A video can be a starting point for a world you can change. Give a coding agent the reference, let it study the frames, and ask it to rebuild the scene as procedural code. Then change the camera, lighting, materials or motion and render it again.
That is the idea behind Video 2 GLSL, the reference-to-shader workflow I have been developing around shader-skills. The interesting part is how much of the process can carry over to the next video.
https://x.com/gokayfem/status/2111907592129671505
Two pieces that work together
shader-skills gives coding agents a library of graphics techniques, working examples and a process for building and checking procedural shaders. Video 2 GLSL adds a reference-driven task: study a clip, derive a specification, construct the scene, then compare the result against the clip.
The skill supports Claude Code, Codex, Cursor, Gemini and other coding agents. Its reusable parts cover shapes, noise, materials, terrain, buildings, water, clouds, lighting and color. You can use them for an original scene, an interactive experiment or a video recreation.
For the standalone Video 2 GLSL target, the deliverable is a single Shadertoy-compatible fragment shader. Geometry is described mathematically, surfaces are procedural, and animation is driven by time. The reference video supplies the visual specification; its frames are not played back inside the shader.
Start by turning the video into a specification
The first step is reliable frame extraction. A video URL or filename gives an agent very little to work with. It needs to inspect what actually appears and how it changes over time.
The reusable prompt asks for a REFERENCE.md before scene code: clip facts, a shot timeline, an object inventory, subject proportions, motion, camera framing, lighting and a sampled palette. It also asks the agent to identify what it cannot reproduce exactly and propose substitutions for approval.
The observations change with the clip. A character shot needs contact points, stride and secondary motion. An architectural shot needs proportions, perspective and camera travel. A water shot needs attention to surface movement, reflections and depth. The same breakdown process gives each one a concrete starting point.
Give the agent tested graphics building blocks
A plausible-looking formula can still produce an unsafe distance field, unstable noise or incorrect color. Those errors become harder to isolate once they sit inside a large scene.
shader-skills currently includes 186 functions across 20 GLSL modules. The agent can search for a technique, call the library function by name, and use the bundler to include its dependencies in one self-contained shader.
The library includes signed distance functions for shapes, operators for combining them, procedural noise and patterns, raymarching helpers, shading, atmosphere and color transforms. Templates show how those pieces work together in complete scenes.
Validation covers compilation in WebGL2 and numerical properties on the GPU: distance-field signs and bounds, noise derivatives, color round trips and other checkable behavior. Rendered examples add a visual check. These checks make the parts more dependable; the final scene still needs inspection.
Build a feedback loop around the renderer
The workflow proceeds through blockout and timing, motion, subject detail, environment detail, light and color, reference matching, and performance. Each milestone produces something visible to judge.
Reference and render should be compared at the same time and framing. Use full frames for composition, crops for object shape and materials, and frame sequences for motion. A single attractive still can hide sliding feet, a camera jump or an interaction that happens too early.
The prompt calls for repeated critique: identify the five biggest differences, fix them, and render again. Measurements support that review. Subject size and horizon position help diagnose camera errors; sampled colors help diagnose grading; motion probes help find discontinuities.
Fast iteration matters here. Low-resolution previews and isolated object views let you correct proportions before committing to expensive renders. Check silhouettes and object proportions before polishing materials and lighting.
Split complex scenes by responsibility
Larger scenes can benefit from several agents working on separate parts: subject, environment, props, motion or lighting. The split should follow the scene's needs.
Shared interfaces keep that work compatible. Agree on scale, coordinates, timing and material identifiers, give each agent ownership of its files, and require evidence from actual renders. An orchestrator can integrate the parts and judge the combined image against the reference.
The reusable lesson is the review structure: someone has to evaluate the whole shot. A detailed building or convincing character can still be wrong for the composition.
What the experiments establish
The work so far includes a floating-island sequence with changing actions, a village sequence with interactions, and an endless street loop. Their requirements differ: event timing, planted contacts, camera transitions and exact repetition each demand different scene logic. Their completion and visual quality also vary.
Across them, the same process remains useful: extract, measure, assemble, render, compare and correct. The repo also contains procedural city, terrain and water examples, while separate interactive experiments explore organic forms and black-hole imagery.
What becomes editable and what remains uncertain
Once a scene exists as code, you can expose controls for camera position, light direction, palette, timing and procedural detail. The street experiment demonstrated alternate camera views from the same constructed set.
Those new views also reveal the limits of reconstruction. A reference only shows part of a world. Hidden surfaces must be inferred, and an AI-generated video may contain changing geometry or inconsistent motion. A convincing match from one angle does not establish a complete, accurate scene.
Performance is another scene-specific result. Detailed raymarched geometry, vegetation, shadows and reflections can be expensive. Measure frame time on the target machine, check different moments in the shot, and report missed targets alongside the remaining visual differences.
Video 2 GLSL is a reusable authoring workflow. Its reliability across arbitrary footage still needs broader evaluation. The current work gives us several concrete experiments and a shared method to improve.
Try it with your own reference
For Claude Code, add the marketplace and install the plugin:
claude plugin marketplace add gokayfem/shader-skills
claude plugin install shader-skills@shader-skills
For other supported agents:
npx skills add gokayfem/shader-skills
Use prompts/video-to-shader-prompt.md, supply a reference clip, an output name and a target machine, and specify what must stay or may be skipped. Review the breakdown before implementation begins.
I want to keep pushing this toward a dependable way to turn visual references into editable procedural scenes. Each reconstruction should leave behind better tools, better tests and techniques the next one can reuse.
Repository: github.com/gokayfem/shader-skills
