Drop an mp4 in the terminal: multimodal coding, demonstrated
The most quietly radical launch demo wasn’t about speed or parallelism. It was a video file becoming a website.
The demo
In Meta’s launch materials, a user drops a fly-through video of a vacation home into the terminal as an mp4 file. Muse Code interprets the video — the rooms, the light, the setting — and produces a visually rich marketing and booking page for the property. No brief, no copywriting pass, no asset list. The video is the specification.
Why this is different from vision support
Plenty of models accept images; the cookbook’s vision-input recipe covers exactly that, and screenshot-to-bugfix is already a familiar workflow. What the demo shows is something further: unstructured, non-code media used as the primary input to a full engineering task, inside the agent loop. The model has to extract intent (this is a rental property that needs to sell itself), derive content (what the pages should say and show), and then execute a normal multi-page build — planning, code, validation — from that understanding.
That collapses the usual pipeline. The traditional path from “here’s a video of the house” to “here’s the booking site” runs through a copywriter, a designer, and a developer, each translating the previous step’s output. A multimodal agent runs the translation chain internally.
Where it fits in real workflows
The vacation-home page is a marketing demo, but the pattern generalizes to inputs engineers actually have: a screen recording of a bug that’s hard to describe in words; a whiteboard photo of an architecture sketch; a competitor walkthrough to study; a design mock exported as an image. Anywhere the truest description of the task is visual rather than verbal, typing a prose approximation into a prompt is lossy compression. Handing over the artifact itself isn’t.
The cookbook’s perception recipes sketch the same direction at smaller scale — chart analysis extracts structured data from images, perception grounding pins interactive annotations to pixel locations in a photo. The mp4 demo is those primitives, composed and pushed to their conclusion.
The honest caveats
A launch demo is a best case. Video interpretation at this depth burns serious tokens — worth remembering that input pricing applies to media too — and “the video is the spec” works best when the deliverable’s shape is conventional, like a property page. But as a statement of direction it’s unambiguous: the input side of coding agents is widening from text to whatever artifact best carries your intent. Try the vision-input recipe in the cookbook to see the primitive; the quickstart gets you to a terminal where you can drop the file.