machinima bit

This commit is contained in:
Brandon Thomas committed 2026-01-31 20:25:29 -05:00
1 parent 67a509dff4
commit 7661bf158a
2 files changed
+16 -3

No files matched your search

@@ -1,10 +1,10 @@
[
{
"slug": "world-models-for-film",
"title": "3D and World Models for Film",
"description": "How to use World Models and 3D Rendering to Get Consistent Film Settings",
"title": "Machinima 2.0: World Models for Consistent AI Filmmaking",
"description": "How to use World Models and 3D Rendering to Get Consistent AI Film Locations and better AI Films.",
"date": "2026-01-30",
"body": "\nTest"
"body": "\n# The \"AI Film Trailer\" Era\n\nYou've seen a lot of these: \n\n@youtube(pGn-1BKo3nY)\n\nFor the past year, it been relatively easy to make montages, \"clip shows\", and trailers\nlike this. While they serve as fun proof-of-concepts and visual eyecandy, you'd be hard \npressed to get a \"good story\" out of this video style.\n\nGood stories have characters that occupy real spaces. They have room to live, breathe, and interact. \n\nYou don't just fast track everyone through a journey from point A to B and have that be the end of things \n(unless you're Game of Thrones). Characters need to spend time in places, to explore interpersonal relationships, \nto overcome challenges. They need places to think, talk, fight and argue, process emotions, grow, and sometimes \ndo absolutely nothing at all.\n\nStories need locations. They're a first class citizen in storytelling.\n\n# Location and Blocking\n\nWe're all filmmakers at ArtCraft. The old school, photons-on-glass kind.\n\nOne of the biggest problems we've had over the past few years with AI video is giving our characters room to occupy. \nSetting them up on set, precisely controlling what's on screen, and maintaining consistency with their location. \nThe majority of AI video right now looks like a series of disconnected clips, which results in \nAI film having that \"film trailer\" montage feel to it. Lots of jumping around, very little \ncontinuity.\n\nWe've figured out how to overcome this problem. And in the spirit of \"Show, Don't Tell\", \nlet me show you what we've discovered. You'll intuit how this technique works immediately:\n\n@youtube(kzvQMdg66Go)\n\nNow let me break down the rationale and the technique...\n\n# Moving the Camera is Important\n\nIn *\"film language\"*, the camera's relationship to the characters conveys meaning. Sometimes we want to give the \nviewer a warm and friendly perspective, or perhaps frame the relationship between two different characters and \nimply a certain power dynamic.\n\nSometimes it's just important to show the location so the viewer can settle in. So the setting feels lived in and \nthe viewer is immersed in it alongside our characters.\n\n# Text is a Coarse-Grained Representation\n\nHere's a simple Nano Banana Pro prompt:\n\n> Sam Altman in a room. He sits at a desk, looking towards a desktop computer on the left. \n> The room is dimly lit. The glow of a computer terminal, sitting in front of him bathes the room in blue light. \n> Behind him, there is a wall with a poster. There’s an AC/DC poster on the left, and a window on the right. \n> Through the window, there’s a view of moonlit trees. \n\nAnd the results: \n\n![Hi Sam!](./images/blog/sam4.png)\n\nWhat happened to the location here? Why are none of these the same? Why are the locations of the poster, \ncomputer, window, and so forth so imprecise? \n\nMoreover, how do we adjust this? Iteratively prompting?\n\nYou already know the answer: text is a poor serialization of the physical world. Text-to-Image is coarse \ngrained. It lacks the ability to precisely convey physical attributes and spatial relationships. You can try, \nbut you'll only get so much precision, and it'll never match the vision in your head.\n\n# Images are Incomplete\n\nThere are multiple ways to move a camera in an image-to-image workflow. You can use a ControlNet, a LoRA, or even \nask an instructive model nicely:\n\n> Rotate the camera behind Sam Altman to show him from behind. Show the desk, computer screen, \n> and wall behind the desk. Show the door to the room. \n\n![Hi Sam!](./images/blog/sam4.2.png)\n\nWhile the lighting and scene elements remain somewhat consistent, we're still ultimately left with the same \nissues we had with text-to-image. We can iterate on editing with words, but it's slow, tedious, and still \nimprecise.\n\n(Why does the desk block the door? What is it even trying to do with that poster?)\n\n# 3D to the Rescue!\n\nOne of the best tools for consistency is 3D. You can position your characters and props in a \"3D set\" exactly as you want them, \nand then move the \"camera\" to any angle, maintaining strong consistency throughout.\n\n![Hi Sam!](./images/blog/sam_previz.png)\n\nPick a \"smart\" model like GPT Image 1.5 or Nano Banana Pro to convert this previz into a photorealistic render (or any style - anime, sci-fi, whatever): \n\n> Suspense movie - live action - night time. Make a photorealistic picture of Sam Altman sitting at a fancy office desk. \n> Use the previz scene as the layout and posing of the shot. The camera and pose of Sam should match this shot. \n> Nighttime shot, moonlit glow. \n\n![Hi Sam!](./images/blog/sam_gpt_image_1.png)\n\nIf desired, you can also attach additional reference images for character designs, wardrobe choices, etc.\n\nAfter Nano Banana Pro lighting adjustments and upscaling (and showing it the previz again to restore the missing desk pad), we get this:\n\n> Make this look more photorealistic. Suspenseful Line truncated
},
{
"slug": "welcome",
@@ -205,6 +205,19 @@ We're still being largely intentional. If there's something we want to see, we n
We're painting with pictures. We're not prompters, but rather auteurs of crafting.
# Machinima 2.0
Do you remember "Machinima"? It's a niche, indie genre of filmmaking that was popular in the early 2000's. Creators would pilot video game characters, carefully controlling for the camera, character movements, and on-screen details in order to tell stories. They would then dub over the footage to breathe life into the narrative.
Popular Machinima series include "Red vs. Blue", from Rooster Teeth Productions, which was filmed using the Halo video game series. Other notable Machinima include "The Strangerhood" (which used The Sims) and "Skibidi Toilet" (which leverages Source Filmmaker).
I mention this obscure artform because as a former Machinima creator myself, it feels like Image-to-Video is the new Machinima. There's the character posting, keyframing, and dubbing, of course, but there's also this notion of using pixels created by someone else's engine. And it's very much in the spirit of machinima to carve new creativity out of these engines by operating at a higher level.
Image-to-image, 3D, and World Models make an entirely new form of Machinima possible:
@youtube(tAAiiKteM-U)
# ArtCraft: Free and Open 3D Filmmaking
ArtCraft is a crafting engine and is available in its entirety on [Github (please star us!)](https://github.com/storytold/artcraft), or try it today on [Mac or Windows](/download).