mirror of
https://github.com/storytold/artcraft.git
synced 2026-11-04 09:28:11 +00:00
wip
This commit is contained in:
Brandon Thomas committed
1 parent
7a3135c020
commit
f423f1ccb8
1 file changed
+45
-55
@@ -10,13 +10,17 @@ You've seen a lot of these:
|
||||
|
||||
@youtube(pGn-1BKo3nY)
|
||||
|
||||
For the past year, it been relatively easy to make montages, "clip shows", and trailers,
|
||||
For the past year, it been relatively easy to make montages, "clip shows", and trailers
|
||||
like this.
|
||||
|
||||
Real stories have characters that need space to live, breathe, and interact. You don't just fast track
|
||||
everyone through a journey from point A to B and have that be the end of things [1]. Characters need to spend
|
||||
time in places, explore interpersonal relationships. They need to take time to think, talk, fight and argue,
|
||||
process emotions, and sometimes do absolutely nothing at all.
|
||||
Good stories have characters that occupy real spaces. They have room to live, breathe, and interact.
|
||||
|
||||
You don't just fast track everyone through a journey from point A to B and have that be the end of things [1].
|
||||
Characters need to spend time in places, to explore interpersonal relationships, to overcome challenges.
|
||||
They need places to think, talk, fight and argue, process emotions, grow, and sometimes do absolutely nothing
|
||||
at all.
|
||||
|
||||
Stories need locations. They're a first class citizen in storytelling.
|
||||
|
||||
# Location and Blocking
|
||||
|
||||
@@ -25,68 +29,60 @@ We're all filmmakers at ArtCraft. The old school, photons-on-glass kind.
|
||||
One of the biggest problems we've had over the past few years with AI video is giving our characters room to occupy.
|
||||
Setting them up on set, precisely controlling what's on screen, and maintaining consistency with their location.
|
||||
The majority of AI video right now looks like a series of disconnected clips, which results in
|
||||
AI film having a sort of "film trailer" feel to it. Lots of jumping around, very little
|
||||
AI film having that "film trailer" montage feel to it. Lots of jumping around, very little
|
||||
continuity.
|
||||
|
||||
We've figured out how to overcome this problem. And in the spirit of "Show, Don't Tell",
|
||||
let me show you what we've discovered. You'll intuit what's up immediately:
|
||||
let me show you what we've discovered. You'll intuit how this technique works immediately:
|
||||
|
||||
@youtube(wJCJYdGdpHg)
|
||||
|
||||
# Text is a Coarse Grained Medium
|
||||
# Moving the Camera is Important
|
||||
|
||||
In *"film language"*, the camera's relationship to the characters conveys meaning. Sometimes we want to give the
|
||||
viewer a warm and friendly perspective, or perhaps frame the relationship between two different characters and
|
||||
imply a certain power dynamic.
|
||||
|
||||
Sometimes it's just important to show the location so the viewer can settle in. So the setting feels lived in and
|
||||
the viewer is immersed in it alongside our characters.
|
||||
|
||||
# Text is a Coarse Grained Representation
|
||||
|
||||
Here's a simple Nano Banana Pro prompt:
|
||||
|
||||
|
||||
|
||||
> Sam Altman in a room. He sits at a desk, looking towards a desktop computer on the left.
|
||||
> The room is dimly lit. The glow of a computer terminal, sitting in front of him bathes the room in blue light.
|
||||
> Behind him, there is a wall with a poster. There’s an AC/DC poster on the left, and a window on the right.
|
||||
> Through the window, there’s a view of moonlit trees.
|
||||
|
||||
What happened to the location here? Why are none of these the same? Why do the locations of the poster,
|
||||
computer, window, and so forth keep changing?
|
||||
And the results:
|
||||
|
||||
|
||||
|
||||
What happened to the location here? Why are none of these the same? Why are the locations of the poster,
|
||||
computer, window, and so forth so imprecise?
|
||||
|
||||
Moreover, how do we adjust this? Iteratively prompting?
|
||||
|
||||
You already know the answer: text is a poor serialization of the physical world. Text-to-Image is coarse
|
||||
grained. It lacks the ability to convey precise physical attributes and spatial relationships.
|
||||
|
||||
This isn't an accident. Most Image-to-Video (I2V) inputs have been created from text sources.
|
||||
Outside of proper nouns, text has limited ability to convey notions of location. It doesn't
|
||||
store any physical attributes, it can't track relative positions between objects, and it's
|
||||
obtuse to describe spatial relationships and appearance.
|
||||
|
||||
In *"film language"*, even the camera's relationship to the characters conveys meaning.
|
||||
|
||||
- Example image
|
||||
|
||||
Most people have been stuck using this.
|
||||
|
||||
Even a thousand words couldn't make these two images match.
|
||||
|
||||
-----
|
||||
They need pauses.
|
||||
They need to occupy spaces, have relationships to other characters, walk around,
|
||||
sit and think. You can't do that with disconnected clips in series.
|
||||
|
||||
It's hard to tell a story like this if we can't even understand where the characters are.
|
||||
|
||||
But that's not telling a story. That's not having your actors live, breathe,
|
||||
and occupy a space.
|
||||
|
||||
-----
|
||||
grained. It lacks the ability to precisely convey physical attributes and spatial relationships. You can try,
|
||||
but you'll only get so much precision, and it'll never match the vision in your head.
|
||||
|
||||
# Images are Incomplete
|
||||
|
||||
Even image references and I2V struggle with location consistency. Images convey no information about what's just outside the frame,
|
||||
so moving the camera results in hallucinations and inconsistencies.
|
||||
There are multiple ways to move a camera in an image-to-image workflow. You can use a ControlNet, a LoRA, or even
|
||||
ask an instructive model nicely:
|
||||
|
||||
- Example image
|
||||
> Rotate the camera behind Sam Altman to show him from behind. Show the desk, computer screen,
|
||||
> and wall behind the desk. Show the door to the room.
|
||||
|
||||
---
|
||||
|
||||
|
||||
While the lighting and scene elements remain somewhat consistent, we're still ultimately left with the same
|
||||
issues we had with text-to-image. We can iterate on editing with words, but it's slow, tedious, and still
|
||||
imprecise.
|
||||
|
||||
(Why does the desk block the door? What is it even trying to do with that poster?)
|
||||
|
||||
# 3D to the Rescue
|
||||
|
||||
@@ -96,30 +92,24 @@ One of the best tools for consistency is 3D. You can position your characters an
|
||||
|
||||
...
|
||||
|
||||
# World Models Make This Easy
|
||||
|
||||
...
|
||||
|
||||
# Object Generation for Props
|
||||
|
||||
...
|
||||
# World Models Make This Easy
|
||||
|
||||
# Putting it all Together
|
||||
|
||||
...
|
||||
|
||||
# ArtCraft: Free and Open 3D Filmmaking
|
||||
|
||||
ArtCraft in its entirety is available on [Github (please star us!)](https://github.com/storytold/artcraft),
|
||||
and we have [downloads for Windows and Mac](/download), with Linux support coming soon.
|
||||
ArtCraft in its entirety is available on [Github (please star us!)](https://github.com/storytold/artcraft)
|
||||
|
||||
We have [downloads for Windows and Mac](/download), with Linux support coming soon.
|
||||
|
||||
You can pay us for image and video compute, or you can bring your own keys and subscriptions
|
||||
without needing to pay us a dime. We offer the ability to log in with MidJourney, Grok,
|
||||
OpenAI/Sora, and other providers. We'll be adding FAL, Replicate, and Google Gemini shortly,
|
||||
and local GPU support is coming soon (subscribe to stay updated).
|
||||
OpenAI/Sora, and several other providers. We'll be adding FAL, Replicate, and Google Gemini
|
||||
shortly, and local GPU support is coming soon (subscribe to stay updated).
|
||||
|
||||
p.s. did you catch the typo(s)? The pixels may be generated, but the text was artisinally hand-crafted.
|
||||
|
||||
— Brandon Thomas, that one silly engineer in your drama club making you Meisner to Tolkien.
|
||||
— Brandon Thomas
|
||||
|
||||
[1] Unless you're Game of Thrones...
|
||||
Reference in new issue
Block a user