AI art

📽️🎞️ HOW-TO for "Second Law"

A short breakdown of how the movie was created using a variety of AI tools integrated into a hybrid workflow.

The core is done in Runaway (text-to-video), but there are others that were essential to create a concise work.

Let's dive in 1/10 🧵

2/10

First thing's first, the context. I wrote about my motivation for making the movie in this X post (no way to post the whole video to Farcaster unfortunately), so I suggest you read it 👇

Computational architect. Some structure, mostly chaos. Co-founder {protocell:labs}

https://protocell.xyz/

2/10

First thing's first, the context. I wrote about my motivation for making the movie in this X post (no way to post the whole video to Farcaster unfortunately), so I suggest you read it 👇

https://twitter.com/LukaPiskorec/status/1763878055993389526

3/10

I had this idea of creating boulders suspended mid-air in realistic landscapes. My go-to text-to-image generator is Midjourney, which I've been using since they launched in 2022. I've invited the Midjourney bot to our own discord server so I can keep all generations in one place.

4/10

Example prompt:

basalt rock cliff floating mid-air, weird vibrant pigmentation, ultra-detailed texture, 35mm color photography --style raw --chaos 20 --aspect 2:3 --v 6.0

Raw parameter removes default Midjourney styling. I also used chaos and weird parameters to get more variation.

No post-processing needed!

5/10

Next, we can turn still images into animated scenes using Runway. Set camera motion / rotation using the interface and add additional movement in the scene using motion brush. You will have to tinker with the parameters somewhat to figure it out, although the interface is very intuitive.

6/10

I've used the free version of Runway which has a watermark below, but it's fully functional even in this free tier!

Great, now we have 4 seconds long animated clips. But what about the story? A compilation of moving clips will just be boring to watch.

7/10

I am a big fan of sci-fi and modern physics, so I thought to incorporate these elements into the story. I used ChatGPT to reformat a quote from Ludwig Boltzmann's "The second law of thermodynamics" and added some parts myself. This made it more concise and suitable for narration.

8/10

Next was turning this text into a voice, for which we used ElevenLabs text-to-speech generator. We synthesized an old-fashioned English sci-fi narrator using clips found online.

Tinkering with the parameters and regenerating helps finding the best sounding clip, as well as some post-editing to adjust the pace.

9/10

To get the background sounds and effects, we used Freesound, although you can also generate sound effects in Runway (and soon in ElevenLabs as well). Who needs stock sites when you can generate everything yourself?

The font used in the movie is an amazing retro-digital CC0 font MonoMEK by MEK.txt

10/10

Finally, video clips, background sounds, narration and the titles were edited in Adobe Premiere, but you can use any video editing software you are comfortable with. Only "special" effects added are the VR glitch effect at the start and the end of the video, and the title oscillations.

Done! 🎞️✂️

✨ 

I figured I should compile a list of links to all the tools (in order of appearance):

www.midjourney.com
runwayml.com
chat.openai.com
elevenlabs.io
freesound.org
objkt.com/collections/KT1WfQkAv9HZCin7bgUD1xQnUreXR6HKEhYK
www.adobe.com/products/premiere

Hope you found this thread useful 🎓