rohki

Here’s a deep dive into how we make our own AI driven anime

🧵 A Thread :

2/14

like any other animated production, we begin with storyboards and scripts. from there, we enter a preproduction phase, where we conduct R&D to build the necessary tech and pipelines for each shot

Follow the story of Hachi, IO and Mitra.

Episode 1 - Intersection - Out Now.

linktr.ee/rohki

3/14

our process is a hybrid of 2.5D and AI-powered rotoscope animation. we start by creating 3D backgrounds and environments using unreal engine. we block out scenes within these sets, determining camera angles and character movements

4/14

we then record a rough dialogue track for each of the characters to determine timing and how we should act out the emotion

5/14

next, we film live action footage of us on a green screen acting out the scenes, which is then used as mocap and processed through various AI pipelines we've built using comfyUI. depending on the shot, we will use 
ViggleAI here instead of mocap

6/14

at this stage, the AI essentially rotoscopes each frame, re-stylizing it to match our character designs and the art style we've established

7/14

we spend a lot of time designing each of the characters using AI and also traditional non-AI design techniques.  we then train our own AI models on these character designs so that the AI understands exactly what our characters look like. these models are known as LORA’s

8/14

for facial animation, we’ve been using an open source program called live portrait. for episode 2, we’ve been experimenting with “act 1” from RunwayML

9/14

we then composite the stylized characters into the scenes using davinci resolve, adding visual effects and color correction to complete the production

10/14

the final piece of the process, and our true specialty, is creating the sound design and composing the music and score. at its heart, RŌHKI is a music project, so this is where we truly get to have fun and bring everything together

11/14

we also sometimes run our 3D background environments through AI as well to further push the style into a specific look to create a more cohesive look. there are also some other consumer AI tools such as RunwayML, Gen3 and LumaLabsAI that we will use to generate various elements or b-role

12/14

this 12 minute episode was the result of months of hard work from a dedicated team of six, often working 14 hour days. while we've found ways to leverage AI to streamline and accelerate parts of the animation process, it remains incredibly labor intensive

13/14

there's a misconception that using AI means we can simply "prompt" our animation into existence, but that couldn't be further from the truth. even with these advanced tools, creating animation still requires a tremendous amount of effort and skill

14/14 

we’re at a pivotal moment where this technology is just beginning to transform animation, and we're thrilled to be at the forefront, pushing the boundaries of what's possible. what is this tech gonna be capable of in a couple years? can’t wait to find out 🩸

Here it is in 4K: https://youtu.be/QLV0sKfxk9s?si=RG_tXijpM3Lknvfj and collect on @zora at

14/14 

we’re at a pivotal moment where this technology is just beginning to transform animation, and we're thrilled to be at the forefront, pushing the boundaries of what's possible. what is this tech gonna be capable of in a couple years? can’t wait to find out 🩸

Here it is in 4K: https://youtu.be/QLV0sKfxk9s?si=RG_tXijpM3Lknvfj and collect on @zora at https://zora.co/collect/base:0x9cd98193f58eef7f78e89caa6cbbebbaf609ffe1/1?referrer=0x99895960B30A34F8431517c4f5B52A1f4b9854a3

2/14

like any other animated production, we begin with storyboards and scripts. from there, we enter a preproduction phase, where we conduct R&D to build the necessary tech and pipelines for each shot
