Meta Movie Gen is a collection of SOTA models designed to deliver advanced video generation and editing capabilities.

🔗: https://ai.meta.com/research/movie-gen/ 

Movie Gen Video: A 30B parameter model capable of creating 16-second videos, trained for both text-to-image and text-to-video tasks. It supports personalized videos using an image and allows for detailed text-based video editing. A 7B parameter upsampler enhances video resolution to 1080p.

Movie Gen Audio: A 13B parameter model that produces 48kHz sound effects, trained on millions of hours of audio. It generates audio based on video content and can extend existing audio in a video.
…

- Utilizes a Transformer architecture with Flow Matching and a compressed latent space for efficient processing.
- Trained on hundreds of millions of videos and images, using a curated dataset for high-quality results.
- Generates videos from text prompts, personalizes videos using an image, allows text-based video edits, and creates synchronized audio.
- Introduces spatial upsampling to achieve 1080p resolution and optimized inference methods.
- technical paper✔️