VAST: Video As Storyboard from Text

Generating videos from textual descriptions poses significant challenges due to the complexity of temporal coherence, spatial detail, and precise control over the appearance and actions of the primary subjects. In this paper, we propose VAST (Video As Storyboard from Text), a two-stage framework that bridges the gap between text-based inputs and high-quality video synthesis. In the first stage, we employ multimodal large models to generate intermediate representations, including pose, segmentation maps, and depth information, based on the textual input. These intermediate representations act as a storyboard that captures the semantic and structural essence of the described scene. In the second stage, we leverage a diffusion-based generative model conditioned on these representations, alongside textual descriptions and the appearance information of the target subject. This enables the generation of videos with precise control over the subject’s position, movement, and visual appearance. Our method demonstrates significant improvements in generating videos that align closely with textual descriptions while preserving high visual fidelity and temporal consistency.