Logo image
Back
AI-powered storyboard to 3D scene generation: a comparative analysis of vision-language models for iterative positioning in Unreal Engine
Thesis   Open access

AI-powered storyboard to 3D scene generation: a comparative analysis of vision-language models for iterative positioning in Unreal Engine

Tyler Varacchi
Master of Science (M.S.), Drexel University
Dec 2025
DOI:
https://doi.org/10.17918/00011222
pdf
Varacchi_Tyler_202513.82 MBDownloadView
PDF Open Access Open Access (License Unspecified)

Abstract

3D scene generation Animation pipelines Artificial intelligence Storyboard analysis Unreal Engine Vision-language models
Hand-drawn storyboards enable rapid visualization and critique of narrative concepts in animation and game development. However, translating these 2D sketches into 3D scenes requires considerable manual effort for camera placement, object positioning, and depth perception. Existing research prototypes convert text-based screenplays into preliminary synthetic scenes but do not efficiently translate visual storyboard sketches into 3D layouts using production asset libraries. This thesis presents StoryboardTo3D, a novel Unreal Engine 5.6 plugin that automates the translation of hand-drawn storyboard panels into fully blocked-out 3D scenes. The plugin employs ChatGPT-4o, LLaVA-13B, and Claude Sonnet 4.5 Extended Thinking for visual analysis, implementing a multi-angle capture system using seven viewpoints (one scout camera repositioned to six angles plus one hero camera, captured sequentially) to provide spatial context for AI positioning. Iterative positioning refinement with AI feedback loops progressively adjusts object placement. This research methodology combined Research Through Design with quantitative performance analysis and qualitative assessment. Development progressed through systematic testing of multiple AI backends. The plugin was created as a minimum viable product with an architecture to scale in the future. This research contributes both a minimum viable product that advances previsualization and novel insights into applying modern AI vision capabilities to creative production pipelines. This work also demonstrates the potential of AI-assisted 3D scene generation from sketches, while documenting the associated failures, designs, and architectures for future production.

Metrics

Details

Logo image