Visual Prompt Engineering for Video Models

Robert Geirhos*, Yuxuan Li*, Thaddäus Wiedemer*, Neha Kalibhat, Zi Wang, Mani Malek, Oyvind Tafjord, Kevin Swersky, Been Kim, Priyank Jaini*
Google DeepMind Logo
*Core team
PDF arXiv LogoarXiv Podcast
TL;DR: Just as text-based prompt engineering systematically improves language model performance, visual prompt engineering (VIPE) serves as a simple, compute-efficient approach to elicit better visual reasoning performance from video models by transforming the input task image (e.g., making abstract sketches photorealistic).
Original Solution Attempt
After Visual Prompt Engineering
Physics VPCT Original
Conjunctive Search Original
Connect Dots Original
Sort 3 Numbers Original
Maze Original
RushHour Original
Physics VPCT
Conjunctive Search
Connect the Dots
Sort 3 Numbers
Maze
RushHour
Physics VPCT VIPE
Conjunctive Search VIPE
Connect Dots VIPE
Sort 3 Numbers VIPE
Maze VIPE
RushHour VIPE
Physics VPCT
Conjunctive Search
Connect the Dots
Sort 3 Numbers
Maze
RushHour

Abstract

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models.

Podcast

Listen to the NotebookLM-generated podcast overview of our paper:

Task Examples

Below, we show examples for each task and split. We show a solution attempt on the original image (which often fails), a solution attempt on the most successful VIPE variant for the task and split, and other VIPE variants obtained by freeform VIPE or atomic concept edit (ACE) VIPE.

Physics VPCT

Predict which bucket the ball will land in after rolling down the ramps.

Text Prompt: "The ball moves down in a physically plausible way, sliding down the obstacles, and ends up in one of the three containers at the bottom. [...]"

Original Solution Attempt Failed
Physics Original
Best VIPE Variant Industrial factory chutes
Success
Physics VIPE

Other Variants

Variant
Ancient stone ruins
Variant
Baking kitchen prep
Variant
Billiard trick shot
Variant
Candy sweets landscape
Variant
Claymation stop motion
Variant
Construction site steel
Variant
Garage rube goldberg
Variant
Macro forest nature
Variant
Marble run toy
Variant
Mini golf course
Variant
Neon cyberpunk arcade
Variant
Office desk supplies
Variant
Retro pachinko machine
Variant
Science chemistry lab
Variant
Steampunk brass mechanism
Variant
Water park slides
Variant
Winter ice slopes
Variant
Wooden physics puzzle
Variant
Zen garden bamboo

Conjunctive Search

Locate the target object defined by a conjunction of features (e.g., color and shape).

Text Prompt: "The two blue balls begin to glow."

Original Solution Attempt Failed
Conjunctive Original
Best VIPE Variant Globe lightbulbs case
Success
Conjunctive VIPE

Other Variants (Freeform)

Variant
Arcade joystick panel
Variant
Billiard table felt
Variant
Candy assortment countertop
Variant
Deep sea bioluminescence
Variant
Dj mixer interface
Variant
Fantasy crystal orbs
Variant
Gemstone jewelry velvet
Variant
Gym equipment wood floor
Variant
Holiday ornaments snow
Variant
Lab plasma balls
Variant
Macro marbles and dice
Variant
Neon mini golf
Variant
Night pool led floats
Variant
Photorealistic 3d shapes
Variant
Pinball arcade bumpers
Variant
Sci fi energy cores
Variant
Synthwave neon grid
Variant
Twilight led bocce
Variant
Vintage toys hardwood

Other Variants (ACE)

Variant
REPLACE flat color of the two blue circles with a 3D glossy material
Variant
REPLACE flat rendering of blue circles with 3D glossy volumetric material
Variant
SET border outline of the two blue balls as a thick white stroke
Variant
REPLACE flat color of blue balls with 3D glossy material
Variant
REPLACE flat material of blue circles with glossy 3D rendering
Variant
SET material of the two blue circles as glossy 3D glass
Variant
SET rendering material of the blue circles as 3D glossy
Variant
SET material of the blue balls as glossy 3D spheres
Variant
SET materiality of the two blue circles as 3D glossy spheres
Variant
REPLACE size of the blue circles with a significantly larger scale
Variant
SET size of the blue circles as significantly enlarged
Variant
SET material of the blue circles as glossy 3D
Variant
SET color saturation of non-target shapes as completely desaturated grayscale
Variant
REMOVE color saturation from all non-target objects
Variant
REPLACE colors of all red circles and blue squares with grayscale
Variant
SET material of the blue circles as glossy 3D spheres
Variant
SET the material of the blue circles as glossy 3D spheres
Variant
REPLACE flat shading of the two blue circles with a glossy 3D material
Variant
SET material of the blue circles as glossy 3D

Connect the Dots

Connect same-colored dots.

Text Prompt: "Connect each pair of same-colored circles with a line."

Original Solution Attempt Failed
Connect Dots Original
Best VIPE Variant Thick black stroke
Success
Connect Dots VIPE

Other Variants (Freeform)

Variant
Aerial flower beds
Variant
Building block baseplate
Variant
Chalk street art
Variant
Corkboard pushpins
Variant
Embroidery hoop
Variant
Frosted cookies baking sheet
Variant
Glowing neon nodes
Variant
Gym floor vinyl spots
Variant
Mobile puzzle ui
Variant
Pcb terminal nodes
Variant
Retro arcade crt
Variant
Round stickers cardboard
Variant
Sewing buttons fabric
Variant
Synth patchbay jacks
Variant
Wax seals parchment
Variant
Wet acrylic dollops
Variant
Whiteboard magnets
Variant
Wooden board game tokens
Variant
Yarn pegboard
Variant
Zen garden stones

Other Variants (ACE)

Variant
REPLACE flat 2D shapes with volumetric 3D spheres
Variant
REPLACE standard hues with highly saturated neon colors
Variant
SET border of each circle as thick outline
Variant
REPLACE solid white background with dark black background
Variant
SET outer border of the circles as a thick black stroke
Variant
REPLACE current background color with a high-contrast dark gray
Variant
SET internal content of the circles as matching text labels
Variant
REPLACE bright white background with a dark slate background
Variant
SET border of each circle as a thick black stroke
Variant
SET border of each circle as a thick white outline
Variant
SET color saturation of the circles as ultra-vibrant
Variant
SET color saturation of the circles as maximally saturated neon
Variant
SET size of the circles as significantly larger
Variant
REPLACE current rendering style with glossy 3D spheres
Variant
REPLACE current background color with solid black
Variant
REPLACE current circle sizes with 40% enlarged versions
Variant
SET color saturation of the circles as highly increased
Variant
SET size of all circles as 30 percent larger
Variant
REPLACE plain white background with a solid black background

Sort 3 Numbers

Sort three numbers in ascending order.

Text Prompt: "The numbers pop and disappear one at a time, in numeric order, starting from the smallest one."

Original Solution Attempt Failed
Sort Original
Best VIPE Variant Retro arcade sprites
Success
Sort VIPE

Other Variants (Freeform)

Variant
Ar notification badges
Variant
Bubblegum bubbles
Variant
Chalk shooting targets
Variant
Dry ice smoke bubbles
Variant
Giant bubble wrap
Variant
Glass lightbulbs
Variant
Glossy candy blobs
Variant
Glowing fireworks
Variant
Holographic data nodes
Variant
Inflatable pool floats
Variant
Latte foam bubbles
Variant
Numbered balloons
Variant
Papier mache pinatas
Variant
Pop art comic splats
Variant
Popping boba pearls
Variant
Soap bubbles
Variant
Squishy jelly cubes
Variant
Vr aim trainer targets
Variant
Water balloons

Other Variants (ACE)

Variant
SET rendering of the numbers as 3D embossed
Variant
REPLACE current plain rendering with a neon sign aesthetic
Variant
REPLACE current background color with a high-contrast dark solid
Variant
REPLACE plain white background with a solid high-contrast dark color
Variant
REPLACE sterile environment with a cohesive chalkboard texture
Variant
SET color of each number as a unique vivid hue
Variant
SET color of each number as a distinct saturated hue
Variant
REPLACE 2D flat typography with 3D volumetric geometry
Variant
REPLACE current flat typography with 3D metallic rendered text
Variant
REPLACE uniform black text color with distinct saturated colors
Variant
SET spatial boundaries of each number as distinct geometric boxes
Variant
SET overall scale of the numbers as uniformly enlarged
Variant
REPLACE white background with a dark high-contrast background
Variant
REPLACE current font weight with an ultra-bold heavy weight
Variant
REPLACE current uniform black text color with distinct vibrant colors
Variant
SET rendering of the numbers as subtle volumetric 3D geometry
Variant
REPLACE current white background with a dark solid color
Variant
SET surface texture of the scene as a green chalkboard
Variant
SET dimensionality of the numbers as subtly embossed 3D

Maze Solving

Find a path from start to finish in various maze sizes.

Text Prompt: "Create a 2D animation based on the provided image of a maze. The red circle slides smoothly along the white path, stopping perfectly on the green circle. [...]"

Split: 5×5 Maze

Original Solution Attempt Failed
Maze 5x5 Original
Best VIPE Variant Classic 2d arcade game
Success
Maze 5x5 VIPE
Other Variants
Variant
Algorithmic pathfinding visualization
Variant
Animated transit map infographic
Variant
Classic flash web animation
Variant
Dark mode gps tracking ui
Variant
Dark mode ui motion
Variant
Digital circuit routing
Variant
Digital rhythm game interface
Variant
Flat 2d physics sandbox
Variant
Flat felt stop motion
Variant
Glossy mobile puzzle interface
Variant
Industrial cam simulation
Variant
Layered 2d motion graphics
Variant
Matte papercraft 3d
Variant
Neon motion graphics
Variant
Precise vector math animation
Variant
Robot vacuum app map
Variant
Sleek 2d vector puzzle
Variant
Smooth arcade tabletop
Variant
Vintage celluloid animation

Split: 7×7 Maze

Original Solution Attempt Failed
Maze 7x7 Original
Best VIPE Variant Classic 2d arcade game
Success
Maze 7x7 VIPE
Other Variants
Variant
Algorithmic pathfinding visualization
Variant
Animated transit map infographic
Variant
Classic flash web animation
Variant
Dark mode gps tracking ui
Variant
Dark mode ui motion
Variant
Digital circuit routing
Variant
Digital rhythm game interface
Variant
Flat 2d physics sandbox
Variant
Flat felt stop motion
Variant
Glossy mobile puzzle interface
Variant
Industrial cam simulation
Variant
Layered 2d motion graphics
Variant
Matte papercraft 3d
Variant
Neon motion graphics
Variant
Precise vector math animation
Variant
Robot vacuum app map
Variant
Sleek 2d vector puzzle
Variant
Smooth arcade tabletop
Variant
Vintage celluloid animation

Split: 9×9 Maze

Original Solution Attempt Failed
Maze 9x9 Original
Best VIPE Variant Neon motion graphics
Success
Maze 9x9 VIPE
Other Variants
Variant
Algorithmic pathfinding visualization
Variant
Animated transit map infographic
Variant
Classic 2d arcade game
Variant
Classic flash web animation
Variant
Dark mode gps tracking ui
Variant
Dark mode ui motion
Variant
Digital circuit routing
Variant
Digital rhythm game interface
Variant
Flat 2d physics sandbox
Variant
Flat felt stop motion
Variant
Glossy mobile puzzle interface
Variant
Industrial cam simulation
Variant
Layered 2d motion graphics
Variant
Matte papercraft 3d
Variant
Precise vector math animation
Variant
Robot vacuum app map
Variant
Sleek 2d vector puzzle
Variant
Smooth arcade tabletop
Variant
Vintage celluloid animation

Split: Irregular Maze

Original Solution Attempt Failed
Maze Irregular Original
Best VIPE Variant Digital rhythm game interface
Success
Maze Irregular VIPE
Other Variants
Variant
Algorithmic pathfinding visualization
Variant
Animated transit map infographic
Variant
Classic 2d arcade game
Variant
Classic flash web animation
Variant
Dark mode gps tracking ui
Variant
Dark mode ui motion
Variant
Digital circuit routing
Variant
Flat 2d physics sandbox
Variant
Flat felt stop motion
Variant
Glossy mobile puzzle interface
Variant
Industrial cam simulation
Variant
Layered 2d motion graphics
Variant
Matte papercraft 3d
Variant
Neon motion graphics
Variant
Precise vector math animation
Variant
Robot vacuum app map
Variant
Sleek 2d vector puzzle
Variant
Smooth arcade tabletop
Variant
Vintage celluloid animation

RushHour

Slide blocks to allow the red car to exit the grid.

Text Prompt: "[...] Plan the minimal sequence of moves needed to free the red car and allow it to exit the parking lot. Output: A video demonstrating the full solution to the puzzle, one move at a time."

Split: Level 2

Original Solution Attempt Failed
RushHour L2 Original
Best VIPE Variant Anodized metal puzzle box
Success
RushHour L2 VIPE
Other Variants
Variant
3d game engine viewport
Variant
Automated mechanical parking garage
Variant
Carved wooden puzzle
Variant
Casual mobile game 3d
Variant
Drone asphalt parking lot
Variant
Macro pcb electronics
Variant
Magnetic whiteboard diagram
Variant
Minimalist 3d diorama
Variant
Neon holographic tactical map
Variant
Port shipping container yard
Variant
Retro pixel art arcade
Variant
Sci fi maglev depot
Variant
Screen recording web ui
Variant
Stop motion felt craft
Variant
Studded plastic brick toy
Variant
Tournament billiard table
Variant
Toy slot car diorama
Variant
Translucent ice blocks on frost
Variant
Vintage cardboard board game

Split: Level 3

Original Solution Attempt Failed
RushHour L3 Original
Best VIPE Variant Sci fi maglev depot
Success
RushHour L3 VIPE
Other Variants
Variant
3d game engine viewport
Variant
Anodized metal puzzle box
Variant
Automated mechanical parking garage
Variant
Carved wooden puzzle
Variant
Casual mobile game 3d
Variant
Drone asphalt parking lot
Variant
Macro pcb electronics
Variant
Magnetic whiteboard diagram
Variant
Minimalist 3d diorama
Variant
Neon holographic tactical map
Variant
Port shipping container yard
Variant
Retro pixel art arcade
Variant
Screen recording web ui
Variant
Stop motion felt craft
Variant
Studded plastic brick toy
Variant
Tournament billiard table
Variant
Toy slot car diorama
Variant
Translucent ice blocks on frost
Variant
Vintage cardboard board game

Citation

@article{geirhos2026visual,
  title={Visual prompt engineering for video models},
  author={Geirhos, Robert and Li, Yuxuan and Wiedemer, Thadd{\"a}us and Kalibhat, Neha and Wang, Zi and Malek, Mani and Tafjord, {\O}yvind and Swersky, Kevin and Kim, Been and Jaini, Priyank},
  journal={arXiv preprint arXiv:2607.XXXXX},
  year={2026}
}