Physical interaction
Mundo renders interactions computed by a physics simulator. We compare it with text-driven video models, a single-image physics pipeline and controllable video models that receive the same simulation.
Setup
- Cases. Twenty interactions: two in each of ten scenes. Six scenes come from the novel view synthesis evaluation (five ScanNet++ rooms and the Tanks and Temples meeting room) with their context photographs; four are single images, two real photographs under Creative Commons licences and two generated. The interactions cover drops, pushes and slides, collisions, throws and bounces, and changed gravity, four of each, on medium and large objects present in the image.
- Camera. The target camera is one of the input views and stays fixed, so every method can take part.
- Mundo. The scene program of the reconstruction is simulated in MuJoCo with the chosen initial velocities, masses, friction, restitution and gravity; the simulated proxy video, the context photographs and a prompt with the simulated timeline drive generation. One run, seed 42.
- Baselines. Text-driven methods receive one shared description of the motion written from the simulation, in the manner of PhysGen3D [1]. PhysGen3D receives the same initial velocities and physical parameters. Wan2.2-Fun-Control and Go-with-the-Flow receive the depth and the optical flow of the same simulation. Official code and default settings, one run each, seed 42.
- Metrics. All videos are cropped to a 4:3 canvas, resized to 512×384 and evaluated over their first five seconds. A vision-language judge scores physical realism, photorealism and semantic consistency between 0 and 1 from the input image, the description and ten evenly spaced frames that span the simulated event (until every object has come to rest, plus half a second; between 1.5 and 5 seconds, the same window for every method), listing its evidence first (PhysGen3D protocol, three runs averaged). Following the simulation is measured by tracking every moving object with SAM 2 [8] and comparing its bounding box and displacement with the simulated object, and by the colour error along the simulated pixel correspondences of moving objects (after PSIVG [4]). Parameter control tracks the object and measures its fall time or its slide distance on the floor.
Results
Physical plausibility (20 cases)
| Method | Input | Vision-language judge [9] | VBench [7] | |||
|---|---|---|---|---|---|---|
| Physical realism ↑ | Photorealism ↑ | Semantic consistency ↑ | Motion smoothness ↑ | Imaging quality ↑ | ||
| Mundo | context photos + simulated proxy | 0.667 | 0.916 | 0.728 | 0.9970 | 0.729 |
| MiniMax-H3 [2] | context photos + text | 0.385 | 0.930 | 0.427 | 0.9947 | 0.744 |
| Wan2.2 TI2V-5B [3] | first frame + text | 0.450 | 0.726 | 0.525 | 0.9941 | 0.733 |
| Wan2.2 I2V-A14B [3] | first frame + text | 0.377 | 0.801 | 0.433 | 0.9919 | 0.743 |
| PhysGen3D [1] | first frame + same physical parameters | 0.422 | 0.720 | 0.523 | 0.9971 | 0.693 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.510 | 0.712 | 0.568 | 0.9967 | 0.729 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.408 | 0.663 | 0.477 | 0.9951 | 0.723 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.492 | 0.652 | 0.577 | 0.9933 | 0.707 |
| Input repeated | reference: no motion | 0.000 | 1.000 | 0.000 | 0.9984 | 0.733 |
Faithfulness to the simulation
Methods conditioned on the same MuJoCo run. Each moving object is tracked with SAM 2 from a box prompt on the first frame and compared with the simulated object every eighth of a second: the overlap of their bounding boxes, the error of the tracked displacement, and the colour error along the simulated motion of the object pixels. The motion ratio compares the tracked displacement with the simulated one; a video that does not move scores near zero. The simulated objects are boxes, so the last column, the overlap of the tracked silhouette with the box silhouette, rewards drawing the box itself and is not ranked.
| Method | Input | Following the simulation | Shape of the proxy | |||
|---|---|---|---|---|---|---|
| Box IoU ↑ | Trajectory error (px) ↓ | Corr. pixel MSE ↓ | Motion ratio → 1 | Mask IoU with the box proxy | ||
| Mundo | context photos + simulated proxy | 0.746 | 14.5 | 0.0213 | 0.99 | 0.562 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.619 | 37.4 | 0.0246 | 0.87 | 0.588 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.652 | 34.0 | 0.0225 | 0.91 | 0.643 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.368 | 86.2 | 0.0446 | 0.79 | 0.277 |
| Input repeated | reference: no motion | 0.250 | 101.2 | 0.0431 | 0.01 | 0.169 |
Parameter control
One parameter changes per sweep; the table gives the measured ratio between the two settings.
| Method | Fall time, 0.16 g / 1 g | Slide distance, μ 0.08 / μ 0.3 | Slide distance, 1.2 / 0.9 m/s | Slide distance, 1.5 / 0.9 m/s | Correct direction | Mean error vs. simulation ↓ |
|---|---|---|---|---|---|---|
| Analytic | 2.50 | 3.75 | 1.78 | 2.78 | – | – |
| Simulation, measured | 3.14 | 4.53 | 1.95 | 3.09 | – | – |
| Mundo | 2.50 | 4.56 | 1.13 | 3.67 | 4/4 | 21% |
| MiniMax-H3 | 0.19 | 0.47 | 0.98 | 0.46 | 0/4 | 80% |
| Wan2.2 TI2V-5B | – | 1.54 | 0.89 | 0.97 | 1/4 | 63% |
| Wan2.2 I2V-A14B | 1.50 | 0.87 | 1.00 | 0.75 | 2/4 | 64% |
Drop (4 cases)
| Method | Input | Vision-language judge | ||
|---|---|---|---|---|
| Physical realism ↑ | Photorealism ↑ | Semantic consistency ↑ | ||
| Mundo | context photos + simulated proxy | 0.833 | 0.933 | 0.900 |
| MiniMax-H3 [2] | context photos + text | 0.000 | 1.000 | 0.000 |
| Wan2.2 TI2V-5B [3] | first frame + text | 0.400 | 0.733 | 0.425 |
| Wan2.2 I2V-A14B [3] | first frame + text | 0.133 | 0.917 | 0.175 |
| PhysGen3D [1] | first frame + same physical parameters | 0.292 | 0.667 | 0.433 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.617 | 0.675 | 0.650 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.392 | 0.596 | 0.558 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.533 | 0.592 | 0.583 |
| Input repeated | reference: no motion | 0.000 | 1.000 | 0.000 |
Push and slide (4 cases)
| Method | Input | Vision-language judge | ||
|---|---|---|---|---|
| Physical realism ↑ | Photorealism ↑ | Semantic consistency ↑ | ||
| Mundo | context photos + simulated proxy | 0.992 | 1.000 | 0.983 |
| MiniMax-H3 [2] | context photos + text | 0.542 | 0.967 | 0.567 |
| Wan2.2 TI2V-5B [3] | first frame + text | 0.667 | 1.000 | 0.667 |
| Wan2.2 I2V-A14B [3] | first frame + text | 0.667 | 1.000 | 0.667 |
| PhysGen3D [1] | first frame + same physical parameters | 0.442 | 0.608 | 0.542 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.558 | 0.792 | 0.592 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.375 | 0.683 | 0.392 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.725 | 0.908 | 0.758 |
| Input repeated | reference: no motion | 0.000 | 1.000 | 0.000 |
Collision (4 cases)
| Method | Input | Vision-language judge | ||
|---|---|---|---|---|
| Physical realism ↑ | Photorealism ↑ | Semantic consistency ↑ | ||
| Mundo | context photos + simulated proxy | 0.100 | 0.925 | 0.150 |
| MiniMax-H3 [2] | context photos + text | 0.108 | 0.933 | 0.133 |
| Wan2.2 TI2V-5B [3] | first frame + text | 0.183 | 0.608 | 0.242 |
| Wan2.2 I2V-A14B [3] | first frame + text | 0.242 | 0.717 | 0.300 |
| PhysGen3D [1] | first frame + same physical parameters | 0.300 | 0.637 | 0.358 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.142 | 0.625 | 0.175 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.258 | 0.600 | 0.325 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.408 | 0.750 | 0.525 |
| Input repeated | reference: no motion | 0.000 | 1.000 | 0.000 |
Throw and bounce (4 cases)
| Method | Input | Vision-language judge | ||
|---|---|---|---|---|
| Physical realism ↑ | Photorealism ↑ | Semantic consistency ↑ | ||
| Mundo | context photos + simulated proxy | 0.650 | 0.783 | 0.733 |
| MiniMax-H3 [2] | context photos + text | 0.525 | 0.775 | 0.633 |
| Wan2.2 TI2V-5B [3] | first frame + text | 0.550 | 0.617 | 0.675 |
| Wan2.2 I2V-A14B [3] | first frame + text | 0.342 | 0.642 | 0.475 |
| PhysGen3D [1] | first frame + same physical parameters | 0.525 | 0.804 | 0.675 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.792 | 0.800 | 0.867 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.733 | 0.842 | 0.808 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.575 | 0.667 | 0.658 |
| Input repeated | reference: no motion | 0.000 | 1.000 | 0.000 |
Changed gravity (4 cases)
| Method | Input | Vision-language judge | ||
|---|---|---|---|---|
| Physical realism ↑ | Photorealism ↑ | Semantic consistency ↑ | ||
| Mundo | context photos + simulated proxy | 0.758 | 0.938 | 0.875 |
| MiniMax-H3 [2] | context photos + text | 0.750 | 0.975 | 0.800 |
| Wan2.2 TI2V-5B [3] | first frame + text | 0.450 | 0.671 | 0.617 |
| Wan2.2 I2V-A14B [3] | first frame + text | 0.500 | 0.729 | 0.550 |
| PhysGen3D [1] | first frame + same physical parameters | 0.550 | 0.883 | 0.608 |
| Wan2.2-Fun-Control 5B [5] | first frame + depth of the same simulation | 0.442 | 0.667 | 0.558 |
| Wan2.2-Fun-Control A14B [5] | first frame + depth of the same simulation | 0.283 | 0.596 | 0.300 |
| Go-with-the-Flow [6] | first frame + flow of the same simulation | 0.217 | 0.342 | 0.358 |
| Input repeated | reference: no motion | 0.000 | 1.000 | 0.000 |
Selected comparisons
Changed gravity ScanNet++ scene, multi-view
Under the weak gravity of the Moon, the right black chair is kicked up into the air, floats slowly upward, then sinks back down and lands softly on the carpet.
Drop ScanNet++ scene, multi-view
The open cardboard box on the desk slides off the edge of the desk and drops onto the gray carpet below.
Push and slide ScanNet++ scene, multi-view
The black and blue office chair rolls out from the desk toward the camera across the carpet and slowly comes to a stop.
Drop single generated image
The wooden crate on the concrete ledge slides forward off its edge and drops onto the ground below.
All cases
Collision ScanNet++ scene, multi-view
The black and blue office chair rolls quickly away from the desk, crashes into the wall on the left and bounces back.
Push and slide ScanNet++ scene, multi-view
The black and blue office chair rolls out from the desk toward the camera across the carpet and slowly comes to a stop.
Drop ScanNet++ scene, multi-view
The open cardboard box on the desk slides off the edge of the desk and drops onto the gray carpet below.
Push and slide ScanNet++ scene, multi-view
The black office chair rolls away from the desk on its own, gliding backwards across the gray carpet before it slows down and stops.
Changed gravity ScanNet++ scene, multi-view
Under the weak gravity of the Moon, the right black chair is kicked up into the air, floats slowly upward, then sinks back down and lands softly on the carpet.
Push and slide ScanNet++ scene, multi-view
The left black chair slides backwards away from the round table across the gray carpet and comes to a stop.
Drop ScanNet++ scene, multi-view
The blue and orange backpack slides off the seat of the chair and drops onto the carpet.
Changed gravity ScanNet++ scene, multi-view
Gravity switches off: the chair and the backpack on it slowly lift off the floor and drift upward in zero gravity, turning gently.
Collision ScanNet++ scene, multi-view
The dark blue office chair rolls across the carpet on its own and crashes into the blue waste bin, which is knocked into the black waste bin beside it.
Throw and bounce ScanNet++ scene, multi-view
The blue plastic waste bin is tossed up into the air, flies over the carpet toward the camera, lands, bounces and tumbles to a stop.
Collision single photograph
The traffic cone on the left is flung across the asphalt and crashes into the other traffic cone, knocking it over.
Throw and bounce single photograph
The traffic cone on the right is tossed up into the air, tumbles, bounces on the asphalt and rolls to a stop on its side.
Drop single generated image
The wooden crate on the concrete ledge slides forward off its edge and drops onto the ground below.
Changed gravity single generated image
Under the weak gravity of the Moon, the wooden crate on the concrete ledge is kicked up, floats slowly forward in a long arc, then drifts down and lands softly on the ground.
Push and slide single generated image
The wooden crate slides across the smooth concrete floor toward the left and gradually comes to a stop.
Throw and bounce single generated image
The cardboard box is tossed up into the air, flips over, lands on the concrete floor, bounces and slides to a stop.
Drop single photograph
The top three suitcases of the stack slide off toward the camera and tumble down onto the platform in front of the stack.
Throw and bounce single photograph
The small brown suitcase on top of the stack is tossed up into the air toward the camera, tumbles, lands on the platform in front of the stack and slides to a stop.
Collision Tanks and Temples scene, multi-view
The first banquet chair at the end of the row slides sideways along the table and knocks into the next chairs in the row.
Changed gravity Tanks and Temples scene, multi-view
Gravity briefly reverses: the banquet chair at the end of the row lifts off the floor and rises into the air, then gravity returns and the chair falls back down onto the carpet and topples.
References
- Chen et al. PhysGen3D: Crafting a miniature interactive world from a single image. CVPR 2025.
- MiniMax. MiniMax-H3 reference-to-video model.
- Wan Team. Wan2.2: open and advanced large-scale video generative models. 2025.
- Foo et al. Physical simulator in-the-loop video generation (PSIVG). CVPR 2026. arXiv 2603.06408.
- Alibaba PAI. Wan2.2-Fun-Control. VideoX-Fun, 2025.
- Burgert et al. Go-with-the-Flow: motion-controllable video diffusion models using real-time warped noise. CVPR 2025.
- Huang et al. VBench: comprehensive benchmark suite for video generative models. CVPR 2024.
- Ravi et al. SAM 2: segment anything in images and videos. 2024.
- Qwen Team. Qwen3-VL technical report. 2025.