Two example scenarios from the human study over prompt sets (Section 4.3 and Appendix I of the revised paper). They are the two scenarios in which the original single prompt and the prompt set differ most in how they order the three models. They illustrate the result and are not its basis; the result over all 15 scenarios is in the table below.
Each example shows the videos of three models for the original prompt and two of its five equivalent rewrites, with the evaluator's physical commonsense (PC) and semantic adherence (SA) scores. A video passes when both are at least 4, as defined by the benchmark. The summary above the videos covers all six prompts. Four raters compared the videos of two models pairwise for three of the six prompts (12 votes per model pair), without knowing which model or prompt produced a video.
| All 15 scenarios, ranking by | Order of the three models | Model pairs ordered as by the raters |
|---|---|---|
| Human raters (share of comparisons won) | Wan2.1-14B 66% > CogVideoX-5B 57% > SDXL+SVD-XT 27% | |
| Prompt set (pass rate) | Wan2.1-14B 41.1% > CogVideoX-5B 36.7% > SDXL+SVD-XT 35.6% | 3 of 3 |
| Original single prompt (pass rate) | SDXL+SVD-XT 46.7% > Wan2.1-14B 40.0% > CogVideoX-5B 33.3% | 1 of 3 |
| Original single prompt (pass or fail) | SDXL+SVD-XT = Wan2.1-14B > CogVideoX-5B | agrees with the raters on 0 of 3 model pairs |
|---|---|---|
| Prompt set (passes out of six prompts) | Wan2.1-14B > CogVideoX-5B > SDXL+SVD-XT (Wan2.1-14B 6, CogVideoX-5B 5, SDXL+SVD-XT 4) | agrees with the raters on 2 of 3 model pairs |
| Human raters | CogVideoX-5B > Wan2.1-14B > SDXL+SVD-XT | CogVideoX-5B over SDXL+SVD-XT: 12 to 0; CogVideoX-5B over Wan2.1-14B: 11 to 1; Wan2.1-14B over SDXL+SVD-XT: 8 to 4 |
| SDXL+SVD-XT | Wan2.1-14B | CogVideoX-5B | |
|---|---|---|---|
| Original prompt shown to raters A leaf blower is pointed at a patch of leaves on a lawn; the leaves are forcefully displaced in a specific direction. | PC 4 · SA 4 · pass | PC 4 · SA 4 · pass | PC 3 · SA 4 · fail |
| Rewrite 3 shown to raters A leaf blower is directed at a patch of leaves on a lawn; the leaves are forcefully displaced in a specific direction. | PC 3 · SA 4 · fail | PC 4 · SA 4 · pass | PC 4 · SA 4 · pass |
| Rewrite 5 A leaf blower is aimed toward a patch of leaves on a lawn; the leaves are forcefully displaced in a specific direction. | PC 5 · SA 3 · fail | PC 4 · SA 4 · pass | PC 4 · SA 4 · pass |
| Original single prompt (pass or fail) | SDXL+SVD-XT = CogVideoX-5B > Wan2.1-14B | agrees with the raters on 0 of 3 model pairs |
|---|---|---|
| Prompt set (passes out of six prompts) | CogVideoX-5B > Wan2.1-14B > SDXL+SVD-XT (CogVideoX-5B 5, Wan2.1-14B 4, SDXL+SVD-XT 3) | agrees with the raters on 2 of 3 model pairs |
| Human raters | Wan2.1-14B > CogVideoX-5B > SDXL+SVD-XT | Wan2.1-14B over CogVideoX-5B: 12 to 0; Wan2.1-14B over SDXL+SVD-XT: 10 to 2; CogVideoX-5B over SDXL+SVD-XT: 8 to 4 |
| SDXL+SVD-XT | Wan2.1-14B | CogVideoX-5B | |
|---|---|---|---|
| Original prompt shown to raters A surfer performs a sharp turn on their board, creating a spray of water. | PC 4 · SA 4 · pass | PC 3 · SA 4 · fail | PC 4 · SA 4 · pass |
| Rewrite 1 shown to raters A surfer makes a sharp turn on their board, creating a spray of water. | PC 4 · SA 3 · fail | PC 4 · SA 4 · pass | PC 4 · SA 4 · pass |
| Rewrite 5 shown to raters A surfer executes a sharp turn on their board, producing a spray of water. | PC 4 · SA 3 · fail | PC 3 · SA 4 · fail | PC 3 · SA 4 · fail |