A published benchmark scores video generation on sixteen axes

The shot

The situation this note examines is a measurement problem, not a shot: how to score the output of a text-to-video model. On 29 November 2023, researchers from Nanyang Technological University, Shanghai Artificial Intelligence Laboratory, the Chinese University of Hong Kong and Nanjing University posted the VBench paper to arXiv, later presented as a CVPR 2024 Highlight. The suite, hosted alongside its project page and code repository, was built specifically because a single aggregate quality score cannot tell a producer whether a model's weakness is in subject continuity, motion, or prompt fidelity.

What the documents show

The paper states that VBench decomposes 'video generation quality' into sixteen dimensions, split into a Quality Score built from subject consistency, background consistency, temporal flickering, motion smoothness, aesthetic quality, imaging quality and dynamic degree, and a Semantic Score built from object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style and overall consistency, as the repository's scoring documentation confirms. Each dimension carries its own tailored prompts and an automatic evaluation method, validated against human preference annotations the authors collected. This is an independent test as scoped here: a repeatable method, named model versions, and comparative results, not a vendor's claim. The paper is explicit about scope. Its limitations section states VBench 'currently does not assess safety and equality dimensions,' and notes that the number of open-sourced text-to-video models available for evaluation was still limited at publication, with image-to-video and other tasks left as future work.

The workflow

As the repository documents it, a team evaluating a model renders the standard VBench prompt list, runs the per-dimension evaluation scripts, and packages the results for the normalization step that turns raw per-dimension numbers into a Quality Score and Semantic Score, which are then weighted into a single Total Score for submission to the public leaderboard. A production team can use the same per-dimension breakdown without submitting anywhere, reading the radar-style profile to see whether a candidate model's actual weakness is temporal flickering or, separately, spatial relationship accuracy, rather than trusting a single number.

What the tool does not change

A sixteen-dimension score is still a proxy measured against a fixed prompt suite, not a judgment of whether a shot serves a scene. The paper's own acknowledgment that it does not score safety or equality leaves that review entirely to the humans running a production. Whether a model's 'aesthetic quality' dimension matches a director's intended look, and whether a generated take is usable in a cut, remain calls the benchmark does not make.

• Which of the sixteen dimensions actually predicts whether a shot will cut into the sequence you need?
• Does the benchmark's standard prompt suite resemble the kind of shots your production actually requires?
• What has the model done on dimensions, like safety, that the authors say the suite does not score at all?

VBench is useful precisely because it publishes what it measures and, in its own limitations section, what it does not; treating its Total Score as a verdict on a model's fitness for a specific production would extend the paper's own claims further than the authors do.