Fugu-MT: arxivの論文翻訳

Summary

The theme centers on preserving object identity, controlling multi-object motion, and evaluating spatial consistency rather than visual quality alone. This week's evidence extends these concerns to multi-shot staging and interactive world models, where generated events must follow explicit rules and persistent hidden state, not merely respond visually to actions.

Situation

Representative Introductions identify object-level consistency as a central constraint on useful generation. ObjectMover repurposes video diffusion priors for single-image object movement, targeting identity preservation, perspective changes, and consistent lighting and shadows, and addresses scarce paired data with game-engine synthetic data plus auxiliary real-video training. LayerT2V addresses colliding multi-object trajectories by generating video in layers, with later layers occluding earlier ones. MVGBench argues that comparing generated views independently against a single ground truth misses 3D inconsistency. It proposes a ground-truth-free 3D consistency metric and reports a trade-off between 3D consistency and image quality, a persistent synthetic-to-real gap, and limited robustness to input perturbations.

Current-week evidence in "State World Models: Execute the Rules, Generate the Pixels" (2610.26811v1) extends the consistency problem to interaction outcomes: identical views and actions can require different results depending on hidden state. Its Stateful Harness separates rule execution from rendering by a frozen video model, using 3D scene previews for spatial guidance and a vision language model to update state from generated clips. Together, this work frames controllability as more than trajectory adherence. Generated content must also preserve spatial relationships and faithfully depict outcomes required by persistent world state.

Infographic (English)

Controllable and Consistent Video Generation situation infographic

Progress

AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark <See Details on Fugu-MT>

AnyGameRenderer applies entity-consistent generative rendering to footage across games. It changes visual appearance while keeping layout, camera, entities, and timing fixed.

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds <See Details on Fugu-MT>

Matrix-Game 4.0 builds real-time playable game worlds on video models. A lightweight game engine maintains world state and executes interaction rules alongside video generation.

StreamEval: Evaluating Interactive Streaming Video Generation Models <See Details on Fugu-MT>

StreamEval introduces a unified benchmark for interactive streaming video generation across five tasks. It finds that high visual quality and identity preservation do not guarantee reliable request execution or persistent-state consistency.

Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation <See Details on Fugu-MT>

Mise-en-Video brings 3D-staged, subject-level control to multi-shot video generation. It reports gains over baselines in video quality, cross-shot consistency, and subject-level controllability.

State World Models: Execute the Rules, Generate the Pixels <See Details on Fugu-MT>

Stateful Harness separates rule-governed world evolution from frozen video rendering. It carries persistent state across turns and shows large gains in rule-compliant outcomes on counterfactual test pairs.

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements <See Details on Fugu-MT>

PVBench quantifies physical plausibility of generated video across 46 evaluation dimensions. It uses object-level measurements such as 6D pose, metric scale, and kinematics instead of appearance or binary semantic judgments.

Outlook

Outlook Summary

Generation is likely to gain tighter object and scene control as it moves toward multi-shot sequences and persistent interactive settings. ObjectMover proposes pose-conditioned training and crossing-object examples, while distillation aims to reduce inference cost. LayerT2V plans newer backbones, but matching trajectory controls to background meaning remains unresolved. Explicit 3D staging and engine-managed state offer complementary approaches. Evaluation should increasingly test spatial, physical, and state consistency rather than appearance alone. Priorities include reliable scoring from few views and broader scene conditions. PVBench proposes testing evaluator components, improving tracking, and using physics-engine reference data to distinguish faithful control and lasting consequences from merely coherent-looking output.

Infographic (English)

Controllable and Consistent Video Generation outlook infographic

Three-Year Movement

Over three years, this path could move video generation toward workflows where an explicit scene record guides rendering and helps reviewers detect continuity failures. The starting point is tighter object control, but this week's reports suggest that scene structure and interaction rules can also exist outside the generator. In the first year, researchers would connect existing editing systems to small scene descriptions and rule-running software, rather than immediately build a unified model. Early trials would ask whether those connections reduce manual repairs while preserving requested actions, especially when objects cross or disappear temporarily from view.

In the second year, shared scene descriptions could connect generation across shots with simple interaction rules, initially in previews containing only a few entities. These descriptions would record which entities exist and how they relate, allowing teams to inspect failures or change renderers without rebuilding the intended scene. Evaluation would need to separate generator errors from measurement errors, because a tracker losing an obscured object does not establish that the generated scene lost it. By the third year, successful pilots could combine an explicit stage and persistent state store with a renderer whose outputs undergo a separate checking step. The central research question would become how to resolve disagreement between the stored record and the image, including when to regenerate or request human approval.

Progress would compound when checks expose specific failures and logs turn those failures into new training examples, making subsequent outputs easier to control and verify. A useful monitoring cue is lower correction effort in matched workflow comparisons, supported by discrepancy reports that explain where the scene record and rendered result disagree. Longer tests should also check whether automated updates gradually invent or lose entities as small errors accumulate across a sequence. This remains a conditional path for constrained production work, not evidence that dependable general simulation will emerge within three years. It would weaken if external records provide no measurable benefit, or if checking consumes the effort saved during generation.

Over three years, this scenario could make correct actions and lasting consequences more important in evaluation, changing which system designs researchers prefer. It builds on tighter object control and explicit scene staging, but the newer editorial-reported findings first need independent reproduction before their reliability can be assumed. During the first year, researchers would compare appearance-based rankings with tests of whether outputs follow instructions and preserve state, meaning earlier events continue to matter. They would also calibrate the tests against scenes with known outcomes, so tracking failures are not mistaken for failures in the generated video.

Practical pilots would measure whether these checks reduce corrections, while separating the time spent rendering from the time required to verify results. In the second year, the decisive shift would be an influential benchmark ranking systems primarily by correct consequences while retaining a minimum visual-quality requirement. That could favor explicit state engines because they store interaction requirements separately from the images, making intended behavior easier to inspect and test. The mechanism is a feedback loop: wider benchmark use attracts more systems, whose difficult cases reveal weaknesses that help researchers improve the measurement process. Production teams could then introduce documented error limits for familiar tasks, allowing selective automation while retaining human review when results remain uncertain.

By the third year, calibration across difficult visual conditions would become a central research problem, since changing visibility or stylization can undermine otherwise useful evaluators. Evaluation might split into physically grounded and appearance-first creative tracks, rather than impose the same expectations on every application or visual style. Within the grounded track, a few groups could connect persistent state with staged rendering in tested workflows, extending validation to interacting entities and shot transitions. A monitoring cue would be broader use of separate consequence reporting alongside reduced correction effort, rather than isolated benchmark gains that do not improve workflows. This path would weaken if ranking differences fall within evaluator error, or if end-to-end models match separated systems at comparable quality and response time.

Over three years, this scenario could shift attention from generating more clips toward producing clips whose correctness can be confirmed at manageable cost. Tighter controls and richer evaluation create the starting point, but the turning point would be evidence that checking outputs becomes a serious bottleneck. In the first year, researchers would compare inexpensive preliminary screens with deeper tests, using engine-generated scenes with known states to calibrate their measurements. A screen would only flag likely problems, not certify correctness, while production prototypes would expose scene structure so reviewers could locate specific failures. Useful early pilots would report both correction effort and the share of outputs confirmed, rather than treating faster generation alone as sufficient progress.

During the second year, benchmark organizers could require disclosure of which outputs received full checks and how closely preliminary screens agree with those checks. Structured systems might then gain an advantage if records of intended events, combined with sampled image checks, reduce the work needed to confirm results. That creates a feedback loop in which cheaper confirmation encourages explicit scene records, and wider use produces reusable evidence for improving later acceptance procedures. Shared evaluation services could support this process, although their operators might gain disproportionate influence over which systems are treated as reliable.

By the third year, repeatable acceptance workflows could become routine for bounded tasks such as multi-shot staging, layered composition, and entity-preserving restyling. Researchers would broaden reference scenes to harder conditions, while production teams would track correction effort per shot and reuse checking records across compatible tools. A division could emerge between systems with confirmed consistency claims and exploratory systems that remain costly to check, without establishing which produces better images. The clearest monitoring cue would be sustained checking backlogs alongside matched comparisons showing lower total effort per correctly accepted output when structured records are used. This scenario would weaken if full evaluation becomes inexpensive, or if image mismatches demand such dense checking that accurate event records save little work.

1-Year / 3-Year Research-Application Infographic

Integrated scenario outlook infographic

References

This page was created using generative AI such as GPT-5, GPT-6.1 Sol, GPT-6 Astra, Claude Opus 5.5, Gemini 3, Grok 4.7, Fugu Ultra, GLM 5.3, Gemini 3.1 Flash Image, GPT Image 2.5 Sunburst, and their higher-end successor versions. No guarantee can be made regarding its contents.