Matrix-Game 4.0 以傳統引擎處理遊戲狀態、影片模型生成畫面
Stop Asking Video Models to Run Physics: Inside Matrix-Game 4.0
Skywork AI 與合作機構的 Matrix-Game 4.0 論文把遊戲狀態管理交給傳統引擎,並由自回歸影片生成器把低多邊形代理畫面轉成最終影格。在單張 NVIDIA RTX 5090 上,系統以 480p 運行於 60 FPS,按鍵至畫面延遲為 112ms。單步生成的穩態吞吐量為 83 FPS,峯值 GPU 記憶體用量為 18.2 GiB。
Over the past two years, neural world models have attempted to turn video generators into complete simulation engines. The premise sounded straightforward: feed keystrokes into a diffusion or autoregressive model, predict the next set of latent frames, and let the neural network learn physics, collision detection, and rendering all at once.
In practice, anyone who has tested pure neural game simulators knows where the setup breaks down. Neural weights make unreliable state machines. Walk down a hallway, turn around, and the doorway you just walked through has vanished or morphed into a bookcase. Shoot an opponent, and the model forgets three frames later whether the damage registered. Worse, generating high-resolution frames while recalculating game mechanics demands massive GPU clusters, leaving playable interactive frame rates out of reach for local hardware.
A new paper from Skywork AI and partner institutions, Matrix-Game 4.0 (arXiv:2610.27999), takes a different architectural approach. Instead of forcing a video model to compute simulation state, the authors split game execution from visual generation through an explicit harness.
The system assigns responsibility across three separate components. First, a lightweight traditional game engine maintains the underlying world state. It tracks player coordinates, inventory, health points, and rigid-body collision boxes deterministically. Second, a multimodal coding agent translates player natural language instructions into concrete game scripts and dynamic scene captions. Third, the lightweight engine renders basic, low-poly wireframes and proxies at high speed. An autoregressive video generator then takes these low-poly proxies, dynamic captions, and memory references to produce the final textured frames.
Because the underlying engine handles coordinate geometry, spatial persistence stays locked down. If you walk past a tree and turn 180 degrees, the tree is still there because the bounding box in the engine never moved. The neural model only needs to perform visual reskinning across the incoming proxy feed rather than inferring where objects belong in three-dimensional space.
To hit interactive speeds, the team trained the renderer on aligned pairs of low-poly wireframes and high-fidelity video footage. They distilled the model into a causal generator that operates in four-frame chunks with one or two denoising steps.
The latency and throughput measurements show the benefit of offloading state logic. Running on a single NVIDIA RTX 5090, Matrix-Game 4.0 outputs 480p at 60 FPS with an end-to-end key-press-to-display latency of 112ms. Single-step generation delivers 83 FPS steady-state throughput while consuming 18.2 GiB of peak GPU memory. With fixed prompts, the system still manages 67 FPS on an RTX 5080 and 58 FPS on an RTX 5070Ti.
For developers building agent harnesses or real-time simulation pipelines, the architectural split provides a clean template. Deterministic state belongs in deterministic code. Trying to train an expensive foundation model to replace a twenty-line collision function wastes compute on a solved problem. Letting the engine handle the state while the neural generator handles the pixels turns an impractical research demo into something that actually runs on a single desktop card.
來源:DEV Community · dev.to