Extending latent world models for multi-view camera setups

Practical Course: Generative AI — From Diffusion to World Models, Technical University of Munich.

The problem

A latent world model plans by imagining: it encodes what it sees into a latent, and a learned dynamics model predicts how that latent evolves under actions. LeWM is the first JEPA world model to do this stably end-to-end from raw pixels — it predicts in representation space rather than pixel space, and avoids collapse through LeJEPA's SIGReg objective, with no reconstruction and no contrastive term.

But it watches through one fixed camera. In 3D manipulation that leaves physical state under-determined: depth along the optical axis is ambiguous, and the manipulator occludes the object it is acting on. No encoder can recover state its input does not determine.

Earlier multi-view world models fix this by fusing views through reconstruction, contrastive learning, or generative video — precisely the machinery a JEPA is built to avoid. VGGT suggests none of it is necessary: alternating frame-wise and global self-attention recovers 3D geometry from unposed views in a single feed-forward pass. So: can a reconstruction-free, feed-forward multi-view encoder give a stable JEPA world model the 3D understanding a single view cannot?

What I changed

One controlled intervention: only the encoder. The predictor, action conditioning, objective and planner are LeWM's, untouched, and the encoder still emits exactly one latent per timestep regardless of how many cameras feed it — so any difference is attributable to fusion.

I extended OGBench with four synchronised cameras — front, two static side views, and a wrist camera on the end effector — and regenerated the Cube pick-and-place dataset with an expert play policy. Resolution, frame-skip and history length match LeWM, so the only difference against the single-view baseline is the camera count.

Three fusion architectures, differing only in how much information crosses between views. cross-view follows VGGT directly: every block alternates a frame-wise stage with a global stage over all tokens from all views. cross-view-camera keeps the frame-wise stage but restricts the global stage to the four camera tokens, one per view, which mix back into the patches at the next frame-wise stage. cls-bottleneck is the compressed reference point: all cross-view flow squeezed through K shared [CLS] tokens.

What I found

The direct test is a linear probe on frozen embeddings predicting ground-truth kinematics. Every multi-view variant beats the baseline on every target, and the gain tracks how under-determined the single view is — end-effector position is already legible from the front camera alone, but yaw, gripper state and cube position rise sharply.

The headline is the third column. cross-view-camera exchanges one token per view and costs no more than four single-view encoders — effectively free — yet matches full patch-level attention on everything except end-effector yaw. Patch-level exchange buys a little orientation detail and little else, at 2.5× the compute.

Legibility also survives imagination. Rolling the predictor forward autoregressively and probing the imagined latents, every fused variant beats the baseline at every horizon. Measuring the drift directly is where the ordering gets interesting: after 16 imagined steps cross-view-camera has accumulated 8.9× its one-step error, against 14.9× for cross-view, 17.3× for the baseline and 23.8× for cls-bottleneck.

That inversion is the most interesting thing in the project. Expressiveness helps perception and hurts prediction. Ordered by cross-view information, cross-view is the most expressive encoder and wins most static probes; ordered by how well the predictor tracks the resulting latent, that ordering reverses. A JEPA optimises both halves jointly, so the encoder can move its own target — extra scene detail then has to be predicted from actions alone. The design criterion is not representational capacity but the predictability of what is represented, which static probing ranks backwards.

Where it does not pay off

Planning is a negative result and I report it as one. With LeWM's unchanged CEM + receding-horizon MPC over 100 goals, cross-view-camera is nominally best at 0.68 against the baseline's 0.61 and the ordering matches the representation results — but every 95% interval overlaps.

The decisive number is the last row: uniform random actions score 0.51. Goals are drawn only 25 steps ahead and many episodes already start inside the success threshold, so half the probability mass is consumed by trivially-solved goals and the remainder cannot resolve a seven-point gap at this sample size. This is an underpowered evaluation rather than evidence that the representations are equivalent — the imagination results, on the same checkpoints without a planner, separate the variants cleanly.

Novel-view decoding tells a similar story. Freezing each encoder and training a decoder for a camera pose absent from training, both decoders recover the novel viewpoint's global geometry but are heavily blurred and differ far less than the probing gaps suggest — plausibly confounded by decoder capacity rather than by what the latents contain.

Takeaways

• A reconstruction-free, feed-forward multi-view encoder does give a stable JEPA world model 3D understanding a single camera cannot supply, and the gain persists over imagination.
• It needs no expensive attention. One camera token per view matches full patch-level attention at a quarter of the cost, while drifting half as fast.
• Whether that converts into control at longer horizons is still open. Lengthening the goal offset and planning horizon together, where the drift curve predicts cross-view-camera should separate, is the follow-up.

Caveats worth stating plainly: one environment, one seed per variant, in exactly the regime where fusion should help most. Views are fixed and extrinsics are never given, so the fusion is learned for one camera rig and would not transfer to a moved one.