Visuo-tactile world model: 20 windows, seen and unseen, short and long

Ten held-out windows and ten training windows, each rolled out at both a single-window horizon and 16 seconds autoregressively — the full {test, train} × {short, long} matrix for two conditioning designs.

Diffusion-Forcing Mixture-of-Transformers over Wan2.2 latents (48 ch, 4 latent frames, 16×16). Both runs: 100 epochs, step 117 199, identical optimizer and seed, last.ckpt — the split warns that val aliases test, so selecting best-epoch=* would be selecting on the reported episodes.

The two models

gelz — the frozen pixel-space Stage-1 flow concatenated into both experts (exactly, via a zero-init projector: a conv is linear in its input channels), plus FiLM from the raw action vector. Its tactile FiLM vector is 11-D in the gel frame: [dp(3), Rot6D(6), log1p(force_N), contact], where +z is the measured gel normal, so channel 2 is press depth and the force rides on that same axis. That matches what the frozen tactile encoder pixc5_D already sees.

vecfilm — the control. Same architecture, but no flow concatenated, the tactile vector is the rigid-frame 9-D step, and no force. It exists because gelz changes three things at once, and without it none of them could be attributed.

Contact force reaches the tactile branch only; every video-branch input (12-ch ActionImage, camera frame, 2-ch flow, action[0:18]) is force-free by construction.

The four cells

short = one 16-frame window: 2 clean latent context frames, 2 generated, decoded to 13 pixel frames of which 5 come from the context. long = 16 s autoregressive sliding-window rollout, keep-1: seed with context, predict a window, keep one new latent frame, advance, repeat — so error compounds over ~22 model calls. test = the two held-out episodes; train = six seen episodes across all three recording dates, so the pair measures the generalisation gap, not just accuracy.

Results — decoded pixel PSNR (dB)

cell	model	camera view	tactile left	tactile right	n
short / test	gelz	24.00	45.41	46.05	10
	vecfilm	23.97	45.58	45.83	10
short / train	gelz	25.50	43.43	45.26	10
	vecfilm	24.90	43.44	44.78	10
long / test	gelz	19.98	32.23	32.34	10
	vecfilm	19.74	32.36	32.63	10
long / train	gelz	21.89	30.54	31.13	10
	vecfilm	21.03	30.66	31.44	10

Generalisation gap (train − test)

horizon	model	camera view	tactile left	tactile right
short	gelz	+1.50	-1.98	-0.79
short	vecfilm	+0.93	-2.14	-1.05
long	gelz	+1.91	-1.69	-1.21
long	vecfilm	+1.28	-1.69	-1.19

Positive = the model does better on episodes it trained on. A large positive number at long horizon and a small one at short horizon is the signature of memorised dynamics rather than memorised appearance.

Rollouts

Ground truth on top, then each model. Frames outlined in red are generated; the unoutlined leading frames are the clean context. Long-horizon strips subsample evenly across the 16 s. PSNR is over the whole clip, computed on the raw decoded tensors — the images are h264, which costs a little fidelity equally in every row.

short horizon — test split

005:0 — sample_000

005:34 — sample_001

005:68 — sample_002

005:102 — sample_003

005:136 — sample_004

006:0 — sample_005

006:150 — sample_006

006:300 — sample_007

006:450 — sample_008

006:600 — sample_009

short horizon — train split

001:0 — sample_000

001:400 — sample_001

001:800 — sample_002

004:0 — sample_003

004:150 — sample_004

005:0 — sample_005

005:400 — sample_006

013:0 — sample_007

013:300 — sample_008

002:0 — sample_009

long horizon — test split