The idea in one page

Fast-WAM is a robot policy. You give it the current camera images, the robot's joint state, and a sentence describing the task, and it produces a short chunk of future actions. It was trained purely by imitation: it watched expert demonstrations and learned to reproduce them. On the LIBERO benchmark it already succeeds on roughly 98% of episodes.

Imitation learning has a well-known ceiling. The policy can only be as good as the demonstrations, it has no idea what to do in situations the demonstrators never reached, and once it makes a mistake it tends not to recover, because recoveries were not in the data. Reinforcement learning (RL) addresses exactly this: the policy tries things in the environment, gets told whether they worked, and is nudged toward the things that worked and away from the things that did not. "Online" RL means this happens in a live loop rather than on a fixed dataset.

WAM-RL is a recent paper that applies online RL to a world-action model, a model that has two coupled parts: a world model that imagines future video, and an action model that reads the world model's internal features and produces actions. WAM-RL's insight is that you should not just fine-tune the action model with RL and leave the world model frozen, because the action model depends on the world model's feature space and the world model has never seen the new situations the RL policy reaches. So WAM-RL does three things at once: it trains the action model with a policy gradient, it fine-tunes the world model on video from successful rollouts, and it regularises the world model's features so the action model does not have the rug pulled out from under it.

This document works out how to do the same thing to Fast-WAM. Most of the machinery transfers directly. One thing does not: Fast-WAM's headline mode does not imagine the future at all when it acts, whereas WAM-RL's reward is defined as the agreement between the imagined future and the real one. Chapter 3 and Chapter 6 deal with this, and it turns out to be the genuinely new scientific question in the whole project.

The plan, in the shortest possible form, is:

1. Replace Fast-WAM's deterministic action sampler with a stochastic one that has a computable probability density, so that we can do policy gradients on it.
2. Roll the policy out in the LIBERO simulator many times, recording everything.
3. Score each action chunk with a reward made of task success plus, optionally, how well the world model's imagination matched what really happened.
4. Update the action model with a PPO-style clipped policy gradient.
5. Update the world model on the successful rollouts' video, with a KL penalty that keeps its features close to where they started.
6. Repeat, and evaluate with the original deterministic sampler so the reported numbers are comparable to the paper's.