Online Reinforcement Learning for Fast-WAM

How to read this

This document explains, from the ground up, how to take the Fast-WAM model (a world-action model that predicts robot actions directly from the current camera image, optionally imagining the future) and continue training it by letting it interact with a simulator, in the spirit of the WAM-RL paper. It is written to be read in order. The early chapters build up the concepts that the later chapters depend on, but every chapter restates what it needs, so you can also jump to the one you care about.

The intended reader is someone who is going to actually implement this. For that reason the emphasis is always on why a design choice is made, followed by the specific tensor shapes, equations, file names, and starting numbers you would need at the keyboard. Where a choice is grounded in a published method, the method is named. Where a choice goes beyond the literature, because Fast-WAM is different enough from the models those papers used, this is said explicitly and the alternatives are laid out as experiments rather than as settled answers.

One warning before you begin. The WAM-RL paper does not publish a hyperparameter table, pseudocode, or the details of how it estimates advantages. Every number in this document that looks like a hyperparameter comes either from related papers (πRL, Flow-GRPO, RLinf, SimpleVLA-RL, DPPO) or from engineering judgement, and Chapter 11 says which is which. Treat them as starting points, not as results.