Chapter 13
Verification checks before the first RL run
about 1 minutes
RL failures are notoriously hard to diagnose after the fact, so each piece should be checked in isolation first. None of these require a simulator.
- Sampler equivalence. With and ,
sample_chainmust reproduceinfer_actionto bf16 tolerance for the same seed. This proves the Flow-SDE step reduces to the existing ODE step. - Log-probability consistency. Sample a chain with the worker-side code, then feed it to the learner-side
logprob_chainwith the same parameters. The recomputed log-probabilities must equal the stored ones, so that the PPO ratio is exactly 1 at the first update. Any mismatch (from compiled versus uncompiled numerics, or a σ-schedule discrepancy) will otherwise show up as a spurious policy change on the first step. - Step-count robustness. Deterministic evaluation of the pretrained checkpoint at on a few tasks should match within a point.
- Data round-trip. Take a successful simulator episode, run it through the Chapter 9 pipeline, and confirm that
training_losson it is in the same range as the training loss on real dataset batches. Also compare one processed frame pixel-for-pixel against what_obs_to_model_inputproduced for the same observation. - KL at initialisation. With the LoRA adapters at their zero initialisation,
L_KLmust be exactly zero, and must be positive everywhere (add a small floor). - Reward sanity. On a handful of successful and failed episodes, the reconstruction reward should be at least weakly correlated with success (WAM-RL reports pixel MSE separates them only weakly, so do not expect a strong signal, but the sign should be right).