Robohouse ’26 Library
Contents

Chapter 13

Verification checks before the first RL run

about 1 minutes

RL failures are notoriously hard to diagnose after the fact, so each piece should be checked in isolation first. None of these require a simulator.

  1. Sampler equivalence. With a=0a = 0 and K=20K = 20, sample_chain must reproduce infer_action to bf16 tolerance for the same seed. This proves the Flow-SDE step reduces to the existing ODE step.
  2. Log-probability consistency. Sample a chain with the worker-side code, then feed it to the learner-side logprob_chain with the same parameters. The recomputed log-probabilities must equal the stored ones, so that the PPO ratio is exactly 1 at the first update. Any mismatch (from compiled versus uncompiled numerics, or a σ-schedule discrepancy) will otherwise show up as a spurious policy change on the first step.
  3. Step-count robustness. Deterministic evaluation of the pretrained checkpoint at K=8K = 8 on a few tasks should match K=20K = 20 within a point.
  4. Data round-trip. Take a successful simulator episode, run it through the Chapter 9 pipeline, and confirm that training_loss on it is in the same range as the training loss on real dataset batches. Also compare one processed frame pixel-for-pixel against what _obs_to_model_input produced for the same observation.
  5. KL at initialisation. With the LoRA adapters at their zero initialisation, L_KL must be exactly zero, and Σold\Sigma_{\mathrm{old}} must be positive everywhere (add a small floor).
  6. Reward sanity. On a handful of successful and failed episodes, the reconstruction reward should be at least weakly correlated with success (WAM-RL reports pixel MSE separates them only weakly, so do not expect a strong signal, but the sign should be right).