Robohouse ’26 Library
Contents

Appendix C

Sources

about 3 minutes

All arXiv identifiers below were checked against arxiv.org on 2026-08-23, and the specific claims attributed to each paper in the text were checked against the paper's HTML version where one exists. Claims that could not be verified are labelled as engineering judgement in the text.

Primary:

  • WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFT. arXiv:2606.17906. https://arxiv.org/abs/2606.17906 — verified: base model (Genie-Envisioner-ACT), Flow-SDE/πRL actor, reconstruction reward rt=sim(x^,x)r_t = \mathrm{sim}(\hat{x}, x) with pixel MSE best among pixel/optical-flow/DINOv2/V-JEPA2, online video SFT on successful trajectories, Gaussian feature KL with EMA covariance, LIBERO-Object 68 → 78 → 82, 8 A800 GPUs for 8 hours. No hyperparameter table, pseudocode, or advantage-estimator details are given.
  • Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666. https://arxiv.org/abs/2603.16666 — architecture, configs, and numbers taken directly from the FastWAM repository and its README.

Methods this document builds on:

  • πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889. https://arxiv.org/abs/2510.25889 — verified: Flow-Noise and Flow-SDE; score −A/τ−((1−τ)/τ)v-A/\tau - ((1-\tau)/\tau)v; SDE drift v+στ2/(2τ)(A+(1−τ)v)v + \sigma_\tau^2/(2\tau)(A + (1-\tau)v); στ=aτ/(1−τ)\sigma_\tau = a\sqrt{\tau/(1-\tau)} with a∈{0.2, 0.5, 0.8}a \in \{0.2,\ 0.5,\ 0.8\}, 0.5 best, 0.2 unstable; mixed ODE–SDE with one randomly sampled stochastic step; K∈{1,2,4,8}K \in \{1,2,4,8\} with 4 as default; chunk as macro-step; PPO with critic and GAE; log-prob is the sum over denoising steps; LIBERO few-shot π₀ 57.6 → 96.1 (Flow-SDE) / 97.6 (Flow-Noise). Learning rates and clip values are in an appendix not verified here.
  • DPPO: Diffusion Policy Policy Optimization. arXiv:2409.00588. https://arxiv.org/abs/2409.00588 — identifier verified; the two-layer MDP and Gaussian per-step transitions are from the body of the paper, which was not re-read for this document.
  • Flow-GRPO: Training Flow Matching Models via Online RL. arXiv:2505.05470. https://arxiv.org/abs/2505.05470 — verified: same SDE update as πRL; training rollouts with 10 steps, inference with 40; a=0.7a = 0.7; KL β=0.04\beta = 0.04 (GenEval/OCR) and 0.01 (PickScore); group size 24; LoRA r=32, α=64; one trajectory-level advantage shared across all denoising steps.
  • ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning. arXiv:2505.22094. https://arxiv.org/abs/2505.22094 — verified: learnable noise injection into the flow path, exact likelihoods at one to four denoising steps (NeurIPS 2025).
  • DSRL: Steering Your Diffusion Policy with Latent Space Reinforcement Learning. arXiv:2506.15799. https://arxiv.org/abs/2506.15799 — verified: RL over the latent-noise space of a frozen behaviour-cloning policy with black-box access only. The abstract does not name the RL algorithm; "SAC" in Chapter 5 is from memory of the body.
  • SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. arXiv:2509.09674. https://arxiv.org/abs/2509.09674 — verified: binary trajectory-level success reward; GRPO with clip-higher (εlow=0.2,εhigh=0.28\varepsilon _{\mathrm{low}} = 0.2, \varepsilon _{\mathrm{high}} = 0.28); no KL; group size 8; batch 64; learning rate 5e-6; dynamic sampling that filters groups with identical outcomes; OpenVLA-OFT (autoregressive) 91.0 → 99.1 on LIBERO.
  • RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models. arXiv:2510.06710. https://arxiv.org/abs/2510.06710 — verified: PPO and GRPO; sparse success reward; chunk-level versus action-level advantage, with action-level better for PPO; OpenVLA-OFT 98.11% on the 130-task LIBERO set.
  • RIPT-VLA: Interactive Post-Training for Vision-Language-Action Models. arXiv:2505.17016. https://arxiv.org/abs/2505.17016 — verified: sparse binary rewards, dynamic rollout sampling, leave-one-out advantage estimation.
  • Self-Imitation Learning. Oh, Guo, Singh, Lee, 2018. arXiv:1806.05635 — the source for "imitate your own successful rollouts"; identifier from memory.
  • Score-Based Generative Modeling through Stochastic Differential Equations. Song et al., 2021. arXiv:2011.13456 — the source of the equal-marginal reverse SDE / probability-flow ODE relation used in Appendix A; identifier from memory.

Code references in the FastWAM repository: src/fastwam/models/wan22/fastwam.py (_build_mot_attention_mask, training_loss, _denoise_action_with_video_cache, infer_action), src/fastwam/models/wan22/schedulers/scheduler_continuous.py, src/fastwam/models/wan22/fastwam_optional_idm.py, src/fastwam/trainer.py, experiments/libero/eval_libero_single.py, configs/sim_libero.yaml, configs/data/libero_2cam.yaml, configs/model/fastwam_optional_idm.yaml.