The full training loop and starting hyperparameters

11.1 One iteration, end to end

ROLLOUT (N worker GPUs, each running env_num LIBERO environments)
  sample tasks with hard-task oversampling; for each sampled (task, seed):
    run G = 8 rollouts from that seed
    per macro-step:
      build state (image, proprio, task id)
      prefill first-frame cache (compiled)
      run K = 8 denoising steps; one randomly chosen step k* ≥ 1 is a Flow-SDE step
        with noise scale a = 0.5, the rest are deterministic ODE steps (πRL mixed sampler)
      store chain x_0..x_K, k*, and log π_old at k*
      execute first 10 actions; render and store the frames at offsets +4, +8
        (and every frame if you will use this episode for video SFT)
      record r^task, done
  write episodes to the rollout buffer (target ≈ 256 episodes per iteration)

LEARNER
  1. reward: for each macro-step, imagine the future (live or frozen world model),
     compute r^rec, normalise, combine: r = r^task + β · r̃^rec; compute returns R_t
  2. advantages: group-normalise within each (task, seed) group; drop degenerate groups
  3. actor phase: 2 PPO epochs over all macro-steps, minibatch 64,
     recompute μ_k with gradients, minimise L_actor (+ λ_ref reference KL if enabled)
  4. world-model phase: 1 epoch over 33-frame windows from successful episodes,
     minimise L_video + λ_KL · L_KL through the LoRA parameters
  5. broadcast new weights to workers; workers drop their compiled graphs
  6. every 5 iterations: deterministic evaluation (a = 0, K = 20) on all 40 LIBERO tasks,
     10 episodes each — this is the number you report

Run it synchronously first: rollouts finish, the learner runs, weights are sent, the next rollouts begin. Asynchronous pipelines of the kind RLinf builds are an optimisation for when the loop works.

11.2 Starting hyperparameters and where they come from

Quantity	Start value	Provenance
Denoising steps during RL, $K$	8	between πRL's default 4 (ablated 1/2/4/8) and Flow-GRPO's 10-train/40-eval
Stochastic steps	1 random step $k^{*} \ge 1$	πRL mixed ODE–SDE sampler
Noise scale $a$ in $g(\sigma) = a\sqrt{\sigma/(1-\sigma)}$	0.5	πRL ablation over {0.2, 0.5, 0.8}; 0.2 unstable, 0.8 degrades
PPO clip $\varepsilon$	0.2 (try 0.2/0.28 clip-higher)	standard PPO; SimpleVLA-RL clip-higher
Group size $G$	8	SimpleVLA-RL (Flow-GRPO uses 24 for images)
Degenerate-group filtering	on	SimpleVLA-RL dynamic sampling; RIPT-VLA
Episodes per iteration	256	engineering judgement (SimpleVLA-RL: batch 64 prompts × 8 = 512 rollouts)
PPO epochs per iteration	2	engineering judgement
Actor minibatch (macro-steps)	64	engineering judgement
Actor learning rate	1e-5	SimpleVLA-RL uses 5e-6 for a 7B policy; scaled up for a 0.7B head
Video-expert LoRA rank / learning rate	64 / 5e-5	engineering judgement (Flow-GRPO uses LoRA r=32, α=64)
Discount $\gamma$	0.99 (0.995 for LIBERO-10)	engineering judgement; binary-reward papers use no discount
Reconstruction weight $\beta$	0.1 after normalisation	unspecified in WAM-RL; tune
Feature-KL weight $\lambda_{\mathrm{KL}}$	0.1	unspecified in WAM-RL; tune
Reference-actor KL $\lambda_{\mathrm{ref}}$	0 (≤ 0.01 if needed)	SimpleVLA-RL removes KL; Flow-GRPO uses 0.01–0.04 for images
Video-SFT batch / windows per successful episode	16 / up to 8	pretraining batch size
EMA momentum for $\Sigma_\theta$	0.99	engineering judgement
Deterministic evaluation steps	20	Fast-WAM default