Updating the actor

8.1 The clipped objective

With stored chains, old log-probabilities, and advantages in hand, the actor update is the standard PPO clipped objective applied at the level of denoising steps (DPPO, πRL). For each macro-step $t$ and stochastic denoising step $k$:

$$\rho_{t,k} = \exp\big( \log \mathcal{N}(x_{k+1}; \mu_k^{\varphi}, \mathrm{std}_k^2) - \log \mathcal{N}(x_{k+1}; \mu_k^{\mathrm{old}}, \mathrm{std}_k^2) \big)$$ $$L_{\mathrm{actor}} = -\,\mathrm{mean}_{t,k} \big[ \min\big( \rho_{t,k} \cdot \hat{A}_t ,\ \mathrm{clip}(\rho_{t,k}, 1-\varepsilon, 1+\varepsilon) \cdot \hat{A}_t \big) \big]$$

The same macro-step advantage $\hat{A}_t$ is broadcast to every stochastic step $k$ of that macro-step (Flow-GRPO states this explicitly; with πRL's single-stochastic-step sampler there is only one $k$ anyway). $\mu_k^{\varphi}$ is recomputed in the learner with the current parameters and gradients enabled, using the stored x_k as input; $\mu_k^{\mathrm{old}}$ is whatever produced the stored $\log \pi_{\mathrm{old},k}$. The clip range $\varepsilon = 0.2$ is the standard PPO value; SimpleVLA-RL uses an asymmetric "clip-higher" range of $\varepsilon _{\mathrm{low}} = 0.2, \varepsilon _{\mathrm{high}} = 0.28$ to encourage exploration, which is a cheap variant to try.

8.2 Keeping the policy near its starting point

There are two ways to stop the actor drifting into nonsense, and you generally want only one of them.

The clip itself bounds how far the policy moves per update and is usually sufficient; SimpleVLA-RL removes the KL term entirely and reports that this helps exploration. If you observe degradation in the deterministic evaluation despite stable training curves, add a KL penalty to the frozen reference actor,

L_actor ← L_actor + λ_ref · mean_{t,k} [ log π_φ − log π_ref ]

Flow-GRPO uses $\beta = 0.04$ on its GenEval and OCR tasks and 0.01 on PickScore; those are for image generation, so treat 0.01 as an upper starting point for a robot policy and reduce if the policy stops moving. A larger value simply turns RL back into imitation of the original model.

A different regulariser is to mix in supervised behaviour cloning on the actions of successful rollouts (self-imitation, in the sense of Oh et al., 2018), which Chapter 9 discusses as an option in the world-model update, since it reuses the same batches.

8.3 Gradient path and what has to be recomputed

The actor's gradient flows through $v_\varphi(x_k, \sigma_k)$, which means through the action expert and through the first-frame K/V cache it attends to. Whether the cache carries gradient depends on a choice:

• If the video expert is frozen or is updated only by the world-model loss (Chapter 9), compute the cache under torch.no_grad() once per batch and let the actor gradient stop there. This is cheaper and is what this document recommends initially; it keeps the actor update and the world-model update cleanly separated, as in WAM-RL.
• If you want the policy gradient to shape the video expert's features directly, let the cache carry gradient. This is more expensive and mixes the two updates; treat it as an ablation.

Either way, the cache must be recomputed in the learner from the stored image with an uncompiled, grad-capable path. Chapter 10 explains why the compiled inference path cannot be reused.