Advantages: deciding which actions were better than expected

A policy gradient pushes up the probability of actions with positive advantage and down those with negative advantage, where the advantage is "how much better than expected was this". WAM-RL does not say how it computes advantages. Two approaches are grounded in the related literature, and this document recommends starting with the first.

7.1 Group-relative advantages (GRPO-style)

This is what SimpleVLA-RL, RLinf, and Flow-GRPO use. For each starting condition (a LIBERO task and an initial-state seed), run $G$ rollouts (say 8) from the same start. Then for each rollout and macro-step,

$$\hat{A}_t = \big( R_t - \mathrm{mean}_G(R_t) \big) / \big( \mathrm{std}_G(R_t) + \varepsilon \big)$$

where the mean and standard deviation are taken over the $G$ rollouts of the same group at the same macro-step index (or, more simply, over the episode returns R_0, broadcast to every step of each rollout; the per-episode version is what SimpleVLA-RL and Flow-GRPO do, assigning one trajectory-level advantage to every step). RIPT-VLA uses the closely related leave-one-out baseline, where each rollout is compared against the mean of the other $G - 1$ in its group.

There is no value network to train, and the baseline is exact for that starting state. The weakness shows up at Fast-WAM's success level: if all 8 rollouts succeed, the advantage is zero for all of them and the group contributes nothing. Two measures deal with this.

• Drop degenerate groups. Skip any group in which every rollout has the same task outcome. They carry no information about which behaviour is better. SimpleVLA-RL calls this dynamic sampling and keeps sampling until each retained group has $0 < #\text{successes} < G$; RIPT-VLA's dynamic rollout sampling does the same.
• Oversample hard tasks. Keep a running per-task success rate from the previous iteration and sample tasks in proportion to something like $(1 - p_{\mathrm{success}}) + 0.05$. At a 98% average, nearly all the signal lives in LIBERO-10 and a handful of spatial tasks, and uniform task sampling would waste most rollouts.

With the dense reconstruction term present, the returns within a group differ even when outcomes agree, so fewer groups are fully degenerate; this is one of the concrete benefits the reconstruction reward could deliver, and Chapter 14 suggests measuring it directly.

7.2 PPO with a critic and GAE

πRL uses a learned value function and generalised advantage estimation. You would add a small value head that takes a pooled version of the first-frame features (for example the mean of the last-layer video tokens) concatenated with the proprio vector and predicts $V(s)$. The value head is trained by regression to the returns, and advantages are $\hat{A}_t = \sum_l (\gamma \lambda )^l \delta _{t+l}$ with $\delta _t = r_t + \gamma V(s_{t+1}) - V(s_t)$.

This is more sample-efficient with dense rewards and does not require grouped rollouts, but it adds a second network that must be kept stable. RLinf-VLA reports that for PPO, assigning advantages per atomic action inside the chunk ("action-level") beats one shared advantage per chunk, whereas for GRPO the chunk-level formulation is used; if you go the PPO route that is worth copying. The recommendation is to start with group-relative, chunk-level advantages and switch only if the reconstruction-reward variance makes learning erratic.

---