Research · Tencent Hy LLM Frontier · July 2026
研究 · 腾讯 HY 大模型 · 2026 年 7 月
Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
陈旧度下的稳定训练：用于稳定异步强化学习的陈旧度自适应信任域
Staleness analysis in asynchronous reinforcement learning: principled origins, diagnostics, and mitigations.
异步强化学习中的陈旧度问题：来源剖析、诊断与缓解方案。
Decoupling rollout from training buys throughput, but the policy that generated a token can then differ from the policy
that trains on it. This blog is a field guide to that mismatch: where it comes from, why it's risky under LLM's reinforcement learning setting
, and what current stabilizers and our proposed method do about it. The theory in one line: the policy-improvement
approximation degrades as the divergence between inference policy $\mu$ and training policy $\pi$ grows. At state $s$,
the divergence can be defined as $D_{\mathrm{TV}}(\mu,\pi)[s]\;=\;\tfrac12\,\mathbb E_{a\sim\mu(\cdot\mid s)}\big|r(a)-1\big|$ — so a hard ratio bound on every
action would bound the divergence — but PPO clipping is only a sampled-objective guardrail, not that hard constraint.
One feasible solution is lets this clipping mechanism adaptive to the action: Staleness-Adaptive Trust Region (SAT)
gives tokens with high observed staleness a tighter directional clip radius,
suppressing future distribution drift via Staleness-based kernel function scaling and Effective contraction factors. In the reported best-checkpoint grid, SAT-GSPO w/ R3 reaches 35.83 at lag = 1 and 34.79 at lag = 8. Under high-staleness training, SAT can achieve higher training performance and lower staleness during training, while several fixed-clip baselines still collapse later in training.
将采样与训练解耦可以换来吞吐，但代价是生成 token 的策略可能已不同于消费它的训练策略。本文是关于这种失配的一份“实战指南”：它从哪里来、在 LLM 强化学习设定下为何具有风险，以及现有稳定方法与我们提出的方法如何应对。从理论层面一句话概括：inference policy $\mu$ 与 training policy $\pi$ 之间的 divergence 越大，策略改进近似越不可靠。对状态 $s$，这一 divergence 可以定义为 $D_{\mathrm{TV}}(\mu,\pi)[s]\;=\;\tfrac12\,\mathbb E_{a\sim\mu(\cdot\mid s)}\big|r(a)-1\big|$， 因此若能对每个动作施加硬比率界，就能约束这一 divergence。与此同时， PPO 裁剪只是采样目标上的一道截断，并不是针对 divergence 本身的硬约束。
一个可行的方案是让这道截断倾听动作：陈旧度自适应信任域（Staleness-Adaptive Trust Region, SAT）给观测陈旧度较高的 token 更紧的方向性裁剪半径，并通过基于陈旧度的核函数缩放与有效收缩因子来抑制未来的分布漂移。在最新报告的最佳检查点汇总中，SAT-GSPO w/ R3 在 lag = 1 达到 35.83，在 lag = 8 达到 34.79。在高异步的训练场景下，SAT 可以实现更高的训练性能与训练期间的较低陈旧度，而若干基线在训练中仍会在后期崩溃。
In one minute一分钟速览
- Asynchronous RL for LLMs decouples rollout, which is executed by SGLang[2], from training, which is executed by Megatron[3], so the two can be parallel; the price is a token-level importance ratio $r_{b,t}$. This ratio is the sampled probability ratio between the current training policy and the rollout or behavior policy at token $t$ of sample $b$, and it defintely departs from $1$.
- 面向大模型的异步 RL 把 rollout 与训练解耦，其中 rollout 由 SGLang[2] 执行，训练由 Megatron[3] 执行，因此二者得以并行；代价是会出现 token 级重要性比 $r_{b,t}$。这个比值表示在样本 $b$ 的第 $t$ 个 token 上，当前训练策略相对 rollout 或 behavior 策略的采样概率比，因此它不可避免地偏离 $1$。
- The performance-bound error is bounded by full-distribution of inference-training policy divergence, with $D_{\mathrm{TV}}(\mu,\pi)\;=\;\tfrac12\,\mathbb E_{a\sim\mu(\cdot\mid s)}\big|r_{b,t}-1\big|$. Here $r_{b,t}$ denotes the sampled token-level ratio; $b$ indexes sampled responses in one batch, and $t$ indexes token positions within response. Here sampled $|r_{b,t}-1|$ can be viewed as a single-sample Monte Carlo estimator of the divergence quantity, with smaller divergence making the approximation tighter.
- 性能下界中的误差项由 inference-training policy divergence 的完整分布控制，其中 $D_{\mathrm{TV}}(\mu,\pi)\;=\;\tfrac12\,\mathbb E_{a\sim\mu(\cdot\mid s)}\big|r_{b,t}-1\big|$。这里 $r_{b,t}$ 表示采样到的 token 级比率；$b$ 表示 mini-batch 中的样本编号，$t$ 表示该样本中的 token 位置。这里采样到的 $|r_{b,t}-1|$ 可以看作这一 divergence 量的单样本 Monte Carlo 估计，而更小的 divergence 会让近似更紧。
- PPO does not enforce that all-action condition. Its fixed clip radius changes the sampled loss: it stops only outward gradients that cross the boundary and keeps the converging pull-back gradient channel.
- PPO 并不会执行上述全动作条件。固定裁剪半径只改变采样损失：它停止越过边界的向外梯度，同时保留收敛方向的回拉梯度通道。
- We propose SAT, which treats $|\log r_{b,t}|$ as an observable staleness proxy and contracts the sign-selected radius for the high-staleness tail. This makes outward clipping activate earlier; it is intended to suppress future $D_{\mathrm{TV}}(\mu,\pi)$ growth.
- 我们提出 SAT：把 $|\log r_{b,t}|$ 作为可观测陈旧度代理，并对高陈旧度长尾收缩由符号选中的半径。这会让向外裁剪更早生效；其目标是抑制未来 $D_{\mathrm{TV}}(\mu,\pi)$ 增长。
- Empirically, in single-seed runs on Qwen3-30B-A3B-Base, SAT-GSPO w/ R3 reports the best AIME24 best-checkpoint avg@8 at both configured lags, reaching 35.83 at lag 1 and 34.79 at lag 8, while several fixed-clip baselines collapse at lag 8. Adaptive clipping and MoE routing replay appear complementary.
- 经验结果来自 Qwen3-30B-A3B-Base 上的单种子实验。SAT-GSPO w/ R3 在两种配置 lag 下都报告了最高的 AIME24 最佳检查点 avg@8，其中 lag = 1 为 35.83，lag = 8 为 34.79；而多个固定裁剪基线在 lag = 8 时崩溃。自适应裁剪与 MoE 路由重放呈现互补效应。
Part 01
第 01 部分
Preliminaries: the asynchronous RL pipeline
预备知识：异步 RL 流水线
Let $b$ index a response, $t$ a token, and $j$ the global trainer update at which that response is evaluated. The rollout engine samples token $(b,t)$ from its actual behavior distribution $\mu_{b,t}$, while Megatron evaluates the current train policy $\pi^{(j)}$ with index $j$. In fixed-cadence pipelines all tokens in a batch share one rollout checkpoint, so we write $\mu_{b,t}=\mu_R$, the engine behavior produced at weights $\theta^{(\ell)}$; only in an idealized policy-only model is this identified with $\pi_{\theta^{(\ell)}}$. In a fully asynchronous pipeline $\mu_{b,t}$ may vary across tokens. The three regimes below differ in how the rollout-to-train version lag is distributed.
记 $b$ 为响应下标、$t$ 为 token 下标、$j$ 为该响应被计算时的全局训练更新下标。采样引擎从 token $(b,t)$ 的实际行为分布 $\mu_{b,t}$ 采样，Megatron 则在当前版本 $j$ 训练策略 $\pi^{(j)}$ 下计算目标。固定节拍流水线中，同一批 token 共用一个 rollout 检查点，因此可简写为 $\mu_{b,t}=\mu_R$，表示引擎在权重 $\theta^{(\ell)}$ 下产生的行为；只有在理想化、仅考虑策略的模型中，才把它等同于 $\pi_{\theta^{(\ell)}}$。完全异步时，$\mu_{b,t}$ 可以随 token 改变。下面三种范式的区别，在于 rollout 到训练的版本滞后如何分布。
Synchronous RL staleness = 0
同步 RL 陈旧度 = 0
update_weights(). Only in the idealized same-distribution abstraction do we write $\mu_R=\pi_{\theta^{(\ell)}}$.update_weights() 把权重广播回去；只有在理想化同分布抽象下才写作 $\mu_R=\pi_{\theta^{(\ell)}}$。Formally, let $\ell$ be the rollout checkpoint index, let $\theta_{\text{roll}}$ and $\theta_{\text{train}}$ be the rollout and trainer weights, and let $N_{b,t}$ be the realized lag of token $(b,t)$.
形式化地，记 $\ell$ 为 rollout 检查点下标，记 $\theta_{\text{roll}}$ 与 $\theta_{\text{train}}$ 分别为 rollout 和训练器的权重，记 $N_{b,t}$ 为 token $(b,t)$ 的实际版本滞后。
$$ j=\ell,\qquad \theta_{\text{roll}}=\theta_{\text{train}}=\theta^{(\ell)},\qquad N_{b,t}=0,\qquad \mu_R=\pi^{(\ell)}\ \text{(idealized)},\qquad r_{b,t}=1\ \text{a.s.} $$At this pre-update behavior point, $r_{b,t}\equiv 1$, so the ratio contributes no off-policy correction; under the usual on-policy estimator assumptions, the surrogate gradient at the behavior policy matches the corresponding policy-gradient estimator. The notation $\text{a.s.}$ in the display means that this equality holds with probability one under the idealized same-distribution abstraction. Here we abstract away perturbations induced by hardware and the train/inference engines. The price is throughput: the two engines never overlap, and one optimizer step costs (rollout) + (trainer) + (broadcast) in serial.
在更新前的行为策略点，$r_{b,t}\equiv 1$，因此比率项不再需要离策略校正；在通常的 on-policy 估计器假设下，行为策略处的替代梯度与相应策略梯度估计器一致。公式里的 $\text{a.s.}$ 表示在该理想化同分布抽象下，这个等式以概率 1 成立。此处我们不考虑硬件和训练推理引擎带来的扰动。代价是吞吐：两个引擎从不重叠，一个优化步的耗时是（rollout）+（训练）+（广播）的串行总和。
Asynchronous RL, batch-wise realized lag ≥ 1
批式异步 RL 实际 lag ≥ 1
Let $\ell$ be the rollout checkpoint index, $n$ the lag at batch handoff, $h$ the within-consumption update offset, and $H$ its maximum. Let $N_{b,t}$ be the realized lag of token $(b,t)$, and let $y_{b,t},s_{b,t}$ be the sampled token and its decoding context.
记 $\ell$ 为 rollout 检查点下标，记 $n$ 为批次交接时的 lag，记 $h$ 为消费该批时的局部更新偏移，记 $H$ 为其最大值。记 $N_{b,t}$ 为 token $(b,t)$ 的实际版本滞后，记 $y_{b,t},s_{b,t}$ 为采样到的 token 及其解码上下文。
$$ \theta_{\text{roll}}\;=\;\theta^{(\ell)},\qquad j=\ell+n+h,\quad h\in\{0,\dots,H\},\qquad N_{b,t}=j-\ell=n+h,\qquad r_{b,t}\;=\;\frac{\pi^{(j)}(y_{b,t}\!\mid\!s_{b,t})}{\mu_R(y_{b,t}\!\mid\!s_{b,t})}. $$All tokens share the behavior checkpoint, while their realized lag at evaluation is $N_{b,t}=n+h$. Write $d_{b,t}=\log r_{b,t}$, $\Delta_{b,t}^{\pi}=\log\pi^{(j)}-\log\pi^{(\ell)}$, and $\Delta_{b,t}^{\mathrm{impl}}=\log\pi^{(\ell)}-\log\mu_R$. Then the observed staleness splits into a policy-lag term and an implementation-mismatch term:
所有 token 共用行为检查点，而计算时的实际 lag 为 $N_{b,t}=n+h$。记 $d_{b,t}=\log r_{b,t}$、$\Delta_{b,t}^{\pi}=\log\pi^{(j)}-\log\pi^{(\ell)}$、$\Delta_{b,t}^{\mathrm{impl}}=\log\pi^{(\ell)}-\log\mu_R$。于是观测陈旧度可拆成策略版本差项与实现失配项：
$$ \log\pi^{(j)}-\log\mu_R\;=\;\log\pi^{(j)}-\log\pi^{(\ell)}\;+\;\log\pi^{(\ell)}-\log\mu_R. $$Here $\log\pi^{(\ell)}$ is Megatron's trainer-side reference at the rollout weights; the Notation panel and §4.f write the same quantity as $\tilde\mu_R$. This two-term equality is exact when all log-probs are evaluated at $(y_{b,t},s_{b,t})$. Engine kernels, MoE routing, and hardware are useful diagnostic sources for $\Delta_{b,t}^{\mathrm{impl}}$, but without additional intermediate distributions they are not a unique additive decomposition.
其中 $\log\pi^{(\ell)}$ 是 Megatron 在 rollout 权重下的 trainer-side reference；在记号面板与 §4.f 中，同一个量也写作 $\tilde\mu_R$。当所有对数概率都在 $(y_{b,t},s_{b,t})$ 上计算时，上述二项分解是精确的。引擎算子、MoE 路由与硬件可以作为 $\Delta_{b,t}^{\mathrm{impl}}$ 的诊断来源，但若不额外定义中间分布，就不能把它们声称为唯一的精确加法分解。
$$ d_{b,t}=\log r_{b,t}=\Delta_{b,t}^{\pi}+\Delta_{b,t}^{\mathrm{impl}}. $$Diagnostics诊断 What can contribute to the observed staleness? 哪些因素可能贡献观测陈旧度？
Policy-lag contribution $\Delta^{\pi}$
策略版本差贡献 $\Delta^{\pi}$
With version lag $n$, rollouts are produced under $\theta^{(\ell)}$ and consumed under $\theta^{(\ell+n)}$. If $g^{(k)}$ denotes the actual parameter-update direction and $\eta$ its step size, a first-order expansion gives
当版本差为 $n$ 时，rollout 由 $\theta^{(\ell)}$ 生成，并在 $\theta^{(\ell+n)}$ 下被消费。若 $g^{(k)}$ 表示实际参数更新方向、$\eta$ 为步长，一阶展开给出
$$\Delta_{b,t}^{\pi}\approx\sum_{k=\ell}^{\ell+n-1}\eta\,\big\langle\nabla_\theta\log\pi(y_{b,t}\!\mid\!s_{b,t}),\;g^{(k)}\big\rangle.$$This first-order expansion shows how earlier optimizer directions enter the later log-ratio. It does not by itself prove a non-zero mean: reward-directional drift is an empirical pattern in our runs, not a consequence of every $g^{(k)}$ sharing one objective.
这一阶展开说明了早先的优化方向如何进入后续对数比；它本身不能证明均值非零。朝奖励方向的漂移是我们实验中的经验模式，并非“所有 $g^{(k)}$ 共享同一目标”就必然推出的结论。
Engine mismatch $\Delta^{E}$
引擎失配 $\Delta^{E}$
Under identical weights, SGLang and Megatron can still disagree because of fused-kernel, attention-kernel, KV-cache, and numerical-precision differences. The relative size of this source is implementation- and model-dependent, which leads to training-inference mismatch.
即使权重完全相同，SGLang 与 Megatron 仍可能因融合算子、注意力算子、KV-cache 与数值精度不同而产生差异；该来源的相对大小取决于具体实现与模型从而导致训练与推理之间的不匹配，从而导致训练与推理之间的不一致。
Router mismatch $\Delta^{R}$ (MoE only)
路由失配 $\Delta^{R}$（仅 MoE）
MoE output at layer $m$ is given below, where $M$ is the total number of experts and $K$ is the number selected by Top-K:
MoE 在第 $m$ 层的输出如下，其中 $M$ 是专家总数，$K$ 是 Top-K 选中的专家数：
$$\mathbf h_{\mathrm{out}}=\sum_{e=1}^M g_e\,\mathcal E_e(\mathbf h),\quad g_e=\frac{I_e\exp(z_e)}{\sum_{e'}I_{e'}\exp(z_{e'})},\quad I_e=\mathbf 1[e\in\text{Top-K}(\mathbf z)].$$The Top-K mask $I$ is a hard function of the router logits; near a routing tie, a small engine difference can change the selected expert subset. Policy lag can amplify this because both the layer input and router weights may move.
Top-K 掩码 $I$ 是路由 logits 的硬函数；在路由分数接近打平时，很小的引擎差异就可能改变专家集合。策略版本差会进一步放大这一现象，因为层输入与路由权重都可能变化。
Hardware-dependent numerical mismatch $\Delta^{H}$
硬件相关数值失配 $\Delta^{H}$
Even with identical code and weights, device-specific kernels and reduction order can change numerical outputs. Its magnitude must be measured for the concrete stack; no universal ordering across devices is implied.
即使代码与权重相同，设备相关算子与归约顺序仍可能改变数值输出。其大小必须在具体软硬件栈上测量，本文不声称不同设备之间存在普适的数量级排序。
Fully asynchronous RL staleness is a random variable
完全异步 RL 陈旧度是随机变量
The dropdown below summarizes the steady-state mean-staleness model and the staleness formulation used for .
下方下拉栏总结了中使用的稳态平均陈旧度模型与陈旧度形式化定义。
Formulas公式 Fully Async-RL: mean staleness & staleness formulation 完全异步强化学习：平均陈旧度与形式化定义
Before reading the mean-staleness formula, we define the symbols that appear in it. Let $n$ denote the realized version lag of a randomly selected active token (a token-level copy of $N_{b,t}$), and let $\bar n=\mathbb E[n]$ denote its mean. We split that mean into a pre-queue contribution $\bar n_{\mathrm{PQS}}$ and an in-queue contribution $\bar n_{\mathrm{IQS}}$. The argument $\rho_{\mathrm{sys}}$ is the system throughput ratio between rollout and training. The symbol $C$ is the rollout concurrency, $M_{\mathrm{tail}}$ is the tail generation length measured in tokens, $B$ is the trainer microbatch size in responses, $Q$ is the queue capacity in responses, and $q_Q=Q/B$ is the queue capacity normalized by one trainer pull.
在读下面的平均陈旧度公式之前，先定义其中出现的符号。记 $n$ 为随机抽取的活跃 token 所对应的实际版本滞后（即 $N_{b,t}$ 在 token 层级上的副本），记 $\bar n=\mathbb E[n]$ 为它的均值。我们把这个均值拆成队列前贡献 $\bar n_{\mathrm{PQS}}$ 与队列中贡献 $\bar n_{\mathrm{IQS}}$。式中的 $\rho_{\mathrm{sys}}$ 表示 rollout 与训练之间的系统吞吐比。符号 $C$ 表示 rollout 并发数，$M_{\mathrm{tail}}$ 表示以 token 数计的尾部生成长度，$B$ 表示训练器每次拉取的 microbatch 响应数，$Q$ 表示队列容量（以响应数计），而 $q_Q=Q/B$ 表示按一次训练器拉取大小归一化后的队列容量。
Mean staleness. We formulize the token-level mean lag as
平均陈旧度。我们把 token 层级的平均 lag 形式化为
$$ \begin{aligned} \bar n\;\triangleq\;\mathbb E[n]\;&=\;\bar n_{\mathrm{PQS}}(\rho_{\mathrm{sys}})+\bar n_{\mathrm{IQS}}(\rho_{\mathrm{sys}}),\\[4pt] &=\;\begin{cases}\dfrac{C\,M_{\mathrm{tail}}}{B}+\rho_{\mathrm{sys}}, & \rho_{\mathrm{sys}}<1,\\[8pt]\dfrac{C\,M_{\mathrm{tail}}}{\rho_{\mathrm{sys}}B}+\dfrac{2q_Q+\rho_{\mathrm{sys}}-1}{2\rho_{\mathrm{sys}}}, & \rho_{\mathrm{sys}}>1,\end{cases}\qquad q_Q\triangleq Q/B. \end{aligned} $$This formulation is not a distribution-free identity. The critical case $\rho_{\mathrm{sys}}=1$ must be handled by the model's boundary convention or an appropriate one-sided limit.
这一形式化表达不是与分布无关的恒等式。临界情形 $\rho_{\mathrm{sys}}=1$ 需要按模型边界约定或合适的单侧极限处理。
Staleness formulation. Following Applied Compute (2026)[19], let $n$ be the realized lag of a randomly selected active token (a token-level copy of $N_{b,t}$), and split it into pre-queue and in-queue components:
陈旧度形式化。沿用 Applied Compute（2026）[19]的做法，令 $n$ 表示随机抽取活跃 token 的实际 lag（即 $N_{b,t}$ 在 token 层级上的副本），并把它拆成队列前与队列中两部分：
$$ n\;\triangleq\;\underbrace{n_{\mathrm{PQS}}}_{\substack{\text{pre-queue: policy versions}\\ \text{elapsed while generating}}}\;+\;\underbrace{n_{\mathrm{IQS}}}_{\substack{\text{in-queue: policy versions}\\ \text{elapsed while waiting}}}. $$Introduce the system-level quantities used by the steady-state model for the mean $\bar n=\mathbb E[n]$:
引入稳态模型用来描述均值 $\bar n=\mathbb E[n]$ 的系统级量：
Here $v_R$ is the aggregate rollout throughput (tokens/s), $v_T$ is the trainer throughput (tokens/s), and $\rho_{\mathrm{sys}}=v_R/v_T$ is the rollout-to-trainer throughput ratio. $C$ is the rollout concurrency, $B=GS$ is the microbatch size in rollouts, and $Q$ (with queue factor $q_Q=Q/B$) is the max queue capacity. $M_{\mathrm{tail}}$ measures the response-length tailness within a prompt group, where $L_i$ is response $i$'s length and $L$ is a generic response length.
其中 $v_R$ 为总 rollout 吞吐（tokens/s），$v_T$ 为 trainer 吞吐（tokens/s），$\rho_{\mathrm{sys}}=v_R/v_T$ 为 rollout 与 trainer 的吞吐比。$C$ 为 rollout 并发度，$B=GS$ 为 rollout 中的微批大小，$Q$ 为最大队列容量（队列因子 $q_Q=Q/B$）。$M_{\mathrm{tail}}$ 衡量同一 prompt 组内响应长度的尾部程度，其中 $L_i$ 为第 $i$ 条响应的长度，$L$ 为通用响应长度。
Pre-queue and in-queue components then read
于是队列前与队列中两部分分别为
$$ \begin{aligned} \bar n_{\mathrm{PQS}}(\rho_{\mathrm{sys}})&\;=\;\begin{cases}\dfrac{C\,M_{\mathrm{tail}}}{B}, & \rho_{\mathrm{sys}}<1\ (\text{rollout-bound}),\\[6pt]\dfrac{C\,M_{\mathrm{tail}}}{\rho_{\mathrm{sys}}B}, & \rho_{\mathrm{sys}}>1\ (\text{train-bound}),\end{cases}\\[10pt] \bar n_{\mathrm{IQS}}(\rho_{\mathrm{sys}})&\;=\;\begin{cases}\rho_{\mathrm{sys}}, & \rho_{\mathrm{sys}}<1,\\[4pt]\dfrac{2q_Q+\rho_{\mathrm{sys}}-1}{2\rho_{\mathrm{sys}}}, & \rho_{\mathrm{sys}}>1.\end{cases} \end{aligned} $$Scope of this article
本文的研究范围
The three regimes above establish the object of study. Concretely, this blog
上述三种范式确定了研究对象。具体而言，本文
- separates version lag (a deterministic configured $n$ or a token-level $N_{b,t}$ with mean $\bar n$) from the observed per-token log-ratio staleness, and uses policy / engine / router effects as diagnostic hypotheses rather than a unique decomposition (Part 02);
- 区分版本差（确定的配置值 $n$，或 token 级随机 lag $N_{b,t}$ 及其均值 $\bar n$）与观测逐 token 对数比陈旧度，并把策略 / 引擎 / 路由效应作为诊断假说，而非唯一分解（第 02 部分）；
- proposes a primary solution — the Staleness-Adaptive Trust Region (SAT) of Part 03 — that couples a directional clip radius to the observed log-ratio staleness. It tightens PPO's sampled-ratio surrogate on tokens with high observed staleness; true $D_{\mathrm{TV}}$ and improvement interpretations remain explicitly conditional;
- 提出核心解法 —— 第 03 部分的陈旧度自适应信任域（SAT）：让方向性裁剪半径与观测对数比陈旧度耦合。它在观测陈旧度较高的 token 上收紧 PPO 的采样比率替代机制；真实 $D_{\mathrm{TV}}$ 与策略改进解释仍明确是条件性的；
- surveys the existing async-RL stability approaches as different choices of behavior policy, target-policy definition, ratio truncation, mask, clip radius, threshold on the same $\log r_{b,t}$ (Part 04).
- 综述现有的异步 RL 稳定方法 —— 它们分别是在同一个 $\log r_{b,t}$ 上对行为策略、目标策略定义、比率截断、掩码、裁剪半径、阈值的不同选择（第 04 部分）。
Notation记号 Notation & Definitions 符号与定义
The following notation is fixed for the remainder of the article.
下列记号在本文其余部分固定不变。
- $b,\;t,\;j$
- Response index $b\in\{1,\dots,B\}$; token index $t\in\{1,\dots,T_b\}$; absolute trainer-update index $j$. A local checkpoint-window offset is written $h$, never $j$.
- 响应下标 $b\in\{1,\dots,B\}$；token 下标 $t\in\{1,\dots,T_b\}$；绝对训练更新下标 $j$。检查点窗口内的局部偏移统一写作 $h$，不再写成 $j$。
- $\ell_{b,t},\;N_{b,t},\;n,\;\bar n$
- Rollout weight version for token $(b,t)$; its realized lag $N_{b,t}=j-\ell_{b,t}$ when version indices are comparable; a deterministic configured or batch-handoff lag $n$; and the token-averaged mean $\bar n=\mathbb E[n]=\mathbb E_{b,t}[N_{b,t}]$, where $n$ denotes the lag of a randomly selected active token. When a batch shares one rollout checkpoint, $\ell_{b,t}\equiv\ell$. In a fixed, single-update batch $N_{b,t}=n$; with batch reuse it can become $n+h$.
- token $(b,t)$ 的 rollout 权重版本；版本下标可比时的实际 lag $N_{b,t}=j-\ell_{b,t}$；确定性的配置值或批次交接 lag $n$；以及 token 平均值 $\bar n=\mathbb E[n]=\mathbb E_{b,t}[N_{b,t}]$，其中 $n$ 表示随机抽取活跃 token 的 lag。若整批共享同一个 rollout 检查点，则 $\ell_{b,t}\equiv\ell$。固定节拍且仅更新一次时 $N_{b,t}=n$；复用批次时可增长为 $n+h$。
lag=1/8in figures图中的lag=1/8- The configured experimental pipeline-lag label (the weight-broadcast / batch-handoff setting), not the autoregressive state $s_{b,t}$ and not a claim that every token's realized $N_{b,t}$ remains constant.
- 实验流水线的配置标签（权重广播 / 批次交接设置），不是自回归状态 $s_{b,t}$，也不表示每个 token 的实际 $N_{b,t}$ 始终不变。
- $\rho_{\mathrm{sys}},\;q_Q$
- System throughput ratio $\rho_{\mathrm{sys}}=v_R/v_T$ and normalized queue capacity $q_Q=Q/B$ in the steady-state staleness model. The subscript distinguishes $\rho_{\mathrm{sys}}$ from GSPO's sequence ratio $\rho_b^{\mathrm{seq}}$.
- 稳态陈旧度模型中的系统吞吐比 $\rho_{\mathrm{sys}}=v_R/v_T$ 与归一化队列容量 $q_Q=Q/B$。下标用于区分 $\rho_{\mathrm{sys}}$ 与 GSPO 的序列比率 $\rho_b^{\mathrm{seq}}$。
- $\pi^{(j)}$, $\pi^{(\ell)}$
- Current train policy evaluated by Megatron at absolute update $j$: $\pi^{(j)}=\pi_{\theta^{(j)}}$. Relative to token $(b,t)$, its version lag is $N_{b,t}=j-\ell_{b,t}$. Likewise, $\pi^{(\ell)}$ is Megatron's evaluation at the rollout weights $\theta^{(\ell)}$ — i.e., the recomputation written $\tilde\mu_R$ below.
- 当前训练策略，即 Megatron 在绝对更新 $j$ 上计算的策略：$\pi^{(j)}=\pi_{\theta^{(j)}}$。相对 token $(b,t)$ 而言，其版本差为 $N_{b,t}=j-\ell_{b,t}$。同理，$\pi^{(\ell)}$ 表示 Megatron 在 rollout 权重 $\theta^{(\ell)}$ 下的计算结果，即下文记作 $\tilde\mu_R$ 的重算分布。
- $B,\;G,\;S,\;T_b$
- Responses per trainer microbatch; sampled responses per prompt (group size); prompts per microbatch, so $B=GS$; and length of response $b$. Formulas using $b=1,\dots,G$ locally re-index the $G$ candidate responses for one prompt.
- 每个训练微批的响应数；每个提示词的采样响应数（组大小）；每微批的提示词数，因此 $B=GS$；以及响应 $b$ 的长度。局部使用 $b=1,\dots,G$ 的公式表示对单个提示词的 $G$ 条候选响应重新编号。
- $x,\;y_b,\;s_{b,t}$
- Prompt; response $y_b=(y_{b,1},\dots,y_{b,T_b})$; autoregressive state $s_{b,t}=(x,y_{b,1},\dots,y_{b,t-1})$.
- 提示词；响应 $y_b=(y_{b,1},\dots,y_{b,T_b})$；自回归状态 $s_{b,t}=(x,y_{b,1},\dots,y_{b,t-1})$。
- $\mathcal V$
- Vocabulary (action space), of size $|\mathcal V|$.
- 词表（动作空间），大小为 $|\mathcal V|$。
- $\mu,\;\mu_{b,t},\;\mu_R$
- The actual autoregressive trajectory behavior $\mu$; its conditional distribution at token $(b,t)$, $\mu_{b,t}(\cdot\mid s_{b,t})$; and the fixed-batch shorthand $\mu_R$ when one rollout policy applies throughout. Fully async policy-level formulas use $\mu$ and its conditionals rather than assuming one fixed checkpoint.
- 真实自回归轨迹行为 $\mu$；其在 token $(b,t)$ 的条件分布 $\mu_{b,t}(\cdot\mid s_{b,t})$；以及整批始终使用同一 rollout 策略时的简写 $\mu_R$。完全异步的策略级公式使用 $\mu$ 及其条件分布，不假设单一固定检查点。
- $\tilde\mu_{b,t},\;\tilde\mu_R$
- Megatron-side recomputation at token $(b,t)$'s rollout weights, and its fixed-batch shorthand when one rollout checkpoint applies throughout. It need not equal the engine behavior $\mu_{b,t}$ because kernels, precision, and MoE routing may differ.
- Megatron 在 token $(b,t)$ 的 rollout 权重下得到的重算分布，以及整批共用一个 rollout 检查点时的简写。由于算子、精度与 MoE 路由可能不同，它不必等于真实引擎行为 $\mu_{b,t}$。
- $r_{b,t}$
- Sampled token ratio $r_{b,t}\triangleq\pi^{(j)}(y_{b,t}\!\mid\!s_{b,t})/\mu_{b,t}(y_{b,t}\!\mid\!s_{b,t})$. The action-wise ratio is $r_{b,t}(a)$; this distinction matters in $D_{\mathrm{TV}}$ bounds.
- 采样 token 的比率 $r_{b,t}\triangleq\pi^{(j)}(y_{b,t}\!\mid\!s_{b,t})/\mu_{b,t}(y_{b,t}\!\mid\!s_{b,t})$。全动作比率写作 $r_{b,t}(a)$；$D_{\mathrm{TV}}$ 界中必须区分两者。
- $\hat A_{b,t}$
- Advantage estimate (e.g. GRPO group-baseline).
- 优势估计（如 GRPO 的组基线）。
- $R(y)$
- Sequence-level scalar reward $R(y)\in[-\xi,\xi]$ with $\xi=\max_y|R(y)|$.
- 序列级标量奖励 $R(y)\in[-\xi,\xi]$，其中 $\xi=\max_y|R(y)|$。
- $d_{b,t},\;d_{b,t}^{\mathrm{sg}},\;|d_{b,t}|$
- Observed log-ratio staleness $d_{b,t}\triangleq\log\pi^{(j)}(y_{b,t}\mid s_{b,t})-\log\mu_{b,t}(y_{b,t}\mid s_{b,t})=\log r_{b,t}$, its stop-gradient copy $d_{b,t}^{\mathrm{sg}}=\operatorname{sg}[d_{b,t}]$, and SAT's staleness proxy $|d_{b,t}|$. The value mixes policy lag with engine/router/hardware mismatch; it is neither the version lag $N_{b,t}$ nor a local $D_{\mathrm{TV}}$ contribution. For action $a$, the corresponding summand in full $D_{\mathrm{TV}}$ is probability-mass weighted: $\tfrac12\mu_{b,t}(a\mid s)|r_{b,t}(a)-1|=\tfrac12|\pi^{(j)}(a\mid s)-\mu_{b,t}(a\mid s)|$.
- 观测对数比陈旧度 $d_{b,t}\triangleq\log\pi^{(j)}(y_{b,t}\mid s_{b,t})-\log\mu_{b,t}(y_{b,t}\mid s_{b,t})=\log r_{b,t}$、其停止梯度副本 $d_{b,t}^{\mathrm{sg}}=\operatorname{sg}[d_{b,t}]$，以及 SAT 使用的陈旧度代理 $|d_{b,t}|$。该数值混合了策略版本差与引擎 / 路由 / 硬件失配，既不等于版本陈旧度 $N_{b,t}$，也不是局部 $D_{\mathrm{TV}}$ 贡献。动作 $a$ 在完整 $D_{\mathrm{TV}}$ 中对应的求和项带有概率质量权重：$\tfrac12\mu_{b,t}(a\mid s)|r_{b,t}(a)-1|=\tfrac12|\pi^{(j)}(a\mid s)-\mu_{b,t}(a\mid s)|$。
- $c_{-,b,t},\;c_{+,b,t}$
- Effective SAT contraction factors after the quantile gate: a factor equals its directional kernel value only when the gate fires, and equals $1$ otherwise. These, rather than the raw kernels, determine the implemented clip radii.
- 分位数门控后的 SAT 有效收缩因子：仅在门控触发时取方向核的值，否则取 $1$。真正决定实现中裁剪半径的是 $c_\pm$，而不是原始核值。
- $\mathcal T_{\mathrm{act}},\;q$
- Active, non-padding token positions in the current microbatch, and the $0.90$ quantile of $|d_{b,t}|$ over that set. $q$ is unrelated to the queue factor $q_Q$.
- 当前微批中的活跃、非 padding token 位置集合，以及该集合上 $|d_{b,t}|$ 的 $0.90$ 分位数。$q$ 与队列因子 $q_Q$ 无关。
- $D_{\mathrm{TV}},\;\bar D_{\mathrm{TV}}$
- Per-state $D_{\mathrm{TV}}$, namely $D_{\mathrm{TV}}(\mu_R,\pi)[s_t]=\tfrac12\sum_a|\mu_R(a\!\mid\!s_t)-\pi(a\!\mid\!s_t)|$, and the occupancy-weighted token sum $\bar D_{\mathrm{TV}}\triangleq\mathbb E_{y\sim\mu_R}[\sum_t D_{\mathrm{TV}}[s_t]]$ used by (2.3).
- 逐状态 $D_{\mathrm{TV}}$ 距离，即 $D_{\mathrm{TV}}(\mu_R,\pi)[s_t]=\tfrac12\sum_a|\mu_R(a\!\mid\!s_t)-\pi(a\!\mid\!s_t)|$，以及 (2.3) 使用的占用加权 token 和 $\bar D_{\mathrm{TV}}\triangleq\mathbb E_{y\sim\mu_R}[\sum_t D_{\mathrm{TV}}[s_t]]$。
- $\mathcal J,\;L'_{\nu},\;\mathcal S,\;\mathcal L$
- True expected return $\mathcal J$; for any trajectory behavior $\nu$, the population surrogate $L'_{\nu}(\pi)=\mathbb E_{y\sim\nu}[R(y)\sum_t(r_t-1)]$ used in the DPPO bound; a sampled surrogate $\mathcal S$ that is maximized; and the minimized loss $\mathcal L=-\mathcal S$. Their signs are kept explicit below.
- 真实期望回报 $\mathcal J$；对任意轨迹行为 $\nu$，DPPO 下界使用的总体替代目标 $L'_{\nu}(\pi)=\mathbb E_{y\sim\nu}[R(y)\sum_t(r_t-1)]$；待最大化的采样替代目标 $\mathcal S$；以及待最小化的损失 $\mathcal L=-\mathcal S$。下文显式区分这些符号及正负号。
- $\mathcal S_{b,t},\;\mathcal S_{b,t}^{\mathrm{PPO}},\;\mathcal S_{b,t}^{\mathrm{SAT}}$
- Per-token sampled surrogate maximized at token $(b,t)$, built from the sampled ratio $r_{b,t}$ and advantage $\hat A_{b,t}$. The batch objective aggregates these tokens: $\mathcal S=\mathbb E_{b,t}[\mathcal S_{b,t}]$ and $\mathcal L_{b,t}=-\mathcal S_{b,t}$. The PPO variant uses the fixed-radius pessimistic min, $\mathcal S_{b,t}^{\mathrm{PPO}}=\min\!\big(r_{b,t}\hat A_{b,t},\ \operatorname{clip}(r_{b,t},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}})\hat A_{b,t}\big)$; the SAT variant replaces the fixed radii with the staleness-adaptive $\tilde\varepsilon_{\text{low},b,t},\tilde\varepsilon_{\text{high},b,t}$ from $c_{\pm,b,t}$ (Eq. (3.5)–(3.6)). $\mathcal S_{b,t}$ is a sampled-token quantity and, unlike $L'_{\nu}$, it is not by itself a policy-improvement lower bound.
- 逐 token 采样替代目标，在 token $(b,t)$ 上待最大化，由采样比率 $r_{b,t}$ 与优势 $\hat A_{b,t}$ 构成。批次目标为其聚合：$\mathcal S=\mathbb E_{b,t}[\mathcal S_{b,t}]$，$\mathcal L_{b,t}=-\mathcal S_{b,t}$。PPO 变体使用固定半径的悲观 $\min$：$\mathcal S_{b,t}^{\mathrm{PPO}}=\min\!\big(r_{b,t}\hat A_{b,t},\ \operatorname{clip}(r_{b,t},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}})\hat A_{b,t}\big)$；SAT 变体则把固定半径替换为由 $c_{\pm,b,t}$ 导出的陈旧度自适应半径 $\tilde\varepsilon_{\text{low},b,t},\tilde\varepsilon_{\text{high},b,t}$（式 (3.5)–(3.6)）。$\mathcal S_{b,t}$ 是采样 token 量，与 $L'_{\nu}$ 不同，它本身并不构成策略改进的下界。
- $\bar d_\pi$
- The reported training–inference mismatch: the experiment's mean absolute sampled-token log-prob difference, averaged over active tokens. It is a diagnostic statistic, not $D_{\mathrm{TV}}$ or a version count.
- 实验报告的训练–推理失配：在活跃 token 上平均的采样 token 对数概率绝对差。它是诊断统计量，不等于 $D_{\mathrm{TV}}$，也不是版本计数。
- $\varepsilon,\varepsilon_{\text{low}},\varepsilon_{\text{high}},\delta$
- PPO clip radii; DPPO / DRPO / TRPO trust-region threshold in $D_{\mathrm{TV}}$ / KL space.
- PPO 裁剪半径；DPPO / DRPO / TRPO 在 $D_{\mathrm{TV}}$ / KL 空间中的信任域阈值。
- $M_{b,t},\;w_{b,t}$
- Per-token gate $M_{b,t}\in\{0,1\}$ or weight $w_{b,t}\in\mathbb R$ for the policy loss.
- 策略损失中的逐 token 门控 $M_{b,t}\in\{0,1\}$ 或权重 $w_{b,t}\in\mathbb R$。
Part 02
第 02 部分
Staleness Analysis in Asynchronous RL
异步 RL 中的陈旧度分析
Part 01 identified the three pipeline regimes; this part quantifies the reinforcement-learning cost of that decoupling. Synchronous RL accepts fully serialized throughput in exchange for strictly on-policy data (in the idealized same-distribution analysis, $\mu_{b,t}=\pi^{(j)}$ and $r_{b,t}\equiv 1$, while engine-level numerical mismatch is taken to be negligible at this scale). Asynchronous RL instead runs SGLang and Megatron in parallel on separate GPU pools, so the trainer continues updating while rollout data are still in flight, which substantially improves training efficiency: the model state is synchronized between the training and inference engines once every configured handoff interval, denoted $n$, while a token's realized lag is $N_{b,t}=j-\ell_{b,t}$. The cost is a systematic parameter mismatch:
第 01 部分确定了三种流水线范式；本部分量化这种解耦对于强化学习的代价。同步 RL 用完全串行的吞吐，换来严格 on-policy 的数据（在理想化同分布分析中 $\mu_{b,t}=\pi^{(j)}$ 且 $r_{b,t}\equiv 1$；引擎数值失配量级可以忽略不计）。异步 RL 则让 SGLang 与 Megatron 在各自的 GPU 池上并行运行，训练器在 rollout 数据尚在途中时继续更新，这无疑大幅度提高了训练效率：每过 lag 步将训练以及推理引擎之间的模型配置同步一次，配置的交接 lag 可记为 $n$，而 token 的实际 lag 为 $N_{b,t}=j-\ell_{b,t}$。代价是系统性的参数失配：
$$ \Delta\boldsymbol\theta_{b,t}^{(j)}\;=\;\boldsymbol\theta^{(j)}\;-\;\boldsymbol\theta^{(\ell_{b,t})},\qquad N_{b,t}=j-\ell_{b,t}\ge 0, $$For the loss, the key consequence is that training is no longer strictly on-policy: behavior data are generated under stale rollout parameters and are then optimized under the current trainer parameters. In other words, the update becomes off-policy at training time, though not offline RL in the fixed-dataset sense. This necessarily changes the token-level ratio. Staleness can be formulized as the observed token-level log-ratio.:
对于损失函数的影响在于它将原本在线的强化学习强制变成离线强化学习，这毋庸置疑地改变了更新时 token-ratio，因而陈旧度可以被表示为观测到的token-level对数比：
$$ d_{b,t}\;=\;\log r_{b,t}\;=\;\log\pi^{(j)}(y_{b,t}\!\mid\!s_{b,t})\;-\;\log\mu_{b,t}(y_{b,t}\!\mid\!s_{b,t}). $$Why $D_{\mathrm{TV}}$ motivates ratio control — and where sampled clipping falls short$D_{\mathrm{TV}}$ 为何引出比率控制 —— 以及采样裁剪的边界
$$r_{b,t}(a)\;\triangleq\;\frac{\pi^{(j)}(a\mid s_{b,t})}{\mu_{b,t}(a\mid s_{b,t})},\qquad r_{b,t}\equiv r_{b,t}(y_{b,t}).\tag{2.1}$$ $$D_{\mathrm{TV}}\!\left(\mu_{b,t},\pi^{(j)}\right)[s_{b,t}]\;=\;\tfrac12\,\mathbb E_{a\sim\mu_{b,t}(\cdot\mid s_{b,t})}\!\left[\left|r_{b,t}(a)-1\right|\right].\tag{2.2}$$Equation (2.2) immediately gives the ideal ratio-to-$D_{\mathrm{TV}}$ bridge. If, at a fixed state, a hard bound on every action satisfies $|r_{b,t}(a)-1|\le\varepsilon$ for all $a$, then it follows that $D_{\mathrm{TV}}\!\left(\mu_{b,t},\pi^{(j)}\right)[s_{b,t}]\le\frac{\varepsilon}{2}$. With asymmetric hard bounds $r(a)\in[1-\varepsilon_{\mathrm{low}},1+\varepsilon_{\mathrm{high}}]$, the same argument gives $D_{\mathrm{TV}}\le\tfrac12\max\{\varepsilon_{\mathrm{low}},\varepsilon_{\mathrm{high}}\}$. This is an all-action statement, not a statement about one sampled token.
式 (2.2) 直接给出理想的token-ratio 和 $D_{\mathrm{TV}}$之间的推导关系。若在固定状态上，对每个动作都有硬约束 $|r_{b,t}(a)-1|\le\varepsilon$，则可得 $D_{\mathrm{TV}}\!\left(\mu_{b,t},\pi^{(j)}\right)[s_{b,t}]\le\frac{\varepsilon}{2}$。对于非对称裁剪 $r(a)\in[1-\varepsilon_{\mathrm{low}},1+\varepsilon_{\mathrm{high}}]$，同理可得 $D_{\mathrm{TV}}\le\tfrac12\max\{\varepsilon_{\mathrm{low}},\varepsilon_{\mathrm{high}}\}$。这是全动作结论，不是关于单个采样 token 的结论。
For the performance bound below, specialize to a fixed trajectory behavior $\mu_{b,t}\equiv\mu_R$; token-varying fully asynchronous behavior requires conditioning on the realized rollout schedule (or augmenting the state with scheduler/version information).
对下面的性能界，先专门考虑固定轨迹行为 $\mu_{b,t}\equiv\mu_R$；token 行为随时间变化的完全异步情形还需对已实现的 rollout 调度取条件（或把调度器 / 版本信息加入增广状态）。
$$\begin{gathered} \mathcal J(\pi^{(j)})-\mathcal J(\mu_R)\;\ge\;L'_{\mu_R}(\pi^{(j)})-4\xi\,\bar D_{\mathrm{TV}}(\mu_R,\pi^{(j)}),\\ \bar D_{\mathrm{TV}}\triangleq\mathbb E_{y\sim\mu_R}\!\left[\sum_tD_{\mathrm{TV}}[s_t]\right]. \end{gathered}\tag{2.3}$$A smaller $D_{\mathrm{TV}}$ term makes this lower bound less pessimistic, so the surrogate is a more trustworthy approximation; it does not by itself imply a better policy because $L'_{\mu_R}$ and optimization error also matter. Equation (2.2) assumes $\pi^{(j)}\ll\mu_{b,t}$; otherwise target mass outside behavior support must be handled separately. Equation (2.3) is the finite-horizon DPPO lower bound[5] under its bounded-reward and compatible trajectory/support assumptions. PPO clipping observes sampled actions and changes their loss; it does not enforce the all-action hard bound above.
更小的 $D_{\mathrm{TV}}$ 项会让这个下界不那么悲观，因此替代目标近似更可信；但它并不单独意味着策略更好，因为 $L'_{\mu_R}$ 与优化误差同样重要。式 (2.2) 假设 $\pi^{(j)}\ll\mu_{b,t}$；否则必须另行处理行为支撑之外的目标概率质量。式 (2.3) 是 DPPO[5] 在奖励有界、轨迹与支撑相容等假设下给出的有限视界下界。PPO 裁剪只观测采样动作并改变其损失，并不会执行上面的全动作硬约束。
Empirically the same picture is reflected by IS-deviation-token fraction, the training–inference log-prob absolute difference, and its KL counterpart: a naive GRPO[8] baseline at configured lag $=8$ produces roughly an order-of-magnitude larger IS deviation than at configured lag $=1$; in those runs this larger mismatch is associated with training instability after only a few good checkpoints, before the first epoch closes.
实证上，IS 偏移 token 占比、训练–推理对数概率绝对差与其 KL 对应量 三个指标呈现出相似图景：Vanilla GRPO[8] 基线在配置 lag $=8$ 时的 IS 偏移比配置 lag $=1$ 时大约高一个数量级；在这些运行中，更大的失配与训练在首个 epoch 结束前、仅取得几个较好检查点后失稳相关。
Section 2.1第 2.1 节 Training instability associated with staleness与陈旧度相关的训练不稳定
The theoretical bound above is only the beginning of the story: its inequality remains valid under its assumptions, but its numerical lower bound becomes looser as the divergence penalty grows. In our runs, sufficiently large lag is also associated with loss of training stability. The main diagnostics are:
上面的理论界只是故事的开头：在其假设成立时，不等式本身仍然有效，但散度惩罚增大时，数值下界会变松。在我们的实验中，足够大的版本差还与训练稳定性丧失相关。主要诊断指标如下：
These symptoms are much weaker in the reported low-lag runs and reappear in the high-lag comparisons. They motivate the rest of the article: Part 03 introduces SAT, whose per-token nominal clip radii respond to the observed staleness distribution, and Part 04 catalogues the existing async-RL stabilizers under the same trust-region lens. The experiments test whether these mechanisms improve stability; the conditional theory in §3.2 should not be read as a causal explanation of every run.
在已报告的低版本差运行中，这些症状明显更弱，并在高版本差比较中重新出现。它们引出本文的其余部分：第 03 部分引入 SAT —— 其逐 token 名义裁剪半径随观测陈旧度分布变化；第 04 部分则在同一信任域视角下梳理现有异步 RL 稳定方法。实验用于检验这些机制是否改善稳定性；§3.2 的条件性理论不应被解读为对每次运行的因果解释。
Part 03
第 03 部分
SAT — Staleness-Adaptive Trust Region
SAT — 陈旧度自适应信任域
SAT (Staleness-Adaptive Trust Region) is a trust-region-motivated sampled-ratio clipping plug-in for PPO-like objectives. It turns static radii $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})$ into per-token directional radii using the observed staleness proxy $|d_{b,t}|=|\log r_{b,t}|$. The construction is batch-relative, direction-aware, and only contracts tokens beyond an adaptive quantile gate. Its name describes the motivation; the implementation is not a hard policy-level trust-region projection.
SAT（陈旧度自适应信任域）是面向 PPO 类目标、由信任域思想驱动的采样比率裁剪插件。它使用观测陈旧度代理 $|d_{b,t}|=|\log r_{b,t}|$，把静态半径 $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})$ 改成逐 token 的方向性半径。该构造以当前微批为相对参考、具有方向性，并且只收缩超过自适应分位数门控的 token。名称表达的是设计动机；实现本身并不是策略级硬信任域投影。
3.1 From observed staleness to an adaptive clip
3.1 从观测陈旧度到自适应裁剪
Equations (2.2)–(2.3) motivate keeping the train policy close to the behavior policy: an ideal support-wide ratio bound controls training-inference divergence $D_{\mathrm{TV}}$, and smaller divergence tightens the performance approximation. PPO replaces that intractable hard constraint with a fixed-radius sampled clipped objective. It is a soft guardrail: seeing one action per visited state cannot certify the full vocabulary distribution, and clipping does not project the learned policy into a ratio-feasible set.
式 (2.2)–(2.3) 给出了让训练策略靠近行为策略的动机：理想的全支撑比率界能够控制训练-推理散度 $D_{\mathrm{TV}}$，而更小的散度会收紧性能近似。PPO 用固定半径的采样裁剪目标替代了这个难以直接执行的硬约束。它只是一道软护栏：每个已访问状态只看到一个采样动作，无法验证完整词表分布；裁剪也不会把学习后的策略投影到比率可行集。
Comparing with vanilla PPO clipping, SAT provides a token-level clipping mechanism behind suppressing future staleness growth.
- Token-level staleness signal. Larger $|d_{b,t}|$ indicates higher training staleness and correlates with instability. SAT therefore narrows the sign-selected nominal endpoint on the high-staleness tail of the current training batch. What it uses here is an observable relative-staleness signal: actions with larger token-level ratios are often the ones that trigger training collapse.
- Staleness-based effective contraction factors By imposing a tighter trust region with Staleness-based effective contraction factors $c_{\pm,b,t}$, SAT suppresses divergent gradient updates and uses $|d_{b,t}|$ as a practical proxy for staleness. On the additionally truncated portion, SAT further ensures that the next-step sampling-ratio deviation stays no larger than under PPO clipping, providing the most direct local mechanism for curbing future growth of training staleness.
与 vanilla PPO clipping 相比，SAT 提供了一种 token-level 的裁剪机制，其背后对应着抑制未来训练陈旧度增长的局部机制。
- Token-level 陈旧度信号。较大的 $|d_{b,t}|$ 往往意味着更高的训练陈旧度，并且与训练不稳定性相关。因此，SAT 会在当前训练批次中陈旧度较高的长尾 token 上收窄由符号选中的名义端点。这里使用的是一种可观测的相对陈旧度信号：那些具有较大 token-level ratio 的动作，往往更容易诱发训练崩溃。
- 基于陈旧度的有效收缩因子。SAT 通过引入基于陈旧度的收缩因子 $c_{\pm,b,t}$ 来施加更紧的 trust region，抑制发散的梯度更新，并将 $|d_{b,t}|$ 作为一种实用的陈旧度代理。对于新增截断的那部分 token，SAT 进一步保证下一步的采样比率偏离不超过 PPO 裁剪，从而给出抑制未来训练陈旧度增长的最直接局部机制。
3.2 SAT pipeline
3.2 SAT 工作流程
PPO-like clipping vs. w/ SAT — one-line comparisonPPO-like 裁剪 vs. w/ SAT — 一行对比
PPO-like clipping uses a fixed nominal clip interval:
PPO-like 裁剪使用固定名义裁剪区间：
$$\begin{gathered}\mathcal S_{b,t}^{\text{PPO}}\;=\;\min\!\Big(r_{b,t}\hat A_{b,t},\ \operatorname{clip}\!\big(r_{b,t},\ {\color{#0a7a3a}1-\varepsilon_{\text{low}}},\ {\color{#0a7a3a}1+\varepsilon_{\text{high}}}\big)\hat A_{b,t}\Big),\\ \mathcal L_{b,t}^{\text{PPO}}=-\mathcal S_{b,t}^{\text{PPO}}.\end{gathered}$$SAT replaces it with a per-token, staleness-adaptive nominal interval:
SAT 把它换成逐 token、陈旧度自适应的名义区间：
$$\begin{gathered}\mathcal S_{b,t}^{\text{SAT}}\;=\;\min\!\Big(r_{b,t}\hat A_{b,t},\ \operatorname{clip}\!\big(r_{b,t},\ {\color{#1c5cab}1-\varepsilon_{\text{low}}\,c_{-,b,t}},\ {\color{#1c5cab}1+\varepsilon_{\text{high}}\,c_{+,b,t}}\big)\hat A_{b,t}\Big),\\ \mathcal L_{b,t}^{\text{SAT}}=-\mathcal S_{b,t}^{\text{SAT}}.\end{gathered}$$Green: PPO's fixed nominal interval. Blue: SAT's contracted nominal interval, driven by the current microbatch's sampled staleness distribution (definitions below).
绿色：PPO 的固定名义区间。蓝色：由当前微批采样陈旧度分布驱动的 SAT 收缩名义区间（定义见下）。
Token-level staleness signal
Token-level 陈旧度信号
Eq. (3.1)We begin with the observable mismatch carried by each sampled token in the current batch. Measure staleness on the sampled action and detach it before constructing the gate and clip limits:
我们先从当前批次每个采样 token 所携带的可观测失配出发。在构造门控与裁剪边界前，先在采样动作上测量陈旧度并停止其梯度：
$$d_{b,t}=\log\pi^{(j)}(y_{b,t}\!\mid\!s_{b,t})-\log\mu_{b,t}(y_{b,t}\!\mid\!s_{b,t})=\log r_{b,t},\qquad d_{b,t}^{\mathrm{sg}}\triangleq\operatorname{sg}[d_{b,t}].\tag{3.1}$$Here $\operatorname{sg}$ denotes stop-gradient; $d_{b,t}^{\mathrm{sg}}$ has the same numerical value as $d_{b,t}$ but no derivative. Otherwise derivatives through the staleness, quantile, and moving clip limits would change the PPO-like gradient analysed below.
$\operatorname{sg}$ 表示停止梯度；$d_{b,t}^{\mathrm{sg}}$ 与 $d_{b,t}$ 数值相同，但不携带导数。否则梯度会穿过陈旧度、分位数与移动裁剪边界，实际梯度就不再是下文分析的 PPO 类梯度。
Directional hill kernel
方向希尔核函数
Eq. (3.2)Next we convert that detached mismatch into a smooth directional scaling rule for clipping. For a positive reference scale $q$, map a non-negative staleness $u$ to a factor in $(0,1]$:
接着我们把这一停止梯度后的失配转成平滑的方向性裁剪缩放规则。对正参考尺度 $q$，把非负陈旧度 $u$ 映射到 $(0,1]$ 中：
$$\psi(u;q)\;=\;\dfrac{1}{1+(u/q)^{2}},\qquad u_{+,b,t}=[d_{b,t}^{\mathrm{sg}}]_+,\quad u_{-,b,t}=[-d_{b,t}^{\mathrm{sg}}]_+.\tag{3.2}$$For fixed $q$, $\psi(0;q)=1$, $\psi(q;q)=\tfrac12$, and $\partial\psi/\partial u\le 0$.
固定 $q$ 时，$\psi(0;q)=1$、$\psi(q;q)=\tfrac12$，且 $\partial\psi/\partial u\le 0$。
Staleness-based kernel function scaling
基于陈旧性的核函数缩放
Eq. (3.3)We then anchor this scaling to the batch tail through a detached empirical quantile.
然后我们通过停止梯度的经验分位数，把这一缩放锚定到当前批次的长尾。
$$\begin{gathered} \widehat F_{|d|}(x) = \frac{1}{|\mathcal T_{\mathrm{act}}|}\sum_{(b,t)\in\mathcal T_{\mathrm{act}}}\mathbf 1\!\left\{|d_{b,t}|\le x\right\},\\ q = \operatorname{sg}\!\left[\inf\{x\ge0:\widehat F_{|d|}(x)\ge0.90\}\right]. \end{gathered}\tag{3.3}$$Thus $q$ is the detached empirical $0.90$ quantile over active, non-padding tokens. The strict gate $|d_{b,t}|>q$ targets upper tail; ties at $q$ can make the realized gate rate smaller than $10\%$.
因此，$q$ 是活跃、非 padding token 上停止梯度的经验 $0.90$ 分位数。严格门控 $|d_{b,t}|>q$ 针对上侧长尾；若在 $q$ 处存在并列值，实际门控率可能低于 $10\%$。
Staleness-based effective contraction factors
基于陈旧度的有效收缩因子
Eq. (3.4)This gate applies contraction only where mismatch is unusually large for the current batch.
这一步只在失配对当前批次而言异常偏大的位置施加收缩。
The explicit $q>0$ branch prevents division by zero when most staleness values are zero. Thus $c_{\pm,b,t}\in(0,1]$, and the raw kernel is never evaluated with a zero reference. If $\mathcal T_{\mathrm{act}}$ is empty, the optimizer step is skipped (equivalently, no SAT factor is evaluated).
显式的 $q>0$ 分支可避免“大多数陈旧度为零”时的除零问题。因此 $c_{\pm,b,t}\in(0,1]$，原始核不会在零参考尺度上求值。若 $\mathcal T_{\mathrm{act}}$ 为空，则跳过该优化步（等价地，不计算 SAT 因子）。
SAT sampled objective
SAT 采样目标
Eq. (3.5)–(3.6)Finally we insert these adaptive radii back into the standard sampled PPO surrogate.
最后我们把这些自适应半径重新代回标准的采样 PPO surrogate。
$$\bar r_{b,t}=\operatorname{clip}\!\left(r_{b,t},1-\tilde\varepsilon_{\text{low},b,t},1+\tilde\varepsilon_{\text{high},b,t}\right).\tag{3.5}$$ $$\mathcal S_{b,t}^{\text{SAT}}=\min\!\left(r_{b,t}\hat A_{b,t},\bar r_{b,t}\hat A_{b,t}\right),\qquad \mathcal L_{b,t}^{\text{SAT}}=-\mathcal S_{b,t}^{\text{SAT}}.\tag{3.6}$$Tokens for which the gate is off use the original radii exactly. A gated token has a strictly smaller nominal interval on the direction selected by the sign of $d_{b,t}$; the opposite endpoint remains unchanged.
门控未触发的 token 完全使用原半径。对触发门控的 token，$d_{b,t}$ 的符号所选方向上的名义区间端点严格收缩，另一侧端点保持不变。
What smaller radius actually changes更小半径究竟改变了什么
For one sampled token, write PPO's nominal interval and SAT's detached interval as $$\mathcal I^{\mathrm{PPO}}=[1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}],\qquad \mathcal I_{b,t}^{\mathrm{SAT}}=[1-\tilde\varepsilon_{\text{low},b,t},\,1+\tilde\varepsilon_{\text{high},b,t}] \subseteq \mathcal I^{\mathrm{PPO}}.$$ Holding the detached radii fixed, $$\mathcal S_{b,t}^{\mathrm{PPO}}=\min\!\left(r_{b,t}\hat A_{b,t},\operatorname{clip}(r_{b,t},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}})\hat A_{b,t}\right),$$ $$\mathcal S_{b,t}^{\mathrm{SAT}}=\min\!\left(r_{b,t}\hat A_{b,t},\operatorname{clip}(r_{b,t},1-\tilde\varepsilon_{\text{low},b,t},1+\tilde\varepsilon_{\text{high},b,t})\hat A_{b,t}\right)\le \mathcal S_{b,t}^{\mathrm{PPO}}.$$
对一个采样 token，记 PPO 类方法的名义裁剪区间与 SAT 的停止梯度区间为 $$\mathcal I^{\mathrm{PPO}}=[1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}],\qquad \mathcal I_{b,t}^{\mathrm{SAT}}=[1-\tilde\varepsilon_{\text{low},b,t},\,1+\tilde\varepsilon_{\text{high},b,t}] \subseteq \mathcal I^{\mathrm{PPO}}。$$ 固定停止梯度的半径后， $$\mathcal S_{b,t}^{\mathrm{PPO}}=\min\!\left(r_{b,t}\hat A_{b,t},\operatorname{clip}(r_{b,t},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}})\hat A_{b,t}\right),$$ $$\mathcal S_{b,t}^{\mathrm{SAT}}=\min\!\left(r_{b,t}\hat A_{b,t},\operatorname{clip}(r_{b,t},1-\tilde\varepsilon_{\text{low},b,t},1+\tilde\varepsilon_{\text{high},b,t})\hat A_{b,t}\right)\le \mathcal S_{b,t}^{\mathrm{PPO}}。$$
Away from clip kinks, the autograd derivative changes only on the newly clipped outward bands: $$\hat A_{b,t}>0,\quad 1+\tilde\varepsilon_{\text{high},b,t}<r_{b,t}<1+\varepsilon_{\text{high}},$$ $$\hat A_{b,t}<0,\quad 1-\varepsilon_{\text{low}}<r_{b,t}<1-\tilde\varepsilon_{\text{low},b,t}.$$ In these bands SAT has already zeroed the diverging gradient while PPO has not. The converging branch $\hat A_{b,t}(r_{b,t}-1)<0$ keeps its pull-back gradient even outside the nominal interval. If an outward ratio is already beyond PPO's original boundary, PPO already clips it, so further shrinkage has no additional gradient effect. Thus SAT matters exactly where it creates a new outward cutoff.
避开裁剪折点时，两者的自动微分导数只在新增的向外裁剪带不同： $$\hat A_{b,t}>0,\quad 1+\tilde\varepsilon_{\text{high},b,t}<r_{b,t}<1+\varepsilon_{\text{high}},$$ $$\hat A_{b,t}<0,\quad 1-\varepsilon_{\text{low}}<r_{b,t}<1-\tilde\varepsilon_{\text{low},b,t}。$$ 在这些区域，SAT 相较于 PPO 将已经把发散方向的梯度归零。收敛分支 $\hat A_{b,t}(r_{b,t}-1)<0$ 即使位于名义区间外也保留回拉梯度。反过来，如果向外比率早已越过 PPO 原始边界，PPO 本来就会裁剪，此时继续缩小半径不会再改变梯度。SAT 真正改变的是“向外截断”出现的位置。
Claim scope结论范围 Derivation and proof of SATSAT 方法的推导与证明
We start with two LLM-regime divergence quantities $$\begin{gathered} D_{\mathrm{TV}}^{\max}(\mu\|\pi) \triangleq \max_{s_t} D_{\mathrm{TV}}\bigl(\mu(\cdot\mid s_t)\|\pi(\cdot\mid s_t)\bigr),\\ \bar D_{\mathrm{TV}}(\mu,\pi) \triangleq \mathbb{E}_{y\sim\mu}\!\left[\sum_{t=1}^{|y|} D_{\mathrm{TV}}\bigl(\mu(\cdot\mid s_t)\|\pi(\cdot\mid s_t)\bigr) \right]. \end{gathered}$$ In the finite-horizon LLM setting ($T$, $\gamma=1$), with $\xi=\max_y |R(y)|$, the policy-improvement gap is lower-bounded by $$\mathcal J(\pi)-\mathcal J(\mu)\ge L'_{\mu}(\pi)-2\xi T(T-1)\bigl(D_{\mathrm{TV}}^{\max}(\mu\|\pi)\bigr)^2,$$ $$\mathcal J(\pi)-\mathcal J(\mu)\ge L'_{\mu}(\pi)-4\xi\,\bar D_{\mathrm{TV}}(\mu,\pi).$$ The max-$D_{\mathrm{TV}}$ form is the finite-horizon analogue of the classical trust-region bound, while the average token-level form is more relevant for long responses and closer to the per-token divergence control used by PPO-style methods. Equivalently, one can read the trust-region idea as maximizing $L'_{\mu}(\pi)$ under a divergence budget such as $D_{\mathrm{TV}}^{\max}(\mu\|\pi)\le\delta$.
这里先给出 LLM 场景下两个关键散度量： $$\begin{gathered} D_{\mathrm{TV}}^{\max}(\mu\|\pi) \triangleq \max_{s_t} D_{\mathrm{TV}}\bigl(\mu(\cdot\mid s_t)\|\pi(\cdot\mid s_t)\bigr),\\ \bar D_{\mathrm{TV}}(\mu,\pi) \triangleq \mathbb{E}_{y\sim\mu}\!\left[\sum_{t=1}^{|y|} D_{\mathrm{TV}}\bigl(\mu(\cdot\mid s_t)\|\pi(\cdot\mid s_t)\bigr) \right]. \end{gathered}$$ 在有限时域 $T$、无折扣 $\gamma=1$ 的设定下，记 $\xi=\max_y |R(y)|$，则策略改进满足 $$\mathcal J(\pi)-\mathcal J(\mu)\ge L'_{\mu}(\pi)-2\xi T(T-1)\bigl(D_{\mathrm{TV}}^{\max}(\mu\|\pi)\bigr)^2,$$ $$\mathcal J(\pi)-\mathcal J(\mu)\ge L'_{\mu}(\pi)-4\xi\,\bar D_{\mathrm{TV}}(\mu,\pi).$$ 前者是经典 trust-region 界在有限时域 LLM 场景下的对应形式；后者直接控制累计 token-level $D_{\mathrm{TV}}$，对长响应更紧，也更贴近 PPO 类方法实际使用的逐 token 散度控制。等价地，也可以把 trust-region 的思想理解为：在 $D_{\mathrm{TV}}^{\max}(\mu\|\pi)\le\delta$ 这类散度预算下最大化 $L'_{\mu}(\pi)$。
The logic here has three layers. Trust regions are motivated because policy-improvement error is controlled by cumulative token-level training-inference divergence divergence. PPO does not enforce this divergence constraint directly; it clips a sampled ratio surrogate instead. SAT stays within the same surrogate view and adaptively shrinks the sampled clipping interval on unreliable tokens. Accordingly, we state three things separately below: the strict sampled-surrogate result, the conditional training-inference divergence interpretation, and the local one-step mechanism by which SAT is intended to suppress future training-inference divergence growth.
这里的逻辑可以分成三层。首先，引入信任域的原因是：LLM 策略改进误差由累计 token 级训练-推理散度控制。其次，PPO 并不直接施加这个散度约束；它实际裁剪的是一个采样比率替代目标。最后，SAT 继承的也是这一路径：它在不可靠 token 上自适应缩小采样裁剪区间。因此，下面分别陈述三件事：严格成立的采样替代项结论、条件性的训练-推理散度解释，以及 SAT 旨在抑制未来训练-推理散度增长的局部一步机制。
Strict statement sampled surrogate contraction
严格结论 采样替代项收缩
$$\begin{gathered} \mathcal I^{\mathrm{PPO}} \triangleq [1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}],\\ \mathcal I_{b,t}^{\mathrm{SAT}} \triangleq [1-\tilde\varepsilon_{\text{low},b,t},\,1+\tilde\varepsilon_{\text{high},b,t}]. \end{gathered}\tag{3.7}$$Because $0<c_{\pm,b,t}\le1$, every sampled token satisfies $\mathcal I_{b,t}^{\mathrm{SAT}}\subseteq\mathcal I^{\mathrm{PPO}}$. This is a statement about nominal clip intervals and the sampled clipped surrogate, not about policy sets. PPO-style clipping can still keep a non-zero pull-back gradient outside the interval, so SAT is not a projection of the policy back into a hard ratio box.
由于 $0<c_{\pm,b,t}\le1$，每个采样 token 都满足 $\mathcal I_{b,t}^{\mathrm{SAT}}\subseteq\mathcal I^{\mathrm{PPO}}$。这是关于名义裁剪区间和采样 clipped surrogate 的结论，不是关于策略集合的结论。PPO 类裁剪在区间外仍可能保留非零的回拉梯度，因此 SAT 不是把策略投影回某个硬 ratio 盒子。
Conditional interpretation only a fixed all-action envelope $|r_s(a)-1|\le \lambda_s(a)$ yields a rigorous $D_{\mathrm{TV}}$ upper bound
条件性解释 固定全动作 $|r_s(a)-1|\le \lambda_s(a)$ 上下界才能严格推导出 $D_{\mathrm{TV}}$ 上界
At a fixed state $s$, let $r_s(a)=\pi(a\mid s)/\mu(a\mid s)$. When $\pi\ll\mu$, $$D_{\mathrm{TV}}(\mu,\pi)[s]=\tfrac12\mathbb E_{a\sim\mu(\cdot\mid s)}|r_s(a)-1|.$$ Hence, if there is a fixed all-action envelope $|r_s(a)-1|\le \lambda_s(a)$ over the action space, then $$D_{\mathrm{TV}}(\mu,\pi)[s]\le \tfrac12\mathbb E_{a\sim\mu(\cdot\mid s)}\lambda_s(a).\tag{3.8}$$ For SAT, one may interpret $\lambda_s(a)$ as a sign-matched adaptive radius, e.g. $\tilde\varepsilon_{\text{high}}(a\mid s)$ when $r_s(a)\ge1$ and $\tilde\varepsilon_{\text{low}}(a\mid s)$ otherwise. If this fixed all-action envelope holds across the action space, adaptive shrinkage would correspond to a tighter $D_{\mathrm{TV}}$ envelope than the base PPO radii.
在固定状态 $s$，令 $r_s(a)=\pi(a\mid s)/\mu(a\mid s)$。当 $\pi\ll\mu$ 时， $$D_{\mathrm{TV}}(\mu,\pi)[s]=\tfrac12\mathbb E_{a\sim\mu(\cdot\mid s)}|r_s(a)-1|。$$ 因而，只要存在全动作 $|r_s(a)-1|\le \lambda_s(a)$ 上下界，就有 $$D_{\mathrm{TV}}(\mu,\pi)[s]\le \tfrac12\mathbb E_{a\sim\mu(\cdot\mid s)}\lambda_s(a)。\tag{3.8}$$ 对 SAT，可以把 $\lambda_s(a)$ 理解为与符号匹配的自适应半径：例如当 $r_s(a)\ge1$ 时取 $\tilde\varepsilon_{\text{high}}(a\mid s)$，否则取 $\tilde\varepsilon_{\text{low}}(a\mid s)$。如果这个固定全动作 $|r_s(a)-1|\le\lambda_s(a)$ 上下界在整个动作空间上成立，那么 adaptive shrinkage 会对应比基础 PPO 半径更紧的 $D_{\mathrm{TV}}$ envelope。
SAT suppressing future $D_{\mathrm{TV}}$ growth
SAT 抑制未来 $D_{\mathrm{TV}}$ 增长
To explain the idea behind SAT's earlier stopping of diverging updates, we consider the most local one-step token-ratio dynamics and treat the token ratio $r_{b,t}$ as a directly optimized scalar: $$r_{b,t,+}^{\mathrm{alg}}=r_{b,t}+\eta\,\partial_{r_{b,t}}\mathcal S_{b,t}^{\mathrm{alg}},\qquad \eta>0,\ \mathrm{alg}\in\{\mathrm{PPO},\mathrm{SAT}\}. $$ On the outward bands newly clipped by SAT, $$\hat A_{b,t}>0,\quad 1+\tilde\varepsilon_{\text{high},b,t}<r_{b,t}<1+\varepsilon_{\text{high}},$$ $$\hat A_{b,t}<0,\quad 1-\varepsilon_{\text{low}}<r_{b,t}<1-\tilde\varepsilon_{\text{low},b,t},$$ the next-step sampled ratio deviation is always smaller than under PPO clipping. In the first case, $$r_{b,t,+}^{\mathrm{PPO}}=r_{b,t}+\eta\hat A_{b,t}>r_{b,t}=r_{b,t,+}^{\mathrm{SAT}},\qquad |r_{b,t,+}^{\mathrm{SAT}}-1|<|r_{b,t,+}^{\mathrm{PPO}}-1|,$$ and in the second case, $$r_{b,t,+}^{\mathrm{PPO}}=r_{b,t}+\eta\hat A_{b,t}<r_{b,t}=r_{b,t,+}^{\mathrm{SAT}},\qquad |r_{b,t,+}^{\mathrm{SAT}}-1|<|r_{b,t,+}^{\mathrm{PPO}}-1|。$$ Therefore, on the additional outward bands clipped by SAT, the next-step sampled ratio deviation is always smaller than under PPO clipping. This gives the most direct local mechanism behind “suppressing future $D_{\mathrm{TV}}$ growth”: the $D_{\mathrm{TV}}$ integrand is $|r-1|$; by cutting outward gradients earlier, SAT weakens the next-step growth of sampled $|r_{b,t}-1|$, thereby suppressing future $D_{\mathrm{TV}}$ growth.
为解释SAT方法的更早停止偏离更新的思想，我们考虑最局部的一步token-ratio动力学，把token-ratio $r_{b,t}$ 视为被直接优化的标量： $$r_{b,t,+}^{\mathrm{alg}}=r_{b,t}+\eta\,\partial_{r_{b,t}}\mathcal S_{b,t}^{\mathrm{alg}},\qquad \eta>0,\ \mathrm{alg}\in\{\mathrm{PPO},\mathrm{SAT}\}. $$ 在 SAT 新增截断的向外区间上，也就是 $$\hat A_{b,t}>0,\quad 1+\tilde\varepsilon_{\text{high},b,t}<r_{b,t}<1+\varepsilon_{\text{high}},$$ $$\hat A_{b,t}<0,\quad 1-\varepsilon_{\text{low}}<r_{b,t}<1-\tilde\varepsilon_{\text{low},b,t},$$ 下一步的采样比率偏离恒小于 PPO 裁剪机制。第一种情况下， $$r_{b,t,+}^{\mathrm{PPO}}=r_{b,t}+\eta\hat A_{b,t}>r_{b,t}=r_{b,t,+}^{\mathrm{SAT}},\qquad |r_{b,t,+}^{\mathrm{SAT}}-1|<|r_{b,t,+}^{\mathrm{PPO}}-1|,$$ 第二种情况下， $$r_{b,t,+}^{\mathrm{PPO}}=r_{b,t}+\eta\hat A_{b,t}<r_{b,t}=r_{b,t,+}^{\mathrm{SAT}},\qquad |r_{b,t,+}^{\mathrm{SAT}}-1|<|r_{b,t,+}^{\mathrm{PPO}}-1|。$$ 所以，在 SAT 新增截断的那部分向外区间上，下一步的采样比率偏离恒小于 PPO 裁剪。这就给出了“抑制未来 $D_{\mathrm{TV}}$ 增长”的最直接局部机制：$D_{\mathrm{TV}}$ 的被积项是 $|r-1|$；SAT 通过提前切掉向外梯度，使下一步采样 $|r_{b,t}-1|$ 的增长被弱化，从而抑制未来的散度增长。
GSPO scope. For GSPO, $\rho_b^{\mathrm{seq}}=\exp(T_b^{-1}\sum_t\log r_{b,t})$ replaces the token ratio in the implementation and is then broadcast to active token positions before quantile computation. Scalar interval containment still holds, but token-level positive and negative log-ratios can cancel, so the token-level conditional $D_{\mathrm{TV}}$ argument above does not directly extend to SAT-GSPO without an additional bound.
GSPO 的适用范围。对 GSPO，实现中用 $\rho_b^{\mathrm{seq}}=\exp(T_b^{-1}\sum_t\log r_{b,t})$ 替换 token 比率，再广播到活跃 token 位置后计算分位数。标量区间包含仍然成立，但 token 级正负对数比可能相互抵消，因此上面的 token 级条件 $D_{\mathrm{TV}}$ 论证不能直接外推到 SAT-GSPO，除非另行给出额外的界。
Figure图示 SAT — illustrative sampled-ratio window (click to expand)SAT — 采样比率窗口示意（点击展开）
3.3 Empirical evaluation: setup
3.3 经验评估：实验设置
We evaluate SAT in a realistic asynchronous pipeline on Qwen3-30B-A3B-Base: a grid of single-seed runs over the GRPO and GSPO bases, each optionally combined with R3, TIS, and SAT, plus a DPPO baseline. In the reported grid, this is the DPPO-$D_{\mathrm{TV}}$ variant, where inline $D_{\mathrm{TV}}$ denotes the total variation divergence, at configured lag ∈ { 1, 8 }. Each cell reports two quantities: the best-checkpoint AIME24 avg@8 and the training-time sampled training–inference mismatch $\bar d_\pi$. Neither metric is the exact full-policy $D_{\mathrm{TV}}$. Section 3.4 reports each run's best logged checkpoint; the per-method figures in Part 04 restrict to the early-training windows stated in their titles, so a few cells there can be lower than in the §3.4 table.
我们在贴近真实的异步流水线中、于 Qwen3-30B-A3B-Base 上评估 SAT：以 GRPO 与 GSPO 为基础方法，分别可叠加 R3、TIS 与 SAT，另加一个 DPPO 基线。本文报告的实现使用的是 DPPO-$D_{\mathrm{TV}}$ 变体，其中行内的 $D_{\mathrm{TV}}$ 表示 total variation divergence，配置 lag ∈ { 1, 8 }，全部为单种子实验。每个格子报告两个量：AIME24 最佳检查点 avg@8，以及训练期采样训练–推理失配 $\bar d_\pi$；两者都不是精确的完整策略 $D_{\mathrm{TV}}$。§3.4 报告每次运行的最佳日志检查点；第 04 部分的各方法小图仅统计其标题所注明的训练早期窗口，因此个别数值会低于 §3.4 的表格。
Setup实验设置 Training data, batching, sequence length, sampling protocol, clip parameters训练数据、批/组大小、序列长度、采样协议、裁剪参数
- Training data. Merged DAPO-Math-17k + Dolci-RL-Zero-Math-7B math set (30,712 prompts = 17,398 DAPO-Math-17k + 13,314 Dolci-RL-Zero-Math-7B), chat template applied, rollout shuffled, data-balanced. Reward:
prime_mathrule-based verifier (multi-pattern answer extraction + Hendrycks-MATH normalization + SymPy symbolic equivalence). Eval: AIME24 (30 problems, 16 samples each, eval every 5 steps, report the average performance across 8 runs). - Batch / group size.
rollout_batch_size= 256 prompts ×n_samples_per_prompt= 16 (GRPO group size 16) = 4096 responses per iteration; one minor step per weight broadcast, so global batch = 4096; 544 rollout iterations. - Sequence length. Rollout / eval max response length = 32k tokens;
max_tokens_per_gpu= 32k with dynamic batching. - Sampling protocol. Temperature = 0.95,
top_p= 1.0,top_k= −1 (full-vocabulary support, no nucleus truncation); weight-broadcast interval = 8 for the configured-lag-8 runs; async pipeline where the rollout engine prepares pipeline step $k+1$ while the trainer runs step $k$. - Clip parameters. Clipping radii $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})=(0.2,0.2)$; Adam ($\beta_1{=}0.9$, $\beta_2{=}0.98$), constant LR $10^{-6}$; KL loss (
low_var_kl, coef $10^{-3}$); entropy coef 0. SAT: hill-kernel exponent $2$ and reference $q$ at quantile level $0.90$. - RL methods (reproducible). All three baselines share one GRPO skeleton and the identical outer config above ($(\varepsilon_{\text{low}},\varepsilon_{\text{high}})=(0.2,0.2)$,
kl-loss-coef$10^{-3}$,low_var_kl, entropy 0, group size 16, std-normalized advantage); only the ratio/KL granularity and sampled clipping/gating rule differ. GRPO (--advantage-estimator grpo): per-token ratio + per-token KL, standard asymmetric PPO clip. GSPO (gspo): same objective but sequence-level ratio/KL — the per-sequence mean KL is all-gathered and broadcast so every token in a sequence shares one ratio. DPPO (dppo): in the reported grid this is the DPPO-$D_{\mathrm{TV}}$ variant, where the inline $D_{\mathrm{TV}}$ denotes total variation divergence. It applies the GRPO clip times a {0,1} hard mask that zeroes only sampled tokens whose local direction is outward and whose sampled $D_{\mathrm{TV}}$ discriminator exceeds its threshold, with--dppo-delta$=10^{-3}$ and--dppo-divergence-type tv(needs rollout log-probs). This sampled discriminator is not the full vocabulary $D_{\mathrm{TV}}$. SAT contracts the shared $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})$ using the ratio scalar of its base objective: a token ratio for GRPO and the broadcast sequence ratio for GSPO. It is bit-identical to the base method when disabled. - Statistical scope. Single-seed runs: each reported cell is one run at a fixed seed; within a matched lag pair, the configured pipeline-lag setting is the deliberately varied factor. We do not attach confidence intervals to individual avg@8 numbers. When two $\bar d_\pi$ means are called “close”, we mean they differ by less than the trailing-25% within-run fluctuation, not a multi-seed significance test.
- 训练数据。DAPO-Math-17k + Dolci-RL-Zero-Math-7B 合并数学集（30,712 条 = DAPO-Math-17k 17,398 + Dolci-RL-Zero-Math-7B 13,314），套用 chat 模板、rollout 打乱、数据均衡。奖励：
prime_math规则式判分（多模式答案抽取 + Hendrycks-MATH 归一化 + SymPy 符号等价）。评测：AIME24（30 题，每题 16 采样，每 5 步评一次，报告 8 次运行的平均性能）。 - 批 / 组大小。
rollout_batch_size= 256 提示 ×n_samples_per_prompt= 16（GRPO group size 16）= 每轮 4096 条响应；每次权重广播一个 minor step，故 global batch = 4096；共 544 轮 rollout。 - 序列长度。rollout / eval 最大响应长度 = 32k token；
max_tokens_per_gpu= 32k，动态批。 - 采样协议。温度 = 0.95，
top_p= 1.0，top_k= −1（全词表支持，不做 nucleus 截断）；lag = 8 的实验使用权重广播间隔 8；异步流水：训练器运行流水线第 $k$ 步时，推理引擎准备第 $k+1$ 步数据。 - 裁剪参数。裁剪半径 $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})=(0.2,0.2)$；Adam（$\beta_1{=}0.9$，$\beta_2{=}0.98$），常数学习率 $10^{-6}$；KL loss（
low_var_kl，系数 $10^{-3}$）；熵系数 0。SAT：hill 核指数为 $2$，参考量 $q$ 的分位水平为 $0.90$。 - RL 方法（可复现）。三种基线共用同一套 GRPO 骨架与上面完全相同的外层配置（$(\varepsilon_{\text{low}},\varepsilon_{\text{high}})=(0.2,0.2)$、
kl-loss-coef$10^{-3}$、low_var_kl、熵 0、组大小 16、std 归一化优势），区别只在比值/KL 的粒度与采样裁剪 / 门控规则。GRPO（--advantage-estimator grpo）：per-token 比值 + per-token KL，标准非对称 PPO clip。GSPO（gspo）：目标相同但比值/KL 为序列级 —— 先 all-gather 完整序列算每序列平均 KL，再广播回每 token，使同一序列内共享一个比值。DPPO（dppo）：在本文报告的实验里使用的是 DPPO-$D_{\mathrm{TV}}$ 变体，其中行内的 $D_{\mathrm{TV}}$ 表示 total variation divergence。它在 GRPO clip 上再乘一个 {0,1} 硬掩码，仅屏蔽局部方向向外且采样 $D_{\mathrm{TV}}$ 判别量超过阈值的采样 token；--dppo-delta$=10^{-3}$、--dppo-divergence-type tv（需 rollout log-prob）。这个采样判别量不是完整词表 $D_{\mathrm{TV}}$。SAT 使用其基础目标的比率标量来收缩共享的 $(\varepsilon_{\text{low}},\varepsilon_{\text{high}})$：GRPO 使用 token 比率，GSPO 使用广播后的序列比率；关闭时与基础方法 bit-identical。 - 统计范围。单种子实验：每个报告数值为固定种子下的单次运行；在成对 lag 对照内，刻意变化的是流水线配置 lag。我们不对单个 avg@8 附加置信区间。当称两个 $\bar d_\pi$ 均值「接近」时，指其差值小于末 25% 步的组内波动，而非多种子显著性检验。
Section 3.4第 3.4 节 The results实验结果
Three comparisons follow: the full best-checkpoint grid (Result 3.4.1), the matching last-epoch mismatch grid (Result 3.4.2), and the mismatch trajectories over training (Results 3.4.3a/b).
下面给出三组对照：完整的最佳检查点网格（结果 3.4.1）、对应的末 epoch 失配网格（结果 3.4.2），以及训练全程的失配轨迹（结果 3.4.3a/b）。
Best observed AIME24 avg@8 across the reported configurations
已报告配置中的 AIME24 最佳 avg@8
| Method方法 | lag = 1lag = 1 | lag = 8lag = 8 |
|---|---|---|
| Base Model (Qwen3-30B-A3B-Base) | 9.38 | |
| GRPO | 31.25 (429) | 30.17 (429, collapse) |
| GSPO | 32.25 (474) | 31.46 (424, collapse) |
| GRPO w/ R3 | 31.83 (314) | 29.58 (214) |
| GSPO w/ R3 | 34.00 (379) | 33.13 (214) |
| GRPO w/ TIS | 31.62 (254) | 31.46 (374) |
| GSPO w/ TIS | 32.88 (319) | 31.21 (279) |
| GRPO w/ TIS w/ R3 | 31.79 (324) | 31.79 (239) |
| GSPO w/ TIS w/ R3 | 31.88 (329) | 32.92 (359) |
| DPPO | 33.96 (324) | 32.71 (289) |
| SAT-GRPO | 32.71 (404) | 32.08 (299) |
| SAT-GSPO | 34.17 (254) | 32.71 (359) |
| SAT-GRPO w/ R3 | 33.75 (284) | 33.13 (289) |
| SAT-GSPO w/ R3 | 35.83 (379) | 34.79 (359) |
Chart柱状图 Bar chart view 上表数据的柱状图视图
Color legend (matches the table above): purple bold = column best (lag=1: SAT-GSPO w/ R3 = 35.83; lag=8: SAT-GSPO w/ R3 = 34.79); underline = column runner-up (lag=1: SAT-GSPO = 34.17; lag=8: SAT-GRPO w/ R3 = 33.13, tied with GSPO w/ R3 = 33.13); red = runs that later collapse at lag=8 (GRPO = 30.17, GSPO = 31.46; see Experiment 4.b). The parenthesized number in each table cell is the step index of the best checkpoint among converged training runs; the word
steps is omitted inside the cell for compactness, and collapse marks runs that later collapse after reaching that best checkpoint.
The SAT family still occupies several of the largest values in both columns. At lag=8, the runs marked red later collapse, while DPPO reports 32.71 and the best SAT w/ R3 configuration remains stable in the logged window.
颜色图例（与上表一一对应）：紫色加粗 = 该列第一名（lag=1：SAT-GSPO w/ R3 = 35.83；lag=8：SAT-GSPO w/ R3 = 34.79）；下划线 = 该列第二名（lag=1：SAT-GSPO = 34.17；lag=8：SAT-GRPO w/ R3 = 33.13，与 GSPO w/ R3 = 33.13 并列）；红色 = lag=8 后期发生崩溃的运行（GRPO = 30.17、GSPO = 31.46；见实验 4.b）。 表格每个单元格括号中的数字表示在训练收敛情况下性能最好的 checkpoint 所对应的 step 编号；为简洁起见，单元格内省略
steps 一词，而 collapse 表示该运行在达到该最佳 checkpoint 之后又发生了崩溃。
SAT 家族仍在两列中占据多席最大值。lag=8 下，标红的运行随后崩溃，DPPO 报告 32.71，而最佳 SAT w/ R3 配置在日志窗口内保持稳定。Per-run last-epoch mean of the training–inference mismatch $\bar d_\pi$
各实验末个 epoch 的训练–推理失配 $\bar d_\pi$ 均值
| Method方法 | lag = 1lag = 1 | lag = 8lag = 8 |
|---|---|---|
| GRPO | 0.0103 | 0.0110 |
| GSPO | 0.0097 | 0.0109 |
| GRPO w/ R3 | 0.0058 | 0.0218 |
| GSPO w/ R3 | 0.0091 | 0.0158 |
| GRPO w/ TIS | 0.0108 | 0.0169 |
| GSPO w/ TIS | 0.0151 | 0.0208 |
| GRPO w/ TIS w/ R3 | 0.0048 | 0.0081 |
| GSPO w/ TIS w/ R3 | 0.0054 | 0.0068 |
| DPPO | 0.0108 | 0.0118 |
| SAT-GRPO | 0.0101 | 0.0108 |
| SAT-GSPO | 0.0095 | 0.0107 |
| SAT-GRPO w/ R3 | 0.0056 | 0.0079 |
| SAT-GSPO w/ R3 | 0.0056 | 0.0076 |
Chart柱状图 Bar chart view 上表数据的柱状图视图
Color legend (matches the table above; lower means less observed sampled mismatch, not a $D_{\mathrm{TV}}$ or return ranking): orange bold = column best (lag = 1: GRPO w/ TIS w/ R3 = 0.0048); purple bold = column best (lag = 8: GSPO w/ TIS w/ R3 = 0.0068); underline = column runner-up (lag = 1: GSPO w/ TIS w/ R3 = 0.0054; lag = 8: SAT-GSPO w/ R3 = 0.0076).
颜色图例（与上表一一对应；数值越低仅表示观测采样失配越小，并非完整 $D_{\mathrm{TV}}$ 或回报排序）：橙色加粗 = 该列第一名（lag = 1：GRPO w/ TIS w/ R3 = 0.0048）；紫色加粗 = 该列第一名（lag = 8：GSPO w/ TIS w/ R3 = 0.0068）；下划线 = 该列第二名（lag = 1：GSPO w/ TIS w/ R3 = 0.0054；lag = 8：SAT-GSPO w/ R3 = 0.0076）。
Training–Inference Mismatch over training — lag = 1
训练全程的训练–推理失配 — lag = 1
interactive可交互Training–Inference Mismatch over training — lag = 8
训练全程的训练–推理失配 — lag = 8
interactive可交互GSPO (lag = 8) leaves the axis after step ≈ 330 and reaches $\bar d_\pi > 6$ by step 504
Part 04
第 04 部分
How does each stability approach work in async RL?
各稳定方法在异步 RL 中如何生效？
SAT turns one dial — the clip radius — using one observable signal. The rest of the async-RL stabilizer landscape can be
read through the same lens: every method below picks a different mathematical object built from the same observed log-ratio
$\log r_{b,t}$ (the behavior denominator, the ratio's granularity, a mask, a threshold, a radius) and modifies it.
Each method follows the same template: (i) which mismatch source it targets, (ii) which mathematical
object it modifies, (iii) a mechanistic or conditional interpretation, and (iv) the reported experiment.
The folded Analysis分析 panels give algebra where it is available and state the assumptions where a formal claim needs them; they are not all policy-improvement proofs.
Click any tile below to expand its panel; click the same tile again to collapse it — only one panel is shown at a time.
SAT 只用一个可观测信号去调一个旋钮——裁剪半径。其余的异步 RL 稳定方法也可以用同一视角来读：下面每种方法都在同一个观测对数比 $\log r_{b,t}$ 之上，选择并修改一个不同的数学对象（行为分母、比率的粒度、掩码、阈值、半径）。每种方法都按同一模板展开：(i) 针对哪个失配来源；(ii) 修改哪个数学对象；(iii) 机制性或条件性解释；(iv) 已报告实验。折叠的分析面板在可行处给出代数，并在形式化结论需要假设时明确写出；它们并不都是策略改进证明。
可以点击不同的方块以展开不同下拉栏；再次点击当前已展开的方块可将其收起，页面上一次只显示一个下拉栏。
Use Rollout-logprobs
使用 rollout 对数概率
Target source. The denominator mismatch between Megatron's token-level rollout-weight recomputation $\tilde\mu_{b,t}$ and the actual engine behavior $\mu_{b,t}$; the switch does not reduce policy-version lag.
针对来源。Megatron 在各 token rollout 权重下的重算分布 $\tilde\mu_{b,t}$ 与真实引擎行为 $\mu_{b,t}$ 之间的分母失配；该开关并不降低策略版本差。
In vanilla async PPO/GRPO, the "old log-probs" term is Megatron's recomputation $\log\tilde\mu_{b,t}$. The switch replaces it with the token log-probs already emitted by SGLang, so the trainer uses the sampled ratio against the actual token behavior conditional $\mu_{b,t}$:
在原味异步 PPO/GRPO 中，"旧对数概率"一项是 Megatron 的逐 token 重算值 $\log\tilde\mu_{b,t}$。该开关将其替换为 SGLang 已经产出的逐 token 对数概率，于是训练器使用以真实 token 行为条件分布 $\mu_{b,t}$ 为分母的采样比率：
$$ \log\tilde\mu_{b,t}(y_{b,t}\!\mid\!s_{b,t}) \;\leftarrow\; \log\mu_{b,t}(y_{b,t}\!\mid\!s_{b,t}), \qquad r_{b,t}^{\text{rollout-lp}} \;=\; \exp\!\bigl(\log\pi^{(j)}(y_{b,t}\mid s_{b,t}) - \log\mu_{b,t}(y_{b,t}\mid s_{b,t})\bigr) \;=\; r_{b,t}. $$This aligns the sampled denominator with the actual behavior log-prob. Under the usual support condition it is the denominator required by the importance-sampling identity. It can nevertheless make the observed ratio more variable because implementation mismatch is now visible rather than absorbed by the recomputation.
这会让采样分母与真实行为对数概率对齐；在通常的支撑条件下，它是重要性采样恒等式所需的分母。但观测比率仍可能更易波动，因为实现失配不再被重算值吸收，而是显式出现。
Analysis分析 What the denominator switch exposes分母切换显式暴露了什么
Split the log-ratio the switch actually optimizes:
把该开关实际优化的对数比率拆开：
$$\log r_{b,t}^{\text{rollout-lp}}\;=\;\underbrace{\log\pi^{(j)}-\log\tilde\mu_{b,t}}_{\text{policy change relative to rollout-weight recomputation}}\;+\;\underbrace{\log\tilde\mu_{b,t}-\log\mu_{b,t}}_{\text{implementation mismatch}}.$$The first term is exactly what PPO's ratio machinery was designed to correct, and it is not touched by the switch. The switch merges the second term into the visible ratio, so PPO actually sees it — that is the "cleanness" part. The variance of the sum depends on both component variances and their covariance, so it is not guaranteed to increase in every system. It did increase in the measurements reported here:
第一项正是 PPO 比率机制所处理的对象，而该开关并未触碰它。开关把第二项并入可见比率，让 PPO 真正“看到”它。两项之和的方差取决于各自方差与协方差，因此不能保证在每个系统中都增大；但在这里报告的测量中确实增大：
$$\begin{array}{lcc} \widehat{\operatorname{Var}}[\,d_{b,t}\,]&\text{configured lag}=1&\text{configured lag}=8\\\hline \text{GRPO}&2.6\times 10^{-4}&1.4\times 10^{-4}\\ \text{GRPO + use-rollout-lp}&3.2\times 10^{-4}&2.7\times 10^{-4}\\ \end{array}$$Scope. Equation (2.3) is a deterministic population bound and contains no sample-complexity constant. Logging the actual denominator changes the estimator used in optimization; the table above is empirical evidence about that estimator, not a modification of the bound itself.
范围。式 (2.3) 是总体层面的确定性下界，本身不含样本复杂度常数。记录真实分母会改变优化中使用的估计量；上表是关于该估计量的经验证据，而不是对下界本身的修改。
Rollout Routing Replay (R3)
路由重放 R3（Rollout Routing Replay）
Target source. (S3) router staleness for MoE.
针对来源。MoE 的 (S3) 路由陈旧量。
The original R3[11] paper (arXiv:2510.11370v2) motivates R3 by a mismatch inside a single global step: SGLang and Megatron produce different Top-K experts on identical weights — a framework-level artifact that can persist even when mini-step $=1$. Lifting R3 into async-RL adds a second axis of drift: even a "framework-consistent" MoE could see a different Top-K at each local offset $h$ among trainer updates $j=\ell+n+h$, i.e. inside $[\ell+n,\ell+n+H]$.
R3[11] 原论文（arXiv:2510.11370v2）的动机是单个全局步内部的失配：在完全相同的权重下，SGLang 与 Megatron 仍可能给出不同的 Top-K 专家，即使 mini-step $=1$ 也可能存在。把 R3 用于异步 RL 场景则多出第二条变化轴：即使框架计算一致，在训练更新 $j=\ell+n+h$ 对应的窗口 $[\ell+n,\ell+n+H]$ 内，不同局部偏移 $h$ 上也可能得到不同 Top-K。
The fix — for every trainer minor step $j=\ell+n+h$, replace the training-side mask by the rollout-time mask, while keeping the softmax denominator built from the current trainer logits so gradients still flow into the router:
修复方式 — 对每个训练小步 $j=\ell+n+h$，把训练侧的专家掩码替换为 rollout 时的掩码，同时 softmax 分母仍由当前训练器 logits 构建，使梯度依旧流入路由器：
Async R3 gate异步 R3 门控
$$ \begin{gathered} g_{\text{replay},e}^{(j)}\;=\;\frac{I_{\text{infer},e}^{(\ell)}\,\exp\!\big(z_{\text{train},e}^{(j)}\big)}{\sum_{e'} I_{\text{infer},e'}^{(\ell)}\,\exp\!\big(z_{\text{train},e'}^{(j)}\big)},\\ \mathbf h_{\text{replay}}^{(j)}=\sum_{e=1}^M g_{\text{replay},e}^{(j)}\,\mathcal E_e^{(j)}(\mathbf h_{\text{train}}),\\ j=\ell+n+h,\qquad h\in\{0,\ldots,H\}. \end{gathered} $$The rollout mask fixes expert membership, while current logits and expert weights preserve gradient flow through the trainer computation. This aligns one discrete routing choice; it does not make the entire forward pass identical to rollout.
rollout 掩码固定专家集合，而当前 logits 与专家权重保留训练侧计算中的梯度流。它对齐的是一个离散路由选择，并不会让整个前向与 rollout 完全相同。
Empirically (Qwen3-30B-A3B), the reported sampled $k_3$ divergence estimate falls from $1.5\times 10^{-3}$ to $7.5\times 10^{-4}$ with R3, close to the $6.4\times 10^{-4}$ dense-model reference. This is a finite-sample diagnostic, not an exact KL identity or a claim that all MoE mismatch is removed.
实证上（Qwen3-30B-A3B），报告的采样 $k_3$ 散度估计在使用 R3 后从 $1.5\times 10^{-3}$ 降到 $7.5\times 10^{-4}$，接近稠密模型的参考值 $6.4\times 10^{-4}$。这是有限样本诊断量，不是精确 KL 恒等式，也不表示所有 MoE 失配都被消除。
Analysis分析 Mechanism and limits of routing replay路由重放的机制与边界
Total variation does not have an exact additive decomposition into “policy + router + engine” terms. With explicitly defined intermediate distributions, the triangle inequality can separate diagnostic comparisons, but it cannot show that replay must reduce the final $D_{\mathrm{TV}}$. R3 makes one narrower statement by construction: the trainer uses the rollout-time Top-K membership.
总变差距离不存在“策略 + 路由 + 引擎”的精确加法分解。若显式定义中间分布，可用三角不等式拆开诊断比较，但这不能证明重放后最终 $D_{\mathrm{TV}}$ 必然减小。R3 在构造上只保证一个更窄的结论：训练器使用 rollout 时的 Top-K 专家集合。
Current router logits, gate weights, hidden states, expert parameters, and downstream layers can still differ. Consequently the router-related contribution is not identically zero, the numerical surrogate is not untouched, and (2.3) is not guaranteed to tighten uniformly. The reported KL reduction is empirical evidence that mask replay helps on this stack.
当前 router logits、门权重、hidden states、专家参数与后续层仍可能不同。因此不能声称路由相关贡献恒为零、数值替代目标完全不变，或 (2.3) 在整个异步窗口上一致收紧。已报告的 KL 降低是该软硬件栈上掩码重放有效的经验证据。
Two useful operating regimes are:
两个有用的运行情形是：
- $n=0,\,h=0$: mask replay isolates framework-level Top-K disagreement at fixed weights.
- $n=0,\,h=0$：掩码重放隔离固定权重下的框架级 Top-K 分歧。
- $n+h\ge1$: replay also fixes expert membership across a rollout-to-train version gap, while all continuous computations remain trainer-current.
- $n+h\ge1$：重放还会跨 rollout 到训练的版本差固定专家集合，但所有连续计算仍使用训练器当前值。
R3 on AIME24 — reported best avg@8 on Qwen3-30B-A3B-Base
R3 在 AIME24 上的表现 — Qwen3-30B-A3B-Base 的已报告最佳 avg@8
Group Sequence Policy Optimization (GSPO) & SeqClip
组序列策略优化（GSPO）与 SeqClip
Target source. (S1) + (S3): move the unit of importance sampling from token to sequence.
针对来源。(S1) + (S3)：把重要性采样的单位从 token 提升到序列。
Instead of reweighting each token by $r_{b,t}$, GSPO reweights each entire response by a length-normalized sequence ratio and clips at the response level. Within one prompt group, the following formulas locally index its $G$ candidate responses by $b=1,\dots,G$. Here $\hat A_b$ is the detached, group-normalized sequence advantage for response $b$; when broadcast to token notation, $\hat A_{b,t}=\hat A_b$ for all $t$ in that response:
GSPO 不再用 $r_{b,t}$ 给每个 token 加权，而是用长度归一的序列比率给整条响应加权，并在响应级别做裁剪。下式在单个提示词组内，把 $G$ 条候选响应局部编号为 $b=1,\dots,G$。其中 $\hat A_b$ 是响应 $b$ 经过组归一化且停止梯度的序列优势；广播到 token 记号时，该响应内所有 $t$ 都满足 $\hat A_{b,t}=\hat A_b$：
$$\rho_b^{\mathrm{seq}}\;=\;\Big(\tfrac{\pi^{(j)}(y_b\mid x)}{\mu(y_b\mid x)}\Big)^{1/T_b}\;=\;\exp\!\Big(\tfrac{1}{T_b}\sum_{t=1}^{T_b}d_{b,t}\Big),$$ $$\mathcal S_{\text{GSPO}}=\mathbb E_{\{y_b\}\sim\mu}\!\Bigg[\frac{1}{G}\sum_{b=1}^G\min\!\Big(\rho_b^{\mathrm{seq}}\hat A_b,\ \operatorname{clip}(\rho_b^{\mathrm{seq}},1-\varepsilon,1+\varepsilon)\hat A_b\Big)\Bigg].$$Length-normalization replaces the unnormalized sum of token log-ratios by their mean, avoiding automatic exponential scaling with response length. It does not make the range mathematically length-independent: token deviations can cancel or reinforce. Sequence-level weighting may smooth isolated token fluctuations, but neither cancellation nor convergence on MoE is guaranteed, and routing replay can still be useful empirically.
长度归一化把 token 对数比的未归一化求和改成均值，从而避免比率随响应长度自动指数缩放。但它并不在数学上保证数值范围与长度无关：token 偏差既可能抵消，也可能同向叠加。序列级加权可能平滑孤立 token 波动，但不能保证抵消或 MoE 收敛；路由重放在经验上仍可能有用。
Analysis分析 GSPO gradient and the SeqClip mechanismGSPO 梯度与 SeqClip 机制
Ignoring clipping for brevity,
为简洁起见忽略裁剪，
$$\nabla_\theta\mathcal S_{\text{GSPO}}=\mathbb E\!\Bigg[\frac{1}{G}\sum_{b=1}^G\rho_b^{\mathrm{seq}}\hat A_b\cdot\frac{1}{T_b}\sum_{t=1}^{T_b}\nabla_\theta\log\pi^{(j)}(y_{b,t}\mid s_{b,t})\Bigg].$$Compared with a token-ratio objective — whose effective per-token weights depend on $r_{b,t}$ and the sign of $\hat A_b$ — GSPO weights all tokens in a response equally by the length-normalized factor $\rho_b^{\mathrm{seq}}$.
与 GRPO 的逐 token 权重不同，GSPO 用长度归一因子 $\rho_b^{\mathrm{seq}}$ 统一加权同一响应内的 token。该公式说明权重粒度的变化，但不单独给出方差或稳定性排序。
SeqClip[18]. A response-level mean log-ratio can still move far from zero when deviations reinforce. SeqClip hard-masks any response whose $\rho_b^{\mathrm{seq}}$ falls outside a tight tolerance:
SeqClip[18]。当 token 偏差同向叠加时，响应级平均对数比仍可能显著偏离零。SeqClip 会把 $\rho_b^{\mathrm{seq}}$ 落在紧容差之外的整条响应硬掩掉：
$$\mathcal S_{\text{SeqClip}}=\mathbb E\!\Bigg[\frac{1}{G}\sum_{b=1}^G\underbrace{\mathbf 1\!\big[\alpha_{\text{seq}}\le \rho_b^{\mathrm{seq}}\le\beta_{\text{seq}}\big]}_{\text{hard sequence-level mask}}\cdot\min\!\big(\rho_b^{\mathrm{seq}}\hat A_b,\ \operatorname{clip}(\rho_b^{\mathrm{seq}},1-\varepsilon,1+\varepsilon)\hat A_b\big)\Bigg].$$Reported production settings use a very tight $[\alpha_{\text{seq}},\beta_{\text{seq}}]=[0.995,\,1.005]$ — orders of magnitude tighter than IcePop's per-token $[0.5,\,5.0]$, because the length-normalized sequence ratio is already close to $1$ under a healthy run.
公开报道的生产配置使用非常紧的 $[\alpha_{\text{seq}},\beta_{\text{seq}}]=[0.995,\,1.005]$ —— 比 IcePop 的逐 token $[0.5,\,5.0]$ 紧几个数量级，因为健康实验中长度归一的序列比率本就非常接近 $1$。
GSPO vs GRPO on AIME24 — reported best avg@8 on Qwen3-30B-A3B-Base
GSPO 对比 GRPO（AIME24）— Qwen3-30B-A3B-Base 的已报告最佳 avg@8
Token-level stabilisers — TIS / IcePop / KPop
token 级稳定器 — TIS / IcePop / KPop
Target source. All three are token-level stabilisers layered on a PPO / GRPO surrogate. TIS weights by the sampled policy ratio $r_{b,t}$ of §1; IcePop and KPop gate on the sampled token's trainer-side vs. rollout-side probabilities — the implementation-mismatch part of the §1 decomposition.
针对来源。三者都是叠加在 PPO / GRPO 替代目标上的 token 级稳定器。TIS 使用 §1 定义的采样策略比率 $r_{b,t}$ 加权；IcePop 与 KPop 则以采样 token 在训练侧与 rollout 侧的概率为门控对象 —— 对应 §1 分解中的实现失配部分。
TIS[12] caps the ratio's contribution at a constant $C>1$ from above (typical $C\in[2,5]$) so extreme overshoot cannot dominate, while passing the exact importance weight through everywhere below the cap (a one-sided continuous truncation). IcePop[13][14] discards tokens whose train-engine / inference-engine probability ratio leaves a two-sided range $[\alpha,\beta]$, keeping the ratio itself as an IS-style weight in range. KPop[15] replaces the ratio geometry with a Bernoulli-KL geometry that scales quadratically at small mismatch and more strongly at large mismatch. This targets a limitation of ratio-only screening: a $10^{-4}\to 10^{-2}$ jump on a long-tail token yields $r_{b,t}=100$ but a $D_{\mathrm{TV}}$/KL contribution of only $\approx 10^{-2}$.
TIS[12] 从上方把比率贡献封顶在常数 $C>1$（典型取值 $C\in[2,5]$），防止极端超冲占据主导；低于上限时则原样保留精确重要性权重（单侧连续截断）。IcePop[13][14] 丢弃训练引擎 / 推理引擎概率比移出双侧区间 $[\alpha,\beta]$ 的 token，区间内则保留该比率本身作为 IS 式权重。KPop[15] 用 Bernoulli-KL 几何替代比率几何：小失配时二次缩放，大失配时惩罚更强。它针对了单看比率的一个局限：长尾 token 上 $10^{-4}\to 10^{-2}$ 的跳变给出 $r_{b,t}=100$，但其对 $D_{\mathrm{TV}}$/KL 的贡献只有约 $10^{-2}$。
Formulas公式 The three masks written in one notation同一套记号写出的三种掩码
TIS — continuous one-sided truncation. Here the effective weight is detached when applied to the log-policy term; $C>1$ is the truncation cap (e.g., $2$ in verl's default, $5$ in the Qwen formulation[12]).
TIS — 连续单侧截断。有效权重乘到 log-policy 项上时停止梯度；$C>1$ 为截断上限（例如 verl 默认取 $2$，Qwen 表述[12]取 $5$）。
$$w_{b,t}^{\text{TIS}}=\min\!\big(r_{b,t},\,C\big),\quad C>1,\qquad \mathcal S_{\text{TIS}}=\mathbb E\!\Big[\sum_{b,t}\operatorname{sg}[w_{b,t}^{\text{TIS}}]\cdot \hat A_{b,t}\log\pi^{(j)}(y_{b,t}\mid s_{b,t})\Big].$$IcePop — two-sided range gate with an IS-style in-range weight. Its discriminant is the sampled token's train-engine / inference-engine probability ratio evaluated at the same rollout weights, $k_{b,t}=\tilde\mu_{b,t}(y_{b,t}\mid s_{b,t})/\mu_{b,t}(y_{b,t}\mid s_{b,t})$ — in our notation, the implementation-mismatch ratio. Both factors are fixed numbers at loss time, so the weight carries no gradient.
IcePop — 双侧范围门控 + 区间内 IS 式权重。其判别量是采样 token 在同一 rollout 权重下的训练引擎 / 推理引擎概率比 $k_{b,t}=\tilde\mu_{b,t}(y_{b,t}\mid s_{b,t})/\mu_{b,t}(y_{b,t}\mid s_{b,t})$——即本文记号中的实现失配比率。两个因子在计算损失时都是常数，因此该权重不携带梯度。
$$M_{b,t}^{\text{IcePop}}=k_{b,t}\,\mathbf 1\!\big[\alpha\le k_{b,t}\le\beta\big],\qquad (\alpha,\beta)=(0.5,\,5.0)\ \text{typical}.$$The weight multiplies the raw PPO/GRPO clip term — in-range tokens keep the IS-style correction $k_{b,t}$, out-of-range tokens are zeroed (some re-implementations reduce this to a pure $0/1$ rejection mask):
该权重乘进原始 PPO/GRPO 裁剪项——区间内的 token 保留 IS 式修正 $k_{b,t}$，越界 token 归零（一些复现实现将其简化为纯 $0/1$ 拒绝掩码）：
$$\mathcal S_{\text{IcePop}}=\mathbb E\!\Bigg[\frac{1}{G}\sum_{b=1}^{G}\frac{1}{T_b}\sum_{t=1}^{T_b} M_{b,t}^{\text{IcePop}}\cdot\min\!\big(r_{b,t}\hat A_{b,t},\operatorname{clip}(r_{b,t},1-\varepsilon,1+\varepsilon)\hat A_{b,t}\big)\Bigg].$$KPop — Bernoulli-KL bidirectional adaptive mask. With $p_{b,t}^{\pi}$ the trainer-side and $p_{b,t}^{\mu}=\mu_{b,t}(y_{b,t}\mid s_{b,t})$ the rollout-side probability of the sampled token, let $\phi>0$ be the KPop divergence threshold and define
KPop — Bernoulli-KL 双向自适应掩码。记 $p_{b,t}^{\pi}$ 为采样 token 的训练侧概率、$p_{b,t}^{\mu}=\mu_{b,t}(y_{b,t}\mid s_{b,t})$ 为其 rollout 侧概率，令 $\phi>0$ 为 KPop 的散度阈值，并定义
$$D_{\mathrm{KL}}^{\mathrm{Bern}}(p\|q)=p\log\!\tfrac pq+(1-p)\log\!\tfrac{1-p}{1-q},$$ $$M_{b,t}^{\text{KPop}}=\mathbf 1\!\big[D_{\mathrm{KL}}^{\mathrm{Bern}}(p_{b,t}^{\pi}\|p_{b,t}^{\mu})\le\phi\big]\cdot\mathbf 1\!\big[D_{\mathrm{KL}}^{\mathrm{Bern}}(p_{b,t}^{\mu}\|p_{b,t}^{\pi})\le\phi\big].$$Mechanistic view. These methods suppress sampled gradient contributions associated with large ratios or divergences. That does not by itself prove that the resulting policy has smaller $\bar D_{\mathrm{TV}}$ in (2.3). A two-sided hard mask also removes any pull-back contribution on a masked sample, whereas direction-aware PPO-style clipping can retain it (see §4.d).
机制视角。这些方法会抑制与大比率或大散度相关的采样梯度贡献；仅凭这一点不能证明更新后的策略在 (2.3) 中具有更小的 $\bar D_{\mathrm{TV}}$。双侧硬掩码还会移除被掩样本上的回拉贡献，而方向感知的 PPO 类裁剪可能保留它（见 §4.d）。
| Method方法 | Discriminator判别量 | Sampled rule采样规则 | Weight $w_{b,t}$ | Direction方向性 |
|---|---|---|---|---|
| PPO | $|r_{b,t}-1|$ | outward cutoff偏离侧截断: $|r_{b,t}-1|=\varepsilon$ | $\{0,1\}$ | asymmetric非对称 |
| TIS | $r_{b,t}$ | $w=\min(r_{b,t},C)$, $C>1$ | $\min(r_{b,t},C)\in(0,C]$ | one-sided单侧 |
| IcePop | $k_{b,t}=\tilde\mu_{b,t}/\mu_{b,t}$ | keep if满足时保留 $k_{b,t}\in[\alpha,\beta]$ | $k_{b,t}\,\mathbf 1[\alpha\le k_{b,t}\le\beta]$ | advantage-agnostic与优势无关 |
| KPop | $D_{\mathrm{KL}}^{\mathrm{Bern}}$ | keep if满足时保留 $D_{\mathrm{KL}}^{\mathrm{Bern}}\le\phi$ (both directions双向) | $\{0,1\}$ | advantage-agnostic与优势无关 |
The essential mathematical difference between IcePop / KPop and PPO / DPPO is that the former's gate is a function of a mismatch magnitude ($k_{b,t}$ or the bidirectional $D_{\mathrm{KL}}^{\mathrm{Bern}}$) only, while the latter's is a function of $\operatorname{sign}(\hat A_{b,t}(r_{b,t}-1))$ as well. Section 4.d derives this asymmetric structure and its async-RL consequences from the ground up.
IcePop / KPop 与 PPO / DPPO 的本质数学区别在于：前者的门控只是失配幅度（$k_{b,t}$ 或双向 $D_{\mathrm{KL}}^{\mathrm{Bern}}$）的函数，而后者的门控还依赖 $\operatorname{sign}(\hat A_{b,t}(r_{b,t}-1))$。§4.d 会从头推导这种非对称结构及其在异步 RL 中的后果。
Token-level stabilisers on AIME24 — reported best avg@8 for TIS variants
token 级稳定器在 AIME24 上的表现 — TIS 系列的已报告最佳 avg@8
Top-p configuration
Top-p 配置
Target source. Support mismatch caused by truncating the behavior sampler while evaluating an untruncated target policy.
针对来源。行为采样器做截断、目标策略却不截断时产生的支撑失配。
During rollout, top_p=p with $p<1$ assigns zero behavior probability outside the selected nucleus. Those
excluded actions are never sampled and therefore do not directly enter a sampled policy loss. The issue is instead
measure-theoretic: if the full-softmax target assigns positive mass outside the behavior support, then
$\pi^{(j)}\not\ll\mu_{b,t}$ and importance sampling from $\mu_{b,t}$ cannot recover expectations over that missing target
mass. A sampled ratio is defined on the nucleus, but a full-policy IS / $D_{\mathrm{TV}}$ argument needs support coverage.
rollout 使用 top_p=p（$p<1$）时，所选 nucleus 之外的行为概率为零。这些动作永远不会被采样，因此不会直接进入采样策略损失。真正的问题是测度支撑：若全 softmax 目标在行为支撑外仍分配正质量，则 $\pi^{(j)}\not\ll\mu_{b,t}$，从 $\mu_{b,t}$ 出发的重要性采样无法恢复缺失的目标质量。采样比率在 nucleus 内有定义，但完整策略的 IS / $D_{\mathrm{TV}}$ 推导需要支撑覆盖。
Fixes修复 Two ways to restore target support coverage恢复目标支撑覆盖的两条路径
MAI-Thinking-1[16] — rollout Top-p mask replay. Cache the Top-p truncation mask $\mathcal T_{b,t}\subseteq\mathcal V$ used at rollout time. During training, before the softmax, set
MAI-Thinking-1[16] — 重放 rollout 的 Top-p 掩码。缓存 rollout 时使用的 Top-p 截断掩码 $\mathcal T_{b,t}\subseteq\mathcal V$。训练时在 softmax 之前置
$$\tilde{\mathrm{logit}}_{b,t}(a)=\begin{cases}\mathrm{logit}_{b,t}(a),& a\in\mathcal T_{b,t},\\-\infty,& a\notin\mathcal T_{b,t},\end{cases}\qquad \pi^{(j)}(a\mid s_{b,t})=\operatorname{softmax}\!\big(\tilde{\mathrm{logit}}_{b,t}\big)(a).$$The target then assigns zero mass outside the recorded behavior nucleus, so $\operatorname{supp}(\pi^{(j)})\subseteq\operatorname{supp}(\mu_{b,t})$ for that masked computation. This repairs the support condition; any KL reduction remains an empirical result.
于是该掩码计算中的目标策略在记录的行为 nucleus 外分配零质量，即 $\operatorname{supp}(\pi^{(j)})\subseteq\operatorname{supp}(\mu_{b,t})$。这修复了支撑条件；KL 是否降低仍是经验结果。
Full-support sampling — disable truncation at rollout time (the configuration used in our experiments, §3.3, and in several reported multimodal training recipes).
全支撑采样 — 在 rollout 阶段关闭截断（即本文实验采用的配置，见 §3.3，也见于一些公开的多模态训练报告）。
$$\text{temperature}=1.0,\qquad \text{top\_p}=1.0,\qquad \text{top\_k}=-1.$$Under these settings the rollout support is the full vocabulary and the training/rollout support disagreement disappears; in reported runs with this configuration, $D_{\mathrm{KL}}(\pi^{(j)}\|\mu_R)$ hovers around and below $\sim 10^{-3}$ — the same ballpark as R3's post-fix regime.
在这组设置下，rollout 的支撑集就是整个词表，训练/rollout 支撑集不再分歧；在采用该配置的已报告运行中，$D_{\mathrm{KL}}(\pi^{(j)}\|\mu_R)$ 稳定在 $\sim 10^{-3}$ 上下 —— 与 R3 修复后的量级相同。
Equality of supports is stronger than necessary. The required direction for target expectations is absolute continuity, $\pi^{(j)}\ll\mu_{b,t}$. Without it, sampled clipping cannot account for target probability mass on excluded actions.
支撑完全相等比必要条件更强。对目标期望而言，需要的方向是绝对连续 $\pi^{(j)}\ll\mu_{b,t}$。若不满足，采样裁剪无法处理目标策略在被排除动作上的概率质量。
Asymmetric clipping mechanism
非对称裁剪机制
Target. Gates that depend on $\operatorname{sign}(\hat A_t(r_t-1))$, distinguishing whether a sampled local update moves its ratio away from or back toward the behavior reference.
目标。依赖 $\operatorname{sign}(\hat A_t(r_t-1))$ 的门控，用于区分某个采样局部更新是在让比率远离行为参考，还是拉回参考。
For the local derivations in §4.d only, we suppress the response index $b$ and write $r_t$, $\hat A_t$, and $\mu_R$ for a fixed-batch behavior reference. Throughout this subsection $\hat A_t$ is detached, and the radii $\varepsilon,\delta$ are positive whenever used.
仅在 §4.d 的局部推导中，我们省略响应下标 $b$，并在固定批行为参考下写作 $r_t$、$\hat A_t$ 与 $\mu_R$。本小节始终把 $\hat A_t$ 视为停止梯度量，并在使用时假设半径 $\varepsilon,\delta$ 为正。
Asymmetric clipping is inherited by PPO, extended by DPPO to divergence space, and softened by SPO / DRPO to preserve out-of-range pull-back. This subsection proves a narrower local property: a two-sided hard mask can discard a pull-back integrand that an advantage-aware asymmetric gate preserves. It does not establish that asymmetry is necessary for every stable async-RL algorithm.
非对称裁剪由 PPO 继承、被 DPPO 推广到散度空间、又被 SPO / DRPO 软化以保留域外回拉。本小节证明一个更窄的局部性质：双侧硬掩码可能丢弃优势感知非对称门控会保留的回拉被积项；它并不证明所有稳定异步 RL 算法都必须使用非对称裁剪。
4.d.i The classical improvement bound → $D_{\mathrm{TV}}$ motivates the discriminator
4.d.i 经典改进界 → $D_{\mathrm{TV}}$ 为判别量提供动机
Under $\gamma=1$, finite horizon $T$, sparse sequence-level reward, DPPO re-derives the LLM version of the TRPO lower bound:
在 $\gamma=1$、有限视界 $T$、稀疏序列级奖励下，DPPO 重推了 TRPO 下界的 LLM 版本：
$$\mathcal J(\pi^{(j)})-\mathcal J(\mu_R)\;\ge\;L'_{\mu_R}(\pi^{(j)})\;- 2\xi T(T-1)\big[D_{\mathrm{TV}}^{\max}(\mu_R,\pi^{(j)})\big]^2,$$ $$\mathcal J(\pi^{(j)})-\mathcal J(\mu_R)\;\ge\;L'_{\mu_R}(\pi^{(j)})\;- 4\xi\,\bar D_{\mathrm{TV}}(\mu_R,\pi^{(j)}).$$Here $D_{\mathrm{TV}}^{\max}(\mu_R,\pi^{(j)})\triangleq\sup_{s\in\operatorname{Reach}(\mu_R)}D_{\mathrm{TV}}(\mu_R,\pi^{(j)})[s]$, where the supremum is over the behavior-reachable states covered by the bound.
其中 $D_{\mathrm{TV}}^{\max}(\mu_R,\pi^{(j)})\triangleq\sup_{s\in\operatorname{Reach}(\mu_R)}D_{\mathrm{TV}}(\mu_R,\pi^{(j)})[s]$，上确界取遍该下界覆盖的行为可达状态。
These bounds motivate the ideal problem of maximising $L'_{\mu_R}$ subject to $D_{\mathrm{TV}}^{\max}(\mu_R,\pi^{(j)})\le\delta$. PPO instead optimizes a sampled clipped objective with a nominal ratio cutoff; it is neither a hard all-action constraint nor a projection onto that feasible set. Raw sampled ratio is also poorly aligned with probability mass on LLM long tails: a $\mu_R=10^{-4}\to\pi=10^{-2}$ jump gives $r_t=100$ (over-clipped) while its sampled-vs-complement sampled $D_{\mathrm{TV}}$ value is only about $10^{-2}$; a $0.99\to 0.80$ shift gives $r_t\approx 0.808\in[1-\varepsilon,1+\varepsilon]$ (escapes clipping) but has sampled $D_{\mathrm{TV}}$ $0.19$. This sampled $D_{\mathrm{TV}}$ is a partition-based lower bound on full-vocabulary $D_{\mathrm{TV}}$, not a full-vocabulary $D_{\mathrm{TV}}$ certificate.
这些下界启发了理想问题：在 $D_{\mathrm{TV}}^{\max}(\mu_R,\pi^{(j)})\le\delta$ 约束下最大化 $L'_{\mu_R}$。PPO 实际优化的是带名义比率截断的采样裁剪目标；它既不是全动作硬约束，也不会把策略投影到该可行集。原始采样比率在 LLM 长尾上也与概率质量变化对齐较差：$\mu_R=10^{-4}\to\pi=10^{-2}$ 的跳变给出 $r_t=100$（被过度裁剪），但其“采样动作 vs. 其余动作”的采样 $D_{\mathrm{TV}}$ 只有约 $10^{-2}$；而 $0.99\to 0.80$ 的移动给出 $r_t\approx 0.808\in[1-\varepsilon,1+\varepsilon]$（逃过裁剪），采样 $D_{\mathrm{TV}}$ 却为 $0.19$。这个采样 $D_{\mathrm{TV}}$ 是基于分区的完整词表 $D_{\mathrm{TV}}$ 下界，不是完整词表 $D_{\mathrm{TV}}$ 证书。
The DPPO variant analysed here moves the implemented sampled-token discriminator from raw ratio space to sampled $D_{\mathrm{TV}}$: the PPO sampled discriminator is $|r_t-1|$ with cutoff $\varepsilon$, whereas the DPPO sampled discriminator is $D_{\mathrm{TV},t}=\big|\pi^{(j)}(y_t\mid s_t)-\mu_R(y_t\mid s_t)\big|$ with cutoff $\delta$.
这里分析的 DPPO 变体把实现中的采样 token 判别量从原始比率空间搬到采样 $D_{\mathrm{TV}}$ 空间：PPO 的采样判别量是带阈值 $\varepsilon$ 的 $|r_t-1|$，而 DPPO 的采样判别量是带阈值 $\delta$ 的 $D_{\mathrm{TV},t}=\big|\pi^{(j)}(y_t\mid s_t)-\mu_R(y_t\mid s_t)\big|$。
This sampled $D_{\mathrm{TV}}$ discriminator is motivated by the full-distribution bound above but is not identical to the vocabulary-level $D_{\mathrm{TV}}$ in (2.2), and does not hard-enforce that policy-level constraint.
这个采样 $D_{\mathrm{TV}}$ 判别量受上方全分布下界启发，但不等同于 (2.2) 的词表级 $D_{\mathrm{TV}}$，也不会硬性执行该策略级约束。
Figure图示 TRPO — fixed $D_{\mathrm{TV}}$-radius trust region (click to expand)TRPO — 固定 $D_{\mathrm{TV}}$ 半径的信任域（点击展开）
4.d.ii The asymmetric structure of PPO's clip
4.d.ii PPO 裁剪的非对称结构
Recall the PPO per-token surrogate $r_t\hat A_t$. Its partial derivative w.r.t.\ $r_t$ is
回顾 PPO 的逐 token 替代项 $r_t\hat A_t$。它对 $r_t$ 的偏导为
$$\frac{\partial}{\partial r_t}\big(r_t\hat A_t\big)=\hat A_t,$$so the desired ascent direction of $r_t$ is $\operatorname{sign}(\hat A_t)$. Taking $r_t=1$ as the reference:
因此 $r_t$ 的期望上升方向是 $\operatorname{sign}(\hat A_t)$。以 $r_t=1$ 为参照：
- Diverging — $\operatorname{sign}(\hat A_t(r_t-1))>0$: this sampled ratio moves away from $\mu_R$ in the reward-improving direction and may cross its ratio/divergence boundary. This local sign alone does not certify a violation of full-policy $D_{\mathrm{TV}}$.
- 偏离 — $\operatorname{sign}(\hat A_t(r_t-1))>0$：该采样比率沿奖励提升方向远离 $\mu_R$，并可能越过相应比率 / 散度边界；仅凭这个局部符号不能判定完整策略 $D_{\mathrm{TV}}$ 已经违规。
- Converging — $\operatorname{sign}(\hat A_t(r_t-1))<0$: $\pi^{(j)}$ sits on the wrong side of the reward objective, and further optimization pulls $r_t$ back toward $1$ — this is the desired behavior.
- 收敛 — $\operatorname{sign}(\hat A_t(r_t-1))<0$：$\pi^{(j)}$ 落在奖励目标的错误一侧，继续优化会把 $r_t$ 拉回 $1$ —— 这正是想要的行为。
Away from clip kinks, expanding the four cases of $\min(r_t\hat A_t,\,\operatorname{clip}(r_t,1-\varepsilon,1+\varepsilon)\hat A_t)$ shows that PPO only zeroes the gradient on diverging updates that have crossed the boundary — never on converging updates, however large $|r_t-1|$ is. At equality, the derivative follows the implementation's subgradient/autograd convention:
在裁剪折点之外，展开 $\min(r_t\hat A_t,\,\operatorname{clip}(r_t,1-\varepsilon,1+\varepsilon)\hat A_t)$ 的四种情形可见：PPO 只在越界的偏离更新上归零梯度 —— 无论 $|r_t-1|$ 多大，收敛更新的梯度从不被归零。边界取等时，导数由具体实现的次梯度 / 自动微分约定决定：
$$\frac{\partial\mathcal S_{\text{PPO}}}{\partial\theta}\Big|_t=\begin{cases}0,&\operatorname{sign}(\hat A_t(r_t-1))>0\wedge|r_t-1|>\varepsilon,\\[3pt]\hat A_t\,\nabla_\theta r_t,&\text{otherwise}.\end{cases}$$PPO's asymmetric mask (canonical form)PPO 的非对称掩码（标准形式）
$$M_t^{\text{PPO}}=\mathbf 1\!\Big[\operatorname{sign}(\hat A_t(r_t-1))\le 0\ \vee\ |r_t-1|\le\varepsilon\Big].$$DPPO reproduces the same structure with the discriminator swapped from ratio space to divergence space:
DPPO 复刻同样的结构，只是把判别量从比率空间换成散度空间：
$$M_t^{\text{DPPO}}=\mathbf 1\!\Big[\operatorname{sign}(\hat A_t(r_t-1))\le 0\ \vee\ D_{\mathrm{TV},t}\le\delta\Big],\quad D_{\mathrm{TV},t}=\big|\pi^{(j)}(y_t\mid s_t)-\mu_R(y_t\mid s_t)\big|.$$Figure图示 PPO — fixed symmetric clip window (click to expand)PPO — 固定对称裁剪窗口（点击展开）
4.d.iii DPPO $D_{\mathrm{TV}}$ gate as a token-adaptive ratio threshold
4.d.iii DPPO $D_{\mathrm{TV}}$ 门控对应的 token 自适应比率阈值
Under sampled $D_{\mathrm{TV}}$, $D_{\mathrm{TV},t}=|\pi^{(j)}(y_t\mid s_t)-\mu_R(y_t\mid s_t)|=\mu_R(y_t\mid s_t)\,|r_t-1|$, so $D_{\mathrm{TV},t}\le\delta$ is equivalent to
在采样 $D_{\mathrm{TV}}$ 下，$D_{\mathrm{TV},t}=|\pi^{(j)}(y_t\mid s_t)-\mu_R(y_t\mid s_t)|=\mu_R(y_t\mid s_t)\,|r_t-1|$，因此 $D_{\mathrm{TV},t}\le\delta$ 等价于
$$|r_t-1|\;\le\;\frac{\delta}{\mu_R(y_t\mid s_t)},$$Thus the detached DPPO sampled $D_{\mathrm{TV}}$ mask can be written with a token-adaptive ratio threshold:
因此，停止梯度的 DPPO 采样 $D_{\mathrm{TV}}$ 掩码可写成 token 自适应的比率阈值：
$$\begin{gathered} \mathcal S_{\text{DPPO-mask}}(\pi^{(j)})=\mathbb E_{y\sim\mu_R}\!\Bigg[\sum_{t=1}^{|y|}\operatorname{sg}\!\left[M_t^{\text{DPPO}}\right]r_t\hat A_t\Bigg],\\ M_t^{\text{DPPO}}=\mathbf 1\!\left[\operatorname{sign}(\hat A_t(r_t-1))\le0\ \vee\ |r_t-1|\le\frac{\delta}{\mu_R(y_t\mid s_t)}\right]. \end{gathered}$$The threshold $\delta/\mu_R(y_t\mid s_t)$ is loose on long-tail tokens (small $\mu_R$) and tight on high-probability tokens ($\mu_R\to1$). Away from threshold kinks, this mask has the same keep/drop gradient regions as a PPO clip with that token-dependent radius. The two objective values are not identical: a hard mask zeroes a rejected term, whereas a clipped surrogate saturates it at a boundary value. In the experiments of §3.4 the DPPO mask is multiplied by the baseline PPO clip, so the effective gate is the intersection of both mechanisms; the adaptive threshold alone does not describe the full experimental objective.
阈值 $\delta/\mu_R(y_t\mid s_t)$ 在长尾 token（$\mu_R$ 小）上宽松，在高概率 token（$\mu_R\to1$）上收紧。远离阈值折点时，该掩码与采用此逐 token 半径的 PPO 裁剪具有相同的梯度保留 / 丢弃区域；但二者的目标函数值并不相等：硬掩码把被拒项置零，裁剪替代目标则在边界值处饱和。§3.4 的实验把 DPPO 掩码乘在基础 PPO 裁剪项上，因此有效门控是两种机制的交集；仅凭自适应阈值不能描述完整实验目标。
Figure图示 DPPO — sampled $D_{\mathrm{TV}}$ threshold scales inversely with $\mu_R(y_t\mid s_t)$ (click to expand)DPPO — 采样 $D_{\mathrm{TV}}$ 阈值与 $\mu_R(y_t\mid s_t)$ 成反比（点击展开）
4.d.iv Why direction-aware gates preserve a pull-back channel
4.d.iv 方向感知门控为何保留回拉通道
Consider a token with $\hat A_t>0$ and $r_t\ll 1$ in a stale-data setting.
考虑陈旧数据场景中一个 $\hat A_t>0$ 且 $r_t\ll 1$ 的 token。
- Under an IcePop-style two-sided range gate ($\alpha=0.5$) applied to this ratio: $r_t<\alpha\Rightarrow M=0$; the pull-back gradient is discarded.
- 在作用于该比率、$\alpha=0.5$ 的 IcePop 式双侧范围门控下：$r_t<\alpha\Rightarrow M=0$，回拉梯度被丢弃。
- Under a KPop-style bidirectional Bernoulli-KL gate at this mismatch level: $M=0$; the pull-back gradient is discarded.
- 在该失配水平下的 KPop 式双向 Bernoulli-KL 门控下：$M=0$，回拉梯度被丢弃。
- Under the DPPO / DRPO direction gate, $\operatorname{sign}(\hat A_t(r_t-1))=-1\le0\Rightarrow M=1$; the sampled pull-back term $\hat A_t\nabla_\theta r_t$ is retained. Its local ratio direction points toward $1$, although a shared-parameter optimizer need not move every token ratio monotonically.
- 在 DPPO / DRPO 的方向门控下，$\operatorname{sign}(\hat A_t(r_t-1))=-1\le0\Rightarrow M=1$，采样回拉项 $\hat A_t\nabla_\theta r_t$ 被保留。其局部比率方向指向 $1$，但共享参数优化器不保证每个 token 比率都单调移动。
Local analysis局部分析 Which sampled integrands each gate removes不同门控移除了哪些采样被积项
The following calculation uses a contextual-bandit reduction, behavior support coverage, and detached advantages and gates; it is exact within that local model away from gate boundaries. Let $P$ be the chosen contextual-state distribution, let $\beta(s)$ be the baseline, set $A(s,a)=R(s,a)-\beta(s)$, and identify the detached estimate $\hat A=A$ within this calculation. The importance-weighted reference integrand is
下面的计算采用上下文老虎机化简、行为支撑覆盖，以及停止梯度的优势与门控；在远离门控边界的这一局部模型内，计算是精确的。令 $P$ 为选定的上下文状态分布，记 $\beta(s)$ 为基线、$A(s,a)=R(s,a)-\beta(s)$，并在该计算中令停止梯度的估计量 $\hat A=A$。重要性加权参照被积项为
$$g_{\mathrm{on}}=\mathbb E_{s\sim P,\,a\sim\mu_R(\cdot\mid s)}\!\big[\,r(\theta)\,\nabla_\theta\log\pi^{(j)}(a\mid s)\,A(s,a)\big],\qquad r(\theta)=\tfrac{\pi^{(j)}(a\mid s)}{\mu_R(a\mid s)}.$$Two-sided advantage-agnostic hard mask. Let $0<r_{\min}<1<r_{\max}$. A gate $M(a)=\mathbb 1[r(\theta)\in[r_{\min},r_{\max}]]$ depends on the ratio magnitude but not on the advantage direction. Writing $\mathbb 1_{\mathcal M}=1-M$ for the masked set, the exact bias relative to $g_{\mathrm{on}}$ is supported on $\mathcal M$:
双侧、与优势方向无关的硬掩码。令 $0<r_{\min}<1<r_{\max}$。门控 $M(a)=\mathbb 1[r(\theta)\in[r_{\min},r_{\max}]]$ 依赖比率幅度，但不依赖优势方向。记 $\mathbb 1_{\mathcal M}=1-M$ 为被掩集合，则相对 $g_{\mathrm{on}}$ 的精确偏差支撑在 $\mathcal M$ 上：
$$\mathrm{Bias}_{\mathrm{2s}}=-\,\mathbb E_{s\sim P,\,a\sim\mu_R(\cdot\mid s)}\!\big[\,\mathbb 1_{\mathcal M}(a)\cdot r(\theta)\,\nabla_\theta\log\pi^{(j)}(a\mid s)\,A(s,a)\big].$$Thus, whenever a converging sample lies in $\mathcal M$ (for example $A>0$ and $r<r_{\min}<1$), the two-sided mask removes its entire reference integrand. This is a pointwise statement about that sample; it does not determine the sign or magnitude of the batch-averaged bias.
因此，只要一个收敛样本落入 $\mathcal M$（例如 $A>0$ 且 $r<r_{\min}<1$），双侧掩码就会移除该样本的完整参照被积项。这是关于该样本的逐点结论，并不能决定批平均偏差的符号或大小。
Asymmetric gate. A gate of the PPO form sets $M=1$ whenever $\operatorname{sign}(A(r-1))\le 0$, so it never masks a converging token. Its bias is supported only on diverging out-of-window tokens:
非对称门控。PPO 形式的门控在 $\operatorname{sign}(A(r-1))\le 0$ 时置 $M=1$，故从不掩掉收敛 token。其偏差只支撑在偏离侧的越窗 token 上：
$$\mathrm{Bias}_{\mathrm{asym}}=-\,\mathbb E_{s\sim P,\,a\sim\mu_R(\cdot\mid s)}\!\big[\,\mathbb 1_{\{\operatorname{sign}(A(r-1))>0,\ |r-1|>\varepsilon\}}\cdot r(\theta)\,\nabla_\theta\log\pi^{(j)}(a\mid s)\,A(s,a)\big],$$This is the defensive truncation implemented by the PPO-style gate: it removes only sampled terms pushing away from $\mu_R$ in the reward-improving direction, and preserves the pull-back term on every converging token within this local model. It should not be confused with TRPO's hard policy-level constraint. (The unbounded-positive limit[20] removes even the diverging-side clip for $A>0$ via a stop-gradient anchor $\tilde r=\pi_\theta/\operatorname{sg}(\pi_\theta)$, whose bias factor $(1-r)$ vanishes pointwise as $r\to1$.)
这就是 PPO 类门控实现的防御性截断：它只移除沿奖励提升方向远离 $\mu_R$ 的采样项，并在该局部模型中保留每个收敛 token 的回拉项；不应把它与 TRPO 的策略级硬约束混为一谈。（无界正向极限[20]甚至通过停梯度锚 $\tilde r=\pi_\theta/\operatorname{sg}(\pi_\theta)$ 对 $A>0$ 移除偏离侧裁剪，其偏差因子 $(1-r)$ 随 $r\to1$ 逐点消失。）
Conclusion. Within the stated local model, a direction-aware gate preserves all samples satisfying $\operatorname{sign}(A(r-1))\le0$, while a two-sided magnitude mask can remove some of them. This explains a useful design property; it does not prove that the discarded terms grow monotonically with staleness or that every stable async method must be asymmetric. ∎
结论。在上述局部模型中，方向感知门控会保留所有满足 $\operatorname{sign}(A(r-1))\le0$ 的样本，而双侧幅度掩码可能移除其中一部分。这解释了一个有用的设计性质；它没有证明被丢项会随陈旧度单调增长，也没有证明所有稳定异步方法都必须非对称。 ∎
4.d.v Softened asymmetric clipping (SPO / DRPO)
4.d.v 软化的非对称裁剪（SPO / DRPO）
On the diverging side, DPPO's asymmetric mask is still a hard $0/1$ gate: immediately after $D_t$ crosses $\delta$, that sampled gradient drops to zero. Converging out-of-range terms remain active through the asymmetric branch. For $\varepsilon>0$, SPO[17] instead softens PPO's diverging-side cutoff with a differentiable quadratic regularizer:
在偏离侧，DPPO 的非对称掩码仍是硬 $0/1$ 门控：$D_t$ 刚越过 $\delta$ 后，该采样梯度便降为零；收敛侧的域外项仍由非对称分支保留。当 $\varepsilon>0$ 时，SPO[17] 用可微二次正则软化 PPO 的偏离侧截断：
$$\mathcal S_{\text{SPO}}(\pi)=\mathbb E\!\Bigg[\sum_t\Big(r_t\hat A_t-\frac{|\hat A_t|}{2\varepsilon}(r_t-1)^2\Big)\Bigg],\qquad r_t^\star=1+\operatorname{sign}(\hat A_t)\varepsilon\quad(\hat A_t\ne0).$$For $\hat A_t\ne0$, the unconstrained scalar stationary point $r_t^\star$ lands exactly on PPO's clip boundary; if $\hat A_t=0$, this per-token term is identically zero and has no unique stationary point. Out-of-range regions with non-zero advantage produce a "pull-back force" toward the boundary. DRPO applies the same recipe on top of DPPO's sampled $D_{\mathrm{TV}}$ space, replacing $\varepsilon$ by the token-adaptive $\varepsilon_t=\delta/\mu_R(y_t\mid s_t)$:
当 $\hat A_t\ne0$ 时，无约束标量驻点 $r_t^\star$ 恰好落在 PPO 裁剪边界；若 $\hat A_t=0$，该逐 token 项恒为零，不存在唯一驻点。优势非零时，域外区域产生指向边界的“回拉力”。DRPO 在 DPPO 的采样 $D_{\mathrm{TV}}$ 空间上如法炮制，把 $\varepsilon$ 换成 token 自适应的 $\varepsilon_t=\delta/\mu_R(y_t\mid s_t)$：
$$\mathcal S_{\text{DRPO}}(\pi^{(j)})=\mathbb E_{y\sim\mu_R}\!\Bigg[\sum_t\Big(r_t\hat A_t\;-\;\frac{|\hat A_t|}{2\delta}\,\mu_R(y_t\mid s_t)\,(r_t-1)^2\Big)\Bigg].$$Derivation推导 Stationary point and $\ell_2^2$ vs. $\chi^2$ regularization驻点分析与 $\ell_2^2$ 对 $\chi^2$ 正则
For $\delta>0$ and $\hat A_t\ne0$, ignoring the probability-simplex boundary and setting $\partial\mathcal S_{\text{DRPO}}/\partial r_t=0$ gives the unconstrained stationary point in probability space. When $\hat A_t=0$, the per-token term is identically zero and no unique stationary point is selected:
当 $\delta>0$ 且 $\hat A_t\ne0$ 时，暂不考虑概率单纯形边界，令 $\partial\mathcal S_{\text{DRPO}}/\partial r_t=0$，得到概率空间中的无约束驻点。若 $\hat A_t=0$，逐 token 项恒为零，不会选出唯一驻点：
$$\pi^{(j)}(y_t\mid s_t)^\star=\mu_R(y_t\mid s_t)+\operatorname{sign}(\hat A_t)\,\delta,$$When this value lies in $[0,1]$, it is the corresponding scalar DPPO sampled $D_{\mathrm{TV}}$ boundary; otherwise the one-coordinate optimum is clipped by feasibility. Because action probabilities share a simplex constraint, this scalar stationary equation does not characterize the full-policy optimum. The gradient weight is nevertheless bounded and continuous:
当该值落在 $[0,1]$ 内时，它就是相应的标量 DPPO 采样 $D_{\mathrm{TV}}$ 边界；否则，单坐标最优值会受可行概率边界截断。由于各动作概率共享单纯形约束，该标量驻点方程并不刻画完整策略最优解。不过梯度权重仍有界且连续：
$$w_t^{\text{DRPO}}=1-\operatorname{sign}(\hat A_t(r_t-1))\,\frac{D_t^{\mathrm{TV}}}{\delta}\ \in\ \Big[1-\tfrac{1}{\delta},\,1+\tfrac{1}{\delta}\Big].$$At a fixed state, substituting $\varepsilon_t=\delta/\mu_R$ into the SPO regularizer changes the implicit penalty from a $\chi^2$-type term with inverse behavior probability to an $\ell_2^2$ term without that factor:
在固定状态下，把 $\varepsilon_t=\delta/\mu_R$ 代入 SPO 正则项，会把含行为概率倒数因子的 $\chi^2$ 型惩罚改成不含该因子的 $\ell_2^2$ 项：
$$\mathbb E_{a\sim\mu_R(\cdot\mid s_t)}\!\big[|\hat A_t(a)|\,\mu_R(a\mid s_t)(r_t(a)-1)^2\big]=\sum_a|\hat A_t(a)|\big(\pi^{(j)}(a\mid s_t)-\mu_R(a\mid s_t)\big)^2\quad(\ell_2^2),$$as opposed to SPO's $\sum_a|\hat A_t(a)|(\pi-\mu_R)^2/\mu_R$ ($\chi^2$). The algebra removes the inverse-behavior-probability factor from this regularizer; improved long-tail stability is the motivation and must still be checked empirically.
而 SPO 对应的是 $\sum_a|\hat A_t(a)|(\pi-\mu_R)^2/\mu_R$（$\chi^2$）。该代数会从此正则项中移除行为概率倒数因子；长尾稳定性改善是设计动机，仍需经验证据检验。
DPPO on AIME24 — reported best avg@8
DPPO 在 AIME24 上的表现 — 已报告最佳 avg@8
Part 05
第 05 部分
Toward agentic asynchronous RL stability
迈向智能体异步 RL 的稳定性
Parts 03–04 use a single-turn presentation. Once the LLM becomes an agent — issuing tool calls, receiving structured responses, and continuing in later rounds — the state distribution also becomes turn- and environment-dependent. This adds diagnostic factors not explicitly conditioned on by the methods above; whether they dominate is an empirical question.
第 03–04 部分采用单轮表述。当 LLM 成为智能体 —— 发起工具调用、接收结构化响应并继续后续轮次 —— 状态分布也会依赖轮次与环境。这增加了上述方法未显式条件化的诊断因素；它们是否占主导是经验问题。
During tool-integrated reasoning the agent receives a tool response, typically structured text such as
<python_output> ... </python_output> enclosing arbitrary program stdout. This structured envelope
may be rare or out-of-distribution relative to the model's pre-training and SFT data. In the reported runs, later-round
contexts are associated with higher mismatch; increased entropy is one hypothesis to test, not an automatic consequence:
在工具融合推理中，智能体接收的工具响应通常是结构化文本，如 <python_output> ... </python_output> 包裹的程序输出；相对模型的预训练与 SFT 数据，这些上下文可能稀有或分布外。在已报告运行中，后续轮次上下文与更高失配相关；熵升高是需要检验的假说，而不是自动推论：
where $H_{b,t}^{(j)}$ is the current train policy's token entropy. The fraction of sampled tokens with $\pi^{(j)}(y_{b,t}\mid s_{b,t})<10^{-3}$ is also reported to increase on non-first-round outputs. This is low train-policy probability, not by itself low behavior probability. Such tail probabilities can make relative changes sensitive to small absolute perturbations, but identifying a behavior-tail mechanism requires logging $\mu_{b,t}$ as well; the observed association does not establish that tool responses cause the mismatch.
其中 $H_{b,t}^{(j)}$ 是当前训练策略的 token 熵。报告还显示，非首轮输出中满足 $\pi^{(j)}(y_{b,t}\mid s_{b,t})<10^{-3}$ 的采样 token 占比上升；这是较低的训练策略概率，不能单独解释为较低行为概率。此类长尾概率会让相对变化对小的绝对扰动更敏感，但若要识别行为长尾机制，还必须同时记录 $\mu_{b,t}$；观测相关性并不能证明工具响应导致了失配。
Evidence证据 Empirical signature: KL-stratified analysis, and "more tool calls, more instability"经验证据：按 KL 分层的分析，以及"工具调用越多越不稳"
Partitioning batches by their vLLM–FSDP KL magnitude:
按 vLLM–FSDP KL 的量级给批次分组：
- Group 1: batches with
vllm-kl$\le 10^{-3}$; - 第 1 组：
vllm-kl$\le 10^{-3}$ 的批次； - Group 2: batches with
vllm-kl$\in[10^{-3},2\times 10^{-2}]$; - 第 2 组：
vllm-kl$\in[10^{-3},2\times 10^{-2}]$ 的批次； - Group 3: batches with
vllm-kl$\in[2\times 10^{-2},10^{-1}]$; - 第 3 组：
vllm-kl$\in[2\times 10^{-2},10^{-1}]$ 的批次；
the reported log-ppl / probability scatter plots show larger mismatch on non-first-round outputs than on first-round outputs — larger mean absolute difference and lower Pearson correlation — across the listed vllm-kl bins. In the notation of §1, the observed ordering is
已报告的 log-ppl / 概率散点图显示：在所列 vllm-kl 分箱中，非首轮输出的失配大于首轮输出 —— 平均绝对差更大、皮尔逊相关更低。用 §1 的记号，观测到的排序为
More tool turns are associated with earlier collapse in the reported runs. In the Qwen3-14B-Base off-policy GRPO comparison (clip_high=0.28, 4 mini-batches), the max_tool_turn=100 run collapses earlier than max_tool_turn=20, alongside increases in gradient norm and vllm-kl. This two-setting comparison is suggestive, not a monotonicity theorem.
在已报告运行中，更多工具轮次与更早崩溃相关。在 Qwen3-14B-Base 离策略 GRPO 比较（clip_high=0.28，4 个 mini-batch）中，max_tool_turn=100 比 max_tool_turn=20 更早崩溃，同时梯度范数与 vllm-kl 上升。这是两个设置的提示性比较，不是单调性定理。
The token-level ratio in the agentic regime becomes
智能体场景下的 token 级比率变为
$$r_{b,t}\;=\;\frac{\pi^{(j)}(y_{b,t}\mid s_{b,t})}{\mu_{b,t}(y_{b,t}\mid s_{b,t})},\qquad s_{b,t}\sim P_E\!\big(\cdot\mid\text{tool interaction at turn }k(b,t)\big).$$Here $k(b,t)$ is the tool-interaction turn associated with token $(b,t)$, and $P_E$ is a schematic environment/tool-response context kernel. It summarizes how the environment helps determine the next autoregressive state; it is not an additional policy in the importance ratio.
其中 $k(b,t)$ 是 token $(b,t)$ 所属的工具交互轮次，$P_E$ 是环境 / 工具响应诱导上下文的示意核。它概括环境如何参与决定后续自回归状态，并不是重要性比率中的另一条策略。
Tool-turn OOD effects and tool-response distribution shift are two additional diagnostic hypotheses for why the observed $d_{b,t}=\log r_{b,t}$ changes across turns. They are not a unique additive decomposition unless counterfactual intermediate policies are explicitly defined.
工具轮次 OOD 效应与工具响应分布偏移，是解释观测量 $d_{b,t}=\log r_{b,t}$ 为何跨轮次变化的两个额外诊断假说。若没有显式定义反事实中间策略，就不能把它们写成唯一的精确加法分解。
The mechanisms in Sections 3–4 do not condition explicitly on tool turn. Because SAT uses the batch-relative reference $q$, a nearly uniform rescaling of all mismatches also rescales the reference and can leave the contraction factors unchanged. A turn-conditioned reference is therefore a plausible extension, not yet a proved remedy.
第 3–4 节的机制没有显式以工具轮次为条件。由于 SAT 使用批相对参考 $q$，若所有失配近似同比例变化，参考尺度也会同比例变化，收缩因子可能保持不变。因此按轮次条件化参考是一种合理扩展，但尚未被证明为必然有效的修复。
Takeaways
要点回顾
- The sampled log-ratio $d_{b,t}=\log r_{b,t}$ is an observable train–behavior staleness proxy. It mixes policy change with implementation effects and is not itself a version count or a unique source decomposition.
- 采样对数比 $d_{b,t}=\log r_{b,t}$ 是可观测的训练–行为陈旧度代理；它混合策略变化与实现效应，不等于版本计数，也不是唯一来源分解。
- $D_{\mathrm{TV}}$ gives the trust-region motivation: $D_{\mathrm{TV}}=\tfrac12\mathbb E_\mu|r-1|$, and sampled $|r-1|$ can be viewed as a single-sample Monte Carlo estimator of this $D_{\mathrm{TV}}$ quantity. This makes it natural to express the trust-region intuition in terms of $D_{\mathrm{TV}}$. A smaller $D_{\mathrm{TV}}$ term tightens the approximation; it does not by itself mean a better policy.
- $D_{\mathrm{TV}}$ 给出信任域动机：$D_{\mathrm{TV}}=\tfrac12\mathbb E_\mu|r-1|$，而采样到的 $|r-1|$ 可以看作这一 $D_{\mathrm{TV}}$ 量的单样本 Monte Carlo 估计，因此用 $D_{\mathrm{TV}}$ 来表达信任域直觉是自然的。更小的 $D_{\mathrm{TV}}$ 项会收紧近似，但本身不代表策略一定更好。
- PPO clipping and the stabilizers built on it modify sampled objectives or gradient regions. They do not hard-enforce the full-policy $D_{\mathrm{TV}}$ constraint.
- PPO 裁剪及其上的稳定方法修改的是采样目标或梯度区域，并不会硬性执行完整策略 $D_{\mathrm{TV}}$ 约束。
- SAT strictly contracts the nominal sampled-ratio surrogate when its gate fires: tokens with high observed staleness reach the outward cutoff earlier, while pull-back gradients remain and ratios already past PPO's boundary see no extra change. Any resulting true-$D_{\mathrm{TV}}$ improvement remains conditional or empirical; SAT-GSPO is an empirical sequence-level extension.
- SAT 在门控触发时严格收缩名义采样比率替代目标：观测陈旧度较高的 token 更早触及偏离侧截断，同时保留回拉梯度；已经越过 PPO 原边界的比率不会再发生额外变化。真实 $D_{\mathrm{TV}}$ 是否因此改善仍是条件性的；SAT-GSPO 是经验性的序列级扩展。
- Agentic async-RL adds turn-dependent state distribution shift. Turn-conditioned diagnostics and references are open directions rather than established guarantees.
- 智能体异步 RL 增加了轮次相关的状态分布偏移；按轮次条件化的诊断与参考尺度属于开放方向，而非既有保证。
Future Directions
未来方向
The items above — especially the last one on agentic async-RL — are the directions we plan to pursue next in the study of asynchronous reinforcement learning for large language models. Multi-turn tool-use rollouts inject rare or out-of-distribution contexts whose interaction with policy lag and implementation mismatch is not yet separated. Testing a turn-conditioned SAT variant and deriving a sequence-level bound that includes environment-dependent state shift are two concrete open problems.
以上要点 — 尤其是最后一条关于智能体异步 RL 的部分 — 是我们接下来针对大模型异步强化学习的探索方向。多轮工具调用 rollout 可能引入稀有或分布外上下文，而它们与策略版本差、实现失配之间的交互尚未被分离。检验按轮次条件化的 SAT 变体，以及推导包含环境相关状态偏移的序列级界，是两个具体开放问题。
In asynchronous RL for LLMs, there is a measurable mismatch between the training policy that generates data and the training policy for rollout data. SAT uses that diagnostic to tighten the directional sampled-ratio guardrail on tokens with high observed staleness, aiming to suppress future drift without discarding pull-back gradients.
大模型的异步强化学习中会在生成数据的training策略与rollout数据的训练策略之间产生可测失配。SAT 用该诊断收紧观测陈旧度较高 token 的方向性采样比率护栏，目标是在不丢弃回拉梯度的同时抑制未来漂移。
References
参考文献
References
参考文献
- [1]Zhu, Z., Xie, C., Lv, X., & slime Contributors. (2025). slime: An LLM post-training framework for RL Scaling. GitHub repository. github.com/THUDM/slime
- [2]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., & Sheng, Y. (2023). SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104. arxiv.org/abs/2312.07104
- [3]Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053. arxiv.org/abs/1909.08053
- [4]Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015). Trust Region Policy Optimization (TRPO). ICML 2015; arXiv:1502.05477. arxiv.org/abs/1502.05477
- [5]Qi, P., Zhou, X., Liu, Z., Pang, T., Du, C., Lin, M., & Lee, W. S. (2026). Rethinking the Trust Region in LLM Reinforcement Learning (DPPO). arXiv:2602.04879. arxiv.org/abs/2602.04879
- [6]Yao, J., Zhou, X., Qi, P., Lee, W. S., Bo, L., & Pang, T. (2026). Rethinking the Divergence Regularization in LLM RL (DRPO). arXiv:2606.09821. arxiv.org/abs/2606.09821
- [7]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms (PPO). arXiv:1707.06347. arxiv.org/abs/1707.06347
- [8]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv:2402.03300. arxiv.org/abs/2402.03300
- [9]Yang, A., et al. (2025). Qwen3 Technical Report. arXiv:2505.09388. arxiv.org/abs/2505.09388
- [10]Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A., Zhou, J., & Lin, J. (2025). Group Sequence Policy Optimization (GSPO). arXiv:2507.18071. arxiv.org/abs/2507.18071
- [11]Ma, W., Zhang, H., Zhao, L., Song, Y., Wang, Y., Sui, Z., & Luo, F. (2025). Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (Rollout Routing Replay, R3). arXiv:2510.11370. arxiv.org/abs/2510.11370
- [12]Zheng, C., Dang, K., Yu, B., Li, M., Jiang, H., Lin, J., Liu, Y., Lin, H., Wu, C., Hu, F., Yang, A., Zhou, J., & Lin, J. (2025). Stabilizing Reinforcement Learning with LLMs: Formulation and Practices (TIS & R3-consistent formulation). arXiv:2512.01374. arxiv.org/abs/2512.01374
- [13]Zhao, X., Liu, Y., Xu, K., Guo, J., Wang, Z., Sun, Y., Kong, X., Cao, Q., Jiang, L., Wen, Z., Zhang, Z., & Zhou, J. (2025). Small Leak Can Sink a Great Ship — Boost RL Training on MoE with IcePop. ringtech.notion.site/icepop
- [14]Hou, Z., Li, Y., Tang, J., & Dong, Y. (2026). Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning (SAO / DIS). arXiv:2607.07508. arxiv.org/abs/2607.07508
- [15]Guo, J., Sun, Y., Huang, Z., Wang, Z., Wen, Z., Zhang, Z., Zhou, J., & Kok, S. (2026). KPop: Taming Training–Inference Mismatch in Reinforcement Learning with Adaptive Masking Regions. ringtech.notion.site/kpop
- [16]Microsoft AI Team. (2026). MAI-Thinking-1: Building a Hill-Climbing Machine. microsoft.ai/pdf/mai-thinking-1.pdf
- [17]Xie, Z., Zhang, Q., Yang, F., Hutter, M., & Xu, R. (2024). Simple Policy Optimization (SPO). arXiv:2401.16025. arxiv.org/abs/2401.16025
- [18]Tencent Hy. Stabilizing RLVR via Token-level Gradient Diagnosis and Layerwise Clipping (SeqClip / GradLoc). hy.tencent.com/research/100015; code: github.com/Tencent-Hunyuan/GradLoc
- [19]Dong, C., Goel, A., & Yu, D. (2026). Predicting and Controlling Staleness in Fully Asynchronous RL Training. Applied Compute Research. appliedcompute.com/research/staleness-in-fully-async-rl
- [20]Fan, C., Liu, P., Huang, J., Liu, S., & Lin, Y. (2026). UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration–Stability Dilemma. arXiv:2607.06987. arxiv.org/abs/2607.06987