Kernel alignment · Importance sampling · Asynchronous RL

REINFORCE Zero

A minimal loss. 20,000 stable updates.

Kernel Alignment Unlocks Stable Asynchronous RL
with Importance Sampling

Trainer–Inference Invariance (TII) aligns rollout and learner execution. Unclipped importance sampling (IS) corrects weight lag, without ratio clipping or sample-rejection gates.

Jian HuHao ZhangYifan ZhangBinfeng XuAo ShenShaokun ZhangYunheng ZouYan BaiJan KautzYi Dong

NVIDIA × Inferact / 2026

Read the paper GitHub Agent skill Molt

ResearchGate ↗ alphaXiv ↗Cite this work ↓

20,000updates: Stable async training
Policy lags 2 & 8¹
2.40×: Training-step throughput
vs. Sync TII · 16 GPUs
64lag: Qwen3-4B + Python
6,000 policy updates
30BMoE: Reasoning with tools
Qwen3-30B-A3B

01 The finding

20,000 updates.
Still learning.

With aligned kernels, the simplest importance-sampling correction sustains asynchronous LLM RL through 20,000 policy updates.

Trainer–Inference Invariance (TII) makes rollout and learner token probabilities agree at identical weights and histories. Ordinary, unclipped token-level importance sampling (IS) corrects the remaining weight lag.

On DeepSeek-R1-Distill-Qwen-1.5B, reasoning quality improves while rollout-to-learner KL stays flat at lags 2 and 8¹. Standard-kernel Async IS shows an 83.33× rise in KL.

Reasoning accuracy improves through 20,000 updates at policy lags 2 and 8, while KL remains flat. Standard-kernel IS shows an 83.33-fold KL increase. — Figure 1 Five-benchmark macro pass@1 with an 8K response budget (left); sampled rollout-to-learner KL (right). Click the figure to enlarge.
¹ “Lag 8” denotes the nominal pipeline setting. Measured rollout-to-learner policy lag averages approximately 10.5 learner updates, with long-response tails reaching 36–38 updates.

Open research. 20,000 stable asynchronous updates with TII and unclipped importance sampling.

02 The mechanism

Two mismatches.
Two different jobs.

Kernel mismatch causes sharp, execution-dependent corrections. Once kernels align, ordinary IS can handle the smoother discrepancy from weight lag.

01 / ALIGN EXECUTION

Remove kernel mismatch.

Different numerical execution can change token probabilities even at identical weights and histories. Batch-invariant kernels and matched forward computation establish TII.

Same weights. Same histories. Matching probabilities.

02 / CORRECT WEIGHT LAG

Let ordinary IS work.

Rollouts use older policy snapshots. Reweight each sampled action token by the current learner probability divided by its recorded behavior probability.

Unclipped token IS. A minimal REINFORCE loss.

logπ_θₖμ_v = logπ_θₖπ_θᵥWeight lag → IS + logπ_θᵥμ_vKernel mismatch → TII

A MINIMAL REINFORCE OBJECTIVE

Align kernels. Correct lag. Keep learning.

\hat{ℒ} (θ) = - \frac{1}{B} \sum_{i = 1}^{B} \frac{1}{T_{i}} \sum_{t = 1}^{T_{i}} A_{i}

\times \frac{π_{θ} (a_{i t} ∣ h_{i t})}{μ_{i} (a_{i t} ∣ h_{i t})}

A_i is the response advantage; T_i counts valid action tokens. The denominator μ_i is the recorded behavior probability.

Differentiate through the current-policy numerator π_θ. Average over action tokens within each response, then over responses.

Smoother weight lag contrasts with fluctuating kernel mismatch. The D3 diagnostic peak-to-median ratio is 1.50 for aligned kernels and 442.54 for standard kernels. — Figure 2 Batch-averaged importance-weight diagnostic D₃, peak / median: **442.54 standard → 1.50 aligned.** The right axis is logarithmic.

03 Why it works

Why TII unlocks
importance sampling.

IS restores the target mean update at a fixed token history. Sampling mismatch can still amplify its variance.

The Gaussian analysis shows how mismatch increases update noise and narrows the stable learning-rate range. The same learning rate can yield bounded error with matched sampling and growing error with mismatch.

TII Suppress execution-induced spikes.
IS Preserve the full correction.

Explore the derivation

In the Gaussian model, matched sampling has bounded mean-square error near 0.30, while mismatch of kappa 0.2 produces error of 68.27 at update 200. — Figure 3 Exact expected squared error. Same learning rate, different sampling mismatch. Logarithmic vertical axis.

The full IS correction

Correct the mean.
Control the update noise.

At a fixed token history, let p_θ be the target policy, q the actual sampler, A a fixed advantage signal, and s = ∇_θ log p_θ. The sampler covers the target support; w = p_θ / q.

01 / TARGET UPDATE

The target covariance.

Under the target policy, the score has zero mean.

g^{⋆} = 𝔼_{p} [A s]

= {Cov}_{p} (A, s)

02 / SAMPLING DISTRIBUTION

Covariance plus score drift.

The uncorrected update includes a score-mean drift term.

𝔼_{q} [A s] = {Cov}_{q} (A, s)

+ 𝔼_{q} [A] 𝔼_{q} [s]

03 / SCORE CENTERING

Center the sampler score.

SC removes score-mean drift and retains the covariance under q.

g_{SC} = {Cov}_{q} (A, s)

Residual covariance bias

g_{SC} - g^{⋆}

= {Cov}_{q} (A, s) - {Cov}_{p} (A, s)

04 / UNCLIPPED IMPORTANCE SAMPLING

Restore the full target update.

IS restores both the zero-mean target score and the target covariance.

𝔼_{q} [w s] = 0

g_{IS} = 𝔼_{q} [w A s]

= {Cov}_{p} (A, s) = g^{⋆}

TII suppresses execution-induced spikes and update noise. Unclipped IS preserves the full correction of the mean update.

04 Experiments

From reasoning
to tool-using agents.

Long training runs, high policy lag, mixture-of-experts models, and visual computer use.

NEMOTRON-3-NANO-OMNI-30B

From tokens
to desktop actions.

A Mamba, attention, and MoE hybrid learns visual computer-use tasks from screenshots, mouse actions, and keyboard input.

OSWorld evaluation · task-weighted pass@1

50.97%Base model56.16%After 428 updates

+5.19 percentage points · 49K-token context with image inputs.

OSWorld training pass@1 and sampled rollout-to-learner KL for REINFORCE Zero through 428 updates and FlashREINFORCE through 194 updates. — OSWorld training Training pass@1 (left) and sampled rollout-to-learner KL (right). REINFORCE Zero: 428 updates; FlashREINFORCE: 194 updates.

OVERLAP GENERATION + OPTIMIZATION

2.40× training-step throughput.

Same 16 GPUs. TII in both runs. Comparable reasoning quality.

164.77 h Sync TII68.55 h REINFORCE Zero

12,000 policy updates; training-step timing excludes evaluation.

05 Agent-driven engineering

Kernel alignment,
automated by agents.

A reusable agent skill automates kernel adaptation, numerical validation, gradient checks, and performance tuning across attention, MoE, and Mamba hybrid architectures.

Prepare the reference.

Map the model’s operators, measure the performance baseline, and build a small proxy that preserves the target computation.

MAP · MEASURE · MODEL

Align & validate.

Match rollout and learner forward passes. Verify gradients and optimizer updates against an independent reference.

ALIGN · CHECK · VERIFY

Scale & optimize.

Bring alignment to the full model, recover throughput, and repeat numerical checks after each optimization.

SCALE · PROFILE · REFINE

06 Paper & resources

Explore the work.

The paper, algorithm guide, and complete alignment skill are publicly available. Training implementation lives in Molt.

PAPER / PDF

REINFORCE Zero

Kernel Alignment Unlocks Stable Asynchronous RL with Importance Sampling

Read the paper

ResearchGateRead and discuss the paper alphaXivExplore the research Algorithm guideUpdate, derivations, and alignment workflow MoltLightweight framework · training implementation

Citation

Build on this work.

Download .bib

@article{hu2026reinforcezero,
  title   = {REINFORCE Zero: Kernel Alignment Unlocks Stable
             Asynchronous RL with Importance Sampling},
  author  = {Hu, Jian and Zhang, Hao and Zhang, Yifan and Xu, Binfeng
             and Shen, Ao and Zhang, Shaokun and Zou, Yunheng
             and Bai, Yan and Kautz, Jan and Dong, Yi},
  year    = {2026},
  url     = {https://github.com/hijkzzz/REINFORCE-Zero}
}

REINFORCE Zero

20,000 updates.
Still learning.