ReOPD — offline multi-turn distillation without environment rollouts

Why does standard on-policy distillation in multi-turn tool environments become computationally intractable as agent training scales? In a multi-turn mathematical reasoning task with Python code execution, evaluating a student policy $\pi_\theta$ against an expert teacher policy $\pi_T$ requires generating fresh candidate actions $A_t$ and obtaining live environment observations $O_t$ across $T$ interaction steps. Because each policy update alters the student's trajectories, the training loop is forced to execute fresh environment rollouts and query teacher target distributions across newly visited interaction histories $H_t$ at every step ^[1][1] Abstract“Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories.”Open source.

The naive attempt to perform on-policy distillation executes this interactive loop live inside the training pipeline. In a standard setup, the student policy generates actions, sends code snippets to a live Python execution server, waits for output messages, and appends the new observation $O_{t+1}$ to build history $H_{t+1}$. Because policy optimization modifies the student parameters at every gradient step, the student history distribution student history occupancy $d_{\theta_{\mathrm{old}}}^t$ constantly changes. Consequently, the training loop cannot reuse past rollouts and must re-run the live Python environment for every updated batch. When scaling this process to complex agentic workloads that combine code interpreters, web retrieval tools, and databases, hosting and managing all execution environments concurrently introduces severe deployment complexity and hardware contention ^[2][2] Section 1“With an increasing number of environments, the operational complexity grows for OPD due to the heavy deployment of the environments.”Open source.

This fully online requirement creates a severe computational bottleneck. Generating fresh environment observations $O_t$ and querying teacher conditionals $\pi_T(\cdot \mid H_t)$ at every update step causes training throughput to collapse. Live execution latency dominates wall-clock time, tool execution servers require massive process concurrency, and hosting large teacher models alongside active environment containers consumes prohibitive memory resources.

A compelling alternative is off-environment prefix replay: replacing live execution with pre-collected teacher trajectories, where observations $O_t$ are drawn from recorded execution traces. Reusing teacher prefixes removes live environment interactions entirely, allowing offline trajectory pools to be shared across diverse tasks. However, this shift creates a new unresolved tension: while replaying static teacher prefixes eliminates environment overhead, it introduces sequence drift between teacher-generated prefixes and on-policy student actions.

Suppose our LLM agent is attempting a multi-turn mathematical reasoning problem using a Python code interpreter. At step $3$, the student policy student policy generates flawed code, producing an execution error and a traceback history that the expert teacher policy never encountered during its own training. If we evaluate the student strictly on its own rollout history from student history occupancy, the training state is perfectly relevant to the student's operational distribution. However, when we query the teacher for a target distribution on this uncalibrated history, its predictions become erratic because the state lies outside the teacher's reliable domain.

Conversely, if we replace the student's rollout with a pre-collected trace from teacher history occupancy, every code block and environment output is pristine. The teacher delivers sharp, low-entropy target distributions, but the student never learns how to handle its own buggy code. This tension exposes the two-sided distribution shift ^[3][3] Section 3 - Two sources of mismatch“Second, even when histories are relevant to the student, the teacher may be unreliable on histories far from its own interaction support.”Open source: prioritizing student relevance by sampling from student history occupancy corrupts teacher supervision, whereas prioritizing teacher reliability by sampling from teacher history occupancy leaves the student fragile to its own mistakes ^[4][4] Section 3 - Extremes analysis“Fully student-on-policy OPD (\(\rho_t=d_{\theta_{\mathrm{old}}}^{t}\) for all \(t\)) zeroes the occupancy term but not the reliability one: student-generated actions can steer the environment into…”Open source.

To quantify this trade-off, we compare the objective against an ideal target ideal improvement target, which evaluates the student on its own environment occupancy student history occupancy against the true optimal step action. When we substitute the ideal target with the teacher teacher policy on an arbitrary effective prefix distribution, the discrepancy between the ideal interactive objective $\mathcal{R}^\star(\theta;\theta_{\mathrm{old}})$ and the replayed distillation loss $\mathcal{L}_\rho(\theta;\theta_{\mathrm{old}})$ is upper-bounded by two distinct mismatch terms ^[5][5] Equation 4“\mathcal R^\star(\theta;\theta_{\mathrm{old}})”Open source.

The first term in the bound measures student occupancy mismatch via total variation distance $\mathrm{TV}(d_{\theta_{\mathrm{old}}}^t, \rho_t)$, scaled by the uniform loss bound bounded loss constant ^[3][3] Section 3 - Two sources of mismatch“Second, even when histories are relevant to the student, the teacher may be unreliable on histories far from its own interaction support.”Open source. This mismatch shrinks to zero only when the effective prefix distribution effective prefix distribution matches the student's live environment occupancy student history occupancy. The second term measures expected teacher unreliability $\mathbb{E}_{\rho_t}[\epsilon_{T,t}^\theta]$, which remains small only when $\rho_t$ draws histories where the teacher's target distribution stays close to the ideal target $q_t^\star$ ^[5][5] Equation 4“\mathcal R^\star(\theta;\theta_{\mathrm{old}})”Open source.

Evaluating either extreme reveals why simple distribution choices fail ^[4][4] Section 3 - Extremes analysis“Fully student-on-policy OPD (\(\rho_t=d_{\theta_{\mathrm{old}}}^{t}\) for all \(t\)) zeroes the occupancy term but not the reliability one: student-generated actions can steer the environment into…”Open source. Selecting $\rho_t = d_{\theta_{\mathrm{old}}}^t$ zeroes out the occupancy mismatch, but as $t$ increases, student errors compound and drive $\mathbb{E}_{d_{\theta_{\mathrm{old}}}^t}[\epsilon_{T,t}^\theta]$ upward as the teacher is queried on unfamiliar code tracebacks. Conversely, selecting $\rho_t = d_T^t$ minimizes teacher unreliability but maximizes occupancy mismatch $2B \mathrm{TV}(d_{\theta_{\mathrm{old}}}^t, d_T^t)$. Because neither purely student-on-policy nor purely teacher-forced history distributions minimize both error terms simultaneously across all interaction steps, effective distillation requires a mechanism that explicitly balances relevance against reliability.

When an LLM agent solves a multi-turn math problem using a Python interpreter, training prefixes set both where the student learns and where the teacher is asked to supervise. If we sample prefixes strictly from the student's history occupancy student history occupancy, the student encounters states produced by its own past interpreter errors. However, as the student drifts off-track into malformed code contexts, the teacher policy teacher policy is queried on uncalibrated histories where its target conditionals become unreliable ^[4][4] Section 3 - Extremes analysis“Fully student-on-policy OPD (\(\rho_t=d_{\theta_{\mathrm{old}}}^{t}\) for all \(t\)) zeroes the occupancy term but not the reliability one: student-generated actions can steer the environment into…”Open source. Conversely, sampling prefixes solely from teacher occupancy teacher history occupancy provides reliable target distributions, but leaves the student untrained on recovery paths. A naive linear mixture $\frac{1}{2} d_{\theta_{\mathrm{old}}}^t + \frac{1}{2} d_T^t$ fails because it assigns mass to low-probability regions of either distribution without preserving the joint likelihood structure of valid multi-turn interaction traces.

To minimize both KL divergences to student occupancy student history occupancy and teacher occupancy teacher history occupancy, we need a formal interpolation between student and teacher history occupancies ^[6][6] Section 3 - Geometric bridge“When the supports overlap, the solution is the geometric bridge”Open source. We define the optimal effective distribution optimal effective history distribution $\rho_t^\star$ as the minimizer of a trade-off objective between student relevance and teacher reliability ^[7][7] Equation 5 - Bridge objective“\rho_t^\star \in \arg\min_{\rho_t}”Open source:

Solving this constrained optimization problem yields the geometric occupancy bridge ^[8][8] Equation 6 - Geometric bridge“\rho_t^\star(h_t\mid x) \propto”Open source:

The parameter $\gamma_t = \frac{\lambda_{\mathrm{stu},t}}{\lambda_{\mathrm{stu},t} + \lambda_{\mathrm{tea},t}}$ operates as a continuous trade-off weight bridge interpolation weight ^[6][6] Section 3 - Geometric bridge“When the supports overlap, the solution is the geometric bridge”Open source. Setting $\gamma_t = 1$ recovers pure student-on-policy roll-in, maximizing relevance at the expense of teacher reliability. Setting $\gamma_t = 0$ collapses the bridge to teacher-forced roll-in, guaranteeing reliable targets while ignoring student drift. Intermediate values $\gamma_t \in (0, 1)$ select prefixes $h_t$ that are both reachable by the student's current interpreter actions and anchored within the teacher's reliable domain. Crucially, the optimal balance shifts across trajectory steps $t$: at early steps, student and teacher occupancies heavily overlap ($\gamma_t \to 1$), whereas deep in a multi-turn reasoning trajectory, compounding errors degrade teacher reliability, pulling the optimal weight toward teacher support ($\gamma_t \to 0$) ^[6][6] Section 3 - Geometric bridge“When the supports overlap, the solution is the geometric bridge”Open source.

Instantiating this optimal geometric bridge directly requires evaluating density ratios between student and teacher occupancies across multi-turn trajectories ^[6][6] Section 3 - Geometric bridge“When the supports overlap, the solution is the geometric bridge”Open source. However, exact density ratios across long sequences incur high variance, making direct sample reweighting unstable over extended horizons.

When an LLM agent executes Python code across multiple interaction turns to solve a math problem, estimating the exact geometric bridge weight at step $t$ requires knowing how likely the student would have reached that specific execution history. Evaluating the exact density ratio between student and teacher history occupancies requires computing the cumulative likelihood ratio along the replayed teacher prefix. Because both policies evaluate on the identical teacher-recorded prefix $h_t = (O_1, A_1, \dots, O_t)$, environment observation probabilities cancel out completely, leaving the per-history prefix likelihood ratio ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source:

Evaluating this ratio reveals a severe practical flaw. Because every historical action $a_s$ was chosen by the teacher, the student policy $\pi_{\theta_{\mathrm{old}}}$ assigns it a lower probability than the teacher $\pi_T$ does, causing $\widehat{r}_t(x,h_t)$ to decay exponentially toward zero as execution traces extend ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source. Estimating the exact normalized bridge weight $w_t(x,h_t) \propto \widehat{r}_t(x,h_t)^{\gamma_t}$ creates extreme variance across individual trajectories, making loss gradients unstable ^[10][10] Equation 7 - Exact weight“\label{eq:exact-weight}”Open source.

However, empirical measurements show that the step index $t$ accounts for almost all variation in $\widehat{r}_t$, so approximating the log-ratio by the average per-step KL divergence gives $\log \widehat{r}_t \approx -(t-1)\bar{c}$, where $\bar{c} = \mathbb{E}_{a_s \sim \pi_T}[\log(\pi_T / \pi_{\theta_{\mathrm{old}}})] \ge 0$ ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source. Exponentiating this approximation connects the bridge parameter $\gamma_t$ directly to the step-decay base ^[11][11] Equation 8 - Kappa map“\label{eq:kappa-map}”Open source:

This mapping replaces noisy per-history density ratios with a clean position-dependent schedule $\omega(t; \kappa) = \kappa^t$, whose normalization across the trajectory horizon yields step sampling probabilities $p_t \propto \kappa^t$. Instead of evaluating every position and multiplying losses by volatile weights, the training pipeline simply draws interaction step $t$ with probability $p_t$ from the offline trajectory pool and applies unweighted distillation on the student's generated action. Early code execution steps—where student prefix drift is minimal—are sampled frequently, whereas late high-shift steps are sampled exponentially less. This concentrates supervision on low-shift prefixes where teacher conditionals are reliable while eliminating per-history likelihood ratio variance ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source.

We must verify whether training off-environment with prefix decay preserves distillation accuracy without executing tool calls.

This mapping replaces noisy per-history density ratios with a clean position-dependent schedule $\omega(t; \kappa) = \kappa^t$ ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source. Normalizing $\omega(t; \kappa)$ across the trajectory horizon yields a step sampling probability $p_t \propto \kappa^t$ ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source. Instead of evaluating every position and multiplying losses by volatile weights, the training pipeline simply draws interaction step $t$ with probability $p_t$ from the offline trajectory pool and applies unweighted distillation on the student's generated action ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source. Early code execution steps—where the student's prefix drift is minimal—are sampled frequently, whereas late steps—where replayed prefixes become high-shift surrogates—are sampled exponentially less ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source. This concentrates supervision on low-shift prefixes where teacher conditionals are reliable, while eliminating per-history likelihood ratio variance ^[9][9] Section 4 - Step decay weight“so the exact weight's depth profile is geometric:”Open source.

We must verify whether training off-environment with prefix decay preserves distillation accuracy without executing tool calls.

When an LLM agent solves a mathematical reasoning problem across multiple interaction turns, executing Python code snippets at each step consumes significant compute and wall-clock time. In standard online on-policy distillation, the student policy student policy must evaluate its own action $A_t \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid x, h_t)$ on-policy at step $t$. To obtain the interaction history prefix $h_t$, online distillation forces the student to execute live tool calls against the environment for every rollout. This creates a severe throughput bottleneck during student optimization. Can we provide genuinely on-policy training signals at step $t$ without executing a single live tool call?

A naive attempt to bypass live execution is pure off-policy teacher forcing—training the student solely to predict offline teacher tokens. But this fails to provide on-policy feedback: because the student never evaluates its own generated actions, it suffers from severe exposure bias when it branches off-trajectory during multi-turn inference.

ReOPD eliminates live environment calls while preserving on-policy evaluation by utilizing an offline teacher trajectory pool teacher trajectory pool. For a supervised step $t$, the interaction prefix $h_t$ is replayed verbatim from a recorded teacher trace, including all previous teacher actions $A_{<t}$ and environment observations $O_{\le t}$. The student does not re-run Python code to reach step $t$. Instead, it receives $h_t$ as a static context and autoregressively samples its own multi-token action $A_t$ ^[12][12] Section 4 - Off-environment prefix construction“For each supervised step \(t\), the prefix \(h_t=(O_1,A_1,\dots,O_t)\) is replayed \emph{verbatim} from the teacher trace -- all earlier actions \(A_{<t}\) and all observations \(O_{\le t}\) are th…”Open source.

Supervision is computed per generated token along the student's action $A_t = (a_t^1, \dots, a_t^{n_t})$. For each token position $j$, the teacher policy teacher policy evaluates its target conditional $\pi_T(\cdot \mid x, h_t, a_t^{<j})$ using the student's own generated tokens $a_t^{<j}$ as context. Because the evaluated contexts $a_t^{<j}$ are sampled directly by the student, the distillation step is genuinely on-policy ^[12][12] Section 4 - Off-environment prefix construction“For each supervised step \(t\), the prefix \(h_t=(O_1,A_1,\dots,O_t)\) is replayed \emph{verbatim} from the teacher trace -- all earlier actions \(A_{<t}\) and all observations \(O_{\le t}\) are th…”Open source. Crucially, because $h_t$ is replayed verbatim from $\mathcal{D}_T$, no action is executed in the environment and no observation is generated ^[13][13] Section 4 - No environment query“No action is ever executed against the environment and no observation is ever generated, so no environment is queried.”Open source.

This off-environment prefix replay achieves zero tool calls during student training, bypassing process memory constraints and code interpreter deployment bottlenecks. Consequently, ReOPD achieves at least a $4\times$ speedup in rollout throughput relative to standard online distillation ^[14][14] Section 5.1 - Efficiency gains“Unlike OPD, which must execute fresh environment rollouts and tool calls during every student update, ReOPD replays teacher-recorded prefixes and therefore uses zero tool calls during student train…”Open source.

While off-environment prefix replay removes the operational bottleneck of online tool calls, it relies entirely on replayed teacher traces. It remains to test whether ReOPD maintains reasoning accuracy across diverse environment domains and under varying teacher-student capability gaps.

In a multi-turn mathematical reasoning problem with a Python code interpreter, an LLM agent executes code across $T$ interaction steps. When standard online on-policy distillation rolls out the student policy student policy in a live execution environment, early student mistakes steer interaction into uncalibrated states where the teacher policy teacher policy was never trained. At these drifted histories, teacher supervision fails because the teacher's conditional action distribution is an unreliable proxy for true solution progress. One might assume that executing live student rollouts is essential for effective on-policy learning; however, when the capability gap between teacher and student is wide, forcing student-on-policy roll-ins actively degrades distillation quality. ReOPD resolves this trap by keeping the roll-in anchored to replayed prefixes replayed interaction prefix from the offline teacher trajectory pool teacher trajectory pool, evaluating the student step action student step action on states where teacher targets remain trustworthy ^[15][15] Section 5.2 - Two regimes in practice“the teacher-anchored pool wins when the gap is large and ties student-on-policy OPD when the teacher is already reliable”Open source.

This structural distinction divides multi-turn distillation into two clear environment regimes ^[15][15] Section 5.2 - Two regimes in practice“the teacher-anchored pool wins when the gap is large and ties student-on-policy OPD when the teacher is already reliable”Open source. On complex mathematical reasoning tasks, where the teacher-student model gap is large, teacher reliability shift dominates. Evaluating a Qwen3-4B student under a Qwen3-4B teacher on Python-assisted math benchmarks, ReOPD achieves an average accuracy of $57.2$, outperforming online OPD's $55.1$. When scaling to a stronger Qwen3-8B teacher, ReOPD widens its advantage over online OPD ($53.7$ vs $51.0$ average accuracy), precisely because teacher unreliability on drifted student rollouts becomes the primary performance bottleneck ^[16][16] Section 5.1 - Single environment math results“With a Qwen3-4B teacher and Qwen3-4B student, ReOPD improves the average from $55.1$ to $57.2$; with a Qwen3-8B teacher, it improves from $51.0$ to $53.7$”Open source. Conversely, in search and question-answering environments where the teacher remains reliable on student-induced histories, student occupancy shift dominates ^[15][15] Section 5.2 - Two regimes in practice“the teacher-anchored pool wins when the gap is large and ties student-on-policy OPD when the teacher is already reliable”Open source. Here, ReOPD essentially matches online OPD ($40.5$ vs $40.6$ average accuracy under a Qwen3-4B teacher) because teacher-anchored prefixes and student-induced histories nearly overlap ^[17][17] Section 5.1 - Search results“With a Qwen3-4B teacher, OPD and ReOPD obtain $40.6$ and $40.5$ average accuracy, respectively;”Open source.

ReOPD also scales seamlessly to multi-environment settings without added operational complexity. Training a single student agent across heterogeneous tool environments normally requires deploying live Python interpreters and search engines concurrently during every gradient update. By decoupling prefix collection from student optimization, ReOPD allows distinct domain teachers to build offline trajectory pools independently, which are then merged into a unified dataset for joint student training. On joint math and search benchmarks, a single student trained via ReOPD achieves $55.3$ math average and $41.0$ search average, matching online OPD while eliminating live environment hosting entirely ^[18][18] Section 5.1 - Multiple environments“ReOPD remains on par with OPD in both domains while avoiding online environment interaction during student training.”Open source. This resolves our central throughline: LLM agents can learn from dense on-policy distillation off-environment with zero live tool calls during student training, achieving at least a $4 imes$ speedup per rollout without degrading reasoning accuracy ^[14][14] Section 5.1 - Efficiency gains“Unlike OPD, which must execute fresh environment rollouts and tool calls during every student update, ReOPD replays teacher-recorded prefixes and therefore uses zero tool calls during student train…”Open source.