Online
InterventionOLIVE

Help at the point where learning gets hard.

A teacher can solve a problem beautifully and still give a student a difficult example to learn from. The student makes different choices, ends up in different situations, and needs help with what follows its own attempt. Training on a fixed collection of complete teacher solutions can leave those situations uncovered.

On-policy distillation (OPD) takes this issue seriously: sample from the student, then use the teacher to supervise those samples. Generalized Knowledge Distillation is one example of this approach. We like the principle of meeting the student where it is.

Many OPD approaches may be unstable when the teacher and student are poorly matched. Recent studies trace failures to mismatch between the teacher and student and some approximations to the token-level objective. Online intervention offers a direct way to supply a connected teacher continuation from the student’s own context. See Zhu et al. and Li et al.

There is also a trajectory problem. In token-level OPD, the teacher can recommend a correction at one position while later targets are still conditioned on the student’s original continuation. The student receives advice without seeing the path that follows that advice. Trajectory-Refined Distillation studies this related problem of prefix failure.

We explored a direct way to supply that path: let the teacher take over from the student’s current context and generate a continuation. We call this Online Intervention, or OLIVE.

The teacher continues. The student learns.

OLIVE repeats a short loop throughout training. The current student begins an attempt. A teacher continues from that prefix. We update the student with ordinary cross-entropy loss on the teacher’s continuation, then collect fresh prefixes from the updated student.

For an agent, the prefix is a history of actions and observations. The teacher takes over interaction with the environment for a few turns, and its actions produce the next observations. We keep that history as context and train only on the teacher’s actions. Student turns and environment observations receive no loss.

The teacher only needs to return text. Its output can be retokenized for the student, so this procedure does not require teacher logits or a shared tokenizer. We can also use a short continuation without asking the teacher to finish or verify the whole solution.

We currently hand control to the teacher after a fixed prefix length. The teacher then demonstrates a continuation from that state. Recovery depends on the prefix still admitting a useful continuation.

This draws on a long history of learning from expert help at learner-visited states, including DAgger and On-Policy Expert Corrections. Our focus here is on refreshing those intervention contexts throughout training and learning from the resulting text.

A useful gain on hard reasoning.

We tested reasoning on 18 synthetic RLVE environments, with difficulty chosen to challenge the students. Both students use thinking, and the teacher is Qwen3-4B-Thinking-2507. OLIVE achieves the highest average accuracy among these methods for both student sizes.

The reasoning OPD baseline uses a top-16 approximation to the teacher distribution.

OLIVE more than doubles the smaller student’s average accuracy. The larger student improves on both metrics relative to its starting point. Across both students, OLIVE achieves the highest avg@8 in the comparison. Offline teacher data has the highest pass@8 for the larger student.

Teacher takeover also helped agents.

Across five interactive environments, OLIVE improves success over both the original student and plain OPD. It also leads the compared student-training methods on four of the five tasks, with Guided OPD slightly ahead on BabyAI. The student is Qwen3-1.7B, learning from Qwen3-32B. These results extend the benefit of teacher continuations to tasks where each action changes what the agent sees next.

ScienceWorld is the clearest example of learning from a very weak starting point: OLIVE reaches 7.5% success where the original student and OPD are near zero. Teacher takeover provides a productive learning signal even when the student rarely completes the task on its own.

Student rows report the student’s task performance. The teacher is shown separately for reference.

Less waiting for the teacher.

Online supervision adds a practical cost: a teacher has to generate while training runs. We overlap teacher continuation for one batch with student prefix generation for the next. This reduces idle time, at the price of training on prefixes from a slightly older student.

In our reasoning experiment, asynchronous OLIVE reduced training time by 23.8% relative to synchronous OLIVE. The asynchronous depth was three updates. Pass@8 changed from 54.4% to 52.2%, while avg@8 remained similar.

A short continuation and overlapping generation made the approach practical in this setting. This gives us a practical starting point for larger training runs, where balancing throughput and prefix freshness will be especially interesting.

Analysis

Learn more while preserving what the student knows.

Refreshing the student’s own prefixes improves transfer and preserves more of its general capabilities. We compared complete teacher solutions, teacher continuations from a fixed initial student, and continuations collected from the changing student throughout training.

For this analysis, all three training conditions use complete responses filtered by a verifier. The fixed-prefix baseline collects the student’s attempts once. OLIVE refreshes those attempts as the student learns, so each new intervention responds to its current behavior.

OLIVE achieves the highest task performance on both metrics. Collecting teacher continuations from a fixed student already helps. Refreshing those prefixes online adds a further gain, connecting the teacher’s demonstrations to the student that is learning from them.

The improvement also comes with better preservation of general capabilities. We evaluated math on AIME25, code on LiveCodeBench v6, instruction following on IFEval, and science on GPQA Diamond. The average accuracy drop across these benchmarks was 0.9 points for OLIVE, compared with 3.4 points for offline full SFT. Online intervention improved the target task while preserving more performance outside it.

This analysis uses complete, verifier-filtered continuations. The main reasoning comparison above uses short, unfiltered continuations. The plots use the paper’s pass@k and avg@k metric labels.

A text-only teacher keeps the student learning.

OLIVE turns generated teacher text into a continuing source of supervision. On ScienceWorld, we trained Qwen3-1.7B with GPT-5.4-mini as the teacher. Offline SFT reused a fixed set of teacher episodes. OLIVE collected new teacher continuations from the student’s latest interaction histories.

Offline SFT is ahead during the first two epochs, then its progress levels off. OLIVE keeps improving through the later epochs and finishes well ahead. Refreshing the context changes what the teacher demonstrates as training proceeds, keeping the supervision useful to the evolving student.

More online attempts also help within a single epoch. In the rollout comparison, OLIVE samples multiple prefixes per prompt during one epoch, while the offline baseline revisits its fixed data for more epochs. OLIVE is ahead at the settings with two and four rollouts per prompt. Fresh attempts give the teacher additional student-generated contexts to continue.

Keep learning as tasks change.

We also tested whether online intervention helps a student continue learning across different environments. Starting from Qwen3-4B, we trained for five epochs on each environment in sequence and then evaluated the resulting policy on all five tasks.

OLIVE finishes ahead on average, with the largest gain on ScienceWorld, the final environment in the sequence. It matches offline distillation on BabyAI and is slightly behind on SearchQA. As earlier training changes the student’s behavior, fresh interventions keep providing demonstrations for the states it now reaches. These results support online supervision as a useful way to sustain learning across tasks.

Larger-scale training is next.

We found online intervention pretty useful. We’re taking OLIVE to larger-scale training soon, and we’re excited to explore its scalability.

Our next experiments will bring OLIVE to larger students and longer training runs. We’re especially interested in how fresh teacher interventions support an increasingly capable student, and how asynchronous generation can keep the training loop efficient as the workload grows.

A teacher’s most useful demonstration can start partway through the learner’s own attempt. OLIVE turns those moments into training data throughout learning. The gains in reasoning and agent performance make us excited to push it further.