Continual Learning Bench — isolating improvement from capability

In the database example, an agent learns which table groups map to product categories and how values are encoded, then answers later questions with fewer queries. Halfway through, a migration makes some of that knowledge stale. ^[1][1] Introduction“A learning agent progressively learns the schema and data---which table group maps to which product category, how prices are encoded---and answers later questions with far fewer queries. Halfway, a…”Open source A high later score still leaves a gap: did sequential experience improve the agent, or did retrieval and prior database competence merely expose an answer it could already produce?

Start with the obvious evaluation: place the earlier schema fact far back in context, then ask for it later. Success establishes that the system can preserve and retrieve information across a long history. It does not establish that the information changed what the system could do. A perfect transcript searcher would pass even if every later answer were already within its static capabilities. This is why the paper argues that recall, long-context question answering, context compaction, and isolated adaptation proxies do not directly test improvement from environment-specific structure. ^[2][2] Introduction“These proxies do not directly test whether a system improves online by learning environment-specific latent structure across related tasks.”Open source

Chaining ordinary tasks does not fix the problem. Suppose each of the 40 database questions uses an unrelated schema. Earlier exploration then contains nothing that helps with a later question. A stronger static model may still score highly, and a weaker model may still score poorly, but neither result measures learning from the sequence. The same defect survives in an iterative benchmark when instances share no hidden rule that experience can uncover and reuse. The paper contrasts this with tasks where latent structure is hidden, task-specific, and transferable across later instances. ^[3][3] Related Work“Without such a structure, a stronger static model may simply do better, preventing the isolation of continual learning. Our work explicitly designs tasks in which shared latent structure exists and…”Open source

Continual learning therefore requires a causal chain: an earlier interaction reveals a hidden, environment-specific regularity; that regularity recurs; retaining it changes a later action; and the changed action improves the outcome. In the running example, discovering one product group’s price encoding should reduce exploration on a later question. After the migration, new evidence should cause stale schema beliefs to be revised rather than replayed. ^[1][1] Introduction“A learning agent progressively learns the schema and data---which table group maps to which product category, how prices are encoded---and answers later questions with far fewer queries. Halfway, a…”Open source Memory is only the carrier; improvement from reusable and revisable latent structure is the target. ^[4][4] Introduction“\bench{} tasks are developed to contain latent structure that is realistic but not known a priori, minimizing the confound of performance reflecting pretrained knowledge. Agents are challenged to l…”Open source

That target creates a harder design problem. Prior experience must be useful enough to improve later performance, yet the task must remain difficult enough that pretrained capability cannot solve every instance immediately. Otherwise the evaluation offers either nothing to learn or no headroom in which learning can appear.

Put forty ordinary database questions in a row and the schedule looks sequential. But suppose each question uses unrelated tables, encodings, and answer rules. A note from question three cannot reduce the work on question eight; history has become storage without leverage. This is why chaining standard benchmark instances fails: such instances are usually diverse and independent, so a sufficiently capable static model can score well without learning from the run. ^[5][5] Task Admission Criteria“Standard benchmark instances are designed to be diverse and independent, and test offline training, meaning a sufficiently capable model could perform well without any online learning.”Open source

A task first needs headroom: initial performance must remain well below the achievable maximum. Otherwise, pretrained competence already supplies the answer, and no observable improvement is available to attribute to online learning. ^[6][6] Task Admission Criteria“Initial performance for any system should be well below the achievable maximum. This means the latent structure to be learned must be task-specific and not recoverable from general offline training…”Open source In the running example, the agent does not receive the Amazon-review schema upfront. Abbreviated columns, category-specific price units, missing tables, and distractor columns make immediate zero-shot efficiency unlikely while leaving the questions solvable through exploration. ^[7][7] Database Exploration“Column names are obfuscated and abbreviated (e.g., \texttt{price -> prc}, \texttt{timestamp -> ts}) and are shared across product categories, masking per-category semantic differences.”Open source

Difficulty alone is not enough. Forty unrelated hard puzzles preserve headroom but still provide nothing reusable. The second criterion is hidden shared structure: recurring schema conventions must span instances, remain initially unstated, and become inferable from experience. ^[8][8] Task Admission Criteria“There must exist discoverable structure (such as codebase structure, schema conventions, opponent strategies) that is \emph{shared} across instances and that a system can exploit for performance im…”Open source Once the agent discovers that Electronics reviews live in a particular table and verified purchases use a particular integer field, that discovery can remove exploratory queries from a later question. The earlier observation changes a later action. After the migration at question twenty, some cached facts stop applying, so the same task also requires selective revision rather than unconditional reuse. ^[9][9] Database Exploration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source

Shared structure still cannot produce learning if the interaction reveals nothing about it. The third criterion is informative feedback: earlier instances must return observations such as query results or errors that can convert an attempted action into actionable knowledge for later instances. ^[10][10] Task Admission Criteria“Earlier instances in the schedule must produce observations that are informative for later ones. The environment must provide a feedback loop that a learning system can exploit (e.g., test failures…”Open source The three conditions form one causal chain: headroom leaves room to improve; hidden shared structure makes early discoveries transferable; feedback lets the agent discover which beliefs deserve transfer. The benchmark's expert validation separately checks realistic workflow, reusable knowledge, and measurable improvement rather than assuming that a plausible sequence is learnable. ^[11][11] Task Validation“Two to three domain experts reviewed each task specification against the criteria below.”Open source Yet admission only establishes that learning is possible. If later database questions happen to be easier than earlier ones, a high or rising reward curve can still credit learning that never occurred.

Question 8 arrives after seven chances to learn the obfuscated Amazon-review schema. If its reward exceeds question 1’s, the tempting explanation is that the agent learned which table and column encode verified purchases. But question 8 may simply require fewer exploratory queries. Reversing the order could reverse the apparent learning curve without changing the agent at all. The paper makes this confound explicit: instance difficulty varies, so a strong fixed system can score highly while a weaker system that uses experience still has lower absolute reward ^[12][12] Reward“When instance difficulty is homogeneous, a rising reward curve is a reliable signal of learning. In practice, however, instance difficulty varies, and different systems have different baseline perf…”Open source.

First run the agent normally through questions 1–8, preserving its conversation, retrieved memories, or other state. On question 8, record stateful reward. This result contains two possible causes: the capability of the underlying system and information accumulated from questions 1–7. A reward alone cannot separate them. The benchmark permits state to persist explicitly, through mechanisms such as retrieved memory, or implicitly in accumulated context ^[13][13] Reward“\textbf{Stateless} ($sl$) systems condition only on the current instance, $\pi_t = \pi(x_t)$, independent of history $\mathcal{H}_{t-1} = \{(x_i, r_i)\}_{i=1}^{t-1}$; by construction, they cannot i…”Open source.

Now rerun the same end-to-end system in stateless mode on question 8 and record stateless reward. The matched comparison keeps the system and instance fixed while changing access to prior history ^[14][14] Gain“For each instance $t$, we define gain as: $g_t = r^{sf}_t - r^{sl}_t$ where $r^{sf}_t$ is the reward achieved by the stateful system and $r^{sl}_t$ is the reward achieved by the same system run in…”Open source ^[13][13] Reward“\textbf{Stateless} ($sl$) systems condition only on the current instance, $\pi_t = \pi(x_t)$, independent of history $\mathcal{H}_{t-1} = \{(x_i, r_i)\}_{i=1}^{t-1}$; by construction, they cannot i…”Open source. It asks what the same system earns without accumulated experience, rather than comparing it with a different system or a different question.

Because the same instance appears in both terms, whatever makes it intrinsically easy or hard appears equally and cancels in the difference ^[14][14] Gain“For each instance $t$, we define gain as: $g_t = r^{sf}_t - r^{sl}_t$ where $r^{sf}_t$ is the reward achieved by the stateful system and $r^{sl}_t$ is the reward achieved by the same system run in…”Open source. In the running example, remembered schema conventions could make the stateful reward higher; stale pre-migration state could make it lower. The instance gain therefore measures the contribution of accumulated state without assuming that retained state helps.

This solves attribution within one instance, but it does not yet make tasks comparable. A reward difference of $0.1$ has different weight on different natural reward scales ^[15][15] Reward“Tasks differ in their natural reward scale as a result, but all satisfy these properties by construction.”Open source. Even on a common scale, a stateless system already near the task ceiling has less remaining room to improve. Per-instance differences isolate state; aggregation still needs to account for scale and available headroom.

Suppose a stateful database agent saves three exploratory queries because it remembers an obfuscated column. Adding that improvement to gains from other tasks seems natural, but a task measured in large reward units will dominate an equally effective improvement measured in small units. Worse, the same raw gain means different things when one stateless baseline is already near the task ceiling. The benchmark’s tasks have different natural reward scales, so summing unadjusted differences cannot support a cross-task ranking ^[15][15] Reward“Tasks differ in their natural reward scale as a result, but all satisfy these properties by construction.”Open source.

Suppose the stateful database agent earns $0.8$ on one question while its stateless run earns $0.5$. The gain is $0.3$. If the scoring rule expressed both rewards in hundredths, the difference would become $30$ without any change in behavior. A raw total would treat that unit conversion as more learning. First average the matched differences over the task schedule.

The mean stateless reward records how much of the task the system could already solve without history. What remains is the distance from that baseline to the task ceiling.

Subtracting the stateless mean from maximum reward produces the available headroom. Because both values use the same task reward units, rescaling the scoring rule multiplies the numerator and denominator together.

For an illustrative database run, let the stateless mean be $0.6$, the stateful mean be $0.7$, and the maximum be $1$. The mean gain is $0.1$, while the available headroom is $0.4$, so normalized gain is $0.25$. Multiplying every reward by $100$ leaves that fraction unchanged. The stateful mean remains valid because the $0.1$ gain is compensated by moving from the $0.6$ baseline to $0.7$, still below the maximum.

A positive normalized gain therefore says that history improved average performance relative to the matched stateless counterfactual. It does not say why. Full conversation history, a curated notepad, and retrieved memories can all produce positive values, yet the metric alone cannot show whether memory management helped more than simply preserving the complete context.

Before the database migration, an agent can learn which table group maps to each product category and how prices are encoded. Halfway through the task, the migration makes some of that knowledge stale. ^[1][1] Introduction“A learning agent progressively learns the schema and data---which table group maps to which product category, how prices are encoded---and answers later questions with far fewer queries. Halfway, a…”Open source The obvious memory strategy is to compress each discovery into a durable note. But compression must choose before future questions are known: keep a broad schema rule and risk carrying it past the migration, or keep a narrow query pattern and risk omitting an observation needed later. A compact memory can preserve the wrong thing.

The benchmark compares that strategy with several ways of retaining experience. Full-context in-context learning, or ICL, preserves the conversation without truncation unless the context window is exceeded. ICL Notepad makes the model rewrite structured notes after each instance. Mem0 extracts memories automatically and retrieves the ten judged most relevant to the current instance, while ACE updates a persistent playbook. Each dedicated mechanism discards or rewrites information; full-context ICL postpones that selection and leaves the model access to the original observations. ^[17][17] Evaluating Frontier Systems on“Full-context ICL preserves the entire conversation history without truncation unless context windows are exceeded, providing a strong context-based baseline. ICL Notepad gives the agent a persisten…”Open source

If curation reliably identified reusable structure, the dedicated systems should capture more available headroom. They did not. Full-context ICL with Claude Sonnet 4.6 achieved the highest aggregate normalized gain among the evaluated systems at $25.4\%$. ICL Notepad with the same model reached $18.2\%$, while ACE ranked tenth at $8.6\%$ despite having the highest reported full-run cost. More broadly, ICL-based systems occupied three of the top five gain positions. The learning medium changed how much prior experience became useful action even when the backbone model stayed fixed. ^[18][18] Results“~\cref{tab:overall-results} shows that full-context ICL with Claude Sonnet 4.6 achieves the highest aggregate normalized reward (22.3\%) and gain (25.4\%) of any evaluated system. More broadly, ICL…”Open source

This is an aggregate result, not a claim that compression always fails. Claude Code placed second in gain at $23.9\%$ and exceeded every ICL configuration on individual Sales Prediction and Database Exploration gains. But no dedicated memory system displaced full-context ICL from the aggregate lead. The ranking establishes that extra memory machinery did not reliably improve reuse; it does not establish that compression caused every deficit. ^[18][18] Results“~\cref{tab:overall-results} shows that full-context ICL with Claude Sonnet 4.6 achieves the highest aggregate normalized reward (22.3\%) and gain (25.4\%) of any evaluated system. More broadly, ICL…”Open source ^[19][19] Results“The frontier on the gain-vs-cost plot is largely occupied by ICL variants, with Claude Code providing the one clear exception: it achieves the second-highest gain (23.9\%) at moderate cost (\$38.6)…”Open source

The aggregate leaves a concrete question unresolved. When the database migrates and some prior knowledge becomes stale, did a system discard structure that remained useful, or did it keep applying a rule that no longer fit? ^[1][1] Introduction“A learning agent progressively learns the schema and data---which table group maps to which product category, how prices are encoded---and answers later questions with far fewer queries. Halfway, a…”Open source The same total gain cannot distinguish those failures.

A database migration makes some learned schema and data facts stale. If performance then falls, one aggregate gain cannot show whether the agent lost transferable structure, misapplied old information, or failed to adapt after receiving new feedback. Systems with the same normalized gain can differ sharply in retention across variants and adaptation within them. ^[1][1] Introduction“A learning agent progressively learns the schema and data---which table group maps to which product category, how prices are encoded---and answers later questions with far fewer queries. Halfway, a…”Open source ^[20][20] Stability–Plasticity Decomposition of Gain“Two systems can demonstrate the same $\widehat{g}$ while differing with respect to these terms: a highly plastic learner can adapt well within each task variant while failing to carry that adaptati…”Open source

The obvious fix is to compare average gain before and after the migration. That still mixes two different tests. On the first question after a variant switch, the agent has no feedback from the new variant; only its accumulated state can help. Later questions follow at least one round of post-migration feedback, so their gains also reflect adaptation. The benchmark therefore partitions each variant into its boundary instance and its remaining within-variant instances. ^[21][21] Variant-based Partition“Boundary instances are the moments at which the system encounters a new variant and has only its prior accumulated state to draw on; within-variant instances follow at least one round of in-variant…”Open source

Boundary gain tests stability. The benchmark compares stateful and stateless mean reward on the first instance of each variant, scales that difference by the fraction of schedule instances at boundaries, and divides by the task-level headroom used for total normalized gain. Because no feedback from the new variant is available yet, a positive component indicates useful transfer from prior state; a negative component indicates that learned information was misapplied. ^[21][21] Variant-based Partition“Boundary instances are the moments at which the system encounters a new variant and has only its prior accumulated state to draw on; within-variant instances follow at least one round of in-variant…”Open source ^[22][22] Decomposition“Using the same task-level headroom denominator $r_{max} - \bar{r}^{sl}$ as in $\widehat{g}$ (\cref{sec:cross_task_norm}), we define the stability component: $$ \widehat{g}_{\text{stab}} \;=\; f_\m…”Open source ^[23][23] Decomposition“The signs of each term can be interpreted in the same way as $\widehat{g}$ from \cref{sec:gain}. For example, a system that retains useful structure across variant boundaries will indicate $\wideha…”Open source

Within-variant gain tests plasticity. It compares stateful and stateless mean reward after each variant’s first instance, scales that difference by the within-variant fraction, and uses the same task-level headroom denominator. These instances follow at least one round of in-variant feedback, so a positive component indicates that accumulated state helps within the new variant. Negative boundary gain followed by positive within-variant gain captures an initially harmful carryover from the old database followed by recovery. Because every instance belongs to exactly one partition, the stability and plasticity components add to total normalized gain. ^[21][21] Variant-based Partition“Boundary instances are the moments at which the system encounters a new variant and has only its prior accumulated state to draw on; within-variant instances follow at least one round of in-variant…”Open source ^[22][22] Decomposition“Using the same task-level headroom denominator $r_{max} - \bar{r}^{sl}$ as in $\widehat{g}$ (\cref{sec:cross_task_norm}), we define the stability component: $$ \widehat{g}_{\text{stab}} \;=\; f_\m…”Open source ^[23][23] Decomposition“The signs of each term can be interpreted in the same way as $\widehat{g}$ from \cref{sec:gain}. For example, a system that retains useful structure across variant boundaries will indicate $\wideha…”Open source

The reported systems separate along these axes: ICL Notepad with Claude Sonnet 4.6 shows the most stable learning but weaker adaptation, while ICL and Claude Code show the most plastic learning; some systems show virtually no stable learning despite gaining from repeated exposure within variants. ^[24][24] Results“With this decomposition, we indeed see significant variance in how much each component contributes to aggregate learning across agents and systems (\cref{fig:learning-decomposition}), with the ICL…”Open source Stored state therefore counts as continual learning only when the matched counterfactual shows selective reuse at boundaries and revision after new evidence. Retention alone can preserve the wrong database, and adaptation alone can repeatedly relearn what memory should have carried forward.