Continual Learning Bench — isolating learning from capability

The database agent answers $40$ questions about obfuscated product tables.^[1][1] Database Exploration“The agent answers a sequence of 40 natural-language questions about an unknown SQLite database containing Amazon product reviews across three product groups: Office Products, Electronics, and Music…”Open source Suppose it scores well from question $1$, then improves further after question $20$. Neither fact proves that earlier questions taught it anything: the base model may already know enough SQL to perform well, and the later questions may simply be easier.^[2][2] Reward“When instance difficulty is homogeneous, a rising reward curve is a reliable signal of learning. In practice, however, instance difficulty varies, and different systems have different baseline perf…”Open source The benchmark’s migration after question $20$ makes this especially awkward because some stored schema facts become stale just as the question set changes.^[1][1] Database Exploration“The agent answers a sequence of 40 natural-language questions about an unknown SQLite database containing Amazon product reviews across three product groups: Office Products, Electronics, and Music…”Open source

The naive test is to plot reward against question number. If the curve rises, call it learning. That inference silently assumes comparable difficulty across the sequence. Remove that assumption and the same curve admits two causal stories. In the first, prior observations reveal that Electronics prices use integer cents, so later answers need fewer exploratory queries.^[1][1] Database Exploration“The agent answers a sequence of 40 natural-language questions about an unknown SQLite database containing Amazon product reviews across three product groups: Office Products, Electronics, and Music…”Open source In the second, later questions happen to require simpler joins. A system with no memory could produce the second curve. High reward can likewise come from static capability: a strong base model may outperform a weaker learner while updating nothing. The paper calls out both varying instance difficulty and different baseline performance as reasons reward alone is insufficient.^[2][2] Reward“When instance difficulty is homogeneous, a rising reward curve is a reliable signal of learning. In practice, however, instance difficulty varies, and different systems have different baseline perf…”Open source

This is the static-capability confound: absolute competence is mistaken for improvement caused by sequential experience. Continual learning sets a stricter causal requirement. Performance must improve because information from earlier online interactions changes how the system handles later instances; neither pretrained competence nor a favorable ordering of questions qualifies. That is why the paper defines continual learning as improvement through sequential experience and designs its metrics to separate learning from prior capabilities.^[3][3] Introduction“Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it.”Open source

The distinction changes what an evaluation must ask. Not “does the database agent score well?” and not even “does its score rise?”, but “would this same agent perform worse on the same question without the useful information accumulated earlier?” Before that comparison can mean anything, the sequence needs a real learning opportunity: earlier observations must reveal task-specific information that pretraining could not supply and later questions can reuse.^[3][3] Introduction“Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it.”Open source

The database agent can answer $40$ questions in order and still learn nothing. If each question concerns an unrelated schema quirk, the transcript is merely longer: success on question $19$ supplies no leverage on question $20$. Chaining independent benchmark instances therefore creates history without creating a reason for history to matter; a sufficiently capable static model can solve the stream without updating.^[4][4] Task Admission Criteria“Importantly, these criteria jointly preclude constructing \bench{} tasks by chaining instances from existing benchmarks. Standard benchmark instances are designed to be diverse and independent, and…”Open source

An admissible task needs three properties. First, headroom: initial performance must sit below an achievable maximum because the hidden structure is task-specific rather than recoverable from offline training alone. Second, shared hidden structure: facts discovered on one instance must recur in later instances, turning earlier work into a reusable model of the environment. Third, informative feedback: earlier actions must return observations—such as query results or errors—that let the system infer that structure and convert it into a better later decision.^[5][5] Headroom criterion“\paragraph{Headroom.} Initial performance for any system should be well below the achievable maximum. This means the latent structure to be learned must be task-specific and not recoverable from ge…”Open source^[6][6] Shared latent structure criterion“\paragraph{Shared latent structure.} There must exist discoverable structure (such as codebase structure, schema conventions, opponent strategies) that is \emph{shared} across instances and that a…”Open source^[7][7] Learning mechanism criterion“\paragraph{Learning mechanism.} Earlier instances in the schedule must produce observations that are informative for later ones. The environment must provide a feedback loop that a learning system…”Open source

Remove any leg and the interpretation breaks. No headroom leaves nothing measurable to improve. No shared structure makes earlier evidence irrelevant. No feedback leaves useful structure present but inaccessible.

Apply the test to the database sequence. The unfamiliar schema leaves headroom: even a strong SQL model does not receive the table structure, encodings, or cross-group differences upfront. Repeated conventions supply shared structure: after exploration reveals which group stores prices in cents or which timestamp format applies, the agent can reuse those facts instead of rediscovering them. SQL results close the loop by exposing whether a guess about a table or column worked. The sequence therefore rewards a causal chain: exploration produces evidence; evidence updates stored schema knowledge; stored knowledge reduces exploratory work on a later question.^[8][8] Database Exploration learning opportunity“The schema is not provided upfront---the agent must discover table structure, encoding conventions, and cross-group differences through exploration. The CL challenge is \emph{efficiency}: a learnin…”Open source^[7][7] Learning mechanism criterion“\paragraph{Learning mechanism.} Earlier instances in the schedule must produce observations that are informative for later ones. The environment must provide a feedback loop that a learning system…”Open source

The migration after question $20$ sharpens the criterion rather than changing it. Some schema and value facts become stale, so later feedback must identify what changed while unchanged conventions continue to carry value. Blindly keeping every fact and blindly deleting the complete record both waste the sequence; the opportunity lies in updating the affected facts while preserving the rest.^[9][9] Database migration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source

These are admission criteria, not evidence that a tested system actually learned. Jointly, they establish an opportunity in which accumulated experience can improve performance and static capability cannot explain the intended improvement.^[10][10] Joint task admission criteria“Any task in an effective continual learning benchmark must satisfy the following criteria. Jointly, they define when measured improvement reflects online learning rather than static model capability.”Open source A system might still ignore the feedback, store the wrong fact, or retrieve nothing useful. The unresolved question is causal: on the same database question, did access to accumulated history change the outcome relative to running the same system fresh?

Suppose the database agent answers question $18$ with one exploratory query after learning the obfuscated schema, while a fresh run needs three. Comparing that agent with a stronger model would mix memory with base capability. Comparing question $18$ with question $2$ would mix learning with question difficulty; the paper warns that varying difficulty and baseline performance can make reward curves mislead.^[2][2] Reward“When instance difficulty is homogeneous, a rising reward curve is a reliable signal of learning. In practice, however, instance difficulty varies, and different systems have different baseline perf…”Open source The migration after question $20$ sharpens the problem: later questions also occur in a changed environment, so chronology is not a control.^[9][9] Database migration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source

The counterfactual—a comparison that changes only the suspected cause—changes one input. Run the same end-to-end system on the same current instance $t$ twice. In the stateful run, the system receives the instance plus interaction history, containing earlier instances and rewards. In the stateless run, it receives the current instance alone. The paper formalizes this through a policy, the rule mapping available inputs to a response: stateful policies condition on history, which may live in retrieved memory or accumulated context, while stateless policies are independent of it.^[11][11] Stateful and stateless policies“\textbf{Stateless} ($sl$) systems condition only on the current instance, $\pi_t = \pi(x_t)$, independent of history $\mathcal{H}_{t-1} = \{(x_i, r_i)\}_{i=1}^{t-1}$; by construction, they cannot i…”Open source

Apply the pair to every database question. The stateful answer earns stateful reward; the stateless answer earns stateless reward. Holding the system, current question, tools, and scoring rule fixed removes those factors from the comparison; the designed difference is access to interaction history. This does not require later questions to resemble earlier ones in difficulty. At question $21$, both runs face the migrated database, but only the stateful policy may condition on pre-migration history.[@state-conditioning @db-drift]

The pair therefore gives two rewards under a controlled history intervention. But “different” is not yet interpretable: a stateful reward above the stateless reward should count as help, equality as no effect, and the reverse as harm. The comparison still needs a signed metric that encodes those three cases.

Consider one of the database sequence’s $40$ questions.^[1][1] Database Exploration“The agent answers a sequence of 40 natural-language questions about an unknown SQLite database containing Amazon product reviews across three product groups: Office Products, Electronics, and Music…”Open source Suppose it happens to be easy: both the fresh and history-carrying runs recognize the relevant table quickly. The stateful reward is $0.9$, but that number cannot show whether stored schema facts helped. The stateless reward might also be $0.9$ because the current question exposes enough clues. Counting only the stateful result would turn ease into apparent learning.

The comparison must preserve the question and the system while changing only access to accumulated state. The stateless reward then becomes a local baseline, not a score from a weaker model or a different question.^[12][12] Gain“For each instance $t$, we define gain as: $g_t = r^{sf}_t - r^{sl}_t$ where $r^{sf}_t$ is the reward achieved by the stateful system and $r^{sl}_t$ is the reward achieved by the same system run in…”Open source

The subtraction is matched differencing: compare two outcomes that share the same system-specific instance difficulty, then remove their shared level. If both runs earn $0.9$, then $g_t = 0$. The question was easy, but accumulated state added nothing. If the stateful run earns $0.9$ while the stateless run earns $0.6$, then $g_t = 0.3$. Both runs faced whatever made the question easy or hard; only the stateful run could reuse earlier schema discoveries. Their difference isolates that access to accumulated state.^[12][12] Gain“For each instance $t$, we define gain as: $g_t = r^{sf}_t - r^{sl}_t$ where $r^{sf}_t$ is the reward achieved by the stateful system and $r^{sl}_t$ is the reward achieved by the same system run in…”Open source^[1][1] Database Exploration“The agent answers a sequence of 40 natural-language questions about an unknown SQLite database containing Amazon product reviews across three product groups: Office Products, Electronics, and Music…”Open source

The sign gives the metric its interpretation. Positive gain means accumulated state improved reward on this instance. Zero gain means it had no measured effect. Negative gain means the stateful run performed worse than the matched stateless run. If their rewards are $0.2$ and $0.7$, respectively, then $g_t = -0.5$. A high absolute reward and positive learning are therefore separate claims; so are a low absolute reward and failure to learn.^[12][12] Gain“For each instance $t$, we define gain as: $g_t = r^{sf}_t - r^{sl}_t$ where $r^{sf}_t$ is the reward achieved by the stateful system and $r^{sl}_t$ is the reward achieved by the same system run in…”Open source

Raw gain solves the attribution problem within one reward scale. It does not yet solve comparison across scales. A gain of $0.2$ may consume most available improvement on one task but little on another, especially when their stateless baselines start at different distances from the maximum. The remaining problem is to measure gain against the headroom each system actually had.

Suppose the database agent gains $0.10$ mean reward from remembered schema facts. That sounds substantial if its stateless baseline is $0.80$ and the task maximum is $1.00$, but modest if the baseline is $0.20$. Comparing raw gains declares a tie even though one agent captures half of the remaining opportunity and the other captures only one eighth.

Let mean stateful reward denote mean reward with accumulated state and mean stateless reward denote mean reward without it. Their difference is mean gain. Learning headroom is the distance from the stateless baseline to the task-defined maximum reward. The benchmark divides the first quantity by the second and calls the result normalized gain.^[13][13] Cross-Task Normalization“Normalized gain is then: $\widehat{g} = \frac{\bar{r}^{sf} - \bar{r}^{sl}}{r_{max} - \bar{r}^{sl}}$ The denominator $(r_{max} - \bar{r}^{sl})$ is the system's own learning headroom—the maximum gai…”Open source

The denominator changes the comparison. With $r_{max}=1.00$, a baseline of $0.20$ leaves $0.80$ headroom, so a gain of $0.10$ gives $\widehat{g}=0.125$. A baseline of $0.80$ leaves $0.20$, so the same gain gives $\widehat{g}=0.50$. The second result is larger because state captured a larger fraction of what remained learnable, not because its raw contribution increased.

Normalization now lets heterogeneous reward scales and baselines contribute on a common fraction-of-headroom scale.^[13][13] Cross-Task Normalization“Normalized gain is then: $\widehat{g} = \frac{\bar{r}^{sf} - \bar{r}^{sl}}{r_{max} - \bar{r}^{sl}}$ The denominator $(r_{max} - \bar{r}^{sl})$ is the system's own learning headroom—the maximum gai…”Open source But averaging across all $40$ database questions erases when the gain occurred. A positive $\widehat{g}$ could reflect durable schema knowledge that survives the migration after question $20$, or repeated adaptation within each stable half followed by failure at the boundary.^[9][9] Database migration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source

Question $21$ arrives after the database migration. The agent has cached that Electronics prices come from the old price column, so the obvious move is to reuse that column. The query may still run because the migration retains the old column, but the new prc_v2 column contains updated prices. Blind reuse can therefore return stale values.^[9][9] Database migration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source^[14][14] Electronics price migration“A new \texttt{prc\_v2} price column is added to the Electronics products table with updated product prices, while the old price column is kept around.”Open source

The opposite policy is safer but wasteful: erase the entire cache and rediscover every table, encoding convention, and category difference. Those discoveries previously reduced the number of exploratory queries, so a complete reset throws away useful structure along with stale facts.^[8][8] Database Exploration learning opportunity“The schema is not provided upfront---the agent must discover table structure, encoding conventions, and cross-group differences through exploration. The CL challenge is \emph{efficiency}: a learnin…”Open source Concept drift is a change in the environment that alters which previously learned claims remain valid. Here the boundary is explicit: the first $20$ questions use the original database, while the remaining $20$ use a migrated database with schema and value changes.^[9][9] Database migration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source

Useful state therefore needs two scopes. Persistent knowledge stores exploration procedures and schema conventions that recur across questions; variant-specific knowledge stores object-level facts such as active table names, column formats, and value encodings. The task rewards reusing discovered schema logic, but migration can rename tables, reformat columns, and invalidate cached information.^[15][15] Database Exploration under migration“A learning agent discovers schema conventions and data idiosyncrasies early and reuses this logic to answer later questions with far fewer queries. The database also undergoes a migration halfway t…”Open source When a query fails or observed values conflict with a cached fact, the agent marks facts about that object as provisional, probes the affected schema, and replaces contradicted entries. Conventions that the new evidence has not challenged remain available. This selective revision avoids both blindly reusing stale facts and discarding all learned structure.

That policy can produce several very different reward paths. One agent may fail on question $21$ and update quickly; another may cross the boundary cleanly but later discard useful structure. Because normalized gain compresses stateful and stateless rewards into run-level means, those paths can yield the same aggregate value.^[13][13] Cross-Task Normalization“Normalized gain is then: $\widehat{g} = \frac{\bar{r}^{sf} - \bar{r}^{sl}}{r_{max} - \bar{r}^{sl}}$ The denominator $(r_{max} - \bar{r}^{sl})$ is the system's own learning headroom—the maximum gai…”Open source The unresolved question is whether gain came from durable transfer at the migration boundary or repeated adaptation after it.

Two agents can finish with the same $\widehat{g}$ while failing in opposite ways: one retains useful structure across variants but adapts slowly, while the other adapts within each variant but fails to carry that learning across switches.^[16][16] Stability–plasticity tradeoff“Two systems can demonstrate the same $\widehat{g}$ while differing with respect to these terms: a highly plastic learner can adapt well within each task variant while failing to carry that adaptati…”Open source In the database task, the migration after question $20$ changes both schema and values, so previously cached facts may become stale.^[9][9] Database migration“The default schedule uses the \texttt{schema\_drift} variant: 20 questions on the original database, then 20 questions after a live migration that introduces a modified database for the agent to de…”Open source Ranking the agents by $\widehat{g}$ alone hides the difference between poor retention and slow adaptation.

The obvious fix is to compare performance before and after the migration, but that still mixes transferred knowledge with learning from post-migration feedback. Instead, boundary gain uses the first question of each variant; in the database schedule, question $21$ is the first post-migration case. No feedback from the migrated variant has arrived yet, so positive gain there indicates useful structure carried across the switch, while negative gain can indicate that stored information was misapplied.^[17][17] Learning deficits by component“We decompose gain into two components using variant boundaries as a natural partition: stability measures gain at the first instance of each new variant, where the system must transfer prior knowle…”Open source^[18][18] Additive gain decomposition“Using the same task-level headroom denominator $r_{max} - \bar{r}^{sl}$ as in $\widehat{g}$ (\cref{sec:cross_task_norm}), we define the stability component: $$ \widehat{g}_{\text{stab}} \;=\; f_\m…”Open source Within-variant gain uses every other question, each of which follows at least one round of feedback from its current variant. Gain there measures plasticity—adaptation from new evidence. An agent can therefore retain useful structure but adapt slowly, or adapt quickly while carrying little across a switch.^[16][16] Stability–plasticity tradeoff“Two systems can demonstrate the same $\widehat{g}$ while differing with respect to these terms: a highly plastic learner can adapt well within each task variant while failing to carry that adaptati…”Open source

These are not two extra scores detached from the aggregate. Every question belongs to exactly one side of the partition. In the $40$-question database schedule, the two variant-opening questions form the boundary subset; the other $38$ form the within-variant subset. Each subset’s mean stateful–stateless difference is divided by the same task-level headroom used for $\widehat{g}$, then weighted by that subset’s fraction of the schedule. Because the fractions cover the complete schedule, the weighted stability and plasticity components sum exactly to $\widehat{g}$. When two totals are equal, a larger boundary contribution must be offset by a smaller within-variant contribution, or vice versa.^[18][18] Additive gain decomposition“Using the same task-level headroom denominator $r_{max} - \bar{r}^{sl}$ as in $\widehat{g}$ (\cref{sec:cross_task_norm}), we define the stability component: $$ \widehat{g}_{\text{stab}} \;=\; f_\m…”Open source

This diagnosis changes what a memory score means. Some evaluated systems showed virtually no stable learning even though repeated exposure within variants produced gain.^[17][17] Learning deficits by component“We decompose gain into two components using variant boundaries as a natural partition: stability measures gain at the first instance of each new variant, where the system must transfer prior knowle…”Open source Matched gain asks whether accumulated state helped, headroom normalization measures how much available improvement it captured, and the decomposition separates help transferred across a switch from help rebuilt after local feedback.^[18][18] Additive gain decomposition“Using the same task-level headroom denominator $r_{max} - \bar{r}^{sl}$ as in $\widehat{g}$ (\cref{sec:cross_task_norm}), we define the stability component: $$ \widehat{g}_{\text{stab}} \;=\; f_\m…”Open source The hook closes with a narrower deficit than storage capacity: accumulated state can introduce spurious generalizations and stale beliefs, and even the strongest evaluated system captured only $25.4\%$ of its available learning headroom.^[19][19] Continual-learning gap“Our evaluation reveals a gap in current systems' ability to continually learn. Naive ICL outperforms dedicated memory architectures on most tasks, and even the best system achieves only 25.4\% norm…”Open source