← Reza Bayat

Paper thread · Long-context sequence modeling ·

Proteus

Incremental Memory Activation for Long-Context Sequence Modeling

Read the paper →

Four hand-inked spheres on a horizontal rule, filling progressively from nearly empty to nearly full.

TL;DR. Most memory models expose their full state from the very first token, so early context spreads across it unchecked and later context can only overwrite what is already there. Proteus schedules that capacity instead: one block is active at first, and more unlock as the context grows. It is a drop-in gate that adds no parameters and no compute, and it improves four state-of-the-art architectures — with the largest gains at the longest contexts.

1/12
Linear-time attentions are reaching frontier scale, their fixed-state memory, however, still saturates as context grows, because all of its capacity is exposed from token 1. Early tokens face no pressure to compress, so they take far more of that capacity than they need. Later tokens inherit a state that is already full.

We introduce incremental memory activation: a fixed-size memory should not expose all of its capacity at once. It should schedule how much is live as the context grows, imposing a bottleneck early that forces the model to summarize rather than memorize, and unlocking fresh capacity later so new information has somewhere to land that isn't already written.

Proteus is the simplest instantiation of that idea. We partition the memory into E blocks and unlock them one at a time as the context advances, gating both reads and writes so locked blocks are neither retrieved from nor updated. The state itself never grows; only the active fraction of it changes. It comes at no extra cost and improves average performance on language modeling and commonsense reasoning (+0.37–1.05 avg. accuracy points). More interestingly, it yields significant gains in long-context handling: up to +8.4 NIAH points at 2× the training context length.

Read more background/method/results below 🧵
First page of the Proteus paper, showing the title, authors, abstract and Figure 1.
2/12
Where that fixed-size state comes from.

Transformers store every token: perfect memory, but quadratic cost.

Recurrent memory models fix this: they compress context into a fixed-size state that updates as tokens flow in.

Think of it as a notebook with limited pages. You have to decide what's worth writing down.
Transformers store every token so memory grows with the sequence, while recurrent models compress context into a fixed-size state. Transformers Store every token t₁ t₁ t₂ t₁ t₂ t₃ t₁ t₂ t₃ t₄ grows t₁ t₂ t₃ t₄ Memory: O(n) · Cost: O(n²) Recurrent models Compress into fixed state M₁ M₂ M₃ M₄ fixed t₁ t₂ t₃ t₄ Memory: O(1) · Cost: O(n) Transformers grow memory with every token. Recurrent models compress it into a fixed-size state.
3/12
The problem: almost every memory model hands you the full notebook from page 1.

Early tokens spread freely across all pages with no compression pressure. By the time later tokens arrive, the notebook is already full of early drafts. Every new line has to be squeezed in, and each one smudges what's already on the page.

This is static memory, and it's the default across the whole family.
A memory bar in which early tokens occupy most of the capacity and later tokens are crowded into what little remains. Static memory: full capacity from token 1 t₁ t₂ t₃ t₄ t₅ t₆ t₇ t₈ … Spread freely, no compression pressure Squeezed into what's left interference Result: memory biased toward early tokens
4/12
Proteus flips this: start with a few pages, unlock more as you go.

We partition memory into E blocks and lock all but the first. Every N/E tokens, one more block unlocks, on a fixed deterministic schedule (E = 16 by default).

Both reads and writes are gated: locked blocks are neither retrieved from nor updated, and keep exactly the values they held.

Note what we're not doing: the memory never grows. Same fixed-size state, same compute. We only schedule which part of it is live.
Four snapshots of the memory state over time, with locked blocks progressively unlocking as each chunk of context arrives. Locked Activated Past Memory state through time Chunk 1 Chunk 2 Chunk 3 Chunk 4 Long context Fresh capacity at every stage: less pollution, less interference
5/12
Why does this work? Two forces, aimed at the two failure modes:

🔹 Early bottleneck → few active blocks means early tokens must summarize, not memorize
🔹 Fresh capacity → later tokens get fresh room to land in, so less of what's stored gets overwritten

Activation is monotone: nothing is ever discarded, and the whole memory is live by the end of the schedule.
Two panels: an early bottleneck forcing compression into one active block, and a freshly unlocked block giving later tokens room to land while past blocks remain active. Early bottleneck Locked Locked Locked Active (B₁) 1/E t₁ t₂ t₃ t₄ Small capacity, must compress Better compression Fresh capacity Locked Fresh (B₃) Past (B₂) Past (B₁) clean used t₅ t₆ t₇ t₈ Less overwriting, less pollution Better retention Together: better compression early, better retention late
6/12
The mechanism is deliberately orthogonal to everything else in the design space. It changes neither the internal objective, nor the optimizer, nor the memory architecture. Only which parameters are exposed at each position.

That's why one gate drops into four very different models: Hope-Attention, SWLA, Comba, and Titans.
Four memory architectures above a single Proteus banner, showing one gating scheme applied unchanged to all of them. what differs: objective · update rule · memory type Titans deep neural memory momentum + forget gate Hope-Attention attention + MLP chain multi-frequency blocks Comba bilinear RNN state closed-loop control SWLA sliding window + linear attention Proteus: one block-wise prefix mask, applied unchanged gates reads and writes · no new parameters · no overhead Four memory designs, one scheduling axis
7/12
And it improves all four, at both scales, with gains from +0.37 to +1.05 points in average accuracy across 8 downstream evaluations. Perplexity drops in nearly every setting too.

Setup: 760M and 1.3B params, trained on 50B and 100B FineWeb tokens with an 8K context window.
Accuracy for four backbones at 760M and 1.3B parameters, each improving with Proteus, by between 0.37 and 1.05 points. baseline + Proteus Avg. accuracy across 8 benchmarks → 760M params · 50B tokens 50 51 52 53 54 Titans +0.71 Hope-Attention +0.84 Comba +0.72 SWLA +0.78 1.3B params · 100B tokens 53 54 55 56 57 58 Titans +1.05 Hope-Attention +0.37 Comba +0.58 SWLA +0.53 accuracy (%) · both panels share one scale (1 pt = same width) · axes offset, not zero-based 8 of 8 improved, every architecture and both scales
8/12
Where it really shows up: long context.

On the hardest single-needle variant (S-NIAH-3, UUID retrieval), Proteus is roughly neutral at 4K, then opens a large gap at 16K. Titans goes 21.4 → 29.8, Hope-Attention 50.0 → 54.4. Comba picks up +7.8 on numeric needles at the same length (13.4 → 21.2).

The multi-needle settings tell the same story. At 16K, Proteus improves all three backbones on multi-key, multi-query and multi-value retrieval alike.
Accuracy gain on S-NIAH-3 at 4K, 8K and 16K context for Titans and Hope-Attention, growing with context length. Proteus accuracy gain (Δ) on S-NIAH-3 UUID retrieval: accuracy gain vs. context length Titans 0 4K 8K 16K 0.0 +1.2 +8.4 Hope-Attention 0 4K 8K 16K −0.2 +0.4 +4.4 The longer the context, the bigger the gain
9/12
Worth being precise about why 16K helps, since the obvious guess is wrong.

We train at 8K, and the schedule finishes inside that window. At 16K, no extra blocks unlock. The memory has been fully active for a while.

The gain isn't extra capacity. It's that a state compressed under an early bottleneck degrades more gracefully past the training length.
10/12
A natural worry: are we buying late-context gains by sacrificing early-context quality?

No. Measuring validation perplexity by token position, Proteus is lower at every one of them. The gap peaks at 6K, then stays open at roughly 0.3 ppl all the way out to 32K — far past the training window. By 8K, the memory is fully active.
Validation perplexity by token position from 2K to 32K, with Proteus below the baseline at every position. Titans Titans + Proteus Validation perplexity by token position lower is better ↓ 17.5 17.0 16.5 16.0 15.5 15.0 8K: training window, memory fully active 2K 3K 4K 6K 8K 10K 12K 14K 16K 20K 24K 28K 32K Context length Lower at every position, no early-context tax
11/12
The schedule isn't a free lunch, and the ablation shows the tension directly.

E = 1 is the base model. Perplexity improves through E = 16, then gets worse at E = 32. Squeeze the first tokens into 1/32 of the memory and the bottleneck costs more than the compression buys.
Validation perplexity against the number of memory blocks E, lowest at E equals 16 and worse at E equals 32. Validation perplexity vs. number of blocks E lower is better ↓ 17.5 17.0 16.5 16.0 15.5 1 4 8 16 32 E (number of memory blocks) base model best bottleneck too tight
12/12
More results in the paper: recall-intensive retrieval, LongBench, and further ablations.

Note: through the Nested Learning view, MLP blocks are themselves associative memory — so the same principle applies to a model's parameters, activated progressively over training rather than over context. We show a proof of concept on Hope-Attention's MLP blocks.

Recap. Static capacity is the default. We think it shouldn't be. Scheduling it is simple, costs nothing, and helps across architectures.

📄 Paper: arxiv.org/pdf/2608.16844
— end of thread —

Citation

If you found this thread useful and would like to cite the paper:

Plain text

Bayat, Reza, Ali Behrouz, Vahab Mirrokni, and Aaron Courville. “Proteus: Incremental Memory Activation for Long-Context Sequence Modeling.” arXiv preprint arXiv:2608.16844 (2026).

BibTeX

@article{bayat2026proteus,
  title   = {{Proteus: Incremental Memory Activation for Long-Context Sequence Modeling}},
  author  = {Bayat, Reza and Behrouz, Ali and Mirrokni, Vahab and Courville, Aaron},
  journal = {arXiv preprint arXiv:2608.16844},
  year    = {2026}
}