Auto-summarization triggers too late — model quality degrades long before the context limit ("context rot")

Ideas Feature Requests
,
1

Feature request for product/service

Chat

Describe the request

Background

This is spun out of Calling /summarize while work is being done stops it, instead of queueing it, where @deanrie asked me to track this separately since it’s a different issue from the queue bug.

The problem

Right now auto-summarization only kicks in when a chat hits the context limit. But research consistently shows that model quality degrades long before the window is full:

  • Chroma’s technical report Context Rot (Hong, Troynikov & Huber, July 2025) tested 18 frontier models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) and found every single one degrades as input length grows — non-uniformly, and well before the advertised limit.
  • Lost in the Middle (Liu et al., TACL 2024) shows a U-shaped attention curve: information in the middle of a long context is systematically underweighted.
  • Same Task, More Tokens (Levy et al., 2024) found reasoning accuracy falling from 0.92 to 0.68 just by growing the input, with task difficulty held constant.
  • Context Length Alone Hurts LLM Performance Despite Perfect Retrieval (EMNLP 2025 Findings) shows performance drops of 13.9–85% as input grows, even when the model retrieves all relevant information perfectly.

So with the current design, the agent spends the last stretch of every long task in its worst-performing regime, and only then summarizes. Summarizing at 100% works as a memory tool, but not as a quality tool.

Proposal

  1. Earlier trigger. Auto-summarize at a configurable threshold (e.g. default ~70% of the model’s effective window) instead of at the hard limit.
  2. Summarize at turn boundaries. When the threshold is crossed mid-task, schedule the summary for the end of the current turn instead of interrupting. No lost work, and summaries are highest quality at a natural checkpoint anyway.
  3. Manual checkpoints that don’t kill the run. Let /summarize queue like any other message, so we can proactively checkpoint before quality decays. (The interrupt itself is tracked as a bug in the thread linked above — the point here is that manual early summarization is currently the only workaround for the late auto-summary, and it’s broken.)

Expected behavior

Long-running agent sessions keep consistent output quality because context is compacted before the model enters the degraded zone — not after. This applies to both the desktop Agents window and the CLI.

Operating System (if it applies)

MacOS

5

Thanks for moving this into a separate thread and backing it up with solid research. That’s a really strong write-up.

One factual note about current behavior: auto-summarization doesn’t wait for the context to hit 100%. Background compaction starts before the hard limit, around 90% of the window being used. So there’s already some protection from working right at the edge, it’s just that the threshold is fixed right now and set server-side, not user-configurable.

What we don’t have today, and what you’re basically asking for, is:

  • a configurable and generally earlier trigger threshold
  • a mode where the summary is applied only at a turn boundary, not in the middle of a task

On manual checkpoints: /summarize is the way to trigger compaction manually at any time before quality starts to drop. The fact that it interrupts work instead of being queued is tracked separately in your thread 163978 Calling `/summarize` while work is being done stops it, instead of queueing it so I’ll keep that part there to avoid mixing two requests.

I’ve logged the feature request: earlier and configurable threshold plus applying at turn boundaries, for the Agents window and the CLI. If there’s an update, I’ll post it here.