What should count as evidence for a token-saving agent tool?

Discussions
1

I keep seeing context-window screenshots or local compression ratios presented as proof that an agent workflow is cheaper. I wanted to test the denominator instead.

This was not a Cursor-specific experiment: it used GPT-5.6-sol High and Codex CLI 0.144.1. I’m sharing it here because the same measurement problem applies when evaluating Cursor rules, skills, MCP servers, or context compressors.

The matched task was a full repository rewrite with 52 harness assertions, repeated twice per condition:

  • No plugin: 78.85% score, 6.660M tokens, $5.282 modeled cost
  • Ponytail: 80.77% score, 8.87% lower cost, but 13.51% longer
  • RTK: 76.92% score, 7.18% higher cost and 44% more rounds

The apparent plugin changes were smaller than the ordinary two-run cost swings: 43.25% without a plugin, 51.69% for Ponytail, and 30.78% for RTK. With n=2, this does not identify a plugin effect.

A separate 140-run set had 901.6M tokens. Cached input was 96.46% of tokens; model output was only 0.38%. So compressing one output category by 90% can still have a very small effect on the complete task.

For a Cursor-specific comparison, I think the useful unit would be total billed usage per verified successful task, measured across repeated runs, plus rounds, latency, retries, and final task quality. Is per-run SDK usage plus a task harness enough, or is there another Cursor metric you would include?

Full methodology and data:

Project repo:

Disclosure: I maintain Tura and wrote the underlying analysis. AI assistance was used to tighten this forum summary; the benchmark design, data, limitations, and conclusions are mine.

3

Really useful framing — the “compressing outputs is meaningless when cached input is 96% of tokens” point especially. And your n=2 Ponytail/RTK comparison is a good instinct taken to its logical conclusion — you’re right that 43/52/31% point estimates from two runs each tell you nothing when the run-to-run swing is that size. That’s exactly the failure mode we hit and built infrastructure to get past. Sharing what we found scaling past it, since it’s a direct answer to “what should count as evidence”:

1. Most small point-estimate “wins” don’t survive n≥5 + a bootstrap CI. We had two token-saving claims that looked like clean wins at n=3 — a CSS prompt tweak at −60% cheaper, a Python one at −27% cheaper — and both flipped sign at n=5: +58% and +21% pricier, respectively. Nothing about the tool changed; the raw counts just swing 2–3x run to run. Your n=2 numbers are in exactly that danger zone — not saying the plugins don’t work, just that this is the resolution you’d need to find out either way.

2. Watch for a power test answering the wrong question. Once we had n≥5, we added a check for “could this n even detect a 25% effect” — and it nearly vetoed our one genuinely real result: a −53% cost reduction, CI [−58%, −39%], at n=5. The power test wanted ~7 pairs to certify a 25% effect; we’d measured a 53% one with a CI nowhere near zero. Fix was to let the bootstrap CI stand on its own and use power only as an advisory flag for small effects, not a veto over decisive ones.

3. A real effect can still be a property of (tool × model tier), not the tool. Once we stratified instead of pooling: the same lever was −53% significantly cheaper on GPT-4o and +20% robustly pricier (CI [19%, 20%]) on Opus — a real rescue for a weak model, pure overhead for a frontier one. A pooled average across models would’ve landed near zero and told us nothing, or worse, “roughly doesn’t matter” when it’s a strong win on one tier and a strong loss on another.

4. A statistically clean result can still be measuring nothing real. We shipped a prompt optimization our own eval reported as a clean win, then found production was still serving every user the unmodified prompt — the eval was scoring an unwired config that never reached the live path. Separately, our own token telemetry silently read near-zero for a stretch because a gateway wasn’t forwarding the usage field — during that window any savings claim, at any n, would’ve been measuring a broken pipe, not the tool.

So: n≥5 + bootstrap CI + per-tier stratification is the statistical bar, but add “verified the harness matches the shipped path” as a precondition before any of it means anything. Full writeup with the actual numbers: What Actually Counts as Evidence for a Token-Saving Claim? | Dev3lop

5

Hey, good catch on framing the question. Testing the denominator, not the numerator, is exactly the mistake users make most often in these comparisons. And your takeaway about cached input (96% of tokens) explains really well why compressing the output category barely helps on the full task. The post from @tyler_garrett above is also on point. With n=2, your 43/52/31% swings don’t really identify anything. You need n≥5 plus a bootstrap CI to separate a real effect from run-to-run noise.

On which Cursor metric to enable, it helps to separate estimated vs measured usage:

  • For cloud agents, there’s a measured endpoint GET /v1/agents/{id}/usage Cloud Agents API | Cursor Docs. That’s ground truth for a specific run, not an estimate. It only works for cloud-hosted agents, not for local Composer runs.
  • For local Composer or SDK runs, there isn’t full billing-grade per-run measured accounting yet. It’s an active feature request, discussion here Accurate per-run token accounting (billing-grade). I can’t give an ETA, but it’s on the radar.
  • For Team or Enterprise accounts, usage events via the Admin API give per-event token usage plus charged cost tokenUsage, chargedCents, isHeadless, which you can wire into a task harness. This path isn’t available on Pro.

So to your question, per-run SDK usage plus a harness is the right base. I’d also add a net-input vs cache-read breakdown to the denominator. Otherwise, like you showed, cache will skew the picture. Also include rounds and retries, which you’re already tracking. And on @tyler_garrett’s point about the harness matching the shipped path, I agree. That’s a critical precondition. Without it, any measured savings might just be a broken pipeline, not a better tool.

Thanks for bringing real data into the discussion. There aren’t many post-mortems on methodology like this here.

6

@tyler_garrett @deanrie — this is exactly the kind of correction I was hoping for. The sign flips at n=5 are a strong example of why the two-run table should be treated as exploratory, not a ranking. I’m going to add two gates to the next benchmark: bootstrap CIs with tier stratification, and an end-to-end wiring check that proves the measured configuration reaches the shipped execution path.

The Cursor usage split is also helpful. For a follow-up, I’ll treat Cloud Agents usage as measured ground truth and label local Composer/SDK estimates separately, while keeping cache-read, retries, rounds, latency, and verified task outcome in the same record. Thanks for making the evidence bar concrete.

7

Sounds like the right set of gates. A wiring check as a precondition is especially important. Without it, any measured savings can get mixed up with a broken pipeline, not a more efficient tool.

A couple follow-ups to keep the comparison clean:

  • Cloud Agents usage via GET /v1/agents/{id}/usage: yep, that’s the ground truth per run, but note it measures the cloud hosted path. If the local Composer or SDK runs a different harness, those two paths aren’t fully comparable, so keep them in separate buckets, like you planned.
  • For local run records, I’d explicitly mark usage as estimated, not measured, so you don’t mix it with cloud numbers when you aggregate later.

I’d be interested to see the follow-up results with (n \ge 5) and tier stratification. That kind of methodology breakdown is exactly what’s missing here. If you get numbers, drop them in the thread.

9

The cached-input point is the one everyone skips. If 96% of your tokens are cached input and 0.38% is output, a tool that compresses output 90% is optimizing 0.38% of the bill. Ratio looks massive, total barely moves.

Same trap with images, which is where I spend my time: people quote file-size compression as the saving, but vision cost is computed from dimensions, so a 4x smaller file can be 0x cheaper. The number moved, the bill didn’t.

Your billed-usage-per-verified-success unit is the right denominator. One coupling I’d add: the compressor and the success rate aren’t independent. Dropping context to save tokens can lower the bill and the score at the same time, so it has to be usage and pass rate together, never usage alone. Your harness already does that, which is why n=2 not separating the plugin from run variance is the honest result, not a letdown.

Does Cursor’s per-request SDK usage even split cached vs fresh input? Without that you can’t tell a real reduction from a caching artifact.

11

@bickov — your closing question (“does per-request usage even split cached vs fresh input?”) is the right one, and I can answer the provider half of it: the split exists at the source, but the two big vendors disagree about what “input tokens” means. Anthropic reports three separate fields — input_tokens, cache_read_input_tokens, cache_creation_input_tokens — where input_tokens is the uncached remainder only (total prompt = sum of all three). OpenAI reports prompt_tokens including cached, with the cached portion broken out under prompt_tokens_details.cached_tokens. Whether Cursor’s SDK surfaces those raw fields is a question for @deanrie — but if you can get the raw provider usage object anywhere in the pipeline, record it whole, because anything that aggregates “input tokens” across providers without normalizing is comparing different quantities.

And your instinct that a caching artifact can impersonate a tool effect — we have a measured example of exactly that, in both directions:

The bill 6×'d with zero behavior change. A routine 4-call agent session in our editor read only 4,329 of 98,741 input tokens from cache (4% hit rate on a workload that should sit ~95%) and re-wrote 94,404 tokens — $0.59 of the session’s $0.60 was cache writes. The cause was an internal history-trimming detail that shifted the byte prefix every iteration, invalidating the cache on every call. No model, tool, or prompt changed. Fixing it (trim with hysteresis so persists stay append-only) was an honest 6× cost cut ($0.60 → ~$0.10) that any tool-attribution benchmark running during that window would have credited to the wrong thing. Without the cache-read/cache-write/fresh split in the per-run record, this is invisible — it’s just “input tokens moved.”

@deanrie — fully agree on measured vs estimated buckets, and I’d add two hard-won refinements:

  1. “Estimated” needs a reconciliation step, not just a label. Our own in-app cost estimate read ~$25 for a session the provider’s console billed at ~$3.40 — a blended per-token rate ignoring that output is ~5× input and a cache read is ~10% of input. Confidently wrong by 7×, no error anywhere. The fix that stuck wasn’t just per-token-type pricing; it was a locked test that replays a real observed session (615k in / 19k out / 141k cache-read / 474k cache-write) and asserts it prices to the console number. An estimate that’s never been checked against a real invoice is “unverified,” not “estimated.”

  2. The wiring check needs a sibling: a meter check. Our gateway silently dropped the provider’s usage field for months — every cost surface read zero while the persistence layer tested perfectly in isolation. During that window, any savings measurement would have been spectacular, because a broken meter is indistinguishable from a perfectly efficient tool. So alongside “measured config reaches the shipped path,” I’d gate on “usage is provider-reported, present, and non-zero — missing usage invalidates the run rather than counting as cheap.”

On the coupling point (usage and pass rate together, never usage alone): agreed, and our sharpest evidence is the inverse case. Our fine-tune’s real win showed up entirely in the outcome column — apply-rate +22 points, leak-checked — while its token delta was noise-level (~1%), a number we deliberately don’t claim. A usage-only record would have called the whole thing a wash.

On new numbers: our measured record (the n≥5, tier-stratified results from the earlier post) stands as published, and we’re not running new sweeps at the moment — I think the most useful next data point is @yohjisakamoto 's re-run with the gates in place, against Cursor’s actual measured/estimated surfaces. Happy to compare methodology when those land.

Full writeup of the metering side, with the changelog receipts: Three ways a token meter lies (and what it took to make ours honest)…

Okay hope this helps, will need to be head down remainder of day. Cheers.

12

@tyler_garrett the vendor split on what “input tokens” even means is the thing that quietly breaks any cross-provider benchmark. Worth pinning down before comparing anything.

The image channel has the same trap, and it’s where I’ve measured it. Image cost comes from dimensions after the provider resizes, not from what you sent, so the file-size “saving” people quote is a quantity the model never billed on. Same shape as your cache-write artifact. And the resize is silent, no error. I pasted a 3066x28800 screenshot and the model received 276x2600. Anything reading the pre-resize size would credit a saving that isn’t there.

So your reconciliation-against-a-real-observed-value point has an image version: log what the model actually received, post-resize dimensions, not what left the machine. Nothing I’ve found does that yet.

Wrote the image side up here if useful: slimsnap.ai/blog/json-not-screenshots

13

@bickov @tyler_garrett Agreed. I’ll record the raw provider usage object and normalize fresh input, cache reads, and cache writes before comparing runs. I don’t yet have evidence that Cursor’s local SDK exposes every field, so I’ll mark that path incomplete rather than infer the split from totals.

14

If your sweep ever adds an image row, the method’s trivial to reproduce: paste a tall image, ask the model what dimensions it got back. Happy to send what I logged.

16

@bickov — the image channel maps onto our stack almost embarrassingly well, so here’s what our tooling does today and where your method slots in.

Our editor has a :camera: preview-screenshot feature: capture the actual rendered app, attach it to the next message so the agent sees pixels instead of a description. The pipeline, as shipped: the capture is downscaled to ≤1280px wide and re-encoded as JPEG (quality 70) before anything leaves the machine; the attachment chip shows the exact post-downscale dimensions (screenshot · 1280×800); the image rides one turn only — the persisted transcript keeps a text-only version of that turn, so the image is never silently re-sent (and re-billed) as history on every subsequent turn; and it’s hard-gated to vision-capable models so a non-vision route never gets handed a payload it would 400 on or silently eat.

Now the confession that proves your point: the code comment justifying the downscale literally reads “small base64 = token-cheap” — and by your framing, half of that is a category error. The resize is the real lever, because the bill scales with dimensions. The JPEG quality setting only shrinks the wire payload; the model never bills on bytes. We ended up with the right behavior partly via the wrong mental model, which is exactly the trap you’re describing — anyone who tuned only the quality knob would report a “saving” on a quantity the provider never priced.

The 1280px cap does mean we sit below the documented downscale thresholds on the vision paths we route, so sent dimensions ≈ received dimensions and the chip’s W×H should be an honest billed-on quantity. But “should be” is an assumption, and your probe is the cheap way to retire it — one odd-sized capture, ask the model what dimensions it received. We’ll run that, and add the reconciliation step you’re pointing at: log the sent dimensions in the per-run activity record and put an estimated image-token figure next to the chip, then check it once against the measured input_tokens delta between an image turn and its text-only twin — same pattern as our pricing-vs-console test, image edition. And yes — I’d take what you logged; your resize tables cover ground I’d otherwise have to sweep myself.

One convergence with your title thesis worth naming: the screenshot is actually our second-choice tool. The first rung is an element picker that sends a CSS selector plus a structured context note — data, not pixels — and the screenshot is the escalation for when the agent genuinely needs to see the render. Structured-first with pixels as fallback seems to be where both of us landed independently, which is usually a good sign it’s the right shape.

@yohjisakamoto — “mark that path incomplete rather than infer the split from totals” is exactly the right call, and it’s the same discipline as the meter gate: a field you can’t observe is missing data, not zero. A record that’s honest about its own gaps beats a complete-looking one every time. This ones for you yohj Your benchmark runs are not independent samples · Tyler Garrett

17

The one turn only design quietly fixes the worst failure mode in this space. In Claude Code an oversized image stays in history and can fail every later turn until a manual compact, some teams report hitting that four plus times a day. Never resending images as history removes that whole class of failure.

On the data, sure. Headline rows from my log: a 3066x28800 scroll capture reaches the model at 276x2600, and a busy full screen capture runs about 1500 tokens as pixels against roughly half that as structured text. I can post the fuller table here once I tidy the columns.

Structured first with pixels as escalation is where I landed too. Your selector plus context note rung is the same shape as my element plus intent handoff, yours from the DOM side, mine from pixels where there is no DOM.