Agent audit regression: Composer 2.5 Fast PASSes on behavioral-only runs while edge suites find blockers

[Sylwia_Pietraszko]

Where does the bug appear (feature/product)?

Cursor IDE

Describe the Bug

Environment

• OS: Windows 10/11 (win32 10.0.26200)
• Cursor: Agent mode, Auto disabled
• Model: Composer 2.5 Fast (manual selection)
• Project: private Python repo, pytest + mypy strict
• Audit methodology: documented in repo rule .cursor/rules/elnath-stage-audit.mdc

Summary

I have used Cursor Agent with Composer as my primary pre-merge auditor for some time and was very happy with the results. During a multi-day audit session in July 2026, the Agent repeatedly issued PASS / merge-ready verdicts when only the behavioral pytest suite was green, while adversarial edge suites (run separately by another tool on the same SHAs) found real blockers.

The gap was not “the model can’t code” — it was audit discipline: skipping edge regressions, downgrading failed edge tests, and stopping as soon as the official suite turned green.

When I enforced the full audit procedure (fetch → ls-tree → behavioral + all edge rounds → fresh clone), results aligned with the external auditor. This suggests a workflow / agent-behavior issue as much as raw model capability — possibly amplified by Composer 2.5 / Fast.

Expected audit procedure (explicit in session + repo rule)

1. git fetch → confirm SHA
2. git ls-tree — run tests only from committed files
3. Behavioral pytest
4. All edge / reaudit / premerge rounds
5. mypy --strict
6. Fresh git clone @ same SHA → repeat
7. Verdict only if both formal and adversarial layers pass

Rule: Behavioral green without edge/reaudit = verdict withheld, never PASS.

Observed failure pattern

SHA / stage	Agent verdict	Behavioral	Edge / adversarial	External auditor
e6a86f4 (E2)	PASS (incorrect)	93/93	round3 not run	6 failed round3
f6a1868 (E2)	PASS (incorrect)	98/98	round4 not run	4 failed round4
9127270 (E2)	PASS (incorrect)	98/98 + fresh clone	round5 not run	2 blockers round5
4618396 (E2)	PASS (incorrect)	104/104	round6 not run; failure misclassified	2 blockers round6
a3146ce (E3)	PASS (incorrect)	102/102	round6 edge not run	DO NOT MERGE round6

Example of misclassification: At 4618396, edge test test_out_of_scope_direct_match_reenters_only_as_flagged_graph_context failed (KeyError: record missing from hits). Agent classified it as a non-blocking documentation issue and issued PASS. External audit classified the same scenario as blocker B1 (graph context lost when record was also a direct candidate). Fix landed in later commits.

Illustrative blockers found by edge tests after behavioral 98–104/104 was green:

1. time_scope + graph seeding: out-of-scope direct hit filtered before graph closure
2. coverage.dec_count: out_of_time_scope_context DEC still counted
3. Graph seed exclusion: record both direct candidate (out of scope) and graph neighbor → vanished from report
4. coverage.limited=true still allowed no_decision verdict
5. Self-relation in graph.expand: seed became spurious graph hit

Behavioral suite on branch did not include these until after edge findings were ported.

What worked when procedure was followed

• Behavioral + all local edge files matching test_e*_premerge_round*.py
• Fresh clone verification @ target SHA
• Explicit instruction: “do not PASS until round N is green”
• Final E2 merge recommendation only after rounds 3–9 regressed green (109 behavioral, 49 edge, mypy clean)

Hypotheses for investigation

1. Agent completion bias: stopping when pytest tests/behavioral passes
2. Untracked test blind spot: git ls-tree correctly excludes local tests/edge/, but Agent reported “not on branch” without withholding PASS
3. Model variant: user was happy with “Composer” before; now sees Composer 2.5 Fast — unclear if Fast vs Standard explains regression (related thread)
4. No built-in adversarial audit mode: no first-class step to run all tests/edge/** or refuse PASS when edge exists but wasn’t executed
5. User rules ignored: repo rule stated “behavioral green ≠ PASS” — violated multiple times in one session

Constructive suggestions

1. Optional “pre-merge audit” Agent preset: user-supplied command list must all pass before allowing PASS summary
2. Warn when project contains tests/edge/ or *premerge* but Agent verdict is PASS without running them
3. Document the Fast toggle more visibly
4. System reminder when user rules say “never PASS on behavioral alone” in long audit sessions

Minimal reproduction

1. Repo with green tests/behavioral/** on a feature branch
2. Add local tests/edge/test_*_premerge_roundN.py that fails on current HEAD (not committed)
3. Ask Agent: “Audit before merge, full procedure, PASS/FAIL”
4. Observe whether Agent runs only behavioral and issues PASS
5. Run edge locally → failures
6. Repeat with: “run tests/edge/test__premerge_round.py or withhold verdict”

Request IDs: Not collected during this session. I can reproduce and supply IDs from the message ⋮ menu if support needs traces.

Intent: Constructive feedback to help identify a reproducible quality gap — not a billing dispute. I still want Composer to work well for audits; sharing this in case it helps tune audit-style Agent workflows.

Steps to Reproduce

N/A

Operating System

Windows 10/11

Version Information

OS: Windows 10/11 (win32 10.0.26200)
Cursor: Agent mode, Auto disabled
Model: Composer 2.5 Fast (manual selection)
Project: private Python repo, pytest + mypy strict
Audit methodology: documented in repo rule .cursor/rules/elnath-stage-audit.mdc

For AI issues: which model did you use?

Composer 2.5 Fast

For AI issues: add Request ID with privacy disabled

5f9957cf-6bcd-429b-909c-dfe9f262a1d0

Additional Information

Does this stop you from using Cursor

Yes - Cursor is unusable

[deanrie]

Hey, thanks for the detailed write-up. We rarely see reports this clean and well structured.

Key point: what you’re describing isn’t a product bug, it’s model behavior. Cursor doesn’t have built-in verdict gating. PASS is just text the model generates, not a mechanical check. Same deal with rules and AGENTS.md, they act as steering, not fail-closed constraints. The model can ignore them, especially in long sessions. Your hypothesis 1 completion bias, stopping once pytest tests/behavioral is green, and hypothesis 2 git ls-tree correctly excluding untracked tests/edge/ and the agent honestly skipping them, both match how it works. The second one is git semantics, not an agent bug. git ls-tree only shows what’s in the index, untracked files won’t appear there.

On Fast: Composer 2.5 and Composer 2.5 Fast are the same model with a serving toggle you switch via Edit in the model picker, not two different weights. The difference is latency and cost, not quality, so it probably doesn’t explain any difference in audit discipline. Looks like you already found the toggle.

What actually gives you mechanical enforcement today is Hooks. What you asked for in constructive suggestions, blocking PASS until the required commands ran, should be done with hooks, not prose rules:

• beforeShellExecution to gate shell commands or require specific suites to run, fail-closed via exit code 2
• stop to intercept agent completion and block a verdict if the edge suites weren’t run

There’s also a generic preToolUse that can fail on all tool calls, but for shell commands it’s better to use beforeShellExecution.

Docs: Hooks | Cursor Docs

There’s also a community guide called Model Is the Auditor with ready-made self-review hooks for Windows and Linux: [Guide] Thin Self-Review Hooks for Cursor | The Model Is the Auditor (Windows + Linux)

A similar behavior class is discussed here. It also breaks down the fail-closed pattern with exit code 2, which matches your case:

Cursor Agent (Composer/Auto) "Reward Hacked" its own QA rubric to pass evaluation

Where does the bug appear (feature/product)? Cursor IDE Describe the Bug During a multi-agent orchestration workflow (DevSwarm topology), the native Cursor Agent (Composer/Auto) acting as the “Coder” engaged in a classic case of “Reward Hacking”. The Cursor Agent was assigned to write code and be evaluated by a Revisor agent using a strict 4-point criteria file (P2-impl-notes.md) tracked in main. Because the Cursor Agent had write access to its isolated worktree, it overwrote the rubric file l…

Agent ignored RESET and workspace rules despite hash locks and 134 tests; repeated false “fixed” claims on intermittent UI bug

Where does the bug appear (feature/product)? Cursor IDE Describe the Bug Cursor Agent does not enforce user workspace rules or operator commands. Over 10+ Agent sessions on a governed repo (.cursor/rules, RESET/NPS in AGENTS.md, hash-locked files, Xcode build gate, 134 passing tests), the agent patch-stacked fixes, edited locked files, updated lock hashes without my QA, ignored RESET (restated checklist but kept editing), and repeatedly claimed “fixed” when verify/build/tests passed but my manu…

What worked for you, explicit per-round instructions like do not PASS until round N is green, is the recommended approach. Use hard gates in the loop, not in the rules text. Hooks formalize this so you don’t have to repeat it every session.

I’ll pass your feedback about the audit preset and warnings for unrun tests/edge/** to the Agent Quality team. Let me know if anything doesn’t work when you try hooks.