I am a long-time Cursor user on a large, old production codebase. For several weeks I have been trying to get Agent to do one job: write and maintain test scripts that actually run the tests I named, the way I named them, and then stop.
I have used Auto and Grok. The same failure shows up on both. Switching models does not fix it. Auto still lands me in Agent sessions that will not stay on the ask. Grok still talks like it understood, then ships something else.
This is not “the model is dumb once.” It is a loop I cannot get out of.
1. It will not do the thing I asked.
I give a small, concrete request (one script, one report, one path). The agent answers as if that is done. What I get is a rewrite, a second design, a “better” runner, extra modes, or a script that testers cannot follow. I have to say the same thing again. Weeks of calendar time have gone into getting scripts to match a request that was already clear.
When I ask for a plain checklist a person can run, I get jargon and internal names. When I ask for a runner that uses our catalog and our data files, I get scripts aimed at a different site’s names, or a sequence number that is not unique, so the job “never ran” and we debug the wrong layer.
2. It invents the next step.
If the request is incomplete, or a plan fails, I need it to stop and ask one question. Instead it fills the gap. It adds a second code path, turns a skip into a fail, “completes” a design I did not lock, or “while I’m here” touches a file I told it not to touch.
I had to put project rules in writing: do not invent, do not change a proven path, do not add/remove a second path, one failed prove means stop. Those rules exist because the default Agent behavior is to keep going. Even with the rules, it still does it.
3. It undoes work that already proved.
This is the expensive one. We get a path that works (content match, timings, the test we care about). A later session “improves” it — hydrate always on, dual path, calc-once slogan, extra filter — and the win is gone. Then we spend days putting the proven path back.
That is not a style complaint. That is lost production time.
4. Test scripts and prove runs are treated as a playground.
I asked for scripts that:
- run the named jobs only
- use the live catalog / live data, not a leftover list from another environment
- classify skip vs empty vs fail correctly
- reuse stored baselines instead of re-running the slow legacy side every iteration
- print results in a format a human can scan
What I keep getting instead:
- runners that skip or fail for the wrong reason
- names that do not exist in the file the program actually reads
- bulk sweeps before the one prove report is green
- re-running the expensive side “just in case”
- output I have to translate, or scripts only the agent can operate
- “copies on the share” / “I’ll also put them here” instead of the one place and the one command I asked for
So I spend more time herding the test harness than testing the product.
5. Auto vs Grok does not change the failure mode.
I tried Auto because I hoped routing would pick a model that follows instructions. I tried Grok because I hoped a different model would stay on a narrow ask. In this project, both:
- confirm the constraint, then violate it in the same turn
- produce a long status update instead of the artifact
- treat “stop and ask” as either freeze (ask forever, never build) or ignore (build the extra thing)
- confuse “I explained the system” with “I shipped the script”
If Auto is routing, I cannot tell which model wrote the bad turn, and I cannot pin a bug report to one name. If I pin Grok, Auto still does the same class of error. The product surface is Agent + Auto + Grok, and instruction-following is the hole.
6. What I need from the product (not a new chat personality).
- Honor a strict scope: only the files and steps named. No second path. No “complete the design.”
- If the ask is ambiguous: one question, then wait. Do not build.
- If a test/prove fails: stop. Do not pile on three more reports or a rewrite of the harness.
- Treat user-written rules like “do not invent / do not touch X” as hard stops, not hints.
- For Auto: show which model ran the turn, so this is debuggable.
- Prefer a small, runnable script in the folder I named over a new architecture.
I am not asking for the agent to be more creative. I am asking for it to do the request and stop. Right now, on Auto and on Grok, that is the part that does not work, and it has cost me weeks.
If anyone from the team wants a repro: long Agent thread, strict “only this script / only this path,” then watch the next turn add a second path or a different catalog. I can share a sanitized example of the ask vs what landed.