Which model leaves you with the least cleanup?

Discussions
1

I’ve had agents fix the reported bug while adding a second way to do something the repo already handles. Then cleaning up the fix becomes another task.

That’s something I’d like to see in model comparisons: the diff you kept, the bits you deleted, and how much steering it took.

If you’re using several models in Cursor, which has been best at working with the code that’s already there? Particularly interested in bug fixes and refactors in established projects.

5

Honestly I don’t think there’s a stable winner. It shifts with every release, and the setup usually matters more than which model you pick.

A few things that cut the “second way to do the same thing” problem with basically any model:

  • Tell it to look before it writes: “Before changing anything, find where this repo already handles X and list the files. Reuse that. Don’t add a new helper unless you can say why the existing one won’t work.”
  • Make that a one-line project rule so you don’t have to remember it every time.
  • Diff budget: “smallest change that fixes the failing case; no refactors; other ideas go in a separate list.”
  • Hand it the failing test or repro first. Then “done” means that test passes, not “looks fixed.”

To compare models fairly: same 3–4 real tickets, identical rules, log lines kept / lines deleted / follow-up prompts. A week of normal work is enough to get a useful read.

7

Hey, thanks for bringing this up. Comparing models by how much of the diff actually stays in the end is a good idea. We don’t really have one consistent leader. A lot changes from release to release, and setup usually matters more than the model choice itself. @uknowhow already gave a good foundation above. I’ll add a couple of things:

  • Plan mode for bug fixes and refactors. The agent first shows a plan, including which files it will touch and what it plans to reuse. If the plan introduces a new helper next to one that already exists, you can catch that before any edits start. It’s cheaper to fix the plan than to clean up the diff later. More here: Plan Mode | Cursor Docs
  • A “reuse first” rule in Project Rules. That way you don’t need to repeat it in every prompt: Rules | Cursor Docs. Just keep in mind that rules are a strong hint, not a guarantee, so reviewing the diff still matters.
  • If you want to compare models in a repeatable way, you can use the SDK to run the same set of tasks with the same prompts and rules. The only thing that changes is the model, and you evaluate the result with your own tests: Run Cursor in your evals | Cursor Documentation. This fits well with the metrics you mentioned, like what stayed, what got removed, and how many follow-up prompts were needed.

It’d be great to see the numbers if someone puts together that kind of comparison on real tasks.

8

I use Obsidian to hold all things ever done, with all rules, changes, removals, archived changes, templates, formats, mistakes, solutions, etc etc

Then I push all tasks to 1 bot who tells another bot to do anything I ask, for coding it pushes to a coding bot that pushes everything out to Cursor. I believe last I checked Cursor it said I had an 87% success rate? I don’t remember.

I do know from my own internal logs of tracking everything, there is maybe 1-3 mistakes per day in total work load of 6-10 hours per day. The mistakes a typically fixed prior to landing on a finished area, and also typically before becoming permanent. The majority of clean up I have to do just comes in the way of my own internal formatting and extreme level of detail control. So the majority of issues are coming from human error caused by granular changes happening in a highly specific area not getting passed down because 1 or more bots are doing work based on older rules so when newer information/data/tasks are coming in they aren’t set up to be in the latest formating and placements I have created.

I also recently yesterday did a complete overhaul of my entire website and complete overhaul of my entire Obsidian Vaults folders and files, zero loss of anything across over 2 thousand files, and very minor issues with labels/tags due to the total name changes of nearly everything and needing to have over 200 scripts paths rewritten. This was broken down into 5 stages and 3 dry runs.

I also have now started building a full workflow diagram that covers 2 separate teams passing tasks to be completed between each other, so it’s very complicated set up that would be assumed to have lots of errors since while bots are working I’m modifying how everything is structured, inputting new ideas, building out projects, and making sweeping changes that effects incoming completed work.

So I think it really just comes down to how a person wants to structure the workflow chart of:

User > input > Bot > output > User

Because that structure matters more so then the actual model a person uses because that structure determines how a task of any kind no matter what it is flows through everything and then is returned. Doesn’t matter if it’s research, data scraping, inputting data to tables, coding, design rendering, making scripts to act as check points.

9

Thanks for the evals link. I’ve ended up having one model clean up another’s work, which makes the time saved hard to judge. I’d want to compare the first patch with what I actually kept after review.

10

Yes, comparing the first patch to what remains after review is the fairest way to do it. To measure that, you need to save the first patch somewhere before anyone starts editing it:

  • With the SDK in cloud, the first patch is already stored separately. Each run through Agent.create gets its own branch, and you can find it in result.git.branches. After review, run git diff --stat <agent branch> <final branch> and you’ll see what was removed or rewritten.
  • Locally, commit the agent’s output right away as a separate commit, for example agent: first patch, then make review changes and fixes in later commits. The diff between that commit and the final state will show your cleanup.
  • When one model cleans up after another, save that cleanup as a separate commit too, and note in the commit message which model did it. That way you can measure how many lines from the first model made it to merge, and how much work went into cleanup. It also makes it easier to see whether that setup is worth it compared to using one model with stricter rules.

If you collect numbers from even a few real tasks, share them here. It’d be interesting to compare.