Why are the edit tools so flaky?

Bug Reports
1

I am always getting messages like
It seems there was an issue creating the file. Let’s try again directly:

or the edit was not applied by the model.

or Model provided invalid arguments to terminal tool.

or It seems our edit didn’t replace the existing date validation code, but instead just added the new code. Let’s look at the full content of the current script:

or It seems that our edit completely replaced the file instead of just updating the date validation section. Let’s recreate the full script with the correct validation:

It seems this should be the easier problem to solve – the model already knows what it wants to do and seems to be all thumbs in changing the code.

This wastes a lot of time and tokens.

2 Likes
3

Yeah, I have experienced this as well, more with gemini-2.5 than Claude-3.7, especially when it with large files (over 700 rows).
In both cases if I see the agent is finding if difficult to make simple edits I ask it to tell me what to change and do it myself.

4

Now it is literally echoing strings >> a shell script because it can’t seem to edit it directly

5

I concur.
One step forward. One step back.

6

came here looking for answers to:

[Model provided invalid arguments to terminal tool.]

wastes so much time! :mantelpiece_clock:

but i suppose we are still moving 10x faster overall? :crossed_fingers: