So, I just tried using agent mode with what I thought was an incredibly simple task. I could have done it in 30 seconds with a regex-replace (and eventually I had to.)
The task is just this: Here’s an SVG files which contains some class attributes which are just simple integers, e.g. class="1". For every instance of class="n", where n is a simple integer, replace it with class="tri-n".
There’s like 20 of them in the whole file. I could do it by hand in a few minutes, and with a regex-replace even faster. But I thought I’d try the model on it, to get used to using agent mode.
Every single attempt resulted in some absolute massacre of the source code. Elements being deleted and added, whole SVG path strings being rewritten into crazy tangled messes, elements being duplicated.
I kept adding more clarification: “do not add or delete any elements. Do not modify anything other than class attributes” and probably ten more attempts to make it just do this one dummy simple task. The best result I got was where I switched it to claude-3.7-sonnet MAX with pay-per-use billing enabled. It did the task, and “only” mangled four other properties.
Was I doing something wrong here? Or is agent based coding just overhyped rubbish, expecting a language model to do anything except copy and paste other people’s work and generate stock value for founders?
755 lines of code, 58KB. So, not trivial, but I’ve definitely seen much larger actual code.
I’m kind of feeling like it’s a training issue. People don’t work in SVG-as-source-code that much, it’s usually treated as an image asset, so there’s not enough good examples for the models to vacuum up.
Find all instances in [filename redacted] where there’s a pattern of class=“n” where n is a simple integer (like class=“1”, class=“2”, etc.) and replace each one with class=“tri-n” (so class=“1” becomes class=“tri-1”, class=“2” becomes class=“tri-2”, etc.)"
After it mangled my source the first time, I followed up with:
Do that again, but only change the classnames, nothing else. Do not add any new elements or change anything that isn’t a class attribute.
It burbled to itself for a few minutes, then ruined the enire file again. My next followup was (this was where I switch to 3.7 max):
Try again. ONLY change class attributes. Any class which is a simple integer should be prefioxed with “tri-”. For example, class="1" would become class="tri-1".
(yes, the typo “prefioxed” was there)
It bungled it again. Admittedly at this point I lost my temper so I’ll be first against the wall when SkyNet takes over:
Look at line 597. You are changing a d attribute for absolutely no reason and in specific contravention of what I asked you to do. DO. NOT. CHANGE. ANYTHING. EXCEPT. CLASS. ATTRIBUTES. CAPEESH???
…and there were a few more rounds after this. Apart from the typo and (finally) losing my temper, what was wrong with my prompt?