I’ve been experiencing weirds bugs here and there with code generation in the chat window, sometimes text pops in weird places, I’ve gotten an “;ain” appearing to appear before, as if “main” was meant to be written but was clipped into the previous line — however, I’ve come to observe the occurrence of another weird bug, that is, sometimes the AI will start writing code, but then, instead of the code progressively being typed down the script, I see an entire window of completed code suddenly pop up, and I wonder what’s going on, until I realize that this is the exact same code as above. Here’s a pronounced example of this:
This is an image of the response to my first request. I asked it to create an entirely new file, to count the total number of tokens in files of my project.
Now, why does the first, entirely new file that the AI give have diff marks? It makes no sense, because this file didn’t actually exist before the AI wrote it.
The answer to that, is because in the next response, the one where I asked the AI to improve the console visuals, the AI seemed to teleport over its previous work:
I could see the generation animation playing, but no new text seemed to be generated, because all that was on my screen was the exact code it had just generated for me in the response before. I then hit the “Apply” button and confirmed to me that there were no changes by replacing every newline, with a new newline, AKA no change at all. But, we could see the diff marks anyway in the new codeblock — and then, surprisingly, in the codeblock of the previous response, which had previously had no such diff markings.
It seems that the code renderer did not differentiate between the old code the AI had written before, and the new one it was working on right now. Perhaps this is an issue with the model not correctly following whatever behind the scenes code window formatting is going on.
This is happening on Claude 3.7 sonnet. I’ve seen this same issue come up before, but in previous times I thought it was just the AI being silly, but now I’m convinced there’s a bad interaction between the AI and the chat system, as I’ve had to get around things like this before by telling the AI something looks wrong with its output on my renderer, but that was only happening on small windows before, so I didn’t realize what the problem was until I saw what was going on with this much longer chunk of code. On a similar note, I’ve seen other models having trouble with the formatting system going on here, notably Gemini 2.0 Flash, which I use for quick requests, stuff like “{edit}” appearing in weird places.
EDIT: Just happened again for the same file:
