My work require long running multistep work, and grok 4.6 is not getting better significantly here, especially on hard problems where it tends to make mistakes. It’s okay to make mistakes on first try, but the problem I have for grok 4.6 is how it deal with the failure.
A typical behavior mode is: It try to tackle with one method, wrote some code, and it didn’t work. then it immediately try something else, with zero dig into why it failed with that method. Maybe a small fix would do, maybe it’s fundementally wrong, but it has zero idea. and the new pathway it choose is often not to deal with real problems but to bypass it. Without the true problem being diagnose and fixed, it would accumulate mistakes even the bypass worked somehow. For hard tasks like the experiment I’m doing, the bypass is directly a total failure, it would never worked, so this strategy would make the work unsolvable.
I try to invent some prompt or skill to change this behavior, yet found it be so strong that it just immediately deviate from my prompt. This is an issue of command following.
For iterative improvement, of experiment, the autonomy is horrible in grok 4.6. when asked to solve a problem, run test and debug, iterate, it would often just implement with the code, and be done, and concluded with: I did this and that, and it still don’t work, the next step would be something. It cannot persist until the goal is really reached.
For experimental work like what i’m doing, the ideal agent should persist on the goal, reach for information and mechanism diagnosis, reasoning to iterate, learn from mistakes and fix accordingly, and iterate. If it need some key information, ask immediately without guessing.
Current agents rely on traditional engeneering, where everything could be pre-planed, and implement step by step. This pattern don’t work for experimental project, where you trial and observe and make improvements by comparing theory with reality.
I did use it to help dev a web based lab, and it went very smooth, it can persist to implement a lot. But it still often claim finished without test it at all. Without run and see, it’s often just have some little bugs.
Biggest problem is that, non of the behavior mentioned above changes according to prompt, agents.md at all, direct command just simply being ignored immediately.
It needs the general capability to trial and error, and dig into it to find the real problem, and solve it, and continue from where it was. Especially when the agent don’t have the capability to guess the reason right. The post training of RL will reward it anyway if it can guess the reason right, or have the problem solved by fit to the problem itself. But we need general capability of problem solving, not overfit on old existing problems.
When each digging is specifically prompted, it can reason pretty well, but to chain those into a multi step autonomous iteration, it cannot. We need improvement here.