We evaluated BugBot across a multi-week pilot and hit the same behavior: findings surface incrementally across re-runs rather than on the first pass, some MRs needed 19–30 runs, and BugBot sometimes leaves its own thread unresolved after the fix lands and re-flags.
Our developers trust the findings. The problem is that a clean pass isn’t a stable signal, so we can’t build review policy on it. Under per-run pricing we also pay for every extra cycle the tool needs to catch what it missed on pass one.
Questions:
Has anything shipped since April that improves first-pass completeness or re-run stability?
Is there, or could there be, a mode that trades latency/cost for a complete, deterministic first pass?
Is the unresolved-thread-after-fix behavior a known bug?
You can now turn on “Incremental Review” to avoid having Bugbot review code that was already reviewed in a previous review. This prevents Bugbot from constantly raising new issues on code that has already been reviewed.
You can also choose the effort level (changing from Default to Deep) for a more thorough analysis.
Out of curiosity, was the root cause mainly the inherent non-determinism of LLMs and the surrounding tooling, or was there something more specific going on? I’d be interested in hearing a quick retrospective on what was happening.
It’s mostly the nondeterminism! If you ask an LLM to find 10 things wrong with your code, it will find 10 things. And if you ask it to find another 10, it will scrounge up 10 more! We aim to ensure the first pass surfaces the most significant issues, then let users decide whether they want the code fully scanned again, or only the changes since the last pass.
