Starting this week, we have been experiencing high volumes of completely invalid bug reports on branches being analyzed by BugBot. The invalid reports cover a range of issues:
referencing code from old commits or MAIN that no longer exists in HEAD of the branch (previously did not experience this at all)
similarly to above, re-raising the same bug report even after that section of code is changed and the originally reported bug is fixed.
complete misunderstanding of the function of the piece of code a bug is raised for (previously for the same code, seemed to have no issues)
in packages with multiple files where different files are being worked on in separate branches… bugbot is raising bugs in code that was not changed in files that have no changes in 1 branch, and not in other branches… a very misleading situation where the same exact script has a magical (often invalid) bug in 1 branch despite no changes, and in another is perfectly fine.
This seems to primarily (for us at least) be an issue with Python and BASH scripts.
Steps to Reproduce
For us, we run bugbot automatically on an open GitHub PR that is marked ready for review, and it re-runs on commit. Essentially, all you do is run the bugbot on a branch of code to reproduce this behavior.
Expected Behavior
I expect the bugbot to raise valid reports, not repeat itself, reference the HEAD of branch, and have a contextual awareness of the section of code it suspects a bug may exist in (so as to determine if that’s true or not based on the use case of the snippet it’s targeting)
No such reference _validate_check_health exists in the code anymore, not around like 317 or within the function defined therein, this class attribute was originally named _validate_check_health and was then replaced by an upgraded version _validate_check_health_threaded but the bugbot continuously threw bugs claiming we were calling the original name, even though that had been fully replaced to calls and class definitions for the new name.
in response, I adjusted the naming convention to attempt to get the bot to “see” a fully different function, to no effect, it still seems to be expecting _validate_check_health which does not exist in code anywhere at HEAD of branch
The second bug it reported did turn out to be a valid report.
The third bug it reported was valid in some use cases, just not in our particular use case.
While the second two bugs reported are new, which represents new behavior from this morning; the first is still the exact same problem we’ve had for the past 24 to 48 hours with reports that seem to originate in a “hallucination”
Here is another example. It provided a summary of the fix as a PR comment / bug instead of in the description. serverGenReqId_e3bd5749-904c-4dfd-bcc1-83366050ecf3
Hey! Yes, we detected a regression that may have affected a few users for a few days. Should be all resolved now Thank you for raising this issue and definitely feel free to tag me if something seems wonky in the future (pls not for like one false positive but if something feels largely different)
Gathered PR context (3s)
Thank you for raising this issue and definitely feel free to tag me if something seems wonky in the future (pls not for like one false positive but if something feels largely different)