Could it be Claude 3.5 instead of Claude 4?

  1. Claude Web has a system prompt, as you can see.
  2. JB AI Assistant has a system prompt, as you can see.

I’ve done this test several times, others have checked it as well with Anthropic Console and you can test it with their API. In about 30% of cases it answers wrong.

I will bring it up again internally but this has been reviewed and tested by devs even checking the models responses as well as making sure the correct model is called.

1 Like