In my rules I ask the model to print a simple status line before each reply telling me which model they are.
This is helpful when I’m running Auto mode, as I don’t know what model is responding to my requests.
Before the latest update; [Model: Claude-Sonnet-4] was the consistent model (sometimes it was 4-5). Since the update; it’s been [Model: claude-3-5-sonnet-20241022] consistently. A noticeable degrade in quality, even if I didn’t have the model indicating rule.
I’m on the Pro plan, so using Auto mode is cost effective. But, since this update, I’ve had to revert a lot more than I did under 4. So I’m definitely losing time and the consistency is all over the place. Sometimes it’s very aggressive and sometimes I’ll make minor tweaks (same request). with 4, it’s usually a more balanced response.
I don’t know if I’ve hit a tier that pushed me down to auto using this lower quality model. I know I can’t control what models are used, but I wish there was a way to atleast set a priority list for the models we’d like auto to choice before going into the C and D tiers.