Opus 5.5 was just released so I have not yet tried it in detail.
But I am intrigued - Anthropic’s benchmark marketing suggests Opus 5.5 is superior to Fable 5.1 and less expensive.
Why use Fable?
It’s definitely not unusual that newer models outperform more expensive older models. And it’s a widely accepted recommendation to use the more expensive models only when absolutely necessary as most tasks don’t require them.
Anyway, I’m sure Fable 5.5 will follow rather sooner than later.
1 Like
I ignore benchmarks, as they tell me nothing I need to know about how well a model actually performs. For example, I’ve been running most of my Cowork sessions on Opus 4.8, even though Opus 5 was available. I found that Opus 5 kept ignoring harness instructions and repeating the same mistakes, and, in general, made any session I ran with it nearly worthless.
Now that Opus 5.5 is out, this model seems to follow instructions better, and I’ve been using it so far with no issues.
Fable 5 and 5.1 have been very good at writing intricate code and do a really good job of analyzing in an Advisor role. Opus is good at these things, too, but I tend to use the Fable models for mission-critical things, as it seems to do a better job of follow-through and handling “what-ifs” that come up in the process.
It is correct that there have been major changes in Opus prompting guidelines (separate entirely from benchmark issues) since Opus 4.7. Starting with version 4.7 Opus needed more strict instructions; Anthropoic viewed it as a “feature” that it would not make assumptions about what you wanted it to do.
https://medium.com/@tentenco/stop-using-old-prompts-the-claude-opus-4-7-hack-guide-1379a9ecdd69
I haven’t had any issues establishing “strict instructions.” Actually, I think I could say that, with Opus 5, the issue wasn’t the lack of clear, strict instructions, but the model’s inability to follow them. I think Opus 5 was broken in some way. When using that model, I would often get, “Wait, I forgot to do this,” or “Oops, this is the 4th time I neglected to check this file,” or similar issues.
Have you (or anyone) done any testing or had anecdotal experience with the effort levels? The default seems to work well for me, as does the next level up (though I don’t notice any difference between them). I did run a task in Fable at the highest level, and it ended up running out the allotment and costing me an extra $60 to finish.
Other than running for hours longer than it should have, I can’t say the extra effort yielded anything beyond increased revenue for Anthropic, since the project results were no better or more revealing than running at the default effort level. I will openly admit that I probably chose poorly on this, but it was something I needed the most accurate results for.
The article was interesting, though old, but highlights one of the issues I think we’re going to face more and more with these updated models. I’ve heavily modified the harness to improve memory and functionality over the default harness, but as Anthropic changes the model’s behavior, I have a feeling I’ll eventually have to start over from scratch.
Yes - its benefit appears to be not necessarily projects that are cognitively “hard” to comprehend but rather projects/queries that will take an extended time to solve and where you do not want the model or harness to give up in the middle of it.
1 Like