I too was puzzled by the response from Claude.
I am using the Anthropic workbench with claude-3-5-sonnet-20241022 (latest)
But it think it has to do more with the freshness of training data.
AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.
Yes. I find it a bit funny how much people care about leaderboards. I see models going up and down, winning this or that benchmark and yet, for me, Sonnet 3.5 still beats the crap out of all of them.
But it think it has to do more with the freshness of training data.
AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.