[GodelNumbering]

Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio...

From Terminal-bench-2.1 details,

> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.

This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.

For reference, in tbench-2.1,

1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)

2. 8 out of 89 tasks allow 8GB RAM

This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1

As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.

[kingstnap]

Why are resource limits considered at all aside from models accidentally fork bombing themselves?

I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?

[GodelNumbering]

Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness in a given situation.

A few examples from memory:

1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).

2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.

3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes

[1] https://github.com/harbor-framework/terminal-bench-2-1/blob/...

[2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...

[solid_fuel]

> Which test cases does this matter for?

The test cases of "don't melt my computer" and "be a good (computational) neighbor"

[rsanek]

This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.

[efromvt]

Out of curiosity, how often are the resource limits the bottlenecks? What do harnesses do to help here - limit parallelism? More efficient tools?

[artrockalter]

The task could be verifiable in the environment so limiting its CPU and RAM could be to discourage brute forcing the answer.

[noobcoder]

Thats what is wrong with close source models, we dont know exactly what we are paying for, a superior base model or a well thought harness for benchmaxing

[kommunicate]

I get your point but I'm not sure it matters all that much.

Did harbor / tb2.1 cap the swap available to docker runs?

There used to be a bug that would allow dockerized instance runs to use more memory than the specs allowed. Some of the original tasks weren't really possible to complete without exploiting swap. Even the oracle solutions didn't pass if you stopped docker from having access to swap.

I think crack-7z-hash and filter-js-from-html had that problem off the top of my head, but i haven't looked at this in months, so i'm not sure

[meric_]

Huh? What are you talking about?

https://www.anthropic.com/engineering/infrastructure-noise

Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either

[GodelNumbering]

What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it

[meric_]

Yes but my point is - Resource limits are a "recommendation" and are not strictly enforced - Significantly boosting resources up to 3 did not statistically shift performance results

Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results