[wongarsu]

Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it

And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases

With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month

[batshit_beaver]

The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.

[kridsdale1]

Models learn. It just costs $10B and 1 year to do what a human does every night.

[gls2ro]

This is not about learning only.

It is about context window too so no matter how much money you put to train your model it can still have degraded performance in following instructions but it will be better to execute the instructions they can follow.

It is also about the harness and how that can help drive the model and pick and choose what to include or not in the context.

These LLMs do not understand the project. For them any next word is good as long as it was picked by the token predictor. It does not have any way to understand but only to do. probability distribution over their vocabulary and if that vocabulary is tainted and lost parts of the original context what is a good candidate there will not match the intention of the initial project.

[wongarsu]

LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models

[8note]

does lora do a good job at teaching the model new things that werent in the training data?

without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with

[irthomasthomas]

No. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.

[didgeoridoo]

There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though.

[0]: https://sakana.ai/doc-to-lora/

[cadamsdotcom]

Is this because of catastrophic forgetting, or is that more or less solved?

Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?

Or something else?

[pixl97]

At least these costs are currently preventing the planets surface from being covered in paperclip maximizers for the moment.

[chrisjj]

That's not a model learning. It is model replacement.

[DoctorOetker]

"You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month"

So the behavior policy should reside in the weights, not in some ever expanding KV cache context, like sticking it in the wrong orifice if the goal is to be reproductive.

Instead of putting the policy document in some tightly crammed RAM, shove it in the weights that already exist, online / post training.

It does make one wonder if there is some way to compute the change to the weights from the computed context (so that we can free it up) without essentially continued "pre" training RMAD after each communication round.

[epolanski]

> Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it

Hmmm, if AI has to grow in the workplace, it has to follow processes to the letter, yet claude code forgets by the second turn my "don't commit" prompt.

Of course claude code is a generic and crap harness over terrific models that can't fit bureaucratic processes, and is increasingly worse at doing so anyway since Opus 4.6 peak.

[dinfinity]

Exactly: The best way to deal with this for humans is to use procedural scripts for different tasks, referring to the relevant bits of the (declarative) documentation.

I would imagine that doing something similar (using agent skills for insurance) would work much better for AI.

[loremium]

isn't it because there are too many contradictions and ambiguity? the reason it works for humans is because we don't apply everything at once either.

[AnimalMuppet]

No, it's more than that. We can't remember everything. I hired, say, two years ago; as part of my onboarding process I had to read a bunch of policy and procedure documents, which were full of stuff that I didn't understand because I wasn't really in the context yet. So at the time, to me, those documents were full of arbitrary text that I didn't really understand. Some of it was rules that I had to follow, but at the time I didn't understand why, so it's just arbitrary rules.

How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.

And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.

[pixl97]

Social effects are a huge part of being human. Quite often we just do what the people around us are doing. In general it's not a bad rule, but can quickly lead go mass casualty events when the crowd has no idea what's going on, or we're messing with dangerous things.

[ActionHank]

That's a great comparison, human vs ai on a wall of text.

The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.

Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.