[joquarky]

I feel like there's a limit on constraints that doesn't necessarily follow the context limits. I've assumed this is "attention heads" which I understand are an independent limitation, but I'm not smart enough to understand all the layers involved in these models so I could be wrong there.

I do observe the same thing. There are a limited number of constraints you can add and once you exceed that, you'll play whack-a-mole if you insist on all of them.

This is why I tend toward a more wu-wei attitude to constraints.

For example:

- Do I really need this constraint?

- How does the agent tend to behave in this scenario it if unconstrained? Is this behavior/result an acceptable pattern for this solution?

- Is the constraint implicitly followed often enough that I can trade spending tokens recovering from a deterministic test that enforces the constraint rather than preemptively state it in the prompt?

If I get into the situation where I need more constraints than can fit in context/attention without the need to regularly play whack-a-mole, then I break the module down into sub-modules with fewer, more specific constraints.