[jarodrh]

Very interesting thoughts. This is a whole discussion on its own.

The ones that I mulled over that I feel have legs in my line of thinking are:

> minimize cost to converge upon sufficiently low error problem

- This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that via sampled prompt logs.

> Evals and OpenInference (OpenTelemetry) might be useful.

- Evals already a given. This is frugon's wheelhouse, however it currently only supports OpenAI-style jsonl. I'll definitely consider adding support for OpenInference.

> A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

- This is an intriguing idea. This would actually enhance/strengthen frugon's stance on the "minimize cost to converge upon sufficiently low error problem". Noted.

[westurner]

> minimize cost to converge upon sufficiently low error problem

Is this a convex optimization or non-convex optimization problem?

> Evals

Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp; https://news.ycombinator.com/item?id=48532642

OpenInference and OpenTelemetry have a schema for agent sessions.

I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perhaps such an index is also useful for agent optimization

> Nonce

Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

[jarodrh]

Down the rabbit hole we go.

> Is this a convex optimization or non-convex optimization problem?

- Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small.

> OpenInference and OpenTelemetry have a schema for agent sessions.

- Good to know for when I look into it.

> Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

- Yes, this is a current weakness in frugon...Hmm, I see, so I'd essentially need to establish a baseline (with whatever the default temp is) + nonce, then compare both spreads. Noted.

Good thread. Give frugon a try on your own logs, and tell me what you think.

[westurner]

As well. Someday costed opcodes and sharded redundancy for all of this;

> No search needed because the problem is discrete and small.

If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function (that assumes or imposes a distance metric to make a metric space).

Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

[jarodrh]

> As well. Someday costed opcodes and sharded redundancy for all of this;

Hmm, noted.

> If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function.

- Yes, what I'm saying is that frugon has a small enough candidate pool that it can all be evaluated at the same time. It doesn't need to search through it.

> Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

- This is exactly the shape the judge prompt takes in frugon.

See: src/frugon/measure.py

Then search: JUDGE_PROMPT_TEMPLATE