[westurner]

Evals and OpenInference (OpenTelemetry) might be useful.

Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well.

Is this the cost to converge problem, the minimize cost to converge upon sufficiently low error problem, or the minimize cost and error problem?

EA methods: mutation, crossover, selection

Gradient descent as a mutation, crossover, and selection pattern; back up when the error/cost stops decreasing for too long and try a different branch.

A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

[jarodrh]

Very interesting thoughts. This is a whole discussion on its own.

The ones that I mulled over that I feel have legs in my line of thinking are:

> minimize cost to converge upon sufficiently low error problem

- This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that via sampled prompt logs.

> Evals and OpenInference (OpenTelemetry) might be useful.

- Evals already a given. This is frugon's wheelhouse, however it currently only supports OpenAI-style jsonl. I'll definitely consider adding support for OpenInference.

> A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

- This is an intriguing idea. This would actually enhance/strengthen frugon's stance on the "minimize cost to converge upon sufficiently low error problem". Noted.

[westurner]

> minimize cost to converge upon sufficiently low error problem

Is this a convex optimization or non-convex optimization problem?

> Evals

Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp; https://news.ycombinator.com/item?id=48532642

OpenInference and OpenTelemetry have a schema for agent sessions.

I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perhaps such an index is also useful for agent optimization

> Nonce

Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

[jarodrh]

Down the rabbit hole we go.

> Is this a convex optimization or non-convex optimization problem?

- Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small.

> OpenInference and OpenTelemetry have a schema for agent sessions.

- Good to know for when I look into it.

> Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

- Yes, this is a current weakness in frugon...Hmm, I see, so I'd essentially need to establish a baseline (with whatever the default temp is) + nonce, then compare both spreads. Noted.

Good thread. Give frugon a try on your own logs, and tell me what you think.

[westurner]

As well. Someday costed opcodes and sharded redundancy for all of this;

> No search needed because the problem is discrete and small.

If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function (that assumes or imposes a distance metric to make a metric space).

Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

[jarodrh]

> As well. Someday costed opcodes and sharded redundancy for all of this;

Hmm, noted.

> If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function.

- Yes, what I'm saying is that frugon has a small enough candidate pool that it can all be evaluated at the same time. It doesn't need to search through it.

> Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

- This is exactly the shape the judge prompt takes in frugon.

See: src/frugon/measure.py

Then search: JUDGE_PROMPT_TEMPLATE