[emporas]

Then you need a Large Arithmetic Model (LAM). We have that, it's called calculator.

The LLM could invoke several command line programs, including calculators or anything else in which a deterministic answer is desirable. Structured outputs for example, people usually mean Json output, but any schema like Xml or Html could be enforced by some command line tools, and when the validation fails, it should double check it's own output and hopefully fix it.