[brotchie]

Text to SQL is the wrong framing of the problem.

Ideally you ask the agent the thing you want to know / business question you want answered.

An agentic loop then runs. The agent can look at the schema, look at any existing SQL scripts available that query similar tables, run a few limit 10 probe queries. Analyze the data, do some joins, check the data again. Show you the data, ask for feedback, etc.

Pure zero / one shot SQL generation isn’t the solution and isn’t how humans do things. We look at the schema, run some queries, do some joins, spot check the totals / row counts, etc.

[efromvt]

I think/hope that most benchmarks have moved to an agentic loop - I'd still call that 'text to sql', since you're going from the business question to one or more SQL queries that provide the answer.

With a loop you can get extremely high results on a clean DB with a clear question. (I'm usually seeing high 9s accuracy.). A messy/ambiguous DB schema then degrades that again (which is what companies actually have, where you get closer to 20-30% rates pre-context engineering), which a clean semantic/presentation layer can bring back up. Then you're just left with the problem of underspecified/poorly formulated question - which you can partially solve with the agent pushing back, but also can be solved by improving the human side of things.