[djhn]

I find SQL and data(bases) in general to be LLM’s Achilles’ heel. Databases are rarely under version control, so the training data only has one half of the knowledge.

My comments are more in the context of OLAP queries and other non-normalised data often queried via SQL.

I train non-LLM transformer models on (older and rarer) datasets, and automating the ingestion of sprawling datasets with hundreds of columns, often in a variety of local languages and different naming conventions adopted over decades, with quite a few duplicated columns…. The LLMs perform badly, it’s nigh impossible to test (for me as a user in prod) and it’s nearly impossible for the LLM companies to test (in training) to RLVR and RLHF this.

[flir]

That's interesting - SQL is one of the places I find them the strongest - I think there must be an insane amount of training data out there for SQL. But mostly I'm asking them for ad hoc report queries. Nobody cares if they're bad SQL, they just want to know how many signups there were in March that didn't tick the marketing box. Sounds like you're pushing their capabilities a lot further than I am though - I just want to perform arbitarily complex queries on 3NF data.

[timacles]

Yeah not sure what this guy is talking about, LLMs excel with queries because the SQL language is pretty small in scope and its easy to test the output. Table structure and relationships are easy to feed to the AI.

> I train non-LLM transformer models on (older and rarer) datasets, and automating the ingestion of sprawling datasets with hundreds of columns, often in a variety of local languages and different naming conventions adopted over decades

All of this sounds like basic data processing

[hunterpayne]

"Nobody cares if they're bad SQL"

Laid off your DBAs I see.

[flir]

Ok, ok. Nobody who matters cares if they're bad SQL ;)

[Tycho]

Just use an LLM to make a good knowledge base for the databases. Based on schema info and production queries. An agent can use that to write queries that work.