Hacker News new | ask | show | jobs
by pu_pu 366 days ago
This is not a difficult problem to solve. We can add the schema, columns and column descriptions in the system prompt. It can significantly improve performance.

All it will take is a form where the user supplies details about each column and relation. For some reason, most LLM based apps don't add this simple feature.
1 comments
joshstrange 366 days ago
It’s not a difficult problem to solve, I did it, last year, with 3.5, it didn’t help. That’s not to say that newer models wouldn’t do better, but I have tried this approach. It is a difficult problem to actually get working.
link
pu_pu 366 days ago
So, I have not tried it on a very complex database myself so I can't comment how well it will work in production systems I have tried this approach with a single Big Query table and it worked pretty well for my toy example.

If by 3.5 you mean ChatGPT 3.5 you should absolutely try it with newer models, there is a huge difference in capabilities.

link
joshstrange 366 days ago
Yes, ChatGPT 3.5, this testing was a while back. I’m sure it has improved but I doubt it’s solid enough for me to trust.

Example/clean/demo datasets it does very well on. Incredibly impressive even. But on real world schema/data for an app developed over many years, it struggled. Even when I could finally prompt my way into getting it to work for 1 type of query, my others would randomly break.

It would have been easier to just provide tools for hard-coded queries if I wanted to expose a chat interface to the data.

link