It can do basic math reasonably well (and this is achieving generation where GPT-3 failed).
Interestingly, asking it to verify itself does resolve bugs sometimes. Managed to fix subtle count() denominator bugs and an inflation-adjustment error with not much hinting on my end.
You can only see it struggle really hard at the end when it tries normalizing month ranges correctly. It seemed to reach conceptual problems over how LAST_DAY() was being used and current debug itself.
I tell it that I'm using <database and version> and give it the relevant DDL statements (e.g. CREATE TABLE, etc) then ask it to write the query to do <x> in plain English. It does surprisingly well.
But!!! the first response is rarely dead-on and instead, just like a junior eng I need to guide it: use (or don't use) SQL construct <x>, make sure to use index <x>, etc.
Example: to sum the values in a JSONB field, GPT desperately wanted to use a lateral join but that would have made for a very awkward set of VIEWs. So instead I directed it to create a function to perform the summation.
https://www.dropbox.com/s/hdhycf7l00d3sx8/gpt4_attempt_sql_q...
It can do basic math reasonably well (and this is achieving generation where GPT-3 failed). Interestingly, asking it to verify itself does resolve bugs sometimes. Managed to fix subtle count() denominator bugs and an inflation-adjustment error with not much hinting on my end.
You can only see it struggle really hard at the end when it tries normalizing month ranges correctly. It seemed to reach conceptual problems over how LAST_DAY() was being used and current debug itself.