[29athrowaway]

I took one of the questions from this article: https://blog.seekwell.io/gpt3

    Instruction: Given an input question, respond with syntactically correct PostgreSQL. Be creative but the SQL must be correct.
    
    Input: how many users signed up in the past month?

And it gave me a pretty nonsensical answer compared to the output from GPT-3 seen in that article.

[29athrowaway]

Because that answer was pretty weak, I resorted to an easier question... taken from the prison IQ test from the movie Idiocracy.

    You have one bucket that holds 2 gallons and another bucket that holds 5 gallons. How many buckets do you have?

The answer was complete nonsense.

https://youtu.be/jbmq9P-8FiM?t=17

[dismantlethesun]

I haven’t properly investigated their dataset they trained on this, but from the responses it gives I would bet that they dataset is skewed towards news articles, press releases, and infomercial or edutainment copy. It does really well when it can write a long impersonal article about something, but doesn’t seem good at the random sort of things GPT3 can do like autocompleting source code.

[datenhorst]

That makes a lot of sense. I played around with it in German and Italian and the results often times read like they were google translated from English. So there must be lots of auto-translated content from spam sites in the corpus, I guess.

[29athrowaway]

GPT3 can answer questions, in comparison this model works a little bit better than a markov chain.

http://joshmillard.com/garkov/