[jameshart]

If you just ask GPT-3 text-davinci-002 to complete

    Create a Python program to reverse a string:

It produces

    def reverse(s): 
        return s[::-1]

And that isn't even the code-specific model.

[bobcostas55]

What happens if you ask it to evaluate the function it generated, with some input?

[jameshart]

Good question.

I just asked text-davinci-002 to complete this prompt:

    Here is some python code:

    def reverse(s): 
        return s[::-1]

    What would this function return for the input "hello"?

And it actually responds:

    The function would return "olleh".

Sadly, when I asked it to try "encyclopedia", it said:

    "aidnocypilecne"

Which is a brave attempt, but not quite right.

To see if it was just making assumptions based on the function name, I tried to trick it with

    def reverse(s): 
        return s + 2

    What would this function return for the input "hello"?

But GPT-3 is too clever for that:

    It would return "hello2".

Also, it even recognizes what the function will actually do, when it's named misleadingly:

    def getLength(s): 
         return s[::-1]

    What would this function return for the input "hello"?

Still produces:

    It would return "olleh".

[jameshart]

One last result:

With this prompt:

    Here is some python code:

    def getLength(s): 
         return a.length

    What is the bug in this code?

GPT-3 says:

    The function is called getLength but it returns the length of the string a, not the string s.

That's it, pack it up, we're done here.

[bobcostas55]

Absolutely fascinating stuff, thanks for trying it out! I just wish we had some way of really understanding what's going on under the hood...