[datastoat]

I'm trying something similar with an introductory Algorithms class.

After we go through Breadth First Search, there's a practical assignment where students are asked to modify the algorithm to return _all_ shortest paths. Then I ask ChatGPT for its solution, and students try to spot its mistakes.

Later, after going through the proof of correctness of Dijkstra's algorithm, I ask ChatGPT for a proof of correctness of its all-shortest-paths algorithm, and again students try to spot what's wrong in the proof. I want students to learn to tell the difference between a bullshit proof and a real proof; in the past I've given them bullshit proofs from real students in exams, but ChatGPT makes the point more nicely.

Finally, students are asked to figure out prompts that will make ChatGPT give a correct algorithm and proof. I haven't managed this myself! I'm looking forwards to seeing what students manage.

Here's a link to lecture notes, including the ChatGPT dialog: https://www.cl.cam.ac.uk/teaching/2223/Algorithm2/alg2-full....

[esperent]

> Finally, students are asked to figure out prompts that will make ChatGPT give a correct algorithm and proof. I haven't managed this myself! I'm looking forwards to seeing what students manage.

Isn't there a probabilistic nature to ChatGPT replies? So even if a student finds a response that gives a correct proof, that doesn't mean it'll work every time. Or am I wrong here?

[datastoat]

You're right, ChatGPT is probabilistic. None of this is graded by the way -- it's all just for fun and bragging rights.

I've asked students to share their full dialog, both prompts and replies, so the whole class gets to see; and I'll invite one or two to talk through their attempts. This is all just a trick to make students engage with "how do you you spot bugs in a proof?", hopefully more than they would from just reading CLRS! Often, students engage well when they're hearing the material from other students.

[graderjs]

I think this is a great idea. I love when teachers do something fun and innovative like this!

[eternalban]

You can set a ceiling on temperature or just simply dictate a low setting to insure repeatable performances by the LLM. This could be on your reading list as well: https://sites.google.com/view/automatic-prompt-engineer

[williamcotton]

Aside from a temperature of 0, which always results in the same completion, the details and translation examples (aka, in-context learning, few-shot) can force very reliable results, say, 8/10 times, meaning a sample-and-vote gives consistent results when the temperature is non-zero.

Edit: I was not in any way rude nor saying anything incorrect.

If you want to see how to do what I’m talking about, here’s an almost finished article describing the above:

https://github.com/williamcotton/empirical-philosophy/blob/m...

[tough]

Upvoted you, don't mind the false flagging

[danuker]

It's an arms race. As students cheat with ChatGPT, teachers should incorporate tactics throwing it off, perhaps using ChatGPT themselves.

I think what you are doing is great! Good luck going forward!

[arcturus17]

Can't they feed the code to ChatGPT and ask it to spot the mistakes, though?

[datastoat]

I tried... I pointed out a problem and asked ChatGPT to fix it, unsuccessfully. I asked it for a proof of correctness, then pointed out a problem in its proof and asked ChatGPT to fix it, again unsuccessfully. (It's all in the notes I linked to.) Perhaps I'm just crummy at prompt engineering; or perhaps this is one of those questions where the only way to engineer a successful prompt is to know the answer yourself beforehand.

[MaxMatti]

I've also had this issue multiple times where ChatGPT provides a flawed answer, is able to identify the flaw when asked but "corrects" it in such a way that the original answer is not changed. I've tried this for code it wrote, for comments on my code and for summaries of texts that I provided.

[danuker]

Reminds me of children trying to speak a word properly, but repeatedly making a mistake in the same way.

[wprl]

I can’t tell if people just don’t understand how ChatGPT works or if there is another reason they are eager to dehumanize themselves and the rest of us along with them.

[curiousllama]

> in the past I've given them bullshit proofs from real students in exams

I'd be so honored to be one of these proofs. "Your wrongness is an elegant balance of instructive error and subtle misunderstanding. Can I save it for posterity? (c-)"