|
|
|
|
|
|
by scarface_74
1045 days ago
|
|
|
That’s the beauty of using ChatGPT for programming questions. You don’t have to have confidence in the answer. You can easily verify correctness by running the code it outputs and testing it. That being said, I realize you do have to be careful about subtle bugs. My canonical example is “write a Python script that returns all of the AWS IAM roles that contain a given list of policies”. The code it usually generates works correctly as long as you don’t have more than 50 roles in your account. It won’t add pagination support unless you spot the bug and tell it. The second question I have is what type of questions do you find are better suited for Stack Overflow than ChatGPT? Are they questions based on post 2021 knowledge? |
|
|
This is not always as easy for all types of questions. It's hard to come up with an example on the spot, but I tried a few queries I searched on SO in the last week, and found one to demonstrate. I was searching for "Where to store JWT in browser?". This is the SO answer [1] for reference. Now to prove my point, I ask the question to ChatGPT (3.5). Here's the chat [2].
On a first glance it looks like ChatGPT may have nailed it, though the information dump is HUGE. Among the 6 options it suggests, the last option "secure cookies" looks "correct". Indeed it is correct in what it says that this can prevent XSS attack. But it is not complete. Because it still does not prevent XSRF attack entirely. So I had to explicitly prompt it to think about XSRF and its response is weird. At first it incorrectly claims that XSRF attack is mitigated, but then in the response body it elaborates that we also need anti-XSRF tokens for complete protection. So I don't know what to make of it. Contrast this with the SO answer which is way more direct.
Honestly, ChatGPT's answer looks like that of a student who is trying to impress some examiner with their knowledge dump by beating around the bush rather than trying to precisely answer the actual question.
[1]: https://stackoverflow.com/questions/27067251/where-to-store-... [2]: https://chat.openai.com/share/c26fee93-5d3d-48e2-a820-297974...