[shoxidizer]

I can't say I was impressed with ChatGPT help when I tried it. I figured quizing it on reading comprehension would be a great task, given that it is a language based model and a skill seemingly in short supply amongst my coworkers and self. After confirming that the specifications of a standard I am implementing were within its knowledge, I tried to have it explain the difference between two parts and it failed so miserably that its understanding of the content was below even my managers for whom this is only something they occasionally review. Any attempt to correct it only resulted in it providing an apology and new misunderstandings. Outside of work, I tried using it to find an old movie, probably from the '60s, about a man refusing to shave his long beard and featuring a scene with him being chased around his home half shaven, but it merely made up scenes about beard shaving for several other movies. Admittedly, I have not tried uploading any of my companies code to give it a less memory based task.

[joe_the_user]

I think reading comprehension is a notable weakness - asking it detailed questions about a long text comes up with lots of hallucinations in my experience.

But it's definitely good at some other things. Writing boiler plate texts of various sorts and giving instructions on how to do certain things notably.

It seems to mostly synthesize common knowledge rather than learning anything. But that can be very useful, a lot of people's job involves doing things like that today.

[dragonwriter]

It sounds like you were testing memorization not reading comprehension.

To test reading comprehension, the source should be in the prompt, not the training set.

[Firmwarrior]

yeah, same boat here.

It's great for generating sample code snippets or refactoring code, but I can't paste my company's intellectual property into it

If I could train a customized version of it on all my company's Slack messages, Jira tickets, e-mails, etc it'd be insanely useful . . . . but I don't think any big company would actually want that, since it wouldn't be able to keep secrets from anyone with access to it