[dayjah]

This is a thin edge of the wedge issue, right? ChatGPT is pretty darn good for most things. I’ve used it extensively for the past 18 months and only in a few cases would I say it “completely lied to me”.

My general rubric is: “would I trust someone on Reddit to correctly guide me on this”. If the answer is “yes” then ChatGPT is likely going to do well. If the volume on a particular subject is low / susceptible to false information then it’ll lie.

Recently it lied hard about how to configure MikroTik routers. I lost many hours. But for a large construction project recently it completely balled out.

Are you doing cutting edge / complicated stuff? Have you examples of where it lies?

[hypeatei]

> Have you examples of where it lies?

No specific prompts, but most were related to the XHR/Fetch specs and behaviors within. It would say "X.Y.Z sections defines this" but that section didn't exist at all and the answer provided was not accurate.

> My general rubric is: “would I trust someone on Reddit to correctly guide me on this”. If the answer is “yes” then ChatGPT is likely going to do well

I see. Well, I don't know if I find that very valuable but if others do, then so be it.

[adamc]

I've asked it for things like book recommendations and gotten:

  - completely made up books
  - real books that were only marginally related
  - real books with really bad reviews

I'd estimate that only 30-40% of the time did I find the results at all useful.

[airstrike]

stop using it like a database that you can query

[bigfudge]

Agreed this is a bad idea in the case you are replying to, but I love ChatGPT as a way to recover the name of a book or film I’ve forgotten. I recently prompted for “a book about nuclear wasteland dominated by a church” and it gave me A canticle for Leibowitz (which is great). I’m not sure how easy that would be any other way.

[airstrike]

absolutely, I've been using it for that at lot as well and it's remarkably well suited for it. there's really no better tool for the job.

it's just that every thread about LLMs in AI invariably has someone complaining about best results from a query best described as `SELECT * FROM...`

[cruffle_duffle]

I wonder how many people are promoting it correctly. You can’t just query it like you might for google or something. It works best with lots of context and back and forth. And yeah, for many things you are going to get directional answers not exact ones (esp with “rote memory” like exact quotes from a book or something.)

[therein]

I don't want to turn this into another Claude lies less than ChatGPT subthread but since you mentioned configuration of MikroTik routers I felt like I should.

ChatGPT lies a lot about RouterOS, I don't know why. Claude helped me a lot on the other hand with all things MikroTik.

[dayjah]

Thanks! I’ll give it a shot when I get to the vlan stuff I’ve on deck