[LittleShaman]

Errors like suggesting incompatible combinations of parameters or hallucinated non-existant params you will easily catch once you try them. But what about more subtle conceptual misunderstandings?

I meant cases like asking how to X and ChatGPT suggesting something similar (and valid) but not what I wanted. Or imagine silent failures like an additional switch that happens to exclude what you wanted.

Not an issue if youre just automating grunt work you know to do yourself of course. But I'm interested in when you arent familiar with a tool.

[Zababa]

Google and stackoverflow have the same issue. Often I won't find the answer I'm looking for, and I'll have to use something that's close to my problem but not exactly the same. ChatGPT has the same issue, but I can tell him what work and what didn't and he'll give me an updated answer.

The alternative to that is to open 15 tabs with stackoverflow, forum threads, github issues, reddit. For the problems I usually encouter, it makes sense to ask ChatGPT first, and if I see that I'm getting nowhere after 2/3 replies I'll fall back on Google, documentation, trying something else.

[LittleShaman]

I've expressed this badly. By "but not what I wanted" I don't mean it suggesting an alternative. I mean it giving an answer claiming to do what I asked for, but which doesn't actually. I would not be able to catch this. I hope this clarifies why I gave that as an example to my uncertainty of ChatGPT giving wrong answers.

On stackoverflow and the like you will know that the question does not apply to you.

[Zababa]

If the answer doesn't do what I want, either I see it and can fix this, or I don't and then the origin of the answer doesn't really matter. That's my experience solving my problems, and I'm sure other people may have different experiences that leads them to different conclusions. But for me, for now, ChatGPT as a first step makes sense.

[ParetoOptimal]

> I mean it giving an answer claiming to do what I asked for, but which doesn't actually. I would not be able to catch this. I hope this clarifies why I gave that as an example to my uncertainty of ChatGPT giving wrong answers.

You decompose the problem further and then tell gpt it was wrong and what you know.

[ParetoOptimal]

> Not an issue if youre just automating grunt work you know to do yourself of course. But I'm interested in when you arent familiar with a tool.

That's valid. I find that most people lump both "gruntwork you know how to do you are automating" and "not familiar with tool and using gpt to figure it out".

In truth, they both require very different strategies. I don't place a lot of importance and have a very high bar of proof required for letting it help me figure things out.

Essentially for figuring things out I make it give me examples I can independently verify to make sure I understand things together at a conceptual level.