[ssharp]

I keep hearing about people using these for coding. Seems like it would be extremely easy to miss something and then spend more time debugging than it would be to do yourself.

I tried recently to have ChatGPT an .htaccess RewriteCond/Rule for me and it was extremely confident you couldn't do something I needed to do. When I told it that it just needed to add a flag to the end of the rule (I was curious and was purposely non-specific about what flag it needed), it suddenly knew exactly what to do. Thankfully I knew what it needed but otherwise I might have walked away thinking it couldn't be accomplished.

[GiorgioG]

My experience is that it will simply make up methods, properties and fields that do NOT exist in well-documented APIs. If something isn't possible, that's fine, just tell me it's not possible. I spent an hour trying to get ChatGPT (4/4o and 3.5) to write some code to do one specific thing (dump/log detailed memory allocation data from the current .NET application process) for diagnosing an intermittent out of memory exception in a production application. The answer as far as I can tell is that it's not possible in-process. Maybe it's possible out of process using the profiling API, but that doesn't help me in a locked-down k8s pod/container in AWS.

[empath75]

I think once you understand that they're prone to do that, it's less of a problem in practice. You just don't ask it questions that requires detailed knowledge of an API unless it's _extremely_ popular. Like in kubernetes terms, it's safe to ask it about a pod spec, less safe to ask it details about istio configuration and even less safe to ask it about some random operator with 50 stars on github.

Mostly it's good at structure and syntax, so I'll often find the library/spec I want, paste in the relevant documentation and ask it to write my function for me.

This may seem like a waste of time because once you've got the documentation you can just write the code yourself, but A: that takes 5 times as long and B: I think people underestimate how much general domain knowledge is buried in chatgpt so it's pretty good at inferring the details of what you're looking for or what you should have asked about.

In general, I think the more your interaction with chatgpt is framed as a dialogue and less as a 'fill in the blanks' exercise, the more you'll get out of it.

[neonsunset]

From within the process it might be difficult*, but please do give this a read https://learn.microsoft.com/en-us/dotnet/core/diagnostics/du... and dotnet-dump + dotnet-trace a try.

If you are still seeing the issue with memory and GC, you can submit it to https://github.com/dotnet/runtime/issues especially if you are doing something that is expected to just work(tm).

* difficult as in retrieving data detailed enough to trace individual allocations, otherwise `GC.GetGCMemoryInfo()` and adjacent methods can give you high-level overview. There are more advanced tools but I always had the option to either use remote debugging in Windows Server days and dotnet-dump and dotnet-trace for containerized applications to diagnose the issues, so haven't really explored what is needed for the more locked down environments.

[BurningFrog]

If I ever let it AI write code, I'd write serious tests for it.

Just like I do with my own code.

Both AI and I "hallucinate" sometimes, but with good tests you make things work.

[bredren]

This problem applies almost universally as far as I can tell.

If you are knowledgeable on a subject matter you're asking for help with, the LLM can be guided to value. This means you do have to throw out bad or flat out wrong output regularly.

This becomes a problem when you have no prior experience in a domain. For example reviewing legal contracts about a real estate transaction. If you aren't familiar enough with the workflow and details of steps you can't provide critique and follow-on guidance.

However, the response still stands before you, and it can be tempting to glom onto it.

This is not all that different from the current experience with search engines, though. Where if you're trying to get an answer to a question, you may wade through and even initially accept answers from websites that are completely wrong.

For example, products to apply to the foundation of an old basement. Some sites will recommend products that are not good at all, but do so because the content owners get associate compensation for it.

The difference is that LLM responses appear less biased (no associate links, no SEO keyword targeting), but are still wrong.

All that said, sometimes LLMs just crush it when details don't matter. For example, building a simple cross-platform pyqt-based application. Search engine results can not do this. Wheras, at least for rapid prototyping, GPT is very, very good.