[jahsome]

I asked it to generate me documentation for a handful Bicep definitions and include links to the official Microsoft Docs. It generated about 50% erroneous links, but once I updated my prompt and fed it the "root" of the MS docs I wanted it to use, it was able to correct all the urls.

At the time it _seemed_ like it wasn't just doing a mindless string replace, because different parts of the url changed on most of the erroneous links. I figured it wasn't going out and spidering the link I gave, but it led me to believe maybe there was _some_ form of index that it simply hadn't prioritized prior, but was able to do with the additional context.

[jimbokun]

Wouldn’t it be wild if it just guessed the URLs correctly, because it had seen so many URLs from Microsoft URLs it was able to correctly predict the form of URLs it hadn’t seen yet?

[colechristensen]

These models have probably been trained on a large portion of the Internet.

The training data however can be a few years old.

I have noticed this when trying to get it to generate some code, it was clearly using considerably old versions.

[codebje]

> I have noticed this when trying to get it to generate some code, it was clearly using considerably old versions.

I have noticed the same thing when trying to use junior devs to generate some code, usually correlating with how old the most popular Stack Overflow question on the subject is.

[colechristensen]

This is how we decided to describe chatgpt’s programming ability the other day. A very fast but somewhat inadequate jr developer. Is amazing really but you also have to ask for the same thing a dozen times and still spend quite a while figuring out what’s wrong with it.

[Godel_unicode]

What I have been doing is asking it for code and then asking it to find bugs in that code. The bug finding works relatively well.