[NewsaHackO]

>I dunno, it seems to be real excited about a VS Code extension that doesn't exist and isn't mentioned in the actual documentation. There's just too many factual errors to list.

There is a folder for a VS Code extension here[0]. It seems to have a README with installation instructions. There is also an extension.ts file, which seems to me to be at least the initial prototype for the extension. Did you forget that you started implementing this?

[0] https://github.com/blopker/codebook/blob/c141f349a10ba170424...

[curl-up]

This thread should end up in the hall of fame, right next to the Dropbox one.

From a fellow LLM-powered app builder, I wish you best of luck!

[raincole]

Yeah, this is a thread worth saving. Even just as an example of multiple people who can't read as well as an LLM.

[oblio]

Plot twist, OP has a doc mentioning it as unreleased.

[sceptic123]

In that folder is CHANGELOG.md[0] that indicates that this is unreleased. I'd say that including installation instructions for an unreleased version of the extension is exactly the issue that is being flagged.

[0] https://github.com/blopker/codebook/blob/main/vscode-extensi...

[NewsaHackO]

You are going to want to reread the file you are quoting buddy. That changelog is indicative that the extension has been released. The Unreleased section seems to list features that are not yet included in the released version of the VS Code extension, and the future plans are features that have not been developed yet.

[throwaway290]

here the maintainer says it doesn't exist. there's basically no way another interpretation is "more correct". presence or files can be not intended for use, deprecated, internal, WIP, etc. this is why we need maintainers.

[NewsaHackO]

Maintainers are not gods, and don't get to rewrite plainly true facts. In the Changelog, it actually says it is a "Initial release of Codebook VS Code extension".

[throwaway290]

compared to an llm they are an authoritative source...

[Phelinofist]

What a plot twist

[NewsaHackO]

It’s funny, I accidentally put a link to the commit instead of the current repo file because I was investigating whether or not he committed it versus he recently took over the project and didn’t realize the previous owner had started one. But he is the one who actually committed the code. I guess LLMs are so good now that they’re stopping developers from hallucinating about code they themselves wrote.

[blopker]

I brought up this issue because I thought it illustrated my previous points nicely.

Yes, there is a VS Code folder in that repo. However, it doesn't exist as an actual extension. It's an experiment that does not remotely work.

The LLM generated docs has confidently decided that not only does it exist, but it is the primary installation method.

This is wrong.

Edit: I've now had to go into the Readme of this extension to add a note to LLMs explicitly to not recommend it to users. I hate this.

[ninininino]

Is it possible that a random person who discovered your repo from Google search would make the same mistake the LLM did and assume it works and not realize it was an unfinished experiment?

[rng-concern]

Yes, and so the value of the persons opinions on the repo is low. Far lower than real documentation written by someone who knows more, that would not have made that mistake.

The value proposition here is that these llm docs would be useful, however in this case they were not.

[NewsaHackO]

>Far lower than real documentation written by someone who knows more, that would not have made that mistake.

But his own documentation did said that there was a VSCode extension, with installation instructions, a README, changelog, etc. From what he said, it doesn't even compile or remotely work. It would be extremely aggravating to attempt to build the project with the maintainer's own documentation, spend an hour trying to figure out what's wrong, and then contact the maintainer for him to say, "oh yeah, that documentation not correct, that doesn't even compile even though I said it did 2 months ago lol." It is extremely ironic that he is so gungho about DeepWiki getting this wrong.

[ninininino]

Yes, this is my point. It seems like the creator was a little bit lazy to create such a full fledged readme.md with so much polish but -entirely neglect to mention the whole thing is broken and unfinished-.

That seems about as annoying as a random wiki mis-explaining your system.

That being said, I am still biased towards empathizing with the library author since contributing to open source should be seen as being a great service already in and of itself, and I'd default to avoiding casting blame at an author for not doing things "perfectly" or whatever when they are already doing volunteer work/sharing code they could just keep private.

[blopker]

This.

The WIP code was committed with the expectation that very few people would see it because it was not linked anywhere in the main readme. It's a calculated risk, so that the code wouldn't get out of date with main. The risk changed when their LLM (wrongly) decided to elevate it to users before it was ready.

It's clear DeepWiki is just a sales funnel for Devin, so all of this is being done in bad faith anyway. I don't expect them to care much.

[NewsaHackO]

>That being said, I am still biased towards empathizing with the library author since contributing to open source should be seen as being a great service already in and of itself, and I'd default to avoiding casting blame at an author for not doing things "perfectly" or whatever when they are already doing volunteer work/sharing code they could just keep private

This is true, and the only reason for this was more so his dismissive view of DeepWiki than a criticism of the project itself or of the author as a programmer. LLMs hallucinate all the time, but there is usually a method to the way they do so. Particularly, for it to just say a repo had a VSCode extension portion with nothing pointing to it would not be typical at all for an LLM like DeepWiki.

[raincole]

Wow. Better advertisement for LLM in three comments than anything OpenAI could come up with.

[lionkor]

It might be internal, unfinished, a prototype, in testing and not yet for public use. It might exist but do something else.

This is not an ad for LLMs. If you think this is good, you should probably not ever touch code that humans interact with.