[graemep]

Wikipedia provides dumps. Probably cheaper and easier than crawling it. Given the size of Wikipedia it would be well worth a little extra code. it also avoids the risk of getting blocked, and is more reliable.

It suggest to me that people running AI crawlers are throwing resources at the problem with little thought.

[voidUpdate]

Maybe they just vibe-coded the crawlers and that's why they don't work very well or know the best way to do it

[ldng]

Maybe they should just ... not "vibe-code" at all then ?

[voidUpdate]

Sounds great to me

[milesrout]

We shouldn't use that term. "Vibe coding". Nope. Brainless coding. That is what it is. It's what beginners do: programming without understanding what they--or their programs--are doing. The additional use of a computer program that brainlessly generates brainless code to complement their own brainless code doesn't mean we should call what they are doing by a new name.