[metadat]

While you are technically correct, these sorts of tarpit cycle patterns should be detected and shed by any half-decent crawling system ("half-decent" is said tongue-in-cheek :p).

Does marginalia struggle or fail to identify and sidestep indexing for these sorts of structures?

[marginalia_nu]

I don't mean to suppose these structures are actually common, I'm using them to illustrate how the concept of the internet having a size falls apart when you start considering the practical matters of counting how big the internet is. It hinges on a model of files-on-a-server that isn't how websites have worked in the last 30-or-so years.

Counting websites, or even delineating where a website starts or ends is difficult, as you can on the one hand have a single server hosting infinite websites like I described. Services like cloudflare also throw a spanner in the works, if you think maybe using server IP would help. Domain name isn't much use either, as that would discount hosting services like neocities.

There's a similar fractal of weird cases with counting documents on a given webserver (and by the extension the internet).