[rsofaer]

The sort of tarpit you're talking about wouldn't even affect this link validator. You really think Stack Exchange should have given up on validating links because Github's robots.txt has:

User-agent: *

Disallow: /

in it?

[lazugod]

They could ask for Github's permission.

[JasonPunyon]

Yeah, we did. http://meta.stackoverflow.com/a/132677/6212