Hacker News new | ask | show | jobs
by thr0wnawaytod4y 1254 days ago
> there is no need to spider/crawl if there are sitemap XML files listing all the content URLs

you never know if these are all URLs tho, so I would always crawl and brute force more paths than trusting sitemap.xml

1 comments

Is that what Bing does. Doubtful.