Hacker News new | ask | show | jobs
Show HN: Tiny Web Crawler (github.com)
2 points by l1feh4ck 2779 days ago
1 comments

It's not entirely correct to remove query parameters from the URL. Query parameters are part of the URL and can make a website show entirely different content.

The Googlebot for example considers these two URL's different pages:

    /mypage?name=name1
    /mypage?name=name2