Hacker News new | ask | show | jobs
by j16sdiz 155 days ago
The worse thing is calendar/schedule. Many crawler tries to load every single day, with day view, week view and month view. Those pages are dynamically generated and virtually limitless