Hacker News new | ask | show | jobs
by delecti 119 days ago
I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.
2 comments
marginalia_nu 119 days ago
Pretty sure every crawler can. You kinda have to go out of your way not to, given how the gzread API looks.

https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LS...

link
1e1a 119 days ago
Could allow only the path to the zip bomb for this user agent.
link
FartyMcFarter 119 days ago
That will work once at most and then quickly get fixed.
link
xp84 119 days ago
Yeah it seems like this team takes a really tough stance on obvious bugs
link
esseph 119 days ago
Are you so sure? :)
link