Hacker News new | ask | show | jobs
by greglindahl 2983 days ago
Er, that's not what I'm saying.

I'm saying that if you get a response from a webserver, you have to think about it being truncated. Period. No matter what the libraries you're using do for the many different cases of problem that can cause truncation.

You seem to have read a lot into my words that I didn't intend to put there.
1 comments
dagenix 2983 days ago
I fully agree that there can always be garbage coming back and that any robust client needs to be aware of that happening. I'm sorry if I read too much into what you were saying, but, it sounded like you were arguing that the change shouldn't be made because there are other ways that corruption could occur that this change wouldn't solve. But, it sounds like you were just warning that this change wouldn't be a panacea? Is that accurate?
link
greglindahl 2983 days ago
I didn't want to express any opinion on the change; I use aiohttp for my python-powered web-scale crawling needs. So yes, it's accurate that I was warning that the change would be incremental and not a panacea.

BTW aiohttp has the problem of currently having a too-strict http protocol parser, so it throws errors for many rare cases of bad webservers, which browsers don't have a problem with. As a crawler, I need to be able to work with whatever a browser will display, ...

link
dagenix 2983 days ago
Well, it sounds like we are in fierce agreement then. Sorry for misunderstanding your comment earlier.

I've been interesting lately in exploring at aiohttp, and your comment about it being too strict is certainly very enlightening. And I want to second your comment about libraries following the lead of what browsers will do. My strong feeling is that the battle over what HTTP libraries should do has been fought and been decided by browsers, and that whatever browsers do is what HTTP libraries should do as well, regardless of how ugly it is (IIRC, when I last checked, browsers did pay attention to the Content-Length header and wouldn't display results that were shorter than it - but if I'm misremembering, I would happily change my position with respect to honoring this header). The purist in me hates to say that, but, the pragmatist wants to get things done and fighting against browser behavior feels counter-productive at this stage.

link