Hacker News new | ask | show | jobs
by underyx 4278 days ago
>Crawling through tor already obfuscates your identity to a certain degree, so we don’t really have to do anything other than cycling User-Agent strings to look different from any other client.

Why is that necessary, though? If Silk Road has any checks in place that protects against scraping, why are those in place?
2 comments
Havoc 4278 days ago
User Agents have proven to be (somewhat) uniquely identifiable under specific conditions so it seems like a reasonable precaution regardless of Silk Road configurations.
link
underyx 4278 days ago
He can specify his own User-Agent string. If he sends the user agent 'silkroadparserbot v0.0.1' with every request, that is going to provide no information at all.

Even if this wasn't, did you think that after one change of UA he would be getting more 'uniquely identifiable' with each new request sent?

link
Havoc 4276 days ago
You're missing my point. Have a look at what the EFF has to say (with cool test too):

https://panopticlick.eff.org/

link
underyx 4274 days ago
I did know about this before, and I suspected that you were referring to this earlier. None of this is applicable to a scraper.
link
xj9 4278 days ago
Shouldn't he just identify as some recent version of Tor Browser?
link