Show HN: I have written a cloud native dark web crawler in Go

Y	Hacker News new \| ask \| show \| jobs

	Show HN: I have written a cloud native dark web crawler in Go (github.com)
	7 points by creekorful 1996 days ago

3 comments

Can you share some examples of what it finds? As long as it is safe to post that.

I have an instance running for about 5 days+ with 1,300,000 docs. It's mostly huge amount of porns, some weird forums & lot of marketplaces.

There's still some good finds like a 'sysadmin' forum with huge amount of good posts & tips.

I'll certainly publish an index of interesting (legal) hostnames on my GitHub account.

Release 1.0.0 will be out very soon. Check it out guys!

nice, what is actually needed to crawl onion sites ?

i am not a go dev, would try it with scala/java

Just a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.

cool i look into that some time.