[ng-henry]

I scraped my favorite blogs and made a graph from the domains that each blog links to.

You can see clusters forming of websites that talk about similar topics, like crypto, rationality, Canada, India, and even postgres!

The visualization was made entirely in webgl with some neat optimizations to render that many lines and circles.

[jseliger]

This is very cool but also not accurate, at least for jakeseliger.com. Henryn.ca lists 0 links from jakeseliger.com to nytimes.com, reason.com, and numerous others that simple search demonstrates are linked to, for example: https://jakeseliger.com/?s=nytimes.com&submit=Search

I put up many links posts, so I probably link to an abnormally large number of sites.

[ng-henry]

Yep this is only for stuff that we've crawled, so we can't detect all of your links. Because we have limited crawling resources, we rate-limit the crawling by domain so we don't get stuck in spider traps.

The current visualization only shows the current state of the crawl, so it won't know about all of the posts.

[TuringNYC]

> I scraped my favorite blogs and made a graph from the domains that each blog links to.

Nice analysis! However, I'm guessing these arent your fav blogs as there are tens of thousands of entries! How did you decide which blogs to index, did you use some central registry of blogs?

[dredmorbius]

<https://news.ycombinator.com/item?id=40137135>

[nickjj]

Thanks a lot for including my site in your list. It was fun to see where it appeared on the map. It was pretty close to RealPython and GitHub.

[dameyawn]

Very neat! So you wrote the graph visualization UI? I see in prior project you used cytoscape - any motivation for doing it yourself this time (vs one of the available libraries)?

[ng-henry]

Yeah I used cytoscape before but it didn't have the full customization that I wanted. Besides the performance issues, there were some problems I couldn't have solved without a custom renderer - if many lines overlap, how should their colors blend? - how to render circles so that they look nice both zoomed in / out - how to avoid it looking like a hairball graph [1]

The nice thing about a personal project is that I can do whatever I like with no constraints, so I built one that's suited for this project and fits my tastes.

[1] https://cambridge-intelligence.com/how-to-fix-hairballs/

[varenc]

This is a really cool project! I'd love to hear more about how your built the front end.

[gala8y]

serendipity heaven, but... how is this map of _your favorite_ blogs?

[ng-henry]

I started off with my favorite blogs and recursively explored from there based on what they linked to.

[gala8y]

ok, got it.