Hacker News new | ask | show | jobs
by evanphx 4487 days ago
It's been discussed many times before, yes. Rubygems usage pattern by our users make any kind of mirror delay unacceptable. We currently run a number of mirrors, configured as caching proxies. I want to get us going on a CDN like Fastly soon because they provide effectively the same functionality but distributed to many, many more POPs that I will ever setup.
2 comments
knyt 4487 days ago
If it's just the sync delay, you could track each mirror's last-updated time and only direct users to a mirror that had synchronized with the master since the package in question was released. Otherwise, serve the content from AWS. Though I'm sure this couldn't beat the service that Fastly's donating.
link
evanphx 4487 days ago
The caching mirror configuration achieves nearly the same thing. In the past, people have wanted to run their own mirrors that we directed people to, but that's got reliability and security issues.
link
sitkack 4487 days ago
Mirrors shouldn't be a security concern, the signatures of packages should come from "headquarters", same goes for reliability, clients should be able to, and SHOULD pull from multiple sites simultaneously.
link
michaelt 4487 days ago
Even if package signing works perfectly, when I connect to a mirror and request a patch for foo, the mirror learns my IP address and the fact I have an as-yet-unpatched version of foo.
link
evanphx 4487 days ago
Very true on the signatures. Using multiple sites isn't necessary though, imho.
link
sitkack 4486 days ago
I could be wrong, but it seems like a nice hack to pull for say 3 mirrors at the same time at some offset into the resource using a range get for say, 16k each. The first one to complete does a pipelined request for another 16k slot and this process continues until the entire asset is downloaded. The fast mirrors would dominate, a small percentage of the bandwidth from slow mirrors would assist and truly slow mirrors would be ignored.
link
iloveponies 4486 days ago
I suspect mirror delay is less of an issue than you might perceive it to be. Many CPAN mirrors manage to stay within tens of seconds/no more than a minute from the main CPAN mirror that PAUSE publishes to.
link