Hacker News new | ask | show | jobs
by dsacco 3241 days ago
I have Verizon Fios set up as a business account to my home. However that bandwidth is for SAN to workstation data processing and analysis. There's only half a gigabit of external internet bandwidth, but I have hundreds of terabytes of data (and high tens of gigabytes are downloaded per day) in local storage.

With 128GB of RAM I can only load small amounts into memory for targeted analysis. Processing the rest of the data requires loading it directly from storage. To improve I/O performance I parallelize the transfer across link aggregated ethernet interfaces. Naturally that would cause disk reads to become a bottleneck; to take proper advantage of the network transfer speeds I hold all data in RAID 0 with 7200 RPM drives.

1 comments

if your IPs are in a contiguous /25 , they can all be blocked with a single firewall rule. And detection is also easy. It only helps with rate limits.

To avoid a block, you need a list of good socks5 proxies

> if your IPs are in a contiguous /25

They're not.

> To avoid a block, you need a list of good socks5 proxies

No; furthermore, that leaks your data to a third party and introduces significant latency.