|
|
|
|
|
|
by breakwaterlabs
1020 days ago
|
|
|
My recent discussions with multiple SAN vendors as well as quoting out cost to DIY storage has that number being far away from "reasonable". I do not claim storage is $5,000/TB but it is substantially higher than the $50/TB you're estimating. It's difficult to estimate the log throughput in this scenario. Cisco on debug all can overload the device's CPU; systems like sssd can generate MB of logs for a single login. All of this is really missing the core issue though. A 2PB system is nontrivial to procure, nontrivial to run, and if you want it to be of any use at all you're going to end up purchasing or implementing some kind of log aggregation system like Splunk. That incurs lifecycle costs like training and implementation, and then you get asked about retention and GDPR.... and in the process, lose sight of whether this thing you've made actually provides any business value. IT is not an ends in itself, and if these logs are unlikely to be used the question is less about dollars-per-developer-hour and more about preventing IT scope creep and the accumulation of cruft that can mature into technical debt. |
|
|
For the 20TB case, you can fit that on 1 to 4 drives. It's super cheap. Plus probably a backup hard drive but maybe you don't even need to back it up.
For the 2PB case, you probably want multiple search servers that have all the storage built in. There's definitely cost increases here, but I wouldn't focus too much on it, because that was more of a throwaway. Focus more on the 20TB version.
> That incurs lifecycle costs like training and implementation
Those don't relate much to the amount of storage.
> and then you get asked about retention and GDPR....
It's 90 days. Maybe you throw in a filter. It's not too difficult.
> if these logs are unlikely to be used
The devs are complaining about the search features, it sounds like the logs are being used.
> preventing IT scope creep and the accumulation of cruft that can mature into technical debt
Sure, that's reasonable. But that has nothing to do with the amount of storage.