[gstaro]

The way I understand this proposal, the idea is to find at which request rate the latency of a storage configuration (eg, 2x Optane) explodes. If your request rate is above that rate, get more RAM. I'm not sure I understand how to figure out how much more RAM? Until the latency goes down I guess?

Also what about diurnal patterns? Can there be a way to dynamically react and save power..

Interesting reference to the morning paper. But don't get how that fits into the story?

https://blog.acolyer.org/2018/10/26/robinhood-tail-latency-a...

[donavanm]

This is well explored space in more tradtional storage land. Look in to cumulative distribution functions. This is going to show you your workload distribution and effective “working set” to optimize for in a time period. This will get you to a place where youre optimizing total cost vs performance as a function.

Youll also want to be cognizant of convex “shoulders” in the distribution that will trip up naive optimization algorithms. I do t have the link offhand, but search for hill climbing in relation to CDFs. Some related work might be in the relatively unexplored “cache insertion” problem area. Check out TinyLFU as an example of knowing what to cache being more beneficial than what to evict.

For more advanced techniques look in to some of the published work from places like Coho Data. They had a great paper back at usenix 2015ish around optimizing placement in dynamic workloads across different storage media.

And lastly experimentation is great to prove a hypotheis, but not the most effective discovery. Youll want to get representative workload traces and use those to replay/simulate against different constraints. Check out Fio and its IO trace capabilities for an example.

[dormando]

1) Super workload dependent unfortunately. Probably "easiest" way is to set up test systems with enough RAM and disk space. Then either slowly reduce the RAM or start from a low point and slowly increase until you hit your target; there's a command for memcached to live-adjust the RAM limit.

2) dunno! For RAM that's super hard. for disk I'm hoping tiering will do something. ie; for on-peak you can add extra NBD space then remove it off-peak.

3) referenced tail latency because most of the testing focuses on displaying latency outliers and how the system generally minimizes them.

[peterwwillis]

I thought the RAM estimation was (1) benchmark your backend, (2) benchmark your memcache<->backend, (3) measure peak traffic performance requirements, (4) provide enough RAM that you reduce the latency to acceptable levels during peak traffic. If you don't have measurements of the first three you're shooting from the hip. It is workload dependent, but the more measurements you have, the better you can estimate. (Used to be mandatory so you could do trend analysis and purchase your new gear in time for inevitably outgrowing the old stuff)

[dormando]

Ah my point of reference was more "how do you go from memcached RAM to RAM + disk?" not "how do you go from no memcached to memcached".

If you don't have any cache layer at all, that's a standard approach.

[peterwwillis]

Wouldn't the RAM + disk just be an extension of that method, though? It's like in the article, you test a subset of the system (RAM+disk) to find its limits, then you compare that to your other measurements and load requirements, and replace RAM with RAM+disk where it's cost and performance appropriate. But I might have missed part of the analysis

[dormando]

The article was significantly cut down for brevity, so any detail of how to find the right balance is beyond the scope :)

The original question was that it can be really hard to determine how much adding RAM will offload from the disk. IE: with extstore recent objects are served by RAM and never disk, so they don't count against your IO balance. If you're coming from a RAM backed system that has 500k requests per second, it's going to take some creative introspection to figure out how much RAM you'd need to keep the disk accesses below 400k, or whatever your target is.

Along those lines I just suggested a simple experiment. which is to just take an existing fully specced instance, add a disk, and test a few RAM settings to see how it looks. Most actual users of extstore have just done that since they're extending/replacing an existing cluster.

There're actually other ways you can simply read the numbers out of memcached but it just takes more effort or familiarity with its internals.

[sitkack]

P99 latency at a constant read/write rate is exactly the test that Aerospike ACT benchmark performs.

https://github.com/aerospike/act

Aerospike has been engineered to perform well in this exact space. P99 <8ms over multiple TB of durable storage under a constant read/write rate.

[xenadu02]

I believe the objective is to replace expensive RAM with less expensive SSD if your workload is suitable for it. It appears there are workloads for which that is an acceptable cost/latency tradeoff.