[dormando]

1) Super workload dependent unfortunately. Probably "easiest" way is to set up test systems with enough RAM and disk space. Then either slowly reduce the RAM or start from a low point and slowly increase until you hit your target; there's a command for memcached to live-adjust the RAM limit.

2) dunno! For RAM that's super hard. for disk I'm hoping tiering will do something. ie; for on-peak you can add extra NBD space then remove it off-peak.

3) referenced tail latency because most of the testing focuses on displaying latency outliers and how the system generally minimizes them.

[peterwwillis]

I thought the RAM estimation was (1) benchmark your backend, (2) benchmark your memcache<->backend, (3) measure peak traffic performance requirements, (4) provide enough RAM that you reduce the latency to acceptable levels during peak traffic. If you don't have measurements of the first three you're shooting from the hip. It is workload dependent, but the more measurements you have, the better you can estimate. (Used to be mandatory so you could do trend analysis and purchase your new gear in time for inevitably outgrowing the old stuff)

[dormando]

Ah my point of reference was more "how do you go from memcached RAM to RAM + disk?" not "how do you go from no memcached to memcached".

If you don't have any cache layer at all, that's a standard approach.

[peterwwillis]

Wouldn't the RAM + disk just be an extension of that method, though? It's like in the article, you test a subset of the system (RAM+disk) to find its limits, then you compare that to your other measurements and load requirements, and replace RAM with RAM+disk where it's cost and performance appropriate. But I might have missed part of the analysis

[dormando]

The article was significantly cut down for brevity, so any detail of how to find the right balance is beyond the scope :)

The original question was that it can be really hard to determine how much adding RAM will offload from the disk. IE: with extstore recent objects are served by RAM and never disk, so they don't count against your IO balance. If you're coming from a RAM backed system that has 500k requests per second, it's going to take some creative introspection to figure out how much RAM you'd need to keep the disk accesses below 400k, or whatever your target is.

Along those lines I just suggested a simple experiment. which is to just take an existing fully specced instance, add a disk, and test a few RAM settings to see how it looks. Most actual users of extstore have just done that since they're extending/replacing an existing cluster.

There're actually other ways you can simply read the numbers out of memcached but it just takes more effort or familiarity with its internals.