[inkyoto]

In Linux, you use cgroups to confine a process to a fixed memory limit that won't cause an OOM in adjacent processes. You can create a separate memory cgroup with a different allowance for each customer depending how much memory is allotted to them per the contract, for example.

You can also use ulimit with a hard memory limit, but cgroups are more fine grained and flexible.

Also do note that the point about the read-only mmap still holds – it can't cause an OOM, it is mmap'd pages pointing to dirty pages that have not been written out to the file system yet that can – on the condition their number is growing uncontrollably and/or fast.

P.S. But allowing a database to cause an OOM is never a good idea to start off with whether memory limits have been set or not – the data loss is unacceptable for a database, and databases have been optimised to operate within pre-configured space constraints that do not even require ulimit/cgroup in the first place. Perhaps fixing the database configuration is a better starting point rather than going down the rabbit hole with mmap(2).

[BohuTANG]

Thank you for the insight.

We manage memory directly within the DBMS, for instance, by setting MySQL's innodb_buffer_pool_size to 1GB. This precise control method ensures that each cloud-based DBMS instance on our large EC2 instances remains within its memory limit without adversely affecting others.

When the internal memory pool limit of the DBMS is reached, we understand that it will spill dirty pages to disk and utilize the write-ahead log to guarantee ACID properties in the event of a crash. This process is predictable and manageable. In contrast, the effects on the DBMS when cgroup memory limits are reached are less clear, introducing uncertainty into system behavior under memory pressure. This unpredictability strongly supports our preference for managing memory within the DBMS itself.

Our experience over a decade with cloud DBMS products has shown that mmap doesn't suit our needs. We prioritize direct memory management methods within the DBMS to ensure system reliability and optimal performance.

[hyc_symas]

What you call optimal performance with InnoDB, I call pathetically slow. http://www.lmdb.tech/bench/memcache/

And if you'd read the reliability references linked in the article, you'd see that MariaDB/InnoDB's reliability is the worst of many tested systems. LMDB's is flawless.

https://lists.openldap.org/hyperkitty/list/openldap-devel@op...

The one fault they found for LMDB, caused by fdatasync not updating the DB filesize, was actually a kernel/filesystem bug (fdatasync is documented to update the filesize) that had already been fixed two years before these guys did their research. Not an LMDB flaw.

[BohuTANG]

Please don't misunderstand me; I've also implemented tokudb_buffer_pool_size for TokuDB because their control over memory is excellent, not falling into the black hole of the page cache. This allows my DBMS to be stable enough. This is also my point in this post, and my discussion ends here.