Hacker News new | ask | show | jobs
by osi 3469 days ago
Where is the data kept?

The very large heaps are partially driven by distributed in-memory storage systems.
1 comments
twic 3469 days ago
Hmm, well if the processing is microservices, presumably the data is in a NoSQL database.

More seriously, though, isn't most large in-memory data storage done off the garbage collected heap? I have to confess i'm actually not sure what anyone is doing with >20 GB of normal objects.

link
sheeshkebab 3469 days ago
Search (elastic/solr), Cassandra, presto, and most other java backend storage and data processing engines need 20+Gb (at least) to be mildly useful. It's not uncommong to go with 64gb+ heaps there too.

Microservices/API won't help with lowering these.

link
osi 3469 days ago
yes, exactly this. and by having scalable GC algos so you don't need to do "off heap" tricks in java, things are much easier.
link
dkersten 3469 days ago
I'm sure people do use huge heaps, but most web applications I know of (microservice or not) store the bulk of their data in a (separate) database. So, unless your application is a database, which might want a huge heap so it can cache a lot in-memory, I don't think huge heaps are really necessary. IMHO, at least.
link
dom0 3469 days ago
Since it's not really possible to do reliable giant user space caches in the POSIX model of things you'd usually delegate to the page cache.
link