[josephg]

> Right, but GC encourages you to not think about memory at all

I’ve come to a new obvious realisation with this sort of thing recently: if you care about some metric, make a test for it early and run it often.

If you care about correctness, grow unit tests and run them at least every commit.

If you care about performance, write a benchmark and run it often. You’ll start noticing what makes performance improve and regress, which over time improves your instincts. And you’ll start finding it upsetting when a small change drops performance by a few percent.

If you care about memory usage, do the same thing. Make a standard test suite and measure it regularly. Ideally write the test as early as possible in the development process. Doing things in a sloppy way will start feeling upsetting when it makes the metric get worse.

I find when I have a clear metric, it always feels great when I can make the numbers improve. And that in turn makes it really effortless bring my attention to performance work.

[tptacek]

Not so much. Here we have an example of a memory pressure problem that's evident only under high load in realistic environments. This is a classic problem with performance engineering: it's usually difficult to do realistic automated load testing. Instead, you end up running lab experiments, which are time-consuming to set up.

The whole post is essentially about how tricky it was to surface the problems their customers were seeing in the field. I'd resist the urge to respond to that with a platitude about automated testing.

[josephg]

Yes it can be difficult to do realistic automated load testing. But I suppose I see this as more evidence that if you're going to do load testing, do it right! In complex systems you often need real world usage data, or your metrics won't predict reality.

I've been running into this a lot writing software for collaborative editing. Randomly generated editing traces work fine for correctness testing. But doing performance testing with random traces is unrepresentative. The way people move their cursors around a text box while editing is idiosyncratic. Lots of optimizations make performance worse with random editing histories, but improve performance for real world data sets.