[teej]

That seems too important to have zero visibility on to me. Just eyeing the graphs, your queue size grew at 750m/s from 17:49 to 17:50. You then starting shedding at 17:50 for 40s. Assuming the ingress rate was roughly linear (which it looks like it was) you shed ~30,000 requests out of 3-4M. Does that not seem high to you?

This system seems great for at most once delivery. I wish I had more problems to solve with that constraint.

[Sikul]

Yep, such is life with new tech. GenStage was 0.3.0 when we started using it. We'll be able to get this visibility once we update to a newer version (hasn't been prioritized).

FWIW, the buffer only fills up to the peak about once a month. Load shedding is the last ditch effort in catastrophic situations so that everything doesn't fall apart.

[henrygrew]

I don't find load shedding as acceptable, perhaps if there was a criteria for dropping based on priority it would make more sense.

[Sikul]

I think it depends on your problem space.