While you can make this as complex as you want, even something as statistical abhorrent as simply measuring 1 run on a build server gives you a lot of bang for your buck.
The main idea is to have some way to keep the pulse and notice the worst regressions.
I usually like to set percentile thresholds: bad performance is much more likely to cause user frustration so I’d set something like p50/90/97th values for your supported scenarios (e.g. on a website, you might say “old phone on 3G”, “laptop on WiFi/cable”, and “desktop on fiber”) and treat blowing the top end budget as the most critical: fix in next release if you can’t meet your 97th percentile targets but otherwise every release should incrementally improve.
For network latency, you can set profiles in things like headless Chrome or use the Linux / macOS traffic shaping globally.
Percentile thresholds IMO is the way to go. You'll find things that don't cause problems at P50 to be problematic at higher percentiles, and the higher percentiles are usually where the user pain lies.
The main idea is to have some way to keep the pulse and notice the worst regressions.