[touisteur]

I've always wondered how people actually get the profiles for Profile-Guided-Optimization. Unit tests probably won't actuate high-performance paths. You'd need a set of performance-stress tests. Is there a write-up on how everyone does it in the wild ?

[mhh__]

You might be surprised how much speedup you can get from (say) just running a test suite as PGO samples. If I had to guess this is probably because compilers spend a lot of time optimising cold paths which they otherwise would have no information about

[CyberDildonics]

That's not how it works. BOLT is mainly about figuring out the most likely instructions that will run after branches and putting them close together in the binary. Unlikely instructions like error and exception paths can be put at the end of the binary. Putting the most used instructions close together leverages prefetching and cache so that unused instructions aren't what is being prefetched and cached.

In short it is better memory access patterns for instructions.

[foota]

I suspect you know this based on the detail in your comment and just missed it, but parent is talking about FDO, not BOLT.

[mhh__]

Yes, but I'm not talking about BOLT

[pgaddict]

Yeah, getting the profile is obviously a very important step. Because if it wasn't, why collect the profile at all? We could just do "regular" LTO.

I'm not sure there's one correct way to collect the profile, though. ISTM we could either (a) collect one very "general" profile, to optimize for arbitrary workload, or (b) profile a single isolated workload, and optimize for it. In the blog I tried to do (b) first, and then merged the various profiles to do (a). But it's far from perfect, I think.

But even with the very "rough" profile from "make installcheck" (which is the basic set of regression tests), is still helps a lot. Which is nice. I agree it's probably because even that basic profile is sufficient for identifying the hot/cold paths.

[foota]

I think you have to be a bit careful here, since if the profiles are too different from what you'll actually see in production, you can end up regressing performance instead of improving it. E.g., imagine you use one kind of compression in test and another in production, and the FDO decides that your production compression code doesn't need optimization at all.

If you set up continuous profiling though (which you can use to get flamegraphs for production) you can use that same dataset for FDO.

[pgaddict]

Yeah, I was worried using the "wrong" profile might result in regressions. But I haven't really seen that in my tests, even when using profiles from quite different workloads (like OLTP vs. analytics, different TPC-H queries, etc.). So I guess most optimizations are fairly generic, etc.

[mhh__]

There are some projects (not sure if available to use in anger) to generate PGO data use using AI.

[touisteur]

I was working some years ago on 'happy path fuzzing', trying to find heuristics to guide through code avoiding all error handling, runtime checks. Never got better results than afl-go or other targeted fuzzing, but you have to know what's your happy path.

Also tried to use previous-version or previous-previous-version coverage ('precise' through gcov, or intel processor trace, or sampled perf traces, down until poor-man's-profiler samples) coupled with program repair tools, and... never managed to jump from fun small toy examples to actual 100+kloc applications. Maybe one day.

[still_grokking]

AI can predict how some code behaves when run?

So AI can predict whether some program halts?

Seriously?

[mhh__]

https://www.researchgate.net/publication/357418662_Profile_G...

What exactly do you think PGO data looks like? The main utility is knowing that (say) your error handling code is cold and your loops are hot, which compilers currently (and so on).

This is indeed unknowable in general but clearly pretty guessable in practice.

[MonkeyClub]

Well spotted! :)

[mhh__]

Barely

[its_bbq]

If I remember correctly, at Google we would run a sampling profiler on some processes in prod to create these profiles, with some mechanism for additional manual overrides

[tdullien]

Google and Meta do in-production profiling. I think that tech is coming to everyone else slowly.