[aseipp]

Trial and error. You need to dial in the right point on the voltage/clock frequency curve for your workloads, AKA "just play some games and look at the results." Just use whatever your overclocking software for your motherboard is, and modify the default curve it has. I use MSI Afterburner and just set a flat clock frequency (plateau) at a certain voltage level to undervolt. I think for NVidia GPUs there's a way to modify the curve with the default tooling, but third party tools like Afterburner can also do it.

You can get great results pretty fast this way. My Mini-ITX build is about as thermally compact as possible given the parts (3080+Ryzen 5600X, NZXT H1), and I'm pushing my PSU to the absolute limits in the stock settings, so undervolting is important for safe power margins since the 3080 can reach ~360W in my testing. I think 30 minutes of tweaking got me something like a +80W power drop for only 10% FPS in Read Dead Redemption 2 @ 4k60fps; I never breach 300W now which is within my personal safety margins, and can native 4k everything.

Some software like Afterburner have "Overclock Scanner" tools that will run benchmarks and repeatedly try to dial these settings in for you, but it really is easier to just modify the curve manually and test your specific workloads.