[fest]

> how have you come to this conclusion / explanation?

Through personal suffering while working on firmware for UAVs (PX4/Ardupilot), CNC controllers (grbl, Marlin), inverted pendulum balancing exercises, RC car control etc.

Sadly, no resources. I guess the best advice I could give on this subject is: increase the visibility of internal state of the system. I have lost track of number of times I've seen people trying to tune PID controller just by outside observation (hmm, looks a bit better than last time) instead of real-time plots of measured process error.

Add high-resolution logging, look at the plots of error and P/I/D terms, have a simulation where you can replay the sensor values and observe the system output.

[jackhack]

You've just summarized the difference between hacking and engineering -- data!

Without data we're just somebody with an opinion. It should be the root of our decisions.