[twawaaay]

You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes.

As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I can add a delay to each response and I can have even something as simple as PID controller regulate the CPU usage by changing the dalay. Larger delay will usually cause the clients to slow down requests (requests being usually a result of previous request completing). This is simple and naive example but this is more or less what I do.

(Of course, in reality, it is much better to just have a backpressure mechanism and whenever possible you should use one rather than try to work around HTTP inadequacy. But you can't always do it, especially if you have a public API.)

I also typically have lots of other controllers. For example something that regulates memory usage by limiting transactions in flight or something that regulates latency as seen by priority clients or database replication rate/delay, or error rates or a bunch of other parameters.

I also routinely take care of babysitting downstream systems like databases or other APIs. I may have a regulator that will automatically start backing off certain types of traffic as a response to increasing error rates or latencies in a downstream system. All this because those downstream systems are usually shit and not designed to deal with overload and it is easier for me to deal with this proactively than do what everybody else does -- keep bugging those people to fix their issues when their evidently don't know how.

[bluepizza]

This is extremely interesting!

I have been trying to move away from dumb rate limiting to a more holistic approach that allows us to make smarter decisions with traffic. Your overview made me intrigued.

Do you have any references you like to use? I am looking at the Wikipedia page, but it's so removed from practical aspects.

[twawaaay]

I don't. There simply isn't any tooling or literature to speak of. I have some experience using control engineering in my electronics projects and that's how I came up with the idea to use it for backend systems. I have researched and developed everything myself. I have used "Modern Control Engineering" by Katsuhiko Ogata, but really, mostly I just learned from the Internet.

My initial motivation was to remove configuration. I have found, historically, that giving people options to configure very complex software more often than not results in problems, especially after original developers leave. More often than not these new people will not understand the implications or interactions between various settings and this will just cause problems. So my aim became to remove any options from the software and make sure it can perform autonomously and recover from wide range of, possibly unknown, situations. Which is exactly what control engineering is about if you think about it!

[KajMagnus]

If one day you'll write a blog post / article about what you're doing, it'd be interesting to read :- )

(What if you start collecting email addresses to people who want to read such an article? And if one day you write one, then you can email them? — My email is in my profile, if you'd like to add it to such a list)

[twawaaay]

Dunno... most people I meet seem to be put off by my software development ideas. I stick to them because they seem to work very well even if it initially creates a lot of friction between me, the team and the management.

Where to start... I think test driven development and unit testing is not giving promised value and instead wastes time and makes software more difficult to refactor and I think functional end to end testing to be much more effective and cost effective. I think code reviews are bad because they don't deliver on promised value and individual craftsmanship (peoples ability to deliver on their own) and pair programming are better. I think microservices to be a wrong approach for 99.9% projects and fixed a bunch of projects by rolling the software into monolyths. I believe bugs can only be truly reduced by taking responsibility for writing correct code in the first place and anything afterwards is expensive and not effective (you can only remove bugs that manifest themselves, everything else stays). I don't compile/run my code multiple times a day -- I write it all in one go, sometimes for weeks, then run it. If it works it means I know what I am doing and if it doesn't -- it is the failure of my process. Where most devs just fix the bug and restart the app I will start an investigation into why my process failed and how I need to fix it -- NTSB-style. I believe that nobody understand what Agile is and the way it is applied is damaging to software industry. I don't believe in linear development progress -- I design my apps top down and at the same time program them bottom up until top down and bottom up meet together. I structure my development process around rewriting the software -- I write the first version and then I will refactor/rewrite to remove any unnecessary complexity until I am happy with it. There is no working software for a long time and then suddenly it is complete. And when it is complete there is no more testing stages, bugs to fix -- it is truly complete.

So you see, I am probably too alien a developer to give advice to general population of developers.

And when I do talk about my ideas it usually ends in flame wars or drowns being downvoted to hell because people tend to downvote anything and everything that does not confirm their existing worldviews.