[0xbadcafebee]

This article is spot on. For $LargeNetworkHardwareVendor we maintained three different automation test frameworks for end-to-end testing. Our tests were more abstract functions that were given arguments for a particular test case. Those were then made into collections of tests that could be re-used. A configuration file allowed QE to build new test cases without programming knowledge. QE would write configs and occasionally one or two of them that could code would modify the test framework. All the tests ran in a scheduler from clusters of test-running manager-servers against globally distributed labs of hardware. While teams did have unit tests and functional tests, the end-to-end test was king (and necessary given the multiple levels of interface for that gear)

A lot of reliability in that system came from being able to quickly iterate on different levels of the system. The easier it was to solve a failure where it's happening, the more likely your bugs can be fixed quickly, so you have a healthy system (as opposed to suffering from entropy and tech debt)