An integration test that creates dummy accounts using every method including SSO and then attempts to bruteforce the password should find “12345678” within an hour.
I think a test like this would also have found the dropbox and macos bugs that let you login to any account by using an empty password:
Edit: Oh, the password was "123qweQWE!@#000000000". Technically doable with an efficient password cracker that favors common patterns. zxcvbn’s entropy estimate says it will take 10^10.5 guesses. That’s 1 week at 50k/s. That’s a hell of an integration test for most software.
Which then opens you up to exactly that class of bugs that also caused this issue. Having test specific code and feature flags and then testing a tweaked version isn't really covering all the cases then.
Just like in this case where a hardcoded password was set to maybe log in through a test based on the naming "test_default".
But only if there's no rate limiting or increasing timeouts for wrong passwords which in most cases exists.