| I have to dealt with debugging flaky tests before on embedded Linux app with multithreads, GUI, audio, video streaming before. I debugged those bugs with custom gdb scripts. The test system will run the gdb scripts to start the app. The gdb script setup breakpoints, log stack trace and local variables automatically for a particular bugs. The debugging environment is scripted and automated. Test failure mark/track the log file with all the gdb info. I schedule the tests to run continuously overnight and in the morning I analyze the log file with gdb info. After that I can add more debug command in gdb script or try fixed the bug if I have enough info. The cycle continue until the bugs were all fixed. I personally think it was very good for try to analyze / fixing the "flaky" tests issue. Even for the the tests that fails 1 in 1000 times, the bugs can still be debug and fixable. It was actually more relaxing for me to debug this kind of problem. 99% of time, the computer do all the works. You guys might want to try it if not done in Google already. |