Hacker News new | ask | show | jobs
by ncmncm 1697 days ago
It would be helpful to be able to present diffs of log output between successful and failing runs of a test.

This is tricky to implement, for several reasons.

Log output is normally timestamped, making every line unique. Those parts of log lines would need to be ignored when comparing between runs.

Log output ordering is often indeterminate, particularly when a test has multiple threads, or interacts with an external service. Often the order of events logged is an essential feature of the difference between a successful and failed run. But some or most order differences are just incidental. The number of logged events may vary incidentally, or significantly. Explaining all these differences in detail to the test system would be too hard. So, the system needs to discover as much as possible of this for itself, and represent these discoveries symbolically. Then, allow a test to be annotated to override default judgments about the diagnostic significance of these features.