[kybernetikos]

I realise that it can occasionally be useful to have a canary that says 'something has changed' as long as its easy to reset. Personally, I'd rather such a canary operated on the visual representation plus interactions, since that is what the user experiences, rather than error on tree differences that produce no effective difference to the end user.

I do think though that neither of these should replace tests that test the actual postconditions of your code, and that the preponderance of false negatives and ease of resetting the canary discourages developers from spending the effort required to properly understand what is going on.

Almost all programming is about erecting firewalls to contain changes, so that as they ripple out from the change site, they hit boundaries beyond which things no longer need to be modified. As part of that, modules document what you can and cannot rely on. If you have a test that breaks because it's relying on some feature of a modules output that is not considered public behaviour, the bug is in your test, not the module that causes the failure.

> In the case you just described they correctly caught that something changed.

It is not the job of tests to tell you that something changed, but that something broke.

Perhaps I'd have less problem if people talked about them as canaries rather than as tests. 'Test' implies that there's something wrong if you fail.