| > I've no idea what downvotes you're referring to, I'm well into the black on that post. As you should be, it's a great contribution. I was referring to the downvotes mentioned in a comment further up. I agree that what you propose is an improvement, but it can be misunderstood to claim that it can prevent _any_ real-world breakage. There will always be aspects not covered by tests and which other people still rely on. I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution. You can argue that your solution significantly shrinks this area of uncertainty while also _defining_ it, which helps when reasoning about what you can depend on - and I agree. But it does not eliminate the gap, and this is what people were pointing out. I was just a little frustrated that the discussion even went there, because I didn't think you were even claiming what they were arguing against. That was happening because I think you left a gap by not addressing it clearly, and wanted to point it out, because people seemed to be taking past each other. |
It's only intended as an improvement to practice. SQLite has a test suite which exhaustively tests every single branch of the code, which D. Richard Hipp wrote on contract to one of his clients. It takes more than a day to run. We might all aspire to such a level of professionalism, but realistically, most programs and libraries will fall short of glory here. And while this exceptional test harness has in fact limited the scope of SQLite bugs a great deal, it hasn't eliminated them entirely.
So with a TestVer system, or whatever we want to call it, there will still be breaking updates, there will still be bugs. But it provides a mechanism for defining the invariant behaviors of a major release number.
It's possible some of the early respondents thought that I considered an appropriate response to a minor change which breaks downstream code to be "lol, too bad". That might happen sometimes, minus the lol we may hope, but more often the response should be more proactive: a revert, adding a test which clarifies the new behavior, something.
The best part of this system is that users of a package can write additional tests of that package and submit them as PRs, if there's some behavior they see the package exhibiting which doesn't appear to be in the test suite. This is easier by far than making changes to the package itself, just add the tests to your own suite, if they pass, make a fork of the repo, add those test to their API suite, submit a PR. Whether they accept the patch or not, you have it in your own test suite, so you'll be informed immediately if a later release breaks it.