[dmurray]

I don't know why this got downvoted; it's at the very least an interesting proposal. Would love to hear a critique arguing that it's a terrible idea.

Existing package managers could even implement it in a completely backwards- compatible way: if you as a package maintainer don't care for it, you simply never add "API tests".

[chacham15]

Its an idealistic view which will almost certainly fail. Test suites, as much as we like to hope that they reflect real usage, mostly dont. A simple example: if function a gets changed from o(n) to o(n^2) but otherwise behaves identically, most test suites will still pass, but if a user has that function in its own inner loop you can go from o(n^2) to o(n^2^2) which can definitely break a lot of things (simple example: transaction was holding lock for too long and so the transaction was aborted). Being able to catch the above is a high bar for a test suite which I'm fairly confident most test suites are way below that.

[samatman]

You're allowed to fix bugs introduced by new releases, you know. It's called a patch release?

Is your claim that a release which introduces bad algorithmic complexity requires a major release to fix in semver? Who thinks this?

[pgphn]

Right. Theres a difference between, “oh no, sorry, WE should fix that in the next patch” vs, “oh no, sorry, YOU should accommodate this incompatibility with a change on your side.”

[falserum]

I would not call it terrible, but I got a “silver bullet” vibe, which it is definitely not.

1. For a library there is API and there are implementation details. What if test depended on implementation detail?

2. What if tests had undisputable bug?

3. Test refactoring requires major release now?

4. Realistically test suite will have some execution paths not covered.

I like the idea of running same tests over multiple versions, to observe changes. But I disagree that it would automate semver. (Maybe in very limited subset of cases)

P.S. Not an actual downvoter, but if I would have downvoted, these would have been the reasons.

[samatman]

> For a library there is API and there are implementation details. What if test depended on implementation detail?

If the test is in the API testset, it's API. If it isn't, it's an implementation detail.

> What if tests had undisputable bug?

If it's in the API testset, time for a major version bump. If not, fix it.

> Test refactoring requires major release now?

Only if you're refactoring the API, as defined by the API testset, thereby producing a breaking change.

> Realistically test suite will have some execution paths not covered.

Doesn't matter. If the behavior isn't in the API testset, it's not a part of the API.

> But I disagree that it would automate semver.

The point isn't to automate semver, I'm not even sure what that would mean. It's to define it, in a useful and objective way.

[dxdm]

The point of the criticism is that defining the version and what constitutes a breaking change like this will still leave people with unexpected breakage in the real world. What you've said so far has not really addressed that point straight on, which might be the reason for the comments and downvotes, I presume.

I don't think your proposed scheme needs to be perfect in that regard, but acknowledging the concern and at least putting it in perspective would probably help.

[samatman]

I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯

SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth.

What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point may have been missed: it isn't every test in your test suite, it's the ones marked "API", only.

This is a strict improvement over social-contract SemVer in two ways: one is that the package manager won't let the maintainers break the API tests without a major version bump. The other is that, if you, as a user, are unsure if some behavior is part of the stable API, you can write and submit a test to that package. If that test is accepted, great: that behavior now cannot change without a major version bump, because, again, the package manager will not bundle the package if that test breaks. Furthermore, even on a major version bump, it is instantly clear if that test is still valid, or not, you can just check before upgrading. If they don't accept the PR, you know that it isn't considered part of the API, so you add the test to your own test suite, so that at least you know quickly what broke if they change it.

[dxdm]

> I've no idea what downvotes you're referring to, I'm well into the black on that post.

As you should be, it's a great contribution. I was referring to the downvotes mentioned in a comment further up.

I agree that what you propose is an improvement, but it can be misunderstood to claim that it can prevent _any_ real-world breakage. There will always be aspects not covered by tests and which other people still rely on.

I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution.

You can argue that your solution significantly shrinks this area of uncertainty while also _defining_ it, which helps when reasoning about what you can depend on - and I agree. But it does not eliminate the gap, and this is what people were pointing out.

I was just a little frustrated that the discussion even went there, because I didn't think you were even claiming what they were arguing against. That was happening because I think you left a gap by not addressing it clearly, and wanted to point it out, because people seemed to be taking past each other.

[samatman]

> I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution.

It's only intended as an improvement to practice. SQLite has a test suite which exhaustively tests every single branch of the code, which D. Richard Hipp wrote on contract to one of his clients. It takes more than a day to run. We might all aspire to such a level of professionalism, but realistically, most programs and libraries will fall short of glory here. And while this exceptional test harness has in fact limited the scope of SQLite bugs a great deal, it hasn't eliminated them entirely.

So with a TestVer system, or whatever we want to call it, there will still be breaking updates, there will still be bugs. But it provides a mechanism for defining the invariant behaviors of a major release number.

It's possible some of the early respondents thought that I considered an appropriate response to a minor change which breaks downstream code to be "lol, too bad". That might happen sometimes, minus the lol we may hope, but more often the response should be more proactive: a revert, adding a test which clarifies the new behavior, something.

The best part of this system is that users of a package can write additional tests of that package and submit them as PRs, if there's some behavior they see the package exhibiting which doesn't appear to be in the test suite. This is easier by far than making changes to the package itself, just add the tests to your own suite, if they pass, make a fork of the repo, add those test to their API suite, submit a PR. Whether they accept the patch or not, you have it in your own test suite, so you'll be informed immediately if a later release breaks it.

[ambicapter]

It looks to me like he neatly addressed all concerns that were brought up, even if one does not agree with the solutions he proposes. I don't see any lack of acknowledgment on OP's part.