I fiddled around a little with the idea of test-driven versioning a while back. Maybe you'd find it interesting. https://github.com/abathur/tdver
I did draft a git-based implementation (https://github.com/abathur/tdverpy), but it just obviously can't be as compelling as one that was part of a language's native tooling/ecosystem could be.
This is quite similar to what I have in mind, yes. Great minds think alike!
I do think that having an API subset of tests is better than basing the system on all tests. Packages should have as many tests as possible, I frequently write tests which I know will break when I do further work on the code, so that I notice when it happens, and because if it happens accidentally it's probably a bug. Wouldn't want a versioning system to have a side effect of making people reluctant to write a test, because it would commit them to the results. I envision tests migrating from the rest of the suite to the API set over time.
I do like that your system completely specifies the meaning of minor and patch numbers, and wonder if there's a way to tweak my proposal so that it does so as well.
I wouldn't try to insist anything I sketched out is the ~right approach. I was just trying to imagine one path through the possibility space, and then reason a bit about what kinds of information it might be able to convey.
There are almost certainly better paths through, and I suspect the idea isn't broadly usable without different kinds of tests that have different kinds of rules. It's probably not helpful to have to increment your major version just because you use snapshot testing and some dependency update causes a trivial shift in the output.
I also fiddled around a little with an idea I called "earmarks", which are basically just version-bound tests. You could use these to express the idea that, say, test_x shouldn't pass or fail until the version is >= a.b.c.
This would make it easy to deprecate an API today and go ahead and ship a test that requires the API to be present and functioning until the next major release but not after. Or, for example, to make a commitment device that asserts the project will hit some doc/lint/typing goals by some clear point.
Since it's an open-ended mechanism, I imagine something like it is the lower-friction way for a real project to explore applying these concepts without full toolchain support.
I do think that having an API subset of tests is better than basing the system on all tests. Packages should have as many tests as possible, I frequently write tests which I know will break when I do further work on the code, so that I notice when it happens, and because if it happens accidentally it's probably a bug. Wouldn't want a versioning system to have a side effect of making people reluctant to write a test, because it would commit them to the results. I envision tests migrating from the rest of the suite to the API set over time.
I do like that your system completely specifies the meaning of minor and patch numbers, and wonder if there's a way to tweak my proposal so that it does so as well.