[notacoward]

This really looks like an example of open source done right. Obviously there are some strong opinions, but the person suggesting the change was pretty gracious about the pushback they got. Since then, stakeholders have had a chance to discuss and agree on a way forward. Nobody is trying to sweep all the "nasty bits" under the rug, like most developers tend to, and there's even mention of regression tests. I've seen few other projects (including but not limited to those where I was a maintainer) handle possibly-disruptive change so well. Kudos.

[pmarreck]

> and there's even mention of regression tests

I've been doing (mostly) full-coverage unit and integration testing since, oh... 2005? At least in the Ruby on Rails and now Elixir/Phoenix development spaces, it's absolutely de rigeur, and has probably saved me countless hours of debugging and simply not breaking stuff that already worked, or validating that things worked the way I expected them to.

The fact that in 2022 someone even has to qualify regression testing with an "even" (as in "EVEN mention of regression testing!") saddens me. Tests reduce developer pain and increase developer productivity, full stop. If you get hit by a bus, someone else who is working on your code will know they didn't break anything thanks to your test suite. Get with the program, folks, it's been decades now since this was known.

[notacoward]

It saddens me too, but it still seems necessary. In my (quite long and varied) experience, most developers do not appreciate the value of regression tests. Therefore, reminders and positive reinforcement are still beneficial.

[R0b0t1]

It is possible to make useless tests.

[notacoward]

Does that seem like a strong argument to you? Of course it's possible to make useless tests. It's possible to be injured by an airbag. It's possible to overdose on a drug that usually saves lives. That doesn't mean any of these things aren't generally beneficial, or that they should be foregone.

As I said recently on Twitter, I've measured the likely cost of production bugs that were fixed early by static analysis, by leaving log messages to mark where the bug would have occurred. I know that's not the same as regression tests, but if anything it's even more of a long shot, more of a hard sell to my fellow developers, and even in that case the value was strongly positive. About a half million 2005 dollars in that case, for less than a fifth of that in license costs and my own time. The ROI for regression tests is likely to be in the same ballpark.

The fact that something can be done poorly by lazy people is in no way an argument against trying to do it well, or even semi-competently, by people who take their profession seriously.

[jhugo]

I think it's a big assumption that the ROI of tests in general is in the same ballpark as the ROI of static analysis.

In my experience, many tests written in commercial software engineering have ~zero or even negative ROI. This mostly applies to micro-level testing like unit tests; macro-level testing like integration tests can be fantastically valuable and I've even come to believe that they're the only type of tests most teams should spend time writing.

[pmarreck]

It depends on how you design your software and where the API surfaces are and how isolated the pieces can be made to be.

Which, if you're writing unit tests at the same time as the unit under test, leads naturally to pro-isolation, pro-modular designs, which are both easier to test, more reliable and generally have a more concise purpose.

[notacoward]

Well, we were talking about regression tests, which are much closer to integration tests than unit tests, so I'm not sure we actually disagree. Writing regression tests is still valuable, and still too rare.

[pmarreck]

Yeah, what exactly are you arguing, here? "assert 1=1" is a useless test, that doesn't invalidate tests whatsoever

[yjftsjthsd-h]

While I agree that it should be a standard feature, it is worth pointing out that operating systems tend to be more difficult and more expensive to run full tests for.

[cortesoft]

Yeah, the scope of things that an operating system needs to be able to do is basically, “all things that can be done on a computer”, so if you are trying to write full regression tests you are never going to hit all the possible combinations.

[yjftsjthsd-h]

It's not even just that, but also that a lot of the things OSs do are painful to artificially test because they're on the edge of hardware and software, or involve building the abstractions that let other software run without worrying about those details. How do you make a CI job that tests that mdevd correctly handles enumerating devices and setting their /dev nodes correctly, when the edge cases are finicky hardware devices and nondeterministic enumeration?

[pmarreck]

Does mdevd not send a signal or command out, and get one back? If so, the hardware behavior can be simulated, as can timing issues.

[notacoward]

You can test the response to a signal that way, but testing whether it was sent when it should have been can get arbitrarily hairy. I've worked places where we had to do things like rig up relays to push physical buttons (or in one case physically pull a cable) to do this level of testing. An adjacent team (in 1994!) had an actual robot that would rove over the a disk-controller circuit board delivering small electric shocks to get the same kind of test coverage on the other end of the SCSI bus. When your code has to deal with devices, which can misbehave in arbitrary ways from the voltage levels on single electrical pulses up to complex protocol violations, simulating purely-software behaviors doesn't cut it. Maybe preach about the cost/benefit ratio of different kinds of testing after you've had to literally build the tests out in the physical world.

[pmarreck]

So because you see testing all the things together as too prohibitive, you balk at testing the individual bits in isolation (hence "UNIT" testing)?

[pmarreck]

That's not an excuse (speaking as someone who has worked on a million-plus-line codebase). Pieces can be broken out and tested. That's the entire point of UNIT testing, it's right in the name

[unethical_ban]

Heh. I'm writing a custom tool for a security product that pulls configs down and looks for deviations from expected config values.

Instead of running the script against the client config and validating it works correctly, I thought to myself "Hey, what if I made a sample configuration with known good and bad values, and have a known result output to quickly validate the script's function?"

I just invented testing. No, large scale programming and devops is not my primary job. Yes, I have built validation before, but it isn't habit and this is a bespoke project so I didn't think about it at first.