[ChrisMarshallNY]

> So far I haven't found a good reason not to write tests...I don't quite see how tooling will get rid of the need for tests.

I write device control software. It’s very difficult to have true automated testing of things like drivers. You can write unit tests for subsystems, like packet parsers, but integration testing generally requires good ol’ “monkey testing.”

“Just write a mock!” Is what I hear all the time.

Mocking a device is a massive project; potentially larger than designing the device, itself. Remember that the mock needs to be of unimpeachable quality, and also needs to do things like simulate adverse signal environments.

DX for that kind of thing can be awful.

As far as basic DX goes...

Most developer tools are wrappers for command-line OS tools, and it shows.

They can also be quite buggy, and we accept this bugginess. I use Xcode, which is quite “crashy.” I am constantly fixing issues by deleting the build folder.

Back to testing...

I prefer test harnesses over unit tests. I write about that here: https://medium.com/chrismarshallny/testing-harness-vs-unit-4...

[henrikeh]

Just a question. Do you imply that the mock of a device should simulate/emulate the entire behavior of the device? Because you definitely don't need to do that and it would, of course, be a vast and hard-to-justify endeavor.

You only need to _mock_ the object, that is emulate behavior on a small scale.

[ChrisMarshallNY]

Yes and no.

It depends on what we are testing.

It can actually be worse than useless to rely on the "perfect world" of unit testing. I have had a nasty shock, when switching from mocked behavior to real-world behavior. I write about that here: https://medium.com/chrismarshallny/concrete-galoshes-a5798a5...

[henrikeh]

I'm not going to argue against your experience and from reading the article it seemed like you made the right call. Perfect world testing is indeed a failure, but I would say that "imperfect" world testing is exactly where mocking is beneficial, because it can reliably produce all sorts of anormal behavior. But again, it obviously depends on the situation and circumstances, and good design probably removes more errors than good testing.

[quelltext]

Sorry, I didn't mean to imply everything can be well tested, but where it can tests make sense even if they don't cover everything (vs. the attitude that testing is pointless unless you can cover all relevant code paths).

Also, I don't think there's as much value in unit tests than other more holistic types of testing, i.e. functional/integration tests.

TBH I read your linked article and still don't quite know what a test harness (in that context) is and in what way it's better and differs exactly in the concrete. Guess I'll have to read up on it elsewhere.

[ChrisMarshallNY]

I appreciate the feedback. I suppose that I should probably add a paragraph that describes what a test harness is.

Something along these lines:

"A test harness is a [usually] simple standalone application that uses the CuT, and presents a user interface. Since it is for testing, it may expose internal metrics and allow the user to access parts of the code that are usually hidden."

[pjc50]

This is very true. Not coincidentally, driver flakiness is an ongoing problem, as is the tendency of OEMs to bundle the actual drivers with a huge ball of mud (NVIDIA, HP are major offenders).

Perhaps some sort of formal spec for the hardware communication API would help, and would also help on the ASIC side, but I can't see how such a thing would be built and popularised. It's a very small balkanised world of driver writing.

[werg]

Too often, "tests are required" -- but as you say it's very hard to test anything meaningful, so developers "test something" and call it a day.