[jschwartzi]

Correct me if I'm wrong, but this seems to run unit tests on the embedded target. Why?

In the embedded space I've found unit tests to mostly be useful for testing logic and algorithms but not terribly useful for testing hardware integration, which is where all the really painful and tedious problems happen. If the unit test passes on the board but fails on your host it's likely due to a compiler bug or a bug in the MCU, and you usually just work around those.

What would be more interesting is creating an environment to support manual and automated integration and system-level testing. I haven't seen such a system in practice because such systems require a very lengthy and painful characterization of the system under test. They also require hardware support, meaning a tool needs to be integrated with the software to interface with and measure the output of the system under test. These can range from simple I2c-based GPIO expanders for monitoring and controlling IO lines to pressure and flow meters for pneumatic measurements, or current probes for instantaneous and aggregate power draw measurements. This is more like laboratory automation so I think a lot of software people try to avoid doing these things.

Do you have any plans to implement APIs that might enable development of such a system in the future?

[sgtnoodle]

I've found a lot of value in running unit tests on target hardware. For one thing, it means you can actually include peripheral drivers in the test. Also, as you said, embedded compilers tend to be more buggy than host compilers. I've actually found compiler bugs through embedded unit testing.

[ctz]

Peripheral unit tests are best run off-target in my experience. We mock out all register accesses for host builds, then can write unit tests that validate we write registers in the right order, handle states that are in-spec but hard to get the real peripheral in to, etc. Things that may work a million times in a row on real hardware, but then regularly fail in the field on a few million devices.

Running them on-target is more of an integration test -- useful too, obviously, but gives a quite low level of assurance.

[jschwartzi]

Yes, exactly. Once you're testing integration with hardware you're integration testing, not unit testing as it's not pure software dictating the outcome of the test.