[jimmyswimmy]

> Another major challenge is to automate device drivers and hardware platform tests because they require the hardware to be tested.

Anyone have any thoughts on how to automate testing of coupled hardware-software systems? This is really hard; we've attacked it in the past by writing hardware simulators in accordance with the ICD. This falls flat once you find that the hardware doesn't precisely match the ICD, and usually it's much cheaper to change the software than the hardware. And at that point, the simulator hasn't actually helped you at all.

I recently built a system which involved several tightly coupled hardware components and we fought many bugs on a tight schedule. It would have been nice to find a good way to think about this beyond the basic hardware-in-the-loop manual testing.

[lliamander]

So, here's a thought experiment:

Start a company who's mission is to provide batch processing of hardware tests for their clients. Test jobs are submitted online with a packaged version of the software and a test script written in some job control language.

This company has a huge warehouse (to start with) of all sorts of hardware. Every port and connector (ethernet, USB, HDMI, PS/2, serial, etc.) is hooked up via a giant network to a central server.

The central server can then run the batch test jobs. It will deploy the software, and can even simulate interactive execution by routing keyboard/mouse signals to the device and scanning the display output signal.

Obviously, sometimes the hardware will have to be reconfigured, thus reducing turn-around time, but clients can pay extra to have hardware set aside that is set-up in their particular configuration.

Eventually they could build up a library of emulators which have been empirically tested to match actual hardware behavior (rather than the spec). Hardware that has been emulated can be tossed to allow room for new hardware. Customers might even be able to run on demand tests in "the cloud" using just the emulators.

---

Basically, I think it's too expensive to do in an ad-hoc basis. You really need a setup that can benefit from the economies of scale.

[rxhernandez]

I can't tell you precisely how it was done but I worked at a company with many different types of hardware that contained many complex configurations of FPGAs, lasers, optics, and microcontrollers, coupled to a computer and they managed to simulate it quite well for what seems like a decade. One of the scientists there was one of the few geniuses I've ever met and they managed to simulate all those devices and a sufficient amount of their variations. So I can confirm it's possible, maybe it just requires an overworked genius?

[yjftsjthsd-h]

So I can confirm it's possible, maybe it just requires an overworked genius?

I believe that, but unless you can find a genius and/or mass-produce their work, does it help the rest of us?

[rxhernandez]

Maybe. Knowing something is possible is sufficient for enough people to give things a try.

I know I've been successful in doing so; I've never built a simulator of this magnitude but I've successfully solved difficult problems with novel solutions simply from hearing it was possible to solve them in a given manner.