Well, in fact you can and you should test embedded software in a test stand. But it shouldn’t be the only thing in your test strategy. In fact, you probably shouldn’t spend more than a third of your test budget on the test stand. Here is the reason why:
A bug found on the test stand is an expensive finding!
Instead, almost all bugs should be found well before the system achieves the state of completeness to make it testable on a test stand. The test stand is at level three (system testing) of the test levels defined by the ISTQB (figure 1).
The cheapest bug is the one never coded – this is why we have coding guidelines and best practises. The next cheapest bug is the one found by the developer who writes code and unit-test side by side. When coding the unit-test, the developer has all the information available, what and why the code goes wrong. He doesn’t lose time on lengthy debugging and re-familiarizing himself with the code.
The next cheapest way to catch a bug is during integration testing. This is the first opportunity to catch bugs that go beyond mere coding errors, such as inaccuracies and omissions in the specification or architectural problems. Never skip integration testing!
And only after basic unit testing and integration testing, the system should be tested on the test stand. At this point, the embedded software will be at a much higher quality level than it otherwise may have been. Make no mistake – issues will still be encountered, but much fewer. The risk of costly re-designs that break schedule and budget is much lower.
Yes, you can test embedded software outside its environment!
But then again we’re talking embedded software, and for the longest time the consensus was that you couldn’t honestly test it outside its environment. And I contend there is some truth in this point of view. But then again, much of a typical embedded program isn’t so embedded after all. Beyond the driver layer, there is much of the same as everywhere else: Data processing, persistence, UI, business logic. Depending on the scale of your system, these things are between 50% and 90% of the software and can be perfectly tested using nothing more than any vanilla xUnit test framework and Visual Studio or Eclipse.
A hardware interface is just a software interface!
And even the hardware-dependent parts can be functionally tested without resolving to outright simulation. With the proper architecture and abstractions, a hardware interface is just a software interface and can be replaced for the purpose of testing. It requires certain skills as well as a modern, testable software architecture. But it requires no specialized tools nor unreasonable effort.
Skills and tools
Here we come to the heart of the matter. As for tools – it remains a problem. Not for any amount of money is it currently possible to acquire a decent embedded development environment with built-in support for modern software development practises, such as unit-testing, integration-testing, build-server and continuous integration server. Current best practise is to get out of the comfort-zone of the vendor-supplied IDE and assemble your own set of tools: Build-system, unit-testing framework (e.g. Unity), mocking framework (e.g. CMock), hardware-in-the-loop testing framework.
And to conclude this post with shameless self-promotion: Here at Zühlke, we have created an industry course to spread the necessary skills. It is geared towards embedded developers and brings them up to speed with state-of-the-art methods of software testing: unit testing, integration testing, testable architectures and available tools.