DevOps in an Embedded World: Challenges for Embedded and IoT Devices

In recent years, the software that runs on embedded or IoT devices has become more and more the feather that tips the scales on whether these devices are successful on the market or if they fail in the customer’s eyes. When looking at companies that disrupted their market with their innovative hardware products, it becomes apparent that they often treat the product with a ‘software-first’ approach on top of Minimal Viable Product on the hardware side. On the other side are companies that often have established their market share over years and work with incremental innovation. In this software development scenario, oodles of resources are consumed, and manual steps are carried out, due to the organic growth of the companies and their processes. Although some companies may use the term “agile” in their development, usually the process phases run consecutively and separated from each other. After completing a phase (whether successful or not), the work results are tossed over the so-called ‘wall of confusion’, true to the motto of ‘devil-may-care’. This procedure results in the following main problems, which can be encountered in the same ways in a wide variety of organisations: 

• Many manual steps, which make the overall process slow and susceptible to errors. 

• Quality assurance and testing as separate phases at the end of the overall process. 

• Security requirements and other non-functional requirements, such as load and performance or maintainability, are only discussed and checked just before final delivery. 

• Lack of collaboration due to separate working areas and phases. 

DevOps in an embedded world: Make experimentation possible and speed up the time-to-market

Since people started to recognise a pattern in these behaviours and problems, a solution has emerged: DevOps.  

DevOps is a mindset, a culture and a series of technical practices. It provides communication, integration, automation and close collaboration between all persons across the whole value stream who are required for the planning, development, testing, provision, approval and maintenance of a product. 

DevOps tools in an Embedded world: automation is key

The goal of DevOps is to speed up the market launch, make experimentation possible, release software at shorter intervals, reduce the lead time for bug fixing and improve the average restoration time.  

While DevOps is a well-established practice in pure software domains, the embedded world has not yet fully embraced the DevOps mindset in many places. One of the many reasons for this lack of adaption is that embedded applications and the employed tools are often tailored to the specific application and its environment. The tools in use are diverse and the range of solutions to achieve some sort of automation is wide. For this reason, many standard DevOps tools and practices can only partially be used in an Embedded environment without adaption. This is especially true for the following 3 steps: 

• Testing 

• Monitoring 

• Deployment 

This blog post will focus on testing, especially automated testing as automation is key in a DevOps setup. 

To understand why automated testing is harder in the world of embedded device development, we need to look at the testing pyramid. The testing pyramid is a framework that helps developers to prioritize their testing efforts. A typical pyramid consists of three or more layers which typically contain at least unit tests, integration tests, and end-to-end tests. In general, tests that are lower in the pyramid, i.e. unit tests, are more isolated and faster and therefore cheap to automate, while tests that are higher in the pyramid, i.e. end-to-end or system tests, are more complex and expensive to automate. 

Success Stories: How to tackle the DevOps challenges for IoT devices

If we want to effectively bring DevOps to the Embedded world, we must facilitate the automation of the more complex system and end-to-end tests. Let’s see how leading manufacturers of IoT devices have tackled this challenge! 

End-to-end testing for a parking meter

The parking meter of Digital Parking is a typical IoT application: Data (e.g. what Vehicles are allowed to park on which parking lot for how long) is acquired and displayed on the device while the same information is available online to facilitate controlling and even provide live utilization data of the parking area. To enable automated end-to-end testing of the full system, a complete device was equipped with additional hardware such that user inputs (e.g. pressing a button or feeding coins) could be triggered on the real hardware.  

Using such a setup, more than 50 test cases can be automated. They range from user interactions, e.g. paying for your parked car, to checking whether data had arrived in the cloud. Also, more sophisticated maintenance and installation use cases are possible to automate. 

This test setup is on one hand used to periodically perform regression tests to ensure the IoT device is running stable, but also to replicate issues discovered in the field. Learn more in our success story "Parking meter for the Internet of Things". 

Power Consumption Tests

To ensure the power efficiency of their devices, Texas Instruments (TI), a global semiconductor design and manufacturing company, has developed a suite of automated power measurement and analysis tools called the EnergyTrace™ software. This software enables developers to measure and analyse the power consumption of their devices in real time, both during development and in the field. 

The EnergyTrace software includes several features that enable developers to automate power consumption testing, including Continuous power profiling, Power measurement automation, Power analysis tools. 

By automating power consumption testing, TI can identify power consumption issues early in the development process and make iterative improvements to the design of their devices. This enables TI to deliver more power-efficient devices that meet the needs of their customers. 

Security Testing for IoT applications 

While security testing on a running system (e.g. pen-testing) is challenging to automate because it involves testing the device's vulnerability to various types of attacks, static testing of code is automatable with reasonable effort. 

Static application security testing (SAST) uses tools (e.g. Sonar Cube) to analyse the source code at compile time for vulnerabilities. When integrated into a CI pipeline (described here for GitLab and GitHub), such tools can be used to even prevent vulnerable code from being committed to the code base. Often, as part of static code analysis, secret scanning is also used to detect whether secrets are mistakenly committed to the source code. These tools help to prevent security issues in the application code (described here for GitLab and GitHub). 

Of course, there may also be security issues in third-party libraries that the application depends on. Software composition analysis (SCA) is checking these dependencies against public databases of known vulnerabilities. Several tools may be used to automate this task and can therefore be run periodically to not miss any newly found vulnerabilities (described here for GitLab and GitHub). 

Discover more possibilities with our "Industrial IoT Starterkit". 

Continuous Integration for a medical oven

For the development and maintenance of a specialized, high-temperature oven for medical products, an automated testing framework has been developed by Zühlke. Building onto the Robot Framework, a test system was implemented such that many system tests could be executed automatically. The tests included user-centric use cases as well as factory use cases like calibration or maintenance cases such as updating the firmware. 

The test automation was finally integrated into the daily development flow and used in a dedicated pipeline such that these system tests would be performed for all changes committed to the code base. This setup greatly enabled the development team to move quicker on new features while having confidence that no regression would slip into a new release. 

Conclusion: DevOps ensures market success and customer satisfaction

Continuous quality assurance of embedded and IoT devices is critical to ensuring their reliability, safety, and functionality. As the software running on these devices becomes increasingly important in determining their success in the market, it is essential to adopt a DevOps approach that prioritizes automated testing and collaboration across all phases of the development process.  

Unfortunately, this automation is not readily available for embedded devices as it requires dedicated effort on the developer side for every new product. In this article, we've shown how far companies go in their efforts to automate their testing for IoT devices which underlines the importance of closing the DevOps cycle and reaping its benefits.  

Ultimately, investing in robust testing processes will not only ensure the success of these devices in the market but also promote user satisfaction and trust in their functionality and security. 

Do you want to explore the possibilities of DevOps? Please get in touch with us!