Case Study: Using a cobot for automating touchscreen testing

How Nextgen ATAM helped a global Automotive OEM test and debug a newly launched in-market infotainment system

“Nextgen's cobot and vison system provided a rapid solution to our field service issue ...that allowed a fix to be developed in weeks, compared to the months of test and analysis that would have been involved in manual debug. ”

Lead Project Engineer, Global Automotive OEM

Most product testing today still involves human engineers testing features and functions manually. According to the research group Global Market Insights most software testing is still performed using human test engineers as shown in the diagram below. This is because test automation requires software engineers to create test frameworks and then write individual scripts for each use case. If the system functionality changes then these test cases need to be updated. Test case maintenance is one of the key challenges facing the software test automation industry.

Product testing involves more complexity than software testing due to the unique physical interfaces for each product. This can range from conventional touch screens to buttons, sliders, rotary controls and pressure sensitive interfaces. In many cases the complexity of instrumenting and manipulating these physical controls presents a barrier to creating an automation solution. Typically the mechanical or electrical manipulation is too complex and cost prohibitive. This is where modern collaborative Robots or Cobot’s can provide a significant benefit. Cobot’s are easier to program and incorporate safety features allowing them to operate alongside humans avoiding the need for large enclosures.

Benefits of Automating Product Testing

Automated product testing provides the design and development team with significant insights into device performance. Based on our experience a balance needs to be established between those features that need to be tested manually and those that can be automated. Manual tests will typically involve 1-5 repetitions of the test case before a result is recorded. With automation we can increase that to hundreds of tests and establish a more intimate understanding of the product behaviour. This is particularly useful when trying to find hidden bugs that can be easily missed during manual testing due to the lack of test coverage or the intermittent nature of the problem. Automated testing is excellent for finding those issues that may only occur say once in 10 or 20 cycles.

Increasing the test cycles through automation enable engineers to develop further insight into product performance. These could be metrics such as responsiveness when the system is loaded.

The graph below shows the time to connect after power cycle using the Bluetooth Low Energy link, over 26 cycles. The connection time can be seen to vary by over 1 second.

Automated testing enables the design teams to accurately measure performance distribution over a high number of test cycles with zero or little incremental effort. In the case above the test could be increased to cover 500 or 1000 cycles with minimal additional cost. This allows the test engineers to focus on the root cause analysis and reduce the time spent performing repetitive manual tests.

Product Testing Using Cobot’s

Cobot’s provide a low cost and adaptable solution to product testing where manipulation and control of a product is required. The main challenge has been the process of programming the Cobot to perform individual actions, called tool paths. Each tool path requires a series of way points to be defined in 3D space for a specific movement, such as pressing a button for example.

The latest Cobot’s feature mechanisms that can be manipulated by the human engineer to teach the desired action, making the task of defining a movement much easier. In other cases more sophisticated tools can be used to define feature planes in 3D space and reference all cobot actions to that plane allowing more sophisticated control and movement. Examples of these are shown later in this study.

One important aspect of modern Cobot’s is their ability to measure applied force at the end effector or touch stylus, this allows us to reproduce real world human interactions such as touch and swipe for example.

Coupling the Cobot with a vision system enables us to expand the test scope and check each function and test result. We can also capture live video of each failure mode while the tests are in-progress.

The case study illustrates how Nextgen's FlexFlow cobot and vison system used automated testing to solve a field service issue in weeks without requiring any incremental manual test effort.

Challenge

An Automotive OEM was receiving reports of calls being dropped from a recently launched infotainment system. This issue was generating consumer complaints on its new vehicle platform. The OEM decided extensive stress testing was needed to deep dive the connectivity issue to reproduce and fix with a system firmware update.

However the OEM’s infotainment security policy meant that the programmable interfaces used for testing development firmware builds were now disabled in the final production firmware builds, ready for market launch.

The OEM team were unable to find manual test resources for this urgent testing and needed a viable solution to automate its testing and requested help from Nextgen.

What We Did

Nextgen’s ATAM FlexFlow cobot and vision system was deployed to control and manipulate the infotainment user interface and perform the required testing on the production firmware.

Using the ATAM FlexFlow cobot system a test sequence was rapidly developed to make and place calls of varying lengths while monitoring the connection and call stability. Our test engineers were quickly able to construct the test cases with Nextgen FlexFlow cobot using the visual point and click macro system to direct and record the cobot movements.

The ATAM FlexFlow codeless test user interface allows anyone to create complex test cases.

Once setup the automated test sequence accessed the phone pairing menu on the infotainment system, using the cobot and vision system to manipulate the HMI to perform the pairing and conduct call tests. The mobile phones were selected and controlled using Nextgen’s seamless smartphone control.

The ATAM FlexFlow cobot vision system was also configured to sequentially control a range of 20 key mobile phones on different networks. The length of each phone test was five hours total with calls of different lengths. The cobot was testing intensively 24 hours a day and the testing completed in 5 days. An in-depth call quality and connection stability profile for each phone was generated.

Success With ATAM FlexFlow cobot

The rapid action of the OEM's engineering team coupled with ATAM FlexFlow cobot vision system was key to the successful and rapid resolution of the field issue. The ease of using the vison system and intuitive point and click interface directed the movements of the cobot to perform complex tasks on the infotainment HMI without a single line of code.

 ATAM FlexFlow's visual, drag-and-drop graphical interface enabled engineers to create and update a detailed test sequence on both the infotainment system head unit and the connected mobile phones without coding. As well as making the creation of test flows both simpler and quicker, it also enables potential problems in a test flow to be identified more easily – leading to faster resolutions and more robust testing practices from start to finish.

Results

The test data was analysed and it quickly became apparent that phones from a certain manufacturer were dropping calls. Bluetooth traces demonstrated a timing mismatch between the phone Bluetooth stack and the expected response timing in the system, leading to phone disconnecting from the infotainment system unexpectedly under some conditions.

The Automotive OEM notified and worked with the phone maker to correct the Bluetooth stack implementation issues in subsequent phone firmware releases, and simultaneously released updated infotainment firmware to work around the unexpected phone connectivity behaviour. This provided an immediate route to satisfy consumers with phone connectivity issues.

“Our bug location and fix to firmware validation team have been aided by Nextgen ATAM to find and fix an unexpected bug fast before it became a major field issue and customer service headache.”

Development Engineer, Global Automotive OEM

Find out more about how Nextgen’s automated testing platform (ATAM) and cobot vison system can help you accelerate interoperability testing.