Case Study: Android Auto Stress Testing

How a major automotive OEM used automated Android Auto testing to resolve connection issues in the field.

A leading automotive OEM began to receive feedback on its Android Auto equipped infotainment system that some users are experiencing problems when connecting their Android mobile device and synchronising Android Auto functions with the head unit. Detailed consumer feedback implies that the infotainment system does not always detect the phone connected to the Android Auto enabled USB port in order to instigate the Android Auto function on the infotainment system.

Additionally a further field issue reported is that during an Android Auto session an outgoing call initiated from the car does not establish the uplink audio, meaning the called party does not hear any audio from the caller when answering the call.

The OEM used Nextgen ATAM Connect automation to investigate the user feedback, deploying stress testing to explore robustness of Android Auto connections and the performance of key functions, such as calling and audio routing.  ATAM is capable of performing automated testing of Android Auto USB connections by connecting and disconnecting multiple connected Android mobile devices using an automated USB inserter mechanism to make connections. The ATAM Connect test setup is also configured to monitor audio quality and routing, audio levels, distortion and also audio stream dropouts down to a resolution of <10ms while in the call.

Any unexpected issues which occur during testing will be captured in detail. ATAM Connect collects and collates a variety of key data logs which detail the exact circumstances of the failure. In this example ATAM Connect logs USB protocol connections and voltages, audio system and mobile device HCI logs, network connectivity, Bluetooth and Wi-Fi activity and creates a fully logged record of test failures for debugging. This series of captured logs also indicates the exact parameters at the time of failure, allowing for an 'instant replay' function for test failures, allowing the exact conditions at the time of failure to be reproduced for investigation and root cause analysis.

Test Setup

The test set up consists of the infotainment head unit and a selection of key Android mobile phones connected to the USB connection control Interface and USB inserter mechanism,  which physically attaches the USB cable to the infotainment system's USB Hub . A Beagle USB analyser is also connected to provide detailed logs of USB traffic.

A diagram of the test setup is show below. The infotainment system USB Hub is connected to USB cable via the USB Inserter. The primary Android mobile phone (DUT) is connected and disconnected via the USB inserter and while connected the Beagle USB analyser logs data transmitted. For the call audio tests ATAM Connect directly connects to the infotainment system’s microphone inputs and audio / speaker outputs. A secondary mobile phone is connected as a slave mobile device as a target for calls from the infotainment system and to provide incoming call tests. ATAM Connect provides full automated control of the mobile phones, including pairing, making and receiving calls and detecting when Android Auto mode is enabled.

How the Test Functions

ATAM Connect automatically connects and disconnects the phone connection from the infotainment system USB port, also measuring the current and voltage drawn by the phone over USB.

During a phone call ATAM Connect monitors the audio levels for the downlink and uplink call audio and measures audio quality, dropouts and reports if any echo is detected. Multiple phones will be connected over the course of testing and the functions of Android Auto on the infotainment system are controlled and monitored using an ATAM custom control interface. 

Nextgen also helped to define the test list of key mobile phones drawn from Nextgen’s extensive global library of devices to ensure a good match and coverage for the devices in use by consumers. Connections to the infotainment system are made using USB and Bluetooth.

The Results

During the stress testing, ATAM Connect was able to replicate the field reports, finding that specific devices were not always detected upon USB connection and so Android Auto did not launch. In a some cases, it was also confirmed that audio was not always correctly routed to the infotainment head unit.

Using ATAM Connect automation multiple phones were rapidly tested for USB and Bluetooth connection, disconnection and calling functions. The test cycles were repeated 1000’s of times, and intermittent failure conditions logged and identified. The test was able to verify and reproduce reported issues across a range of mobile devices and scenarios. The frequency of the occurrence of these issues was found to be below >7% - low enough to be missed in manual development test processes but high enough to account for the level of field issues reported by end users. Additionally some additional device specific failures were found and logged which had not previously been reported.

During the test process ATAM Connect collects multiple logs when any unexpected issues are detected. The precise test failure scenarios along with detailed phone, Bluetooth and USB connection logs are collated and filed for debugging and rectification.

Conclusion

Nextgen ATAM Connect automation stress testing reproduced the issues reported by customers and identified specific devices that were failing for the problems observed. Using the collected reports and logs the precise conditions and scenarios required to reproduce the failures were identified, allowing the root cause of issues to be understood and provided recommendations to fix via firmware update.