Case Study: Critical Firmware Updates for True Wireless Headphones

How Nextgen ATAM helped a global OEM extend its market share by testing crucial over-the-air firmware updates for wireless headphones.

“Nextgen ATAM gave us the power to test a wider range of functionalities over a larger number of devices in only half the time. What made it even better was that it integrated with our existing systems, using the APIs we’re utilising for our build server on one side and for our deployment server on the other side. It fitted into our existing infrastructure in a way that streamlined our operation and delivered significant tangible benefits.”

Global Headphone OEM

True wireless headphones are one of the world’s most in-demand consumer electronic devices, with a CAGR of 21% anticipated between 2020 and 2026. By 2026, the global market is projected to be worth $42.1 billion. But with that success comes significant pressure for OEMs.

These tiny devices are effectively mini computers, with functions extending well beyond delivering good audio from connected devices. On most models, Bluetooth and inbuilt microphones allow the user to take calls. Automatic sensing saves power if the headphones are removed from the wearer’s ears.

Integrations with smart-home assistants are common. And with users often spending hundreds of dollars on their headphones, the expectation is that they will work first time, every time.

Challenge

A headphone OEM approached Nextgen to ensure the successful interoperability of its latest wireless headphones.

As the complexity of the device’s functionality increased, so too did the potential for things to go wrong. Like many of the world’s leading headphone manufacturers, this OEM relied on firmware updates issued through a companion app to ensure in-market products remained compatible with both existing and new devices.

We were challenged to devise an automated interoperability testing regime that checked all aspects of this OEM’s over-the-air firmware updates. Our tests had to ensure ongoing performance and compatibility for a large and growing user base. And they had to be completed relatively quickly, because firmware updates were issued frequently.

Our challenge was three-fold: ensuring excellent test coverage, maximising the speed of the testing process, and replacing manual testing with a fully-automated testing process that, if successful, could push the firmware update to the OEM’s deployment server without human involvement.

What We Did

Our engineers designed an automated interoperability testing set-up that saw our market-leading and patented automation platform Nextgen ATAM (Automated Test Application Manager) take the strain.

Nextgen ATAM was configured at the heart of the system. It automatically checked the OEM’s build server for firmware updates. If it found a firmware update, it then ran a full testing programme across a suite of 10 reference mobile devices specifically chosen to represent a mix of manufacturers, OS levels and Bluetooth stacks. The tests checked audio quality and integrated wireless connectivity to a far greater depth than previous manual testing regimes.

The three graphs below illustrate some of the audio quality tests in action:

1. Dropout in the audio stream showing interruptions in an audio waveform.

2. This caused a frequency change failure.

3. This caused a frequency change failure, so even though the gap was very short, it was still detected as a failure.

Planning for Success With ATAM Flexflow

ATAM Flexflow was central to the system’s success. This visual, drag-and-drop graphical interface enables engineers to create a detailed test sequence without struggling with coding. As well as making the creation of test flows both simpler and quicker, it also enables potential problems in a test flow to be identified more easily – leading to faster resolutions and more robust testing practices from start to finish.

Results

Thanks to Nextgen ATAM, the headphone OEM was able to halve the time it took to test each firmware build while simultaneously doubling the number of mobile devices tested. On top of this, the depth of testing was significantly increased. Automation looked at the performance and compatibility of each firmware update across several functions, including calls, music streaming, audio quality and connectivity.

For each firmware update, typical questions answered through this automated testing programme included:

• Did pairing work without glitches or difficulties?
• Was the connection maintained over long use periods?
• Was the audio being produced accurately in each headphone?
• Were there any interruptions during the streaming of audio?
• Was the audio correctly produced, or did it suffer from too much bass?
• Did the companion app still work?
• Could users still make and receive phone calls?
• Did all previous functions remain intact with each firmware update (regression testing)?

Nextgen ATAM was able to automatically check the build server for any potential firmware updates. If any were ready for testing, it then automatically completed the testing programme across a range of reference mobile devices. And if the firmware update was found to be successful, Nextgen ATAM automatically placed it on the deployment server ready for immediate distribution to the user base through the companion app.

With rock-solid firmware updates, the OEM gained market share as confidence grew among end users that its products were reliable, long-lasting and high-performing – even as new devices continued to emerge on the smartphone market.

“Our firmware validation process has been completely streamlined by Nextgen ATAM. We’re no longer constrained by having to have people physically in the office to carry out these tests. We can now send firmware updates to all our users at once, confident in their performance and without feeling any risk to our brand equity and global reputation.”

Global Headphone OEM

Find out more about how Nextgen’s automated testing platform (ATAM) can help you accelerate interoperability testing.