In-Car Voice Recognition (VR) & Voice Assistant (VA) Automated Testing

Voice assistants are now embedded as part of the in-car experience, and the applications are growing, potentially offering a safe way to receive assistance, navigate, communicate, entertain and control car functions hands free.  Ensuring that the voice assistance works as expected for every driver in the car environment is challenging, but the rewards for both safety and driver satisfaction are huge.

Recent advances in Cloud computing and AI technologies have enabled a new generation of voice assistants to help consumers choose music in the home and control their domestic appliances and utilities. Also many of the first level support helplines use voice assistants to guide customers and address basic customer support questions. Consequently, today’s consumers especially the younger generation are quite comfortable using voice technology.

Advanced voice assistants are now integrated in to the automobile HMI to manage vehicle functions or program a destination in to the navigation system amongst other things. Voice control provides a safe and convenient solution to controlling the complex HMI functions available on modern automobiles. This is most relevant as modern HMI systems all switch to the proverbial tablet format containing a multitude of features and functions. Sometimes a simple task as adjusting the cabin temperature can become a challenge when the user has to navigate through the HMI. Using a voice assistant to ‘change the temperature to 23 degrees’ provides a convenient and safe option.

However unlike the home, the in-car environment presents unique challenges that need to be addressed. This article explores the merits of automating the testing of Voice Assistants and the associated benefits.

In-Car environment

The In-car environment presents some unique challenges to testing voice assistants. Firstly the acoustic environment can change significantly with the settings of the Heating and Ventilation system. During the rapid demist process the ventilation fans tend to be on maximum speed generating significant background noise.

Similarly as the vehicle speed increases the road noise component (including sub-sonic noise) will increase, potentially affecting the performance of the voice assistant system. Other factors such as noise from passengers complicate the operating audio spectrum for in-car voice assistants.

Manual versus Automated VR Testing

During development the tuning of the voice assistance system is performed in prototype vehicles with human test subjects. This process can be expensive and time consuming to cover the range of languages and driving conditions involved. To save time 2-3 test subjects need to be in the vehicle at the same time as its driven over the test route. With the outbreak of the Covid-19 global health emergency such drive tests could not be possible due to social distancing rules.

Although human test subjects are the ideal candidate to test the voice assistant system the logistics to mange their availability and ensure the subject is consistent with the pronunciation of each voice command can be challenging.

Considerations for automation

To automate the voice assistant test process we need to consider three factors;

(1) Library of test phrases: These are pre-recorded audio speech files collected from human subjects in a controlled audio environment. Ideally recording made in a vehicle travelling at different speeds are the most suitable. Once the files are categorised we have the basics to exercise the voice assistant system.

(2) In-vehicle environment: The in-vehicle acoustic environment can be affected by vehicle speed, number of occupants, HVAC settings and other audio sources. Using a real vehicle is the ideal environment for automated testing. The vehicle will remain static throughout the test.

(3) Road Noise: This is a significant factor in the performance of in-vehicle voice control systems coupled and coupled other sources such as transmission and engine vibrations will affect performance. High quality in-vehicle recording can provide a good reference to simulate vehicle noise using loudspeakers positioned close to vehicle exterior or mixed with the test audio.

The typical test set up for an in-vehicle automated voice assistance test system is shown in the diagram below.

The automation platform such as Nextgen’s ATAM Connect solution is responsible for selecting the test files and checking the IVI system voice assistant output response against each input command. The automation system will need to verify the audio response and determine if a test passed or failed. Audio and video recordings are collected for each test cycle.

Automated Voice Recognition & Voice Assistant Testing

Automation enables faster time to market and significantly reduces the overall costs of voice assistant testing. Testing can continue 24x7 as part of a CI/CD development program and allows the test engineers to tune the test program on the fly.

Find out more about Nextgen ATAM Connect

It would be great to talk with you about how robust test automation helps ensure VR/VA voice functionality in a wide range of use cases applications, contact us today.