Using VR to understand how acoustic perception works in complex environments

Many people find it difficult to hear in classrooms, open-plan offices or near busy streets. In order to understand how acoustic perception works in such complex environments, leading experts of the University of Oldenburg, Germany, count on virtual reality (VR). As part of the new priority program Audictive ("Auditive Cognition in Interactive Virtual Environments"), they set out to clarify how complicated auditory processes such as perception, attention and memory can be investigated using new, interactive VR methods. The German Research Foundation (DFG) is funding three projects led by acoustics Experts Dr. Stephan Ewert, Prof. Dr. Volker Hohmann, Prof. Dr. Steven van de Par and neuropsychologist Prof. Dr. Stefan Debener with about 830.000 euros over the next three years.

The sound of virtual rooms

The first project, led by Prof. Dr. Mathias Blau of the University of Applied Sciences in Oldenburg together with van de Par, aims to find out experimentally whether room acoustics can be realistically simulated in a virtual environment. The team also wants to explore how visual perception influences the acoustic experience. Moreover, the researchers are going to investigate how audio rendering in virtual worlds affects social anxiety. That is, for example, whether the fear of giving a talk in front of a large number of people can be reproduced in a virtual environment and thus be studied more easily. Researchers of the University of Regensburg are also involved in the project.

The second project, led by Ewert, aims to find out how volunteers assess distances and movements using their sense of hearing. This is particularly important in situations with limited vision or when objects are outside the field of vision and the ear acts as a warning system. To this end, the team is going to place test persons in different virtual environments using VR glasses.

The acoustic stimuli will be presented either by headphones or loudspeakers. In the experiments, the researchers will vary the type of acoustic sources, their distance and intensity. Van de Par and Dr. Virginia Flanagin of the University Hospital in Munich are also principal investigators in this project. This research relies on the acoustic simulation system RAZR, developed in the groups of both Ewert and van de Par. It creates virtual acoustic environments that are very close to reality.

Chatting with virtual characters

The third project, led by Debener and Hohmann, will also be based on experimental studies in virtual environments. The aim is to understand how healthy people manage to filter out the voice of their current conversation partner from a variety of sound sources. This ability goes beyond just hearing or understanding speech, but requires the ability to control attention adequately. Hohmann, who heads a collaborative research center on hearing acoustics, and his team has developed a powerful acoustic stimulation system called TASCAR. This platform provides virtual characters not only with a voice, but also with realistic lip movements.

It is also capable of reproducing common everyday conversational situations. To find out how the test subjects control their attention, the team wants to measure the electrical activity in the test persons' brain with a mobile EEG device developed by Debener's research group. The aim of the cooperation is to establish mobile EEG technology in VR and, for example, to investigate the visual stimuli that help to direct attention to the desired sound source.

AUDICTIVE is one of 14 priority programs established by the DFG this year. It combines the disciplines of acoustics, cognitive psychology and computer science.