• Xi Peng, Tandon School of Engineering, NYU & University of North Carolina, Chapel Hill


• Kenneth Chen, Tandon School of Engineering, NYU


• Iran Roman, Tandon School of Engineering, NYU & Music and Audio Research Lab, NYU


• Juan Pablo Bello, Tandon School of Engineering, NYU & Music and Audio Research Lab, NYU


• Qi Sun^†, Tandon School of Engineering, NYU


• Praneeth Chakravarthula^†, University of North Carolina, Chapel Hill

     

Figure 1: Azimuth-based audio perceptual acuity guided sound source clustering. (a) visualizes our model-predicted human auditory perception of spatial discrimination threshold along azimuth eccentricity in degrees. (b) illustrates the model-derived audio source clustering method. Based on listeners’ heading direction (assuming forward here), we cluster audio sources that are spatially indistinguishable. Clusters were highlighted by colors corresponding to the human’s minimum audible angle. The number of sound sources for each cluster is marked in the figure. Our measurement shows a 53% computational saving at the presented scene.

Abstract

Realistic spatial audio rendering improves immersion in virtual environments. However, the computational complexity of acoustic propagation increases linearly with the number of sources. Consequently, real-time accurate acoustic rendering becomes challenging in highly dynamic scenarios such as virtual and augmented reality (VR/AR). Exploiting the fact that human spatial sensitivity of acoustic sources is not equal at azimuth eccentricities in the horizontal plane, we introduce a perceptually-aware acoustic “foveation” guidance model to the audio rendering pipeline, which can integrate audio sources that are not spatially resolvable by human listeners. To this end, we first conduct a series of psychophysical studies to measure the minimum resolvable audible angular distance under various spatial and background conditions. We leverage this data to derive an azimuth-characterized real-time acoustic foveation algorithm. Numerical analysis and subjective user studies in VR environments demonstrate our method’s effectiveness in significantly reducing acoustic rendering workload, without compromising users’ spatial perception of audio sources. We believe that the presented research will motivate future investigation into the new frontier of modeling and leveraging human multimodal per- ceptual limitations — beyond the extensively studied visual acuity — for designing efficient VR/AR systems.