Late last year, Apple was granted a patent for AR headphones which would enable the listener to accurately determine the exact location of sounds in a scene from what are known as binaural recordings. This would have applications in both real-life video – like conference calls – and in augmented reality scenes.
A newly-published Apple patent application shows this type of binaural recording being made with an iPhone …
Binaural recordings are specifically designed to be listened to with headphones, and are intended to give a very immersive experience, where you can clearly hear sound coming from a specific direction.
Making binaural recordings is currently achieved with specialist microphones, many of them even embedded into a mannequin head to give the most realistic sound properties possible. These recordings accurately capture the way sound is affected (or ‘shadowed’) by the head and shape of the ears.
One compromise approach is to embed microphones into the earcups of headphones. This doesn’t achieve the same level of realism, but can still be convincing.
Achieving the same thing with closely-spaced microphones cannot be achieved by hardware alone, but Apple’s patent application (spotted by Patently Apple) describes using two cameras and four microphones to capture the sound, and then using software processing to emulate a true binaural recording.
It’s likely that you’d need to pair the two technologies for maximum benefit – that is, listen to the recordings made with the tech in this patent application through headphones which use the tech described in the previous one. However, it may be that ordinary headphones would still deliver a somewhat convincing experience.
While this approach is compelling for, for example, full virtual reality applications, in which a user can interact both visually and audibly in a virtual environment, in traditional video capture applications three dimensional sounds can distract the viewer from the screen. In contrast, monophonic or traditional stereophonic recordings may not provide a sufficient sense of immersion […]
An aspect of the disclosure is directed to a method for producing a spatially biased sound pickup beamforming function, to be applied to a multi-channel audio recording of a video recording. The method includes generating a target directivity function. The target directivity function includes a set of spatially biased head related transfer functions.
A left ear set of beamforming coefficients and a right ear set of beamforming coefficients may be generated by determining a best fit for the target directivity function based on a device steering matrix. The left ear set of beamforming coefficients and the right ear set of beamforming coefficients may then be output and applied to the multichannel audio recording to produce more immersive sounding, spatially biased audio for the video recording.
The major benefit of the approach described in the earlier headphone patent is that turning your head wouldn’t affect perceived location.
Our usual disclaimer applies: the company patents a great many things that never make it to market. Apple is, however, believed to be working on some kind of AR headset dubbed Apple Glasses, expected to launch within the next 2-3 years.
