To start off 2021, we invited sound designers Em Halberstadt and Millie Wissar from ‘A Shell in the Pit’ to pick apart and unpeel the many layers of sound in VR. Here’s a wholesome revisit, or call them closely-taken notes, on the aspects of sound that our guest speakers unblurred.
As Em and Millie have teamed up on several sound projects in the past and have experience working on the sound of the MOR itself, the talks were made all the more fun as attendees toured some of the works that they spoke about. While the focus of what Em shared was the spatialization of sound by breaking sound down into components and talking about the use of references from story-elements, Millie elaborated on how Ambisonic Sound works.
In VR, sound immerses you in the midst of the story of a space. Sound is inherently spatial. It creates a more complete sense of immersion than in the real world, as it envelopes you and lets you hear all the different layers of sound coming from a specific direction. There's a reason meditation techniques focus on sound so much, to bring about a sense of presence - as it surrounds you in a moment in time. It fits in with VR perfectly, where using it as a spatial element guides the audience through an experience with sound cues. When people hear something in the distance or behind them, they're naturally drawn towards it. So we are directing a gaze, or body movements subconsciously. This helps create a seamless experience of a story.
It is very important in VR, when you have an entire world around you, that you know where to look. Regardless of whether a piece of VR art is animated or static - sound adds movement, life and an element of time. This makes it a powerful tool for storytelling. When you design these little snippets of time, you allow them to be discovered throughout a space, really bringing a viewer into that world. It creates an experience that's interactive. And in VR, such interactions with the body are what create a feeling of being immersed.
Several ways of being creative with it were scattered throughout Em’s talk. She took us through examples where accomplishing different kinds of immersion was the ambition. Some different methods or elements sound designers use - and have been put to use throughout the museum, and in almost every art piece - were delved into.
The first kind of immersion is designed using ambient sound. So you can hear it, as it kind of surrounds you. Most of the sound you hear in the museum are elaborate ambiences. Every world created within the museum has what gets called a sound bed - which reflects on its environment. For Alex's Sci-Fi World by Matt Schaefer, rain is the bed that grounds the viewer, and it's created using four mono files placed in your headset.
As the mono files are made to follow the camera with six relative positions (and they don't rotate with the camera), you can turn your head inside of the ambience. Next, there are spot ambiences and loops that are placed all around in the space for the viewer to discover. The spray painter in Alex’s Sci-Fi World is one example of a spot. As you get closer to such spots, the volume fades up - following what's called an attenuation curve. You can set the distance on these curves such that bigger sounds can be heard from further away and draw you in, and smaller sounds are more like secrets to discover. You won't hear them unless you come really close to engage and trigger it to unravel.
Such use of sound encourages people to explore more, as once they realize that they can find these sorts of instances their experience starts feeling playful. Spots work like little stories, as they are details that help give life and movement to a world. Like that little conversation between bugs in Alex’s Sci-Fi World. They can be uneventful, but could still work as entire sequences of conversations that ebb and flow in the space. Even if it's not a linear story - the feeling of life, and the emotion of the sound can be quite powerfully driven.
Perhaps the most distinctly separate element in sound design is music. In some cases, the music may be diegetic. This means that it is sound that lives within the world. A technique used to suggest this is known as futzing, which is the process of making sound seem like it's coming from a radio, a phone - which in the scene Em shared (during the talk) was used to design the sound for a band. Em recommended that music is made diegetic in VR, so that there is the option to move away from it. While music with no source can take you out of the feeling of an environment, it can also compensate its presence by carrying an emotion. It's when non-diegetic music is overused, that an overloaded numbness gets caused - which could be unpleasant. Unless it's made more atmospheric - like an ambience. The ambience in the lobbies of the MOR is a bit of a cross between music and ambience, for example. There’s a mix of scatter sounds (which are non-looping sounds, triggered at random locations in time) and ambient sounds. Sounds of birds, gusts of winds or other environmental sounds are another example of ambience which add a sense of space that's alive. Such a suggestion of life can be deteriorated if sounds are looped - because life is random and doesn't loop.
Em shared Skeletons by Liz Edwards to give another example of a spot. A spot can be heard from further away to drag the viewer towards the source. The cackles and laughs in this piece are an example of what are called SFX (short for sound effects). Sound effects are interactive sounds - which in VR one can find many ways to get creative with as much as in non-VR. In the case of the piece in question, the movement of the body through the door gets used as a cue that triggers a big laugh and draws the viewer towards the spot.
The direction of the viewer’s gaze can also be used as a trigger in VR. When you look at each of the skeletons, a randomly selected laugh gets triggered. It's less conventional than having the sound direct the gaze, but that makes things real fun. This was a deliberate choice for the piece, to not let viewers realize what's happening. Yet the sound gives this feeling that the skeletons are aware of what the viewers are doing. This brings them to life as they are as active as the viewers are in a transactional gaze. Creating the story using the body of the viewer can be powerful. This can also be seen in another piece by Liz called Ghouls. In this case, the sculpture is brought to life with literal breathing that follows the animation in the piece. However, this doesn’t mean that sound effects need movement or animation of their own to generate a sound. The effect could be used to encourage a viewer to move closer to the characters or properties in the artwork. The bed ambience in this piece brings you in to feel an aura of creepiness.
We often hear people saying they don't know how to talk about sound, they struggle to describe it. If you're wanting to communicate with a sound designer and help them understand the story you want to tell using sound, the following is some language-level advice that could help.
Experimenting with making sound yourself helps as a starting point. It is often hard to find words for the sound qualities you are wishful for - but you could try to ask the following questions:
-How do you want people to feel when experiencing your piece?
The range can be broad, but you can begin mapping it out. Are you looking for something that feels positive, dark, moody, sad, or relaxed? Pretty much any emotion you can think of is a great piece of direction.
-What kind of design?
When speaking of design, the sound could be (for example) organic by design. By which you could suggest that the sound is grounded, or based on natural elements like water, fire, earth, wind, glass, metal, or wood. A contrary could be synthetic sound - for which there may be a synthesis of sounds. Such sound is often created for sci-fi textures.
For something more high tech, texture can be created by manipulating sound waves. In terms of language around the sound itself, different frequencies of the same sound can communicate different feelings. Lower frequencies can be calming, while very high frequencies can be unnerving. It’s common, but may not always be the case. It can be considered useful though while mapping out experiments.
-References are always great!
Provide your sound designer with some references. There could be vocal examples. Try making the sound using your voice. This may feel silly but it's very legitimate to use your voice to suggest your sound design requirements. The entire ambience, the bass of the museum, is in fact made using just Em’s breathing!
-Learn the names of some simple effects
Several sorts of effects exist and are spoken of colloquially. Learning their names is a quick trick of the trade. Like ‘reverb’ or ‘delay’ - which often translate sound into magical or otherworldly effects.
-Descriptors or Textures
Using some descriptors like ethereal, floaty, tactile, realistic, grounded, crunchy, aggressive, impactful, or subtle could give your sound designer some context. You can also use words that convey emotional tone, so as to describe the sound you’re seeking.
---
Em spoke about Immateria by Cabibbo, which serves as an example of adding sound to something abstract. The piece is so texturally rich and outlandish, that it could be made to sound like anything. For Em the starting point was tone - and what the piece makes her feel.
“I almost always start my design with organic sounds. So I recorded a bunch of weird stuff with some chimes and a banjo. To make it sound otherworldly. I reverse a lot of the sound out of a ton of reverb and effects. These are just looping sounds, but what you can't hear in this video is that the piece is reacting to a voice which could be both premises. But there are always technical restraints with experimental art and we weren't able to get the sound tied to that reaction. So I tried to design this with as much movement as possible.”
Therefore, a sound designer thinking spatially thinks about the experience of hearing what is in front of and behind you. Sound positioned, as such, is known as a scatter sound. The sounds in Cabibbo’s piece randomly sweep in, to and fro, in all directions. There are multiple spots on the string piece, which amplify the same sound as the viewer moves. It's made to be narrow so that you can kind of follow it along. And while all these seemingly separate pieces are combined into one installation, making use of similar source material - they're manipulated and recorded in different ways to capture various movements (while remaining cohesive). Wherever possible, Em has added spaces that let the viewer discover the feeling of difference as they get entered. The cubes in the piece are where Em wanted to bring in a cozy feeling, which closes in as a contrast to the open space of the rest of the piece. There are instances of spot sounds that communicate movement, like a stream of bubbles. The viewer can float around in the bubbles, and can also hear them as you approach them - which makes you enter their stream and start floating with them.
A piece that Em created the art and the sound for, called Capsule is an example of another application of sound for VR, that opens up the conversation of recording techniques. She made use of binaural microphones for this piece. These microphones record sound relative to where the microphone is, or the listener is by going into each ear. As they are designed to record how humans hear, they take the human head into account.
Em’s piece Capsule really makes a case for how these microphones are at capturing the essence of a memory, because it sounds like you're listening through someone else's perspective. As the slivers of stories held by Capsule are fleeting and fragmented - like in a memory casket - they embrace the listener in a manner that makes you tune into a shard of recorded time. Em adds that these may not always be suited for VR. But as Capsule was created gesturally to suggest that the bubbles in the sculpture are spaces for listening, the spatial aspects of the piece come together because of a decision to stop and let the memories take a sonic hold.
Ambisonics is a technology that was created back in the 1970s, yet there was never a commercial use for it. That being until VR and 360 videos came along - which as formats in themselves require 360° audio. The basic approach of ambisonics is to treat an audio scene as a full 360° sphere of sound coming from different directions towards a center point. This also includes locating sound sources that are above or below the listeners ear-level. To be able to listen to ambisonics, the listener/sound designer needs a decoder. The decoder that Millie Wissar uses is called Dear VR Pro, Ambi Micro. The MOR has a decoder of its own too, which is why listening to ambisonics inside the museum with headphones is made possible.
Millie worked on bringing the sound of the film How Was Your Day into the MOR for the show in November 2020 titled Body Clock. The short film by Edward Madojemu is a 360° sci-fi animation, which requires 360° sound. Millie had been wanting to work on ambisonics for a while, and she shared how this project was a great way for her to learn ambisonics and immersive storytelling. Despite being worked on in this manner, Ambisonic sounds feel and sound way more natural than sound in the real world. This, she explained, is because of how when we step into the real world, we are surrounded by ambiences that keep us in their hold even if we look away.
“The work begins with ambience in the scene. The footsteps of the characters running, the robot jumping, the machines moving. The ambiences are able to glue all the smaller sound effects together. They also tell you where you are, and set the mood. They give you information about the location. You could be looking anywhere, but the ambiences will stay enveloped around you.”
Well-designed ambiences can essentially form a detailed narrative about the scene independently. The plugin used for putting together the sounds for How Was Your Day is called Dear VR Pro, alongside Reaper, which is a software Millie uses to edit sound. Once the sounds felt ready, she put all the render files with the video player back in Virtual Desktop. It was important to reference in VR and not using a flat screen, to be able to tell the distances of the objects moving in space. You need to be in VR to reference VR! Following some research, she found that this setup would be ideal when working with visual cues and if ‘mixing’ is involved. Mixing is when sound designers move sounds around according to what can be seen in the picture and then balancing the sounds together so that they make sense in the story. Another tool that can be useful (that Millie mentions in the talk) is called Dear VR Spatial Connect. While she didn’t make use of it, she shared what it can do for a similar scenario involving mixing. It is designed to connect your headset to your editing software. So, one would still need to have Dear VR Pro and Dear VR Micro to be able to listen to the ambisonic files binaurally and to be able to pan the objects in 360°.
Millie shared a scene from Madojemu’s film where there is a crash, to show how she applied her research about these tools. While the scene visually omits the actual crash (with a black screen) and transitions to a frame where the characters end up after the crash, this scene is important for how it closes the circular narrative. The fact that in this scene, there was no given visual cue to follow, it gave Millie an opportunity to creatively play around with Dear VR Pro. She took sounds of the crash and moved them around ‘the head’ of the listener using this tool. She did this a bunch of times, recorded the results, and then applied the one that made the closest sense to the story and the details in the scene.
What is important to note is that when it comes to sound in an immersed narrative, the experience cannot be keyed as ‘watching’ a piece. The viewer is ‘entering an experience’ and ambisonics sound is capable of enhancing that feeling of entering and being enveloped. Millie confirmed, in conclusion, how all of this theory actually works.
The advice from both our guest speakers privileges playing with sound through immersed observation. Using some of these techniques and tools could surely serve as a technical starting point, but step zero (applicable irrespective of VR) is the resolve around becoming an effective communicator, developing awareness around the referential feelings that can conjure and amalgamate in space for a listener to brush shoulders with.
