As an audio designer my focus is players audio perception in interactive game environments. How a player is immersed into the game play and how sounds encourage the player to interact. I know that the topic of game audio is huge. There is not much research on the area and there are still many questions regarding subjects such as:

• Sound engines
• Adaptive music
• Dialogue
• How to avoid repetitions
• Player profiles

At Nordic Game Jam – we do not have the time to create a state-of-the-art First-Person Shooter – we have less than 48 hours. So that is the reason why I have chosen to focus on casual and edutainment games with examples from some of the projects that I have worked on.

This article is divided into 4 parts:

• Immersion & Interaction
• Casual games
• Edutainment
• Conclusion

My main objective is how you can tell a story with few and meaningful sound effects, jingles and music compositions, when you are working with game design and/or technical limitations. For instance if you are working on audio design for interactive TV applications – your technical limitation is the hardware. On the other hand – if you are working on audio design for a PC game for children – your limitation is the game design and the storage medium.

IMMERSION & INTERACTION
Why is game audio interesting? And how does player actually perceive audio?

Audio as a physical event
When we look at the graphic interface – it is a 2-dimensional representation or simulation of a 3D graphic environment, displayed on a flat TV or PC screen. But audio is not a simulation. Audio is in fact the only true 3-dimensional physical event that happens during game play. Audio has the properties of pressure and frequency, as it propagates through air and occupies an expanding volume. So the sound wave travels through air from the headphones or TV monitors to our ears. This physical event makes us react physically by making haptic spasmodic inputs on the controller.

Acoustic spaces
When we talk about audio as a physical event during game play – we also have to talk about acoustic spaces. There are 2 kinds of acoustic or resonating spaces that we perceive when we play games:

• The perception of a real resonating space – which is the physical space that player is placed in – listening to the output from TV or PC monitors.
• The perception of a virtual resonating space – which is the graphical space combined with the sound output from game engine.

It is the union of these two resonating spaces that makes player immerse physically into the virtual game. Meaning that the player is the centre of the gaming event as an immobile figure and s/he is in control of the acoustic space represented on screen.

Audio as a story through references
Audio is not only a physical event. It also gives us information based on our references. So it is audio as a physical event in combination with the referential elements of sound that makes us immerse into the game play.

In for instance First-Person shooters audio is very important because you have to use your audio perception skills to navigate to a certain place or to localise an enemy. As a game player we use audio as a cue to interact during the game play.

Some sounds have ‘alarm’ functions and some sounds have the function of putting us in a certain mood. So sound and music design can contain many elements of information that can be interpreted by the game player – either as something that player has to act upon or something that player has to feel or recognise.

There are many terminologies on the meaning of sound effects and music. But I will go through some of the different functions of sound and music supported by the screenshots and audio examples below.

CASUAL GAMES
Working on limited platforms such as the Set-Top-Box – audio is also important, because you have to tell a story with short and very precise sounds. As an audio designer you have to put a lot of information in a sound file that is less than 1 second in duration.

Another restriction is that the STB can only play one sound at a time. I will also in this article explain the implications of the absence of sound and what this means for players audio experience.

The controllers that players are using on this type of games is a TV remote, where they press arrow buttons and a ‘SELECT’ button to play.

Casual Games – Example 1
In the 1st example – the graphical setting gives us associations to the Wild West. The game play is like ‘Bejeweled’.

The player is “The Gambler” and has to play against different opponents. The intention is to make the opponent lose all his money – so when you make a match with gold bars or silver coins – the opponent loses money.

Gamblers Quest – Audio extract from game play

Sound design on this game is realistic recorded sounds used in a different context:

• Card flop: Executed when navigating on menu, this sound supports players action on screen. It supports the players control of the game
• Bottle toast: Executed when player gets a match
• Cash register – Executed when there is a match with gold bars and the opponent loses money

The jingle is composed imitating piano roll music and gives us instantly references to Western movies and/or silent movies. So the intention of the audio design is to give player a perception of being in a ‘saloon’ in a historical setting from the beginning of the 20th century.

This also means that sounds can give us information on where we on:

• A historical time line: It can give us a clue on whether we are in the past or present
• Location: Based on our references, we know that we are in the USA and not in China

But sound and music design also relies on player’s prior experience – if player has never seen a Western movie before or does not know where USA is, s/he would probably just interpret the sounds as being ‘realistic’ and will maybe not acknowledge the humour in them.

Casual Games – Example 2

Example 2 is from a puzzle game, where the visuals are a futuristic universe. As a player you have to turn some connectors to make the electric spark enter the receptor. When player enters the receptor – the level is complete.

Ben10 – Audio extract from game play

The audio design in this puzzle game is ‘imitated reality’ sounds. We do not know how the future sounds, so we imitate the reality. We make the sounds according to how we think it will sound like in the future.

There is a positive feedback sound, which is executed when the electric spark enters a receptor. The audio has an ascending pitch. The negative feedback sound, which is executed when the electric spark dies, imitates an electric circuit short-cut.

The most important sound is when player is rotating the connectors. The function of this sound is to give player instant feedback on the interaction that is performed on screen. The focus is to tell player that s/he is in control of the game.

This game can give player associations to other science fiction entertainment media, but if the player has seen the original cartoon episode – that this game is based on – then s/he will recognize the style of sound design and appreciate the similarity of the ‘mechanic’ and ‘electric’ sound universe.

Casual Games – Example 3

In this last casual game example the graphical setting is an open sea. The game is a puzzle and as a player you have to navigate to collect corks to get points. When you have collected the message bottle – you have completed the level. The challenge is to avoid dangers such as the pirate ships. This example is from the first level of the game, so I want you to focus on the absence of sound.

Flapjack – Audio extract from game play

All sounds are in cartoon style. The positive feedback sounds are:

• Musical bell: Executed when the adventure bar is full
• Cork: Executed when player picks up a cork. Sound character can be interpreted as realistic, but because it is used in a different context it goes into the group of ‘cartoon’ sounds

The negative feedback sounds are:

• Car horn: Executed when player attempts to go to a place outside the highlighted area. The sound has an alarm function.
• Canon shot: Executed by Pirate ship. The sound has an alarm function as it is the enemy that player has to be aware of and avoid

Jingle is composed using a single accordion instrument. It gives player references to the original cartoon episode or players prior experience from similar ‘sailor’ cartoons, movies or music, where the accordion instrument is used.

But the most interesting part in this audio extract is the long pause before the last sound execution. It is the absence of sound. Silence can sometimes be very effect full in games – like for instance in horror, but in this example it feels a bit tedious.

Basically my point is that it is important to think the sound into the game at a very early stage. Ask questions like:

• What impact is audio going to have on the game player?
• How can we integrate audio in the interaction in the best possible ways?

So it is important to think of audio as a merged part of game design and interaction – and not just an effect.

EDUTAINMENT GAMES
The next 3 examples are from an edutainment game from the Pixeline series developed by Krea Medie. In these examples I would like to explore how you can use music and ambience in 2D graphics and how you can create a spatial depth in the aural perception.

Edutainment – Example 1

The first example is a screenshot with an extract from the background ambience music. The story is about the girl Pixeline who is beamed up to an UFO. From the UFO she will help her friends to explore different parts of the human body – such as the blood veins, the ears, brain, heart and so fort. In this example I would like to focus on how the background music is adding a spatial depth to the 2D graphics.

UFO ambience – Audio extract from background music and sound

The background ambience is a mixture of ‘bleeping’ sounds and music. The intention of the ‘bleeping’ sounds is to create a ‘resonating room’ that supports the many buttons that are present graphically on screen – but also to support the flying mini UFO

My intention with the music was to add a spatial depth to the 2-dimensional graphics. To give listener a perception of an open space – a place with ‘no roof’. So I had to aurally stretch the 2D graphic representation into something that gives player the perception of a deep space. To create that feeling of open space – I used contrasting timbres such as ‘natural’ vs. ‘synthetic’ sounds in the music score.

Bright bell sounds and cymbals with long decay times where also used to create a sense of an ‘open space’. I wrote the composition in low and high frequency notes, and I also used quartal harmony to give the listener a perception of an open tonal direction.

Edutainment – Example 2

In this example Pixeline is going to explore the blood veins in the human body – and the game play is about navigating inside the blood veins and avoid things such as cholesterols.

Music supports the level design with fast and slow tempo:

• When blood veins are wide – music is in high tempo and player can move quickly
  Blood Veins 1
  

• When blood veins are narrow – music is slow and player has to be careful
  Blood Veins 2
  

The water stream sounds in the music score give player the perception of being ’under water’ and a perception of an ’enclosed’ space. There are also synthetic sounds in music score imitating ‘submarine’ and ‘whale’ sounds. This gives player references to action entertainment movies and cartoons. These musical effects also help player to establish a perception of being on an ‘underwater’ mission.

Edutainment – Example 3

This level is about the ear and how we perceive sounds. The game play takes place in a detective agency with 3 characters: the Brain, the Eye and the Ear. These 3 characters have to co-operate by guessing the sounds in the right order. My focus in this audio example is how musical instruments are used as sound effects in the music score.

Spy Game – Audio extract from background music

Music is composed using a harmonic progression that is very common for agent/action movies – such as James Bond or Bourne Identity. The intention is to give player references to these previous entertainment action and suspense movies with elements of a story line that has to be revealed or a puzzle that has to be solved. There is not much animation on the level – so to create a sense of being in an office – I added some percussion effects in music score. There is for instance the ‘brrrrrrr-ding’ sound that imitates an old type writer.

CONCLUSION
So what is the conlusion? We have established that audio is:

• A physical 3-dimensional event
• The physical gate through which player immerse into the virtual environment
• Telling stories through references

We should really think of audio as action. When we make inputs on the controller – we get an instant audio feedback from the game engine – that we again react and act upon through the controller.

Interactive audio design can add those extra details to the game play that can not always be said in words or in graphics. It helps us immerse to the virtual environment through the resonating acoustic spaces but also through the story telling elements that we interpret based on our references from real life or from other entertainment media.