VidWorks Entertainment New Developer Training Program

Tutorial #7

Sound & Music

It's official. We're done with graphics. I believe we've taught you everything we can. From here on, it's all up to you. At least graphics-wise it is. Although once in a while, maybe Dwayne might drop in with some clever new technique. Maybe.

Anyways, onto Sound & Music!

Sound

For sound and music, we'll be using the Java Sound API, which is already part of the JDK and JRE. The Java Sound API is very easy to use, and at the same time, very powerful. You can find it in packages javax.sound.sampled (for WAVs, AIFFs, AUs, and eventually, MP3s) and javax.sound.midi (for, um, MIDIs).

The core of the sampled sound package is a class called AudioSystem. As far as you and I are concerned, we will always be dealing with this class statically and as a utility for getting access to other sound resources.

Now, usually, we'll be playing sounds off a file. On Windows, this is usually a WAV file. In order to read the file off the disk, we'll be using something called an AudioInputStream. The AudioSystem class has this method that we can use:

static AudioInputStream getAudioInputStream(File file)

But it's not enough to just have opened an audio file. We need something to play that file for us. In Java Sound, we'll use an object called a Clip. We can obtain a clip using the following AudioSystem method:

static Clip getClip()

Now, we have an AudioInputStream and a Clip. How do we load the AudioInputStream into the Clip? With this Clip member function:

void open(AudioInputStream stream)

Finally, to actually play the sound:

void start()

Note that the start method sets the sound to play in the background. The current thread can continue to run while the sound is playing. Also note that the sound playing doesn't constitute an actual thread, and if all other threads end, the sound will automatically end as well.

To stop the sound, we have this method:

void stop()

Note that the stop method only stops the sound. It doesn't rewind the position, so if you start the sound back up again, it'll continue playing from where it left off. It's more of a pause than the stop. If you want to reset the position, you can always use:

void setFramePosition(int frames)

void setMicrosecondPosition(long microseconds)

where the first position is zero.

Also, after you're done using the Clip, be sure to close() both the Clip and the AudioInputStream.

Anyways, a code sample:

/* SoundDemo.java */

import java.io.*;

import javax.sound.sampled.*;

public class SoundDemo {

public static void main(String[] args) {

try {

// get the sound clip

AudioInputStream myStream =

AudioSystem.getAudioInputStream(new File("tada.wav"));

Clip myClip = AudioSystem.getClip();

myClip.open(myStream);

// play the clip and wait

myClip.start();

System.out.println("Playing sound...");

Thread.sleep(3000);

System.out.println("The thread wakes up...");

// close the sound clip

myClip.stop();

myClip.close();

myStream.close();

} catch (Exception e) {

System.out.println(e.getMessage());

}

}

}

(No screenshot will be provided since this is command line and is very non-graphical. In fact, I foresee not having any more screenshots for the rest of the series.)

See? I told you Java Sound was very easy. Anyways, the code sample has some println's to demonstrate that the playing of the sound occurs outside of the current thread of execution. Thus, we have to put the thread to sleep if we don't want the Clip to immediately stop after starting up again.

Playing Sounds Simultaneously

This is a very trivial extension. If we want to play two sounds simultaneously, all we have to do is open up two AudioInputStreams and two Clips, and Java will automatically do it for us (assuming our hardware can handle it--most can).

I'll jump right into the code sample:

/* SoundDemo2.java */

import java.io.*;

import javax.sound.sampled.*;

public class SoundDemo2 {

public static void main(String[] args) {

try {

// get first sound clip

AudioInputStream stream1 =

AudioSystem.getAudioInputStream(new File("tada.wav"));

Clip clip1 = AudioSystem.getClip();

clip1.open(stream1);

// get second sound clip

AudioInputStream stream2 =

AudioSystem.getAudioInputStream(new File("scream.wav"));

Clip clip2 = AudioSystem.getClip();

clip2.open(stream2);

// start both clips

clip1.start();

clip2.start();

// wait while sounds play

Thread.sleep(3000);

// close the sound clips and streams

clip1.stop();

clip2.stop();

clip1.close();

clip2.close();

stream1.close();

stream2.close();

} catch (Exception e) {

System.out.println(e.getMessage());

}

}

}

Playing Sounds Sequentially

In order to play two sounds, one right after the other, we need to know when the first sound ended. There are two ways to do this: the first is to ask how long the first sound is, then Thread.sleep()'ing for that exact amount. Unfortunately, that's rather cumbersome and clunky.

Fortunately, there's a much better way. Remember events? That's right, we can get an event when a sound stops (or starts, is opened, or is closed for that matter).

First, we need to have our class implement the LineListener, which has the following method:

void update(LineEvent event)

The LineEvent object has the following methods:

long getFramePosition() - returns the next frame to be played, orr AudioSystem.NOT_SPECIFIED

Line getLine() - note that a Clip is a sub-class (well, technically sub-interface) of a Line

LineEvent.Type getType() - can be LineEvent.Type.OPEN, LineEvent.Type.START, LineEvent.Type.STOP, or LineEvent.Type.CLOSE

Using these 3 methods, we can find out which Line (in our case, Clip) generated the event, what kind of event it was, and where in the audio is the Clip.

To register our class as a LineListener, we can call the following method on our Clip:

void addLineListener(LineListener listener)

Now, we know everything we need to know to play sounds sequentially, so I'll give you this code sample:

/* SoundDemo3.java */

import java.io.*;

import javax.sound.sampled.*;

public class SoundDemo3 implements LineListener {

AudioInputStream stream1, stream2;

Clip clip1, clip2;

// returns true if successful

public boolean initDemo() {

try {

// get first sound clip

stream1 =

AudioSystem.getAudioInputStream(new File("tada.wav"));

clip1 = AudioSystem.getClip();

clip1.open(stream1);

// get second sound clip

stream2 =

AudioSystem.getAudioInputStream(new File("scream.wav"));

clip2 = AudioSystem.getClip();

clip2.open(stream2);

// add as a line listener

clip1.addLineListener(this);

clip2.addLineListener(this);

} catch (Exception e) {

System.out.println(e.getMessage());

return false;

}

return true;

}

public void update(LineEvent event) {

Line eventLine = event.getLine();

LineEvent.Type eventType = event.getType();

if (eventLine == clip1 && eventType == LineEvent.Type.STOP) {

// clip 1 has stopped, start clip 2

clip2.start();

} else if (eventLine == clip2 && eventType == LineEvent.Type.STOP) {

// clip 2 has stopped, clean up and go home

clip1.close();

clip2.close();

try {

stream1.close();

stream2.close();

} catch (IOException e) {

System.out.println("Unknown error while closing streams!");

} finally {

System.exit(0);

}

}

}

public static void main(String[] args) {

// initialize the demo

SoundDemo3 demo = new SoundDemo3();

if (!demo.initDemo()) {

System.out.println("Error initializing demo!");

System.exit(1);

}

// start the first clip

demo.clip1.start();

// give the sounds time to play

// if they don't give an event when they stop, timeout after 15 seconds

try {

Thread.sleep(15000);

} catch (InterruptedException e) {

System.out.println("Unknown error in main()");

}

}

}

First of all, note that we had to move almost everything out of our main() method. Why? Because only an instantiated object can be a LineListener, so we had to move everything to a non-static context.

Just a brief explanation: in our update() method, if we detect that clip 1 has stopped, we play clip 2. If we detect that clip 2 has stopped, we close all our Clips and AudioInputStreams and quit.

Finally, notice that we have to sleep at the end of our main() method. This is because the threads that play the sounds aren't actually threads, so if main() ends, the program ends and the sounds stop. There is most likely a way to change this behavior. However, that is outside the scope of this tutorial and will be left to the reader if he or she is interested in the subject.

Looping Sounds

What if you want to play a sound in a loop? Well, we could just wait for a sound stopped event and then reposition the Clip and start it up again. However, there is a much simpler solution that's actually more powerful.

The Clip object has these following loop related methods:

void setLoopPoints(int startFrame, int endFrame) - the first frame is 0; you can also use -1 for the end

void loop(int count) - loops for the specified number of timess, or loops continuously if count is equal to Clip.LOOP_CONTINUOUSLY

Now, often times, we don't know anything about the frames in the sound file, only about the time. Ideally, we should be able to specify loop points in terms of microseconds or something. Failing that, we could just use these Clip member methods to derive a conversion from time to frames:

int getFrameLength()

long getMicrosecondLength()

Now, you should know enough to understand this code sample without any explanation:

/* SoundLoop.java */

import java.io.*;

import javax.sound.sampled.*;

public class SoundLoop {

public static void main(String[] args) {

try {

// get the sound clip

AudioInputStream myStream =

AudioSystem.getAudioInputStream(new File("tada.wav"));

Clip myClip = AudioSystem.getClip();

myClip.open(myStream);

// calculate frames per second

double fps = myClip.getFrameLength() * 1.0

/ myClip.getMicrosecondLength();

// set the loop points

// we want to loop from 0.5 seconds to 1 second

// (500,000 microseconds to 1,000,000 microseconds)

int startFrame = (int) Math.floor(500000 * fps);

int endFrame = (int) Math.floor(1000000 * fps);

myClip.setLoopPoints(startFrame, endFrame);

// loop the clip and wait

myClip.loop(Clip.LOOP_CONTINUOUSLY);

Thread.sleep(15000);

// close the sound clip

myClip.stop();

myClip.close();

myStream.close();

} catch (Exception e) {

System.out.println(e.getMessage());

}

}

}

Playing MP3s

Everyone likes MP3s. It's decent quality music at a reasonable size, just like the MIDI of 10 years ago. Honestly, it's a fair comparison. For anyone who doesn't believe that MP3s will fall out favor in 10 years, look up Lossy Compression on the internet.

Now, issues aside, we'd like very much to be able to play MP3s in our Java applications. Unfortunately, Java by default doesn't provide any support for MP3s. Fortunately, Sun released an MP3 plug-in in November 2004. I know it says Java Media Framework on it, but the plug-in also grants this MP3 ability to Java Sound as well.

But wait! You just installed the plug-in and took SoundDemo.java and replaced tada.wav with [your favorite MP3].mp3, and it said not supported. What gives?

Well, if you look at the error message more closely, you'll see that it's because it has an unknown frame size and bit rate and stuff like that. This is because the MP3 is still encoded. We need to decode the MP3 into a more playable form.

First, let's get the format of the current AudioInputStream. Yes, I am aware that the error message claimed that Java Sound doesn't know anything about the format, but it left out a few details--namely sample rate and number of channels. We'll need these numbers to generate new numbers for the format.

Now, the format is stored in an object called AudioFormat, which you can retrieve from the AudioInputStream by calling

AudioFormat getFormat()

Next, we want to create a new AudioFormat based on the old AudioFormat. The AudioFormat constructor of interest is:

AudioFormat(AudioFormat.Encoding encoding, float sampleRate, int sampleSizeInBits, int channels, int frameSize, float frameRate, boolean bigEndian)

Now, once we create a new AudioFormat, we can decode our AudioInputStream by calling this static method on AudioSystem:

static AudioInputStream getAudioInputStream(AudioFormat targetFormat, AudioInputStream sourceStream)

where targetFormat is the AudioFormat we just created and sourceStream is our original AudioInputStream.

Using this information, we can in general use the following block of code to do all our MP3 related work in Java Sound:

AudioFormat myFormat = myStream.getFormat();

AudioFormat decodedFormat =

new AudioFormat(AudioFormat.Encoding.PCM_SIGNED,

myFormat.getSampleRate(),

16,

myFormat.getChannels(),

myFormat.getChannels() * 2,

myFormat.getSampleRate(),

false);

AudioInputStream decodedStream =

AudioSystem.getAudioInputStream(decodedFormat, myStream);

Be sure to open the decoded stream rather than the original stream.

Now, splicing these 3 lines of code into SoundDemo.java will get us this code sample, which plays MP3s!

/* MP3Demo.java */

import java.io.*;

import javax.sound.sampled.*;

public class MP3Demo {

public static void main(String[] args) {

// command line output so it looks like the program is doing something

System.out.println("Loading MP3...");

try {

AudioInputStream myStream =

AudioSystem.getAudioInputStream(new File("[your favorite MP3].mp3"));

// this is the extra work it takes to decode the MP3

AudioFormat myFormat = myStream.getFormat();

AudioFormat decodedFormat =

new AudioFormat(AudioFormat.Encoding.PCM_SIGNED,

myFormat.getSampleRate(),

16,

myFormat.getChannels(),

myFormat.getChannels() * 2,

myFormat.getSampleRate(),

false);

AudioInputStream decodedStream =

AudioSystem.getAudioInputStream(decodedFormat, myStream);

// load the MP3 into a clip and play it

Clip myClip = AudioSystem.getClip();

myClip.open(decodedStream);

myClip.start();

Thread.sleep(60000);

System.out.println("Cutting MP3 off short!");

myClip.stop();

myClip.close();

myStream.close();

// note the extra stream that we need to close

decodedStream.close();

} catch (Exception e) {

System.out.println(e.getMessage());

}

}

}

It should be trivial to play two or more MP3s in sequence or to loop an MP3.

Also, I'm not providing you with an MP3 to play with for this example, so be sure to replace [your favorite MP3] with, um, your favorite MP3.

Playing MIDI

Many of you young ones laugh, but MIDI is still useful. (Why else would Java Sound still support it?) Unless you're shelling out dollars on a server or a real host and assuming you're not still on dial-up (which a surprisingly large number of people are), MIDIs are good for saving disk space and bandwidth. In the disk space you could put a single MP3, you could have anywhere from 50 to 250 MIDIs. Wait until today's MP3 becomes tomorrow's MIDI, and the real MIDI is good and dead. Then, you can laugh when you read this section.

(And, you know, somehow I doubt MIDI will truly die, because most amateur composers don't have access to an orchestra. Plus, if you shell out about a thousand dollars on a good keyboard, MIDI sounds pretty damn close to godly. Everyone in Trance and Techno does it.)

In any case, I should get off my soapbox and continue with the tutorial.

javax.sound.midi parallels javax.sound.sampled in many ways. Where sampled sound has AudioSystem, Java MIDI has MidiSystem. In sampled sound, we needed a sound file (AudioInputStream) and the resource to play that file (Clip). In Java MIDI, we also have MIDI files (Sequence) and MIDI players (Sequencer). (Be sure not to confuse a Sequence with a Sequencer. I'm sure everyone can see that they are very different, but typos still occur.)

Anyways, to get a Sequence from a MIDI file, we can ask the MidiSystem using the following static method:

static Sequence getSequence(File file)

We can also use the MidiSystem to get a Sequencer:

static Sequencer getSequencer()

Now, unlike a Clip, a single Sequencer can be used for multiple MIDI Sequences. We don't actually need to pass it the Sequence until after we've already opened the Sequencer. First, call open() on the Sequencer to reserve its resources. Then, we can load the Sequence with this:

void setSequence(Sequence sequence)

setSequence() can be called as many times as you'd like in the lifetime of the Sequencer. This makes playing several different MIDIs a lot easier than playing several different Clips (although never at the same time).

Finally, the Sequencer has start() and stop(), just like a Clip. They even have the exact same behavior, namely when the Sequencer starts playing, it plays the MIDI Sequence in a separate pseudo-thread, and the stop() method acts more like a pause than a stop.

Finally, I'll show you the code sample. You'll see that Java MIDI parallels sampled sound very well.

/* MidiDemo.java */

import java.io.*;

import javax.sound.midi.*;

public class MidiDemo {

public static void main(String[] args) {

try {

// get Sequencer

Sequencer mySequencer = MidiSystem.getSequencer();

mySequencer.open();

// get MIDI file

Sequence mySequence

= MidiSystem.getSequence(new File("DwayneJeng-Donuts.mid"));

mySequencer.setSequence(mySequence);

// start the MIDI and wait

mySequencer.start();

System.out.println("Starting the MIDI...");

Thread.sleep(60000);

System.out.println("Cutting MIDI off short!");

// stop and close the MIDI Sequencer

mySequencer.stop();

mySequencer.close();

} catch (Exception e) {

System.out.println(e.getMessage());

}

}

}

Finally, like sampled sound, MIDIs can loop as well. Here are the related Sequencer methods:

void setLoopCount(int count) - can be set to Sequencer.LOOP_CONTINUOUSLY to loop continuously

void setLoopStartPoint(long tick) - note that a tick is usually equivalent to a quarter note

void setLoopEndPoint(long tick) - the MIDI Sequence starts with 0; -1 can be used for the end of the Sequence

In addition, in MIDI, there are event handlers. However, these events are a lot more complicated than the events for sampled sound and generally include MIDI Events. (Yes, everything in a MIDI Sequence is technically an event.) You can look up ControllerEventListener and MetaEventListener if you're interested, but these interfaces are outside the scope of our tutorial.

Conclusion

That concludes the sound and music tutorial. Now, you know everything you need to know about the basics of sound and music in Java. Next time: Networking!

Homework Assignment!

Add sound effects and a music soundtrack to one of the graphical programs you developed in one of the previous tutorials.