Ever since our apartment got a 7.1 surround sound home theater setup, I thought it would be pretty cool to find a way to create a surround sound mix to listen to on our system. I'll start off by saying that I haven't found a complete solution yet, but I'll use this post to outline a lot of the open source tools I've tried and hopefully provide a starting point for your own research if you want to try something similar.
I normally use Pro Tools LE on OS X as my main DAW, but surround mixing costs a lot more through the DV toolkit or the Complete Production Toolkit. I've been looking for a reason to try out the open source DAW Ardour (http://ardour.org/), and it seemed that by being so open, there should be a few plugins and options that could extend it to create 5.1 surround sound.
Ardour supports multichannel outputs from every channel, but the multichannel panner has some odd behavior. After a decent amount of experimentation, I was never able to get it to work the way I wanted. Ambisonics, a surround format based on XYZ location and uses a decoder to apply the format to specific speaker configurations, seemed like a possible solution. The format seems to work best in quad or octal speaker configurations and I considered trying to use ambisonics for the front left, front right, and surround left and surround right speakers. This would help make it easy to do the music and effects, while having manual control of the center and LFE effects channels. The downside of the ambisonics approach was that I couldn't get any of the decoders to work. There are some plugins and encoders that were included with ardour, but I couldn't figure out their use. this page: ( http://ardour.org/node/2804) gives some good info on using ambisonics, but I couldn't get the ambdec decoder to install from source on OS X.
After all the exploration, I decided to record a few guitar tracks and use ac3jack (http://sonosaurus.com/ac3jack/) to create a 5.1 ac3 file. Using jack (http://jackaudio.org/), I routed the individual tracks to different surround channels. It was very easy to create the ac3 file, but I could not find a way to create a DVD to play the surround mix on our home theater setup. iDVD and iMovie only support stereo audio and the cheapest solution I found seemed to be Roxio Toast Titanium 10. DVD audio tools (http://dvd-audio.sourceforge.net/) looked promising, but I didn't try it out. If anyone has an easy solution to burning a surround encoded file to DVD for playback, let me know. I'll keep researching and experimenting and I hope this posts serves as a good jumping off spot for anybody who wants to enjoy creating surround sound on open source software.