Found this on a site when I was helping out someone with this same issue, hope it helps...
The PS3 will only output surround-sound to one port. It will output 5.1 or 7.1 to HDMI, if your TV tells the PS3 that it supports surround-sound. The PS3 will also output 5.1 to the digital-out (optical) port of the PS3. If your TV is capable of outputing surround-sound, then it should already be connected to the surround-sound speaker system; and the PS3 will send the surround-sound signal through the HDMI port. If your TV does not output surround-sound, then you need to connect the surround-sound system directly to the PS3; either via optical cable or via HDMI using an AV receiver that supports HDMI pass-through. If you have an HDMI pass-through receiver, then you just connect the PS3 to an HDMI input on the receiver, and the connect the HDMI output on the receiver to the TV.
If you use an optical cable to connect to a home theater system or an AV receiver, then you need to change the audio output settings to send the PS3 audio to the optical port. In the XMB, select (Settings>Sound Settings>Audio Output Settings>Digital Out (optical)). After you do this, there will be no sound out of the TV at all when you are displaying PS3 video. An AV receiver with HDMI pass-through is the best option, as that is the only way to get 7.1 surround-sound. The digital-out port is good enough for 5.1 surround-sound. You can't get surround sound with just one set of RCA (red and white) jacks at all. You need several RCA cables to get 5.1 surround-sound, but the PS3 doesn't support that configuration directly.