You will not get 5.1 out of the red and white...but you already knew that. But you already knew that. Most receivers have a different setup for digital and audio, but I'm assuming that you knew that as well.
Im guessing that your audio converter does not work. I used a CO2 for a while before I got a good receiver, and still may use it for long runs. You mentioned you have an optical cable hooked to the converter and a coax from there. Any power to the converter? You don't get something for nothing. Are you sure it isn't backwards? that is a coaxial source to a optical receiver? Any way to check this on a known working coaxial reciever and eliminate some options for error?
Setting the 5.1 flag for me just made my surround sound work, not enabled or disabled the digital connection. You still should get sound with the optical cable no matter the setting in the dashboard
On a side note, I'm surprised that optical has become so popular. On short runs, its lossier than a proper copper coaxial interconnect, and it's often cheaper for consumers for both the interconnects and devices. On longer runs (50' and more) optical wins by a long shot in quality but who can afford a 50' optical cable? ;-) While I wished I could have the 360 with copper, im afraid my receiver only has two coaxial inputs, so I''l use them for music and movies ;-)