Ok, an answer to my own question

I've managed to get AC3 sound working too:
1. I've set AC3 Filter (1.01a) to the 3/2+SW 5.1 channels profile
2. To increase the volume I've set the master gain to max. (20) and the voice and surrounds gains to 10
3. I edited the Transcode 360 16x9 profile to use the
Windows Media Audio 9.1 Professional codec
4. I set the audio format to 384 kbps, 48 kHz, 5.1 channel 24 bit CBR
I also set the video resolution to 3500K bps (giving a total of 3893 Kbps).
And since I'm using it mostly for US TV-shows, I've set the frame rate to 23.976 fps.
My P4 3.2 HT Ghz that does the transcoding runs at about 60% now (on both logical CPU's).
I have tried this, and I still don't get AC3 5.1 on my receiver! just stereo PCM.