I've got a similar problem.
Im on a SDTV 480i, with component inputs (btw there's a difference between composite/yellow and component even on a SDTV, less bleed and easier to see text)
Starting my xbox regular (with regular ms dash) it works fine, plays all regular games fine with dolby digital by using the Xbox brand component cables and a optical cable.
The problem occurs when I start up my xbox using Unleashx or xbmc. The X green flubber screen is fine but then it changes out the resolution to one my TV cant handle, the result is black and white lines.

Audio works fine though my optical cable
Ive been though all the setting and disables all settings over 480i. Ive changed the settings in the MS dash, in UnleashX, and XBMC. Everything is set to Ntsc and standard or 4:3 resolution.
BTW everything works fine when using the standard A/V composite cables that came with the xbox.
I just want to get either unleashx, or xbmc working with the component cables.