Not sure about your TV but on mine, only certain inputs support HDTV.
Additionally, I'm sure you know this but I'll mention it for reference, you need a NTSC xbox - or have used the video switcher util to switch to NTSC mode.
PS, I can/have switched to the 480p, 720p & 1080i modes on XBMC settings screen without problem - however, on my older plasma, the 720p & 1080i modes would be garbled - it didn't support them - garbled screen is a good sign it is not supported or you are using wrong inputs.
My TV: Hitachi 42pd5200
Input: Component (AV1)
XBOX: Pal V1 - set to NTSC (with HDTV enabled in dash)
XBMC Resolution set: 720p
Screen Quality: Fantastic!
Hope this helps. <