Forgot to add the section about the characters, see below.
1. Set up - Extract the contents of the Xmugen-linux folder (mugen folder, linuxboot.cfg) to your E: partition. Put the default.xbe into a folder called Mugen (or whatever you care to call it) in your Emu/apps folder. DO NOT rename the original mugen folder that contains the chars, stages, etc as it must be called exactly "mugen". It will not load anything else.
2. Linux is case sensitive - EVERYTHING must be exactly typed as the file/folder is named. When I first tried this out, I had my mugen folder looking like this: "Mugen" . I then realized it was case sensitive so I renamed it to: "mugen" and it worked fine. Same goes for your characters. If the folder is called "SFA3 Ryu", then in your select.def it must reference "SFA3 Ryu", not "sfa3 ryu" or "SFA3 ryu".
3. Screenpack usage - We don't know why yet, but you can't reference a screenpack folder in your mugen.cfg motif option as you normally can, or else mugen will crash. For now, you can back up the files in the /data/ folder you want to replace (normally select.def, system.def, and sometimes fight.def, etc) to a backup folder or something, and then just copy in the screenpack files to /data/.
3. Display options - I'm running my mugen on a HDTV, using the standard 640x480 resolution in the linuxboot.cfg with no problems. Theoretically you should be able to get by the "press y to change resolution" screen by setting your display depth to 16 instead of 32, but for some odd reason this gives me an error.
4. Characters - If you're like me and you have a lot of characters, you probably want to copy over an existing Winmugen select.def to the xbox. Now if you've used a mugen gui to setup your characters (I use Mugen Windows Configurator) you'll notice that your characters won't show up once you boot the xbox mugen. Reason is, after running a windows based mugen config, your select file might look something like this:
"SF2 Ken\SF2 Ken.def,stages/SF2 Ken.def,order=7,music=,includestage=0;Ken (SF2)
SF2 Ryu\SF2 Ryu.def,stages/SF2 Ryu.def,order=5,music=,includestage=0;Ryu (SF2) "
Well in the linux mugen it should look like this:
"SF2 Ken/SF2 Ken.def,stages/SF2 Ken.def,order=7,music=,includestage=0;Ken (SF2)
SF2 Ryu/SF2 Ryu.def,stages/SF2 Ryu.def,order=5,music=,includestage=0;Ryu (SF2)"
The linux version doesn't like the backslashes so do a find and replace all to fix this.