In my opinion, using the 360 to watch low-def files is pointless when you have a modified Xbox (original). For low def files, Xbox Media Center can natively stream divx/xvid natively. The 360 cannot do this- you have to either re-encode your files or transcode them on the fly (which is sometimes problematic).
With that said, the original Xbox / XBMC cannot decode high definition files. The file types are supported, yes, but the processor is far too slow to decode the files.
The Xbox 360 is powerful enough to decode high definition video- but filetype support is shaky. You need to encode your files to WMV-HD, H264, or MPEG2 (I believe).
I prefer WMV-HD, as outlined in my tutorial below (signature). A friend of mine encodes on his Mac to H264, but I think my method gives slightly better quality. We're going to do some controlled testing on this soon to find out which of the 3 methods is the best, based on output quality:size ratio and ease of encoding.
Bottom line?
Low def files play natively, with no hassle on the XBMC. For high def, you need to do some converting and use the 360. This doesn't require modding your 360 though.