XBMC4XBOX does support both the x264 codec and mp4 containers, you'll just get bad shuttering when you get into HD resolutions. I really only experience crashing when using scripts/plugins like Navi-X, so that should not be too much of an issue for you.
The best way to play HD content is through UPnP using a program that supports transcoding. TVersity and PlayOn are two that I'd recommend. They both allow you to stream local content for free and virtually all files will play. Moreover, they can also be purchased to get near Navi-X like functionality. PlayOn really stands out in this aspect by supporting both major video sites and user made plugins.
You can also install more ram from a dead xbox. It's really worth it if you have the opportunity.