I would suppose the answer you're looking for is this:
N64 emulation is not 100% finished and perfected by any means. However, it is a superb way along, and obviously we all reap and enjoy the software to its [current] fullest capabilities.
N64 emulation improves by the furthered work of the coder(s), and increased understanding of the hardware that is being emulated attained by the coder(s). So, there's still more to learn, and still more to attempt to implement to attain further progress in N64 emulation.
Now, the aim for emulation isn't necessarily speed as you assume (playability on everything), but rather perfection (for it to run everything frame-by-frame as it should). Possibly better explained, to emulate every last facet of the hardware under consideration - in this case, the N64. Obviously, this is quite tricky!
Now, although being a relatively powerful console, the XBOX does chug when using high-end filters on the N64 emulator, and it always will. However, the xbox clearly can provide playable framerates, and can very much be fun!
The hardware limitations exist, but they aren't so severe to keep you from playing. This means that there are also many software limitations, or "bugs" as you want to call them, to work out. Theonly way this is fixed is mentioned above. So rendering/sound problems are not a result of xbox deficiencies. But don't make haste to call the coder(s) deficient in any way! They are working hard and doing a spectacular job.
All you can do is either lean a lot about programming and the system's hardware and help with the code to speed up the process, or just sit tight and enjoy what you have now.
I hope this is what you were asking, and I answered your question.
-Bender