Some of the different builds dont have controller support.Then I think there is a build that I ran that you have to replug the controller back in for it to be recognized.I know xUbuntu you have to have the mouse.I just ran Xebian the other day but for the life of me I can not remember if I could use a controller or not.
I can tell you that xdsl runs everything just fine and is twice as fast as anything else.I would try to run it and see if you still have the problem.
