Diable firewall software if you have it running on your own PC (i.e. Norton Firewall, Zone Alarm, just to name a couple). Describe your network setup in more detail...what type of router are you using, are you using static/dynamic IPs, have you tried to connect to your XBOX from OUTSIDE your home on a remote network, etc.
Also, how do you know you're even connecting to your XBOX if you cannot "see" any files? If you're running EvolutionX as your dashboard go into System Utils/settings and verify that FTP is enabled and that you're FTP'ing into the correct IP address.
File/directory listing from FlashFXP when you're connected from your own LAN should be instant, no delays whatsoever. If you have some sort of whacked firewall setup that would cause FTP problems. Zone Alarm will prevent FTP access unless you have your XBOX IP listed as part of the "safe zone" in the Zone Alarm config.