i ran into a simililar problem when i first modded my xbox.
(Note: this gets kinda technical. I'm basically saying to have a direct connection between the computer and xbox, or to use a SWITCH!!!! ...and to use XBMC.)
I have a linksys HUB. (5 port) (I should get another switch for this though)
connected to the hub is my laptop computer, my Desktop computer, my modded xbox, and my unmodded xbox. (for Xbox Live

)
on the UPLINK port I had my connection to a SWITCH which connected everything else in my house, including the internet.
anyways, connecting to my modded xbox from either computer was really
slow.
i finally figured out that what i should do is:
only have my modded xbox and the connecting computer connected, and the UPLINK.
I turned on the xbox and computer, let them get their dynamic IP addresses from my DHCP server (router). I disconnected the UPLINK port.
Now the only things on the HUB were the xbox and the computer, and they had their IP addresses, and transfers were really fast. (9mb/s) if i plugged in another cable, during mid-transfer, the speed would shoot down to about 20kb/s.
so whenever i transfer files, i just disconnect everything else on my hub, (but making sure that both xbox and comp have a valid IP address first) and it works great.
I once used my 24port switch just to test it, (because switches have a "smart" technology built into them) and had every port connected (yes, i am a geek big enough to have that many computers ) and had all those extra computers doing endless pinging (to create extra network traffic), and transfer speeds were not bad, 5-6mb/s
my advice to everyone: use a SWITCH when creating a network! Not just your router or hub. SWITCH's have "smart" technology that know where the data is going, so unlike a hub, it only sends the info there instead of to all of the connections.
i also found the XBMC has the most reliable FTP Server bulit into it: FileZilla Server.
I use the FileZilla Client on my computers to FTP.