There are so many variables that I really couldn't tell you what your problem is, but I think I do have an idea though.
How are you connecting to your XBox?
If you're going from the computer to a router, and then from the router to the XBox, then I'm not really sure what the problem is, but if you're trying to go stright from your computer to the XBox you need a crossover cable. If you don't have a crossover cable, it's possible to make your own with the cable you've already, but you'd have to Google that since I've never tried it before. Probably easier just to go through a router or buy a crossover cable from newegg.com.
Word of advice to ya... not sure which dashboard you're using, but in every one there should be a way that you can check inside the settings to see which IP address your XBox has been assigned. That should make determining if you even have a valid connection in the first place anyhow. Plus you won't be pissing in the wind when you're inputing the IP address in the FTP program either.
Also, you are aware that you have to use the username xbox and password xbox, right?
Couldn't tell you anything about Vista except that I would downgrade if at all possible, lol. Good luck!