I am actually looking into setting up a similar setup. I currently have a Debian Linux box with ~1.5 TB of hard drive space (not raided, just jbod) that all my storage sits on. All disks are IDE whichI am growing tired of and am looking to turn a mini itx box I have sitting aorund into a NAS.
I think your setup is actually overkill for just a file server. I understand that the CPU is going to be doing XOR computation, but thats still overkill in my opinion. I'm gonna have a 1ghz cpu with 256mb ram doing pretty much the same thing. I already bought a hotswappable sata enclosure (
link) so I'm just gonna need drives to fill it up with and a sata-raid controller. Looking at getting 4x500GB (SATA-II of course) or more, depends how rich I feel at the time.
As for the OS... There are free NAS/SAN unix-based OS's out there. Check out
FreeNAS (which is based on FreeBSD) and
OpenFiler Linux. They both support pretty much any transfer protocal you might want to use and I think they both support Active Directory integration, but for home use you probably wont need that. Everything is set up web-based besides for an initial setup I believe.
If you are looking into exploring linux a bit more and want to set up everything manually with a regular distro of linux I'd say try out Ubuntu 6.06. I use Debian which is similar, but many people find Ubuntu easier for new linux users. There is plenty of help out there on Google for it and it can get you on your way pretty quickly.
As soon as I get my hardware I can provide a little more insight on both Openfiler and FreeNAS.