Print

[heinrich]

QUOTE (Ace25 @ Nov 11 2003, 05:37 PM)

I already spoke with Paulb via PMs and he told me this is NOT a LBA48 issue and he thinks it may be a FATX issue. I absolutely agree with his conclusion. What I can't seem to find is documentation on FATX and how it works, file and/or directory limitations, etc. Anyone else have anything on FATX?

This is very possible, as anyone 'in the know' know's, there are limits to the # of files on a partition.  Unfortunately, the xbox, and the FATX filesystem is more an 'extension' of fat32 that allows longer filenames.  The XDK docs only give limited documentation on FATX as the devs of games dont really need to worry about these problems.  I think it is safe to assume that the limits are the same as FAT32.

FAT32 supports the follow:
4,177,920 TOTAL files on a "volume" (a partiton, this means 4.1 million for EACH of the drives C E F G X Y Z)
65,534 files/subfolders PER folder.  This means your F:Games folder can only have a max of 65 thousand files.  IE: 65 games with 1,000 files each.  Of course, games vary greatly in this, Project Gotham Racing 2 is over 10,000 files, while other games have less than 10.

Now, in my F:Games folder, I currently have 107,867 files in 5771 subfolders.  I have not noticed any problems with games.  Does this mean that there arent any?  I dont know, I havent played through every game 100% to know that they are problem free.
Even assuming that they are all ok, thats only in ~116gig of space, someone with a 320gig drive, which gives them close to 300gig on F would mean, with the same ratio of files:space, would have 278,966 files, which if there was a limit around there, I would guess it to be around the 256,000 mark (i know, this # is not exact, but I'm typing as im thinking, so just stfu )

QUOTE

i agree Ace25, maybe if we cannot find info on FATX is there a way we can impliment a different type of file system? Possibly FAT32?

As stated above, the limits for FAT32 are more strict as far as filename length, as well as total partition size, and # of files, this would be taking a step backwards.

Also, please 'newbs', if you dont really understand what is being said here, please dont respond, just wait and see where everyone gets with this problem.

[heinrich]

EDIT4: Talked to TJ from avalaunch, and it seems that the limit of files per 1 folder is 4096, and that is in the FS.
Soo, it seems we are reaching the peak usuage of the FATX FS, and without another one heavily in the works, the only thing that I can think of as a temp solution would be to create more partitions.  Not the most efficient as far as space utilization, but there really isnt anything left to do.
EDIT5: the above is now confirmed.

QUOTE

The maximum number of files and subdirectories per directory is 4,096. However, any directory with more than 768 files is going to give slower file access than a directory with 768 files or less, due to the technique that the Xbox file system uses to store directory listings. Additionally, directories are not sorted, so it requires an O(n) search to find and open a file. The more files in a directory, the longer it takes to open a file in that directory.

The above is © 2000-2003 MS Corporation. All rights reserved.

[heinrich]

From what a I remember, the current patch for LBA48 will look on the hard drive for a partition table, so in reality, there is the possibility for a "partition magic" for the xbox.  I dont have the skills to do write this, I im not sure if there is any active xbox dev that does, that has the time.  I know BenJeremy has suggested that this could cause problems with dashboards....I'll PM him and ask him to check out this thread.

[heinrich]

Also, for anyone who wants to check their # of files/folder, you can use boxplorer, "check free space" will return the above values.

[BenJeremy]

QUOTE (heinrich @ Nov 15 2003, 06:57 PM)

From what a I remember, the current patch for LBA48 will look on the hard drive for a partition table, so in reality, there is the possibility for a "partition magic" for the xbox.  I dont have the skills to do write this, I im not sure if there is any active xbox dev that does, that has the time.  I know BenJeremy has suggested that this could cause problems with dashboards....I'll PM him and ask him to check out this thread.

The problem with LBA48 and some programs, such as dashboards, that are set up to initialize G: (Partition 7) is with X2 BIOSes, which seem to fill the P7 info with something that causes the system to hang when trying to initialize P7 (G:). This, for some reason, does not happen on the Evo-X BIOSes.

In MXM, for example, I now look at the size of F:, and unless it's EXACTLY the size expected in a G:-enabled setup (around 129GB), it does not initialize G:. Fiddling with the partition table size of F: would hose that method up badly (but I did provide a way to hardcode the G: initialization)

So basically, messing around with the partition tables to adjust the size of F: with a G: makes life a bit more difficult for dashboard and utility apps.

[SilntBob]

heinrich,  

I started a separate thread a month or so ago, asking paulb about a similar issue

The thread is here:
http://forums.xbox-s...T&f=41&t=108657

I was wanting to make custom partition tables to assist with doing some custom Linux work.  But the same reasoning could still apply here.

The one important thing to note, is that a "Partition Magic" for the xbox is much more difficult than a "fdisk" for the xbox.  A Partition Magic like application would need the ability to not only modify partition size, but it would have to be able to intelligently consolidate and relocate data to facilitate partition resizing.  That is a big first step.

It seems that building custom partition tables is achievable.  paulb had included some code for me, but I do not have enough xbox coding experience to make it work.  The code is in the thread above.

[heinrich]

Thanks SilntBob

One thing though, team xecuter decided to leave out the part of paul's lba48 code that looked at partition0 in their 4979.xx bios, so any type of 'partition magic' would not be compatible with that bios.

[EncryptedBytes]

Rather than breaking a chunk of disk off for another partition, it'd make more sense to code support for loading ROMs from multiple directories in the emulator.  An emulator could read roms from maybe 4 directories, then merge the list together for display to the user...

[CaliSurfer008]

[CaliSurfer008]

I am expecting to get a Western Digital 250gig in sometime hopefully before Christmas and will be using it in my xbox. From what I've read, looks like I will use F & G partitions althought I would really like to just create 1 large F Partition...but looks like files start gettings corrupted

I'll letcha know!

[Cutriss]

QUOTE

Still, I wonder what the maximum partition size is?  We know that LBA48 supports a large disk, but the filesystem has to support it as well...  Files are generally accessed using an offset from the start of the partition - So, now that the drive supports over 137GB; the question becomes will the filesystem support it.  If I was MS, why create a filesystem that can handle 4TB per partition when I know the BIOS *will* never support it and I'm only putting in a 10GB drive max?

It's not that it's a partitioning scheme they designed *for* the Xbox, so much as one they adapted from FAT32 to meet the Xbox's needs. Therefore, allowing for a 4TB partition is futureproofing.

QUOTE

Now, assuming that we have a limitation in the partition size imposed by FATX; what is the limit on the number of partitions that we can create?  I thought that IDE only supports 4 primary partitions, but we have already exceeded that - of course these are probably not your typical primary partitions... We are at 7 partitions, but do partitions 8, 9, & 10 also work?

Well, not all your partitions must be primary. You can have three primary and one extended with multiple logical volumes, which I suspect is the case here (not necessarily three primary, but that logical volumes are being used). Logical volumes are constructed using a linked list, so that the partition table of each knows how big it is, and how big the remainder of the extended partition is (and where it starts). You can't do direct access to logical volumes - You have to get the head of the extended partition, and then jump ahead through each partition table until you find the correct one.

QUOTE

With all the knowledge and smart people here, I'm sure that somebody will calculate the actual max partition size - a post up above already mentioned the maximum files and directories that a single directory can hold...

Well, that's the thing. There is no *one* size, unless there's *also* an addressing limit in the partitioning system (my assertion, which heinrich seems to disagree with). A lot of the people I'd read reports of having trouble with their partitions didn't have problems until they passed around the 280 GB mark, which is just at 2^38.

If there *isn't* an addressing limit, then you can't really say that there is a hard ceiling on drive sizes, since the problem isn't tied to how large your files are, but rather how many of them you have.

QUOTE

As Paul mentioned a long time ago, we need to get away from hard coding partions and assuming drive letters... this should either be something that the BIOS provides to the applications or a standard partition table that defines the drive letters and partitions for each application.

Agreed. There's no reason, other than shortsightedness, that apps should be developed that need to be "patched" later to support the G: drive, or any other drive for that matter. Apps should be drive/directory agnostic.

This is a point that I strongly disagree with Xport about.

[oz_paulb]

QUOTE (BenJeremy @ Nov 17 2003, 02:18 AM)

QUOTE (heinrich @ Nov 15 2003, 06:57 PM) From what a I remember, the current patch for LBA48 will look on the hard drive for a partition table, so in reality, there is the possibility for a "partition magic" for the xbox.  I dont have the skills to do write this, I im not sure if there is any active xbox dev that does, that has the time.  I know BenJeremy has suggested that this could cause problems with dashboards....I'll PM him and ask him to check out this thread. The problem with LBA48 and some programs, such as dashboards, that are set up to initialize G: (Partition 7) is with X2 BIOSes, which seem to fill the P7 info with something that causes the system to hang when trying to initialize P7 (G:). This, for some reason, does not happen on the Evo-X BIOSes. In MXM, for example, I now look at the size of F:, and unless it's EXACTLY the size expected in a G:-enabled setup (around 129GB), it does not initialize G:. Fiddling with the partition table size of F: would hose that method up badly (but I did provide a way to hardcode the G: initialization) So basically, messing around with the partition tables to adjust the size of F: with a G: makes life a bit more difficult for dashboard and utility apps.

BenJeremy -

You say that some dash's have problems when they "initialize" G:/Partition7.  Is it that they have a problem when G: doesn't exist (a non-LBA48 or non-G: setup)?  The bios/kernel is returning 'garbage' size/location info for the invalid partition, causing problems?

If that's the case, there are ways to discover whether/not LBA48 is installed, and which partitions are valid (using DeviceIOControl commands).  Of course, it's not documented anywhere (it's stuff in my LBA48 source, which I haven't gotten around to 'officially' releasing yet), but it's something do-able.

BTW, I've hesitated on releasing my LBA48 stuff because I wanted to first 'clean up' the code and add some comments/documentation.  Because of my job, I haven't had any time to mess with Xbox stuff.  So, I think I'll go ahead and release the code 'as is' - it may be helpful to some developers out there.

- Paulb

[oz_paulb]

My thoughts on ~280GB limitation:

As I've said from the beginning, LBA48 allows for partitions up to about 2TB, but I suspect that there are FATX limitations that make the max partition size smaller.  But, it should be possible to create as many partitions you like - each smaller than the FATX limitation - with no problems.  The LBA48 code supports more than just partition6/partition7 (F:/G:).  I forget the number - maybe 4-6 more partitions are allowed in the current code (this could be increased - up to (I think) either 10 or 20 partition limit (per drive) in the kernel).


I suspect that the FATX limitation is in the FAT (File Allocation Table).  This table is at the start of the drive, and gets initialized at format time.  As you add files, the space they take up gets marked in the FAT.

There are at least 2 different 'types' of FAT tables on a FATX system (FATX16/FATX32).  One is for smaller partitions, another for larger.  At the time of formatting the partition, the FAT 'type' is chosen and the FAT table is initialized with entries for each 'cluster' on the partition.

So, the FAT table must be of the right 'type', and must have enough entries for (total size of partition / size of 'cluster').

According to this page: Xbox Linux FATX info

a 'cluster' on the Xbox is always 512 bytes.  So, the FAT table must be large enough to hold (total partition size / 512) entries.   Each FATX32 entry is 4 bytes (32bits), so the FAT table byte size would be: ((total partition size / 512) * 4) bytes.

If someone has a way of dumping a formatted disk, they could confirm that a newly formatted FATX partition on a large (> 280GB) drive is large enough for the partition.  Keep in mind that it's possible that the utility/utilities used to format the drive are at fault - not an underlying filesystem problem.  This would be the best scenario, since that can easily be fixed.  I haven't read the entire thread: is there a common format utility used in all cases of 'corruption'?  Keep in mind that some format utilities (Slayer's, for example) make use of EvoX's own format code (so multiple utilities could really all be the same one).

Another possibility: the max size of a 'cluster #'.  On older FAT systems, as drives grew larger, the size of a cluster was increased.  Part of the reason for this was that there was a max cluster # size (16 bits, I think, so 65535 max clusters).  If cluster sizes are made larger, then 65535 clusters represents a larger amount of hard drive space.

The Xbox-Linux doc I referred to above says that the cluster size is fixed @ 512 bytes.  Maybe we're hitting a max cluster # limit.  If so, then maybe it'll be possible to figure out how to get FATX to use a larger cluster size.  I really doubt this, but since (I think) there are some 'undefined/unknown' fields in the FATX partition header, we may find that one of these values can be tweaked to use larger cluster sizes.

The negative side for larger cluster sizes means that the minimum file size now increases (a 1 byte file on FATX now takes 512 bytes of space - it would take more space if the cluster size is bigger).  This means that if you've got lots of small files, you end up wasting lots of drive space.  This is the big trade-off on increasing cluster size.  But, it's what people have lived with on older FAT systems - we may have to live with it on FATX (assuming we can figure out how to make it use larger clusters).

My job has kept me very busy lately - no real time for any Xbox experimentation.  I also don't have a drive large enough to reproduce these problems (I have a 200GB which works fine).  I'll try to offer input/help on this issue, but I can't promise that I'll be able to dedicate much time on it.

I hope this is helpful.

- Paulb

[Cutriss]

That's why you could temporarily create 4 GB FAT partitions until MS rolled out FAT32 - by increasing the cluster size, the net addressable space increases, but reduce system efficiency (since complete clusters are transferred on a read, and as you mention, a 16385 byte file would use 32768 bytes in 2 clusters).

2^24 = 16.7M clusters (16777216)

16.7M clusters * 16384 bytes/cluster = 274877906944 bytes, or 274 GB.

Perhaps a 24-bit addressing range on clusters.

So, a proper test would be to see first if the cluster size is modifiable in the partition table (and format it accordingly, though I don't think any Xbox utilities currently have an adjustable cluster size in formatting options), and then fill it with 1 GB dummy files, and see if it can be reliably used beyond 274 GB.

[oz_paulb]

I agree with your #'s, and think it should be tested.

It does seem odd to me that there would be a 24-bit limit on # of clusters.  The old 16-bit limit made sense (size of 'int' on 16-bit computers), but the next logical limitation would be 32-bits.

It's definitely worth trying.  As you said, existing format utilities probably don't deal with changing cluster sizes.  Increasing the cluster size (always a power of 2, I believe) would decrease the overall size of the FAT (since the # of FAT entries would decrease).  That would change where the 'root' directory is placed on the partition.

I took a quick look at the kernel code that reads the 'FATX' header.  It does look like it tries to use the cluster size specified in the header (at least it loads it from the header into another structure).  I haven't followed the code through far enough to see if it's used throughout - I suspect it is.

- Paulb