Print

[node21]

QUOTE(nevernow @ Feb 4 2010, 12:51 PM)

I see. That's a sensible approach, for sure. But this makes me understand the problem even less. :|
Could you tell me what you read in an XBLA game, other than the eight-digit ID at offset 360?
For example, does the title displayed in the list (not the management view) come from the CSV file or from the game? The unlock status is also wrong... I'm suspecting that some critical data are not in a standard position in the container (but then the game title in the list appears correctly in v0.5).

Sure, but this is from memory, since I don 't have the code in front of me. :-)  Also, this is subject to change in version 0.7 as I'm re-organizing a bunch of stuff.  

The first scene that comes up where you "Scan" hasn't even read the xbla_titles.csv file yet.  So, everything you see there is coming from the file itself.  While sucking that data in, it also pulls the titleID from offset 0x360.  The determination of it's "locked" status is grabbed at the same time.  I'd have to check the code, but there could *easily* be an issue there... There are now 2 functions in the codebase that check if a title is "unlocked" (XBLA, and DLC), and 2 functions in the codebase that "unlock" content (the old way, and the new way).  I'm certain that I never call the old "unlock" code...but it's possible that I'm still using the old "XBLA" check for the lock status, and maybe that is somehow getting confused?  My intention was to not use that code either (and only use the new check) but I might have forgotten to change it?  Anyway, there's no harm in it aside from the visual indicator being wrong (which is wrong enough to warrant a fix in the next version, so I'll check this when I see the code again).

Once you are in the collection management screen, everything that you see comes from the xbla_titles.csv file.  It determines if you "have" a title by searching for it (by titleID) in the datastructure that was filled out when you did the initial scan back on the first scene.

[nevernow]

Thanks for sharing all this.

Here's some more observations:

In v0.5, with the original CSV file with two SC entries, in the "unlock view", SC appears as unlocked, correct title, but can't be played (gray box suggesting to download the game again). The game is playable from NXE.
In the collection management view, it appears twice (as expected). The CC line of the CSV file results in a "Have" entry, the 1A line is "Missing".
This means that when you check the CSV against the datastructure, the datastructure does not have a 5841081A entry. I have verified that the CC line status comes from Rocket Riot, not Super Contra (actually obvious since you compare titleids).
But assuming that the entries from the datastructure come from the initial scan, this does not make much sense (to me, at least). Do you only ever read the titleid from offset 0x360?

In v0.6b, in the "unlock view", SC's title is garbage and status is locked, which leads me to think that you have changed the way the scan works (I remember at least something about the license bytes has changed).

My next question is: what exactly do you read in the initial scan? If you only read the titleid at offset 0x360, which is correct in SC, I guess the id would go into your datastructure and the management view would show the game as a "Have".

[node21]

Unfortunately, I can't remember exactly what changed between 05 and 06.  But I remember a massive re-write...so anything could have happened.

What's most troubling is that you are reporting the title as coming up as garbage in .06.  That shouldn't happen, but is probably related to the oddness that I've noticed with seekg on the 360.  Sometimes it doesn't go where you would expect it to go before doing a read.  However, I would expect all titles to be screwed up then... *sigh*

During the initial scan, it does this:

CODE

getContentType(F); // is this XBLA, or DLC?
getTitleID(F);          // read the titleID at offset 0x360
getWTitleString(F);  // get the embedded Title at offset 0x411
if ContentType == ARCADE
    check if it is unlocked (the old way, confirming my earlier suspicion that I got this part wrong)
if ContentType == DLC
    getWTitleStringFor(F);   // get the embedded Title that this DLC is "for" at offset 0x1691
    check if it is unlocked (the right way)

There is no other place to read the titleid from...it is only in one place.

=-=-=-=-=-

And now, a minor update from a short coding session last night.

1)  I'm having major issues with XUI controls (specifically text controls) not doing what I expect them to do, but that's another post...
2)  I'm starting to believe that "Collection Management" for DLC will never really work.  There are *TONS* of titles that are on my hard drive (and I don't have much DLC), that just do not match to what is scraped from the MS site.  Whatever database drives the MarketPlace site seems to have no real correlation to what is actually written inside the DLC at offset 0x1691 and offset 0x411.  In many cases, it seems to be related to the fact that the marketplace tags "cover" songs (not the original artist) with an asterisk (*).  In other cases, I think something must be wrong with the comparison code since a lot of the missed titles have an apostrophe (') in them.  But, there are some cases where they just don't match...  I'm sticking with it, for now, to see if it will ultimately settle down to a small number of inconsistencies, but it isn't looking good for now... If I could just get past issue #1 I'd be able to really concentrate on this!
3) My 360 VGA cable arrived in the mail, so now I don't have to continually kick the entire family off of the main TV every 2 minutes to test a build.  That was getting old for all of us. :-)

This post has been edited by node21: Feb 5 2010, 03:08 PM

[nevernow]

QUOTE(node21 @ Feb 5 2010, 03:07 PM)

Unfortunately, I can't remember exactly what changed between 05 and 06.  But I remember a massive re-write...so anything could have happened.

What's most troubling is that you are reporting the title as coming up as garbage in .06.  That shouldn't happen, but is probably related to the oddness that I've noticed with seekg on the 360.  Sometimes it doesn't go where you would expect it to go before doing a read.  However, I would expect all titles to be screwed up then... *sigh*

During the initial scan, it does this:

CODE

getContentType(F); // is this XBLA, or DLC?
getTitleID(F);          // read the titleID at offset 0x360
getWTitleString(F);  // get the embedded Title at offset 0x411
if ContentType == ARCADE
    check if it is unlocked (the old way, confirming my earlier suspicion that I got this part wrong)
if ContentType == DLC
    getWTitleStringFor(F);   // get the embedded Title that this DLC is "for" at offset 0x1691
    check if it is unlocked (the right way)

There is no other place to read the titleid from...it is only in one place.

- if you're using seekg with offsets, and something is indeed wrong (or at least non-standard) in SC, this might explain both the garbage title and the wrong unlock status detection. Or not?

- isn't the title string at offset 0x412? I mean, 411 is blank and 412 is where the fist character is.

- I have found that in SC, the hex sequence "584181A" appears two times; sequence "584108" four times in total. In other games, the full titleid sequence appears several times.
That's why I asked if you only ever read 0x360. Still can't figure why it won't match the CSV file. I am 100% that at 0x360, SC reads "5841081A".

QUOTE

2)  I'm starting to believe that "Collection Management" for DLC will never really work.  [...]

About this issue, what do you think of a complete change of approach? I mean, instead of relying on a scrape of xbox.com, using that as a basis and building a cooperative database where internal labels from DLC items and scrape results are correctly matched? Something like this:

DLC item parse:
- titleid 12345678
- title label "Rock Band - Guns And Roses - Sweet Child O' Mine"

xbox.com scrape:
- title name "Rock Band: Guns & Roses - Sweet Child Of Mine"

Not an exact match -> need for a manual association. Which - in theory - could happen either locally and individually, through a fuzzy search by XM360 showing similar matches, or cooperatively on a database/wiki website. What do you think?

[node21]

Not really thinking about workarounds yet...this XUI issue is taking all of my time.

[nevernow]

Ok. Best of luck with it!

[node21]

0.7 is in beta testing now.

Version 0.7 -- Feb 6th, 2010
* Further massive rewrites :-)
* Automatically scan on launch, showing progress
* That ugly fixed-width font is gone, except where needed (the XBLA Collection Manager list)
* Support for DLC "Collection Management" - it's integrated into the "Show DLC" scene.  Use LB and RB to change views.  Note there are *lot's* of missed matches, mostly because what is *inside* the DLC files doesn't match what is on the marketplace web site.  Because of this, there is also the ablity to show content which you have, but wasn't scraped from the web site.
* There is no support for exporting a DLC "report" yet.  It will come, if there is interest.
* Like scraping for XBLA, there is an included scrapeMS_DLC.pl perl script.  It takes a very long time to complete.  You shouldn't need it often.  XM360 will search for a file called DLC_titles.csv before using the one included in the package.
* There's still no button to generate the XBLA_report.txt.  It does it any time you leave the XBLA Collection Management screen, so expect it to be a little slow.

If the beta goes well, it should be released this weekend...

[Midnight Tboy]

nice one thanks...I'll have a play about with it a bit later.

DLC Report would be great.

wonder by way of scraping the MS site for both DLC and XBLA....presumably it'll be better some point to perhaps set to download a scraped file weekly from a hosted site rather than wait to scrape it itself - XBLA takes quite a while to scrape when I've tried it - so can imaging the DLC one will take ages beyond that.

[node21]

QUOTE(Midnight Tboy @ Feb 6 2010, 02:25 PM)

nice one thanks...I'll have a play about with it a bit later.

DLC Report would be great.

wonder by way of scraping the MS site for both DLC and XBLA....presumably it'll be better some point to perhaps set to download a scraped file weekly from a hosted site rather than wait to scrape it itself - XBLA takes quite a while to scrape when I've tried it - so can imaging the DLC one will take ages beyond that.

Actually, I don't think scraping the marketplace is going to be a feasible solution in the long run...as you can see in version 0.7beta, there are so many mismatches that it probably isn't worth the effort.  What we need is our own database of DLC with the *actual* internal string.  But, I have no place, and no inclination to host this data.  If someone does, then XM360 count connect to it, and always have "live" data.

[node21]

version 0.7


Version 0.7 -- Feb 10th, 2010
* Further massive rewrites :-)
* Automatically scan on launch, showing progress
* That ugly fixed-width font is gone, except where needed (the XBLA Collection Manager list)
* Support for DLC "Collection Management" - it's integrated into the "Show DLC" scene. Be sure to use LB and RB to change views. Note there are *lot's* of missed matches, mostly because what is *inside* the DLC files doesn't match what is on the marketplace web site. Because of this, there is also the ablity to show content which you have, but wasn't scraped from the web site.
* There is a button to write the DLC report. It takes a while, and only includes information on DLC for which you have at least one title.
* Like scraping for XBLA, there is an included scrapeMS_DLC.pl perl script. It takes a very long time to complete. You shouldn't need it often. XM360 will search for a file called DLC_titles.csv before using the one included in the package.
* There's a button to generate the XBLA_report.txt instead of doing it all the time.
* Entirely new support for Title Updates. Including the ability to Delete them.
* Support for deleteing XBLA titles.
* Support for deleteing DLC titles.

[Midnight Tboy]

wicked stuff - got a bunch more DLC to copy over.  Will have to take a look at the generated reports to see how it displays...to see if any use for working out the missing DLC and such like.

The obv trouble with the DLC lists being so large, will be all the c**p that MS sell like sport logos, and individual clothes items......some games on the xbox.com marketplace have dozens and dozens of useless junk( NHL games for example)....spanning over a dozen web pages.

Hopefully in the future there'll be some way to filter out all the unwanted junk like that to only show things like addon packs, and things that actually add elemends to a game.

[node21]

QUOTE(Midnight Tboy @ Feb 10 2010, 06:23 PM)

wicked stuff - got a bunch more DLC to copy over.  Will have to take a look at the generated reports to see how it displays...to see if any use for working out the missing DLC and such like.

The obv trouble with the DLC lists being so large, will be all the c**p that MS sell like sport logos, and individual clothes items......some games on the xbox.com marketplace have dozens and dozens of useless junk( NHL games for example)....spanning over a dozen web pages.

Hopefully in the future there'll be some way to filter out all the unwanted junk like that to only show things like addon packs, and things that actually add elemends to a game.

The way it works now is it only spits out information on DLC for titles that you actually already have some DLC...In other words, if you don't have any of the DLC for an NHL game, none of it's DLC will show in your report.  

I have an idea on how to improve this even though we aren't getting good title matches.  Just like the ability to mark XBLA content as "Ignore", I think I'm going to add the ability to mark a DLC title as "Have".  That way, even if the title doesn't match, you can tell it that you actually have the content.  We'll see how that works out....

[aubrey_76]

QUOTE(node21 @ Feb 10 2010, 05:03 PM)

version 0.7
Version 0.7 -- Feb 10th, 2010
* Further massive rewrites :-)
* Automatically scan on launch, showing progress
* That ugly fixed-width font is gone, except where needed (the XBLA Collection Manager list)
* Support for DLC "Collection Management" - it's integrated into the "Show DLC" scene. Be sure to use LB and RB to change views. Note there are *lot's* of missed matches, mostly because what is *inside* the DLC files doesn't match what is on the marketplace web site. Because of this, there is also the ablity to show content which you have, but wasn't scraped from the web site.
* There is a button to write the DLC report. It takes a while, and only includes information on DLC for which you have at least one title.
* Like scraping for XBLA, there is an included scrapeMS_DLC.pl perl script. It takes a very long time to complete. You shouldn't need it often. XM360 will search for a file called DLC_titles.csv before using the one included in the package.
* There's a button to generate the XBLA_report.txt instead of doing it all the time.
* Entirely new support for Title Updates. Including the ability to Delete them.
* Support for deleteing XBLA titles.
* Support for deleteing DLC titles.

I am not seeing v.07 in the "usual places" Has this been uploaded somewhere else or is it coming out later tonight. Thanks and cant wait to try this out.

[node21]

QUOTE(aubrey_76 @ Feb 10 2010, 07:51 PM)

I am not seeing v.07 in the "usual places" Has this been uploaded somewhere else or is it coming out later tonight. Thanks and cant wait to try this out.

Not there yet.  PM on the way.

[aubrey_76]

QUOTE(node21 @ Feb 10 2010, 08:21 PM)

Not there yet.  PM on the way.

Thank you very much. I like it....this app just gets better and better.....again thanks for all the hard work