Print

[veekay]

Have you done any tests on the automatic scraping time?  Hopefully there will be a way to force a full rescrape?  

Now if only there was a way to scrape a list of all xbox titles and their barcodes

[veekay]

Barcodes for game library.  Was a pita doing all my movies even with a barcode scanner.  Figured they would at least hide a UPC somewhere for each game, but nothing that I have found

[danthaman673]

Will there be any way to force region specific scrapings?

[node21]

QUOTE(danthaman673 @ Jun 17 2010, 12:32 AM)

Will there be any way to force region specific scrapings?

I can try to expose 2 things:  The "region", and the main site URL.  If people want to futz with that to come up with the right region & URL for themselves, they can try.

[node21]

Okay, DLC and XBLA scrapers are completely added to xm360_server now.

I just have to fix xm360 now to support DLC where there are multiple titles with the same filename.  ACE COMBAT is a prime example.  XM360 used to only list one title per filename...that's fixed, but now I have to deal with updating the UI when that file is deleted (now that there can be many per filename, I've gotta go find them all and clean up).  Also, things like fetching the file from a remote box...now it's gotta mark all DLC with the same filename as being "LOCAL".

That's pretty simple, so version 1.7b is almost certainly coming out this weekend.

[node21]

Version 1.7b -- Jun 19th, 2010
* NOTE NOTE NOTE: Version 1.7b is initially being released as a password protected file.  Donators have been emailed the password.
* NOTE NOTE NOTE: After a few days(?) it will be released for non-donators.
*
* Fix DLC where multiple DLCs are actually contained in the same file.  They will now display properly, get transfered properly, and get deleted properly.
* Fix ignored DLC not being remembered between session.  I broke this a couple of versions back, I think.
* Changed code so that if you have multiple files queued up, it will only transfer one at a time.  Three at a time was cool, but one at a time is faster.
* Removed the "new unlock" that was in 1.7 until I can get a handle on why it didn't always work.  May be re-introduced later.
* Finally, and maybe most interestingly, the old perl scripts for scraping DLC and XBLA content have been rewritten in C++, and are now included as features of xm360_server.exe

Usage: XM360server.exe /? - list options
       XM360server.exe /D - update or create dlc_titles.csv in the current directory and exit
       XM360server.exe /X - update or create xbla_titles.csv in the current directory and exit
       XM360server.exe /L {locale} - optionally specify the locale (defaults to en-US).  Only useful before /D or /X
       XM360server.exe /U {URL} - optionally specify the URL (defaults to catalog.xboxlive.com).  Only useful before /D or /X
       XM360server.exe - with no parameters runs the server (doesn't exit)

Sample:
XM360server.exe /X  (update XBLA_titles.csv and exit)
XM360server.exe /U catalog.xboxlive.com /L en-US /D (update DLC_titles.csv and exit, URL and Locale specifically overridden)

* One of the key features of this is that it is no longer silent while running.  It will keep you updated on what it is working on.  Sometimes the MS server takes a long time
*   to respond, so it's best to just be patient.
*   Also, the first line of the generated CSV file is now the date that it was run on.  This allows *subsequent* runs to only UPDATE the content that MS says has changed.  This
*   saves CONSIDERABLE time.  XM360 has been modified to accept CSV file either with the date, or without...so old CSV files (or ones generated by the perl scripts) will not break
*   XM360...however, that's only true of version 1.7b and above...anything earlier will choke on a CSV file with a date at the top.

Enjoy!
-node21

[veekay]

This should be interesting to test the new features.  Hopefully a remote fresh command via the server will be in the future

So far the new scrape is great - love the extra info.  Is it logged to a file anywhere aside from the csv?

[veekay]

File sizes are much smaller with this new method.  

Some of the symbols seem to cause issues in the scraped files, is the program able to convert those back correctly?

Examples:

[Dead Space™]
[Dante's Inferno™]
[NHL® 10]
[TC’s H.A.W.X]

[veekay]

QUOTE(node21 @ Jun 19 2010, 02:20 PM)

Can you be more specific?  What feature are you hoping for?
 

[veekay]

Thanks for the UTF8 info - just had to change that option when importing them into access.  I'm guessing the reports that are exported aren't in the same format since they don't have the same symbols in them?

Was anything changed that would affect the Cache folder?  Mine went from 600+ to 64 after updating and refreshing.  Might not be related, but the only thing done in the past few days were this and 1.7

Gotta love the edit option while it lasts..  just doing XM360server.exe /D > file.txt gives the output I had wanted, so at least that part was simple!

[veekay]

Ok, finally was able to confirm something that I had wondered about for a while.  

If you have no DLC of a certain item, the game won't even show up on the DLC report.  So if you are missing all DLC for any title, that title won't show on the list under the missing section.  

I guess this would explain why so many things never showed when I was checking for missing content.

Also, when running the new scraper the resulting file ends up missing a lot of items versus what you include in the release.  3731 in yours and when I run the file I get 2178.  I know an example of difference is Rock Band - 158 songs on mine and 1115 on yours.  When running as an update it "updates" some Rock Band tracks each time, always the same ones.

XBLA scraper results come out the same as your file.

[danthaman673]

Well done again my friend !!
 
That's awesome going to save heaps of mucking around etc..

The only thing I have to winge about is : When I was trying to upload all my DLC (Before my I updated .csv) It stalled on a blue(unrecognised) title for Forza 3 (I'm pretty sure it hadn't been unlocked yet) It may have been corrupt (I will confirm this shortly) But I just thought I'd mention it as it would be cool to have a push/get cancel as it sux when a transfer stalls and one has to go back to dash etc.. etc..

It took me a while to figure out that the switches were case sensitive and not to cut and paste from the example that had the .exe after the command name.. But when I figured that out it all worked like a dream!! 10/10 It added quite a few titles that have been missing for some time.

 Thanks again!!!

We will donate again when we can , Dan

[veekay]

QUOTE(node21 @ Jun 20 2010, 05:47 AM)

Nothing changed about the cache folder.  I'm guessing that cache2 was left alone, and the system cleaned out your cache?

What edit option?



Yes, that is by design.  If you read the heading it says something like "Titles for which you have some DLC".  You can always toggle that left side list with LB to see other filters.

The only reason the scraper would get less, in my opinion, is if you are running it from a different region, or MS is filtering it, somehow, by your IP's location.  Where are you located?  It is expected that it will update the same thing over and over again if you are updating on the same day as the change.  Since I only compare dates, and not times...  Also, the "update" marker that I look at is at the title level, not at the DLC level.  So, if *any* change is made to *any* "Rock Band" track (including if someone just rates one of the songs), then the updater will refresh ALL Rock Band DLC.

-node

[veekay]

QUOTE(node21 @ Jun 20 2010, 02:50 PM)

No current way of doing what you are asking...but why no look at the dlc_titles.csv directly?  If you know the title that you care about, all of the DLC for that title will be obvious in that file.

 

[node21]

While working on version 1.8, I found that the XBLA scraper included in version 1.7b gets the ratings wrong on some titles.  So, if you use the ratings, and actually care that they are accurate you should probably stick with the perl version of the scraper, as much as that sucks.  I'm not going to worry about fixing 1.7b, as you might imagine.