Well, you knew this was coming– xbfuse. This is a program that leverages the wonderful FUSE paradigm to mount a Microsoft Xbox disc filesystem — the so-called XDVD filesystem — under Linux. I hammered out the bug mentioned in yesterday’s post (sure enough, a 64-bit offset was being demoted to a signed 32-bit quantity at one point, and that matters for filesystems this large). This is what the program looks like in action:

$ xbfuse Halo-3.iso mnt/

$ ls -al mnt/
total 4
dr-xr-xr-x  6 melanson melanson       0 2007-11-07 20:00 .
drwxr-xr-x 47 melanson melanson    4096 2007-11-10 17:31 ..
dr-xr-xr-x  2 melanson melanson       0 2007-11-07 20:00 bink
-r--r--r--  1 melanson melanson 8929280 2007-11-07 20:00 default.xex
dr-xr-xr-x  5 melanson melanson       0 2007-11-07 20:00 maps
dr-xr-xr-x  2 melanson melanson       0 2007-11-07 20:00 $SystemUpdate
dr-xr-xr-x  2 melanson melanson       0 2007-11-07 20:00 waves
-r--r--r--  1 melanson melanson  561152 2007-11-07 20:00 WaveShell-Xbox.dll
-r--r--r--  1 melanson melanson  724992 2007-11-07 20:00 WavesLibDLL.dll

$ ls -al mnt/bink/
total 0
dr-xr-xr-x 2 melanson melanson        0 2007-11-07 20:00 .
dr-xr-xr-x 6 melanson melanson        0 2007-11-07 20:00 ..
-r--r--r-- 1 melanson melanson 77940860 2007-11-07 20:00 attract_1_60.bik
-r--r--r-- 1 melanson melanson 61324440 2007-11-07 20:00 attract_2_60.bik
-r--r--r-- 1 melanson melanson 72829508 2007-11-07 20:00 attract_3_60.bik
-r--r--r-- 1 melanson melanson 69631000 2007-11-07 20:00 credits_60.bik
-r--r--r-- 1 melanson melanson 21163412 2007-11-07 20:00 intro_60.bik

$ fusermount -u mnt/

So Halo 3 uses Bink files, some very high resolution ones, rather than any Xbox-specific multimedia formats, like XMV. Actually, Bungie (the company behind Halo) may have a history with Bink, as I seem to recall that the FMV for the PC demo version of Halo was also Bink (or at least one promotional file).

I actually just thought to look up whether there are other options for mounting Xbox filesystem images under Linux. The format certainly seems to be of much greater interest than, say, GameCube filesystem images. I did find a project called Mount ISO Image that is supposed to be able to handle XDVD filesystems. Though I can’t really figure out if it’s a KDE application, a script, or a KDE script.

I took a slightly different approach to writing this one. All in all, I suppose the result is much simpler than gcfuse. The GameCube filesystem is an odd beast and required a lot of custom hacks to parse all of the data structures. However, writing xbfuse scared me more because I had to write 2 mutually recursive functions: After loading the volume descriptor, call xbfs_recurse_directory(), which then calls xbfs_recurse_file_subtree(), which calls not only itself, but also xbfs_recurse_directory() when a file entry happens to be a directory. I thought about writing xbfuse in such a way that it would traverse the data structures on demand when loading a file, since the data structures are laid out to be conducive to binary searching. I also thought about only loading the first level of the directory tree, and loading other levels on demand. But in the end, I just went with the full tree load at the outset and finally squashed the 64 -> 32-bit bug and the program seems to work quite well.

What next? Wii discs seem to use a different format than the GameCube discs and I would like to find out what that is. Plus, I am still dogged by the slightly custom Dreamcast ISO-9660-style format. There is a lot of interesting Sofdec media on those Dreamcast games. And it only takes about 26 hours to rip the contents of a Dreamcast disc onto your PC, provided that you have the right serial cable.

Related Posts:

• gcfuse
• Xbox Multimedia, Finally!

At long last, I wrote that FUSE driver for the Microsoft Xbox DVD filesystem. I will hopefully get around to releasing it, just as soon as I can test it a little bit more. I am not completely confident with the new driver but that might be because the filesystem that I was using to test the program appears to have a number of problems and might be corrupted (or it could very well be that Xbox 360 games use a slightly different filesystem layout than Xbox games, or that I’m not correctly handling 32-bit thresholds on large filesystem images). When I started studying other smaller Xbox filesystems, things seemed to go quite smoothly.

So did I finally find some samples of the fabled XMV format? Yes! At long last. Far and away, however, the Bink format appears to dominate, at least on the sample of games that I surveyed.

I found 4 Xbox games that include XMV media: Thrillville, Xiaolin Showdown, Monopoly Party, and Lego Star Wars 2: The Original Trilogy. You can find samples at the usual place. The last game on that list surprised me– Lego Star Wars is an aggressively multi-platform franchise. While the disc only encodes the intro movie as an XMV file (and in both PAL and NTSC variations), this would imply to me that versions for other platforms are encoded in that platform’s preferred SDK format, i.e., MPEG-2 for PS2, THP or H4M for GameCube, and who knows for Windows. Typically, cross-platform games take the easy way out using portable middleware solutions like Bink.

A little bird once told me that XMV was on the horizon for Xbox multimedia and that it would be straight ASF files with WMV2 video and MS ADPCM audio. If that’s true, then it looks like the WMV2 puzzle was solved just in time. These XMV files certainly are not stock ASF files. Curiously, the most notable signature is ‘Xbox’ spelled, ahem, in little endian notation in bytes 12-15: ‘xobX’. Width and height are encoded at bytes 20-23 and 24-27, respectively. I see sample rate data at bytes 40-43. I tried to find framerate data by comparing the headers of the PAL vs. NTSC Lego SW demo movies. I was hoping to find fields with 50 or 25 for PAL and 60 or 30 for NTSC. No such luck. However, I have also read that the movies are 29.97 fps, so the framerate data might be encoded as floating- or fixed-point numbers.

The Lego Star Wars 2 game has lots of interesting stuff to peruse. Since it is LucasArts software, it should be no surprise that human-readable scripts play a role. Indeed, there is a whole directory of scripts, such as this simpler file scripts/Attack.scp:

state Base {
        Conditions {
                if GotOpponent == 1 goto StartAttack
        }
        Actions {
                BreakFormation
        }
}

state StartAttack {
        Conditions {
        }
        Actions {
                KeepWeaponOut
                SetState "Attack"
        }
}

state Attack {
        Conditions {
                if GotOpponent == 0 goto Base
        }
        Actions {
                EngageOpponent "goalrange 1.5" "firerange 3"
        }
}

Thrillville is also from LucasArts and has various markings to indicate that the Lua language is involved, as in other LucasArts titles such as Grim Fandango.

Lego Star Wars 2 also contains a large number of .wavm files. These could be headerless PCM or ADPCM data. What was that FFmpeg incantation for converting headerless, raw PCM to a container format, manually specifying parameters?

One more item– Call of Duty 3. This uses a format with the extension XBV and it begins with the signature ‘AFMV’. I don’t have many other clues on it, but I have thrown together a MultimediaWiki page on the matter and uploaded some samples, as is customary.

Related Posts:

• gcfuse
• gcfuse, With Executable Support

I’m just going to keep guessing; it’s much easier than digging up actual, empirical data. I’m starting to come to grips with the idea that the number of valid sectors contained on a DVD disc is something that’s interpreted by the optical drive firmware and enforced by the same (as is the case for RPC-2 DVD region encoding). If that’s true, there is no point in using direct SCSI access to beg for sectors beyond the magic 6,992 limit on a standard Xbox disc. Ways around this? There’s the Xbox-Linux FTP trick alluded to in my cursory post on the matter…

Read the rest of this entry »

After yesterday’s post I received a tip that I may wish to try getting at the Xbox disc data using the low-level Linux SCSI layer– the facility commonly known as SCSI generic, or simply ‘sg’. In order to make use of this facility, it is necessary to configure your Linux kernel with the right support (CONFIG_CHR_DEV_SG, CONFIG_BLK_DEV_IDESCSI, but not CONFIG_BLK_DEV_IDECD). Download and install the SCSI generic utilities from here (which do not compile on my AMD64 machine; fortunately, the x86-compiled utilities and associated shared library drop into the system perfectly). There are about 30 little sg* utilities in the package. One that caught my attention was sg_dd– your typical ‘dd’ Unix command, but for interacting with a SCSI device. I tested it out on a typical Xbox DVD:

$ sudo sg_dd if=/dev/sg0 of=dvd bs=2048 bpt=1 count=20
20+0 records in
20+0 records out

And, indeed, the beginning of the disc has the markings of a typical DVD filesystem. So let’s try to read beyond the 6,992-sector limit on a standard Xbox disc:

$ sudo sg_dd if=/dev/sg0 of=dvd bs=2048 bpt=1 count=2 skip=6991
reading: SCSI status: Check Condition
 Fixed format, current;  Sense key: Illegal Request
 Additional sense: Logical block address out of range

 Raw sense data (in hex):
        70 00 05 00 00 00 00 0a  00 00 00 00 21 00 00 00
        00 00
plus...: Driver_status=0x08 [DRIVER_SENSE, SUGGEST_OK]
sg_read failed, seek=1
Some error occurred,  remaining block count=1
1+0 records in
1+0 records out

I had to deal with all this SCSI nonsense in a previous life and I admit that this output is conjuring some sour memories. Anyway, it seems that the stock sg_dd command can not read beyond what the DVD header structure ostensibly reports as the size of the disc. There are a number of other utilities I need to investigate for potential. Failing that, it’s possible — with at least a modicum of effort — to program the sg layer. There may yet be a solution.

Related posts:

• Xbox Media Redux
• Studying Xbox Multimedia

Pursuant to my post last week regarding Xbox media, I received some suggestions to throw some leading Windows utilities at the problem, notably Nero and IsoBuster. They both report the same as my results in Linux.

IsoBuster:

Nero:

I’m sure there were a lot of people out there who already knew this stuff. But it’s hard to find specifics on the internet (sort of like heavy technical details of reverse engineering).

…at least, that’s what I would like to do. I bought a used Xbox game nearly 2 years ago with the intention of opening up the disc and studying the contents for interesting multimedia formats. I picked up a few more titles some months ago because I heard these games had some unique multimedia targets. Today, close to 5 years after the Xbox’s initial launch, I tried to examine the disc contents. I ran into a few problems.

I had always heard that Xbox discs have a standard DVD-video structure at the front in order to accommodate the eventuality that the disc may be placed into a standard DVD player. Sure enough, treating the disc like a DVD shows a 13-second, totally X-treme Xbox video followed by a still screen instructing the user, in several languages, to play the disc in the proper game console.

If you use standard CD/DVD system tools to study the Xbox disc, it will be reported as having 6,992 2048-byte sectors, a little over 14 megabytes of DVD-video data. This page on Xbox-Linux describes the Xbox DVD filesystem (XDVDFS) format. However, the markers discussed in that document (“MICROSOFT*XBOX*MEDIA”) do not occur in this 14-megabyte data segment. I imagine the game data starts immediately after this segment. As the document explains, the Xbox DVDs have a fraudulent TOC which tells standard DVD-ROM drives that the disc is much shorter than it actually is. Trying to seek past that point manually just results in an unceremonious EOF.

A brief perusal of BitTorrent sites reveals that many people have absolutely no trouble ripping Xbox games for the purpose of distribution. How do they do it? Apparently, by modding an Xbox, running Linux on it, logging into the Xbox, loading a game disc into the Xbox’s drive, FTP’ing to another computer, and reading the raw game sectors from the disc using the Xbox’s drive, which apparently uses a different set of rules for reading. Not being in possession of an Xbox and having no desire to invest in one at this time, I am at a bit of a loss to study these discs, unless there is a solution I am overlooking.

Based on the filesystem format doc, I am quite confident I could write a FUSE module for browsing the filesystem in short order, based largely on my gcfuse code.