Tag Archives: filesystems

Of Filesystems and Codecs

I have been hanging out at the Linux Foundation Collaboration Summit. One theme I have heard tossed around is the matter of filesystems– ongoing filesystem research, the need to upgrade standard filesystems in Linux, etc. I admit that I don’t spend a lot of time thinking about filesystems (except when I’m writing FUSE drivers for filesystems that lack wide appeal). The filesystem is something that’s just “there” and should just work. Indeed, I have never had a major problem with any filesystem I have used while it is still considered modern. It is only when the next generation comes along that I understand the faults in the previous generation (journaled filesystems helped me understand that extensive integrity checking at boot time doesn’t have to be necessary; anything beyond FAT16 helped me understand that 8.3 filenames didn’t have to be the standard).

But there is a category of obsessed individuals who spend a lot of time thinking about filesystems and measuring what they’re doing and figuring out how they could be doing things better. And it’s a good thing that we have these people around, even though most of us largely view filesystems as a transparent cog in the machine of daily computing.

This got me to thinking about how it’s probably very likely that most computer users view multimedia codecs the same way that I view filesystems. An AVI file might contain Cinepak or MPEG-4 part 2 video, or any of 100+ video codecs. Most users don’t have a reason to care about the difference. This may help to explain why some people (not particularly well-versed in multimedia technology) take it for granted that Theora could easily replace H.264 in all applications where the latter is in use today.

They’re both video codecs, right?

xbfuse

Well, you knew this was coming– xbfuse. This is a program that leverages the wonderful FUSE paradigm to mount a Microsoft Xbox disc filesystem — the so-called XDVD filesystem — under Linux. I hammered out the bug mentioned in yesterday’s post (sure enough, a 64-bit offset was being demoted to a signed 32-bit quantity at one point, and that matters for filesystems this large). This is what the program looks like in action:

$ xbfuse Halo-3.iso mnt/

$ ls -al mnt/
total 4
dr-xr-xr-x  6 melanson melanson       0 2007-11-07 20:00 .
drwxr-xr-x 47 melanson melanson    4096 2007-11-10 17:31 ..
dr-xr-xr-x  2 melanson melanson       0 2007-11-07 20:00 bink
-r--r--r--  1 melanson melanson 8929280 2007-11-07 20:00 default.xex
dr-xr-xr-x  5 melanson melanson       0 2007-11-07 20:00 maps
dr-xr-xr-x  2 melanson melanson       0 2007-11-07 20:00 $SystemUpdate
dr-xr-xr-x  2 melanson melanson       0 2007-11-07 20:00 waves
-r--r--r--  1 melanson melanson  561152 2007-11-07 20:00 WaveShell-Xbox.dll
-r--r--r--  1 melanson melanson  724992 2007-11-07 20:00 WavesLibDLL.dll

$ ls -al mnt/bink/
total 0
dr-xr-xr-x 2 melanson melanson        0 2007-11-07 20:00 .
dr-xr-xr-x 6 melanson melanson        0 2007-11-07 20:00 ..
-r--r--r-- 1 melanson melanson 77940860 2007-11-07 20:00 attract_1_60.bik
-r--r--r-- 1 melanson melanson 61324440 2007-11-07 20:00 attract_2_60.bik
-r--r--r-- 1 melanson melanson 72829508 2007-11-07 20:00 attract_3_60.bik
-r--r--r-- 1 melanson melanson 69631000 2007-11-07 20:00 credits_60.bik
-r--r--r-- 1 melanson melanson 21163412 2007-11-07 20:00 intro_60.bik

$ fusermount -u mnt/

So Halo 3 uses Bink files, some very high resolution ones, rather than any Xbox-specific multimedia formats, like XMV. Actually, Bungie (the company behind Halo) may have a history with Bink, as I seem to recall that the FMV for the PC demo version of Halo was also Bink (or at least one promotional file).

I actually just thought to look up whether there are other options for mounting Xbox filesystem images under Linux. The format certainly seems to be of much greater interest than, say, GameCube filesystem images. I did find a project called Mount ISO Image that is supposed to be able to handle XDVD filesystems. Though I can’t really figure out if it’s a KDE application, a script, or a KDE script.

I took a slightly different approach to writing this one. All in all, I suppose the result is much simpler than gcfuse. The GameCube filesystem is an odd beast and required a lot of custom hacks to parse all of the data structures. However, writing xbfuse scared me more because I had to write 2 mutually recursive functions: After loading the volume descriptor, call xbfs_recurse_directory(), which then calls xbfs_recurse_file_subtree(), which calls not only itself, but also xbfs_recurse_directory() when a file entry happens to be a directory. I thought about writing xbfuse in such a way that it would traverse the data structures on demand when loading a file, since the data structures are laid out to be conducive to binary searching. I also thought about only loading the first level of the directory tree, and loading other levels on demand. But in the end, I just went with the full tree load at the outset and finally squashed the 64 -> 32-bit bug and the program seems to work quite well.

What next? Wii discs seem to use a different format than the GameCube discs and I would like to find out what that is. Plus, I am still dogged by the slightly custom Dreamcast ISO-9660-style format. There is a lot of interesting Sofdec media on those Dreamcast games. And it only takes about 26 hours to rip the contents of a Dreamcast disc onto your PC, provided that you have the right serial cable.

Related Posts:

Xbox Multimedia, Finally!

At long last, I wrote that FUSE driver for the Microsoft Xbox DVD filesystem. I will hopefully get around to releasing it, just as soon as I can test it a little bit more. I am not completely confident with the new driver but that might be because the filesystem that I was using to test the program appears to have a number of problems and might be corrupted (or it could very well be that Xbox 360 games use a slightly different filesystem layout than Xbox games, or that I’m not correctly handling 32-bit thresholds on large filesystem images). When I started studying other smaller Xbox filesystems, things seemed to go quite smoothly.


Microsoft Xbox logo

So did I finally find some samples of the fabled XMV format? Yes! At long last. Far and away, however, the Bink format appears to dominate, at least on the sample of games that I surveyed.

I found 4 Xbox games that include XMV media: Thrillville, Xiaolin Showdown, Monopoly Party, and Lego Star Wars 2: The Original Trilogy. You can find samples at the usual place. The last game on that list surprised me– Lego Star Wars is an aggressively multi-platform franchise. While the disc only encodes the intro movie as an XMV file (and in both PAL and NTSC variations), this would imply to me that versions for other platforms are encoded in that platform’s preferred SDK format, i.e., MPEG-2 for PS2, THP or H4M for GameCube, and who knows for Windows. Typically, cross-platform games take the easy way out using portable middleware solutions like Bink.

A little bird once told me that XMV was on the horizon for Xbox multimedia and that it would be straight ASF files with WMV2 video and MS ADPCM audio. If that’s true, then it looks like the WMV2 puzzle was solved just in time. These XMV files certainly are not stock ASF files. Curiously, the most notable signature is ‘Xbox’ spelled, ahem, in little endian notation in bytes 12-15: ‘xobX’. Width and height are encoded at bytes 20-23 and 24-27, respectively. I see sample rate data at bytes 40-43. I tried to find framerate data by comparing the headers of the PAL vs. NTSC Lego SW demo movies. I was hoping to find fields with 50 or 25 for PAL and 60 or 30 for NTSC. No such luck. However, I have also read that the movies are 29.97 fps, so the framerate data might be encoded as floating- or fixed-point numbers.

The Lego Star Wars 2 game has lots of interesting stuff to peruse. Since it is LucasArts software, it should be no surprise that human-readable scripts play a role. Indeed, there is a whole directory of scripts, such as this simpler file scripts/Attack.scp:

state Base {
        Conditions {
                if GotOpponent == 1 goto StartAttack
        }
        Actions {
                BreakFormation
        }
}

state StartAttack {
        Conditions {
        }
        Actions {
                KeepWeaponOut
                SetState "Attack"
        }
}

state Attack {
        Conditions {
                if GotOpponent == 0 goto Base
        }
        Actions {
                EngageOpponent "goalrange 1.5" "firerange 3"
        }
}

Thrillville is also from LucasArts and has various markings to indicate that the Lua language is involved, as in other LucasArts titles such as Grim Fandango.

Lego Star Wars 2 also contains a large number of .wavm files. These could be headerless PCM or ADPCM data. What was that FFmpeg incantation for converting headerless, raw PCM to a container format, manually specifying parameters?

One more item– Call of Duty 3. This uses a format with the extension XBV and it begins with the signature ‘AFMV’. I don’t have many other clues on it, but I have thrown together a MultimediaWiki page on the matter and uploaded some samples, as is customary.

Related Posts: