Here is the link to the samples RSS feed [ http://samples.mplayerhq.hu/samples-rss.xml ]. Also, here is the Python source code I threw together for the task.

I just want to check: I’m not the only person who still relies on RSS these days, right? The tech press has been cheerfully proclaiming its demise for some time now. But then, they have been proclaiming the same for Adobe Flash as well.

I’m no expert in RSS. If you have any suggestions for how to improve the features presented in the feed, please let me know. And, of course, keep the samples coming. This script should help provide more visibility for a broader audience.

Mario and Flashback Samples
Thanks to LuigiBlood who sent in some samples that allowed me to test out my new script for automatically syncing the repositories and updating the samples RSS feed. First, there are CPC multimedia files from the Japanese 3DO port of Flashback: The Quest for Identity. Then, there is an Interplay MVE file on the CD version of Mario Teaches Typing in which the video doesn’t decode correctly.

LuigiBlood also sent in another file from the latter game. It’s big and has the extension .AV. It could be a multimedia file as it appears to have a palette and PCM audio inside. But there’s no header and I’m a bit unsure about how to catalog it.

This 1993 CD-ROM makes proud use of multimedia files. What kind? There’s a movies/ directory with 17 .mov files. It would be way too simple if these were QuickTime files, though. These represent a custom format, and video-only since a separate sounds/ directory contains .snd files with filenames corresponding to the .mov files. The .snd files are actually Creative Voice (a.k.a. VOC) files. As for this MOV format, wiki page and samples.

I was also surprised to find the binary ultrasnd.exe file among the drivers on the disc. The Gravis UltraSound was released in 1992. The sound setup utility does not have an option for the GUS, however. No matter since DOSBox has great SB/Pro/16 emulation.

I’m also a bit puzzled about why the DOSBox screenshots are 720 x 480 (posted here are various cropping and resizings).

Things went pretty smoothly. We suspected that there was an integer field that indicated the frame rate, but 18 fps is a bit strange. I kept fixating on a header field that read 0x41F00000. Where have I seen that number before? Oh, of course — it’s the number 30.0 expressed as an IEEE 32-bit float. The 4XM format pulled the same trick.

Hexadecimal Easter Egg
I know I finished the game years ago but I really can’t recall any of the clips present in the samples directory. The file mgs1-60.vp3 contains a computer screen granting the player access and illustrates this with a hexdump. It looks something like this:

Funny, there are only 22 bytes on a line when there should be 32 according to the offsets. But, leave it to me to try to figure out what the file type is, regardless. I squinted and copied the first 22 bytes into a file:

 1F 8B 08 00   85 E2 17 38   00 03 EC 3A   0D 78 54 D5
 38 00 03 EC   3A 0D

And the answer to the big question:

$ file mgsfile
mgsfile: gzip compressed data, from Unix, last modified: Wed Oct 27 22:43:33 1999

A gzip’d file from 1999. I don’t know why I find this stuff so interesting, but I do. I guess it’s no more and less strange than writing playback systems like this.

Lords of Midnight
Check out the box copy scan for Lords of Midnight in MobyGames. In particular, I’d like to call your attention to this little blurb:

Ahem, “Journey through an immense world — the equivalent of 8 CD-ROMs.” Yet, when I procured the game, it only came on a single CD-ROM. It’s definitely a CD-ROM (says so on the disc) and, coming from 1995, certainly predates the earliest DVD-ROMs (which can easily store 8 CD-ROMs on a disc). Thus, I wanted to jump in a see if they were using some phenomenal compression in order to squeeze so much info into 600 or so megabytes.

I was surprised to see the contents of the disc clocking in at just under 40 megabytes. An intro movie and an outro movie account for 75% of that. Format? None other than that curious ASCII anomaly, ARMovie/RPL with Escape 122 codec data.

Cyclemania

Cyclemania is one of those FMV backdrop action games, but with a motorcycle theme. I had a good feeling I would find some odd multimedia artifacts here and the game didn’t disappoint. The videos are apparently handled using 3-4 discrete files per animation. I’ve documented my cursory guesses and linked some samples at the new MultimediaWiki page.

Interplay ACMP
This is unrelated to this particular acquistion, but I was contacted today about audio files harvested from the 1993 DOS game Star Trek: Judgment Rites. The files begin with the ASCII signature “Interplay ACMP Data”. This reminds me of Interplay MVE files which begin with the similar string “Interplay MVE File”. My theory is that these files use the ACOMP compression format, though I’m still trying to make it fit.

Wiki and samples are available as usual if you’d like to add your own research.

SNES game cartridges — being all hardware — were at liberty to expand the hardware capabilities of the base system by adding new processors. The most well-known of these processors was the Super FX which allows for basic polygon graphical rendering, powering such games as Star Fox. It was by no means the only such add-on processor, though. Here is a Wikipedia page of all the enhancement chips used in assorted SNES games. A number of them mention compression and so I delved into the emulators to find the details:

• The Super FX is listed in Wikipedia vaguely as being able to decompress graphics. I see no reference to decompression in emulator source code.
• DSP-3 emulation source code makes reference to LZ-type compression as well as tree/symbol decoding. I’m not sure if the latter is a component of the former. Wikipedia lists the chip as supporting “Shannon-Fano bitstream decompression.”
• Similar to Super FX, the SA-1 chip is listed in Wikipedia as having some compression capabilities. Again, either that’s not true or none of the games that use the chip (notably Super Mario RPG) make use of the feature.
• The S-DD1 chip uses arithmetic and Golomb encoding for compressing graphics. Wikipedia refers to this as the ABS Lossless Entropy Algorithm. Googling for further details on that algorithm name yields no results, but I suspect it’s unrelated to anti-lock brakes. The algorithm is alleged to allow Star Ocean to smash 13 MB of graphics into a 4 MB cartridge ROM (largest size of an SNES cartridge).
• The SPC7110 can decompress data using a combination of arithmetic coding and Z-curve/Morton curve reordering.

No, I don’t plan to implement codecs for these schemes. But it’s always comforting to know that I could.

Not directly a compression scheme, but still a curious item is the MSU1 concept put forth by the bsnes emulator. This is a hypothetical coprocessor implemented by bsnes that gives an emulated cartridge access to a 4 GB address space. What to do with all this space? Allow for the playback of uncompressed PCM audio as well as uncompressed video at 240x144x256 colors @ 30 fps. According to the docs and the source code, the latter feature doesn’t appear to be implemented, though; only the raw PCM playback.

The Setup
One side effect of running this ridiculously niche interest blog at the intersection of multimedia, reverse engineering, and game hacking is that people occasionally contact me for assistance on those very matters. So it was when one of my MobyGames peers asked if I can help to extract some music from a game called Aztec Wars. The game consists of 2 discs, each with a music.xbe file that contains multiple tunes and is hundreds of megabytes large.

That’s all the data I received from the first email. At first I’m wondering what makes people think I have some magical insight into cracking these formats with such little information. Ordinarily, I would need to have the entire data file to work with and possibly the game binaries. But I didn’t want to ask him to upload hundreds of megabytes of data and I didn’t feel like downloading it; commitment issues and all.

But then I gathered a little confidence and remembered that the .xbe files are probably just Game Resource Archive Formats (GRAF) which are, traditionally, absurdly simple. I asked my colleague to send me a hexdump of the first kilobyte of one of the .xbe GRAFs ('hexdump -C -n 1024 music.xbe > file') as well as the total file size of the GRAF.

The Hexdump
The first music.xbe file is 192817376 bytes large. These are the first 1024 144 bytes (more than enough):

00000000  01 00 00 00 60 04 00 00  14 00 00 00 01 00 00 00  |....`...........|
00000010  0d 00 00 00 48 00 00 00  94 39 63 01 1c a4 21 03  |....H....9c..¤!.|
00000020  7a d2 54 04 04 28 ad 05  d8 88 fd 06 d8 88 fd 06  |zÒT..(­.Ø.ý.Ø.ý.|
00000030  2a 6e 46 08 2a 6e 46 08  2a 6e 46 08 2a 6e 46 08  |*nF.*nF.*nF.*nF.|
00000040  50 13 2f 0a e0 28 7e 0b  52 49 46 46 44 39 63 01  |P./.à(~.RIFFD9c.|
00000050  57 41 56 45 66 6d 74 20  10 00 00 00 01 00 02 00  |WAVEfmt ........|
00000060  44 ac 00 00 10 b1 02 00  04 00 10 00 64 61 74 61  |D¬...±......data|
00000070  fc 13 63 01 00 00 00 00  00 00 00 00 00 00 00 00  |ü.c.............|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

The Challenge
Armed with only the information in the foregoing section, figure out a method for extracting all the audio files in that file and advise on their playback/conversion. Ideally, this method should require minimal effort from both you and the person on the other end of the conversation.

The Resolution
The reason I ask is because I came up with a solution but knew, deep down, that there must be a slightly easier way. How would you solve this?

The music files in question are now preserved on YouTube (until they see fit to remove them for one reason or another).

I tried to recall how this screenshot came to exist. Had I actually created a functional KDE frontend to Nosefart yet neglected to release it? I think it’s more likely that I used some designer tool (possibly KDevelop) to prototype a frontend. This would have been sometime in 2000.

However, this screenshot prompted me to revisit Nosefart.

Nosefart Background
Nosefart is a program that can play Nintendo Sound Format (NSF) files. NSF files are files containing components that were surgically separated from Nintendo Entertainment System (NES) ROM dumps. These components contain the music playback engines for various games. An NSF player is a stripped down emulation system that can simulate the NES6502 CPU along with the custom hardware (2 square waves, 1 triangle wave, 1 noise generator, and 1 limited digital channel).

Nosefart was written by Matt Conte and eventually imported into a Sourceforge project, though it has not seen any development since then. The distribution contains standalone command line players for Linux and DOS, a GTK frontend for the Linux command line version, and plugins for Winamp, XMMS, and CL-Amp.

The Sourceforge project page notes that Nosefart is also part of XBMC. Let the record show that Nosefart is also incorporated into xine (I did that in 2002, I think).

Upgrading the API
When I tried running the command line version of Nosefart under Linux, I hit hard against the legacy audio API: OSS. Remember that?

In fairly short order, I was able to upgrade the CL program to use PulseAudio. The program is not especially sophisticated. It’s a single-threaded affair which checks for a keypress, processes an audio frame, and sends the frame out to the OSS file interface. All that was needed was to rewrite open_hardware() and close_hardware() for PA and then replace the write statement in play(). The only quirk that stood out is that including <pulse/pulseaudio.h> is insufficient for programming PA’s simple API. <pulse/simple.h> must be included separately.

For extra credit, I adapted the program to ALSA. The program uses the most simplistic audio output API possible — just keep filling a buffer and sending it out to the DAC.

Discovering GME
I’m not sure what to do with the the program now since, during my research to attempt to bring Nosefart up to date, I became aware of a software library named Game Music Emu, or GME. It’s a pure C++ library that can essentially play any classic video game format you can possible name. Wow. A lot can happen in 10 years when you’re not paying attention.

It’s such a well-written library that I didn’t need any tutorial or documentation to come up to speed. Just a quick read of the main gme.h header library enabled me in short order to whip up a quick C program that could play NSF and SPC files. Path of least resistance: Client program asks library to open a hardcoded file, synthesize 10 seconds of audio, and dump it into a file; ask the FLAC command line program to transcode raw data to .flac file; use ffplay to verify the results.

I might develop some other uses for this library.

Back In The Day
As some of you are likely aware, there was a dark age of online communication that predated the era of widespread internet access. This was known as “The BBS Age”. People dialed into these BBSes using modems that operated at abysmal transfer speeds and would communicate with other users, upload and download files, and play an occasional game.

BBS software evolved and perhaps the ultimate (and final) evolution was Galacticomm’s MajorBBS (MBBS). There were assorted games that plugged into the MBBS, all rendered in glorious color ANSI graphics. One of the most famous of these games was Tele-Arena (TA). TA was a multiplayer fantasy-themed text adventure game. Perhaps you could think of it as World of Warcraft, only rendered as interactive fiction instead of a rich 3D landscape. (Disclaimer: I might not be qualified to make that comparison since I have never experienced WoW firsthand, though I did play TA on and off about 17 years ago).

TA was often compared to multi-user dungeons — or MUDs — that were played by telneting into internet servers hosting games. Such comparisons were usually unfavorable as people who had experience with both TA and MUDs were sniffy elitists with internet access who thought they were sooooo much better than those filthy, BBS-dialing serfs.

Sorry, didn’t mean to open old wounds.

Modern Retelling of A Classic Tale
Anyway, my friend Ron Kinney is perhaps the world’s biggest fan of TA. So much so that he has re-implemented the engine in Java under the project name Ether. He’s in a similar situation as the ScummVM project in that, while the independent, open source engine is fair game for redistribution, it would be questionable to redistribute the original data files. That’s why he created an AreaBuilder application that generates independent game data files.

Ironically, you can also telnet into a server on which Ron hosts an instance of Tele-Arena (ironic in the sense that the internet/BBS conflict gets a little blurry).

I hope that one day Ron will regale us with the strangest tales from the classic TA days. My personal favorite was “Wrath of a Sysop.”

When mounted in Linux (as UDF), the disc is reported to contain 2.8 GB of data, so it has to be a DVD. 810 MB of that is dedicated to the movies/ directory. Multimedia format? Just plain, boring MPEG files (very YouTube-friendly– here’s the opening animation). Deeper digging reveals some more subdirectories called movies/ that, combined, occupy the lion’s share of the disc space. Additionally, there are several single-frame .m2v files in a directory called iframes/ which are used to encode things like load screens.

There are more interesting data files including .zbm files for images and fonts, and .zwf files for audio. I suspect that these stand for zipped bitmap and zipped wave file, respectively. They can’t be directly unzipped with ‘gunzip’. Some of the numbers at the start of some files lead me to believe they can be easily decompressed with standard zlib facilities.

Based on the binary files on the Gemz disc, I couldn’t find any data on what CPU this system might use. A little Googling led me to this page at the Video Game Console Library which pegs the brain as a Mediamatics 6811. Some searching for that leads me to a long-discontinued line of hardware from National Semiconductor.

The Console Library page also mentions that the games were developed using the Lua programming language. Indeed, there are many Lua-related strings in the game’s binaries (‘zlib’ also makes an appearance).

Background
Like many other programmers, I initially gained interest in programming because I desired to program video games. Not content to just plunk out games on a PC, I always had a deep, abiding ambition to program actual video game hardware. That is, I wanted to program a purpose-built video game console. The Sega Dreamcast might be the most ideal candidate to ever emerge for that task. All that was required to run your own software on the unit was the console, a PC, some free software tools, and a special connectivity measure.

The Equipment
Here is the hardware required (ideally) to build software for the DC:

• The console itself (I happen to have 3 of them laying around, as pictured above)
• Some peripherals: Such as the basic DC controller, the DC keyboard (flagship title: Typing of the Dead), and the visual memory unit (VMU)
  
  
  
• VGA box: The DC supported 480p gaming via a device that allowed you to connect the console straight to a VGA monitor via 15-pin D-sub. Not required for development, but very useful. I happen to have 3 of them from different third parties:
  
  
  
• Finally, the connectivity measure for hooking the DC to the PC.
  There are 2 options here. The first is rare, expensive and relatively fast: A DC broadband adapter. The second is slower but much less expensive and relatively easy to come by– the DC coder’s cable. This was a DB-9 adapter on one end and a DC serial adapter on the other, and a circuit in the middle to monkey with voltage levels or some such; I’m no electrical engineer. I procured this model from the notorious Lik Sang, well before that outfit was sued out of business.
  
  
  

Dealing With Legacy
Take a look at that coder’s cable again. DB-9? When was the last time you owned a computer with one of those? And then think farther back to the last time to had occasion to plug something into one of those ports (likely a serial mouse).

A few years ago, someone was about to toss out this Belkin USB to DB-9 serial converter when I intervened. I foresaw the day when I would dust off the coder’s cable. So now I can connect a USB serial cable to my Eee PC, which then connects via converter to a different serial cable, one which has its own conversion circuit that alters the connection to yet another type of serial cable.

Bits is bits is bits as far as I’m concerned.

Putting It All Together
Now to assemble all the pieces (plus a monitor) into one development desktop:

The monitor says “dcload 1.0.3, idle…”. That’s a custom boot CD-ROM that is patiently waiting to receive commands, code and data via the serial port.

Getting The Software
Back in the day, homebrew software development on the DC revolved around these components:

• GNU binutils: for building base toolchains for the Hitachi SH-4 main CPU as well as the ARM7-based audio coprocessor
• GNU gcc/g++: for building compilers on top of binutils for the 2 CPUs
• Newlib: a C library intended for embedded systems
• KallistiOS: an open source, real-time OS developed for the DC

The DC was my first exposure to building cross compilers. I developed some software for the DC in the earlier part of the decade. Now, I am trying to figure out how I did it, especially since I think I came up with a few interesting ideas at the time.

Struggling With the Software Legacy
The source for KallistiOS has gone untouched since about 2004 but is still around thanks to Sourceforge. The instructions for properly building the toolchain have been lost to time, or would be were it not for the Internet Archive’s copy of a site called Hangar Eleven. Also, KallistiOS makes reference to a program called ‘dc-tool’ which is needed on the client side for communicating with dcload. I was able to find this binary at the Boob! site (well-known in DC circles).

I was able to build the toolchain using binutils 2.20.1, gcc 4.5.1 and newlib 1.18.0. Building the toolchain is an odd process as it requires building the binutils, then building the C compiler, then newlib, and then building the C compiler again along with the C++ compiler because the C++ compiler depends on newlib.

With some effort, I got the toolchain to build KallistiOS and most of its example programs. I documented most of the tweaks I had to make, several of them exactly the same as this one that I recently discovered while resurrecting a 10-year-old C program (common construct in C programming of old?).

Moment of Truth
So I had some example programs built as ELF files. I told dc-tool to upload and run them on the waiting console. Unfortunately, the tool would just sort of stall, though some communication had evidently taken place. It has been many years since I have seen this in action but I recall that something more ought to be happening.

Plan B (Hardware)
This is the point that I remember that I have been holding onto one rather old little machine that still has a DB-9 serial port. It’s not especially ergonomic to set up. I have to run it on my floor because, to connect it to my network, I need to run a 25′ ethernet cable that just barely reaches from the other room. The machine doesn’t seem to like USB keyboards, which is a shame since I have long since ditched any PS/2 keyboards. Fortunately, the box still has an old Gentoo distro and is running sshd, a holdover from its former life as a headless box.

Now when I run dc-tool, both the PC and DC report the upload progress while pretty overscan bars oscillate on the DC’s monitor. Now I’m back in business, until…

Plan C (Software)
None of these KallistiOS example programs are working. Some are even reporting catastrophic failures (register dumps) via the serial console. That’s when I remember that gcc can be a bit fickle on CPU architectures that are not, shall we say, first-class citizens. Back in the day, gcc 2.95 was a certified no-go for SH-4 development. 3.0.3 or 3.0.4 was called upon at the time. As I’m hosting this toolchain on x86_64 right now, gcc 3.0.4 can’t even be built (predates the architecture).

One last option: As I searched through my old DC project directories, I found that I still have a lot of the resulting binaries, the ones I built 7-8 years ago. I upload a few of those and I finally see homebrew programming at work again, including this old program (described in detail here).

Next Steps
If I ever feel like revisiting this again, I suppose I can try some of the older 4.x series to see if they build valid programs. Alternatively, try building an x86_32-hosted 3.0.4 toolchain which ought to be a known good. And if that fails, search a little bit more to find that there are still active Dreamcast communities out there on the internet which probably have development toolchain binaries ready for download.