Monthly Archives: November 2007

XIntra8 In FFmpeg

Rejoice! Thanks to the inimitable multimedia hacker Allan Smithee, FFmpeg now supports the XIntra8 coding scheme! Why is this important? Also known informally as the J-frame (and X8Intra and IntraX8), the XIntra8 coding mode has long been the missing piece of the Microsoft WMV2 puzzle and is apparently also used in certain variations of WMV3.

Update: Check out Kostya’s rant on the matter for better details on exactly what XIntra8 is, and why it is so painful.

This has been a longstanding problem for FFmpeg’s open source WMV2 decoder. You would be watching a WMV2-encoded video and suddenly, obnoxiously, there would be a severe glitch where you could watch blocks of incorrect colors moving around the screen. For example:


Britney Spears -- Not A Girl, Not Yet A Woman -- XIntra8 blocky decode

The XIntra8 is a different type of intraframe from the usual I-frame found in WMV2. Since the decoder could not handle the data, the policy was to just copy over the previous interframe and proceed with more frames, and hope that a regular I-frame was not too far in the future.

But now, I can finally properly watch this WMV2 encode of Britney Spears old music video for “Not A Girl, Not Yet A Woman”:


Britney Spears -- Not A Girl, Not Yet A Woman -- XIntra8 correct decode

Oh, don’t try to claim that you don’t have an extensive collection of her works. It’s okay to state that you have amassed the collection strictly for the academic purpose of multimedia study. That’s my story and I’m sticking to it.

Sofdec Support

FFmpeg now includes support for the Sofdec middleware format thanks to Aurel Jacobs and Mans Rullgard, as well as everyone who has made FFmpeg’s MPEG video decoding what it is today. Sonic the Hedgehog salutes you:



Sonic Adventure 2: Battle
Sonic Adventure 2: Battle

Sofdec is a multimedia middleware format that was used heavily on the Sega Dreamcast. Indeed, if you booted a DC game, there might have been a 50/50 chance that you would see Sofdec’s technology insignias among the many obligatory corporate logos. Sofdec still survives to this day and is seen on various GameCube games (often developed by Sega’s subsidiary houses). It probably runs on all the other consoles as well. In fact, I see that MobyGames maintains a game group for CRI-using games.

The thing about Sofdec files is that they are fundamentally MPEG files with MPEG-1 video. The only thing special about them is that they are packaged with a custom ADPCM format called CRI ADX. I checked out the new FFmpeg support with files from a variety of games. One of the biggest problems is blocky output. On files from certain games (for example, F-Zero GX, Resident Evil 4, and Starfox Assault), it almost appears that only DC data is being decoded. FFmpeg does not report any decoding problems. The result is something like this:


Blocky GameCube logo

I am pretty sure that the above is supposed to be the official GameCube logo, which looks like this:


Nintendo GameCube Logo

There are also a few videos from Resident Evil: Code Veronica X on the GameCube that display as a square 320×320 frame, which is not correct:


Resident Evil: Code Veronica X-- Claire Redfield, square aspect ratio

I wonder if aspect ratio information is stored inside this file format, or if perhaps the data is in some other place in the game’s data. Not all of the videos in the game are like this, though.

Many new samples are available in the usual place.

Palette Communication

If there is one meager accomplishment I think I can claim in the realm of open source multimedia, it would be as the point-man on palette support in xine, MPlayer, and FFmpeg.


Palette icon

Problem statement: Many multimedia formats — typically older formats — need to deal with color palettes alongside compressed video. There are generally three situations arising from paletted video codecs:

  1. The palette is encoded in the video codec’s data stream. This makes palette handling easy since the media player does not need to care about ferrying special data between layers. Examples: Autodesk FLIC and Westwood VQA.
  2. The palette is part of the transport container’s header data. Generally, a modular media player will need to communicate the palette from the file demuxer layer to the video decoder layer via an out-of-band/extradata channel provided by the program’s architecture. Examples: QuickTime files containing Apple Animation (RLE) or Apple Video (SMC) data.
  3. The palette is stored separately from the video data and must be transported between the demuxer and the video decoder. However, the palette could potentially change at any time during playback. This can provide a challenge if the media player is designed with the assumption that a palette would only occur at initialization. Examples: AVI files containing paletted video data (such as MS RLE) and Wing Commander III MVE.

Transporting the palette from the demuxer layer to the decoder layer is not the only be part of the battle. In some applications, such as FFmpeg, the palette data also needs to travel from the decoder layer to the video output layer, the part that creates a final video frame to either be displayed or converted. This used to cause a problem for the multithreaded ffplay component of FFmpeg. The original mechanism (that I put into place) was not thread-safe– palette changes ended up occurring sooner than they were supposed to. The primary ffmpeg command line conversion tool is single-threaded so it does not have the same problem. xine is multi-threaded but does not suffer from the ffplay problem because all data sent from the video decoder layer to the video output layer must be in a YUV format, thus paletted images are converted before leaving the layer. I’m not sure about MPlayer these days, but when I implemented a paletted format (FLIC), I rendered the data in higher bit depths in the decoder layer. I would be interested to know if MPlayer’s video output layer can handle palettes directly these days.

I hope this has been educational from a practical multimedia hacking perspective.

Android Multimedia

I honestly do not understand much about Google’s new Android platform. But if I gather correctly from the surrounding hype and press releases, it’s the answer to all of my prayers and will make every single one of my dreams come true.

If you’re like me (and, let’s face it, if you read this blog then you probably are), you are interested in the multimedia features of this proposed platform.


Android phone logo image

The literature that I have perused thus far has not made any remarkable claims with respect to multimedia capabilities amidst all the other wish-granting facets of the phone framework. The above picture, which seems to be the mascot image, specifically sports that ubiquitous symbol of multimedia playback — the rightward-facing arrow in a circle.

Since “open source” is a key selling point of this platform, what are they going to use for a general multimedia backend? I can hardly imagine.