Here’s an honest engineering question: If you were tasked to make compressed video play back on a Sega Dreamcast video game console, what video format would you choose? Personally, I would choose RoQ, the format invented for The 11th Hour computer game and later used in Quake III and other games derived from the same engine. This post explains my reasoning.
Video Background
One of the things I wanted to do when I procured a used Sega Dreamcast back in 2001 was turn it into a set-top video playback unit. This is something that a lot of people tried to do, apparently, to varying degrees of success. Interest would wane in a few years as it became easier and easier to crack an Xbox and install XBMC. The Xbox was much better suited to playing codecs that were getting big at the time, most notably MPEG-4 part 2 video (DivX/XviD).
The Dreamcast, while quite capable when it was released in 1999, was not very well-equipped to deal with an MPEG-type codec. I have recently learned that there are other hackers out there on the internet who are still trying to get the most out of this system. I was contacted for advice about how to make Theora perform better on the Dreamcast.
Interesting thing about consoles and codecs: Since you are necessarily distributing code along with your data, you have far more freedom to use whatever codecs you want for your audio and video data. This is why Vorbis and even Theora have seen quite a bit of use in video games, “internet standards” be darned. Thus, when I realized this application had no hard and fast requirement to use Theora, and that it could use any codec that fit the platform, my mind started churning. When I was programming the DC 10 years ago, I didn’t have access to the same wealth of multimedia knowledge that is currently available.
Requirements Gathering
What do we need here?
- Codec needs to run on the Sega Dreamcast; this eliminates codecs for which only binary decoder implementations are available
- Must decode 320×240 video at 30 fps; higher resolutions up to 640×480 would be desirable
- Must deliver decent quality at 12X optical read speeds (DC drive speed)
- There must be some decent, preferably free, encoder readily available; speed of encoding, however, is not important; i.e., “take as long as you need, encoder”
Theora was the go-to codec because it’s just commonly known as “the free, open source video codec”. But clearly it’s not suitable for, well… any purpose, really (sorry, easy target; OW! stop throwing things!). VP8/WebM — Theora’s heir apparent — would not qualify either, as my prior experiments have already demonstrated.
Candidates
What did the big boys use for video on the Dreamcast? A lot of games relied on CRI’s Sofdec middleware which was MPEG-1 video and a custom ADPCM format. I don’t know if I have ever seen DC games that used MPEG-1 video at a higher resolution than 320×240 (though I have not searched exhaustively). The fact that CRI used a custom ADPCM format for this application may indicate that there wasn’t enough CPU power left over to decode a perceptual, transform-based audio codec alongside the 320×240 video.
A few other DC games used 4X Technologies’ 4XM format. The most notable licensee was Alone in the Dark: The New Nightmare (DC version only; PC version used Bink). This codec was DCT-based but incorporated 16-bit RGB colorspace into its design, presumably to optimize for applications like game consoles that couldn’t directly handle planar YUV. AITD:TNN’s videos were 640×360, a marked improvement over the typical Sofdec fare. I was about to write off 4XM as a contender due to lack of encoder, but the encoding tools are preserved on our samples site. A few other issues, though: The FFmpeg decoder doesn’t seem to work correctly as of this writing (and nobody has noticed yet, even though it’s tested via FATE).
What ideas do I have? Right off the bat, I’m thinking vector quantizer (VQ). Vector quantizers are notoriously slow to compress but are blazingly fast to decompress which is why they were popular in the early days of video compression. First, there’s Cinepak. I fear that might be too simple for this application. Plus, I don’t know if existing (binary-only) compressors are very decent. It seems that they only ever had to handle small videos and I’ve heard that they can really fall over if anything more is demanded of them.
Sorenson Video 1 is another contender. FFmpeg has an encoder (which some allege is better than Sorenson’s original compressor). However, I fear that the wonky algorithm and colorspace might not mesh well with the Dreamcast.
My thinking quickly converged on RoQ. This was designed to run fullscreen (640×480) video on i486-class hardware. While RoQ fundamentally operates in a YUV colorspace, it’s trivial to convert it to any other colorspace during decoding and the image will be rendered in that colorspace. Plus, there are open source encoders available for the format (namely, several versions of Eric Lasota’s Switchblade encoder, one of which lives natively in FFmpeg), as well as the original proprietary encoder.
Which Library?
There are several code choices here: FFmpeg (LGPL), Switchblade (GPL), and the original Quake 3 source code (GPL). There is one more option that I think might be easiest, which is the decoder Dr. Tim created when he reverse engineered the format in the first place. That has a very liberal “do whatever you like, but be nice and give me credit” license (probably qualifies as BSD).
This code is no longer at its original home but the Wayback Machine still had a copy, which I have now mirrored (idroq.tar.gz).
Adaptation
Dr. Tim’s code still compiles and runs great on Linux (64-bit!) with SDL output. I would like to get it ported to the Dreamcast using the same SDL output, which KallistiOS supports. Then, there is the matter of fixing the longstanding chroma bug in the original sample decoder (described here). The decoder also needs to be modified to natively render RGB565 data, as that will work best with the DC’s graphics hardware.
After making the code work, I want to profile it and test whether it can handle full-frame 640×480 playback at 30 frames/second. I will need to contrive a sample to achieve this.
Unfortunately, things went off the rails pretty quickly when I tried to get the RoQ decoder ported to DC/KOS. It looks like there’s a bug in KallistiOS’s minimalistic standard C library, or at least a discrepancy with my desktop Linux system. When you read to the end of a file and then seek backwards to someplace that isn’t the end, is the file still in EOF state?
According to my Linux desktop:
open file; feof() = 0 seek to end; feof() = 0 read one more byte; feof() = 1 seek back to start; feof() = 0
According to KallistiOS:
open file; feof() = 0 seek to end; feof() = 0 read one more byte; feof() = 1 seek back to start; feof() = 1
Here’s the seek-test.c program I used to test this issue:
EOF
Speaking of EOF, I’m about done for this evening.
What codec would you select for this task, given the requirements involved?
Thanks a lot, it looks very promising !
How would 320×240 MJPEG stack up?
Also, I’d love to see a few more VP8 speed tests with different encoder settings. Tests with all filtering disabled, and with intra-frames only would be interesting.
@Brooss: Actually, MJPEG didn’t even occur to me. That should be a trivial solution to wire up since KallistiOS has libjpeg built in. But it seems that it would suffer from the YUV intermediate step (either a proper RGB conversion or the awkward YUV packing).
Then again, the prevailing FMV format on GameCube games is motion JPEG running at 60 fps, and that system has a PPC-based CPU running at a little under 500 MHz. No word on how or if it had to do colorspace conversion after decoding.
Motion PNG might also be a contender if done right (i.e., the source data lends itself to PNG compression and all PNGs are in 16-bit RGB and decoded straight to an RGB565 bitmap).
RoQ seems like a decent choice, but SVQ1 looks simple enough based on what I’ve read of the decoder and encoder in libavcodec. I’m not familiar enough with the limitations of the Dreamcast to call it though.
Regarding Cinepak, I thought I’d take the opportunity to plug the encoder I wrote for it a while back, since that might be fun to play with. It RD:s the number of strips, choice of modes (per strip and per MB) and codebook sizes, but only does so based on PSNR.
See http://titan.codemill.se/~tomhar/cinepakenc.patch
First, I’d avoid trying to use SDL on the Dreamcast. It tends to be slow and unstable. Everything with it is done with direct writes to the framebuffer (using the version of SDL that’s included with KOS). Then again, just for something that’s rendering a video with a one-pass write to the framebuffer, this is probably not much of a problem…
Second, are you using KOS-1.2.0, or are you using a version pulled from the SVN? If you’re using a SVN version, then your feof bug is probably not in KOS proper, but in Newlib, as KOS doesn’t even have its own minimal libc anymore.
@Tjoppen: Awesome! I can’t believe someone finally wrote an open source Cinepak encoder. I can’t wait to try it out.
I have a weird feeling about SVQ1 because of its colorspace, YUV 4:1:0 planar. Then again, it would probably be possible to render onto a YUV422 packed texture during the primary decode operation.
I smell a bake-off coming between various codecs. The contenders include Cinepak, SVQ1, RoQ, motion JPEG, and motion PNG.
@BlueCrab: I’m using KOS 1.2.0, the last real release. I have never touched the SVN version.
Can you tell me where there is an active community of DC developers? If enough people are working on their own KOS forks (I already have a laundry list of mods I’ve had to make to KOS 1.2.0), perhaps we should collaborate. Or perhaps there is a thriving community already that I need to get involved with.
@Mike: Retro codec shootout? :)
In case it isn’t obvious from the code, the encoder can only be controlled with -qscale. I think the roqvideo encoder has the same problem though. You can also enable it to try different number of strips by changing MAX_STRIPS. Finally, it also supports grayscale encoding.
One other possible contender could be MxPEG. It’s basically MJPEG with the ability to skip macroblocks (zero motion, like cinepak). There’s a decoder on FFmpeg-devel.
Mike: Bluecrab and I are the last active maintainers of KOS, and it seems he beat me to the punch here. KOS 1.2.0 is really not suggested for use anymore, and the SVN version (despite the typical stigmas attached to ‘bleeding edge’) is really preferable in all cases. There simply hasn’t been an official ‘release’ due to indecisiveness.
You can find most active DC developers on the forums at dcemulation.org (which is how we found this article), or on IRC at #dreamcastdev on irc.freenode.net.
As to forks, there aren’t really many, there have been many developers through the years who have posted their own mods/fixes/features which have been later integrated into the main SVN. Again, we’d welcome hearing about any modifications you’ve made, though it would probably be best to check out the SVN versions of KOS first as there have been significant changes since 1.2.0.
Multimedia Mike Says:
March 9th, 2011 at 12:12 am
@Brooss: Actually, MJPEG didn’t even occur to me. That should be a trivial solution to wire up since KallistiOS has libjpeg built in. But it seems that it would suffer from the YUV intermediate step (either a proper RGB conversion or the awkward YUV packing).
————————————————————-
Well it seems the PVR does support YUV4:2:0 colorspace!
Take a look at this doc on the DC hardware:
[code]3.6.1.2 YUV Textures
YUV textures are expressed by 16 bits per texel, and one data item (YUV422 data) corresponds to two adjacent texels in the horizontal direction. The Y data for the left texel and the U data for both texels are collectively referred to as “Y0U” data, and the Y data for the right texel together with the V data for both texels are referred to as “Y1V†data. Each YUV data element is specified by an unsigned 8-bit value from 0 to 255.
MPEG data (YUV420 data in macro block units) is converted into YUV texture data by passing the data through the YUV data converter in the Tile Accelerator. (Refer to section 3.8.1.) [/code]
[code]3.8.1 YUV-data Converter
The Tile Accelerator has a YUV-data converter that converts YUV420- or YUV422-format data into YUV422-format texture data in macro block units (16 pixels ´ 16 pixels), and then stores the data in texture memory. DMA transfers from system memory to the TA are performed one macro block of YUV data at a time. The transfer order of each type of data must be as shown below.
For YUV420 format
(1) Transfer U data for 16 ´ 16 pixels (64 bytes).
(2) Transfer V data for 16 ´ 16 pixels (64 bytes).
(3) Transfer Y data for 16 ´ 16 pixels (256 bytes).
The U and V data are each stored in the internal buffer one time; once half of the Y data has been loaded into the internal buffer, transfer to texture memory begins. Once texture data for 16 ´ 8 pixels has been output, the other half of the Y data is loaded into the internal buffer and then the texture data for the remaining 16 ´ 8 pixels is transferred to texture memory.[/code]
Theres more about it, too much to paste just take a look at the doc.
If someone can get this function of the PVR working it would be amazing.
Sorry heres the link to the document
(redacted)
@Quzar: Actually, I saw a link to here in IRC, not on the forums. ;-)
@Mike: Certainly if you have any interesting contributions, I’d be willing to take a look at them. I’ve been rather busy with stuff for work and class lately, but I’d certainly take a look when I have the time and see about committing any good modifications. Also, as Quzar said, the SVN version has changed quite a bit since 1.2.0. I’d definitely recommend it strongly over 1.2.0, but it’ll involve compiling a new toolchain. The KOS SVN has a Makefile that can be used for building the new toolchain along with all the patches needed to GCC and Newlib.
@PH3NOM: That certainly looks like a document that Sega probably didn’t want distributed outside of Sega. I’ve come across it before, and pretty much archived it and left it alone. I’d like to actually keep it that way, myself, just out of principle.
@BlueCrab: I have no way to remove the link, sorry…
I just got a little excited there when I read that the DC has a dedicated pipeline for converting YUV 4:2:0 textures, which nobody has been able to utilize yet.
@PH3NOM: Since you indicated you wanted the link removed, I have done so.