Implementing A Testbed For The Palettized Animation Video Codec

I decided it is time to stop talking about what I going do and start talking about how I am going to do it. I want to try my hand at creating a new video codec. This is my first attempt at such a feat. I need to write both the encoder and decoder simultaneously. I need to develop some idea about which coding theories I wish to test and how I will go about testing them. I need test vectors for encoding.

But first and foremost, I need a name for this new codec. I have decided on “Palettized Animation Video Codec.” That is what its name shall be unless someone can come up with a better name. The fourcc shall be ‘PAVC’ which is yet unclaimed (although ‘PDVC’ is claimed according to fourcc.org). Since this is an article about PAVC, I need to lead in with some cool shots of old video games, if only as a reminder of this endeavor’s challenge:

Crystalis, NES
Widely regarded as the finest video game ever produced

Crystalis 		Solar Jetman

Solar Jetman, NES
Perhaps the toughest Nintendo game to get the hang of

Things I need to start this experiment:

  • Test vectors: I need some sample media to encode.
    Done: I have already used emulator tools to capture raw RGBA32 video frames into AVI files. I have test vectors from NES and SNES games. And I can easily generate many more, hard drive space permitting (1 second of a raw NES video requires over 14 megabytes).
  • Codec development infrastructure: I am using the FFmpeg open source project for developing this codec. As a first step, I will make a stand-in codec module that is actually identical to RGBA32. The codec skeleton will simply be a passthrough and will not modify the video data in any way.
  • Codec validation infrastructure: Write out a sequence of RGB bitmaps generated from the raw test vectors. Develop a method for validating whether the new encode->decode process yields the same data.
  • Develop a palette transport model: The first step of encoding the first frame of video (or a keyframe) will be to build a new palette based on the new data. Do this by scanning the first frame’s pixels and storing each new color. Also, count each palette occurrence since this data might be useful in later video coding concepts. After settling on a palette table, figure out the minimum number of bits needed to represent the palette index. For example, an NES frame should have no more than 32 unique colors. Thus, a palette index (frame pixel) only needs 5 bits of resolution. Next, convert the frame to a palette index map.
  • Develop a basic coding model: Encode a keyframe with a header that specifies number of colors to unpack and the resolution of pixel indices. Encode each pixel using that number of bits. Write a decoder to take apart the basic keyframe.

Screenshots from the MobyGames database.