Regarding my toy VP8 encoder, Pengvado mentioned in the comments of my last post, “x264 looks perfect using only i16x16 DC mode. You must be doing something wrong in computing residual or fdct or quantization.” This makes a lot of sense. The encoder generates a series of elements which describe how to reconstruct the original image. Intra block reconstruction takes into consideration the following elements:

I have already verified that both my encoder and FFmpeg’s VP8 decoder agree precisely on how to reconstruct blocks based on the predictors, coefficients, and quantizers. Thus, if the decoded image still looks crazy, the elements the encoder is generating to describe the image must be wrong.

So I started studying the forward DCT, which I had cribbed wholesale from the original libvpx 0.9.0 source code. It should be noted that the formal VP8 spec only defines the inverse transform process, not the forward process. I was using a version designated as the “short” version, vs. the “fast” version. Then I looked at the 0.9.5 FDCT. Then I got the idea of comparing the results of each.

input: 92 91 89 86 91 90 88 86 89 89 89 88 89 87 88 93

• libvpx 0.9.0 “short”:

  forward: -314 5 1 5 4 5 -2 0 0 1 -1 -1 1 11 -3 -4
  inverse: 92 91 89 86 89 86 91 90 91 90 88 86 88 86 89 89
  

• libvpx 0.9.0 “fast”:

  forward: -314 4 0 5 4 4 -2 0 0 1 0 -1 1 11 -2 -5 
  inverse: 91 91 89 86 88 86 91 90 91 90 88 86 88 86 89 89 
  

• libvpx 0.9.5 “short”:

  forward: -312 7 1 0 1 12 -5 2 2 -3 3 -1 1 0 -2 1 
  inverse: 92 91 89 86 91 90 88 86 89 89 89 88 89 87 88 93 
  

I was surprised when I noticed that input[] != idct(fdct(input[])) in some of the above cases. Then I remembered that the aforementioned property isn’t what is meant by a “bit-exact” transform– only that all implementations of the inverse transform are supposed to produce bit-exact output for a given vector of input coefficients.

Anyway, I tried applying each of these forward transforms. I got slightly differing results, with the latest one I tried (the fdct from libvpx 0.9.5) producing the best results (to my eye). At least the trees look better in the Big Buck Bunny logo image:

The dense trees of the Big Buck Bunny logo using one of the libvpx 0.9.0 forward transforms

The same segment of the image using the libvpx 0.9.5 forward transform

Then again, it could be that the different numbers generated by the newer forward transform triggered different prediction modes to be chosen. Overall, adapting the newer FDCT did not dramatically improve the encoding quality.

Working on the intra 4×4 mode encoding is generating some rather more accurate blocks than my intra 16×16 encoder. Pengvado indicated that x264 generates perfectly legible results when forcing the encoder to only use intra 16×16 mode. To be honest, I’m having trouble understanding how that can possibly occur thanks to the Walsh-Hadamard transform (WHT). I think that’s where a lot of the error is creeping in with my intra 16×16 encoder. Then again, FFmpeg implements an inverse WHT function that bears ‘vp8’ in its name. This implies that it’s custom to the algorithm and not exactly shared with H.264.

I know people are anxious to see what happens next with my toy VP8 encoder. First and foremost, I corrected the encoder’s DC prediction. A lot of rules govern that mode and if you don’t have it right, error cascades through the image. Now the encoder and decoder both agree on every fine detail of the bitstream syntax and rendering thereof. It still encodes to a neo-impressionist mosaic piece, but at least I’ve ironed the bugs out of this phase:

I also made it possible to adjust the quantization levels inside the encoder. This means that I’m finally getting some compression out of the thing, vs. the original approach of hardcoding the minimum quantizers.

So I’m stubbornly plugging away at my toy VP8 encoder and I managed to produce this gem. See if you can spot the subtle mistake:

The misplaced color plane resulted from using the luma plane stride where it was not appropriate. I fixed that and now chroma planes are wired to use to the same naive prediction algorithm as the luma plane.

Also, I fixed the entropy encoder so that end of block conditions are signaled correctly (instead of my original, suboptimal hack to just encode all zeros). I was disappointed to see that this did not result in a major compression improvement. Then again, I’m using the lowest possible quantization settings for this outing, so perhaps this is to be expected.

Sigh… 4×4 luma prediction is next. Wish me luck.

I’ve been toiling away as a multimedia technology generalist for so long that it’s easy for me to forget that not everyone is as versed in the minutiae of the domain as I am. But I recently experienced what it’s like to be such an outsider when I posted about my toy VP8 encoder, expressing that it’s one of the hardest things I have ever tried to do. I heard from a number of people who do have extensive experience in video encoding, particularly with the H.264 and VP8 codecs. Their reactions were predictable: What’s so hard? Look, you might be a little too immersed in the area to really understand a relative beginner’s perspective.

And to all the people who suggested that I should get the encoder into FFmpeg ASAP: Are you crazy?! Did you see what the first pass of the encoder produced? Do you have lower standards than even I do?

Not Giving Up
I worked a little more on the toy encoder. Remember that the above image is what I’m hoping to encode somewhat faithfully for this experiment. In my first pass, I attempted vertical prediction for all planes. For my next pass, I forced the chroma planes to mid-level (which results in a greyscale image) and played with the 16×16 luma prediction modes. When implementing an extremely naive algorithm to decide which 16×16 prediction mode would be the best for a particular block, this is what the program produced:

For fun, here is what the image encodes to when forcing various prediction modes:
Continue reading →