Many thanks to VMware for contributing to the existing MultimediaWiki description for their own VMware Video codec. This is a a lossless codec used for efficiently and losslessly capturing action from virtualized computer desktops running under VMware’s Workstation product. The community had worked out a number of the codec details but VMware people have filled in the gaps.
Sam Hocevar — who I last saw with his groundbreaking counter-CAPTCHA research named PWNtcha — has been working on a fuzzing program called zzuf. Fuzzing is when you create random, implicitly fraudulent data and feed it as input into a program which is supposed to process the data and make sure that the program fails gracefully when chewing on the pathological data. I learned about this technique some time ago and have always wanted to launch it against our body of open source multimedia software. Sam has done just this. None of the multimedia players tested fared particularly well.
FFmpeg devs have already corrected a number of the issues uncovered by this cursory fuzz test. According to the program’s description, zzuf intercepts file operations and changes random bits in the program’s input. This is a great start to the process. However, it will likely only attack the demuxer layer of the various multimedia applications. A deeper method would be to fuzz, e.g., individual chunks of codec data on the way from the demuxer to its corresponding decoder. This could probably be rigged up in FFmpeg fairly easily (just be sure to make the behavior deterministic so any bugs can be reproduced).
I have embarked on an ambitious new project, with a new blog to match: Gaming Pathology. Remember all those games I have picked up on the cheap so I can study their multimedia? I have managed to amass in excess of 500 games. I decided that this year I would try to play one of them each day. My early focus is on games that are missing data from the MobyGames database. As vast as the database is, I still have at least 80 games out of 500+ that don’t exist in the tables at all. That fact is changing steadily.
Anyway, I thought it would be pertinent to announce the Gaming Pathology blog here since I know that this blog has readers interested in game hacking. Sometimes, my reviews on the new blog include theories or empirical evidence of the underlying technologies, algorithms, and formats that the games use.
And lots of pretty screenshots and videos as well.
Karim sends word of a new lossless audio codec under development.
The codec is wryly and appropriately entitled Yet Another Lossless Audio Codec (YALAC) that was originally named TAK. I had that backwards: YALAC was renamed TAK. Obligatory new MultimediaWiki page.
That’s not the most interesting part of Karim’s email. His email notified me of this post which was the first indication I received that there is a entire weblog devoted to lossless audio coding — The Lossless Audio Blog! And I thought this blog was niche. The blog’s sidebar mentions Sony’s ATRAC as being lossless, which I was unaware of (rather, a different variant called ATRAC Advanced Lossless). Also, DTS-HD and Dolby True-HD are listed as lossless codecs.
It’s amazing how much activity there is in the lossless audio codec field. I’ll be keeping an eye on that blog, as should you, the multimedia tech enthusiast.
YouTube, as you probably know, is a wildly popular website that simply hosts videos that users can view by using Adobe’s Flash Player as a client. A significant part of its popularity comes from its accessibility. Users don’t have to know much about multimedia formats. They just upload whatever file they have and YouTube magically converts it to the format that they need internally in order to serve it out to other users.
I just started uploading videos to YouTube recently and decided to see just how flexible YouTube is in terms of the input formats that it accepts. In other words, I’ve been throwing lots of different files at YouTube and seeing what sticks. YouTube turns out to accept a remarkable array of obscure multimedia formats, which just happens to be my specialty. How does it handle all of these formats? I have heard rumors that it uses FFmpeg on the backend. Based on my experiments, it is more likely that it is using MEncoder, if they are using any pre-made bit of open source software. It could also be that they are using their own homegrown piece of software.
Either way, I have a strong feeling that FFmpeg contributors could feel justified in adding “Freelance YouTube Software Engineer” to their resumes/CVs. What makes me suspect this? Because I wrote FFmpeg code for a number of the atypical multimedia formats that YouTube supports and because of that, I know where the bugs are. When certain videos from the vast sample collection are converted, they exhibit the same bugs that they do under xine, FFmpeg’s ffplay, or MPlayer. However, ffplay and MPlayer behave differently on certain files and that fact offers clues about whether YouTube’s backend software might be using FFmpeg or MEncoder.
The first video I tried to upload was encoded with DOSBox’s ZMBV codec. FFmpeg has supported this codec since early last year. YouTube declined to convert it. Hmm, maybe it doesn’t use FFmpeg after all. So I went through a process with FFmpeg to convert the video to MPEG-4 with minimum quality loss and upload it, since YouTube really likes MPEG-4. Then I realized that, since I didn’t instruct FFmpeg otherwise, FFmpeg inserted its default ISO MPEG-4 video FourCC of ‘FMP4′. YouTube had no problem with that. I then tried a sample of one of my very favorite formats– Sega FILM. YouTube took it! The A/V sync was a little off, but it still worked. That’s when I went to work seeing what else YouTube could handle.
YouTube’s behaviors for Interplay MVE, Westwood VQA, 4XM, and ZyGo are all precisely consistent with MPlayer 1.0rc1 but not with FFmpeg, at least not the latest SVN copy. But I keep coming back to my baby, Sega FILM/CPK. I worked hard to sort out the details of this format and I have found a lot of quirks in different files. The different media programs that support FILM/CPK handle the quirks differently. Examine this bizarre video, for example, which involves a butterfly and a semi-automatic firearm. After the butterfly goes down, there is supposed to be more dialog while the video remains static. But the video conversion ended early. This behavior is consistent with MPlayer but not FFmpeg.
This somewhat reminds me of nmap’s method of detecting the network stack implementation for a machine on the other end of the internet– see how the stack handles a series of poorly-defined network packets.
Also, check out YouTube’s conversion of sonic3dblast_intro.avi from this location. It shows that YouTube’s software at least has Reimar Döffinger’s fix to the Duck TrueMotion v1 decoder (the great-great-great granddaddy of On2′s VP6 video codec) as outlined in this post (compare the image and note the difference in how it looked before the fix). This would indicate that YouTube’s version of the FFmpeg library — in whatever backend software they are using — is from after November, 2005 (when Reimar submitted the Duck TM1 upgrade) and before February, 2006 (when Kostya first submitted the ZMBV decoder).
One more important data point is that YouTube can also convert Indeo 4 and Indeo 5 video, and probably other codecs that are only supported, reliably, via Win32 binary codecs. FFmpeg can’t do that out of the box while MEncoder can.
I’m a little confused by the fact that a particular Id RoQ file, when converted by YouTube, has video a little ahead of the audio, while MPlayer plays the audio a bit ahead of the video. FFmpeg’s ffplay has proper sync. Though it’s essential to note that if YouTube is using code from sometime between the 11/2005-2/2006 timeframe, the current FFmpeg/MPlayer code I am experimenting with can not possibly be guaranteed to produce the same results.
I would like to take this opportunity to say that I’m not at all upset, bitter, or jealous to learn that YouTube is probably using free and open source software that I helped write in order to build one of the most popular sites on the internet which was recently purchased for a non-negligible amount of money by some search engine outfit. Further, they are perfectly within their rights to do this, and such usage adheres to the GPL/LGPL. I just found this to be a fascinating exercise and I’m amused to see how far this software has gotten. Further, I hope the new Wiki page can somehow help people create better YouTube videos (because a lot of YouTube video contributors could use the help).
Besides, now I work on the client-side software that helps to drive YouTube’s success.
IMDb Studio Briefing carries a news snippet today about new Blu-Ray discs that can play in Sony PlayStation 3 units but not in standalone players: Sony Encounters New Blu-ray Glitches. It seems that the new discs use some system called BD-Java for processing extras and additional features. Just when you thought multimedia tech couldn’t get more complicated and bloated. So now players have to have some kind of Java VM?
As usual, Wikipedia is on top of it.