FFmpeg is quite an amazing program. There’s a certain smugness that comes with being involved with it. That can lead to a bit of complacency followed by shock when realizing that you’re not as good as you thought you were.

That happened to me recently when I realized the official libtheora decoder is significantly more performant than FFmpeg’s Theora decoder. I suddenly wondered if this was true in any other departments, i.e., if FFmpeg is slower than other open source libraries that are dedicated to a single purpose. Why do I care? Because I started to wonder if FFmpeg would simply come to be known as the gcc of multimedia processing.

Is it good or bad to be compared to gcc in this way? Depends; gcc has its pros and cons. A colleague once succinctly summarized these trade-offs thusly: “You can generally count on gcc to generate adequate code for just about any platform.” Some free software fans grow indignant when repeated benchmarks unequivocally show, e.g., Intel’s proprietary compilers slaughtering gcc’s various versions. But what do you expect? gcc spreads the effort around to all kinds of chips while Intel largely focuses on generating code for chips that they invented and know better than anyone else. Frankly, I’ve always admired gcc for being able to do as well as it does.

But does it have to be that way with FFmpeg? “You can generally count on FFmpeg to be able to decode a particular format fairly quickly and encode to a wide variety of formats with reasonable quality.” That’s certainly the case currently regarding Theora (it can decode the format, just not as efficiently as libtheora). What about some other notable formats? I think some tests are in order.

Read the rest of this entry »

FFmpeg crossed the 20,000 commit threshold today. Mans captured the distinction when he submitted an ARM NEON optimization for int32_to_float_fmul_scalar(). Does that warrant a prize? Diego presented the statistics:

It took 7 years to get to r10000, but only two more to get to r20000.
FFmpeg is approaching warp 6 :-)

Today was also the day I noticed that YouTube upgraded their backend conversion system somewhere along the line. Nearly 3 years ago, I started poking at YouTube to see what kind of multimedia files it can convert and cataloged my findings at the MultimediaWiki.

Today, I was clicking around on some of my old videos and noticed that this video which came from an Ogg Theora source now looks correct. Actually, according to the comments (and I receive enough between all my videos that I rarely pay attention to any of them), this was working over a year ago.

It’s interesting to note that this means that YouTube/Google keeps all of the source material that users upload. When it was time to recode, they obviously had to go back to the original material.

I found that CSCD, KMVC, 3iv2, ZMBV, and VP6 video codecs all work; Vivo files, Westwood v2 VQAs, Real files with RV40, and the bastardized FLIC files from The Magic Carpet are all fine as well; Wing Commander III MVE files, id CIN files, and Interplay MVE files all transcode with audio but with either missing or glitched video.

Sorry if I seem a bit sentimental about this but it all still amazes me. When I was writing the bulk of the subsystems for all manner of bizarre formats circa 2001-2003, I never could have imagined that there would be a website that would take the weird video formats as input and convert them to a standard video format for anyone to view.

A little over a year ago, Sun was making rumblings about a brand new video codec that they were hoping to design from the ground up using known-good (read: patent-unencumbered) coding algorithms. This was to be called the Open Media Stack (OMS). I wrote a post about, made an obligatory MultimediaWiki page about it, and then promptly forgot all about it.

Today, by way of a blog post by Opera’s Bruce Lawson describing why HTML5′s <video> tag is, well, stalled (to put it charitably), I learned that Sun’s OMS team has published at least 2 specs, the latest one being posted a few weeks ago on June 9, 2009. As he notes, the proposed Oracle buyout of Sun puts the OMS project’s status in limbo.

The spec page links to forum threads where interested parties can discuss issues in the spec. There aren’t many comments but the ones that exist seem to revolve around the codec’s articial resolution limitations. For my part, I wonder about how to encapsulate it into a container format for transport. The format specifies a sequence header that is 96 bits (12 bytes) in length, though there are provisions for (currently unused) extended data as well as free-form user data. The sequence header would be categorized as extradata in an AVI file or in a MOV file’s video stsd atom. Successive frames should be straightforward to store in a container format since the coding method only seems to employ intra-and inter-frames. Each frame begins with a header which specifies a 37-bit timestamp in reference to a 90 kHz clock. This allows for a recording that’s just over 1 week in length. It’s also one of those highly redundant items if this format were to be stuffed in a more generalized container format.

Anyway, the main video algorithm uses arithmetic and Golomb coding for its entropy coding; 8×8 pixel macroblocks which can either be an entire block unto themselves or subdivided into 4 4×4 sub-blocks; colorspace is YUV 4:2:0 planar; specifies bit-exact 2×2, 4×4, and 8×8 transforms and quantization procedures; spacial prediction using left, top-left, top, and top-right blocks; precisely-specified 1/4-pel motion compensation. All in all, it appears relatively simple; the 0.91 spec (annexes and all) weighs in at a mere 96 pages.

Naturally, there are no reference implementations or samples available. This got me to wondering about how one goes about creating a reference implementation of a codec (or a production implementation, for that matter). The encoder needs to come first so that it can generate samples. Only then can a decoder be written. Ideally, the initial encoder and decoder should be 2 totally different programs, written by 2 different people, though hopefully working closely together (speedy communication helps). There is wisdom in the FFmpeg community about not basing the primary encoder and decoder on the same base library after we reverse engineered one of the Real video codecs and found a fairly obvious bug that occurred in both sides of the codec.

I think I know one way to ensure that the encoder and decoder don’t share any pieces– write them in different computer languages.

I’m still wondering what kind of application would need to record video continuously for up to a week. How about a closed-circuit security camera? With a terabyte drive, it could store video for a week assuming a bitrate of 1.5 Mbits/sec. That’s roughly the same bitrate as the original MPEG-1 standard. If this coding method compresses more efficiently than MPEG-1, this might be a plausible application.

FFmpeg has been accepted as a mentoring organization in the Google Summer of Code 2009 program. This is the program’s 5th year and FFmpeg’s 4th year in the program. Thanks as always to the big ‘G’.

To refresh, the GSoC pays students $4500 for a summer’s worth of work on qualified open source projects. FFmpeg is one of those projects. FFmpeg team members mentor various multimedia-related projects. Our current project proposals are listed in the wiki. And, as with the previous 2 years, we have some stringent requirements for our students– namely, a student must complete a small FFmpeg task, refining the solution based on critical feedback until it is part of the mainline codebase.

I personally haven’t committed to any mentor possibilities. I wonder if I should propose the Theora encoder again, which didn’t quite pan out last year? The thing is practically written already in my mind.

If you’re reading this, the multimedia.cx DNS change has propagated, yet again, for another hosting service. But that’s a venomous tale for another blog post. FFmpeg v0.5 hit the web yesterday, right about the time that my new hosting service started having serious problems. Thus, I was unable to observe the occasion at the time.

But now I’m able to let it sink in a bit more… wow. I can’t even remember the last time we had a release and to be honest, I don’t really want to look it up because it’s a little embarrassing to think about. Thanks to all my co-devs who made the release what it is. But thanks especially to Diego Biurrun who stuck his neck out and pushed hard for this release effort starting a little over a month ago. Somehow, we got the release out.

Rest assured, we are taking notes this time around in order to make the process easier next time around. This is the dawn of a new era of open source multimedia (here’s hoping, anyway).

Don’t look now, but the idea of making a formal FFmpeg release in the very near future is gaining traction over on the FFmpeg-devel list. There is a determined and growing team of individuals committed to this goal in the near future. To that end, there is even a Wiki page listing the steps that need to occur leading up to a release.

There is some debate about what kind of release schedule we should commit to. Some take the stance that we should follow the model that Wine apparently follows by releasing every 2 weeks. Really, every new weeks? I had never heard of that before. It strikes me as a bit severe. I personally would like to see a little more time between releases, but see them often enough that we get really good at it, and it becomes part of our shared community mindset.

After all, there are a few projects out there that depend on FFmpeg.

I admit, whenever I get to thinking about an FFmpeg release, I get a little nervous about the ensuing discussion. Not the discussion about whether or not a release should happen; that’s a foregone conclusion because it will happen. No, the item that makes me lose sleep is the biggest bikeshed hue discussion of them all — how should we version the upcoming release of FFmpeg? I don’t think I even want to get involved in that one.

Wow, it seems like only yesterday that I downloaded the newly open sourced On2 VpVision source code package and started reverse engineering an English language description of the VP3 video coding algorithm. Well, actually, that was nearly 7 years ago. VP3 eventually formed the basis of the Xiph Theora video codec. And today Theora is pleased to announce that the codec is finally, well… final. It’s out. No more alpha/beta phases. The codec is ready for primetime use and should be conquering the digital media frontier in short order.

You know, just like Vorbis.

I haven’t forgotten about that goal of converting Interplay MVE files to a more modern format. In fact, I have been periodically updating my FFmpeg and libx264 snapshots in order to take another stab at the problem. The crashing issue I experienced before turned out to be a known FFmpeg-x264 interaction issue that was being actively discussed on the ffmpeg-devel mailing list at the time I was experiencing the problem. Robert’s guides helped, too, at least in the not-crashing department.

But… I don’t know… sometimes I can’t shake the feeling that x264 is an elaborate hoax perpetrated by a number of my open source multimedia colleagues.

I think the foregoing movie was generated using the default FFmpeg preset, but I got similarly awful results for all the profiles. I guess libx264 is working, and achieving miraculous compression rates to boot. Maybe a little too much for my goals, though. The resulting files decode better in Apple’s QuickTime Player than they do with FFmpeg’s decoder– makes me wonder what those 119 H.264 decoder tests in the FATE suite are even useful for.

For his FFmpeg Summer of Code project, Jai Menon successfully completed an encoder for the Apple Lossless Audio Codec (ALAC). As you can see, Jai’s encoder is competitive with Apple’s iTunes, as well as dBpoweramp, another program that implements an ALAC encoder:

Thanks to Benjamin Larsson for setting up the graph. We’re rooting for the cyan bar to be the lowest one of each bar quartet. The code is already in the mainline tree.

The 13 samples I used for this test come from the OpenMusic CD and showcase a reasonable diversity of genres. The raw rips are available here.

The MOOBEX project was really the first real exposure I had to the Amarok application. For the uninitiated, Amarok is a rather full-featured KDE-based music playing application and has been around since 2003. Back when I used Linux more exclusively, I stuck to xine directly for all of my media playback needs.

How was my first impression of Amarok? Well… it’s hard to say. The program was different on each distribution that I tried. In OpenSUSE, I was thoroughly frustrated when I first tried to play a file through Amarok, or rather, when I selected an audio file through the file manager and just let the system do its thing. Amarok popped up and just sort of stared at me. Okay, I recognize that I’m throwing a bunch of formats toward Amarok that the distro has administratively determined to be “legally unsafe”. But I should still get some obvious feedback to that effect. Oh, wait, look way down there in the status bar:

When I launched an audio file on another KDE-based distro installation (Mandriva), I expected the same behavior from Amarok. To my surprise, the program threw a far more brazen error. Ubuntu has Amarok version 1.4.9.1. Mandriva sports 1.4.8. The Amarok version in Mandriva exhibited the superior UI characteristic. I understand the source of the unsupported codecs. But I wonder about the dialog discrepancy. I am not sure if this dialog behavior is a version specific thing in Amarok, or a configuration option in Amarok, or an dialog characterstic configurable in the hosting KDE libraries. It’s a minor matter, to be sure, but it still makes me wonder how individual distributions would be able to devalue something like the Adobe Flash Player if it were open source and allowed distros to make “choice” changes to the functionality.

I’m sure that the Amarok developers worked very hard to produce the best music player they could. And it probably hurts to see their fruits of their labor mangled by various distributions to the point of practical worthlessness. I know the feeling based on my work on xine.

But here’s a huge annoyance with Amarok which I suspect is a core behavior: stop should mean STOP! When I did get a file to play and then tried to stop playback, the audio kept right on going long enough to make me think that something was wrong. Eventually, the audio started to gently fade out. This is not reasonable behavior. The expected action for a stop button is an immediate halt.