The FFmpeg project made a formal release yesterday. You can download version 0.6 from the project’s download page.

I’ve actually been seeing more news items about this today than I would have expected. This is most likely because the 0.6 release is named “Works with HTML5″. Let it never be said that nerds don’t know marketing. The team really latched on to the hottest buzzword going right now.

The name of the release refers to FFmpeg’s new native support for Google’s WebM format. It can mux and demux the WebM container, and decode the Vorbis audio, all natively (FFmpeg’s Vorbis encoder has been demoted to “experimental” for this release and it is recommended to enable libvorbis for encoding). But the big news is that this release can support Google’s libvpx natively for VP8 encoding and decoding, without having to apply any other patches.

Getting libvpx to compile still might be a bit tricky. Fortunately, the first pass of a native, independent VP8 decoder is currently in review on the ffmpeg-devel list.

At the recent Linux Foundation Collaboration Summit, I was chatting with someone heavily involved in a major open source SQL database program. I asked how he feels about the interminable hype surrounding so-called NoSQL databases. Among other sentiments, he mentioned that he felt positive regarding the fact that there is competition in the open source database arena once more which is driving much innovation.

Of course, I have to frame everything through my multimedia lense and I pondered the notion of competition in open source multimedia development. Some of my fondest memories of open source development come from 2002-2003 when the xine and MPlayer teams were engaged in open war, trying to create the best open source multimedia player. The competition was a major factor.

These days, the deepest multimedia development occurs on the FFmpeg project. FFmpeg is pretty much a category-killer. There’s nothing else like it in the open source or proprietary worlds (and if you think you know of a competitor, it’s probably using FFmpeg as its backend). Would things move faster if there were serious competitors to FFmpeg?

Thanks to Daniel Verkamp for contributing a Kega video (KGV1) decoder to FFmpeg. I was about to demand samples for testing until I looked up what Kega is — a Sega game console emulator — when I realized that it would be more fun to create my own (be advised that only the Windows version of Kega presently supports the AVI encoding option). Then I looked up the Wiki page and realized that there is, in fact, one sample on record at the archive. Well, I went ahead and made my own sample anyway. I used the mountainside attract-mode scene from the Genesis game Strider, one of my favorite sequences in any video game. It’s in the samples directory.

I am holding off on adding a FATE test; there’s still an endian issue (PPC configs disagree with x86 configs). I’m also a little puzzled as to why FFplay insists on playing the video as 320×240 even though the video is encoded as 640×480. For that matter, I’m also bewildered trying to understand why Kega renders video as 640×480 by default; that’s not a native resolution for any of its emulated consoles.

I have been reading way too many statements from people who confidently assert that Google will open source all of On2′s IP based on no more evidence than… the fact that they really, really hope it happens. Meanwhile, I have found myself pettily hoping it doesn’t happen simply due to the knowledge that the FSF will claim total credit for such a development (don’t believe me? They already claim credit for Apple dropping DRM from music purchases: “Our Defective by Design campaign has a successful history of targeting Apple over its DRM policies… and under the pressure Steve Jobs dropped DRM on music.”)

But for the sake of discussion, let’s run with the idea: Let’s assume that Google open sources any of On2′s intellectual property. Be advised that if you’re the type who believes that all engineering problems large and small can be solved by applying, not thought, but a mystical, nebulous force called “open source”, you can go ahead and skip this post.

The Stack

Read the rest of this entry »

Today was the day: Kostya committed his Bink video decoder to FFmpeg. Here’s just one little screenshot:

Of course, this is just one Bink file out of the literal thousands of software titles that have incorporated Bink video (the above comes from Indiana Jones and the Emperor’s Tomb for Windows). For this reason, it’s entirely possible that the Bink video decoder (not to mention the Bink audio decoder and the Bink file format demuxer) might not cover all the cases out there. This is especially relevant considering intel I have received from a guy who has talked to the guy who invented Bink and described the development process. The upshot is that there could conceivably be a lot of custom Bink versions out there. That’s why Kostya hopes for a lot of testing with as many different Bink files that people can throw at this system. To that end, I started with my old Multimedia Exploration Journal and did a text search for every game that I recorded as using Bink.

Just think: The next time that YouTube and assorted other video uploading services update their video conversion backends, they can finally be flooded with Bink videos. (I know it seems silly, but I sometimes feel like my biggest contribution to open source multimedia has been to allow people to upload to YouTube video files that they found on their old Sega Saturn CD-ROMs).

As for FATE, is it plausible to get a basic decoding test staged at this point? I ran a simple sample through my RPC testing tool and learned that the video output is bit exact across platforms. Test staged.

(Aside: Thanks to Vitor Sessak, Valgrinder extraordinaire, for locating a memory bug in the Musepack v7 demuxer. Since I created and staged a v7 sample at the same time I staged a sample for the Musepack v8 demuxer, I have already activated a Musepack v7 demuxing test.)

Here’s a project for someone that likes text processing and searching puzzles: Find a simple, efficient method for comparing my list of DOS/Windows games (here’s the HTML list and here it is in CSV) against the big list of known Bink titles and find all the Bink games in my PC game collection. I have already harvested samples from: Alien vs. Predator Gold Edition, Disney’s Atlantis, Gabriel Knight 3, Gods & Generals, Halo 3 (Xbox 360), In Cold Blood, Indiana Jones and the Emperor’s Tomb, Monsters Inc. Wreck Room Arcade, Starlancer, Tony Hawk Pro Skater 2, Uru: Ages Beyond Myst.

Ronald Bultje has been a long-time contributor to a variety of open source multimedia projects. He was keen to try his hand at reverse engineering and implementing an undiscovered codec. Most people start simple, but Ronald went for a vocoder (significantly more complex than the piddly little ADPCM codecs I started with). He has completed his reverse engineering of the Windows Media Audio 9 Voice algorithm and committed a decoder for FFmpeg. If you’re interested in the technical details, check out Ronald’s blog posts on the matter: Codec Woes and WMA Voice Codec Dissection.

Here is a WMA Voice file being played in FFplay using Michael’s spectrum visualization (now the default audio visualization):

We have a solution to the problem of making QuickTime/MP4 files streamable. It’s called qt-faststart. The solution has problems which we have tried to remedy over the years. Recently, I proposed another patch to another problem. But can we obviate the need for qt-faststart entirely in favor of a more integrated solution? Is that even a good idea?

Every so often, the FFmpeg project receives a bug report about qt-faststart operating incorrectly– it would mysteriously no-op and output a blank file. Each time, we have to dredge up our recollections of what causes this and how to fix it. Turns out that the problem is always caused by users manually compiling the utility (‘gcc qt-faststart.c -o qt-faststart’) which will produce incorrect operation on 64-bit platforms. The solution is to build it with FFmpeg’s build system (after running ‘./configure’, run ‘make tools/qt-faststart’). I even wrote that down in the header comments of qt-faststart.c.

Then I smacked myself hard for expecting average end users to actually read source code comments. It’s bad enough that they have to compile a program in the first place. For the average user, it’s laudable that they figured out enough to run ‘gcc’ manually. When the compiler didn’t complain, that’s reason for optimism.

I decided to modify qt-faststart.c so that it fails to compile via a simple gcc build command while printing out a helpful error message. Then I got to pondering the classic problem of muxing a streamable QT/MP4 file in the first place. Here’s what I’m thinking:

Estimating Header Space
If the duration of an input file is known at the outset, it should be reasonable to estimate how much space the moov header will need. Develop a formula based on the input file’s duration, video output frames per second, and target audio codec characteristics, and decide how much space to set aside. The more frames there will be in the target file, the more header bytes will need to be set aside for entries in the various sample tables. At this phase, calculate the amount of space to set aside for all specified metadata. Add a little space to the computed header size for good measure, create a new file, and jump straight ahead to the position indicated by that size to start writing the mdat atom. After the mdat atom has been laid down, write down the moov atom plus a free space atom to make up any size difference.

Naive Fallback
If the input format does not specify its total duration (perhaps a live source or it might be from any of a number of file formats for which there is simply no efficient way to compute duration without decoding the whole file), then the whole of qt-faststart could be effectively integrated into the QT/MP4 muxer as a post-processing phase.

Is This A Good Idea?
I get the impression that FFmpeg is a major player in the world of video conversion. Further, QT/MP4 is pretty much the ubiquitous standard these days. I worry about changing a fundamental bit of the way the biggest tool creates QT/MP4 files. There must be many toolchains and installations out there which already perform the “mux; qt-faststart” sequence. Will changing this behavior hurt anything? qt-faststart doesn’t do anything to a file that is already streamble; it doesn’t even create a blank file. So modifying FFmpeg to directly create streamable QT/MP4 files will break programs that expect to run ‘FFmpeg && qt-faststart’.

One alternative would be to add streamable remuxing as another command line option. But that somewhat ruins the user-friendliess aspect of creating the desired streamable files per the default mode of operation.

I don’t have any answers right now and certainly no time to code a prototype (nor inclination, unless I’m darn sure the idea would be accepted into the codebase).

See Also:

• Improving qt-faststart

Executive Summary: I tried to get a Broadcom CrystalHD chip to work in Linux. I came close to being successful. The chip, kernel driver, and userspace library all work. The example app that would have been the payoff… not so much. I document my process in this post in case others need assistance, or can lend assistance in the final step.

There was some news recently about Broadcom open sourcing code related to a video decoding chip. The brand name here is apparently “CrystalHD” and the chip in question is the BCM-70012. I came into possession of one of these and endeavored to make the open source software surrounding it work in Linux.

The first issue is installing the hardware. It’s a PCI Express mini card which has the same form factor as a PCIe mini wireless networking card. So if your computer can host such a wireless card, it can also hold this thing. Allegedly. First, I tried to place it in the empty PCIe-mini slot in my MSI Wind Nettop. No go. The machine refused to boot up (it would power up but never beep to indicate that it’s really ready to run). Removing the card made the problem go away.

So determined was I to make this chip work that I actually took apart my dear Eee PC 701, ripped out the wireless card and replaced it with the Broadcom card. Deciding it would be too much trouble to attempt to re-attach the keyboard and touchpad ribbons, I realized I could just use USB peripherals.

Resigned to the notion that I just foolishly destroyed my 2 year old Eee PC, I threw the switch anyway and was quite surprised to see it boot up normally. An ‘lspci’ command indicates a new Broadcom multimedia controller hanging off the PCI bus. It’s not pretty but it’s breathing:

So let’s talk software. Broadcom released the driver as open source. To many in the open source community, this is tantamount to, “Okay, done deal! What else needs to be open sourced?” Not so fast. There’s no documentation in the whole package (user-wise, anyway; the libcrystalhd API is thoroughly documented in header comments). So I will describe my experiences with the software.

Read the rest of this entry »

UPDATE, 2010-06-17: This information is now maintained via the FFmpeg Metadata page on the MultimediaWiki.

While creating my automated game archiving solution, I wanted to automatically tag ALAC/M4A files with appropriate metadata while encoding using FFmpeg. Then I realized I didn’t know how to do that. I do remember that FFmpeg recently supplanted a series of specific metadata command line options (like -title and -album) with a highly generalized metadata framework that is accessed by the option ‘-metadata <key>=<value>’. The official documentation provides this lonely (and, I came to realize, useless NO-OP; more on why later) example of the option’s usage:

    `-metadata key=value'

      Set a metadata key/value pair. For example,
      for setting the title in the output file:

      ffmpeg -i in.avi -metadata title="my title" out.flv

So this option allows you to specify absolutely any key/value metadata pair you can imagine. However, a specific muxer won’t necessarily recognize it. How can I know the specific keys that, e.g., the MOV/MP4 muxer honors? 2 methods spring to mind: Trial and error (worked out well for me at first) and reading the code (which was made a good deal easier when I already knew a few of the keys that worked).

I think it would be a good idea to document the specific metadata keys that each muxer in FFmpeg recognizes. This blog post will lead by example. This data comes from SVN revision 20910, current as of 2009-12-21.

QuickTime/MOV/MP4/M4A/et al.

I have constructed the following table of the metadata keys that libavformat/movenc.c honors. The low-level identifier column lists the atom name that the format uses to encode the data on disc, which is not interesting to most readers. For the interested but uninitiated, the notation, e.g., ‘\251nam’ indicates a 4-byte code consisting of the byte A9 in hexadecimal (or 251 in octal) followed by the ASCII characters ‘n’, ‘a’, and ‘m’.

Read the rest of this entry »

FFmpeg is quite an amazing program. There’s a certain smugness that comes with being involved with it. That can lead to a bit of complacency followed by shock when realizing that you’re not as good as you thought you were.

That happened to me recently when I realized the official libtheora decoder is significantly more performant than FFmpeg’s Theora decoder. I suddenly wondered if this was true in any other departments, i.e., if FFmpeg is slower than other open source libraries that are dedicated to a single purpose. Why do I care? Because I started to wonder if FFmpeg would simply come to be known as the gcc of multimedia processing.

Is it good or bad to be compared to gcc in this way? Depends; gcc has its pros and cons. A colleague once succinctly summarized these trade-offs thusly: “You can generally count on gcc to generate adequate code for just about any platform.” Some free software fans grow indignant when repeated benchmarks unequivocally show, e.g., Intel’s proprietary compilers slaughtering gcc’s various versions. But what do you expect? gcc spreads the effort around to all kinds of chips while Intel largely focuses on generating code for chips that they invented and know better than anyone else. Frankly, I’ve always admired gcc for being able to do as well as it does.

But does it have to be that way with FFmpeg? “You can generally count on FFmpeg to be able to decode a particular format fairly quickly and encode to a wide variety of formats with reasonable quality.” That’s certainly the case currently regarding Theora (it can decode the format, just not as efficiently as libtheora). What about some other notable formats? I think some tests are in order.

Read the rest of this entry »