The Split
The biggest deal in open source multimedia was the matter of the project split. Where once stood one project (FFmpeg) there now stands two (also Libav). Where do things stand with the projects now? Still very separate but similar. Both projects obsessively monitor each other’s git commits and prodigiously poach each other’s work, both projects being LGPL and all. Most features that land in one code base end up in the other. Thus, I refer to FFmpeg and Libav collectively as “the projects”.

Some philosophical reasons for the split included project stagnation and development process friction. Curiously, these problems are fond memories now and the spirit of competition has pushed development forward at a blinding pace.

People inside the project have strong opinions about the split; that’s understandable. People outside the project have strong opinions about the split; that’s somewhat less understandable, but whatever. After 5 years of working for Adobe on the Flash Player (a.k.a. the most hated software in all existence if internet nerds are to be believed on the matter), I’m so over internet nerd drama.

For my part, I just try to maintain some appearance of neutrality since I manage some shared resources for the open source multimedia community (like the wiki and samples repo) and am trying to keep them from fracturing as well.

Apple and Open Source
It was big news that Apple magnanimously open sourced their lossless audio codec. That sets a great example and precedent.

New Features
I mined the 'git log' of the projects in order to pick out some features that were added during 2011.

First off, Apple’s ProRes video codec was reverse engineered and incorporated into the multimedia libraries. And for some weird reason, this is an item that made the rounds in the geek press. I’m not entirely sure why, but it may have something to do with inter-project conflict. Anyway, here is the decoder in action, playing a video of some wild swine, one of the few samples we have:

Other new video codecs included a reverse engineered Indeo 4 decoder. Gotta catch ‘em all! That completes our collection of Indeo codecs. But that wasn’t enough– this year, we got a completely revised Indeo 3 decoder (the previous one, while functional, exhibited a lot of code artifacts betraying a direct ASM ->C translation). Oh, and many thanks to Kostya for this gem:

That’s the new Origin Xan decoder (best known for Wing Commander IV cinematics) in action, something I first started reverse engineering back in 2002. Thanks to Kostya for picking up my slack yet again.

Continuing with the codec section, there is a decoder for Adobe Flash Screen Video 2 — big congrats on this! One of my jobs at Adobe was documenting this format to the outside world and I was afraid I could never quite make it clear enough to build a complete re-implementation. But the team came through.

Let’s see, there are decoders for VBLE video, Ut Video, Windows Media Image (WMVP/WMP2), Bink audio version ‘b’, H.264 4:2:2 intra frames, and MxPEG video. There is a DPX image encoder, a Cirrus Logic AccuPak video encoder, and a v410 codec.

How about some more game stuff? The projects saw — at long last — an SMJPEG demuxer. This will finally allow usage and testing of the SMJPEG IMA ADPCM audio decoder I added about a decade ago. Funny story behind that– I was porting all of my decoders from xine which included the SMJPEG ADPCM. I just never quite got around to writing a corresponding demuxer. Thanks to Paul Mahol for taking care of that.

Here’s a DFA playback system for a 1995 DOS CD-ROM title called Chronomaster. No format is too obscure, nor its encoded contents too cheesy:

There’s now a demuxer for a format called XMV that was (is?) prevalent on Xbox titles. Now the projects can handle FMV files from many Xbox games, such as Thrillville.

The projects also gained the ability to play BMV files. I think this surfing wizard comes from Discworld II. It’s non-computer-generated animation at a strange resolution.

More demuxers: xWMA, PlayStation Portable PMP format, and CRI ADX format; muxer for OpenMG audio and LATM muxer/demuxer.

One more thing: an AVX-optimized fast Fourier transform (FFT). If you have a machine that supports AVX, there’s no way you’ll even notice the speed increase of a few measly FFT calls for audio coding/decoding, but that’s hardly the point. The projects always use everything on offer for any CPU.

Please make me aware of features that I missed in the list!

Continuous Testing
As a result of the split, each project has its own FATE server, one for FFmpeg and one for Libav. As of the new year, FFmpeg has just over 1000 tests while Libav had 965. This is one area where I’m obviously ecstatic to see competition. Some ad-hoc measurements on my part indicate that the total code coverage via the FATEs has not appreciably increased. But that’s a total percentage. Both the test count and the code count have been steadily rising.

Google Summer of Code and Google Code-In
Once again, the projects were allowed to participate in the Google Summer of Code as well as Google Code-In. I confess that I didn’t keep up with these too carefully (and Code-In is still in progress as of this writing). I do know that the project split occurred after FFmpeg had already been accepted for GSoC season 2011 and the admins were gracious enough to allow FFmpeg and Libav to allow both projects to participate in the same slot as long as they could both be mature about it.

Happy New Year
Let’s see what we can accomplish in 2012.

I’ve actually been seeing more news items about this today than I would have expected. This is most likely because the 0.6 release is named “Works with HTML5″. Let it never be said that nerds don’t know marketing. The team really latched on to the hottest buzzword going right now.

The name of the release refers to FFmpeg’s new native support for Google’s WebM format. It can mux and demux the WebM container, and decode the Vorbis audio, all natively (FFmpeg’s Vorbis encoder has been demoted to “experimental” for this release and it is recommended to enable libvorbis for encoding). But the big news is that this release can support Google’s libvpx natively for VP8 encoding and decoding, without having to apply any other patches.

Getting libvpx to compile still might be a bit tricky. Fortunately, the first pass of a native, independent VP8 decoder is currently in review on the ffmpeg-devel list.

Of course, I have to frame everything through my multimedia lense and I pondered the notion of competition in open source multimedia development. Some of my fondest memories of open source development come from 2002-2003 when the xine and MPlayer teams were engaged in open war, trying to create the best open source multimedia player. The competition was a major factor.

These days, the deepest multimedia development occurs on the FFmpeg project. FFmpeg is pretty much a category-killer. There’s nothing else like it in the open source or proprietary worlds (and if you think you know of a competitor, it’s probably using FFmpeg as its backend). Would things move faster if there were serious competitors to FFmpeg?

I am holding off on adding a FATE test; there’s still an endian issue (PPC configs disagree with x86 configs). I’m also a little puzzled as to why FFplay insists on playing the video as 320×240 even though the video is encoded as 640×480. For that matter, I’m also bewildered trying to understand why Kega renders video as 640×480 by default; that’s not a native resolution for any of its emulated consoles.

But for the sake of discussion, let’s run with the idea: Let’s assume that Google open sources any of On2′s intellectual property. Be advised that if you’re the type who believes that all engineering problems large and small can be solved by applying, not thought, but a mystical, nebulous force called “open source”, you can go ahead and skip this post.

The Stack

I like thinking about specific multimedia problems on this blog because people who read this blog often know beans about multimedia technology and are equipped to think about the issues involved. So imagine that Google open sources On2′s video codecs. What does that mean? Free web video for all! Well, not so fast. There are a few issues to consider. While most observers can only see as far as the video codec portion, there are, at the very least, 3 things to consider when putting together a video for consumption:

1. Video codec — how are the pictures compressed and encoded?
2. Audio codec — presumably, you want some sound to go along with those moving pictures
3. Container format — there has to be a method for tying together audio and video for delivery and playback

So, Google open sourcing On2′s video codecs would address the first need. What about the second need? Obviously, MP3 and AAC — which represent the standards today — would just get the open video argument back to square one. “VORBIS!!!” would be the reflexive cry of the open source checklister crowd. Indeed, this just might work. Vorbis has problems and shortcomings but is generally considered a reasonable replacement for MP3, WMA, and AAC, at least for desktop computing applications (we’ll think outside of the desktop arena a little later). Actually, there is another option: Did you know that On2 dabbled in audio codecs at one time? They developed 2 IMA ADPCM variants (DK4 and DK3) which wouldn’t be seriously considered for this application. They also have something they simply called Audio for Video Codec (inspired). I know we have samples of the latter somewhere.

How about the last component on the list, the container format? How about Ogg? Ogg is not usually considered a general-purpose container format. It would require extension to be capable of storing any new codec (I argue that a container is not general-purpose if a demuxer has to have special code to handle every codec that could possibly be inside, which is pretty much how Ogg works). The prevailing container format on the standard MPEG side is the QuickTime-derived MP4 format. This could hold new On2 data (and probably does so in various applications today), but I’m not sure about encoding Vorbis data.

This is the part of the post where I learned something new (which is a big reason I like writing these blog posts). My theory was that there was no agreed-upon method for storing Vorbis audio data in QuickTime/MP4. But if there were something resembling a standard on this front, FFmpeg would implement it. So I tried transcoding an audio file to a Vorbis encoded .MOV or .MP4 and not only did both take, both played back in FFplay.

Mobile Considerations

Okay, so the MP4/VPn/Vorbis stack might just be crazy enough to work… in the desktop realm. There’s the emerging world out there collectively called “mobile” — cell phones, netbooks, tablet computers, and more — and they all want to be able to show video. They often do this with dedicated processors that are designed to decode the standard MPEG codecs (H.264 and AAC). I’ve read from various observers who wave away this problem of decoding a new stack on mobile by invoking 3 magic letters: ‘DSP’. This is a good example of how a little knowledge can be dangerous. There seems to be a general misconception that DSPs (digital signal processors) are extraordinary devices that can be programmed to accelerate the decoding of any video or audio codec. The reality isn’t so simple.

I confess, I don’t know to what extent today’s generation of mobile devices employ general-purpose, programmable DSPs vs. custom MPEG ASICs but that would make for an interesting survey. While I’m confident that any device in the “mobile” category could likely be programmed to play data encoded in the MP4/VPn/Vorbis stack, the issue then becomes battery usage– how long can the custom programmed system play video vs. how long could the same unit play by pushing MPEG data through the custom ASICs? Again, don’t know, but it would be interesting to find some numbers or develop experiments to test this.

The Google/YouTube Factor

Observers obsess about the Google/YouTube angle in all this, which is understandable due to the large mindshare that YouTube controls when it comes to web video. “Control the YouTube video format, control the future of the internet,” or so the popular sentiment seems to go (be advised, however, that YouTube doesn’t necessarily represent as much online video viewing as you might think). One received bit of wisdom is that Google already stores multiple copies of every video in various formats, so one more shouldn’t be any big burden. Our thoughts are not Google’s thoughts, neither are Google’s ways our ways (I’m sure I read that somewhere). It’s hard to know exactly how YouTube operates, but we can always poke at it to find clues. Outside of Google, few people are as interested in YouTube’s operation as Dark Shikari of x264 fame and here are some interesting tidbits he has empirically determined. Of interest to this discussion is the following: “They could have saved 30-50% on bandwidth by offering an H.264 High stream for PC viewers, but they didn’t because it would have required they keep around a separate stream.” This indicates that Google doesn’t necessarily consider disk space to be limitless, at least not in the disk space vs. bandwidth trade-off calculation. I know for a fact that YouTube also keeps around the original copies of all the videos that have ever been uploaded (because certain videos that were not transcoded properly years ago are now correct). However, YouTube only needs to keep one copy of the original around (and maybe a backup somewhere) while transcoded videos need to be shipped off to any number of content distribution nodes.

This doesn’t rule out the possibility that Google could push an MP4/VPn/Vorbis stack as the new web video standard and maintain parallel MPEG & open copies of each YouTube video. But it’s not a foregone conclusion that it would be a drop in the bucket for Google’s resources. As a programmer for a large technology company, I endure more than my fair share of outside advice regarding what my company ought to do for the benefit of everyone else except the company. I can only imagine how frustrating it must be for Google people to constantly read declarations along the lines of, “Google needs to spend untold millions of its own money in order to do what I believe is right,” especially considering that whole “don’t be evil” charter/moral bludgeon.

In Closing

Anyway, I just wanted to think about these issues from a purely technical standpoint, a perspective that I really don’t see discussed anywhere else (except when I read Dark Shikari try to talk sense into online forums where video discussions abound). I don’t have all the facts but I enjoy not only searching for them, but also figuring out how to search for them.

Check out Silvia Pfeiffer’s recent blog post: Google’s challenges of freeing VP8. In addition to wondering about some of the same technical issues I have outlined here, she addresses the tough legal and patent issues surrounding a possible open sourcing. And none of this even begins to address the political and social issues surrounding the adoption of such a standard.

Of course, this is just one Bink file out of the literal thousands of software titles that have incorporated Bink video (the above comes from Indiana Jones and the Emperor’s Tomb for Windows). For this reason, it’s entirely possible that the Bink video decoder (not to mention the Bink audio decoder and the Bink file format demuxer) might not cover all the cases out there. This is especially relevant considering intel I have received from a guy who has talked to the guy who invented Bink and described the development process. The upshot is that there could conceivably be a lot of custom Bink versions out there. That’s why Kostya hopes for a lot of testing with as many different Bink files that people can throw at this system. To that end, I started with my old Multimedia Exploration Journal and did a text search for every game that I recorded as using Bink.

Just think: The next time that YouTube and assorted other video uploading services update their video conversion backends, they can finally be flooded with Bink videos. (I know it seems silly, but I sometimes feel like my biggest contribution to open source multimedia has been to allow people to upload to YouTube video files that they found on their old Sega Saturn CD-ROMs).

As for FATE, is it plausible to get a basic decoding test staged at this point? I ran a simple sample through my RPC testing tool and learned that the video output is bit exact across platforms. Test staged.

(Aside: Thanks to Vitor Sessak, Valgrinder extraordinaire, for locating a memory bug in the Musepack v7 demuxer. Since I created and staged a v7 sample at the same time I staged a sample for the Musepack v8 demuxer, I have already activated a Musepack v7 demuxing test.)

Here’s a project for someone that likes text processing and searching puzzles: Find a simple, efficient method for comparing my list of DOS/Windows games (here’s the HTML list and here it is in CSV) against the big list of known Bink titles and find all the Bink games in my PC game collection. I have already harvested samples from: Alien vs. Predator Gold Edition, Disney’s Atlantis, Gabriel Knight 3, Gods & Generals, Halo 3 (Xbox 360), In Cold Blood, Indiana Jones and the Emperor’s Tomb, Monsters Inc. Wreck Room Arcade, Starlancer, Tony Hawk Pro Skater 2, Uru: Ages Beyond Myst.

Here is a WMA Voice file being played in FFplay using Michael’s spectrum visualization (now the default audio visualization):

Every so often, the FFmpeg project receives a bug report about qt-faststart operating incorrectly– it would mysteriously no-op and output a blank file. Each time, we have to dredge up our recollections of what causes this and how to fix it. Turns out that the problem is always caused by users manually compiling the utility (‘gcc qt-faststart.c -o qt-faststart’) which will produce incorrect operation on 64-bit platforms. The solution is to build it with FFmpeg’s build system (after running ‘./configure’, run ‘make tools/qt-faststart’). I even wrote that down in the header comments of qt-faststart.c.

Then I smacked myself hard for expecting average end users to actually read source code comments. It’s bad enough that they have to compile a program in the first place. For the average user, it’s laudable that they figured out enough to run ‘gcc’ manually. When the compiler didn’t complain, that’s reason for optimism.

I decided to modify qt-faststart.c so that it fails to compile via a simple gcc build command while printing out a helpful error message. Then I got to pondering the classic problem of muxing a streamable QT/MP4 file in the first place. Here’s what I’m thinking:

Estimating Header Space
If the duration of an input file is known at the outset, it should be reasonable to estimate how much space the moov header will need. Develop a formula based on the input file’s duration, video output frames per second, and target audio codec characteristics, and decide how much space to set aside. The more frames there will be in the target file, the more header bytes will need to be set aside for entries in the various sample tables. At this phase, calculate the amount of space to set aside for all specified metadata. Add a little space to the computed header size for good measure, create a new file, and jump straight ahead to the position indicated by that size to start writing the mdat atom. After the mdat atom has been laid down, write down the moov atom plus a free space atom to make up any size difference.

Naive Fallback
If the input format does not specify its total duration (perhaps a live source or it might be from any of a number of file formats for which there is simply no efficient way to compute duration without decoding the whole file), then the whole of qt-faststart could be effectively integrated into the QT/MP4 muxer as a post-processing phase.

Is This A Good Idea?
I get the impression that FFmpeg is a major player in the world of video conversion. Further, QT/MP4 is pretty much the ubiquitous standard these days. I worry about changing a fundamental bit of the way the biggest tool creates QT/MP4 files. There must be many toolchains and installations out there which already perform the “mux; qt-faststart” sequence. Will changing this behavior hurt anything? qt-faststart doesn’t do anything to a file that is already streamble; it doesn’t even create a blank file. So modifying FFmpeg to directly create streamable QT/MP4 files will break programs that expect to run ‘FFmpeg && qt-faststart’.

One alternative would be to add streamable remuxing as another command line option. But that somewhat ruins the user-friendliess aspect of creating the desired streamable files per the default mode of operation.

I don’t have any answers right now and certainly no time to code a prototype (nor inclination, unless I’m darn sure the idea would be accepted into the codebase).

See Also:

• Improving qt-faststart

There was some news recently about Broadcom open sourcing code related to a video decoding chip. The brand name here is apparently “CrystalHD” and the chip in question is the BCM-70012. I came into possession of one of these and endeavored to make the open source software surrounding it work in Linux.

The first issue is installing the hardware. It’s a PCI Express mini card which has the same form factor as a PCIe mini wireless networking card. So if your computer can host such a wireless card, it can also hold this thing. Allegedly. First, I tried to place it in the empty PCIe-mini slot in my MSI Wind Nettop. No go. The machine refused to boot up (it would power up but never beep to indicate that it’s really ready to run). Removing the card made the problem go away.

So determined was I to make this chip work that I actually took apart my dear Eee PC 701, ripped out the wireless card and replaced it with the Broadcom card. Deciding it would be too much trouble to attempt to re-attach the keyboard and touchpad ribbons, I realized I could just use USB peripherals.

Resigned to the notion that I just foolishly destroyed my 2 year old Eee PC, I threw the switch anyway and was quite surprised to see it boot up normally. An ‘lspci’ command indicates a new Broadcom multimedia controller hanging off the PCI bus. It’s not pretty but it’s breathing:

So let’s talk software. Broadcom released the driver as open source. To many in the open source community, this is tantamount to, “Okay, done deal! What else needs to be open sourced?” Not so fast. There’s no documentation in the whole package (user-wise, anyway; the libcrystalhd API is thoroughly documented in header comments). So I will describe my experiences with the software.

Download the Linux driver package and unpack it.

Have some kernel source to build against. You don’t actually need a full source tree installed; just the headers will do. To install the appropriate header package on an Ubuntu system, execute:

  apt-get install linux-headers-`uname -r`

When installing a linux-headers package, it’s necessary to specify a kernel version. `uname -r` will determine your Linux kernel version and substitute it into the command line.

If you’re going the route of installing only kernel headers like I did, you will also need to create a symlink from /lib/modules/<version>/build to the new headers tree. This should do the trick:

  ln -s /usr/src/linux-headers-`uname -r` /lib/modules/`uname -r`/build

With that infrastructure in place, change directory into crystalhd/driver/linux. Important step: Run ‘dos2unix *’ in this directory. There are DOS-style line endings running around in here that will foul up the build process. Run ‘autoconf’. Then run ‘./configure’. The whole process looks sort of like this:

$ dos2unix *
$ autoconf
$ ./configure
checking for ld... ld
configure: creating ./config.status
config.status: creating ./Makefile

Time to build: ‘make’. This should look something like:

$ make
make -C /lib/modules/2.6.30.5-ep0/build SUBDIRS=/home/melanson/crystalhd/driver/linux modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.30.5-ep0'
  CC [M]  /home/melanson/crystalhd/driver/linux/crystalhd_lnx.o
  CC [M]  /home/melanson/crystalhd/driver/linux/crystalhd_misc.o
  CC [M]  /home/melanson/crystalhd/driver/linux/crystalhd_cmds.o
  CC [M]  /home/melanson/crystalhd/driver/linux/crystalhd_hw.o
  LD [M]  /home/melanson/crystalhd/driver/linux/crystalhd.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/melanson/crystalhd/driver/linux/crystalhd.mod.o
  LD [M]  /home/melanson/crystalhd/driver/linux/crystalhd.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.30.5-ep0'

‘make install’ (as root, of course) to get the module where it needs to go and then ‘modprobe crystalhd’ to activate the driver. After the modprobe command, the last line of my ‘dmesg’ log looks like:

  Broadcom 70012 Decoder 0000:01:00.0: setting latency timer to 64

We’re not done yet, though. There are 2 provided scripts (bcm_70012_???.sh) which are supposed to load the module and create a device node. The scripts failed to run. Instead, I sorted out the right command for my machine:

mknod -m 666 /dev/crystalhd c 251 0

The major number (251) comes from ‘grep crystalhd /proc/devices’.

Then we move over to the user-land system library (crystalhd/linux_lib/libcrystalhd). A simple ‘make && sudo make install’ should do the trick here.

Then there’s the matter of the example code (crystalhd/examples). Whoever wrote these samples was not in close communication with the people who wrote the rest of the library. It’s reasonable that the code did work at some point. But whoever made this final package reorganized the directory structure and didn’t test this code since there are all kinds of problems when running the simple build script included.

Things went off the rails at this point.

The build script doesn’t build as-is due to incorrect paths. I changed the build script to look like this:

g++ -I ../include/ -I ../linux_lib/libcrystalhd/ -I/usr/include -D__LINUX_USER__ -lcrystalhd -lpthread hellobcm.cpp -o hellobcm

This is to be run from the examples directory. Doing so, however, indicates that hellobcm.cpp has several problems. Upshot: I altered the start of the file to look like this (added 2 headers, subtracted one, added U8 and U32 typedefs which might only work on x86_32):

#include <stdlib .h>
#include <stdio .h>
#include <stdint .h>
#include <string .h>
#include <semaphore .h>
#include "bc_dts_types.h"
#include "bc_dts_defs.h"
#include "libcrystalhd_if.h"
//#include "bc_ldil_if.h"
#include <iostream>
#include <fstream>
#include <sys /shm.h>

typedef unsigned char U8;
typedef unsigned int U32;

#define TRY_CALL_1(func, p1, errmsg) \
[...]

That gets the test program to compile. Running it results in a complaint that firmware can’t be located in a preordained location:

[...]
DtsGetFirmwareFiles:Ctx->FwBinFile is /lib/firmware/bcmFilePlayFw.bin
Failed to Open FW file
[...]

All right, so it wants the firmware files in very specific places (this is actually due to that libcrystalhd user-land library). These firmware files live in crystalhd/firmware/fwbin/70012. Moving the firmware files to /lib/firmware (hey! there are a bunch of firmware files there; I never knew that), makes hellobcm get farther before tossing out a cryptic error. A little investigation reveals that the source is looking for a hardcoded .264 video file (home directory paths implicate a ‘jarod’ and a ‘davilla’, perhaps the same davilla who authored the XBMC blog post).

So I need to feed it a raw .264 video file. All right, Broadcom, let’s see what you’ve got. Chew on the MV1_BRCM_1 H.264 sample (src19td.IBP.264 from here), one of the meanest conformance samples in the ITU suite. Incidentally, it’s from Broadcom (hence, “BRCM”).

Things go… well, confusingly. The program emits a lot of debugging info which largely emanates from the libcrystalhd.so library (that’s also the code which downloads the firmware at initialization). I’m still massaging the code (various other parameters seem to be hardcoded based on the sample) but when it does run successfully, it claims to decode 136/257 frames from the sample. And it does so while occupying the CPU for about a quarter of a second.

For comparison, it takes FFmpeg 16.6 seconds of CPU time to decode this sample using the following command line:

  ffmpeg -benchmark -i src19td.IBP.264 -f yuv4mpegpipe -y /dev/null

It is, of course, inappropriate to make comparisons before I can validate that this example is decoding anything correctly (and it’s a specious comparison anyway). It should be noted that 16.6s amounts to “less than realtime” for a sample that’s 10 seconds long. So that’s the time to beat on my Eee PC 701 equipped with an Intel Celeron M CPU running at 620 MHz… wait, no, ACK! /proc/cpuinfo reports the full 900 MHz! I accidentally overclocked the thing. Or rather, de-underclocked it. Oops. Per my understanding, the only negative implication of that is less battery time.

This same CrystalHD code seems to have been copy and pasted for crystalhd-for-osx. The same example programs live in that tree. I doubt they compile.

Would CrystalHD driver support be useful for FFmpeg? It doesn’t really seem like a good fit. But then, FFmpeg already supports other hardware decoding features such as XvMC and VDPAU.

While creating my automated game archiving solution, I wanted to automatically tag ALAC/M4A files with appropriate metadata while encoding using FFmpeg. Then I realized I didn’t know how to do that. I do remember that FFmpeg recently supplanted a series of specific metadata command line options (like -title and -album) with a highly generalized metadata framework that is accessed by the option ‘-metadata <key>=<value>’. The official documentation provides this lonely (and, I came to realize, useless NO-OP; more on why later) example of the option’s usage:

    `-metadata key=value'

      Set a metadata key/value pair. For example,
      for setting the title in the output file:

      ffmpeg -i in.avi -metadata title="my title" out.flv

So this option allows you to specify absolutely any key/value metadata pair you can imagine. However, a specific muxer won’t necessarily recognize it. How can I know the specific keys that, e.g., the MOV/MP4 muxer honors? 2 methods spring to mind: Trial and error (worked out well for me at first) and reading the code (which was made a good deal easier when I already knew a few of the keys that worked).

I think it would be a good idea to document the specific metadata keys that each muxer in FFmpeg recognizes. This blog post will lead by example. This data comes from SVN revision 20910, current as of 2009-12-21.

QuickTime/MOV/MP4/M4A/et al.

I have constructed the following table of the metadata keys that libavformat/movenc.c honors. The low-level identifier column lists the atom name that the format uses to encode the data on disc, which is not interesting to most readers. For the interested but uninitiated, the notation, e.g., ‘\251nam’ indicates a 4-byte code consisting of the byte A9 in hexadecimal (or 251 in octal) followed by the ASCII characters ‘n’, ‘a’, and ‘m’.

Further, the MOV muxer encodes libavformat version string into the ‘\251too’ field. Here’s an example that ties together all of the presently supported metadata options, even a few video options that don’t make sense for an audio file:

  ffmpeg -i track05.wav \
    -metadata title="Track #5" \
    -metadata author="Unknown Artist" \
    -metadata composer="Composer Unknown" \
    -metadata album="Tracer Video Game Soundtrack" \
    -metadata year="1996" \
    -metadata track="5" \
    -metadata comment="This is redbook CD audio track #5 from the Windows 95 game \"Tracer\"" \
    -metadata genre="Game Soundtrack" \
    -metadata copyright="Copyright 1996 Future Endeavors, Inc." \
    -metadata description="Nifty techno background tune for a futuristic video game" \
    -metadata synopsis="Hey, is this thing on? This is where the 'synopsis' field shows up." \
    -metadata show="Tracer" \
    -metadata episode_id="102" \
    -metadata network="Some network" \
    -acodec alac \
    -y track05.m4a

The lavf version shows up in the Summary tab of the informational dialog:

This is the Info tab of the same dialog:

Here is the video tab:

And this is where the “synopsis” key shows up (selecting “Show Description” from the context menu):

I don’t know where the “copyright” or “network” strings appear in various iTunes dialogs. But the strings are definitely encoded in the file.

ASF/WMV/WMA

FFmpeg’s ASF muxer honors the following metadata keys:

• “title”
• “author”
• “copyright”
• “comment”

AVI

FFmpeg’s AVI muxer honors the following metadata keys and maps them to these FourCCs in the file header:

• “Title” -> “INAM”
• “Artist” or “Author” -> “IART”
• “Copyright” -> “ICOP”
• “Comment” -> “ICMT”
• “Album” -> “IPRD”
• “Genre” -> “IGNR”
• “Track” -> “IPRT”

Further, if the bit-exact flag is not explicitly specified (using ‘-flags bitexact’, the muxer will also encode the libavformat version string into the “ISFT” field.

Matroska

Per my reading of libavformat/matroskaenc.c, the muxer only supports 3 keys. This surprises me– I always thought MKV was supposed to be a container format to end all container formats; maybe FFmpeg’s muxer just doesn’t support many of the tags available; or perhaps the format embeds some other metadata format. Whatever the case, these are the only metadata keys that the muxer supports:

• “title”
• “description”
• “language”

MP3

According to libavformat/mp3.c, the following metadata keys are supported:

• “title”
• “author”
• “album”
• “year”
• “comment”
• “track”
• “genre”

MPEG Transport Streams

The libavformat/mpegtsenc.c file indicates that the muxer honors the metadata keys “title” and “language”.

NUT

FFmpeg’s NUT muxer honors the following metadata keys:

• “title”
• “author”
• “copyright”

Realmedia

FFmpeg’s Realmedia muxer encodes a “CONT” chunk by concatenating certain metadata values specified on the command line. These are the recognized metadata keys:

• “title”
• “author”
• “copyright”
• “comment”

Example:

  ffmpeg -i track05.wav \
    -metadata title="This is the title" \
    -metadata author="Made by Me" \
    -metadata copyright="Copyright 2009 Me"
    -metadata comment="An exercise in Realmedia metadata" \
    -y track05.rm

This is what the start of the file looks like in a hex editor:

0040   00 01 00 03  43 4F 4E 54  00 00 00 5F  00 00 00 11  ....CONT..._....
0050   54 68 69 73  20 69 73 20  74 68 65 20  74 69 74 6C  This is the titl
0060   65 00 0A 4D  61 64 65 20  62 79 20 4D  65 00 11 43  e..Made by Me..C
0070   6F 70 79 72  69 67 68 74  20 32 30 30  39 20 4D 65  opyright 2009 Me
0080   00 21 41 6E  20 65 78 65  72 63 69 73  65 20 69 6E  .!An exercise in
0090   20 52 65 61  6C 6D 65 64  69 61 20 6D  65 74 61 64   Realmedia metad
00A0   61 74 61 4D  44 50 52 00  00 00 9B 00  00 00 00 00  ataMDPR.........

Odds ‘n Ends

The SDP muxer honors the “title” metadata key.

The SoX native format muxer honors “comment”.

Observations and Conclusions

The craziest part I notice is that the example provided in the official documentation showed how to set metadata for a FLV file. However, the FLV muxer does not honor any metadata keys. This could be that FLV metadata is highly free-form and wholly dependent on what the client player SWF cares about. The official FLV spec describes the onMetaData tag which allows a FLV to encode metadata which will be presented to an ActionScript program via Netstream.onMetaData. Really, though, this just means that this should be the easiest format for metadata support; just iterate through each key/value pair and write it to an onMetaData tag.

I was slightly disturbed by the case inconsistency seen between a few of the muxers. A few of them (like AVI) specify “Title” while most specify “title”. However, digging in libavformat/metadata.c reveals that the default behavior is to treat the metadata keys as case-insensitive unless a certain flag is specified, which is never specified.

For completeness, it looks like we should add some other fields as well. For the MP4 muxer: ‘\141ART’ (Album Artist, vs. Artist which is ‘\251ART’), ‘\251grp’ (Grouping), ‘\251lyr’ (Lyrics), ‘tvsn’ (Season Number) and ‘tves’ (Episode Number), and perhaps any other minor options, if the mood strikes. If you see some omissions with metadata types for other formats, you may want to chime in. I also notice that Ogg and FLAC are not hooked up to the metadata API while both formats support such information.