Yearly Archives: 2009

Multithreaded FFmpeg Programming

As briefly mentioned in my last Theora post, I think FFmpeg’s Theora decoder can exploit multiple CPUs in a few ways: 1) Perform all of the DC prediction reversals in a separate thread while the main thread is busy decoding the AC coefficients (meanwhile, I have committed an optimization where the reversal occurs immediately after DC decoding in order to exploit CPU cache); 2) create n separate threads and assign each (num_slices / n) slices to decode (where a slice is a row of the image that is 16 pixels high).

So there’s the plan. Now, how to take advantage of FFmpeg’s threading API (which supports POSIX threads, Win32 threads, BeOS threads, and even OS/2 threads)? Would it surprise you to learn that this aspect is not extensively documented? Time to reverse engineer the API.

I also did some Googling regarding multithreaded FFmpeg. I mostly found forum posts complaining that FFmpeg isn’t effectively leveraging however many n cores someone’s turbo-charged machine happens to present to the OS, as demonstrated by their CPU monitoring tool. Since I suspect this post will rise in Google’s top search hits on the topic, allow me to apologize to searchers in advance by explaining that multimedia processing, while certainly CPU-intensive, does not necessarily lend itself to multithreading/multiprocessing. There are a few bits here and there in the encode or decode processes that can be parallelized but the entire operation overall tends to be rather serial.

So this is the goal:

Mac OS X Activity Monitor showing FFmpeg using more than 100% CPU

…to see FFmpeg break through the 99.9% barrier in the CPU monitor. As an aside, it briefly struck me as ironic that people want FFmpeg to use as much of as many available CPUs as possible but scorn the project from my day job for being quite capable of doing the same.

Moving right along, let’s see what can be done about exploiting what limited multithreading opportunities that Theora affords.

First off: it’s necessary to explicitly enable threading at configure-time (e.g., “–enable-pthreads” for POSIX threads on Unix flavors). Not sure why this is, but there it is.

Continue reading →

Happy 20,000; New YouTube Engine

FFmpeg crossed the 20,000 commit threshold today. Mans captured the distinction when he submitted an ARM NEON optimization for int32_to_float_fmul_scalar(). Does that warrant a prize? Diego presented the statistics:

It took 7 years to get to r10000, but only two more to get to r20000.
FFmpeg is approaching warp 6 :-)

Today was also the day I noticed that YouTube upgraded their backend conversion system somewhere along the line. Nearly 3 years ago, I started poking at YouTube to see what kind of multimedia files it can convert and cataloged my findings at the MultimediaWiki.

Today, I was clicking around on some of my old videos and noticed that this video which came from an Ogg Theora source now looks correct. Actually, according to the comments (and I receive enough between all my videos that I rarely pay attention to any of them), this was working over a year ago.

It’s interesting to note that this means that YouTube/Google keeps all of the source material that users upload. When it was time to recode, they obviously had to go back to the original material.

I found that CSCD, KMVC, 3iv2, ZMBV, and VP6 video codecs all work; Vivo files, Westwood v2 VQAs, Real files with RV40, and the bastardized FLIC files from The Magic Carpet are all fine as well; Wing Commander III MVE files, id CIN files, and Interplay MVE files all transcode with audio but with either missing or glitched video.

Sorry if I seem a bit sentimental about this but it all still amazes me. When I was writing the bulk of the subsystems for all manner of bizarre formats circa 2001-2003, I never could have imagined that there would be a website that would take the weird video formats as input and convert them to a standard video format for anyone to view.

Optimizing Away Arrows

Google released the third version of their year-old Chrome browser this past week. This reminded me that they incorporate FFmpeg into the software (and thanks to the devs for making various fixes available to us). Chrome uses FFmpeg for decoding HTML5/video tag-type video and accompanying audio. This always makes me wonder, why would they use FFmpeg’s Theora decoder? It sucks. I should know; I wrote it.

Last year, Reimar discovered that the VP3/Theora decoder spent the vast majority of its time decoding the coefficient stream. He proposed a fix that made it faster. I got a chance to check out the decoder tonight and profile it with OProfile and FFmpeg’s own internal timer facilities. It turns out that the function named unpack_vlcs() is still responsible for 44-50% of the decoding time, depending on machine and sample file. This is mildly disconcerting considering the significant amount of effort I put forth to even make it that fast (it took a lot of VLC magic).

So a function in a multimedia program is slow? Well, throw assembly language and SIMD instructions at the problem! Right? It’s not that simple with entropy decoders.

Reimar had a good idea in his patch and I took it to its logical conclusion: Optimize away the arrows, i.e., structure dereferences. The function insists on repeatedly grabbing items out of arrays from a context structure. Thus, create local pointers to the same array and save a bunch of dereferences through each of the innumerable iterations.

Results were positive– both OProfile and the TSC-based internal counter showed notable improvements.

Ideas for further improvements: Multithreading is all the rage for video decoders these days. Unfortunately, entropy decoding is always a serial proposition. However, VP3/Theora is in a unique position to take advantage of another multithreading opportunity: It could call reverse_dc_prediction() in a separate thread after all the DC coefficients are decoded. Finally, an upside to the algorithm’s unorthodox bitstream format! According to my OProfile reports, reverse_dc_prediction() consistently takes around 6-7% of the decode time. So it would probably be of benefit to remove that from the primary thread which would be busy with the AC coefficients.

Taking advantage of multiple threads would likely help with the render_slice() function. One thing at a time, though. Wish me luck with presenting the de-dereferencing patch to the list.

Python Bit Classes

Here’s a little project of absolutely no use to anyone (a specialty of mine, as if you didn’t know): Pure Python classes for writing and reading bitstreams. This was just one of those things where I was sitting around wondering what it would take to accomplish, and a cursory Google search didn’t reveal anything useful (though it’s probably out there, in all likelihood), so I sat down and pounded out the code.

To what end? Oh, I don’t know– reimplement FFmpeg in Python; go crazy. Behold brute force bit banging in Python: