Breaking Eggs And Making Omelettes

Topics On Multimedia Technology and Reverse Engineering


Archives:

Google’s YouTube Uses FFmpeg

February 8th, 2011 by Multimedia Mike

Controversy arose last week when Google accused Microsoft of stealing search engine results for their Bing search engine. It was a pretty novel sting operation and Google did a good job of visually illustrating their side of the story on their official blog.

This reminds me of the fact that Google’s YouTube video hosting site uses FFmpeg for converting videos. Not that this is in the same league as the search engine shenanigans (it’s perfectly legit to use FFmpeg in this capacity, but to my knowledge, Google/YouTube has never confirmed FFmpeg usage), but I thought I would revisit this item and illustrate it with screenshots. This is not new information– I first empirically tested this fact 4 years ago. However, a lot of people wonder how exactly I can identify FFmpeg on the backend when I claim that I’ve written code that helps power YouTube.

Short Answer
How do I know YouTube uses FFmpeg to convert multimedia? Because:

  1. FFmpeg can decode a number of impossibly obscure multimedia formats using code I wrote
  2. YouTube can transcode many of the same formats
  3. I screwed up when I wrote the code to support some of these weird formats
  4. My mistakes are still present when YouTube transcodes certain fringe formats

Longer Answer (With Pictures!)
Let’s take a video format named RoQ, developed by noted game designer Graeme Devine. Originated for use in the FMV-heavy game The 11th Hour, the format eventually found its way into the Quake 3 engine as well as many games derived from the same technology.

Dr. Tim Ferguson reverse engineered the format (though it would later be open sourced along with the rest of the Q3 engine). I wrote a RoQ playback system for FFmpeg, and I messed up in doing so. I believe my coding error helps demonstrate the case I’m trying to make here.

Observe what happened when I pushed the jk02.roq sample through YouTube in my original experiment 4 years ago:



Do you see how the canyon walls bleed into the sky? That’s not supposed to happen. FFmpeg doesn’t do that anymore but I was able to go back into the source code history to find when it did do that:



Academic Answer
FFmpeg fixed this bug in June of 2007 (thanks to Eric Lasota). The problem had to do with premature colorspace conversion in my original decoder.

Leftovers
I tried uploading the video again to see if the problem persists in YouTube’s transcoder. First bit of trivia: YouTube detects when you have uploaded the same video twice and rejects the subsequent attempts. So I created a double concatenation of the video and uploaded it. The problem is gone, illustrating that the backend is actually using a newer version of FFmpeg. This surprises me for somewhat esoteric reasons.

Here’s another interesting bit of trivia for those who don’t do a lot of YouTube uploading– YouTube reports format details when you upload a video:



So, yep, RoQ format. And you can wager that this will prompt me to go back through the litany of unusual formats that FFmpeg supports to see how YouTube responds.

Posted in General | 19 Comments »

19 Responses

  1. Anonymous Says:

    Heh, but not a few hours ago I was wracking my brain over what decoder Google was using that could possibly have been interpreting my upload’s framerate as double its actual value, when Jason confirmed it as a bug in FFMPEG’s tbc for H.264. Sure enough, MPEG-4 ASP processed fine, and so did H.264 with certain features disabled (presets below –slow, strangely).
    A rather timely post, in my opinion!

  2. Anonymous Says:

    they do not use the same version of ffmpeg for all videos, also not sure they only use ffmpeg. based on the format/container detected they use different binaries, i.e. there theora decoder is still broken and uses a version of fftheora that is more than 2 years old.
    if you upload theora in mkv it uses a newer theora decoder.

  3. astrange Says:

    The H.264 bugs vary based on whether you use Main or High profile, I believe.

    (I can’t think of a single reason this makes sense.)

  4. Peter Says:

    I wonder how often youTube updates their libavcodec/libavformat dependencies.

  5. Multimedia Mike Says:

    @Peter: As other commenters/experimenters have indicated, their backend architecture might be more erratic than we expected. It also must be noted that YouTube likely uses binary solutions in order to decode various proprietary codecs. In fact, my initial conclusion 4 years ago was that they actually based their transcoding backend of MEncoder, which of course implies a heavy dose of FFmpeg.

  6. clone2727 Says:

    I had a little fun with this myself now. The 11th Hour (and other Groovie games) use a JPEG block (1012) as a keyframe and the FFmpeg (and id’s too, even) decoder don’t yet handle. Two other chunks are in old Groovie ones that ScummVM supports: http://scummvm.svn.sourceforge.net/viewvc/scummvm/scummvm/trunk/engines/groovie/roq.cpp?revision=55850&view=markup

    So, I ripped out a test video from the demo and tried to play it in FFplay. No dice. I then tried to upload it to YouTube. YouTube’s decoder recognized it as ROQ alright… but then has bombed since – http://img217.imageshack.us/img217/6299/screenshot20110209at105.png

    It will be at http://www.youtube.com/watch?v=_J2wHJgi5AE if YouTube ever finishes processing the video. I’ll check back in a couple months.

  7. Coderjoe Says:

    A few months back, I played with uploading lossless formats to youtube. The two I tried were HuffYUV and FFV1, both in AVI, with PCM audio. Both worked quite well, aside from the sheer absurdity of the file sizes of the 7 second test clips.

    I don’t remember if I tried H.264 lossless or not. If I did, it was rejected.

  8. RC Says:

    On2 had a Flash video encoder called Flix.
    Flix has a Linux version.
    Flix uses MPlayer (not MEncoder) to decode videos, outputs to yuv4mpeg and pcm streams, which arr piped to their actual Flash video format encoder.
    You can get the On2 Flix patches for Mplayer from their website (or at least I could a year ago).
    Google bought On2.

    I think it’s safe to assume Youtube uses “On2 Flix for Linux” for the video conversion, perhaps a custom modified version of it…

  9. clone2727 Says:

    @RC: Except that this converting “phenomenon” was noticed before Google bought On2?

  10. RC Says:

    @clone2727: I didn’t suggest Google bought a random company for no reason, then after the fact decided to start using their products. When Google bought On2, and everyone was scratching their heads as to why they might have done that, I was suggesting this same answer.

  11. compn Says:

    i’d love to work on youtube’s failed samples. or even just get the fourccs in youtube’s failed samples.

    that picsearch list was fun.

    btw mike have you tried other video hosting sites like megavideo, bliptv, brightcove, dailymotion, vimeo? i’m sure they are all using ffmpeg somewhere in the process.

    i know facebook uses ffmpeg:
    http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2009-June/071146.html
    “I’m currently working for Facebook, which is at this point probably the world’s largest user of ffmpeg in terms of videos processed (a few million a week).”

    your code is used all over the internet! you’ve hit it big now!

  12. Multimedia Mike Says:

    @compn: I don’t have a lot of ready contacts at all the video sites, at least none that I know of.

  13. compn Says:

    i meant have you seen if they use more updated ffmpeg versions than youtube? :P

  14. Multimedia Mike Says:

    @compn: I know it sounds like fun, but no, I haven’t conducted similar tests on other video upload sites.

  15. YouTube aprovecha la potencia de FFmpeg Says:

    [...] parece demostrarlo un interesante artículo rescatado del pasado mes de febrero en el que su autor explicaba cómo logró deducir que Google [...]

  16. Tom Limoncelli Says:

    The comparison to the Google/Bing controversy implies that Google has done something wrong. I don’t think that’s what you meant to do. It must be quite a feather in your cap to know that your code is used so ubiquitously.

    Would you be surprised to learn that Google uses cpython instead of re-writing their own Python interpreter?

  17. Tsz Ming WONG Says:

    Actually, few years ago, I have mailed Google about what tricks they have used in FFmpeg in encoding some version of WMV format (since the trunk version of FFmpeg at that time had failed to encode those files correctly, while YouTube is okay), they refused to provide me the information I needed and finally I solved the problem using MEncoder.

  18. Multimedia Mike Says:

    Hmm? Why is the post seeing interest suddenly? Oh, got posted on Hacker News.

    @Tom: This is true, using FFmpeg for this backend application is not improper. It was hard to word this article not to imply that. The only part that reminded me was the interesting methodology Google used.

    And you’re right about the feather/cap thing– people who know me know that I never shut up about the fact that Google uses some of my code. :-)

    @Tsz Ming WONG: In my original experiments on the matter, I concluded empirically that Google was probably using MEncoder, which makes heavy use of FFmpeg (as nearly all modern multimedia applications do).

  19. Peter Theta Says:

    The best part of sharing the Bing anecdote and this story is that it reiterates how there’s no substitute for actually performing the craftwork of one’s trade. Good read!

Leave a Comment

Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.