Actual IM conversation:

[me]: ever use git?
[them]: Why would I do such a thing?
[me]: peer pressure, because all of your co-devs
      told you it was the cool thing to do

I thought that developing the software that drives FATE would serve as a good opportunity to learn the Git source control software. Foolish.

Git terrifies me. Thing is, I make mistakes. Lots of mistakes. I need a source control management system (SCM) that is sympathetic to my incompetence. As it stands, when I make a mistake, I have to dig through 140 git-* commands on my system to try to guess which one just might offer a shimmering hope of redemption. If I choose poorly, I will only exacerbate the situation as well as pollute the official history log. Such was the case when I tried to revert one particular commit. I can’t remember how that worked out exactly. I guess I got the correct code back eventually, but the log file tells a sordid tale.

More recently, I edited a file but decided I didn’t want the changes; I wanted the previous committed version back. Perhaps use git-revert, like most other SCMs? Goodness, no. Maybe git-reset? Guess again. Turns out git-checkout is what I was looking for (thanks, Mans). Now, I have made the mistake of using git-commit in such a way that actually committed more files than I thought it would (serves me right for following examples and not reading the pedantic documentation first). Now I find myself wanting to undo the commit for one particular file but not actually lose the changes.

Here’s a solution that can’t fail: ‘rm -rf .git/’, followed by a re-reading of how to initialize a local Subversion repository. And whose idea was it to tag revisions with random 160-bit hex codes like 488dfe6a946bbbbb4e095a5d758ad9808f7336b1? (Yeah, I know, they’re SHA-1 codes or some such. I don’t care; it’s still not human-friendly). I hope FFmpeg never gets around to making the switch.

If you have visited the FATE server, the first thing you will notice is how abysmally slow the main page is. That’s because of the absurd amount of queries required to put together that concise summary page. It’s hard to get people to take the system seriously when the front page takes 30-60 seconds to load. Look, what can I tell you? I’m a multimedia hacker, not a DBA.

But it all changed tonight. That’s right– it’s all about caching! Give the FATE Server another look. Give it many looks, in fact, because the main page will load immediately. Be advised that the cache is only updated every 15 minutes, so the trade-off is not quite as much instant gratification.

I think it’s time to face up to the fact that this whole parallelization fad is probably not going to go away. There was a recent thread of ffmpeg-devel regarding the possibility of ‘porting’ FFmpeg to something called the Nvidia Tesla. This discussion rekindled a dormant interest I have regarding what optimization possibilities could be in store for the Cell processor on board the Sony PlayStation 3, and whether there should be effort directed toward making FFmpeg capable of using such features.

I finally took some time to read through many of the basic and advanced tutorials on offer and finally have a feel for what the system is set up to do. Unfortunately, it’s not always clear what these parallel architectures are capable of, a situation only exacerbated by vague, impenetrable marketing materials. Too many people confuse the Cell architecture with a homogenous multiprocessor environment, as is common today, which is simply not the case. In order to take advantage of the machine’s full power, an app has to be written with a special awareness of the fact that the Cell has a primary core (PPE) and 6 little helper coprocessors (SPEs), as is half-heartedly illustrated above. The PPE is a dual-threaded general-purpose CPU (64-bit PowerPC) and can do anything. Meanwhile, each SPE is essentially another 64-bit PPC that has its own pool of 256 kilobytes of memory (LS) and a special memory controller (MFC) that coordinates contact with the outside world. To take advantage of the SPEs, the PPE has to load programs into their memory space and tell them to execute the code. The Cell also features DMA facilities to efficiently shuttle data between main memory and the SPEs’ local memory, and there are mailbox facilities and interrupts to facilitate communication between the PPE and the SPEs.

I don’t know about a general parallelized architecture for FFmpeg that would take advantage of multiple architectures like Cell and Tesla (because I still can’t figure out how Tesla is supposed to work). However, in a media playback application, it might be possible to assign one SPE the task of decoding perceptual audio. Another SPE might be performing inverse transform operations for a video codec, while another SPE does postprocessing and yet another handles YUV -> RGB conversion. On the opposite end, it seems reasonable that SPEs could be put to work at tasks like motion estimation for video encoding.

Would this qualify as a Google Summer of Code project for FFmpeg? There is precedent for this– see “Development assistant for the ‘Ghost’ audio codec” which was essentially a lab rat for Monty’s (of Vorbis fame) newer audio coding ideas. Fortunately, a prospective student would not require a PS3 for this project; just a Linux machine. For it seems that IBM has a freely downloadable tool called the Cell Simulation Environment. I’m still working on getting the program running (it’s distributed as an RPM and is most happy on a Red Hat system).

I am a little surprised that there is not a PS3 Media Center project, in the spirit of the Xbox Media Center, at least not that I have been able to locate via web searches. I have been pondering the technical plausibility of such an endeavor. It almost seems as though the PS3 gives the guest OS just enough of a confined playground environment that it can’t possibly blossom into a reasonably high-end enviroment. While real-time video playback must be possible, is it possible to run at, say, full 1080p resolution at 30 fps? With all of the processing power, I trust that the Cell can handle any kind of video decoding, though I heard an unsubstantiated rumor once that it takes the PPE and 4 SPEs to decode HD H.264 video from a Blu-Ray disc. The PS3′s native HD player would have a slight advantage since it would presumably use the video hardware’s full feature set, which likely allows the PS3 to pass through raw 12-bit YUV data to be handled by the video hardware, in one way or another. In Linux under the hypervisor, you basically get to play with a big RGB frame buffer. That means that not only to you have to convert YUV -> RGB, but you also have to shuffle 2.5x as much raw video data to the video memory for each frame. That works out to upwards of 250 MB of data shuffling each second ((1920 * 1080 pixels/frame) * (4 bytes/pixel) * (30 frames/second)). I have read conflicting sources about whether it’s possible for Linux under the PS3 hypervisor to DMA data from main RAM to system RAM. Some sources contend that there is work ongoing while other sources claim that this feature was fixed in later firmware revisions (i.e., no longer possible).

One possible dealbreaker in the proposal to use the PS3′s guest OS mode to install Linux and a general purpose media player is that, from everything I have read, the hypervisor only allows the guest OS to output stereo audio. This might be a long shot, but perhaps it would be possible to transcode super-stereo (more than 2 channels) audio to Dolby Pro Logic II to be sent out to a capable decoder module. Hey, it’s sort of like true surround sound.

If you are interested in the hard technical details of running Linux on a PlayStation 3 and programming its Cell Processor, this directory at kernel.org seems to be fairly authoritative on the matter. The latest iteration of the tech documents (dated 2008-02-01) are here.

If you keep up with FATE as obsessively as I do, you may have noticed that I got Intel’s C compiler (icc) into the build farm. It was a struggle, but I finally made it happen. The compiler is distributed as an RPM. but the x86_32 build machine is Ubuntu. I googled and found a number of blog posts describing how to install it on Ubuntu. I went the route of using the alien program to convert the RPM to a DEB, installing it, manually modifying the ‘icc’ shell script to point to the correct INSTALLDIR and updating the ld configuration to point to the right libraries. Finally, I installed the free-for-non-commerial-use license file in one of the many acceptable locations and I was off.

When I first started fighting with icc about a month ago, the compiler was on version 10.1.008. It is now on 10.1.012, indicating that I may need to update it almost as frequently as the the SVN version of gcc used on the farm.

I also tried to get the x86_64 version of the compiler running on the appropriate build machine. When I try to run ‘icc’, I get one of the most annoying and confusing UNIX errors known to exist:

  -bash: /opt/intel/cce/10.1.012/bin/iccbin: No such file or directory

Even though I can see that file quite clearly exists.

But at least icc is running for x86_32, and FFmpeg is compiling fine and running the same series of tests as all the gcc versions. Personally, I have never put a lot of stock in the optimizing prowess of proprietary compilers. I have seen a few too many that need to have their optimizers disabled because they are so obviously buggy. However, icc demonstrates some clear speedups over gcc based on FATE testing. If you open a build record page for an icc run in one window or tab, and then open a build record page for a gcc run in another, you can see that the icc-built binary generally runs faster. This is particularly notable on longer tests.

This exercise also reminds me that the SVN versions of gcc build slow binaries, at least on x86_32. I am wondering if this has to do with the way I am building the compiler, or if 4.3 will actually build substantially slower binaries?

And yes, I plan one day to deploy an easier way of comparing build performance over time, and for various platforms.

Today’s post over on Coding Horror is called There Ain’t No Such Thing as the Fastest Code. The post discusses the idea that no matter how much you manage to hyper-optimize your code, even down to the assembly level, it could always be faster. Strange that this topic should arise since there is currently a discussion over on ffmpeg-devel regarding a faster C-based fast Fourier transform (FFT).

I guess I had just been taking it for granted that the matter of a fast C-based FFT was a closed issue. How wrong I was. It seems that there is an algorithm answering to the name of djbfft that offers some marked speedups over FFmpeg’s current C-based FFT.

The ensuing discussion for the Coding Horror article is a typical debate on optimization trade-offs (e.g., time investment vs. resulting speed-up). However, while the common argument is that computing hardware is so ridiculously cheap, powerful, and abundant that there is no reason to waste precious time on optimization, there is also the ironic trend back to less capable machines. Like my Asus Eee PC. Trust me, I am suddenly keenly aware of modern software bloat.

My favorite comment comes from no-fun:

…the more people which claim optimisation is worthless, well, that just means that I can charge more since my particular expertise is almost impossible to find. And I’ll take the big $$$, thank you very much. Yeah, I agree with y’all, optimisation is crap. Dont bother learning it.

We multimedia hackers tend to be quite secure in our reasoning for optimization. After all, I challenge anyone to decode 1080p H.264 video in realtime using pure Java or C# code (no platform-specific ASM allowed).

Tonight, I finally implemented and deployed a build history browsing option for the FATE Server. Yeah, FATE just got a little uglier and crowded tonight in the process. But if you visit the main page and see that one of the build boxes is red instead of its recommended green, you can browse the corresponding history to learn approximately which SVN revision broke the build. I say “approximately” because the commits can get stacked, e.g., a build/test cycle is occurring and 3 new changes are committed before the machine has a chance to check out code again and build. In that case, the history browser will at least provide links to each of the 3 commits in the online SVN browser which allows a developer to inspect each change message that went into a build.

I hope to implement a similar history browser for test results so that a developer can analyze which SVN commit broke a particular test.

With these history features, I think I am finally finishing with the most basic useful features. Too bad the website looks so bad with its retro-1995 feel. Unfortunately, I have roughly zero experience in making websites pretty. What I care more about is that the website is functional for the nominal FFmpeg developer. I’m disappointed with how crowded the main page is becoming. However, I have had plans from the beginning about how to distill that data down to a few essential pieces of information. Plus, I have ideas about how to make the front page orders of magnitude faster to load (hello, caching strategy). Though I might be the only person who really cares as I obsessively check the FATE status throughout the day.

When the FATE initiative went live, I asked the guru how to handle the H.264 conformance suite samples– should I just dump all of them into the database whether they were working or not, or should I only enter the samples that worked with the current version of FFmpeg? His answer was far more complicated than I could have anticipated:

1. Enter all currently working samples
2. If a particular H.264 conformance vector used to work with FFmpeg, add the sample and enter a new issue in the tracker
3. Otherwise, don’t add the test yet

Whoa. As you know, I got task #1 accomplished relatively easily. Now I’m back to take on task #2.

Hypothesis: most of the code that can make or break the H.264 decoding process lives in files named libavcodec/h264*. Thus, test the sample suite against every single FFmpeg revision in which one of those files was touched.

  for file in libavcodec/h264*; do svn log $file; done |
  grep "^r.*lines$" |
  sed -e 's/^r\([0-9]*\).*$/\1/' |
  sort -n |
  uniq

That produces just over 400 different FFmpeg revisions that need testing. I had better get started early.

Algorithm outline:

• create a script that takes the above revision list and the directory full of H.264 conformance vectors
• create a list of standard test names based on the convention already in the database
• query the database to obtain a complete list of all tests known to work currently
• remove the working tests from the list of all tests
• for each revision:
• • get the FFmpeg code corresponding to that revision
  • build FFmpeg, and use ccache to hopefully gain a little speedup in the process
  • test FFmpeg against all of the non-working samples, output results in a CSV format: “revision, 0, 0, 0, 1, 0, 0, 0, 0, 0,…”; this should facilitate analysis and serve to illustrate that the non-working samples have been broken from the get-go

Hey, computing cycles are cheap, right? Perhaps the same ambitious strategy can be employed as a one-time brute force method to learn when other FFmpeg components broke so that they can be fixed and subsequently tested via FATE. And there’s no reason I have to do this on my own; I know certain FFmpeg developers who like to brag about their cumulative 27 or so underworked CPU cores laying around their flats (you devs know who you are).

When I upgraded my web hosting plan last summer from 800 MB of online storage to 1/2 TB, I wondered what I could possibly use all that extra space for. The FATE Server is stepping up to the task and presently — somehow — occupies 1/2 GB of space. This is not a problem in and of itself since it’s only 1/1000 of my total allotment. However, it makes a regular, responsible backup schedule difficult to keep. I have toyed with the idea of hosting the database operation on my own hardware and bandwidth. I’m pretty sure I’m the primary user of this database anyway. Having the database under local authority would also likely allow for greater flexibility and configurability for the underlying engine.

As always, I have plans to add many, many, many more tests. There are various public MPEG conformance suites for different codecs, each consisting of tens or hundreds of samples. There is FFmpeg’s internal regression suite which ought to be run and verified for each build. By my accounting, ‘./ffmpeg_g’ is invoked over 300 times when running ‘make test’; I suspect each of those invocations would be a separate test in the database. Whenever I think of getting down to it and starting to enter individual test specs into the database using my custom PHP tool, I always step back, glance at the magnitude of the task, and instead start outlining a script that will automatically process the test series for me, and with fewer mistakes.

However, there are yet more problems. There are only 76 tests currently. Logging the 76 individual test result records nominally takes 10-15 seconds. Yes, I use one MySQL connection, but with 76 separate INSERT queries. It would probably be more efficient to concatenate them into one INSERT query with 76 records. However, it would probably be even better to parameterize the data, compress it, and POST it via HTTP to a custom CGI script on the server that could uncompress it and perform the INSERT locally and more efficiently, ideally. This would solve firewall problems and library problems as outlined in a previous post and allow for more diverse platform expansion in the future.

Finally, I also need to be able to test tests before deploying them. That’s right — test tests. I.e., enter a new test, or a series of new tests, into a staging area and be able to run a special script to verify that I got all the basic details right such as sample paths and FFmpeg command line parameters. None of this nonsense about dumping in a new test spec and waiting until the next SVN commit to see if I got it all correct. Or worse yet, artifically starting a new build/test cycle with a document update SVN commit. Out of all the problems examined in this post, this should be the easiest to take care of.

Thanks for putting up with yet another edition of Multimedia Mike’s research notepad.

Someone asked me a question in email today that I thought I would pass on to the broader group. Are there any general methods for attaching general metadata — e.g., EXIF data — to individual video frames?

At first thought, this strikes me as a container-level matter. However, there could be a video codec that embedded metadata in each frame and could be stored in any generalized container format.

Personally, I have always been more aware of issues surrounding content playback vs. content creation which I admit is somewhat of a weakness in my overall multimedia knowledge. The submitter had in mind data along the lines of absolute timestamps and GPS coordinate information to be included with every frame. This may seem a tad excessive but you can never underestimate other peoples’ requirements.