Breaking Eggs And Making Omelettes

Topics On Multimedia Technology and Reverse Engineering


Archives:

Deadline Extension; GSoC Qualification Showcase #2

March 31st, 2008 by Multimedia Mike

Today was supposed to be the the last day that students could apply to participate in Google’s 2008 Summer of Code. That deadline, however, along with all of the program’s deadlines, have been pushed back by a week. The new submission deadline is next Monday, April 7. The new deadline for the FFmpeg mentoring crew to pick students in April 18, so be sure to have your qualification code in SVN well before then in order to be considered for a project slot.

Regarding qualification tasks, 2 more students have qualified. First, the more graphically interesting: Eli Friedman completed an RPL/ARMovie demuxer and Escape-124 video decoder:


ffplay playing RPL/Escape-124 data

This is the container/codec combo that was seen in the first Tomb Raider PC game. ARMovie is a format that was apparently used on Acorn RISC PCs. If anyone happens to have any ARMovie files that don’t come from Eidos games, I would be interested it seeing them.

Next is a demuxer/decoder pair for the Amiga IFF format that supports the PCM and Fibonacci encodings, prepared by Jai Menon. No real graphics of course, so here’s ffplay’s graphical interpretation of the PCM amplitude:


ffplay playing an IFF/8svx file

There is still time to both apply for GSoC 2008 and work on a qualification task. I have been scavenging the MultimediaWiki for game-related formats that have long been documented but not re-implemented in FFmpeg. Hopefully, the supply will hold up for all the students who want to try their hand at FFmpeg.

Posted in Open Source Multimedia | 7 Comments »

Good FATE Housekeeping

March 26th, 2008 by Multimedia Mike

The front page of FATE may not look all that different, save for some new tests every few days. But I assure you I am doing a lot of work on the backend to improve the whole thing. I finally deployed the optimizations outlined in this post. That put the brakes on the unrestrained growth of the database (2.5 GB in as many months). I also thought of another –ahem– groundbreaking optimization: If the client-side test script compresses the stdout or stderr for a build record and the compressed size is larger than 64K, there’s really no reason to send it over and store it in the database– the database won’t be able to decompress the data block anyway after it is truncated to 64K. This is the case for every single icc build. Intel’s C compiler insists on operating in a pedantic mode that I can’t seem to disable, despite the options documentation. Fortunately, I have been logging total stderr line count for for several weeks now (though I haven’t made it available via the web interface yet).

So what to do about this enormous database? At first, I suspected that all those specious icc stderr blobs had a significant impact. But no: that’s only occupying 13 MB. Still, that useless data is now gone.

The real space pigs are the cumulative stderr/stdout blobs logged prior to my space-saving epiphany:

  1. mysql> SELECT
  2.   SUM(LENGTH(stdout)),
  3.   SUM(LENGTH(stderr))
  4.   FROM test_result;
  5. +---------------------+---------------------+
  6. | SUM(LENGTH(stdout)) | SUM(LENGTH(stderr)) |
  7. +---------------------+---------------------+
  8. |          1337031227 |          1096300800 |
  9. +---------------------+---------------------+

Whoa. So that’s where all the space went. That’s right: ~1.3 GB and ~1.1 GB, respectively. Getting rid of the stderr blobs is pretty straightforward. I don’t care about stderr if the test was successful:

  1. mysql> UPDATE test_result
  2.   SET stderr=NULL
  3.   WHERE test_passed=1;

The stdout blobs are a bit trickier. If the test failed, the stdout is always a keeper, just like stderr. If the test succeeded, I still want the stdout data if the corresponding test spec listed the expected stdout as being NULL– that means the stdout is supposed to be retained for logging purpose. This is presently used for tracking the assorted binaries’ filesizes over time and I don’t want to drop that data. So it seems I need data from another table in order to make this query work. Sounds like a job for an JOIN operation, which didn’t seem to be possible per the raw syntax spec. I’m glad I turned out to be wrong as indicated by this blog post. So I cloned a small test table and went to work prototyping it backwards and forwards, lest I lose 3 months of precious filesize data.

  1. mysql> UPDATE test_result result
  2.   INNER JOIN test_spec spec
  3.   ON result.test_spec=spec.id
  4.   SET result.stdout=NULL
  5.   WHERE result.test_passed=1
  6.   AND spec.expected_stdout IS NOT NULL;

Things are a bit more manageable now and I’m happy to report that I have a fighting chance of implementing a regular, automated backup schedule for FATE:

  1. +---------------------+---------------------+
  2. | SUM(LENGTH(stdout)) | SUM(LENGTH(stderr)) |
  3. +---------------------+---------------------+
  4. |             2046425 |             4292739 |
  5. +---------------------+---------------------+

I hope you have enjoyed yet another installment of The New Adventures Of The Amazingly Inept DBA.

Posted in FATE Server | Comments Off on Good FATE Housekeeping

2008 GSoC Qualification Showcase #1

March 22nd, 2008 by Multimedia Mike

The application process for Google’s Summer of Code 2008 season is not even open yet, but people interested in participating with FFmpeg are already busy on their qualification tasks.

Ramiro Polla submitted a system that can demux captured MSN video streams and play them back using a video codec known as Mimic. Here is the system in action:


I [heart] FFmpeg

Sascha Sommer has completed a playback system for the RL2 file format, which figured heavily into a CD-ROM game called Voyeur:


Voyeur FMV title

There are still some qualification tasks left unclaimed on the FFmpeg GSoC page. And if we run out, we can make more, so don’t despair.

Posted in Open Source Multimedia | 3 Comments »

GSoC Multimedia Competition

March 21st, 2008 by Multimedia Mike

The 2008 Google Summer of Code participating organizations have been selected. Like last year, I wanted to survey what other multimedia-type projects are doing. Fortunately, instead of clicking on each individual project in the official Google listing to figure out which ones might be vaguely multimedia-related, the GenMAPP project (also a participating organization) has organized the various projects by category.

Multimedia category projects include BBC Research, FFmpeg, GStreamer, Neuros Technology, XBMC, and Xiph.org. Tangential to multimedia are the 2 TV & Video category projects, Dirac Schroedinger (separate from BBC) and VideoLan.

Check out XBMC’s SoC project page. If you have been active with FFmpeg’s own SoC page, it should seem charmingly familiar. Eh, it’s all GFDL, I guess.

Then there is the Audio & Music category: Atheme, Audacity, CLAM, Mixxx, and XMMS2. I hadn’t heard of Atheme before. I can’t quite nail down what it is they do, but they seem to have a number of ambitious software projects under their umbrella. And one of their proposed SoC endeavors is “Support for RealAudio: Implement an input plugin for the RealAudio codec, preferably with support for streaming as well as files.” Seems a bit understated.

These are some other projects that caught my eye at a cursory glance:

Speaking of Neuros Technology, this is the first time I have heard of them. They produce an open platform as a digital media center. GSoC participants will receive one of the items. Tantalizing. No such perks for working on FFmpeg. But I would like to remind prospective GSoC participants that FFmpeg offers valuable real world experience in the form of working long, thankless hours for a set of abusive, anti-social, impossible-to-please bosses on a rarely acknowledged piece of backend software. This is training you don’t get in school.

The application process begins bright and early on Monday morning (March 24). And don’t forget your qualification task.

Posted in Open Source Multimedia | 3 Comments »

« Previous Entries