Monthly Archives: September 2008

FATE Will Return

The FATE server started getting frustrating and dispiriting to maintain, so I decided to scrap it a little while ago. But I have since started to heavily revise the infrastructure so it can come back online. I have been sitting on a pile of brainstorms about how to make the system work better. Once I finally got down to implementing the changes, it sort of snowballed and I thought of even more improved ways that the various pieces could work together.

But this growth is not without its associated pains. I largely blame PHP for this. Whenever I have a bad day at work, I just remind myself that things could be a lot worse. For example, it could be my job to write PHP code full time. I have lots of gripes with the language, but a few new ones due to this experience.

There must be 7 different ways to interface to one library I want to use, and I can’t get any of them to work. And since it’s all server-side, it’s incredibly difficult to diagnose why the server is having trouble.

PHP is hyper-paranoid about security. When you GET or POST data, PHP’s site-specific (that I can’t change) setting is to escape quotes and backspaces before it makes the data available to you, whether you like it or not. And I don’t like it. I really don’t want the data escaped, but I can’t turn it off. The manual states that the next version of PHP will remove this annoyance.

But there’s no point in complaining about PHP. As Jeff Atwood eloquently expressed, PHP Sucks, But It Doesn’t Matter. It’s still serves as the backbone of some of the most important sites on the internet. And I know I will eventually coerce PHP to be the backbone of FATE once more.

Python isn’t blameless in this either. I need a key feature that, for once, is not provided by the expansive Python standard library (even though the library handles everything else associated with this type of functionality). A few hackers around the net have attempted to fill in the missing piece but I haven’t successfully adapted their code yet.

On the plus side, I should mention that I have gotten FATE running on Mac OS X. It’s currently watching FFmpeg SVN and performing build/test cycles while saving the data locally. That was the easy part. Getting the data to the server is the troublesome part and the foregoing issues described are all components of that problem.

Baldur In Bulk

I got those Baldur’s Gate videos converted to something more modern. The problem turned out to be in the Interplay MVE demuxer code I wrote long ago for FFmpeg. Once upon a time, timestamps in FFmpeg were supposed to be in reference to a 90 kHz clock. Thanks to Pengvado for pointing out that my demuxer still made that assumption. Fixing the demuxer seems like a lot of work right now. So at this point in the exercise, I opted to simply hard code 15 fps for the framerate.

So I got that transcoding process underway, finally. And I made an interesting discovery along the way. I have a colleague who has this quote on his office whiteboard:

Baldur's Gate Nietzsche quote

I can only conclude that said colleague is a huge Baldur’s Gate fan.

Prerequisites for the transcoding operation (basic Kubuntu 8.04 virtual machine):

  • install the libfaac-dev package
  • download and manually compile YASM (required by x264 and the latest YASM packaged by Ubuntu is not bleeding edge enough)
  • download and compile the latest x264 snapshot; configure with –enable-shared
  • get the latest SVN of FFmpeg
  • configure and build FFmpeg with: configure –enable-gpl –enable-postproc –enable-avfilter –enable-avfilter-lavf –enable-swscale –enable-libx264 –enable-libfaac; I don’t really know if all the filter options are strictly necessary for this exercise but I’m used to them by now

So my process for transcoding in bulk after installing this software is:

  • use my Python script (, listed at the end of this post) to split the BIF resource into its constituent MVE files:
    $ MovieCD1.bif
    extracting file #0 at offset 132, 29654204 bytes, to 'MovieCD1.bif-0.mve'
    extracting file #1 at offset 29654336, 6530954 bytes, to 'MovieCD1.bif-1.mve'
  • bulk transcode:
    for mve in `ls *.mve`
      ffmpeg -y -i $mve \
      -acodec libfaac -ab 128k \
      -vcodec libx264 -vpre hq -b 500k -bt 500k \
      `basename $mve .mve`.mp4

The resulting files are highly competitive, size-wise, against the original MVE files. At first, I was monkeying with the bitrate because there were some annoying artifacts in the high motion areas. But then I watched the original videos using ffplay and realized that those artifacts are artifacts in the source material.

Continue reading

Hero Of The Samples Archive

Check out compn’s project/goal, listed on his MultimediaWiki user page, which he affectionately refers to as the Sample Challenge:

  1. start with older files
  2. google the filename to see if it was posted to the mplayer/ffmpeg mailing list
  3. do a search at and to see if it was posted there
  4. if no text file exists, try playback using mplayer and ffplay
  5. move sample to appropriate place and create/modify bugreport or delete

For extra credit, write up a brief description in the MultimediaWiki with cursory findings.

Godspeed, compn.

Migrating iTunes Libraries

I have been migrating from Windows XP to Mac OS X as my primary home computing solution. Why? Because I’m looking for entirely new reasons to hate computers, reasons that Linux and Windows just cannot provide.

The migration process has made me very aware of data investment and the cost of migrating platforms when you have been using one for a long time. This applies to closed and open software alike. When switching from Linux to Windows XP and then to Mac OS X, I have stayed with Mozilla’s Firefox web browser and Thunderbird mail client. Why not try more integrated solutions such as Apple’s Safari and Mail apps? Frankly, because I already have so much data invested into Mozilla’s solutions, and because it is absolutely trivial to migrate an entire installation between machines and even platforms (find the main data directory, copy the thing wholesale to its counterpart location on another machine, and update the top-level profiles.ini file with the correct directory path).

Apple’s iTunes presented a bigger challenge. I scanned many blogs for a solution and finally hit upon something that worked. I’m pretty sure I’m expected to blog about exactly how I got my iTunes library migration to work.

Problem statement:
Migrate an existing iTunes library from Windows XP to Mac OS X. Latest version of all OSes and iTunes at the time of this writing (iTunes 8.0(35) on Mac; whatever the latest iTunes 8.0 on Windows is).

What worked:

  • On Windows iTunes, “Consolidate library” — make sure the entire library is under iTunes management
  • browse to “My Documents\My Music\” and copy the entire iTunes\ directory up to a shared network drive, whole
  • On Mac OS, exit iTunes if it is running
  • go to Music/ directory in home directory and delete existing iTunes directory (this assumes you have not accumulated any media there yet or else this will destroy it)
  • copy iTunes/ directory from shared network drive, whole, into the Music/ directory
  • this is iTunes 8 so delete the iTunes Library Genius.itdb file that just got copied over or else suffer an iTunes #13026 error
  • important, “secret-sauce”-style step: press and hold the Option key while invoking iTunes; this will make iTunes prompt for an existing library; navigate to and select Music/iTunes/iTunes Library.itl

And that’s how I solved the problem for myself. I don’t know why it didn’t occur to me to hold down some random meta-key while invoking the program to attain the desired behavior. Somehow, magically, iTunes is able to sort out all of the music paths. I don’t know what format the .itl file takes but my best guess is that it must store paths relative to the top level iTunes directory. What I know is that all the media shows up correctly as does the proper metadata, a.k.a., my year+ data investment.

Anyway, if you want to try to replicate my steps, I hope you’re migrating fresh and don’t have anything valuable on the destination machine.

What didn’t work:

  • Monkeying with the plaintext XML file. Have I ever mentioned how much XML annoys me? Several blogs and forums offered the solution of copying all media but none of the database files, just the XML files with manually modified absolute path names. This actually seems quite reasonable but perhaps it’s iTunes 8 that isn’t fooled by this.
  • Transfer via iPod: This is evidently an official Apple solution but is unworkable for me since my library is larger than my iPod.
  • Transfer via series of optical discs: This is another official Apple solution. Allegedly, it backs up your library and associated metadata to a series of DVDs and another computer with iTunes is just supposed to notice the disc and start restoring. I tried this. iTunes wrote the first of a series of DVDs. Then it prompted me for the second while it kept spinning the DVD up and down. When it finally surrendered the first disc and I inserted the second disc… iTunes just sort of forgot what it was doing and blew off the remainder of the backup operation.
  • Probably a few other failed approaches that I am blocking out right now…