Category Archives: Python

Designing A Download Strategy

The uncommon video codecs list mentioned in the last post is amazing. Here are some FourCCs I have never heard of before: 3ivd, abyr, acdv, aura, brco, bt20, bw10, cfcc, cfhd, digi, dpsh, dslv, es07, fire, g2m3, gain, geox, imm4, inmc, mohd, mplo, qivg, suvf, ty0n, xith, xplo, and zdsv. There are several that have been found to be variations of other codecs. And there are some that were only rumored to exist, such as aflc as a codec for storing FLIC data in an AVI container, and azpr as an alternate FourCC for rpza. We now have samples. The existence of many of these FourCCs has, in fact, been cataloged on FourCC.org. But I was always reticent to document the FourCCs in the MultimediaWiki unless I could find either samples or a binary codec.

But how to obtain all of these samples?

Do you ever download files from the internet? Of course you do. Do you ever download a bunch of files at a time? Maybe. But have you ever had to download a few thousand files?

I have some experience to guide me in this. Continue reading →

Baldur In Bulk

I got those Baldur’s Gate videos converted to something more modern. The problem turned out to be in the Interplay MVE demuxer code I wrote long ago for FFmpeg. Once upon a time, timestamps in FFmpeg were supposed to be in reference to a 90 kHz clock. Thanks to Pengvado for pointing out that my demuxer still made that assumption. Fixing the demuxer seems like a lot of work right now. So at this point in the exercise, I opted to simply hard code 15 fps for the framerate.

So I got that transcoding process underway, finally. And I made an interesting discovery along the way. I have a colleague who has this quote on his office whiteboard:


Baldur's Gate Nietzsche quote

I can only conclude that said colleague is a huge Baldur’s Gate fan.

Prerequisites for the transcoding operation (basic Kubuntu 8.04 virtual machine):

  • install the libfaac-dev package
  • download and manually compile YASM (required by x264 and the latest YASM packaged by Ubuntu is not bleeding edge enough)
  • download and compile the latest x264 snapshot; configure with –enable-shared
  • get the latest SVN of FFmpeg
  • configure and build FFmpeg with: configure –enable-gpl –enable-postproc –enable-avfilter –enable-avfilter-lavf –enable-swscale –enable-libx264 –enable-libfaac; I don’t really know if all the filter options are strictly necessary for this exercise but I’m used to them by now

So my process for transcoding in bulk after installing this software is:

  • use my Python script (parse-bif-graf.py, listed at the end of this post) to split the BIF resource into its constituent MVE files: 
    $ parse-bif-graf.py MovieCD1.bif
extracting file #0 at offset 132, 29654204 bytes, to 'MovieCD1.bif-0.mve'
extracting file #1 at offset 29654336, 6530954 bytes, to 'MovieCD1.bif-1.mve'
[...]
  • bulk transcode: 
    for mve in `ls *.mve`
do 
  ffmpeg -y -i $mve \
  -acodec libfaac -ab 128k \
  -vcodec libx264 -vpre hq -b 500k -bt 500k \
  `basename $mve .mve`.mp4
done
[...]

The resulting files are highly competitive, size-wise, against the original MVE files. At first, I was monkeying with the bitrate because there were some annoying artifacts in the high motion areas. But then I watched the original videos using ffplay and realized that those artifacts are artifacts in the source material.

Continue reading →

Psychotron Story Arc

I’m still on this game hacking kick. Today’s target is The Psychotron, perhaps the most painful interactive movie game I have ever seen. Thing is, I have to admit that I have never actually played this game; I am judging the game purely on the plain Cinepak/PCM AVI files that comprise the key feature of the game. I don’t even own the full game, only a demo. Granted, I have tried to play the Psychotron demo but could never make it work. These 1994 multimedia-heavy games…


Psychotron gangsters

Whereas the conversations in Flash Traffic: City of Angels were packed in a quasi-compressed binary format, the conversation tree in Psychotron is stored in a series of .SEG files that look like the file S5H1PD1.SEG here:

\scene5\s5h1Pd1.avi*from 23 to 283*3*Fold.*10*S5h1r1.seg*Bet a large amount.*1000*s5h1r2.seg*Bet a small amount.*500*s5h1r3.seg*

Pretty straightforward to figure out. It also helps me understand a technical matter about the game that irritated me so much when I viewed the FMV files standalone– when you view the game’s files in a standalone player, you will see that they invariably start paused with tracking lines (apparently digitized from a VCR). I am guessing that the game uses the field “from 23 to 283” to describe which frames of the file are meant for human consumption.

Anyway, I wrote a Python script to print out a Graphviz-compatible spec to map out the game demo’s dialog tree. Warning! There might be spoilers in the tree!


Pyschotron demo flowchart
Click for full image
Click for Graphviz source

Ah, who am I kidding? No one cares about spoilers in this alleged story. It is interesting to note that 2 of those trees don’t correlate to any of the scenes on the disc.

The Python source code for generating the graph is below. I finally ordered the full game from an eBay seller the other day so I am wondering if the graphing utility will be applicable to the full game.

Continue reading →

Libsndfile Survey; CAFF

I have been sitting on the results of this little experiment for a month now. I was in a bit more of a hurry during the qualification period for this year’s Summer of Code because I knew it would eventually yield a bumper crop of suitable qualification tasks. But that time has long passed.

The task at hand is to systematically survey the types of files that libsndfile can create, see what it can do that FFmpeg can’t, and make a plan to get those formats into FFmpeg starting with listing them as small tasks.

Libsndfile comes with several small example utilities (something can FFmpeg could probably use instead of, or to supplement, one big utility). One such program is sndfile-convert

I had to modify sndfile-convert since not all of the supported formats were enumerated in its case-switch statements. The result of the above command was 120 unique format/codec combinations. I have uploaded all 120 cow moo samples (taken from the OpenOffice media suite, handy on my Eee PC when I was running this experiment).

How many of these 120 files can FFmpeg decode? I compiled FFmpeg with GSM 6.10 support and used the following command to print out FFmpeg return code and filename for each sample:

for file in `ls`; do
  ffmpeg -i $file -f wav - 1> /dev/null 2> /dev/null;
  echo $? $file; 
done

The count is: 26 supported, 94 unsupported. Time to get to work. You wouldn’t believe how many different ways there are to wrap raw PCM data with a basic file header for storage and transport.

Actually, closer inspection will reveal that FFmpeg is not necessarily ready to support all of these file formats since a number of them contain 24- or 32-bit integer PCM, or 32- or 64-bit floating point PCM; these are longstanding FFmpeg TODO items.

Libsndfile is actually the first program I have encountered that handles Apple Core Audio Format Files (CAF or CAFF). I haven’t even found an Apple program that creates these, at least not among the offerings bundled with Mac OS X 10.5. Now that I have created some CAFs, I see that Apple’s QuickTime Player plays them handily.

FFmpeg also doesn’t support multichannel (more-than-stereo) audio very robustly in its present incarnation. Yeah, that’s another item on the TODO list. Check out the complete specs for CAFF, however. I think if we made it a goal to support CAFF to its fullest (save perhaps for its pulse-width modulation provisions), FFmpeg’s audio handling would reign supreme at the end of the journey.

For the curious, these are the 26/120 files that FFmpeg can decode at this time:

Continue reading →