Monthly Archives: December 2009

Microsoft Jingle Bells

I acquired an MP3 all the way back in 1997 or 1998 which is a parody of Jingle Bells holiday tune that laments Microsoft-branded bloatware. Mildly humorous at the time, the song now serves as a technology time capsule. The internet seems unsure of who wrote or performed the song or when it was recorded, but the lyrics give a clue about its vintage. The singer complains that MS Word takes a whole 60 megabytes of RAM to run and occupies 900 megabytes of disc space.

Nine-tenths of a gig, biggest ever seen,
God, this program’s big: MS Word 15!
Comes on ten CDs, and requires–damn!
Word is fine, but jeez, 60 megs of RAM?!
Oh! Microsoft, Microsoft, bloatware all the way!
I’ve sat here installing Word, since breakfast yesterday!
Oh! Microsoft, Microsoft, moderation, please.
Guess you hadn’t noticed: Four-gig drives don’t grow on trees!

This clearly hearkens back to a time when 4 gigabyte drives were considered premium. I’m trying to remember when 4 GB drives were introduced and would have commanded a prohibitive price. Whenever it was, it still strikes me as ironic considering that I am typing this on my Linux-based Eee PC 701 which is equipped with 4 GB of storage which is just barely enough for Ubuntu 9.10 to tread water (ran out of space today, in fact, and I had to scramble to make room to keep working).

Supplying FFmpeg With Metadata

UPDATE, 2010-06-17: This information is now maintained via the FFmpeg Metadata page on the MultimediaWiki.

While creating my automated game archiving solution, I wanted to automatically tag ALAC/M4A files with appropriate metadata while encoding using FFmpeg. Then I realized I didn’t know how to do that. I do remember that FFmpeg recently supplanted a series of specific metadata command line options (like -title and -album) with a highly generalized metadata framework that is accessed by the option ‘-metadata <key>=<value>’. The official documentation provides this lonely (and, I came to realize, useless NO-OP; more on why later) example of the option’s usage:

    `-metadata key=value'

      Set a metadata key/value pair. For example, 
      for setting the title in the output file:

      ffmpeg -i in.avi -metadata title="my title" out.flv

So this option allows you to specify absolutely any key/value metadata pair you can imagine. However, a specific muxer won’t necessarily recognize it. How can I know the specific keys that, e.g., the MOV/MP4 muxer honors? 2 methods spring to mind: Trial and error (worked out well for me at first) and reading the code (which was made a good deal easier when I already knew a few of the keys that worked).

I think it would be a good idea to document the specific metadata keys that each muxer in FFmpeg recognizes. This blog post will lead by example. This data comes from SVN revision 20910, current as of 2009-12-21.

QuickTime/MOV/MP4/M4A/et al.

I have constructed the following table of the metadata keys that libavformat/movenc.c honors. The low-level identifier column lists the atom name that the format uses to encode the data on disc, which is not interesting to most readers. For the interested but uninitiated, the notation, e.g., ‘\251nam’ indicates a 4-byte code consisting of the byte A9 in hexadecimal (or 251 in octal) followed by the ASCII characters ‘n’, ‘a’, and ‘m’.

Continue reading

PS3 Notes

I have been working (and occasionally playing) with my PlayStation 3 recently. I upgraded the 80 GB internal hard drive to a 1/2 TB one. Since I have the old 80 GB HD laying around, of course I have to plug it in and see if there’s anything familiar about the data. It’s a short exploration: As you might suspect, the HD is completely impenetrable. No partition table reported through Linux fdisk. No human-readable strings can be seen when running ‘strings’ over the raw HD sectors. Based on forum postings I have read where one PS3 HD can’t successfully be transplanted to another PS3 (and have all the data accessible; the HD could still be reformatted fresh to work in another PS3), I’m guessing that every sector is encrypted with a key derived at least partially from a unique ID embedded in each console. That’s all the effort I plan to put into this exercise. Next stop for this HD is my Eee PC 701 which is currently struggling to run Ubuntu Linux on a mere 4 GB SSD.

I downloaded a free movie trailer through the PlayStation store. When I inspected the information through the PS3’s XMB menu, the filetype was reported as “MNV”. A little Googling ties this format into the paid content format of the PS3 store. I’m not especially confident about this format since the trailer that I downloaded doesn’t even play correctly on the PS3. The video stutters back and forth, almost as though it’s swapping pairs of frames during playback: 1, 0, 3, 2, 5, 4, 7, 6, etc. The XMB allows me to “backup” this media. This option needs to be distinguished from “copy”, which is sometimes an option. “Copy” implies an unlocked version that can be copied onto removable media and used anywhere. “Backup” implies that it can be copied onto removable media but is still keyed to — and can only be used on — this console. I backed it up and was able to inspect the data on the USB drive. It turns out that the MNV file is still a stock MP4 but with custom DRM. When FFmpeg is aimed at this file, this is the result:

[h264 @ 0x1004000]AVC: nal size -2055117847
[h264 @ 0x1004000]no frame!
[...repeated many times...]
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1002600]max_analyze_duration reached

Seems stream 0 codec frame rate differs from container frame rate:
 48000.00 (48000/1) -> 23.98 (24000/1001)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from
 '/Volumes/KINGSTON/PS3/EXPORT/VIDEOBKP/20091220-220733-00000001/20091220-220733-00000001.001':
  Metadata:
    major_brand     : MGSV
    minor_version   : 20842393
    compatible_brands: MGSVmp42isom
  Duration: 00:01:46.64, start: 0.000000, bitrate: 8651 kb/s
    Stream #0.0(und): Video: h264, 2205 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 48k tbc
    Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 264 kb/s
    Stream #0.2(eng): Audio: aac, 48000 Hz, 5.1, s16, 395 kb/s
    Stream #0.3(und): Data: mp4s / 0x7334706D, 759552 kb/s
Video pixel format is unknown, stream cannot be decoded

I remember some patches flying around the FFmpeg-devel list recently which would allow the program to print warnings and bail out if it encountered a known DRM scheme. When I shove an Apple-encrypted file through FFmpeg, it doesn’t tell me anything special so I don’t think the patch is in yet. However, FFmpeg should probably detect this type of DRM file as well.

Better Compression For ISOs

Pursuant to my project idea of archiving a bunch of obscure, old, CD-ROM-based games, I got an idea about possibly compressing ISO-9660 filesystems more efficiently than could be done with standard lossless compressors such as gzip, bzip2, and lzma. This is filed under “outlandish brainstorms” since I don’t intend to get around to doing this anytime in the near future, but I wanted to put the idea out there anyway.

Game developers throughout the era of optical disc-based games have been notoriously lazy about data efficiency. A typical manifestation of this is when one opens a disc to find dozens or hundreds of sizable, uncompressed WAV audio files. Same goes for uncompressed graphical assets. General lossless compressors are able to squeeze some bytes out of uncompressed audio and video data but specialized algorithms (like FLAC for audio and PNG for images) perform better.

Here’s the pitch: Create a format that analyzes individual files in an ISO filesystem and uses more appropriate compression algorithms. If there’s a 16-bit PCM WAV file, use FLAC to compress it. If there’s an uncompressed BMP file, compress as a PNG file. Stuff them all in a new file with an index and a mapping to their original files.

As an additional constraint, it obviously needs to be possible to reconstruct a bit-exact ISO from one of these compressed images. So the index will need to store information about what sector ranges different files occupied.

The only other attempt I have seen for a specialized ISO compression format is the CISO format used for compressing ISO filesystems for storage on flash memory to be read in PSP units. That format uses zlib to compress sector by sector which has the advantage of being able to mount and use a CISO image in place without decompressing the entire filesystem. That should be possible for this filesystem a well by using a FUSE driver. Mount the filesystem and the driver presents the file tree upon request and decompressed the files on demand.

Many compression algorithms have assorted methods of operation depending on what is appropriate for a particular range of data. This scheme is merely an extension of that principle. I have no idea if this idea would hold water in the general case. But thanks to my archiving brainstorm, I expect I will have quite a lot of data to analyze.

See Also: