In this context, the oscilloscope is used to draw the time-domain samples of an audio wave form. I have written such a plugin as part of the xine project. However, for that project, I didn’t have to write the full playback pipeline– my plugin was just handed some PCM data and drew some graphical data in response. Now I’m trying to write the entire engine in a standalone program and I’m wondering how to get it just right.

This is an SDL-based oscilloscope visualizer and audio player for Game Music Emu library. My approach is to have an audio buffer that holds a second of audio (44100 stereo 16-bit samples). The player updates the visualization at 30 frames per second. The o-scope is 512 pixels wide. So, at every 1/30th second interval, the player dips into the audio buffer at position ((frame_number % 30) * 44100 / 30) and takes the first 512 stereo frames for plotting on the graph.

It seems to be working okay, I guess. The only problem is that the A/V sync seems to be slightly misaligned. I am just wondering if this is the correct approach. Perhaps the player should be performing some slightly more complicated calculation over those (44100/30) audio frames during each update in order to obtain a more accurate graph? I described my process to an electrical engineer friend of mine and he insisted that I needed to apply something called hysteresis to the output or I would never get accurate A/V sync in this scenario.

Further, I know that some schools of thought on these matters require that the dots in those graphs be connected, that the scattered points simply won’t do. I guess it’s a stylistic choice.

Still, I think I have a reasonable, workable approach here. I might just be starting the visualization 1/30th of a second too late.

Here are the students, projects, and mentors for this summer:

1. Andrew D’Addesio is working on an Opus Decoder, mentored by Justin Ruggles
2. Guillaume Martres is working on an HEVC video decoder, mentored by Mashiat Sarker Shakkhar
3. Jan Ekström is working on an LGPL Ut Video encoder, mentored by Kostya Shishkov
4. Jordi Ortiz is working to rewrite avserver, mentored by Luca Barbato
5. Samuel Pitoiset is working on an RTMP[E|S|T|TE] protocol implementation, mentored by Martin Storsjö

Wish them luck– these are some ambitious projects.

I found myself staring at the file again today and Googled the filename. This led me to a few Japanese sites which also contained working URLs for a few more .vg2 samples. Some other clues led me to a Russian language forum where someone had linked to a site that had Win32 codec modules that could process the files. The site was defunct but the Internet Archive Wayback Machine kept a copy for me, as well as copies of several more .vg2 samples from a defunct Japanese site previously involved with this codec.

Sometimes this internet technology works really well. But I digress.

Anyway, through all this, I finally found a clue: XVD. and wouldn’t you know, there is already a basic page on the MultimediaWiki describing the technology. In fact, while VG2 is a custom container, the MultimediaWiki states that the video component has a FourCC of VGMV, and there is already a file named VGMV.avi in the root V-codecs/ samples directory, something I vow to correct (that’s a big pet peeve of mine– putting samples in the root V-codecs/ or A-codecs/ directories).

XVD… XVD… XVD… why does that sound so familiar? Oh, of course; there is a company named XVD and they have an office in the Bay Area which I have passed on numerous occasions, like this morning:

<

Someone originally connected with the multimedia technology in question operates a website which contains an unofficial history of the XVD tech. At first, I was wondering if the technology was completely defunct (and should therefore be open sourced). But if XVD’s solutions page (dated 2010) is to be believed, the technology is still in service, and purported to be better than H.264 and VC-1: “The current generation of XVD video compression technology provides better video quality at any given data rate than standards-based codecs (H.264 or VC-1) with four times lower encoding complexity (when compared with H.264 Main Profile).”

If they say so. For my part, I’m just happy that I have finally figured out what this lamborghini_850kbps.vg2 is so that I can properly catalog it on the samples site, which I have now done, along with other samples and various codecs modules.

This episode reminds me that there’s a branch office of Zygo Corporation close to my home (though the headquarters are far, far away). The companies you see in Silicon Valley. Anyway, long-time open source multimedia hackers will no doubt recognize Zygo from the ZyGo FourCC & video codec transported in QuickTime files that was almost decode-able using an H.263 decoder.

I may never learn what Zygo’s core competency actually is, but I will always remember their multimedia tech every time I run past their office.

I decided to dust off that old copy of The 11th Hour and have another go at it.

Baseline
The game consists of 4 CD-ROMs. Each disc has a media/ directory that has a series of files bearing the extension .gjd, likely the initials of one Graeme J. Devine. These are resource files which are merely headerless concatenations of other files. Thus, at first glance, one file might appear to be a single RoQ file. So that’s the source of some of the difficulty: Sending an apparent RoQ .gjd file through a RoQ player will often cause the program to complain when it encounters the header of another RoQ file.

I have uploaded some samples to the usual place.

However, even the frames that a player can decode (before encountering a file boundary within the resource file) look wrong.

Investigating Codebooks Using dreamroq
I wrote dreamroq last year– an independent RoQ playback library targeted towards embedded systems. I aimed it at a gjd file and quickly hit a codebook error.

RoQ is a vector quantizer video codec that maintains a codebook of 256 2×2 pixel vectors. In the Quake III and later RoQ files, these are transported using a YUV 4:2:0 colorspace– 4 Y samples, a U sample, and a V sample to represent 4 pixels. This totals 6 bytes per vector. A RoQ codebook chunk contains a field that indicates the number of 2×2 vectors as well as the number of 4×4 vectors. The latter vectors are each comprised of 4 2×2 vectors.

Thus, the total size of a codebook chunk ought to be (# of 2×2 vectors) * 6 + (# of 4×4 vectors) * 4.

However, this is not the case with The 11th Hour RoQ files.

Longer Codebooks And Mystery Colorspace
Juggling the numbers for a few of the codebook chunks, I empirically determined that the 2×2 vectors are represented by 10 bytes instead of 6. Now I need to determine what exactly these 10 bytes represent.

I should note that I suspect that everything else about these files lines up with successive generations of the format. For example if a file has 640×320 resolution, that amounts to 40×20 macroblocks. dreamroq iterates through 40×20 8×8 blocks and precisely exhausts the VQ bitstream. So that all looks valid. I’m just puzzled on the codebook format.

Here is an example codebook dump:

ID 0x1002, len = 0x0000014C, args = 0x1C0D
  0: 00 00 00 00 00 00 00 00 80 80
  1: 08 07 00 00 1F 5B 00 00 7E 81
  2: 00 00 15 0F 00 00 40 3B 7F 84
  3: 00 00 00 00 3A 5F 18 13 7E 84
  4: 00 00 00 00 3B 63 1B 17 7E 85
  5: 18 13 00 00 3C 63 00 00 7E 88
  6: 00 00 00 00 00 00 59 3B 7F 81
  7: 00 00 56 23 00 00 61 2B 80 80
  8: 00 00 2F 13 00 00 79 63 81 83
  9: 00 00 00 00 5E 3F AC 9B 7E 81
  10: 1B 17 00 00 B6 EF 77 AB 7E 85
  11: 2E 43 00 00 C1 F7 75 AF 7D 88
  12: 6A AB 28 5F B6 B3 8C B3 80 8A
  13: 86 BF 0A 03 D5 FF 3A 5F 7C 8C
  14: 00 00 9E 6B AB 97 F5 EF 7F 80
  15: 86 73 C8 CB B6 B7 B7 B7 85 8B
  16: 31 17 84 6B E7 EF FF FF 7E 81
  17: 79 AF 3B 5F FC FF E2 FF 7D 87
  18: DC FF AE EF B3 B3 B8 B3 85 8B
  19: EF FF F5 FF BA B7 B6 B7 88 8B
  20: F8 FF F7 FF B3 B7 B7 B7 88 8B
  21: FB FF FB FF B8 B3 B4 B3 85 88
  22: F7 FF F7 FF B7 B7 B9 B7 87 8B
  23: FD FF FE FF B9 B7 BB B7 85 8A
  24: E4 FF B7 EF FF FF FF FF 7F 83
  25: FF FF AC EB FF FF FC FF 7F 83
  26: CC C7 F7 FF FF FF FF FF 7F 81
  27: FF FF FE FF FF FF FF FF 80 80

Note that 0x14C (the chunk size) = 332, 0x1C and 0x0D (the chunk arguments — count of 2×2 and 4×4 vectors, respectively) are 28 and 13. 28 * 10 + 13 * 4 = 332, so the numbers check out.

Do you see any patterns in the codebook? Here are some things I tried:

• Treating the last 2 bytes as U & V and treating the first 4 as the 4 Y samples:
  
  
  
• Treating the last 2 bytes as U & V and treating the first 8 as 4 16-bit little-endian Y samples:
  
  
  
• Disregarding the final 2 bytes and treating the first 8 bytes as 4 RGB565 pixels (both little- and big-endian, respectively, shown here):
  
  
  
• Based on the type of data I’m seeing in these movies (which appears to be intended as overlays), I figured that some of these bits might indicate transparency; here is 15-bit big-endian RGB which disregards the top bit of each pixel:
  
  
  

These images are taken from the uploaded sample bdpuz.gjd, apparently a component of the puzzle represented in this screenshot.

Unseen Types
It has long been rumored that early RoQ files could contain JPEG images. I finally found one such specimen. One of the files bundled early in the uploaded fhpuz.gjd sample contains a JPEG frame. It’s a standard JFIF file and can easily be decoded after separating the bytes from the resource using ‘dd’. JPEGs serve as intraframes in the coding scheme, with successive RoQ frames moving objects on top.

However, a new chunk type showed up as well, one identified by 0×1030. I have never encountered this type. Where could I possibly find data about this? Fortunately, iD Games recently posted all of their open sourced games at Github. Reading through the code for their official RoQ decoder, I see that this is called a RoQ_PACKET. The name and the code behind it are both supremely unhelpful. The code is basically a no-op. The payloads of the various RoQ_PACKETs from one sample are observed to be either 8784, 14752, or 14760 bytes in length. It’s very likely that this serves the same purpose as the JPEG intraframes.

Other Tidbits
I read through the readme.txt on the first game disc and found this nugget:

        g)      Animations displayed normally or in SPOOKY MODE

                SPOOKY MODE is blue-tinted grayscale with color cursors, puzzle
                and game pieces.  It is the preferred display setting of the
                developers at Trilobyte.  Just for fun, try out the SPOOKY
                MODE.

The MobyGames screenshot page has a number of screenshots labeled as being captured in spooky mode. Color tricks?

Meanwhile, another twist arose as I kept tweaking dreamroq to deal with more RoQ weirdness: After modifying my dreamroq code to handle these 10-byte vectors, it eventually chokes on another codebook. These codebooks happen to have 6-byte vectors again! Fortunately, I was already working on a scheme to automatically detect which codebook is in play (plugging the numbers into a formula and seeing which vector size checks out).

Copyright 2003, and labeled as PC ONLY. Each disc claims to have 2 episodes. So are these some sort of video discs? Any gaming elements? I dove in to investigate.

So, it turns out that there are some games on these discs, done in Flash Player (which tells me that these were probably available on the web at some point). Here’s a shooting gallery game from the first disc:

As promised by the CD-ROM copy, the menu does grant access to 2 classic G.I. Joe episodes. Selecting either one launches this:

Powered by C-ezy? Am I interpreting that correctly? Anyway, the video player goes fullscreen and looks fine (given the source material). I can’t capture screenshots and controls are limited to: space for pause, ESC to exit player, and up/down to control volume. No seeking and certainly no onscreen controls. Pretty awful player.

Studying the first disc, I find a 550 MB file with the name 5859Hasbro.egm. Coupled with ep58.cfg and ep59.cfg files in the same directory, I gather that the disc has G.I. Joe episodes 58 and 59 (though the exact episodes, “There’s No Place Like Springfield” parts 1 and 2, are listed on Wikipedia as being episodes 154 and 155; but who’s counting?). The cfg files contain this text:

ep58.cfg:
EGM_GIJOE.exe
5859Hasbro.egm /noend /track:0 /singletrack 

ep59.cfg:
EGM_GIJOE.exe
5859Hasbro.egm /noend /track:1 /singletrack

The big EGM file starts with the string “Egenie Player”. After that, I see absolutely no clues. The supporting EGM_GIJOE.exe file has some interesting strings: “Decore Bits Per Pixel” (I know I have seen “Decore” used to mean “decoding core” in some libraries), “Egenie Player – %s, Version:%s”, “4th June 2002″, a list of common FourCC tags seen in AVI files, “Brought to you by Martin, Patrick Bob and Bren” (do you suppose “Patrick Bob” is one person’s name?), a list of command line options…

Aha! A URL: http:\\www.e-genie.tv (yep, backslashes, not forward slashes). e-genie.tv seems to redirect to mygenie.tv, which… doesn’t appear to be strictly related to video technology these days.

Aside: They have all of the artpacks archived at Github.

The full corpus of nearly 4000 artpacks contains over 146,000 files. Versus my sampling of 350 artpacks and 13,000 files that covered all but 45 lines of the ansi.c source file, the full corpus has files to exercise… 6 more of those lines. Whee. This means that there are files which exercise the reverse and concealed attributes, all 3 “erase in line” modes, and one more error path (which probably wasn’t a valid file anyway).

Missing features mostly cluster around different video modes, including: 320×200 (25 rows), 640×200 (25 rows), 640×350 (43 rows), and 640×480 (60 rows); on the plus side, nothing tripped the “unsupported screen mode” case. There are no files that switch modes during playback.

I guess statistical sampling theory holds out here– a small set of randomly chosen files would do a fine job covering code. But this experiment is about finding the statistical outliers.

Current Process
When adding a test that depends on a sample (like a demuxer or decoder test), it’s ideal to add a sample that’s A) small, and B) exercises as much of the codebase as possible. When I was studying code coverage statistics for the WC4-Xan video decoder, I noticed that the sample didn’t exercise one of the 2 possible frame types. So I scouted samples until I found one that covered both types, trimmed the sample down, and updated the coverage suite.

I started wondering about a method for finding the optimal test sample for a given piece of code, one that exercises every code path in a module. Okay, so that’s foolhardy in the vast majority of cases (although I was able to add one test spec that pushed a module’s code coverage from 0% all the way to 100% — but the module in question only had 2 exercisable lines). Still, given a large enough corpus of samples, how can I find the smallest set of samples that exercise the complete codebase?

This almost sounds like an NP-complete problem. But why should that stop me from trying to find a solution?

Science Project
Here’s the pitch:

• Instrument FFmpeg with code coverage support
• Download lots of media to exercise a particular module
• Run FFmpeg against each sample and log code coverage statistics
• Distill the resulting data in some meaningful way in order to obtain more optimal code coverage

That first step sounds harsh– downloading lots and lots of media. Fortunately, there is at least one multimedia format in the projects that tends to be extremely small: ANSI. These are files that are designed to display elaborate scrolling graphics using text mode. Further, the FATE sample currently deployed for this test (TRE_IOM5.ANS) only exercises a little less than 50% of the code in libavcodec/ansi.c. I believe this makes the ANSI video decoder a good candidate for this experiment.

Procedure
First, find a site that hosts a lot ANSI files. Hi, sixteencolors.net. This site has lots (on the order of 4000) artpacks, which are ZIP archives that contain multiple ANSI files (and sometimes some other files). I scraped a list of all the artpack names.

In an effort to be responsible, I randomized the list of artpacks and downloaded periodically and with limited bandwidth ('wget --limit-rate=20k').

Run ‘gcov’ on ansi.c in order to gather the full set of line numbers to be covered.

For each artpack, unpack the contents, run the instrumented FFmpeg on each file inside, run ‘gcov’ on ansi.c, and log statistics including the file’s size, the file’s location (artpack.zip:filename), and a comma-separated list of line numbers touched.

Definition of ‘Optimal’
The foregoing procedure worked and yielded useful, raw data. Now I have to figure out how to analyze it.

I think it’s most desirable to have the smallest files (in terms of bytes) that exercise the most lines of code. To that end, I sorted the results by filesize, ascending. A Python script initializes a set of all exercisable line numbers in ansi.c, then iterates through each each file’s stats line, adding the file to the list of candidate samples if its set of exercised lines can remove any line numbers from the overall set of lines. Ideally, that set of lines should devolve to an empty set.

I think a second possible approach is to find the single sample that exercises the most code and then proceed with the previously described method.

Initial Results
So far, I have analyzed 13324 samples from 357 different artpacks provided by sixteencolors.net.

Using the first method, I can find a set of samples that covers nearly 80% of ansi.c:


0 bytes: bad-0494.zip:5
1 bytes: grip1293.zip:-ANSI---.---
1 bytes: pur-0794.zip:.
2 bytes: awe9706.zip:-ANSI───.───
61 bytes: echo0197.zip:-(ART)-
62 bytes: hx03.zip:HX005.DAT
76 bytes: imp-0494.zip:IMPVIEW.CFG
82 bytes: ice0010b.zip:_cont'd_.___
101 bytes: bdp-0696.zip:BDP2.WAD
112 bytes: plain12.zip:--------.---
181 bytes: ins1295v.zip:-°VGA°-. н
219 bytes: purg-22.zip:NEM-SHIT.ASC
289 bytes: srg1196.zip:HOWTOREQ.JNK
315 bytes: karma-04.zip:FASHION.COM
318 bytes: buzina9.zip:ox-rmzzy.ans
411 bytes: solo1195.zip:FU-BLAH1.RIP
621 bytes: ciapak14.zip:NA-APOC1.ASC
951 bytes: lght9404.zip:AM-TDHO1.LIT
1214 bytes: atb-1297.zip:TX-ROKL.ASC
2332 bytes: imp-0494.zip:STATUS.ANS
3218 bytes: acepak03.zip:TR-STAT5.ANS
6068 bytes: lgc-0193.zip:LGC-0193.MEM
16778 bytes: purg-20.zip:EZ-HIR~1.JPG
20582 bytes: utd0495.zip:LT-CROW3.ANS
26237 bytes: quad0597.zip:MR-QPWP.GIF
29208 bytes: mx-pack17.zip:mx-mobile-source-logo.jpg
----
109440 bytes total


A few notes about that list: Some of those filenames are comprised primarily of control characters. 133t, and all that. The first file is 0 bytes. I wondered if I should discard 0-length files but decided to keep those in, especially if they exercise lines that wouldn’t normally be activated. Also, there are a few JPEG and GIF files in the set. I should point out that I forced the tty demuxer using -f tty and there isn’t much in the way of signatures for this format. So, again, whatever exercises more lines is better.

Using this same corpus, I tried approach 2– which single sample exercises the most lines of the decoder? Answer: blde9502.zip:REQUEST.EXE. Huh. I checked it out and ‘file’ ID’s it as a MS-DOS executable. So, that approach wasn’t fruitful, at least not for this corpus since I’m forcing everything through this narrow code path.

Think About The Future
Where can I take this next? The cloud! I have people inside the search engine industry who have furnished me with extensive lists of specific types of multimedia files from around the internet. I also see that Amazon Web Services Elastic Compute Cloud (AWS EC2) instances don’t charge for incoming bandwidth.

I think you can see where I’m going with this.

See Also:

• Running this experiment on 4000 artpacks / 146,000 files

Some other recent developments in the projects that I wanted to call out: An encoder for the Apple ProRes encoder from Kostya; XWD (X window dump) image decoding and encoding from Paul B. Mahol; a Sun rasterfile encoder from Aneesh Dogra.

And then there’s the new playback system for CDXL files, also courtesy of Paul B. Mahol. I wasn’t familiar with this format until I wrote this post, which is surprising, given the format’s vintage. This was a CD-ROM FMV format favored for Amiga computers. Here it is in all its 160x120x10fps glory:

That’s the amigaball.cdxl sample available in the repository. The sample is 3835910 bytes large and plays for about 24 seconds. This yields a data rate of about 159 kbytes/second. So, yeah, single-speed CD-ROM FMV.

GhettoRSS
https://github.com/multimediamike/GhettoRSS
I describe this as a true offline RSS reader. Technically, it’s arguably not a true offline RSS reader. Rather, it does what most people actually want an offline RSS reader to do.

I wrote this about 2 years ago when I had a long daily train ride with a disconnected netbook. I quickly learned that I couldn’t count on offline RSS readers simply because most RSS feeds to not contain much meat. Thus, I created a program that follows URLs in RSS feeds, downloads web pages and supporting images and CSS files, and caches them in an offline database which can be read via a local web browser.

I wrote more information about this little project 2 years ago (here is part 1 and here is part 2). I fixed a few bugs in preparation for posting it but I probably won’t work on this anymore since I don’t have any use for it (the commute is long gone, but I didn’t even use it when I was commuting because I decided I just didn’t care enough to read the feeds on the train).

xbfuse
https://github.com/multimediamike/xbfuse
This is a FUSE module for mounting Xbox/360 optical disc filesystems. Here is when I first discussed it. The tool has had its own little homepage for a long time. This tool has seen some development, as I learned from Googling for “xbfuse”. Regrettably, no one who has modified the tool has ever contacted me about it (at least, not that I can recall). This is unfortunate because the patches I have seen floating around which fix my xbfuse for various installations usually boil down replacing many occurrences of an include path in the autotool-generated build system. There is probably a simpler, cleaner fix.

gcfuse
https://github.com/multimediamike/gcfuse
Written prior to xbfuse, this is a FUSE module for mounting GameCube optical disc filesystems. I first discussed this here and here. This tool has not seen too much direct development although someone eventually used it as the basis for WiiFuse which, as you can predict, mounts optical disc filesystems from Nintendo Wii games.

Immediately, I delved in expecting to find Xan-encoded AVI files that would play perfectly using FFmpeg/Libav. Instead, I found a directory labeled flics/ that indeed has a lot of AVI files, but not in Xan. The programs attempt to interpret them as raw RGB. The strangest thing is the first frame often looks correct, if upside down:

The first file I peered inside had the video FourCC ‘RRV1′. Searching for this led me to this discussion forum where people have already been hacking on this very format (Origin games invariably get a heap of lasting love). The forum participants have observed that 3 codecs are in play in this flics/ directory, including ‘RRV1′, ‘RRV2′, and ‘JYV1′, which apparently correspond to the initials of certain developers. The reason that the programs identify the files as raw RGB is because the FourCCs don’t appear everywhere that they’re supposed to. Additionally, there are several trailers for other Origin/EA games stored in Cinepak format elsewhere on the disc.

It seems that I’m the person who added this title to the Xan wiki page, obviously with no first-hand evidence to back it up. Meanwhile, the forum participants speculate that the files are descended from the old Autodesk FLIC format (which would explain why they live in a directory called flics/). Corroborating strings extracted from the CRUSADER.EXE file include “FlicWait”, “FlicPlayer”, “Flic %s not found.”, “flicpath”, and “FLICPLAY.C”.

The disc also features a sound/ directory which contains AMF files. Suxen Drol already documented these on the wiki as Asylum Media Format files. The disc contains an ASYLUM.DLL file as well as a utility called MOD2AMF.EXE. The latter works beautifully on a random MOD file I had laying around. The AMF file is a bit larger.

Samples for all 3 FourCCs can be found here, while the AMF files and associated utilities are here.