I’m trying to figure out how to write a software oscilloscope audio visualization. It’s made more frustrating by the knowledge that I am certain that I have accomplished this task before.

In this context, the oscilloscope is used to draw the time-domain samples of an audio wave form. I have written such a plugin as part of the xine project. However, for that project, I didn’t have to write the full playback pipeline– my plugin was just handed some PCM data and drew some graphical data in response. Now I’m trying to write the entire engine in a standalone program and I’m wondering how to get it just right.

This is an SDL-based oscilloscope visualizer and audio player for Game Music Emu library. My approach is to have an audio buffer that holds a second of audio (44100 stereo 16-bit samples). The player updates the visualization at 30 frames per second. The o-scope is 512 pixels wide. So, at every 1/30th second interval, the player dips into the audio buffer at position ((frame_number % 30) * 44100 / 30) and takes the first 512 stereo frames for plotting on the graph.

It seems to be working okay, I guess. The only problem is that the A/V sync seems to be slightly misaligned. I am just wondering if this is the correct approach. Perhaps the player should be performing some slightly more complicated calculation over those (44100/30) audio frames during each update in order to obtain a more accurate graph? I described my process to an electrical engineer friend of mine and he insisted that I needed to apply something called hysteresis to the output or I would never get accurate A/V sync in this scenario.

Further, I know that some schools of thought on these matters require that the dots in those graphs be connected, that the scattered points simply won’t do. I guess it’s a stylistic choice.

Still, I think I have a reasonable, workable approach here. I might just be starting the visualization 1/30th of a second too late.

So, the projects are participating in the Google Summer of Code for the 2012 season. (While Libav is the project officially accepted to particular, I still refer to the projects because FFmpeg will also benefit).

Here are the students, projects, and mentors for this summer:

1. Andrew D’Addesio is working on an Opus Decoder, mentored by Justin Ruggles
2. Guillaume Martres is working on an HEVC video decoder, mentored by Mashiat Sarker Shakkhar
3. Jan Ekström is working on an LGPL Ut Video encoder, mentored by Kostya Shishkov
4. Jordi Ortiz is working to rewrite avserver, mentored by Luca Barbato
5. Samuel Pitoiset is working on an RTMP[E|S|T|TE] protocol implementation, mentored by Martin Storsjö

Wish them luck– these are some ambitious projects.

I downloaded a multimedia file a long time ago (at least, I strongly suspected it was a multimedia file which is why I downloaded it). It went by the name of ‘lamborghini_850kbps.vg2′. I have had it in my collection for at least 7 years. I couldn’t remember where I found it. I downloaded it before it occurred to me to take notes about this sort of stuff.

I found myself staring at the file again today and Googled the filename. This led me to a few Japanese sites which also contained working URLs for a few more .vg2 samples. Some other clues led me to a Russian language forum where someone had linked to a site that had Win32 codec modules that could process the files. The site was defunct but the Internet Archive Wayback Machine kept a copy for me, as well as copies of several more .vg2 samples from a defunct Japanese site previously involved with this codec.

Sometimes this internet technology works really well. But I digress.

Anyway, through all this, I finally found a clue: XVD. and wouldn’t you know, there is already a basic page on the MultimediaWiki describing the technology. In fact, while VG2 is a custom container, the MultimediaWiki states that the video component has a FourCC of VGMV, and there is already a file named VGMV.avi in the root V-codecs/ samples directory, something I vow to correct (that’s a big pet peeve of mine– putting samples in the root V-codecs/ or A-codecs/ directories).

XVD… XVD… XVD… why does that sound so familiar? Oh, of course; there is a company named XVD and they have an office in the Bay Area which I have passed on numerous occasions, like this morning:

<

Someone originally connected with the multimedia technology in question operates a website which contains an unofficial history of the XVD tech. At first, I was wondering if the technology was completely defunct (and should therefore be open sourced). But if XVD’s solutions page (dated 2010) is to be believed, the technology is still in service, and purported to be better than H.264 and VC-1: “The current generation of XVD video compression technology provides better video quality at any given data rate than standards-based codecs (H.264 or VC-1) with four times lower encoding complexity (when compared with H.264 Main Profile).”

If they say so. For my part, I’m just happy that I have finally figured out what this lamborghini_850kbps.vg2 is so that I can properly catalog it on the samples site, which I have now done, along with other samples and various codecs modules.

This episode reminds me that there’s a branch office of Zygo Corporation close to my home (though the headquarters are far, far away). The companies you see in Silicon Valley. Anyway, long-time open source multimedia hackers will no doubt recognize Zygo from the ZyGo FourCC & video codec transported in QuickTime files that was almost decode-able using an H.263 decoder.

I may never learn what Zygo’s core competency actually is, but I will always remember their multimedia tech every time I run past their office.

I received this 3-disc set of G.I. Joe CD-ROMs today:

Copyright 2003, and labeled as PC ONLY. Each disc claims to have 2 episodes. So are these some sort of video discs? Any gaming elements? I dove in to investigate.

So, it turns out that there are some games on these discs, done in Flash Player (which tells me that these were probably available on the web at some point). Here’s a shooting gallery game from the first disc:

As promised by the CD-ROM copy, the menu does grant access to 2 classic G.I. Joe episodes. Selecting either one launches this:

Powered by C-ezy? Am I interpreting that correctly? Anyway, the video player goes fullscreen and looks fine (given the source material). I can’t capture screenshots and controls are limited to: space for pause, ESC to exit player, and up/down to control volume. No seeking and certainly no onscreen controls. Pretty awful player.

Studying the first disc, I find a 550 MB file with the name 5859Hasbro.egm. Coupled with ep58.cfg and ep59.cfg files in the same directory, I gather that the disc has G.I. Joe episodes 58 and 59 (though the exact episodes, “There’s No Place Like Springfield” parts 1 and 2, are listed on Wikipedia as being episodes 154 and 155; but who’s counting?). The cfg files contain this text:

ep58.cfg:
EGM_GIJOE.exe
5859Hasbro.egm /noend /track:0 /singletrack 

ep59.cfg:
EGM_GIJOE.exe
5859Hasbro.egm /noend /track:1 /singletrack

The big EGM file starts with the string “Egenie Player”. After that, I see absolutely no clues. The supporting EGM_GIJOE.exe file has some interesting strings: “Decore Bits Per Pixel” (I know I have seen “Decore” used to mean “decoding core” in some libraries), “Egenie Player – %s, Version:%s”, “4th June 2002″, a list of common FourCC tags seen in AVI files, “Brought to you by Martin, Patrick Bob and Bren” (do you suppose “Patrick Bob” is one person’s name?), a list of command line options…

Aha! A URL: http:\\www.e-genie.tv (yep, backslashes, not forward slashes). e-genie.tv seems to redirect to mygenie.tv, which… doesn’t appear to be strictly related to video technology these days.

The projects (FFmpeg / Libav) just got a WMA lossless decoder. For those keeping score, this means that there are open source methods for decoding every single one of Microsoft’s proprietary audio codecs (Windows Media Audio, or WMA): WMA v1, WMA v2, WMA9/Pro, WMA Voice, and now WMA lossless. Currently, it’s only advertised to decode 16-bit audio (no 24-bit). Also, when I first tried it a few days ago, it didn’t decode the very end of the single sample file I concocted many years ago (luckynight.wma). But that might be cleared up by now.

Some other recent developments in the projects that I wanted to call out: An encoder for the Apple ProRes encoder from Kostya; XWD (X window dump) image decoding and encoding from Paul B. Mahol; a Sun rasterfile encoder from Aneesh Dogra.

And then there’s the new playback system for CDXL files, also courtesy of Paul B. Mahol. I wasn’t familiar with this format until I wrote this post, which is surprising, given the format’s vintage. This was a CD-ROM FMV format favored for Amiga computers. Here it is in all its 160x120x10fps glory:

That’s the amigaball.cdxl sample available in the repository. The sample is 3835910 bytes large and plays for about 24 seconds. This yields a data rate of about 159 kbytes/second. So, yeah, single-speed CD-ROM FMV.

I jump around between projects. A lot. Over the years, I have employed various methods for retaining state or context as I switch to a different project. Here’s a quick survey and a general classification of their effectiveness.

Good

• Evernote: This is a cloud-based note-taking service that has a web client, Mac and Windows clients, and clients for just about ever mobile platform out there. I have an account and access it via the web interface as as the Windows, iOS, and Android clients. I really like it.

Okay

• Series of text files: I have been doing this for a very long time. I have many little note-filled directories here and there that are consistently migrated to new machines but generally forgotten about. This isn’t a terrible method but can be unwieldy when you work on lots of different machines. I’m still tracking down all these directories and importing them into Evernote.

Bad

• Layout of desktop windows: I have a habit of working on one project in a set of windows on one desktop space and another project in a second set of windows in another space, etc. Oh, this makes me shudder just thinking about it, mostly because of living in constant fear of a power failure or some other inadvertent reset (darn you, default config’d Windows Update) that wipes the state clean (sure, all of the work might have been saved, but I was relying on those windows to be set up in just the right manner to remind me of all the things I was working on). These days, I force myself to reboot at least once a week so I can’t get too deep into this habit. When it’s time to change projects, I write up exactly what I was doing and where I left off and stick it in Evernote.
• Open browser windows: I guess it’s common to have many, many tabs open in one’s web browser in this day and age. Like many, I use open tabs as a stack of items to read. The state problem comes when a few of the open tabs represent TODO items. Then I start living in fear that the browser might crash or be restarted in an unexpected way and I struggle to recall what 3-5 important TODO items were that I had opened in separate tabs (on top of a stack of less important items). Again, I try to shut down the browser frequently in order to break this tendency. TODO items are better filed in Evernote.
• Unsaved data in a text editor: Okay, this is just sloppy on my part, shoving temporary data into a text editor window thinking it’s supremely ephemeral. The problem comes when it’s linked to one of the many tasks on my desktop that might be bumped down a few priority levels; when finally returning to the context-free data, I’m at a loss to explain what it’s for. Evernote gets it, once more, with a more thorough description of what was going on.
• Email inbox: I make an effort to ensure that my email inbox has the fewest number of messages possible. Once things are dealt with, they get filed away elsewhere. This implies that things in my inbox require action. Some things have a habit of hanging around, though. Longer items now get described in better detail and filed away in Evernote.
• Classic paper: Thanks to Derek in the comments for reminding me of this one. Paper is a reliable standby but it can get unwieldy when Post-It Notes litter your work area. Further, it can be problematic when you have multiple physical work areas.
• Shell history: Another method I rely on entirely too often. This is when I count on a recipe of command line incantations living on in the history buffer of my Unix shell (generally Bash). What sequence of git commands allowed me to do XYZ? Let’s check the shell history– I sure hope it’s still in there.

Conclusion
I guess what I’m trying to say here is that I really like Evernote. If you have similar troubles with retaining state, try it out. I hear there are many other services similar to it with slightly varying feature sets (people rave about Microsoft OneNote). So there are plenty of options and something out there is surely a fit.

Evernote has a free tier and a premium tier. For my meager note-taking needs, I don’t come anywhere close to the free tier’s limit but I decided to pay for a premium subscription simply because I feel like I derive so much value from the service.

One downside, however, is that I seem to be doing a lot less blogging since I got on Evernote earlier this year (though it is where I author most of these posts nowadays; I especially like that I have a notebook labeled “Posted” whose incrementing count reminds me that I am getting some stuff out there). I originally started this blog as a sort of technical journal in order to organize notes and projects in a central location. It’s strange to think that if Evernote existed in 2005, I might never have had a reason to start this blog.

I bought an Android tablet a few months ago. It is less expensive than the best tablets but no where near the bottom end of the market. I think it’s pretty good. However, one downside is that it’s not “certified” to use Google’s official marketplace. That would seem to be somewhat limiting, however…

Enter Amazon’s Android App Store
Amazon got into the business of selling Android Apps some time ago. I started experimenting with this on a Nexus One phone that Google gave me. When I installed the App Store on the Android tablet and logged in, I was pleasantly surprised to see all of my Amazon apps ready for downloading onto the tablet.

So I have an App Store for use with this Android tablet.

Anyway, the reason I bring this up is because I managed to screw up this tablet in an unusual and humorous manner. You might be wondering if an app downloaded from the Amazon App Store requires the App Store to be present in order to run. The answer is: Oh yeah! It works like this:

This means that if — perhaps out of curiosity, for example — you login to the Amazon App Store, download an app, install it, and then subsequently log out of the App Store or uninstall it altogether, the downloaded app will decline to run until you log back into the store.

Here’s the thing– I wanted to provide a minimal level of security for my Android tablet. At the very least, I wished to lock the Amazon App Store itself since Amazon is famously (and, let’s face it, understandably) reluctant to deliberately add any friction to their shopping processes. I.e., without any external protection app, the App Store app would allow anyone to purchase any app using my tablet.

So I purchased App Protector Pro from the Amazon App Store and it worked quite well. By default, it also password protects against modifying any system settings as well as installing new apps.

So, here’s where I screwed up: App Protector Pro was doing its faithful duty and I uninstalled the Amazon App Store as an experiment. Suddenly, no apps obtained from the App Store would work unless I reinstalled the App Store. Okay, fair enough, except for one thing– App Protector Pro wouldn’t run without the App Store. Well, it did, it started to, tried to, but then exited. So I couldn’t re-install the App Store:

Oops

I eventually learned how to perform a factory reset of the unit which solved the problem. And, as indicated earlier, all of my apps were available for me to re-download.

Modding, Cyanogen-style
Open source aficionados will likely point out that there are alternate firmware options which allow me to take control of my Android tablet in a free and open manner. Among these options is CyanogenMod. After I got stuck in the situation described above, I thought I would have to resort to such an option.

On the plus side, researching alternative firmware options is what taught me to boot the device into a recovery mode and ultimately restore to a factory default setting. But if you’ll allow me to indulge in a mini-rant regarding accessibility of open source software: I was more than a little frustrated in trying to understand what CyanogenMod could possibly offer me. Their homepage says it’s “an aftermarket firmware”. I’m not entirely sure what that means or how it can benefit me. Fortunately, they have a full feature list linked from the front page. They are, in order: Lockscreen gestures, phone goggles, OpenVPN, incognito mode, themes support, and DSP equalizer. I can’t say that any of those really add any value for me. I’d love to know if CyanogenMod supports Google Android Market and various other Google apps (such as maps and GMail). That’s a question that I can’t seem to find the answer to.

The themes feature opens another old wound for me. Back around 1999 when I was first getting into Linux in a serious way, I remember that themes were a big theme at the Linux User Groups I would attend. I also remember lots are online articles at the time that emphasized how highly customizable the Linux desktop was in comparison to Windows 9x. I was bothered for 2 reasons: First, I thought there were more pressing problems that needed to be addressed in Linux; and second, none of these customization options seemed particularly straightforward; many apparently required hours of compiling and tinkering.

Small digression. Anyway, back to CyanogenMod, I was glad to see that they prominently display a button in order to “View Video Tour”. Ah, internet video has us so spoiled these days. I was eager to see this aftermarket firmware in action to see what it could do for me. However, the link leads to… a forum post? The thread seems to discuss how it would be a cool idea if the community could put together a video tour. At this point, the investigation just seems bizarre. It feels like a bunch of kids doing their best to do things the grown-up way.

Okay, sorry, rant over. I try to stay positive these days. I’m sure the CyanogenMod folks are doing great, fun, and interesting work on their project. The problems they choose to solve might lack mainstream appeal, however.

Free iPad
Ultimately, I recently unloaded the little Android tablet because, well… when a free iPad comes your way, lower spec tablets feel a little silly to keep around. Yeah, it’s great to play around with. Though here’s one unsettling thing I noticed about Apple’s App Store. While browsing for worthwhile games to indulge in, I noticed that they had a section for “Top Grossing Games”. This was a separate list from the “Top Apps” charts. I found the list weird for 2 reasons: 1) Why do I care which games are raking in the most cash? How does this communicate value to me, personally? Seriously, why would I base a purchasing decision around which vendor has earned the most money?

Anyway, let’s move on to reason #2 this was scary: Most of the games in this list had a price of FREE. One of them was that Capcom Smurfs game that stirred up controversy some months ago because of kids making unsupervised in-app purchases of virtual smurfberries. I tend to think that a top-grossing, free to play game is probably one that heavily encourages in-app purchases. Strange how this emerging trend actually encourages me to seek out games from the “top paid” list vs. “top free”.

A little while ago, I got a big head over the fact that I owned and controlled the feared and revered MPlayer samples archive. This is the repository that retains more than a decade of multimedia samples.

Conflict
Where once there was one multimedia project (FFmpeg), there are now 2 (also Libav). There were various political and technical snafus regarding the previous infrastructure. I volunteered to take over hosting the vast samples archive (53 GB at the time) at samples.mplayerhq.hu (s.mphq for this post).

However, a brand new server is online at samples.libav.org (s.libav for this post).

Policies
The server at s.libav will be the authoritative samples repository going forward. Why does s.libav receive the honor? Mostly by virtue of having more advanced features. My simple (yet bandwidth-rich) web hosting plan does not provide for rsync or anonymous FTP services, both of which have traditionally been essential for the samples server. In the course of hosting s.mphq for the past few months, a few more discrepancies have come to light– apparently, the symlinks weren’t properly replicated. And perhaps most unusual is that if a directory contains a README file, it won’t be displayed in the directory listing (which frustrated me greatly when I couldn’t find this README file that I carefully and lovingly crafted years ago).

The s.mphq archive will continue to exist — nay, must exist — going forward since there are years’ worth of web links pointing into it. I’ll likely set up a mirroring script that periodically (daily) rsyncs from s.libav to my local machine and then uses lftp (the best facility I have available) to mirror the files up to s.mphq.

Also, since we’re starting fresh with a new upload directory, I think we need to be far more ruthless about policing its content. This means making sure that anything that is uploaded has an accompanying file which explains why it’s there and ideally links the sample to a bug report somewhere. No explanation = sample terminated.

RSS
I think it would be nifty to have an RSS feed that shows the latest samples to appear in the repository. I figure that I can use the Unix ‘find’ command on my local repository in concert with something like PyRSS2Gen to accomplish this goal.

Monetization
In the few months that I have been managing the repository, I have had numerous requests for permission to leech the entire collection in one recursive web-suck. These requests often from commercial organizations who wish to test their multimedia product on a large corpus of diverse samples. Personally, I believe the archive makes a rather poor corpus for such an endeavor, but so be it. Go ahead; hosting this archive barely makes a dent in my fairly low-end web hosting plan. However, at least one person indicated that it might be easier to mail a hard drive to me, have me copy it, and send it back.

This got me thinking about monetization opportunities. Perhaps, I should provide a service to send HDs filled with samples for the cost of the HD, shipping, and a small donation to the multimedia projects. I immediately realized that that is precisely the point at which the vast multimedia samples archive — with all of its media of questionable fair use status — would officially run afoul of copyright laws.

Which brings me to…

Clean Up
I think we need to clean up some samples, starting with the ones that were marked not-readable in the old repository. Apparently, some ‘samples’ were, e.g., full anime videos and were responsible for a large bandwidth burden when linked from various sources.

We multimedia nerds are a hoarding lot, never willing to throw anything away. This will probably the most challenging proposal to implement.

I played with the ‘file’ utility a long time ago because I wanted to make it recognize a large number of multimedia formats. I had trouble getting my changes to take. But I’m prepared to try again after many years.

Aiming at the Corpus
In my local mirror of the MPlayerHQ samples archive, I find 9853 unique files. So I run all of them through the ‘file’ command:

  'find /path/to/samples -type f -print0 | xargs -0 file --no-pad'

My Ubuntu installation has file v5.04. I also tested against 5.07 and the latest, 5.08. Here is the number of files each version was unable to identify (generically marking as ‘data’):

5.04  1521
5.07  1405
5.08  1501

That seems like a regression for v5.08 until I dug into the details and saw quite a few items like this, indicating that the MPEG detection could use some work:

-mov/mov-demux-infinite-loop.mpg: DOS-executable (
+mov/mov-demux-infinite-loop.mpg: data
-image-samples/UNeedQT4.pntg: DOS-executable (
+image-samples/UNeedQT4.pntg: data

Workflow
These are just notes to myself and perhaps anyone else who wants to add new file formats to be identified by the ‘file’ command.

First, download either the latest release from the FTP or clone from Github. Do the usual unpack, ‘./configure’, ‘make’ routine. To use this newly-built version and its associated magic file:

  ./src/file --magic-file magic/magic.mgc <file>

To add a new format for ID, first, run the foregoing command to ensure that it’s not already identified. Then, check over the files in magic/Magdir and see which one might pertain to what you’re doing (it’s unlikely that your format will merit a new file in this directory). For example, for this round, I modified animation, audio, iff, and riff. Add or modify existing specs based on the copious examples in the directory and by consulting the appropriate man page (‘man 5 magic’).

Finally, run ‘make’ again which will regenerate the magic file. Invoke the above command again to use the modified magic file.

Before and After
On a selection of formats taken from the samples archive (renamed and cut down to a kilobyte because detection typically only relies on the first few bytes), here is the “before”:

amv:            RIFF (little-endian) data
armovie:        data
bbc-dirac:      data
interplay-mve:  data
mtv:            data
nintendo-thp:   data
nullsoft-video: data
redcode:        data
sega-film:      data
smacker:        data
trueaudio:      data
vqa:            IFF data
wavpack:        data
wc3-mve:        IFF data
wtv:            data

And the “after”:

amv:            RIFF (little-endian) data, AMV
armovie:        ARMovie
bbc-dirac:      BBC Dirac Video
interplay-mve:  Interplay MVE Movie
mtv:            MTV Multimedia File
nintendo-thp:   Nintendo THP Multimedia
nullsoft-video: Nullsoft Video
redcode:        REDCode Video
sega-film:      Sega FILM/CPK Multimedia, 320 x 224
smacker:        RAD Game Tools Smacker Multimedia version 2, 320 x 200, 100 frames
trueaudio:      True Audio Lossless Audio
vqa:            IFF data, Westwood Studios VQA Multimedia, 418 video frames, 320 x 200
wavpack:        WavPack Lossless Audio
wc3-mve:        IFF data, Wing Commander III Video, PC version
wtv:            Windows Television DVR Media

After rerunning ‘file’ on the mphq corpus using the modified magic file, only 1329 files remain unidentified (down from 1501).

Going Forward
As mentioned, MPEG detection could probably be strengthened. However, a major weakness is QuickTime/MP4. Many files are not detected, probably owing to the many ways that QuickTime files can begin.

I have never claimed to be especially proficient at math. I did take Advanced Placement calculus in my senior year of high school. While digging through some boxes, I found an old grade report from that high school year. I wondered what motivated me to save it. Maybe it’s because it offered this clue as to why I can’t perform adequately in math class:

Mystery solved: I did not wear proper P.E. attire to calculus class.