Sometimes, when I don’t feel like doing anything else, I look at that Apple SMC video encoder again.

8-bit Encoding
When I last worked on the encoder, I couldn’t get the 8-color mode working correctly, even though the similar 2- and 4-color modes were working fine. I chalked the problem up to the extreme weirdness in the packing method unique to the 8-color mode. Remarkably, I had that logic correct the first time around. The real problem turned out to be with the 8-color cache and it was due to the vagaries of 64-bit math in C. Bit shifting an unsigned 8-bit quantity implicitly results in a signed 32-bit quantity, or so I discovered.

Anyway, the 8-color encoding works correctly, thus shaving a few more bytes off the encoding size.

Encoding Scheme Oddities
The next step is to encode runs of data. This is where I noticed some algorithmic oddities in the scheme that I never really noticed before. There are 1-, 2-, 4-, 8-, and 16-color modes. Each mode allows encoding from 1-256 blocks of that same encoding. For example, the byte sequence:

  0x62 0x45

Specifies that the next 3 4×4 blocks are encoded with single-color mode (of byte 0×62, high nibble is encoding mode and low nibble is count-1 blocks) and the palette color to be used is 0×45. Further, opcode 0×70 is the same except the following byte allows for specifying more than 16 (i.e., up to 256) blocks shall be encoded in the same matter. In light of this repeat functionality being built into the rendering opcodes, I’m puzzled by the existence of the repeat block opcodes. There are opcodes to repeat the prior block up to 256 times, and there are opcodes to repeat the prior pair of blocks up to 256 times.

So my quandary is: What would the repeat opcodes be used for? I hacked the FFmpeg / Libav SMC decoder to output a histogram of which opcodes are used. The repeat pair opcodes are never seen. However, the single-repeat opcodes are used a few times.

Puzzle Solved?
I’m glad I wrote this post. Just as I was about to hit “Publish”, I think I figured it out. I haven’t mentioned the skip opcodes yet– there are opcodes that specify that 1-256 4×4 blocks are unchanged from the previous frame. Conceivably, a block could be unchanged from the previous frame and then repeated 1-256 times from there.

That’s something I hadn’t thought of up to this point for my proposed algorithm and will require a little more work.

Further reading

• My first entry regarding my toy SMC encoder

How do you take a 24-bit RGB image and convert it to an 8-bit paletted image for the purpose of compression using a codec that requires 8-bit input images? Seems simple enough and that’s what I’m tackling in this post.

Ask FFmpeg/Libav To Do It
Ideally, FFmpeg / Libav should be able to handle this automatically. Indeed, FFmpeg used to be able to, at least at the time I wrote this post about ZMBV and was unhappy with FFmpeg’s default results. Somewhere along the line, FFmpeg and Libav lost the ability to do this. I suspect it got removed during some swscale refactoring.

Still, there’s no telling if the old system would have computed palettes correctly for QuickTime files.

Distance Approach
When I started writing my SMC video encoder, I needed to convert RGB (from PNG files) to PAL8 colorspace. The path of least resistance was to match the pixels in the input image to the default 256-color palette that QuickTime assumes (and is hardcoded into FFmpeg/Libav).

How to perform the matching? Find the palette entry that is closest to a given input pixel, where “closest” is the minimum distance as computed by the usual distance formula (square root of the sum of the squares of the diffs of all the components).

That means for each pixel in an image, check the pixel against 256 palette entries (early termination is possible if an acceptable threshold is met). As you might imagine, this can be a bit time-consuming. I wondered about a faster approach…

Lookup Table
Read the rest of this entry »

WARNING: This post is a bunch of nostalgia. Feel free to follow along if you recall the DOS days of the early-mid 1990s.

Also, if you’re interested in the Gravis Ultrasound Experience CD-ROM, you can download the compressed ISO image here (137 megabytes).

I finally let go of my Gravis Ultrasound MAX sound card a little while ago. It felt like the end of an era for me, even though I had scarcely used the card in recent memory.

The Beginning
What is the Gravis Ultrasound? Only the finest PC sound card from the classic DOS days. Back in the day (very early 1990s), most consumer PC sound cards were Yamaha OPL FM synthesizers paired with a basic digital to analog converter (DAC). Gravis, a company known for game controllers, dared to break with the dominant paradigm of Sound Blaster clones and create a sound card that had 32 digital channels.
Read the rest of this entry »

Remember Intel’s custom flavor of H.263 cleverly named I.263? I think I have finally found an application that used it thanks to a recent thrift shop raid– Intel Video Phone:

The root directory of the disc has 2 copies of an intro.avi video. One copy uses Intel Indeo 3 video and PCM audio. The other uses I.263 video and an undetermined (presumably Intel-proprietary) audio codec — RIFF id 0×0402 at a bitrate of 88 kbits/sec for stereo, 22 kHz audio. The latter video looks awful but is significantly smaller (like 4 MB vs. 25 MB).

This is the disc marked as “Send it to a friend…”. Here’s the way this concept was supposed to operate:

• You buy an Intel Video Phone Camera Pack (forgotten page courtesy of the Internet Archive) which includes a camera and 2 CDs.
• You install the camera and video phone software on your computer.
• You send the other CD to the person whom you want to be able to see your face when you’re teleconferencing with them.
• The other party installs the software.
• The 2 of you may make an internet phone call presumably using commodity PC microphones for the voice component; the person who doesn’t have a camera is able to see the person who does have a camera.
• In a cunning viral/network marketing strategy, Intel encourages the other party to buy the physical hardware as well so that they may broadcast their own visage back to the other person.

If you need further explanation, the intro lady does a great job:

I suspect I.263 was the video codec driving this since Indeo 3 would probably be inappropriate for real time video applications due to its vector quantizing algorithm.

So I’m watching a fairly recent music video for a song named “XXXO” from an artist named M.I.A. when I’m suddenly assaulted by this imagery:

… and I enter nervous convulsions. You see, while this might seem to be an odd video effect to the casual viewer, to a multimedia hacker, it appears to be deliberately antagonistic. To anyone who has written a video codec, this scene looks like an entire casserole of video bugs, combining creeping plane offsets errors, chroma problems, and interlacing havoc. The craziest part is to realize that this is probably some kind of standard video effect / filter type. Upon a repeat viewing, I realized that the entire video sort of looks like an amateur video editor’s first week using video software.

Elsewhere in the video, a YouTube-style video frame vortex highlights the proceedings. I guess I need to come to terms with the fact that the ubiquitous player frame is just part of the digital Zeitgeist now:

Vintage Video Strangeness
I’m a long-time music video junkie but I have a tendency of examining them entirely too closely. I first saw Paula Abdul’s video for “Cold-Hearted” when I was just starting to understand multimedia technology and how it interacted with emerging home computers. Imagine how confused I was when I tried to make sense of the actions performed by our eMaestro “Chuck” whom Paula has instructed to “hit it”. First, he hits a key followed by 3 quick strikes on a second key:

Then, the “start music” action is apparently bound to a particular key on the electronic keyboard:
Read the rest of this entry »

After I posted about my initial encounter and frustration with Google’s Native Client (NaCl) SDK and took a deep breath, I realized that I achieved an important proof of concept– I successfully played music using the NaCl SDK audio output interface. Then I started taking a closer read through the (C-based set of) header files and realized I might be able to make a go of it after all. I had much better luck this time and managed to create a proper Native Client interface that allows for controlling playback, presenting metadata, and toggling individual voices (a fascinating tool for studying classic game music).

I haven’t bothered to post the actual plugin because, really, what’s the point? I started with NaCl SDK 0.3 which requires Chrome 12, which means terribly limited reach, even among Chrome users. At least, that was true when I restarted this little project. Chrome 12 was formally released this past week. Chrome development really does move at breakneck pace.

Anyway, here is a static screenshot of what the plugin currently looks like:

Not pretty, but it does the job.

Dev Journal
Various notes based on this outing:

• Portability: I tested my plugin using Chrome 12 on 64-bit Windows, Mac, and Linux. Mac and Linux both work; Windows does not.
• Build System: SDK 0.3 is still lacking in its ability to compile .cpp files (instead of .cc files); necessary because libgme is C++ using .cpp files. This requires some build system modification.
• Getting the interfaces: This is where I got tripped up the first time around. get_browser_interface() from their example actually refers to a parameter passed in through the PPP_InitializeModule() function. The SDK’s template generator renames this to get_browser().
• Debugging: I feel unstoppable once I have a printf() mechanism available to me during development. To that end, console.log() from JavaScript outputs to Chrome’s built-in JavaScript console log while putting printf() statements in the actual NaCl plugin causes the messages to show up in ~/.xsession-errors on Linux/X.
• Size Matters: The binaries generated with the NaCl 0.3 SDK are ridiculously huge. The basic “Hello World” example in C compiles to binaries that are 6.7 MB and 7.8 MB for the 32- and 64-bit builds, respectively. This made me apprehensive to build a full version of SaltyGME that contains all the bells and whistles offered by the library. However, all of the GME code compiled into the binary adds very little size. Curiously, the C++ version of “Hello World” only ranges from 1.8-2.0 MB for 32- and 64-bit. Is there some kind of C tax happening here? Note that running ‘strip’ on the resulting .nexe files (they’re ELF files, after all) brings the sizes down into the C++ range, but at the cost of causing them to not work (more specifically, not even load).
• No Messaging: The NaCl SDK is supposed to have a messaging interface which allows the NaCl plugin to send asynchronous messages up to the hosting page. When I try to instantiate it, I get a NULL. I’m stuck with the alternative of polling from the JavaScript side to, e.g., determine when a song has finished loading via the network.

That’s all I can think of for now. I may work on this a little more (I’d like to at least see some audio visualization). Maybe Google will enable NaCl per default sometime around Chrome 21 and this program will be ready for prime time by then.

See Also:

• My first attempt at this same project

Have you heard of Google’s Native Client (NaCl) project? Probably not. Basically, it allows native code modules to run inside a browser (where ‘browser’ is defined pretty narrowly as ‘Google Chrome’ in this case). Programs are sandboxed so they aren’t a security menace (or so the whitepapers claim) but are allowed to access a variety of APIs including video and audio. The latter API is significant because sound tends to be forgotten in all the hullabaloo surrounding non-Flash web technologies. At any rate, enjoy NaCl while you can because I suspect it won’t be around much longer.

After my recent work upgrading some old music synthesis programs to use more modern audio APIs, I got the idea to try porting the same code to run under NaCl in Chrome (first Nosefart, then Game Music Emu/GME). In this exercise, I met with very limited success. This blog post documents some of the pitfalls in my excursion.

Infrastructure
People who know me know that I’m rather partial — to put it gently — to straight-up C vs. C++. The NaCl SDK is heavily skewed towards C++. However, it does provide a Python tool called init_project.py which can create the skeleton of a project and can do so in C with the '-c' option:

./init_project.py -c -n saltynosefart

This generates something that can be built using a simple ‘make’. When I added Nosefart’s C files, I learned that the project Makefile has places for project-necessary CFLAGS but does not honor them. The problem is that the generated Makefile includes a broader system Makefile that overrides the CFLAGS in the project Makefile. Going into the system Makefile and changing "CFLAGS =" -> "CFLAGS +=" solves this problem.

Still, maybe I’m the first person to attempt building something in Native Client so I’m the first person to notice this?

Basic Playback
At least the process to create an audio-enabled NaCl app is well-documented. Too bad it doesn’t seem to compile as advertised. According to my notes on the matter, I filled in PPP_InitializeModule() with the appropriate boilerplate as outlined in the docs but got a linker error concerning get_browser_interface().

Plan B: C++
Obviously, the straight C stuff is very much a second-class citizen in this NaCl setup. Fortunately, there is already that fully functional tone generator example program in the limited samples suite. Plan B is to copy that project and edit it until it accepts Nosefart/GME audio instead of a sine wave.

The build system assumes all C++ files should have .cc extensions. I have to make some fixes so that it will accept .cpp files (either that, or rename all .cpp to .cc, but that’s not very clean).

Making Noise
You’ll be happy to know that I did successfully swap out the tone generator for either Nosefart or GME. Nosefart has a slightly fickle API that requires revving the emulator frame by frame and generating a certain number of audio samples. GME’s API is much easier to work with in this situation — just tell it how many samples it needs to generate and give it a pointer to a buffer. I played NES and SNES music play through this ad-hoc browser plugin, and I’m confident all the other supported formats would have worked if I went through the bother of converting the music data files into C headers to be included in the NaCl executable binaries (dynamically loading data via the network promised to be a far more challenging prospect reserved for phase 3 of the project).

Portable?
I wouldn’t say so. I developed it on Linux and things ran fine there. I tried to run the same binaries on the Windows version of Chrome to no avail. It looks like it wasn’t even loading the .nexe files (NaCl executables).

Thinking About The (Lack Of A) Future
As I was working on this project, I noticed that the online NaCl documentation materialized explicit banners warning that my NaCl binaries compiled for Chrome 11 won’t work for Chrome 12 and that I need to code to the newly-released 0.3 SDK version. Not a fuzzy feeling. I also don’t feel good that I’m working from examples using bleeding edge APIs that feature deprecation as part of their naming convention, e.g., pp::deprecated::ScriptableObject().

Ever-changing API + minimal API documentation + API that only works in one browser brand + requiring end user to explicitly enable feature = … well, that’s why I didn’t bother to release any showcase pertaining to this little experiment. Would have been neat, but I strongly suspect that this is yet another one of those APIs that Google decides to deprecate soon.

See Also:

• I eventually got a second wind and made this work using the C interface.

Google's Chrome browser has made me phenomenally lazy. I don't even attempt to type proper, complete URLs into the address bar anymore. I just type something vaguely related to the address and let the search engine take over. I saw something weird when I used this method to visit Archive Team's site:

There's greater detail when you elect to view more results from the site:

As the administrator of a MediaWiki installation like the one that archiveteam.org runs on, I was a little worried that they might have a spam problem. However, clicking through to any of those out-of-place pages does not indicate anything related to pharmaceuticals. Viewing source also reveals nothing amiss.

I quickly deduced that this is a textbook example of website cloaking. This is when a website reports different content to a search engine than it reports to normal web browsers (humans, presumably). General pseudocode:

You can verify this for yourself using the wget command line utility:


$ wget --quiet --user-agent="Mozilla/5.0" \
http://www.archiveteam.org/index.php?title=Geocities -O - | grep \<title\>
<title>GeoCities - Archiveteam</title>

$ wget --quiet --user-agent="Googlebot/2.1" \
http://www.archiveteam.org/index.php?title=Geocities -O - | grep \<title\>
<title>Cheap xanax | Online Drug Store, Big Discounts</title>


I guess the little web prank worked because the phaux-pharma stuff got indexed. It makes we wonder if there's a MediaWiki plugin that does this automatically.

For extra fun, here's a site called the CloakingDetector which purports to be able to detect whether a page employs cloaking. This is just one humble observer's opinion, but I don't think the site works too well:
Read the rest of this entry »

I recently learned that iD -- you know, the famed game company -- has a series of summer tech camps:

All I care to know is: Where were programs like these when I was 7-18 years old? Born too early, I tell ya.

That reminds me that the Google Summer of Code, 2011 Edition is getting underway soon. I guess it's like summertime computer camp for college-aged students. FFmpeg / libav is once again a part of the program with 10 slots awarded by Google. Here are the students, projects, and mentors.

Wish them luck.