Breaking Eggs And Making Omelettes

Topics On Multimedia Technology and Reverse Engineering


Archives:

Sequencing MIDI From A Chiptune

April 27th, 2013 by Multimedia Mike

The feature requests for my game music appreciation website project continue to pour in. Many of them take the form of “please add player support for system XYZ and the chiptune library to go with it.” Most of these requests are A) plausible, and B) in process. I have also received recommendations for UI improvements which I take under consideration. Then there are the numerous requests to port everything from Native Client to JavaScript so that it will work everywhere, even on mobile, a notion which might take a separate post to debunk entirely.

But here’s an interesting request about which I would like to speculate: Automatically convert a chiptune into a MIDI file. I immediately wanted to dismiss it as impossible or highly implausible. But, as is my habit, I started pondering the concept a little more carefully and decided that there’s an outside chance of getting some part of the idea to work.

Intro to MIDI
MIDI stands for Musical Instrument Digital Interface. It’s a standard musical interchange format and allows music instruments and computers to exchange musical information. The file interchange format bears the extension .mid and contains a sequence of numbers that translate into commands separated by time deltas. E.g.: turn key on (this note, this velocity); wait x ticks; turn key off; wait y ticks; etc. I’m vastly oversimplifying, as usual.

MIDI fascinated me back in the days of dialup internet and discrete sound cards (see also my write-up on the Gravis Ultrasound). Typical song-length MIDI files often ranged from a few kilobytes to a few 10s of kilobytes. They were significantly smaller than the MOD et al. family of tracker music formats mostly by virtue of the fact that MIDI files aren’t burdened by transporting digital audio samples.

I know I’m missing a lot of details. I haven’t dealt much with MIDI in the past… 15 years or so (ever since computer audio became a blur of MP3 and AAC audio). But I’m led to believe it’s still relevant. The individual who requested this feature expressed an interest in being able to import the sequenced data into any of the many music programs that can interpret .mid files.

The Pitch
To limit the scope, let’s focus on music that comes from the 8-bit Nintendo Entertainment System or the original Game Boy. The former features 2 square wave channels, a triangle wave, a noise channel, and a limited digital channel. The latter creates music via 2 square waves, a wave channel, and a noise channel. The roles that these various channels usually play typically break down as: square waves represent the primary melody, triangle wave is used to simulate a bass line, noise channel approximates a variety of percussive sounds, and the DPCM/wave channels are fairly free-form. They can have random game sound effects or, if they are to assist in the music, are often used for more authentic percussive sounds.

The various channels are controlled via an assortment of memory-mapped hardware registers. These registers are fed values such as frequency, volume, and duty cycle. My idea is to modify the music playback engine to track when various events occur. Whenever a channel is turned on or off, that corresponds to a MIDI key on or off event. If a channel is already playing but a new frequency is written, that would likely count as a note change, so log a key off event followed by a new key on event.

There is the major obstacle of what specific note is represented by a channel in a particular state. The MIDI standard defines 128 different notes spanning 11 octaves. Empirically, I wonder if I could create a table which maps the assorted frequencies to different MIDI notes?

I think this strategy would only work with the square and triangle waves. Noise and digital channels? I’m not prepared to tackle that challenge.

Prior Work?
I have to wonder if there is any existing work in this area. I’m certain that people have wanted to do this before; I wonder if anyone has succeeded?

Just like reverse engineering a binary program entails trying to obtain a higher level abstraction of a program from a very low level representation, this challenge feels like reverse engineering a piece of music as it is being performed and automatically expressing it in a higher level form.

Posted in Outlandish Brainstorms | 8 Comments »

8 Responses

  1. lockecole2 Says:

    Not quite the NES, but it appears that gbsplay (http://gbsplay.berlios.de/) has a module to write out a MIDI file.

  2. Multimedia Mike Says:

    @lockecole2: Thanks for the tip. Game Boy .gbs files are a target of interest.

  3. kvance Says:

    Linus Ã…kesson wrote a “theme finder” for the HVSC that works on similar principles: http://www.linusakesson.net/themes/how.php

  4. Reimar Says:

    You’ve talked about the JavaScript issue before but it sounded more like a performance question. I had the impression that asm.js might actually help a lot with that. The approach would at least in theory even allow supporting the use of SSE from JavaScript.
    Maybe I am hopelessly optimistic, but maybe there is a chance that Native Client is a dead end? Also, I heard that there’s some work on reworking Native Client to use LLVM IR instead so that the same code can run on e.g. ARM and x86. I haven’t paid much attention though.

  5. Multimedia Mike Says:

    @Reimar: I have looked at asm.js, based on claims that it can get the performance penalty down in the range of 3-5x of plain C, rather than the usual orders of magnitude slower. I’m not sure I believe the assertion of accessing SIMD predictably from JS.

    I don’t think NaCl is a lasting approach either. However, the work I’m doing with the NaCl port right now might be a springboard for porting to individual app architectures (trying to avoid JS here, obviously).

  6. Reimar Says:

    The asm.js approach should allow for very predicatable SIMD, though making it work for different architectures will be a challenge.
    A stupid implementation would e.g. detect when JS calls a function called vector_add, check if that vector_add function is the one from you SIMD asm.js and if so emit a SIMD vector add instruction during JIT instead.
    Of course there are a lot of details involved, but it doesn’t seem like huge issue – coming up with a cross-platform SIMD instruction set to use might be though.

  7. D Says:

    yo, speaking of Native Client, the GMA player hasn’t been working for me in the past week or so, giving an error of “Native Client is not allowed”. Unfortunately, I don’t know when exactly this error started happening, but I haven’t done or installed anything Chrome-related in the past few months except fail to stop Chrome from auto-updating. Is this likely a problem on my end somewhere?

  8. Multimedia Mike Says:

    @D: Indeed. This is apparently a regression (possibly a new feature) introduced in Chrome 27, which automatically rolled out a few weeks ago. I’m trying to work around the problem.

    To fix it temporarily, you have to go to chrome://flags and enable NaCl for all websites. This step should not be necessary, but that’s what I’m trying to fix.

    Sorry for the inconvenience.