Back in the glory days of the old 8-bit Nintendo Entertainment System, I used to read my Nintendo Power magazine issues religiously. One issue that stood out to me, and that I never forgot about, is one that briefly mentioned Treasure Master (volume 26, July 1991, Robin Hood: Prince of Thieves issue).

This will be the first NES game where you can actually win prizes by playing it to the finish! The Game Pak will be released in the fall, and everyone will have time to master it. When you finish the game, you’ll reach a point where you can enter a speical password. In February of 1992, American Softworks will reveal the password that will allow you to play a secret portion of the game. The first person to finish the secret level and call in will win a $25,000 bond. Many of those who finish later will still be eligible for prizes– the sooner you finish, the better the prize. As you can imagine, the code is top secret, and the password system has been proven uncrackable by MIT and the CIA. (emphasis added)

click for larger image

So the gimmick for this substandard, side-scrolling, run & jump game was that players would buy it, play it, and build up skill in a few months. Then, after everyone has had a chance to get really good at the game, the company would release a password that would unlock a secret stage. Play through this stage and you could win big prizes.

Naturally, the part about the uncrackable code caught my attention. I didn’t even realize the CIA provided such a vetting service. Anyway, I didn’t hear anything else about the game again. Many years passed and I saw the game in a used video game shop. I figured it was time to revisit the uncrackable code issue since I know a little more about computers and codes now. And I also know better than to accept an uncrackable code claim at face value.

How uncrackable is the code? The introductory blurb from Nintendo Power raises some questions, many of which can be answered by the Treasure Master instruction manual. The full details of the contest are provided. Apparently, the password was to be announced on MTV on April 11, 1992, probably during a commercial. Players would enter the password, along with their game cartridge’s unique serial number, in order to unlock the special levels of the game. Upon completion of the levels, the game would return a special winning number that the player could give to the operator on the other end of a 1-900 phone number to try to win a prize.

To be fair, the official manual doesn’t mention the uncrackable code nonsense. The James Bond material highlighted in the Nintendo Power preview can probably be attributed to some Marketing/PR flunkie at American Softworks working more closely with the magazine than their own engineering department.

Also, Nintendo really wants you to know that they have nothing to do with the contest:

Threat Model

What does the game need to protect against? Presumably, the game needs to prevent a determined party from playing the levels before the appointed time, and it certainly needs to keep the prize-winning code secret. A determined attacker would need to have the tools and knowledge to dump out all of the ROM data, analyze it, and perhaps modify it and run the changes. This wasn’t exactly common knowledge back in 1991, but it certainly could have been done.

One idea I had was that the unlock code to be released would be a symmetric cipher key that would decrypt the level data. However, that would only be plausible if Treasure Master were the type of NES cartridge in which all of the game’s graphic (CHR) data was stored in the code (PRG) space and shuttled the graphical data to CHR RAM on demand. According to the FCE Ultra emulator, these are the characteristics of the cartridge hardware:

 PRG ROM:    8 x 16KiB
 CHR ROM:   16 x  8KiB
 Mapper:  1
 Mirroring: Horizontal

The important point here is that the game had CHR ROMs. Still, it’s possible that the key could be used to decrypt some data from the PRG side in the NES’ 2K of work RAM, data that is necessary to present the final level. This data includes tiling information for the background layout and sprite arrangement.

Password Strength

So there are 2 inputs when playing for the grand prize: A serial number and a password. The serial number is 8 digits long and can include 0-9. The password is 24 characters long and can include 32 different characters, including 0-9, exclamation point (!), and any of the 21 English language consonants. Since each character represents 5 bits of information (2⁵ = 32), 24 * 5 = 120 bit password. That should have easily been a long enough key length to thwart even the most determined Nintendo nerd from brute forcing the password space, at least by the standards of 1991′s consumer computing strength.

Other Threats

The game’s manual warns against using game-altering devices as it might cause the game to output a bad number. The #1 game-altering device for the NES was the Galoob Game Genie.

The password entry screen also warns gravely that a misentered code can result in disqualification. I took this to mean that the game might have a battery backup and only allows players to try a certain number of codes, perhaps in a certain time frame. This could lead to all kinds of thorny technical and usability issues, which don’t arise because the game doesn’t have a battery.

Best Guess

My hypothesis is that the secret password is the public half of a public/private key pair. One application for this key could be to decrypt a block of data needed for displaying the final level, a block of data that was encrypted with the analogous secret key. When the end of the secret level is reached, the game uses the public key to encrypt the serial number and displays the resulting ciphertext to the user (encoded at 5 bits/character, just like the original password). This ciphertext is telephoned into the 1-900 winners’ hotline where the operator has a computer application that was built to decrypt the ciphertext with the private key. This yields the game’s serial number. This is then checked against the official database since participants were required to register their serial numbers in order to be eligible for the contest.

The secret password is known to be 3HDJL9DNQV2WYTV4S91RXR86. Wikipedia has a record of what the prizes were but does not discuss any other outcome from the contest. Not just any serial number will suffice, unfortunately, so there must be a validity pattern for the serial number, sort of like credit card numbers. I don’t know of any valid serial numbers, and it seems likely that the only way to find one would be to reverse engineer the game.

Using the model above would not prove completely uncrackable, per the game’s constraints. If an attacker figured out the model and knew that the serial number was just going to be encrypted using the secret key as a password run through, e.g., the RSA algorithm, he could write a program that just simulated that behavior as soon as the public key was announced. However, perhaps the game could have a secret hash algorithm that was encrypted along with the secret data which would hash the serial number and then encrypt it. Actually, at that point, the encryption with the public key might be superfluous; just call the winning phone number and tell them your game serial number along with the special number (serial number computed with secret hash algorithm) and the operator’s program could verify the number based on that information alone.

And now that I put it like that, I realize that the public key idea probably isn’t necessary. Protect the secret game data on the cartridge with a symmetric key algorithm. Many such algorithms would be much faster and lighter-weight than a public key algorithm.

Thanks for reading through all of that. I have actually been sitting on this article for many years, waiting for an opportunity to flesh it out, though I am not sure if I will ever have time to get down to reverse engineering the game’s program.

There is a Windows program based on — what else? — FFmpeg called Wii Video 9 that handily converts anything into the accepted multimedia format. From the Sony PlayStation 3 and PSP to the Apple iPod and now the Nintendo Wii, the trend is for consumer electronics devices to be able to play user videos. But they usually support more advanced formats like MPEG-4 and H.264. I’m guessing that part of the Wii’s comparably low price is directly attributable to lower multimedia technology licensing costs.

I don’t know why I find it so humorous to read the Wii Video 9 forum posts, such as “Why are the converted files so HUGE?” I think we all need to respect that the deeper aspects of multimedia technology are not second nature to the general populace. One day, this stuff may truly be transparent and ubiquitous. Until then, Monsieur Scandragon has a point when he rants, “It doesn’t matter how bad it is at playing compressed video.. 10 seconds at BAD quality should NEVER be 25MB!!”

I have been too lazy to try the program myself. I wonder if it’s normal MJPEG data, or if it is unescaped data like that seen in the THP format common on the Nintendo GameCube. I also wonder about the audio format. Presumably, if they were too cheap to license MPEG-4 or H.264 video codecs, they’re not going to license MP3 or AAC either. I am guessing either straight PCM or their own custom ADPCM format.

• THP files seem to be the most prevalent
• H4M files have the markings of a hierarchical vector quantizer (like the letters ‘HVQ’); HVQ is also used in Sorenson Video 1 and LucasArts video
• Electronic Arts games have a VID format which has similar markings as a number of their other evolved game multimedia formats
• One game has 11,500+ Ogg Vorbis files; some of them appear to correspond via base filename to files with the extension .cib; I tend to think that these are corresponding video files
• Sofdec files, as commonly seen on Sega Dreamcast games
• DSP files, often paired as L and R files for stereo effect
• occasional BIK files

Plus, there are a number of other audio files, probably encoded with a custom ADPCM format common on GameCube games.

Unsurprisingly, very few technical details are available. The technology page brags about how tweakable the parameters are, but mentions nothing about coding algorithms. Still, it’s interesting that these A/V codecs were designed and optimized for the Gameboy Advance.

Related Post:

• Survey Of GBA Coding Methods

It’s weird to think that I was building an elaborate search mechanism to index approximately 760 items (the total number of games in the database), though I had hoped to expand the database into something MobyGames-like in due time. Keep in mind that at the outset, while the data was managed in a MySQL database, the data was exposed online through a series of static HTML pages; this was 1998 and it was a little difficult and expensive to get database hosting on the internet. Thus, I could not directly query a database, and I had limited CGI scripting facilities.

Introduction

I present the approach that I took in developing a simple search engine for the Internet NES Database as well as the thinking and reasoning behind the design decisions. It’s not highly sophisticated, but if you don’t know anything about search engine operation, this should give you a starting point.

When I started developing my video game database in early 1998 I knew that it would be much more usable if it featured a search engine. I had no idea how to write a search engine. Web searches on topics such as “search engine theory” turned up thousands of pages claiming to reveal the secrets of getting high ranks in popular search engines. I realized that I was on my own. So I sat down and thought really hard about how to write a search engine.

Analysis

The first thing I did was identify exactly what it was that I expected my search engine to do. In other words, I needed to anticipate how my site’s visitors would be using my search engine. I anticipated that they would first think to input the title of their favorite NES game, such as “The Legend of Zelda”. Therefore, if a user inputs “The Legend of Zelda” into the search engine, it needs to be able to find the page with that game.

Of course, if the user had to input the exact string “The Legend of Zelda” with all of the spacing and capitalization shown, the search engine would be of very limited use. A user might forgo capital letters and input “the legend of zelda”. He might omit “the” and simply input “legend of zelda”. He may be in a big hurry and also not know where the CAPS LOCK key is and input “ZELDA”. The search engine still needs to be able to associate those search strings with the title “The Legend of Zelda”.

It became immediately obvious that the search operation needed to be case-insensitive so that “zelda” and “ZELDA” are treated the same. I also needed to associate a lot of title permutations with the title. As shown in the above example, “zelda”, “legend of zelda”, and “the legend of zelda” all needed to be associated with “The Legend of Zelda”. I decided to take this idea to its logical extreme and associate all of these strings with the title:

• the
• legend
• of
• zelda
• the legend
• legend of
• of zelda
• the legend of
• legend of zelda
• the legend of zelda

Do you see the pattern? Many years after creating the search engine, I learned that this is called a powerset. If you have a set of four symbols, { a, b, c, d } then the powerset is:

• a
• b
• c
• d
• ab
• bc
• cd
• abc
• bcd
• abcd

If I had known that I was computing something called a powerset I might have been able to find an algorithm to do it. As it was, I spent a long time getting it right.

Examination of the powerset for “The Legend of Zelda” reveals that some of the strings, like “the” and “of”, are pretty useless and can be eliminated. The remaining strings need to be associated with “The Legend of Zelda”. How do we do this? The verb “associated” indicates that this would be a good job for associative arrays, a.k.a. hash arrays, hash tables, database hashes, hashes, etc. We’ll call them hashes in this document.

A hash is a collection of associated key/value pairs. For example, we could have a hash that maps (associates) fruits with their usual colors. An abstract representation would look like this:

If you query this FruitColor hash with the key “banana”, you will get the value “yellow”. If you query with the key “grape”, you will get the value “purple”, and so on.

Now let’s apply the hash concept to the search engine. We’ll map each string in the title powerset (minus words like “the” and “of”) to the title:

If you query this hash with the key “legend of zelda” or just “zelda”, you will get the value “The Legend of Zelda”.

So far, so good. I decided to try another title, such as “Zelda II: The Adventure of Link”. I didn’t like the look of that colon character in there. I decided that in addition to not dealing with uppercase characters, the search engine would also not deal with anything except lowercase characters and numbers. Therefore, the colon character will be stripped out.

Now we lowercase the title, strip out non-alphanumeric characters (including spaces), compute the powerset and map the powerset strings to the title using a hash. The result looks like this:

You get the idea and I don’t feel like typing out the rest of the permutations. That’s a computer’s job. Notice that “zelda” occurs again. It occurred when we were operating on “The Legend of Zelda”. Therefore, the string “zelda” needs to map onto both of the titles, “The Legend of Zelda” and “Zelda II: The Adventure of Link”:

Implementation

Then I moved on to the actual implementation. There is a variety of standards and libraries for database hash files on UNIX platforms. I don’t know much about the differences/advantages/disadvantages of one compared to another. Currently, I’m using GNU gdbm. I wrote a Perl script which interfaced to the MySQL database and collected all of the titles and game IDs. I used the IDs because the games are referenced in the database by their IDs rather than their actual titles. For each title string, the Perl script lowercases the string, strips out the non-alphanumeric characters, computes the powerset and maps the significant resulting strings (tosses out “the”, “a”, “and”, “of”, etc.) to the game ID.

The original search.cgi script, which was the basis for the original database search engine, was written in Perl. It simply took the user’s input string, lowercased the string, stripped out non-alphanumeric characters and used it as a key in the database hash file. If the key wasn’t mapped to any values, the search script reported that the search was unsuccessful. If the key was mapped to one game ID, the CGI script used an HTTP “Location:” header to redirect the web browser directly to the relevant game page. Finally, if the key returned multiple game IDs, the IDs were used as keys into another database hash file which mapped game IDs to game titles. Then the search engine would list the titles returned from the search.

Refinements

The search.cgi script also included a facility to log all of the search queries, as well as their success rates, to a file. This was done so I could analyze search queries that database users were actually attempting and identify weaknesses in the search scheme.

One problem that I noticed right away was the number of variations of the word ‘and’ used in game titles. The word could be represented as an ampersand (&) or the letter ‘n’, possibly with an apostrophe before, after, or before and after. For example:

• Crash ‘n’ The Boys Street Challenge
• A Boy & His Blob
• Tombs And Treasure
• Disney’s Chip ‘n Dale Rescue Rangers
• Fire N Ice

In the original search scheme, a query of “crash and the boys” would return no results because the search engine database hash contained the string “crashntheboys”, but not “crashandtheboys”. I needed a standard “and”. Therefore, before stripping out non-alphanumeric characters, the Perl script that builds the hash file first replaces occurrences of ” & “, ” n “, ” ‘n “, ” n’ “, and ” ‘n’ ” with the string ” and ” (note that the spaces before and after those strings are important). Similarly, when the search.cgi script prepares a user’s query for the hash, it performs the same replacement.

Examining the log of search queries revealed a huge problem regarding sequel games where the sequel number is denoted as a Roman numeral, e.g., Castlevania II: Simon’s Quest and Castlevania III: Dracula’s Curse. Users would enter the search string “Castlevania 3″. The search engine hash database didn’t contain a string “castlevania3″; it does contain the string “castlevaniaiii”. Therefore, the search engine reported that there were no search results. I felt this was very poor search engine behavior since it’s very unambiguous (from a human perspective) what the user was looking for. I solved this problem by converting all numbers to Roman numerals in both the hash database and the search queries. Therefore, Super Mario Bros. 3 is represented in the hash file as “supermariobrosiii”. The game “1942″ is not internally represented as “1942″, but as the Roman numeral “mcmxlii”. When a user inputs the search query “Castlevania 3″, the search engine converts the string into “castlevaniaiii” and queries the hash. The proper page could now be fetched.

Finally, it became clear that although the strict searching mechanism was effective, the search engine needed to have a looser searching mechanism as well. I came to refer to these concepts as the primary and secondary searches. The primary search takes a user’s search query, lowercases it, converts numbers to Roman numerals, strips out all non-lowercase alphabetic characters and uses the resulting string to query for game IDs in the hash file. If that returns 0 or more than 1 results, the search engine attempts a secondary search which consists of lowercasing the search query, converting numbers to Roman numerals and stripping out all non-lowercase alphabetic and non-space characters. Then, the search engine splits the query up by the spaces and queries the hash for each individual word. Then it returns a list of unique results (if there were any). As an example, a confused visitor might query for “Mega Ninja Gaiden” which would fail the primary search. However, the secondary search would return all of the games with “mega” in the title, as well as games with “ninja”, and games with “gaiden”. As a more realistic example, it wouldn’t be uncommon for a visitor to search for “super mario 3″ which is a common conversational way of abbreviating Super Mario Bros. 3. Again, even though it’s clear from a human’s perspective what the user is looking for, the primary search would fail miserably. But the secondary search would find the right title (and a number of other titles) if it queried the hash for “super”, then “mario”, then “iii” (remember that “3″ will be converted to a Roman numeral).

The Current Search Engine

The latest iteration of the database, online as of January, 2001, contains the capability to search through game keywords and creators as well as titles. There are two hashes for each category: The first hash maps permuted strings of the mangled powerset to IDs and the second maps IDs back to full names or titles. A Perl script named makedb.pl periodically generates all of the database hash files from the MySQL database. The search engine consists of a PHP script named search.php.

Conclusions

I realize that this is not the best search engine design. I haven’t studied the specifics of other search techniques (mostly because when I started my own search engine, there were no examples to study). However, it’s important to recognize that this search scheme works for me and my meager database and I haven’t yet needed to come up with a better scheme. I analyzed the specific problem that I had and developed an adequate solution. Maybe a similar solution will work for you too.

Related posts:

• The Data of Zelda
• The Format of Zelda

For example, when mounting the first disc image of one of my few GC games that I have actually completed, Metal Gear Solid:

$ ls -al mount/
total 1
dr-xr-xr-x 4 melanson users         0 Jul 15  2005 .
drwxr-xr-x 7 melanson users       760 Aug 26 21:48 ..
-r--r--r-- 1 melanson users        95 Jul 15  2005 .metadata
dr-xr-xr-x 4 melanson users         0 Jul 15  2005 audio
-r--r--r-- 1 melanson users 426387456 Jul 15  2005 demo.dat
-r--r--r-- 1 melanson users   1988128 Jul 15  2005 metal-gear-solid-the-twin-snakes-exe.dol
-r--r--r-- 1 melanson users      6496 Jul 15  2005 opening.bnr
dr-xr-xr-x 3 melanson users         0 Jul 15  2005 shared
-r--r--r-- 1 melanson users 198715392 Jul 15  2005 stage.dat

The executable file is metal-gear-solid-the-twin-snakes-exe.dol. The filename is a little long, which can happen since it is derived from the game title in the disc metadata, which can be nearly 1000 characters long. The GC executable format is known as DOL, probably short for Dolphin which was the codename of the GameCube during development.

I recognize that I’m likely the only person on the planet who cares about this utility but, hey, it’s my blog and what are blogs for if not to tell the world about the tedious minutiae of an individual’s life?

Related post:

• gcfuse

What on earth for? I’ve heard tales of strange and wonderful FMV formats on those petite GameCube DVDs and I just had to know for myself. One game I’m playing right now is Metroid Prime, which has visuals that certainly appear to be pre-rendered multimedia files. Let’s find out:

$ gcfs metroidprime.gcm gcm/

$ ls gcm/
Audio         Metroid4.pak  NESemu.rel    SlideShow.PAK   metroid5.pak
AudioGrp.pak  Metroid6.pak  NESemuD.rel   TestAnim.Pak    opening.bnr
GGuiSys.pak   Metroid7.pak  NESemuP.rel   Tweaks.Pak
Metroid1.pak  Metroid8.pak  NoARAM.pak    Video
Metroid2.pak  MidiData.pak  SamGunFx.pak  client_pad.bin
Metroid3.pak  MiscData.pak  SamusGun.pak  default.dol

$ ls gcm/Video/
00_first_start.thp            08_GBA_fileselect.thp  attract9.thp
01_startloop.thp              AfterCredits.thp       creditBG.thp
[...]

Right away, a new multimedia format– THP. The GC-Linux project already has documentation about this MJPEG-like format. Samples, of course, are available for your inspection.

Click for a larger image, and to guess which game is covered by the level 8 password on the sticky note

Now I’m playing with power. They put a tremendous amount of work into that cover. Passports for not only the Principality of NES but also the Republic of SNES. I guess in the early 1990s, nothing said “top secret” quite like a portable phone. Luckily, the book features passwords for Solar Jetman, the present object of my password infatuation. I wonder if the official password validator accepts the secret password comprised of all ‘Q’s, or if that’s handled by a special case.

Not only is Solar Jetman covered in the book but when I opened the book a carefully folded piece of paper slid out. It contained a number of very neatly written passwords, including ones for every world in Solar Jetman! It doesn’t look like my handwriting, plus the paper includes passwords for games that I never would have been caught dead playing. What a mystery. It’s almost like someone meant for me to find these clues and take up the cause of researching these ancient Nintendo password systems.

The password book contains passwords for a number of games where the only information carried in the password is what level the player was on. For a number of such games, I did a quick string check through the respective ROM data for the passwords. It looks like no coders bothered to use straight string comparison techniques for password validation.

One can only guess what sort of international espionage thrillers influenced the book’s artists, but their conceptualization of incognito (and airplane markings) involved a lot of pink:

Click for larger image of Codename: Pink Gamer