Author Archives: Multimedia Mike

Playing With File

I played with the ‘file’ utility a long time ago because I wanted to make it recognize a large number of multimedia formats. I had trouble getting my changes to take. But I’m prepared to try again after many years.

Aiming at the Corpus
In my local mirror of the MPlayerHQ samples archive, I find 9853 unique files. So I run all of them through the ‘file’ command:

  'find /path/to/samples -type f -print0 | xargs -0 file --no-pad'

My Ubuntu installation has file v5.04. I also tested against 5.07 and the latest, 5.08. Here is the number of files each version was unable to identify (generically marking as ‘data’):

5.04  1521
5.07  1405
5.08  1501

That seems like a regression for v5.08 until I dug into the details and saw quite a few items like this, indicating that the MPEG detection could use some work:

-mov/mov-demux-infinite-loop.mpg: DOS-executable (
+mov/mov-demux-infinite-loop.mpg: data
-image-samples/UNeedQT4.pntg: DOS-executable (
+image-samples/UNeedQT4.pntg: data

Workflow
These are just notes to myself and perhaps anyone else who wants to add new file formats to be identified by the ‘file’ command.

First, download either the latest release from the FTP or clone from Github. Do the usual unpack, ‘./configure’, ‘make’ routine. To use this newly-built version and its associated magic file:

  ./src/file --magic-file magic/magic.mgc <file>

To add a new format for ID, first, run the foregoing command to ensure that it’s not already identified. Then, check over the files in magic/Magdir and see which one might pertain to what you’re doing (it’s unlikely that your format will merit a new file in this directory). For example, for this round, I modified animation, audio, iff, and riff. Add or modify existing specs based on the copious examples in the directory and by consulting the appropriate man page (‘man 5 magic’).

Finally, run ‘make’ again which will regenerate the magic file. Invoke the above command again to use the modified magic file.

Before and After
On a selection of formats taken from the samples archive (renamed and cut down to a kilobyte because detection typically only relies on the first few bytes), here is the “before”:

amv:            RIFF (little-endian) data
armovie:        data
bbc-dirac:      data
interplay-mve:  data
mtv:            data
nintendo-thp:   data
nullsoft-video: data
redcode:        data
sega-film:      data
smacker:        data
trueaudio:      data
vqa:            IFF data
wavpack:        data
wc3-mve:        IFF data
wtv:            data

And the “after”:

amv:            RIFF (little-endian) data, AMV 
armovie:        ARMovie
bbc-dirac:      BBC Dirac Video
interplay-mve:  Interplay MVE Movie
mtv:            MTV Multimedia File
nintendo-thp:   Nintendo THP Multimedia
nullsoft-video: Nullsoft Video
redcode:        REDCode Video
sega-film:      Sega FILM/CPK Multimedia, 320 x 224
smacker:        RAD Game Tools Smacker Multimedia version 2, 320 x 200, 100 frames
trueaudio:      True Audio Lossless Audio
vqa:            IFF data, Westwood Studios VQA Multimedia, 418 video frames, 320 x 200
wavpack:        WavPack Lossless Audio
wc3-mve:        IFF data, Wing Commander III Video, PC version
wtv:            Windows Television DVR Media

After rerunning ‘file’ on the mphq corpus using the modified magic file, only 1329 files remain unidentified (down from 1501).

Going Forward
As mentioned, MPEG detection could probably be strengthened. However, a major weakness is QuickTime/MP4. Many files are not detected, probably owing to the many ways that QuickTime files can begin.

Started Programming Young

I have some of the strangest memories of my struggles to jump into computer programming.

Back To BASIC
I remember doing some Logo programming on Apple II computers at school in 5th grade (1987 timeframe). But that was mostly driving turtle graphics. Then I remember doing some TRS-80 BASIC in 7th grade, circa 1989. Emboldened by what very little I had learned in perhaps the week or 2 we took in a science class to do this, I tried a little GW-BASIC on my family’s “IBM-PC compatible” computer (they were still called that back then). I still remember what my first program consisted of. Even back then I was interested in manipulating graphics and color on a computer screen. Thus:

10 color 1
20 print "This is color 1"
30 color 2
40 print "This is color 2"
...

And so on through 15 colors. Hey, it did the job– it demonstrated the 15 different colors you could set in text mode.

What’s FOR For?
That 7th grade computer unit in science class wasn’t very thick on computer science details. I recall working with a lab partner to transcribe code listings into a computer (and also saving my work to a storage cassette). We also developed form processing programs that would print instructions to input text followed by an “INPUT I$” statement to obtain the user’s output.

I remember there was some situation where we needed a brief delay between input and printing. The teacher told us to use a construct of the form:

10 FOR I = 1 TO 20000
20 NEXT I

We had to calibrate the number based on our empirical assessment of how long it lasted but I recall that the number couldn’t be much higher than about 32000, for reasons that would become clearer much later.

Imagine my confusion when I would read and try to comprehend BASIC program code I would find in magazines. I would of course see that FOR..NEXT construct all over the place but obviously not in the context of introducing deliberate execution delays. Indeed, my understanding of one of the fundamental building blocks of computer programming — iteration — was completely skewed because of this early lesson.

Refactoring
Somewhere along the line, I figured out that the FOR..NEXT could be used to do the same thing a bunch of times, possibly with different values. A few years after I had written that color program, I found it again and realized that I could write it as:

10 for I = 1 to 15
20 color I
30 print I
40 next I

It still took me a few more years to sort out the meaning of WHILE..WEND, though.

The Fastest Way To Learn Assembly Language

I saw an old StackOverflow thread linked from Hacker News asking how to whether it’s worthwhile to learn assembly language and how to go about doing so. I’d like to take a stab at the last question.

The fastest way to learn an assembly language is to reverse engineer something. Seriously, start with something that you know (like a C program that you wrote yourself) and take it apart. The good news is that assembly language is very simple and you will get a lot of practice in a short amount of time with RE.

So here’s how you do it:

  • Take a simple program in C and build it with your tool chain, whether GNU gcc on Linux, Xcode on Mac, or MSVC on Windows. Also, make sure to turn on debugging symbols during compilation (this will help annotate the disassembly).
  • On Linux, use objdump: objdump -d program_binary
  • On Mac, use otool: otool -tV program_binary
  • On Windows: I admit, I’m a bit fuzzy on this one– I’m quite certain there’s a standard MSVC tool that prints the assembly listing.

Anyway, look at the disassembled code and find the main() function. Work from there. Whatever the first instruction is, look it up on Google. You’ll likely find various CPU manuals that will explain the simple operation of the instruction. Look up the next unfamiliar instruction, then the next. Trust me, you’ll become an ASM expert in no time.

Good luck!

Physical Calculus Education

I have never claimed to be especially proficient at math. I did take Advanced Placement calculus in my senior year of high school. While digging through some boxes, I found an old grade report from that high school year. I wondered what motivated me to save it. Maybe it’s because it offered this clue as to why I can’t perform adequately in math class:



Mystery solved: I did not wear proper P.E. attire to calculus class.