Improving qt-faststart | Breaking Eggs And Making Omelettes

As this seems to be the homepage of ‘qt-faststart’ according to Google, here are some other useful links:

Moving right along…

It’s weird to think that I may have already written the most popular piece of free software that I will ever program– qt-faststart. It’s easily the most ported of all my programs– at the very least, there are native versions for Mac OS X (Cocoa wrapper) and Adobe AIR.

All that qt-faststart does is take an Apple QuickTime file that looks like this:

  [ 'mdat' QT data atom ]    [ 'moov' QT info atom ]

and rearranges it to look like this:

  [ 'moov' QT info atom ]    [ 'mdat' QT data atom ]

Why is this even necessary? In order to stream a QT file via HTTP, the moov atom needs to appear before the mdat atom. So why not write it that way when it is created? That’s tricky to do without 2 passes through the file. Most QT muxers only do the first pass. The qt-faststart utility does the second pass.

I have long maintained that the biggest flaw with Apple’s QuickTime file format is that chunk offsets are specified as absolute file offsets. If they were specified as offsets relative to the start of their mdat chunk, the moov chunk would be easily relocatable.

Are there ways to improve qt-faststart? One thing that would probably be nifty, and perhaps considered standard in this day and age, would be to compress the moov atom. moov atom compression has been around for a long time and operates with the standard zlib algorithm. It seems straightforward enough– take the moov atom from the end of the file, patch the offsets, compress the atom, and put it at the front. But wait– the chunk offsets will all be different since the moov atom is compressed. So, how about compressing the moov atom, assessing the size of the compressed moov atom, patching all of the moov atom’s chunk offsets, and then compressing? But… that would alter the size of the compressed atom since the data has changed. But it probably does not change much. I suspect this is why QuickTime specifies such atom types as ‘free’, ‘junk’, and ‘skip’. These are empty space atoms. Thus, the revised qt-faststart algorithm would probably look like this:

Find and load moov atom at end of QT file.
Compress moov atom to obtain rough size.
Declare moov atom size to be compressed size + factor n.
Patch moov atom’s chunk offset atoms to reflect calculated size.
Compress and write moov atom.
Write empty space atom with the difference between estimated and actual compressed sizes.
Copy remainder of QT file.

I wonder what factor n should be? I will probably have to determine this empirically. Or rather than a fixed number or percentage, I wonder if there should be an iterative process for converging on a somewhat optimal compressed size? Not that it matters a great deal; the size of the moov atom generally pales in comparson to the size of the actual multimedia data. It makes one ponder why the moov atom can be compressed in the first place. Alex once proposed that it may provide a layer of data integrity. According to the zlib tech notes, CRCs are used above the compression.

One bit of trivia about the program: qt-faststart does not take into account compressed moov atoms to begin with. I had always considered this a TODO item. However, it has occurred to me that compressed moov atoms probably only ever occur at the beginning of a file. At the very least, I have never received a complaint about qt-faststart being unable to process compressed moov atoms at the end of a file.

30 thoughts on “Improving qt-faststart”

Reimar October 19, 2007 at 1:32 am

I guess it would be completely reasonable to try all values of compressed size from 0 to uncompressed as offsets to the header values and the choosing whichever would be smallest. At least if the header is not more than 1 MB or so.
I also is possible to abort when the “assumed size” becomes larger than the best result you already had.
Since it’s zlib you could of course also take into account the maximum compression ratio, IIRC that is not really good for zlib.

2ge March 10, 2008 at 2:59 am

Thanks for this beatiful software, I really appreciate that. Just one idea – maybe it should work with STDIN, so I can make something like: cat file.mov | qt-faststart – out.mov

Thanks.

Multimedia Mike Post authorMarch 10, 2008 at 7:28 am

Not a bad idea. Just the other day, I was thinking that it would also be good if qt-faststart could pipe the output to STDOUT. This would help for automated testing of the tool, at least.

2ge March 11, 2008 at 2:23 am

superb,when we can await new version ? :) Because it should be really nice to do ffmpeg … | qt-faststart – out.mov

Thanks a lot!

Anonymous August 24, 2008 at 11:07 am

i am getting the following error:

could not allocate 0x20 byte for ftyp atom

what does this mean?

Multimedia Mike Post authorAugust 24, 2008 at 6:47 pm

It means something is very wrong. Email me and we’ll try to solve the problem.

Anonymous August 25, 2008 at 6:39 am

thanks. email sent.

victor September 23, 2008 at 3:19 am

Can you share the solution of “could not allocate 0x14 byte for ftyp atom” in here?
My OS is CentOS 4.6.Thank you very much

Multimedia Mike Post authorSeptember 23, 2008 at 6:58 pm

I’ll share my solution as soon as someone can furnish better details.

cdub October 1, 2008 at 9:43 pm

I’m having a problem with file sizes. Has this been checked on large file sizes, or is there something that I need to take into consideration? It works at 6mb, lose progressive video but keeps progessive audio at around 15 mb, and lose all progressiveness at 50 mb. I’m getting one “patching stco atom” message for the small ones, two “patching stco atom” for the medium one, and three “patching stco atom” for the biggest one. Would that have something to do with it? I’m in a bind – would LOVE to see this work for me. Thanks!

Multimedia Mike Post authorOctober 1, 2008 at 10:04 pm

Are you using my old version of the tool (available from this site) or the one packaged with FFmpeg? The FFmpeg one is preferable.

cdub October 2, 2008 at 8:25 am

I checked the one packaged with ffmpeg and I see no differences in the code I am using. So I must have got that version, unless I can’t strip it out – is there anything in that version dependent on the full ffmpeg package? Have you heard of this problem before?

Multimedia Mike Post authorOctober 2, 2008 at 10:09 am

Sounds like you might have an ancient version of FFmpeg. The qt-faststart currently packaged with FFmpeg is significantly changed from my original code. Get the latest SVN of FFmpeg.

Neil November 30, 2008 at 8:24 pm

This is a life saver. I didn’t know about MOOV atom until someone told me that my huge .mov files might have the MOOV atom at the end of the file. MP4box was suggested. It worked but I wasn’t good in DOS batch scripting.

Found qt-faststart on a forum. So I searched my machine and found qt-faststart.c from my ffmpeg source code. Compiled it. Tried it and worked pretty well.

So all I had to do to fix our 100+ fight .mov files was

for i in *.mov;do qt-faststart $i $i.mp4;done;

Took a coffee break, after few minutes, BAM. Done!!! :D

Thank you so much Mike!

Pingback: Auttomatic Releases WordPress.com Video Server Code | gaarai.com

Daniel January 22, 2009 at 6:28 pm

Did you find a solution for the ‘could not allocate 0Ã—20 byte for ftyp atom’ problem?

I’m having the same thing, server load goes 40+, and memory usage for qt-faststart hits 80%+ before top stops responding. Server has 2GB RAM so I don’t think that’s an issue.

Daniel January 23, 2009 at 3:27 pm

I tried the first version is SVN and it works. Hopefully you can find the cause of the problem because it’s scary using a copy this old!

Pingback: doomicile » flash videoplayer und mp4 video

Pingback: H264 Flash/Flex streaming « notboring dev blog

Pingback: qt-faststart.exe binary for windows « notboring dev blog

Bryan September 9, 2009 at 3:04 pm

It works great when it works!

I just wish it wouldn’t hang on corrupted files, and instead died with an error message to stderr.

pim September 29, 2009 at 2:54 am

Same problem, program hanging on corrupted files, would be nice to have a timeout param ;)

Chris October 8, 2009 at 6:54 am

Same problem here, program hanging with 100% cpu usage. any fix/workaround for this?

klappy October 30, 2009 at 9:30 am

@pim, @Chris

personally i wrap qt_faststart with both timelimit and nice.
`timelimit -t 1800 nice qt_faststart …`

this allows qt_faststart to run for a maximum of 30 minutes in an extreme case.
the nice command also allows for the command to be run at a low priority and to refrain from locking the system up, using idle resources.

especially if this machine is to perform other tasks, this is also most useful when scripting sequential usage with ffmpeg (which is also wrapped with nice) in which the script may be running more than once at a time.

Pingback: FFmpeg x264/mp4 Video Encoding auf Debian Lenny Â» Server Â» Debian Root