Breaking Eggs And Making Omelettes

Topics On Multimedia Technology and Reverse Engineering


Archives:

Improving qt-faststart

October 18th, 2007 by Multimedia Mike

As this seems to be the homepage of ‘qt-faststart’ according to Google, here are some other useful links:

Moving right along…

It’s weird to think that I may have already written the most popular piece of free software that I will ever program– qt-faststart. It’s easily the most ported of all my programs– at the very least, there are native versions for Mac OS X (Cocoa wrapper) and Adobe AIR.


Classic Apple QuickTime logo

All that qt-faststart does is take an Apple QuickTime file that looks like this:

  [ 'mdat' QT data atom ]    [ 'moov' QT info atom ]

and rearranges it to look like this:

  [ 'moov' QT info atom ]    [ 'mdat' QT data atom ]

Why is this even necessary? In order to stream a QT file via HTTP, the moov atom needs to appear before the mdat atom. So why not write it that way when it is created? That’s tricky to do without 2 passes through the file. Most QT muxers only do the first pass. The qt-faststart utility does the second pass.

I have long maintained that the biggest flaw with Apple’s QuickTime file format is that chunk offsets are specified as absolute file offsets. If they were specified as offsets relative to the start of their mdat chunk, the moov chunk would be easily relocatable.

Are there ways to improve qt-faststart? One thing that would probably be nifty, and perhaps considered standard in this day and age, would be to compress the moov atom. moov atom compression has been around for a long time and operates with the standard zlib algorithm. It seems straightforward enough– take the moov atom from the end of the file, patch the offsets, compress the atom, and put it at the front. But wait– the chunk offsets will all be different since the moov atom is compressed. So, how about compressing the moov atom, assessing the size of the compressed moov atom, patching all of the moov atom’s chunk offsets, and then compressing? But… that would alter the size of the compressed atom since the data has changed. But it probably does not change much. I suspect this is why QuickTime specifies such atom types as ‘free’, ‘junk’, and ‘skip’. These are empty space atoms. Thus, the revised qt-faststart algorithm would probably look like this:

  • Find and load moov atom at end of QT file.
  • Compress moov atom to obtain rough size.
  • Declare moov atom size to be compressed size + factor n.
  • Patch moov atom’s chunk offset atoms to reflect calculated size.
  • Compress and write moov atom.
  • Write empty space atom with the difference between estimated and actual compressed sizes.
  • Copy remainder of QT file.

I wonder what factor n should be? I will probably have to determine this empirically. Or rather than a fixed number or percentage, I wonder if there should be an iterative process for converging on a somewhat optimal compressed size? Not that it matters a great deal; the size of the moov atom generally pales in comparson to the size of the actual multimedia data. It makes one ponder why the moov atom can be compressed in the first place. Alex once proposed that it may provide a layer of data integrity. According to the zlib tech notes, CRCs are used above the compression.

One bit of trivia about the program: qt-faststart does not take into account compressed moov atoms to begin with. I had always considered this a TODO item. However, it has occurred to me that compressed moov atoms probably only ever occur at the beginning of a file. At the very least, I have never received a complaint about qt-faststart being unable to process compressed moov atoms at the end of a file.

Posted in Open Source Multimedia | 30 Comments »

30 Responses

  1. Reimar Says:

    I guess it would be completely reasonable to try all values of compressed size from 0 to uncompressed as offsets to the header values and the choosing whichever would be smallest. At least if the header is not more than 1 MB or so.
    I also is possible to abort when the “assumed size” becomes larger than the best result you already had.
    Since it’s zlib you could of course also take into account the maximum compression ratio, IIRC that is not really good for zlib.

  2. 2ge Says:

    Thanks for this beatiful software, I really appreciate that. Just one idea – maybe it should work with STDIN, so I can make something like: cat file.mov | qt-faststart – out.mov

    Thanks.

  3. Multimedia Mike Says:

    Not a bad idea. Just the other day, I was thinking that it would also be good if qt-faststart could pipe the output to STDOUT. This would help for automated testing of the tool, at least.

  4. 2ge Says:

    superb,when we can await new version ? :) Because it should be really nice to do ffmpeg … | qt-faststart – out.mov

    Thanks a lot!

  5. Anonymous Says:

    i am getting the following error:

    could not allocate 0×20 byte for ftyp atom

    what does this mean?

  6. Multimedia Mike Says:

    It means something is very wrong. Email me and we’ll try to solve the problem.

  7. Anonymous Says:

    thanks. email sent.

  8. victor Says:

    Can you share the solution of “could not allocate 0×14 byte for ftyp atom” in here?
    My OS is CentOS 4.6.Thank you very much

  9. Multimedia Mike Says:

    I’ll share my solution as soon as someone can furnish better details.

  10. cdub Says:

    I’m having a problem with file sizes. Has this been checked on large file sizes, or is there something that I need to take into consideration? It works at 6mb, lose progressive video but keeps progessive audio at around 15 mb, and lose all progressiveness at 50 mb. I’m getting one “patching stco atom” message for the small ones, two “patching stco atom” for the medium one, and three “patching stco atom” for the biggest one. Would that have something to do with it? I’m in a bind – would LOVE to see this work for me. Thanks!

  11. Multimedia Mike Says:

    Are you using my old version of the tool (available from this site) or the one packaged with FFmpeg? The FFmpeg one is preferable.

  12. cdub Says:

    I checked the one packaged with ffmpeg and I see no differences in the code I am using. So I must have got that version, unless I can’t strip it out – is there anything in that version dependent on the full ffmpeg package? Have you heard of this problem before?

  13. Multimedia Mike Says:

    Sounds like you might have an ancient version of FFmpeg. The qt-faststart currently packaged with FFmpeg is significantly changed from my original code. Get the latest SVN of FFmpeg.

  14. Neil Says:

    This is a life saver. I didn’t know about MOOV atom until someone told me that my huge .mov files might have the MOOV atom at the end of the file. MP4box was suggested. It worked but I wasn’t good in DOS batch scripting.

    Found qt-faststart on a forum. So I searched my machine and found qt-faststart.c from my ffmpeg source code. Compiled it. Tried it and worked pretty well.

    So all I had to do to fix our 100+ fight .mov files was

    for i in *.mov;do qt-faststart $i $i.mp4;done;

    Took a coffee break, after few minutes, BAM. Done!!! :D

    Thank you so much Mike!

  15. Auttomatic Releases WordPress.com Video Server Code | gaarai.com Says:

    [...] notes also indicate that you’ll need to have a working install of FFmpeg and qt-faststart (the readme.txt file says qu-faststart, but it’s actually qt-faststart). To install FFmpeg, [...]

  16. Daniel Says:

    Did you find a solution for the ‘could not allocate 0×20 byte for ftyp atom’ problem?

    I’m having the same thing, server load goes 40+, and memory usage for qt-faststart hits 80%+ before top stops responding. Server has 2GB RAM so I don’t think that’s an issue.

  17. Daniel Says:

    I tried the first version is SVN and it works. Hopefully you can find the cause of the problem because it’s scary using a copy this old!

  18. doomicile » flash videoplayer und mp4 video Says:

    [...] (kommt mir nicht auf die platte, so ein mist) oder dem public domain programm qt-faststart von Mike, das in dem Multimedia-Software Wunder ffmpeg enthalten ist. die anwendung von qt-faststart ist [...]

  19. H264 Flash/Flex streaming « notboring dev blog Says:

    [...] qt-faststart – moves the moov atom to the start of a h264 file to make it streamable [...]

  20. qt-faststart.exe binary for windows « notboring dev blog Says:

    [...] This tool was *not* written by me, say thanks to the author that can be found at http://multimedia.cx/eggs/improving-qt-faststart/. [...]

  21. Bryan Says:

    It works great when it works!

    I just wish it wouldn’t hang on corrupted files, and instead died with an error message to stderr.

  22. pim Says:

    Same problem, program hanging on corrupted files, would be nice to have a timeout param ;)

  23. Chris Says:

    Same problem here, program hanging with 100% cpu usage. any fix/workaround for this?

  24. klappy Says:

    @pim, @Chris

    personally i wrap qt_faststart with both timelimit and nice.
    `timelimit -t 1800 nice qt_faststart …`

    this allows qt_faststart to run for a maximum of 30 minutes in an extreme case.
    the nice command also allows for the command to be run at a low priority and to refrain from locking the system up, using idle resources.

    especially if this machine is to perform other tasks, this is also most useful when scripting sequential usage with ffmpeg (which is also wrapped with nice) in which the script may be running more than once at a time.

  25. FFmpeg x264/mp4 Video Encoding auf Debian Lenny » Server » Debian Root Says:

    [...] Es kann zu Problemen kommen wenn man MP4s mit x264 Streamen möchte. Man erkennt es wenn erst das ganze Video runter geladen wird, bevor es anfängt zu spielen. Das liegt an einer falschen Reihenfolge des Header/Data Teils im Video. Sollte man dieses Problem haben, kann qt-faststart helfen. qt-faststart wird mit ffmpeg ausgeliefert. Mehr Informationen gibt es hier. [...]

  26. 33t Says:

    how would i know qt faststart is working ?

  27. Optimizing Files for YouTube – RupaSharma.org Says:

    [...] we prefer the “moov” atom in the beginning of the file – also known as “fast start” or a “flattened” MP4 / MOV file. There are tools available on the web to flatten [...]

  28. How to Make MP4 Progressive with qt-faststart Says:

    [...] the name of this software says this program helps you move the important meta info from the end to the beginning of the file. [...]

  29. Video, Video, Vide…DOH! « Powered by Haiku Says:

    [...] encoded and are concerned that the metadata is not at the start, you can use another tool such as qt-faststart to move the metadata [...]

  30. MaTc-Online Blog » Further Video Discoveries Says:

    [...] achieve this I found the software qt-faststart – http://multimedia.cx/eggs/improving-qt-faststart/ it can take an mp4 and move the information around to make it start streaming when played from a [...]