Per-Frame Metadata

Someone asked me a question in email today that I thought I would pass on to the broader group. Are there any general methods for attaching general metadata — e.g., EXIF data — to individual video frames?


video frame metadata

At first thought, this strikes me as a container-level matter. However, there could be a video codec that embedded metadata in each frame and could be stored in any generalized container format.

Personally, I have always been more aware of issues surrounding content playback vs. content creation which I admit is somewhat of a weakness in my overall multimedia knowledge. The submitter had in mind data along the lines of absolute timestamps and GPS coordinate information to be included with every frame. This may seem a tad excessive but you can never underestimate other peoples’ requirements.

8 thoughts on “Per-Frame Metadata”

  1. Reimar

    Well, obviously you can use any of the image codecs like JPEG or PNG, almost all containers support MPNG and especially MJPG.

  2. Lúcio Corrêa

    First of all, thank you for answering my mail.

    And thanks again for posting it here (you even created an image to explain it!)

    So, as I told you in the mail, I know it’s possible to that with the FLV container, but does ffmpeg support H.264 in FLV in ffmpeg?

    What about the metadata, is it possible to store that too using ffmpeg?

    Thank you again!

  3. Robert Swain

    An XML container could do it easily. Just add GPS and absolute timestamp tags somewhere within each frame header. :)

    On a more serious note, is the FLV container intended to contain h.264? I got the impression that h.264 for flash was supposed to be contained in mp4 or mov.

  5. Aurel

    Something like this was already discussed on mplayer-dev-eng (IIRC).
    A good suggestion was to store GPS data in a separate stream.
    Basically, you could simply store a gpx file as if it were a
    text subtitle (embeded in Matroska for example). Just split each
    gpx track point into its own subtitle packet and store the gpx
    header inside the track extradata.
    That way, you could embed a gpx file inside Matroska, and extract
    it from Matroska almost lossless.
    This also directly associate GPS coordinates to video frames
    and audio packets (when you encounter a gpx packet, consider it
    is current coordinates and it applies to every audio/video
    packets up to next gpx packet).

  6. Lúcio Corrêa

    I thought it was possible to use h.264 in FLV container. Sorry, my mistake.

    Thanks for the suggestion, Aurel. I’ll try Matroska.

Comments are closed.