Description of the IBM Ultimotion (ULTI) Video Codec by Mike Melanson (mike at multimedia.cx) and Konstantin Shishkov (cannonball at bw-team.com) v1.0: September 23, 2004 ======================================================================= NOTE: The information in this document is now maintained in Wiki format at: http://wiki.multimedia.cx/index.php?title=IBM_UltiMotion ======================================================================= Copyright (c) 2004 Mike Melanson & Konstantin Shishkov Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". Contents -------- * Introduction * Algorithm * Gradient Fill Patterns * Appendix A: Converting Ultimotion Scaled YUV To Correct YUV * References * ChangeLog * GNU Free Documentation License Introduction ------------ IBM Ultimotion (henceforth ULTI) is a video codec typically seen in AVI files, though there is no reason it could not be used in other container formats. Algorithm --------- The ULTI algorithm operates natively on a scaled-down YUV 4:1:0 planar colorspace. This means that for each 4x4 block of pixels, each of the 16 pixels has a luminance (Y) sample and the entire block shares one blue chrominance sample (Cb or U) and one red chrominance sample (Cr or V). The colorspace is actually non-linear quantized to fixed values from 2 tables. Indices into these tables (64 possible values for luminance and 16 for chrominance) are transmitted in the final bitstream. The ULTI algorithm codes a frame by dividing it into a series of 8x8 blocks. Each 8x8 block is subdivided and coded as 4 4x4 blocks. The 4 sub blocks, 0-3, are coded in the following arrangement: 0 3 1 2 Each block can be coded using the same chrominance samples, or each sub block can be coded with its own pair of chrominance samples. Each sub block has 16 luminance samples that can be coded using a variety of methods. To decode a frame of Ultimotion data, follow this process: initialize uniq and mode flags to 0, initialize modifier to 0 for each 8x8 block in the frame, iterating from left -> right, top -> bottom: get the next byte in the stream (b0) if (b0 & 0xF8) == 0x70, handle escape code: if b0 is 0x70 modifier = next byte in stream note: only modifier values of 0 or 1 are valid if b0 is 0x71 set uniq flag if b0 is 0x72 toggle mode flag if b0 is 0x73 get the next byte in stream (b1) skip next (b1) blocks in the output frame (unchanged from previous frame) else chrominance byte = 0 if either the mode or uniq flags are set color flag = 1, each sub block has a different chrominance byte uniq = 0 else color flag = 0, all 4 sub blocks use the same chrominance byte if b0 is non-zero chrominance byte = next byte from stream for each 4x4 sub block in the block, arranged in the sub block pattern: extract sub block coding mode from b0: bits 7-6: sub block 0 coding mode bits 5-4: sub block 1 coding mode bits 3-2: sub block 2 coding mode bits 1-0: sub block 3 coding mode if coding mode is non-zero and color flag is 1 chrominance byte = next byte from stream coding mode 0: skip sub block (unchanged from the previous frame) coding mode 1: paint the luminance samples using a gradient pattern based on one 6-bit sample coded in the stream get next byte from stream (b0) bits 7-6 of b0 indicate gradient fill pattern: 0 = gradient fill pattern 0 1 = gradient fill pattern 2 2 = gradient fill pattern 6 3 = gradient fill pattern 4, swap Y0 with Y3 and Y1 with Y2 after unpacking Y samples bits 5-0 of b1 indicate luminance sample Y0 Y1 = Y0 if gradient fill pattern is pattern 0 Y2 = Y0 + 1 (saturated to an upper limit of 0x3F) Y3 = Y2 else Y2 = Y3 = Y0 fill luminance samples according to Y0..Y3 and the gradient fill pattern (see section Gradient Fill Patterns) coding mode 2: paint the luminance samples using a gradient pattern based on either 4 6-bit samples coded in the stream or 4 6-bit samples retrieved from a 16,384-entry codebook using an index coded in the stream if modifier is non-zero get the next 3 bytes from the stream as a big endian 24-bit number: Y0 = bits 23-18 Y1 = bits 17-12 Y2 = bits 11-6 Y3 = bits 5-0 fill luminance samples according to Y0..Y3 and gradient fill pattern 8 (see section Gradient Fill Patterns) else (modifier is 0) get the next 2 bytes from the stream as a big endian 16-bit number: gradient fill pattern = bits 15-12 luminance codebook index = bits 11-0 Y0..Y3 are set according to the codebook entry (see References for Ultimotion luminance codebook) if bit 3 of the gradient fill pattern is set, swap Y0 with Y3 and Y1 with Y2; clear bit 3 of the gradient fill pattern fill luminance samples according to Y0..Y3 and the gradient fill pattern (see section Gradient Fill Patterns) coding mode 3: flexible coding mode that can paint the luminance samples with 16 6-bit samples coded in the stream, or with a gradient fill pattern and 4 6-bit luminance samples coded in the stream, or with an arbitrary pattern using 2 luminance samples and a set of flags coded in the stream if modifier is non-zero get the next 16 6-bit numbers from the stream as the 16 luminance samples for the sub block, thus consuming 96 bits, or 12 bytes from the stream if modifier is 0 get the next byte in the stream (b0) if bit 7 of b0 is set gradient fill pattern = bits 6-4 of b0 get the next byte in the stream (b1) Y0 = ((b0 & 0x0F) << 2) | (b1 >> 6) Y1 = bits 5-0 of b1 Y2 = next byte in stream & 0x3F Y3 = next byte in stream & 0x3F fill luminance samples according to Y0..Y3 and the gradient fill pattern (see section Gradient Fill Patterns) else flagsA = b0 flagsB = next byte in the stream Y0 = next byte in stream & 0x3F Y1 = next byte in stream & 0x3F the 16 bits between flagsA and flagsB represent the pattern used to place either Y0 or Y1 into the luminance grid A7 A6 A5 A4 A3 A2 A1 A0 B7 B6 B5 B4 B3 B2 B1 B0 for example, if bit 3 of flagsB is 1, Y1 would be used in the lower left corner of the sub block, else Y0 finally, output sub block: bits 7-4 of the chrominance byte represent the red chrominance value for the sub block while bits 3-0 represent the blue value convert the 16 luminance values and the 2 chrominance values to a proper YUV colorspace using the conversion tables in Appendix A Gradient Fill Patterns ---------------------- The ULTI codec has 9 gradient fill patterns for painting 4x4 sub blocks of luminance data using 4 luminance samples. Gradient Pattern 0: 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 Gradient Pattern 1: 1 2 3 3 0 1 2 3 0 1 2 3 0 0 1 2 Gradient Pattern 2: 1 2 3 3 1 2 2 3 0 1 1 2 0 0 1 2 Gradient Pattern 3: 2 3 3 3 1 2 2 3 0 1 1 2 0 0 0 1 Gradient Pattern 4: 3 3 3 3 2 2 2 2 1 1 1 1 0 0 0 0 Gradient Pattern 5: 3 3 3 2 3 2 2 1 2 1 1 0 1 0 0 0 Gradient Pattern 6: 3 3 2 2 3 2 1 1 2 2 1 0 1 1 0 0 Gradient Pattern 7: 3 3 2 1 3 2 1 0 3 2 1 0 2 1 0 0 Gradient Pattern 8: 0 0 1 1 0 0 1 1 2 2 3 3 2 2 3 3 Appendix A: Converting Ultimotion Scaled YUV To Correct YUV ----------------------------------------------------------- To convert Ultimotion 6-bit luminance values to proper luminance samples, use the following conversion table: unsigned char ulti_lumas[64] = { 0x10, 0x13, 0x17, 0x1A, 0x1E, 0x21, 0x25, 0x28, 0x2C, 0x2F, 0x33, 0x36, 0x3A, 0x3D, 0x41, 0x44, 0x48, 0x4B, 0x4F, 0x52, 0x56, 0x59, 0x5C, 0x60, 0x63, 0x67, 0x6A, 0x6E, 0x71, 0x75, 0x78, 0x7C, 0x7F, 0x83, 0x86, 0x8A, 0x8D, 0x91, 0x94, 0x98, 0x9B, 0x9F, 0xA2, 0xA5, 0xA9, 0xAC, 0xB0, 0xB3, 0xB7, 0xBA, 0xBE, 0xC1, 0xC5, 0xC8, 0xCC, 0xCF, 0xD3, 0xD6, 0xDA, 0xDD, 0xE1, 0xE4, 0xE8, 0xEB}; To convert Ultimotion 4-bit chrominance values to proper chrominance samples, use the following conversion table: unsigned char ulti_chromas[16] = { 0x60, 0x67, 0x6D, 0x73, 0x7A, 0x80, 0x86, 0x8D, 0x93, 0x99, 0xA0, 0xA6, 0xAC, 0xB3, 0xB9, 0xC0}; References ---------- XAnim http://xanim.polter.net/ Ultimotion Video for Windows Decoding DLL, ultimo.dll Ultimotion Luminance Codebook: http://mplayerhq.hu/cgi-bin/cvsweb.cgi/~checkout~/ffmpeg/libavcodec/ulti_cb.h?content-type=text/x-cvsweb-markup&cvsroot=FFMpeg ChangeLog --------- v1.0: September 23, 2004 - initial release GNU Free Documentation License ------------------------------ see http://www.gnu.org/licenses/fdl.html