Clientside MySQL Compression
January 24th, 2008 by 
Multimedia Mike
I figured out yesterday's problem and the upshot is that x86_32 builds using gcc 2.95.3 have been reinstated for the FATE Server. So the nostalgic, sentimentalist users of FFmpeg should be happy to know the test suite is still being run through the old school compiler.
For reference, this is how to compress data using Python so that it will be suitable for insertion into a MySQL table (and so that MySQL will be able to decompress it with its built-in UNCOMPRESS() function):
PYTHON:
-
import zlib
-
from struct import *
-
-
# pack: the '<' symbol indicates little endian;
-
# the 'l' means treat the quantity as a long (i.e., 4 bytes)
-
compressed_string = pack('<l', len(raw_string)) + zlib.compress(raw_string, 9)
It can make an impressive difference, particularly with highly redundant text as is seen with compiler output. For example:
SQL:
-
mysql> SELECT
-
LENGTH(stderr) AS encoded,
-
LENGTH(UNCOMPRESS(stderr)) AS decoded
-
FROM ...
-
-
+---------+---------+
-
| encoded | decoded |
-
+---------+---------+
-
| 20831 | 1056570 |
-
+---------+---------+
Posted in FATE Server, Python | 
No Comments »