{"id":3972,"date":"2012-11-28T22:49:45","date_gmt":"2012-11-29T06:49:45","guid":{"rendered":"http:\/\/multimedia.cx\/eggs\/?p=3972"},"modified":"2020-07-25T23:01:04","modified_gmt":"2020-07-26T06:01:04","slug":"adventures-in-unicode","status":"publish","type":"post","link":"https:\/\/multimedia.cx\/eggs\/adventures-in-unicode\/","title":{"rendered":"Adventures in Unicode"},"content":{"rendered":"

Tangential to multimedia hacking is proper metadata handling. Recently, I have gathered an interest in processing a large corpus of multimedia files which are likely to contain metadata strings which do not fall into the lower ASCII set. This is significant because the lower ASCII set intersects perfectly with my own programming comfort zone. Indeed, all of my programming life, I have insisted on covering my ears and loudly asserting “LA LA LA LA LA! ALL TEXT EVERYWHERE IS ASCII!”<\/strong> I suspect I’m not alone in this.<\/p>\n

Thus, I took this as an opportunity to conquer my longstanding fear of Unicode. I developed a self-learning course comprised of a series of exercises which add up to this diagram:<\/p>\n


\n\"\"
\n<\/center><\/p>\n

Part 1: Understanding Text Encoding<\/strong>
\nPython has regular strings by default and then it has Unicode strings. The latter are prefixed by the letter ‘u’. This is what ‘\u00f6’ looks like encoded in each type.<\/p>\n

\r\n>>> '\u00f6', u'\u00f6'\r\n('\\xc3\\xb6', u'\\xf6')\r\n<\/pre>\n
A large part of my frustration with Unicode comes from Python yelling at me about UnicodeDecodeErrors and an inability to handle the number 0xc3 for some reason. This usually comes when I’m trying to wrap my head around an unrelated problem and don’t care to get sidetracked by text encoding issues. However, when I studied the above output, I finally understood where the 0xc3 comes from. I just didn’t understand what the encoding represents exactly.
\n<\/p>\n
I can see from assorted tables that ‘\u00f6’ is character 0xF6 in various encodings (in Unicode and Latin-1), so u’\\xf6′ makes sense. But what does ‘\\xc3\\xb6’ mean? It’s my style to excavate straight down to the lowest levels, and I wanted to understand exactly how characters are represented in memory. The UTF-8<\/a> encoding tables inform us that any Unicode code point above 0x7F but less than 0x800 will be encoded with 2 bytes:<\/p>\n
\r\n 110xxxxx 10xxxxxx\r\n<\/pre>\n
Applying this pattern to the \\xc3\\xb6 encoding:<\/p>\n
\r\n            hex: 0xc3      0xb6\r\n           bits: 11000011  10110110\r\n important bits: ---00011  --110110\r\n      assembled: 00011110110\r\n     code point: 0xf6\r\n<\/pre>\n
I was elated when I drew that out and made the connection. Maybe I’m the last programmer to figure this stuff out. But I’m still happy that I actually understand<\/em> those Python errors pertaining to the number 0xc3 and that I won’t have to apply canned solutions without understanding the core problem.<\/p>\n
I’m cheating on this part of this exercise just a little bit since the diagram implied that the Unicode text needs to come from a binary file. I’ll return to that in a bit. For now, I’ll just contrive the following Unicode string from the Python REPL:<\/p>\n
\r\n>>> u = u'\u00dc\u00f1\u00ec\u00e7\u00f4?\u00e9'\r\n>>> u\r\nu'\\xdc\\xf1\\xec\\xe7\\xf4\\u0111\\xe9'\r\n<\/pre>\n
Part 2: From Python To SQLite3<\/strong>
\nThe next step is to see what happens when I use Python’s SQLite3 module<\/a> to dump the string into a new database. Will the Unicode encoding be preserved on disk? What will UTF-8 look like on disk anyway?<\/p>\n
\r\n>>> import sqlite3\r\n>>> conn = sqlite3.connect('unicode.db')\r\n>>> conn.execute(\"CREATE TABLE t (t text)\")\r\n>>> conn.execute(\"INSERT INTO t VALUES (?)\", (u, ))\r\n>>> conn.commit()\r\n>>> conn.close()\r\n<\/pre>\n
Next, I manually view the resulting database file (unicode.db) using a hex editor and look for strings. Here we go:<\/p>\n
\r\n000007F0   02 29 C3 9C  C3 B1 C3 AC  C3 A7 C3 B4  C4 91 C3 A9\r\n<\/pre>\n
Look at that! It’s just like the \\xc3\\xf6 encoding we see in the regular Python strings.<\/p>\n
Part 3: From SQLite3 To A Web Page Via PHP<\/strong>
\nFinally, use PHP (love it or hate it, but it’s what’s most convenient on my hosting provider) to query the string from the database and display it on a web page, completing the outlined processing pipeline.<\/p>\n