Breaking Eggs And Making Omelettes

Topics On Multimedia Technology and Reverse Engineering


Feigning Transactions

March 20th, 2008 by Multimedia Mike

There is some collateral damage showing up on FATE due to that indexing solution deployed last week, a side effect that should also be obvious to anyone with one or two computer science courses behind them: Indexing a column of a table means that the index must be updated during each insert which makes the insert operation slower. The net effect is that it increases the opportunity that the FATE page cache generator might erroneously report too few tests corresponding to a build record. A previously yellow box (meaning that one or more tests failed) is green but closer inspection reveals only 21/21 tests succeeded.

What is happening is that, when a FATE build/test cycle is complete for a configuration, the script enters a new build record and, if the build succeeded, proceeds to enter a new test result record for each of the (currently) 111 tests. When the page cache update script kicks in on its usual 15-minute interval, a client script might still be inserting build records, leading to a race condition. I mitigate this possibility with the following algorithm:

  query the test result set corresponding to the a build record
  current_count = items in test result set
  while (1):
    query test results again
    new_count = items in test result set
    if current_count == new_count:
      current_count = new_count
      wait 4 seconds and try again (up to 10 times, then quit the script;
        it'll try from cron again anyway)

The heuristic actually works quite well. However, sometimes the server is extremely bogged down for one reason or another and the insert operations are occurring more than 4 seconds apart, or perhaps the client lost connection before it could enter all the test results.

The proper solution to this would be database transactions. MySQL is not renowned for its transactional support. True, version 5 is supposed to support them, and I am on version 5. But it requires special configuration that I don’t know how to perform and am not sure if I even have the access privileges to set up. But I have determined empirically that transactions are not supported with my current database and configuration (method: create a new table; insert a new record; start a transaction; update the record; query the record from a different session; the record has been updated, ergo, the START TRANSACTION was not honored).

Idea: Feign a transaction by adding a new field to the build record table, a boolean flag to indicate if the record is complete. When the build record is first entered, the flag is cleared. Only after all the test results have been entered will the flag be manually set true. Using this method, FATE will easily be able to find build records that were completed. This has the downside of leaving specious, “zombie” data in the tables and I will probably need to create a process for periodically cleaning the cruft in lieu of proper transaction/rollback support.

A perfect hack solution, I hope. We make do with what we have because MySQL is just so fast and free.

Posted in FATE Server | 5 Comments »

FFmpeg Hazing Ritual

March 17th, 2008 by Multimedia Mike

The pilot for an American TV show called Greek was a free download on Apple iTunes recently. I’m just as eager as the next open source software developer to brainlessly give a try to free stuff, so I checked it out. The show centers around some participants in the Greek-lettered fraternity and sorority system present on many college campuses. Hazing plays a role.

FFmpeg transliterated to Greek alphabet

This caused me to consider FFmpeg and the Google Summer of Code in the context of fraternities. GSoC is a college activity, like the Greek system. Participation might help your career along, post-school (an alleged rationale for joining a fraternity). And if you want to be initiated into the FFmpeg brotherhood, you are required to submit to a ritual known as the qualification task.

This would be a good time to mention that FFmpeg has been accepted into the GSoC for a third year in a row. Students who have any interest in working on a summer FFmpeg project on Google’s dime need to make their interest known on the ffmpeg-devel list and publicly claim a qualification hazing ritual.

Also, it seems that the x264 project wants in on some of the GSoC action, as indicated by their new adjunct Wiki page. This only creates ever more exciting opportunities. Wouldn’t you like to be a part?

Posted in Open Source Multimedia | 8 Comments »

An Object Lesson In Database Optimization

March 14th, 2008 by Multimedia Mike

I have a tendency to regard a database engine as a black box. I just formulate my queries and count on the engine to make them fast, somehow. I think this is similar to the faith that people tend to place in language compilers– just write the C code and the compiler will just magically optimize it. And if the code isn’t fast enough, maybe you should use a higher optimization level. Of course, a computer scientist ought to be able to analyze algorithmic running efficiency and spot opportunities for theoretical improvement, rather than relying on the compiler to insert a faster machine instruction here or there.

I started reading up on MySQL optimization strategies. There are a few things to understand about how the database works under the covers, things that are quite intuitive to anyone who has a semester of data structures and algorithms coursework. The FATE database is getting slower as it grows larger. The table growing the fastest is test_result. Each build currently generates 111 new rows, one for each active test specification.

mysql> SELECT COUNT(test_spec) 
       FROM test_result 
       WHERE build_record=6742;
| COUNT(test_spec) |
|              111 |
1 row in set (4.12 sec)

Read the rest of this entry »

Posted in FATE Server | Comments Off on An Object Lesson In Database Optimization

Unholy Alliance

March 10th, 2008 by Multimedia Mike (his actual domain name), the father of the WordPress blogging system, snapped this photo at the SxSW event and it gave me a cold chill for some reason:

Blu-Ray/Java Segway

I did a little searching and realized that I had already been exposed to the idea that Blu-Ray was colluding with Java. Now it occurs to me to wonder: Has there been demand for free multimedia players to support the Java functionality necessary to play Blu-Ray discs?

Posted in Java | 5 Comments »

« Previous Entries Next Entries »