{"id":3077,"date":"2010-12-31T19:40:34","date_gmt":"2011-01-01T03:40:34","guid":{"rendered":"http:\/\/multimedia.cx\/eggs\/?p=3077"},"modified":"2010-12-31T19:40:34","modified_gmt":"2011-01-01T03:40:34","slug":"a-better-process-runner","status":"publish","type":"post","link":"https:\/\/multimedia.cx\/eggs\/a-better-process-runner\/","title":{"rendered":"A Better Process Runner"},"content":{"rendered":"

I was recently processing a huge corpus of data. It went like this: For each file in a large set, run 'cmdline-tool <file>'<\/code>, capture the output and log results to a database, including whether the tool crashed. I wrote it in Python. I have done this exact type of the thing enough times in Python that I’m starting to notice a pattern.<\/p>\n

Every time I start writing such a program, I always begin with using Python’s commands module<\/a> because it’s the easiest thing to do. Then I always have to abandon the module when I remember the hard way that whatever ‘cmdline-tool’ is, it might run errant and try to execute forever. That’s when I import (rather, copy over) my process runner from FATE, the one that is able to kill a process after it has been running too long. I have used this module enough times that I wonder if I should spin it off into a new Python module.<\/p>\n

Or maybe I’m going about this the wrong way. Perhaps when the data set reaches a certain size, I’m really supposed to throw it on some kind of distributed cluster rather than task it to a Python script (a multithreaded one, to be sure, but one that runs on a single machine). Running the job on a distributed architecture wouldn’t obviate the need for such early termination. But hopefully, such architectures already have that functionality built in. It’s something to research in the new year.<\/p>\n

I guess there are also process limits, enforced by the shell. I don’t think I have ever gotten those to work correctly, though.<\/p>\n","protected":false},"excerpt":{"rendered":"

I was recently processing a huge corpus of data. It went like this: For each file in a large set, run ‘cmdline-tool <file>’, capture the output and log results to a database, including whether the tool crashed. I wrote it in Python. I have done this exact type of the thing enough times in Python […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[],"class_list":["post-3077","post","type-post","status-publish","format-standard","hentry","category-python"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t