Comments on: iOpenAppStoreMetaData

By: Multimedia Mike

Multimedia Mike — Sat, 05 Sep 2009 17:32:48 +0000

@John: I hope to make a few revisions to the system in the near future and then publish the SQLite database file on an ongoing basis. That should be more useful to people than publishing the harvesting utilities, which can take a long time to do their job and would only stress Apple’s servers (and cause them to implement roadblocks to doing this automatically).

By: John

John — Fri, 04 Sep 2009 07:16:27 +0000

Hi, I would love to get the code for downloading all the app details from the appstore, or the protocol you discovered that is used. I would like to create a database myself, and run some queries on it. I am particularly interested in the biggest publishers, and your list only goes up to 30.
thanks, John

By: Multimedia Mike

Multimedia Mike — Fri, 07 Aug 2009 16:27:10 +0000

It actually wasn’t that quick — I started early in July. :-) It was an on and off project, like most of the things I do.

By: Andrew

Andrew — Fri, 07 Aug 2009 15:22:09 +0000

Hey, neat stuff, I’m amazed by the quick-nature of this effort, and the amount of work you did :D You should have mentioned this on the list, I’m going to post this as some nice news :)

By: Multimedia Mike

Multimedia Mike — Wed, 05 Aug 2009 21:59:53 +0000

The problem is that this is a sophisticated SQL database with 2 tables instead of the pedagogical 1. I have to JOIN the appstore table with the genre table when breaking things down by genre and I don’t know how to eliminate duplicate rows (since apps can be placed in many genres).

I should probably just post the entire SQLite database in order to live up to the “Open” part of iOpenAppStoreMetaData.

By: SvdB

SvdB — Wed, 05 Aug 2009 08:53:57 +0000

Where is the difficulty in calculating the median? It’s nothing more than sorting all the elements and picking the middle one (or average of the middle two if there is an even number).
Is it that you need to be able to do it from SQL? A quick search with Google for “sql median” gave me this page, which has an almost ready to use query: http://www.1keydata.com/sql/sql-median.html

And thanks for adding a preview button. I’ve seen several WordPress blogs with one (or something similar, like a preview which is updated while you type), so I know that the plugins exist.

By: Multimedia Mike

Multimedia Mike — Wed, 05 Aug 2009 05:33:39 +0000

I couldn’t think of an easy way to calculate the correct median, so I took the easier way out and removed it from the pages. I also got rid of the mode since it’s pretty obvious from looking at the graph.

By: Multimedia Mike

Multimedia Mike — Mon, 03 Aug 2009 19:20:29 +0000

I’ll look into the preview button, especially at the request of one of my most frequent commenters. :-) There are probably dozens of plugins and extensions to do this in WordPress.

My understanding of “median” is a little bit off. I’ll get that corrected soon.

By: SvdB

SvdB — Mon, 03 Aug 2009 11:22:16 +0000

My personal experience on the development side of databases is small, but I suspect that your problem with large data in a database has more to do with SQLite than relational databases in general. Though I’d probably prefer keep my large data externally anyhow, if just for easy access outside of the application.

As for transforming XML documents to HTML with RegExps: “yuck”. Nuff said.

[quote cite="Mike]Have you ever gotten one little spark of an idea and started to research and prototype it, only to have it snowball into something absolutely unrecognizable in a short period of time[/quote]
Yes. In fact, I think my projects progress fastest when they “evolve” rather than when I design them up front.

Ceterum censeo you need a preview button with the comment form.

By: SvdB

SvdB — Mon, 03 Aug 2009 09:27:49 +0000

Your median price calculation is wrong. On http://multimedia.cx/iOpenAppStoreMetaData/7001.html well over 400 of the 801 apps are priced at $0.99, but the median is listed as $4.99.
Also, showing the standard deviation may be a nice addition.