Someone recently asked on the ffmpeg-devel mailing list for nightly binary builds of FFmpeg and its associated libraries to be made available for download. The idea seemed to gain traction on the list so I began thinking of how we can make this a reality. Since FATE clients are already building FFmpeg binaries day and night, it’s not a huge leap to consider extending the infrastructure slightly to include packaging the binaries and sending them to a central server. So let’s talk process:
- After building binaries as part of FATE, make a compressed bundle (.tar.bz2 for Unix, .zip for Windows) and ship the package right here to multimedia.cx. Hey, I have the space to spare (an alleged 500 GB with the current hosting plan, of which I am currently barely using 6 GB, and nearly half of that is FATE data). I also have FTP account options which should facilitate transfer.
- The filename would follow the naming convention of “ffmpeg-bin-svn-OS-arch-compiler.ext”. So, for example, the build of SVN 18005 from Linux / x86_32 / gcc 4.2.4 would be: ffmpeg-bin-svn18005-linux-x86_32-gcc-4.2.4.tar.bz2.
- What should be in the package? The FFmpeg binary programs and the associated libraries, of course. But in what hierarchy? A flat hierarchy doesn’t strike me as a good idea (unzip and potentially trash existing files in your current directory). Instead, package the files in, e.g., ffmpeg-svn18005-linux-x86_32-gcc-4.2.4/.
- There should be a standard, auto-generated README.txt file in the same directory. Actually, perhaps it should be a simple HTML file since I envision it should contain — in addition to basic information such as its revision and the time it was built — a report of which tests passed and which failed, along with links to each report in the FATE database.
- Create a web interface that allows users to browse among the latest FFmpeg binary builds. Out of everything presented in this brainstorm, this is the step that actually gives me some pause since I don’t exactly know how I would implement it off the top of my head.
Feedback welcome. Otherwise, these are likely to be the decisions I run with for the first iteration of this plan, when I get around to it (think of this as the public comment period).
good idea, but how much builds are you going to keep. are they going to be rotated, to reduce space and clutter?
how often would new build be posted? for each svn revision?
Sounds good, I need to use FFmpeg for some video file conversions (at least getting a video file to a raw state) and this would save me compiling it myself for the three different OS’s I use so I can use the latest version and also provide feedback :)
The other question is how do you determine what libraries are linked in (e.g. libmp3lame, libspeex, etc.). I think FATE builds today don’t link any, but if you changed that, some way of communicating it would be good.
@yoshi314: Yes, new builds posted at each SVN revision. It would probably add up to a tremendous amount of storage. But then again, I am supposed to have 500 GB with this hosting plan. I will probably need some method for gracefully deleting older archived versions (delete the packages and probably update a database that maintains build and path information).
@Andrew: Your idea motivates the brainstorm that I should log information about each submitted package to a database that can be easily and automatically queried via HTTP.
@Art: That’s a good point– the readme should stipulate what compilation options went into the build, thus identifying which libraries are linked; i.e., none.
All told, that latter point makes me suspect that this scheme won’t be that useful for people who want fully working and useful, pre-built FFmpeg binaries and libraries– they won’t be able to encode MP3, AAC, or H.264.
The -bin- in the name is superfluous, imo.
linux-x86_32-gcc-4.2.4 already says that something was compiled.
Apart from that, a great idea: imo, sources chosen like this should be our releases in the future.
Carl Eugen
instead of libs cluttering up everything
why not go enable static? one binary. possibly stripped
to save some space. maybe even upx compressed (for windows)?
@compn: Fortunately, the FATE installations are already creating stripped binaries by virtue of configuring with –disable-debug. As for libraries, the FATE clients build them to make sure they work (and we have caught a few problems because of this); building a monolithic static library is easier. I really don’t think it makes a big difference — at least size-wise — whether the package contains a single binary or a binary with 8 lib files alongside.
And UPX compression– we’re going to compress the archive anyway, so why?
I’d go for 7z compression to save you some bandwidth usage (probably more critical than space).
Besides reducing installation headaches, the packages with binaries statically compiled should be smaller than the ones using shared libraries, as the linker will have hopefully removed any unused portions of the libraries.
upx for faster loading of the binary, i think.
[benchmars needed]
also strip can be used to remove all/unused symbols
or more than just debug symbols so it may help… more benchmarks needed :)