I finally had a revelation as to why so building software can be so difficult– because build systems are typically built on programming languages that you don’t normally use in your day to day programming activities. If the project is simple enough, the build system usually takes care of the complexities. If there are subtle complexities — and there always are — then you have to figure out how to customize the build system to meet your needs.
First, there’s the Makefile. It’s easy to forget that the syntax which comprises a Makefile pretty well qualifies as a programming language. I wonder if it’s Turing-complete? But writing and maintaining Makefiles manually is arduous and many systems have been created to generate Makefiles for you. At the end of the day, running ‘make’ still requires the presence of a Makefile and in the worst case scenario, you’re going to have to inspect and debug what was automatically generated for that Makefile.
So there is the widespread GNU build system, a.k.a., “the autotools”, named due to its principle components such as autoconf and automake. In this situation, you have no fewer than 3 distinct languages at work. You write your general build instructions using a set of m4 macros (language #1). These get processed by the autotools in order to generate a shell script (language #2) called configure. When this is executed by the user, it eventually generates a Makefile (language #3).
Over the years, a few challengers have attempted to dethrone autotools. One is CMake which configures a project using its own custom programming language that you will need to learn. Configuration generates a standard Makefile. So there are 2 languages involved in this approach.
Another option is SCons, which is Python-based, top to bottom. Only one programming language is involved in the build system; there’s no Makefile generated and run. Until I started writing this, I was guessing that the Python component generated a Makefile, but no.
That actually makes SCons look fairly desirable, at least if your only metric when choosing a build system is to minimize friction against rarely-used programming languages.
I should also make mention of a few others: Apache Ant is a build system in which the build process is described by an XML file. XML doesn’t qualify as a programming language (though that apparently doesn’t stop some people from using it as such). I see there’s also qmake, related to the Qt system. This system uses its own custom syntax.
My experience with Scons is that the fact that it uses Python (with no limitations) is actually incredibly annoying. Not because it makes things difficult; but because far too many projects end up adding custom Python code to it and the build system becomes incomprehensible.
There’s a new kid on the block called Tundra from the game studio Dice, which uses Lua for its configuration.
Only partly related, but pretty funny on it’s own: http://blog.gerv.net/2011/09/build-tool-name-shortage/
@thana: Hey, sounds like my alphabet of tracing: http://multimedia.cx/eggs/alphabet-of-tracing/
True, though– build systems don’t lend themselves to creative naming.
On the topic of uncommon build systems, two stand out to me:
tup ( http://gittup.org/tup/ ) because it shows how everybody does dependencies wrong.
redo ( https://github.com/apenwarr/redo ) because the build scripts are so elegant.
We need a new tool called “lovemake”
Yes, (GNU) Make is turing-complete.
Curiously, I had the reverse revelation: At some point I realized that most people don’t seem to consider build system work programming.
People would immediately cringe at duplicating parts 10+ times in C/Python/Java, but they have no qualms to do it in their build system. Their taste instincts don’t kick in and they do not look for ways to avoid duplication and ugliness. Unsurprisingly, it shows in many build system implementations, which work only some of time and even then more by sheer luck than anything else.
What also seems to confuse people about Make is that it is not an imperative programming language, but a declarative one. Since imperative languages are far more widespread and many programmers don’t know any declarative languages, that seems to create additional hurdles and possibly not let people realize that they are actually programming.
Heh, you failed to mention my least-favorite tool, imake, which plagued the X-11 source tree for so long. (And prompted me to write a paper in 1991 about how I eliminated it http://highlandsun.com/hyc/#Make .)
Honestly, I dunno how people got so lazy. Back in the 80s we would post source code on Usenet. You would read the README file. You would edit the Makefile and tweak one or two platform-specific macros, and then you’d be off and running. Nowadays, people type “configure”, it spits out an error message, and they’re off to the mailing lists asking “why didn’t this build” without even reading the damn error messages.
The fact that Scons is written in Python is an immediate dealbreaker for me. My experience with bitbake in OpenEmbedded was way too painful; eating up 500MB of RAM and 100% CPU just to begin the build process, before even the first line of code has been compiled, is ridiculous.
BSD Make worked great. GNU Make works great. If people actually took the time to learn Gmake fully (which honestly doesn’t take that much time), we wouldn’t be in this stupid mess of multiple half-baked build systems that all run slower than molasses.
tup sounds pretty interesting, may have to look into it further.
> people type “configureâ€, it spits out an error message, and they’re off to the mailing lists
Honestly who could blame them? With stuff like autotools being around there’s no chance for an ordinary person to fix it.
If you look at the code they generate it seems obvious they never even made an attempt to make the generated code readable.
I still think that adding a code generated to a project that spits out an unreadable mess is one of the worst things you can do to a project.
If it’s C code that even with -Wall and -Wextra spits out hundreds of warnings then good night, and better hold on to the compiler binary you are using right now because the next version probably won’t compile that mess into working code.
@hyc: Funny you should mention imake. I was struggling with the autotools this past week and felt a cold shudder when ‘imake’ suddenly popped into my mind. I remember using it to build something a long, long time ago and I remember it was horrific (and I don’t think I got the software built). Then I went back to quietly working with autotools, dealing with all the frustrations, because I realized the situation could be much worse.
To your point about half-baked build systems: Personally, I can’t get indignant about them anymore. Build systems are complicated because building software is complicated. If you have a simple “Hello world” program, that will be simple to build. Anything larger introduces its own unique complexities.
Aside: hyc, I saw on your resume that you ported the ‘MPEG-3’ codec to Atari ST. Is that referring to the MPEG-1, layer 3 audio codec, a.k.a., MP3?
I do not understand why people complain so much about imake. It solved the problem it was intended for adequately. Granted, it was not a general-purpose build tool, so attempts to use it as such were doomed to fail. Imake was based on the premise that the system would provide details for a number of configuration parameters, mostly but not necessarily related to X11, allowing generation of makefiles with the correct flags etc for the system. In some ways, it is not entirely unlike the idea of pkg-config, which also works well as long as the system config files are good and comes crashing down when they are not, as is all too often the case.
@Mike yes, MPEG-1, Layer 3 audio. This was something like 16 years ago, everyone was mis-naming it MPEG-3 back then. I could fix it, but leaving it that way reminds me of the timeframe. ;)
re: imake – eh – investing so much time/energy into a single-use tool is also a waste. Waste heaped on top of waste.
I believe one of the reasons to make source code open is to allow other people to *learn* from it, not just to allow them to *use* it. Back in the day, it was pretty much a given that the people who received your posts were techies not only capable of reading and understanding whatever code came along, but who were naturally inclined to take the time to do so. These days it’s a given that most people who use open source do so simply because they can use it without sending anyone any money; they don’t care how it works and they are not inclined to learn. That may be the normal, natural progression of a technology into society, but I have my doubts. I think there’s too much of a consumer mindset; people don’t realize that they can and should be creating this stuff themselves and contributing to the whole.
I seem to have gone off on a tangent.
At this point, what’s so hard about improving build systems is that your next Great Idea is just one of many in a sea of noise. It will take a lot of dramatic gains to get people interested enough in a new tool to give it critical mass to take off.
You seem to be ignoring the fact that a lot (most?) of software is written in “higher” languages – Java, C#, Scala etc. The idiomatic build systems (Ant+Ivy/Maven, MSBuild, sbt respectively) are actually very advanced and do a lot of the work “automagically”.
That being said, the only build system I’ve been at all impressed with lately is sbt; it’s completely Scala/Java-centric, but extremely powerful and elegant at that.