Yet Another Build Server

Thoughtworks has finally released a successor to their venerable Cruise Control - Cruise build server. The UI certainly looks nice and it seems quite flexible. There is even a free version (which is limited to two computers), which is great.

What is not clear though is how this product is different from AntHill, Buildforge, Bamboo, TeamCity, Gauntlet and the likes. The field is certainly becoming crowded - and I haven't even mentioned numerous open-source contenders. All these products seem to be doing the same thing - organizing your build scripts, interfacing with version control, running builds on a distributed "build server farm", collecting statistics, publishing reports and providing UI for all these functions.

All these features are important and useful. Ironically, however, what build servers don't do is automatically building or deploying your software. You still have to write Ant or Maven scripts, define and manage configuration parameters (using properties, XML, environment variables), deal with different environments (if I'm not mistaken, AntHill is the only product that has an explicit concept of an environment). For a complex project this could be a lot of work. Granted, every project is unique (if not, just use the default Maven configuration and you're done), so this could be a tough nut to crack. It should however be possible for a build server to have enough intelligence to infer how to build a project directly from the code base.

Going beyond Builds with Build Servers

A build server provides great benefits to a development team, including:

• Convenient way of sharing build results and artifacts.
• Ability to run builds without having to mess with command line parameters.
• Tight integration with version control.
• History and statistics.
• Accountability and audit (e.g., who ran what build).

In most instances build servers, such as Hudson, Continuum or LuntBuild are used mostly for continuous integration and development builds. But they can be used for much more than that. A build server provides a generic framework for performing any kind of deployment or operations-related activity, including deployment of an application into acceptance and production environments, configuring of an app server (e.g., setting up data sources and JMS destinations) and even app server monitoring.

Several things make it possible. Build servers have flexible scheduling capabilities allowing to schedule any kind of repetitive activity, such as “pinging” a server every 15 minutes. Integration with version control makes it easier to manage the scripts. Very often, scripts for administration or configuration tasks are created in an ad-hoc fashion with multiple versions scattered around different servers and directories. A build server is able to check out a specific version of a script from a version control system and execute it. In other words, a build server helps treating admin scripts same way with business applications. Additionally, build servers can publish logs. This is powerful, since it lets anyone (who has permissions) see results of configuration activities or monitoring scripts. Finally, build servers do a great job of notifying all interested parties via e-mail, RSS, IM and other means of communication.

Security could be another important benefit of leveraging a build server outside of development builds. Usually, access to production environments is tightly controlled. In many organizations, developers have no access to production whatsoever. Build server can help enforce this policy yet give developers visibility into the details of production deployment. This can be done by creating a separate build configuration just for production deployments.

All the benefits that I mentioned in the beginning of this post also apply to non-build activities. For example, the ability to see history of all production deployments can be very valuable.

Not all build servers are created equal, especially open source ones. Security, for one, could be supported and implemented very differently in different products. Support for various version control features, such as labels and branches, also varies. In other words, careful evaluation is required before embarking on an effort of expanding the build server beyond development. But more importantly, this effort requires buy-in from operations and support organization and other groups that will become users of the build server.

As an example, with one of our clients we’ve been able to implement a number of interesting non-build functions using a build server (we started with LuntBuild and now migrated to Hudson):

• Deployment of pre-built EAR/WAR files into test and production environments.
• Configuration of application servers. In our case, most of the app server resources are shared, so we run the configuration script separately from application deployments. In fact, we treat configuration as a separate “unit of release” with its own lifecycle.
• Monitoring of application servers. We have a script which tests health of application servers using JMX.
• Log scanning. We’ve implemented a custom log scanning tool which “greps” logs for various patterns. The build server sends notification if a pattern is found in the logs.

Many of these functions can be accomplished using various commercial system management and monitoring products. These products however can get expensive and very often organizations simply don’t have enough licenses to cover all environments. So a combination of custom scripts with a build server provides a viable alternative.

Secret Weapon of LAMP Applications

I’m surprised by low traction of LAMP applications in an enterprise (I use the LAMP acronym loosely as a catch-all for PHP, Ruby and Python apps). For most large organizations Java EE still reigns supreme. While developers and analysts debate performance and “enterprisey” features (or the lack thereof) of the LAMP stack, there is one aspect that is often overlooked – LAMP infrastructure is much more simple than a typical Java EE application server, hence its operations and maintenance is greatly simplified. And of course operations and maintenance is a big chunk of IT budget of any organization; in many shops it is actually the biggest part of the budget (60% is the average according to Forrester).

It usually goes like this. Data center operations are outsourced, and so data center personnel has to provide the first line of defense for all production problems. Data center folks are not developers, so NullPointerException does not mean much to them. But they have to be able to figure out who to call when they see the NullPointerException.

Here is an example of an error message from an application running under WebSphere Portal. This message is 318 lines long and is completely inscrutable to all but a hardcore WebSphere Portal developer or administrator. The most ironic part is that in spite of multiple “caused by” in the text, the message tells us nothing about the actual root cause of the problem, which, most likely, is a classloader conflict. As a sidebar, why can’t an app server at least give me a warning during deployment about a potential class loader problem (especially since all the app servers, even tomcat, add dozens of jars to the classpath)?

On the other hand, here is an example of an error message I get from a python app implemented using django:

Validating models...
djtest.order: name 'test' is not defined
1 error found.

So which error message do you think will be more palatable to an operations person looking at the logs?

I know I’ve exaggerated a bit – my django app is extremely simplistic at this point. However, it is true that many Java EE app servers and applications do extremely poor job at exception logging.

Even more obvious benefit of LAMP is availability of the source code. In Java EE world, the common practice is not to include the source code into WAR and JAR files. In many instances, the code is compiled without debug information. Even if the source code and line numbers are available, finding the right file takes some digging since we have to deal with multitude of JAR, EAR and WAR files. Not to mention that the same class can reside in multiple jars.

So if I was the person who had to respond to the “site is down” calls late at night, I’d vote for PHP.