NT was obviously a high-intensity, almost a death march kind of project and so the builds had to be churned out at a quick pace:

...the number of builds grew from a couple to a half dozen some week…

This may not sound like much, but since NT was getting quite big and complex, it kept the guys in the build lab busy. The builds were so critical that at some point the technical lead of the project, Dave Cutler, had to take over the build lab. This, however, did not improve the way builds were done. One of the members of the build team remembers:

He is not giving us the list, he’s basically saying, ‘Go to this directory and sync this file.’ He’s saying, ‘Pick up this file, do this, do that’.

We have all these cowboy developers, just slinging code like crazy, calling out: “We need another build!”

We’d think we were all done for the day, then test the build and it wouldn’t boot. We’d run around looking for the programmer whose code broke it.

I don’t think this situation was unique to Microsoft back then. But I also think that the attitude toward CM and development process automation has changed over the last 16 years. Today, automated builds is pretty much the norm for all but the smallest projects. Continuous integration and automated testing is becoming widespread. There is a dizzying array of build systems, build servers, version control systems and other CM and development tools.

There is a long way to go however. Implementing solid build/deployment and release management automation is still hard. Most large projects end up having to dedicate multiple highly skilled people to solving this problem. Home-grown script-based automation is still pretty much the state of the art. This is going to change. The tools will become more intelligent and advanced. I hope it won’t take another 16 years.

From what I understand, YAML is popular in the Ruby world and it is used for various PHP projects. However, it is almost unknown in Java/J2EE circles. Which is a shame. While annotations somewhat limited the spread of “XML hell” in Java applications, XML still remains a de-facto file configuration format. I would venture to say that except for few outliers, YAML would be a better option as a format for configuration files. Why is it the case? One reason is that YAML format simplifies application support. Developers often say that they don’t care about readability of XML since they use IDE or editors that hide the complexity of XML. Indeed, being able to work with XML in a nice tree view-based editor is appealing. But this does not work when application configuration needs to be quickly analyzed and potentially updated on some remote machine that most likely only has VI or notepad (which is usually the case in production environments, which I find very ironic – shouldn’t the production machine have the most advanced editors and analysis tools to make troubleshooting as efficient as possible?) in response to some production problem. For configuration files, readability and ease of understanding is the key.

Of course, there is also an old trusty property/name-value format. It is, however, very limited, since it does not support any kind of nesting or scoping. So all properties become essentially global and haven’t we learned already that global variables is not a good thing?

YAML, on the other hand, allows for expressing arbitrary complex models. Anything that can be expressed in XML can also be expressed using YAML.

On the downside, YAML does not have a very broad ecosystem. There are not that many editors that support YAML. There is a YAML Eclipse plugin, but in only gives color highlighting, no validation (here is another plugin which I have not tried yet). There is no metadata support, at least for Java, although there is a schema validator for Ruby (its Java port seems to be dead). There is also no XSLT equivalent.

There are two YAML parsers for Java – jvyaml and JYaml. They kinda work, but there is certainly room for improvement in terms of error messages and just the ability to detect and reject an incorrect document. Since YAML is supposed to be a language with minimal learning curve, the parsing has to be intuitive and bulletproof.

I still think that despite the shortcomings YAML is the way to go. Perhaps I will give a closer look to one of these parsers and see if I can tweak it a bit.

An “environment” is just a collection of resources which could include middleware and OS/filesystem resources. In the simplest case, an environment for a J2EE web application consists of a single application server. Complex applications consisting of multiple components could utilize many different resources, including several different middleware products (e.g., app server, Web server, messaging infrastructure, ESB).

For any IT organization it is important to know how their resources are used. ITIL has a concept of CMDB that’s supposed to contain all IT resources. However, the granularity of CMDB implementations is usually too high (typically, a server level) which makes it difficult to use for software development. Also, CMDB is not really integrated with development processes and tools, it’s kind of a thing on its own.

Ideally, the environment concept must be supported by development, version control, change management and build/deploy tools. Environment metadata can be used to automatically install an application in a given environment. Testing tools can use this metadata to generate “smog” tests or to adjust existing test cases (e.g., by using different URLs/endpoints). There should be a capability to produce various reports showing what version of what application is installed in what environment.

Sadly, all these wonderful features are mostly missing from modern development and CM tools. Developers rely on scripting and informal use of environment variables. Essentially, today, each application has its own “selfish” view of what an environment is. This makes providing consistent operations and support difficult. This is especially true in virtualized environments where each logical environment may consist of many different VMs.

Case in point are build servers and build tools. I looked at several build servers and found explicit environment support only in AntHill. All the others I looked at (several, I won’t name them) omit the environment concept completely (except for some lame support of environment variables). To me, this is really odd. While build servers have their root in continues integration, their key selling point in an enterprise is actually release management (at lease for commercial products; there are many great open source build servers to choose from if continuous integration is the only goal). So how can a release process be automated if the foundation of this process is missing entirely from the tool that’s supposed to help with the automation?

Build tools are the same way. There is some deployment support in both Maven (Cargo plugin) and Ant, but no way of supporting environment as an entity.

This, however, could finally give a shot in the arm to the long stagnated SOA projects. What makes me say that? Well, as I’ve been advocating all along, SOA represents an inexpensive way of adding flexibility and achieving new goals. SOA does not have to require significant new investments. Done smartly, it could be a great way of implementing new functionality without ripping the existing systems apart, which is undoubtedly won’t fly in light of highly constrained budgets.

To prove this point, in spite of lackluster economy, the demand for SOA architects has been booming.

There is one caveat to this view – organizations need to stop approaching SOA from the infrastructure standpoint (i.e., what ESB product do I need?) and start using it as a solution to real business needs. I’m not entirely sure that it’s going to happen, but bad economy could be just the motivation for that.

• Jython path is now handled by a separate JythonPath task.

• Jython interpreter is now scoped to Ant project. This means that you can have multiple Jython calls withing the same project that share common imports and variables.

• Jython task now supports nested text, similar to the “script” task.

Specifically, the task provides the following benefits:

• More graceful exception handling. Jython code invoked using "script" could generate long error stack that includes the Java stack trace of the "script" task. This is especially painful if you invoke other Ant tasks from Jython (you can do it using MyArch PAnt utility) that add their own Java stack trace. MyArch Jython task produces a brief readable error stack. I personally think that unwieldy error traces is a big issue in the industry and it's far from innocuous.

• You can use Ant properties as parameters (the task makes them available in the local namespace of the calling script).

• Ant "project" object is passed in the global variable "antProject" (available from the built-in namespace). In other words, "antProject" is available to all modules/functions invoked using "jython".

• Convenience "import" attribute.

• There is also a JythonPath task for easily setting jython path from Ant path.

• Jython interpreter is initialized once per Ant project. All scripts invoked from the same Ant project reuse the same built-in namespace. So you can define variables and imports in one call and use them in a subsequent call.

• Task name ( the name that prefixes all console output from Ant for a given task) is generated automatically based on the supplied Python code.

Example:

Ant code:

<jythonPath pythonPathRef="python.path" />
<property name="testProp" value="testVal" />

<jython>
print "Property from ant:", testProp
# define a var that we can use in other scripts
s="test"
</jython>

<jython>
print "Var created earlier: ",s
</jython>

<jython  import="from testmodule import *" exec="test(testProp)"  />

"testmodule" python code:

def test (prop):
    print "Passed parameter: ",prop
    print "Test property: ",antProject.properties["testProp"]

Jython task attributes:

• exec - python code to execute. This has to be a single line, e.g., mod.fun() although you could combine multiple statements separated by ";".
• import - a convenience attribute for providing "import" statement. Its only purpose is to make the task invocation more readable. Alternatively, you can have "import" as part of the"exec",e.g., exec="import mod;mod.fun()".
• execfile - execute a python script. This has to be an absolute path to a python file.

JythonPath task attributes:

• pythonPath - python.path to use.

• pythonPathRef - the python.path to use, given as reference to a PATH defined elsewhere.

Jython task is part of myarch-antutil project. Download jython task.

You may also be interested in PAnt - Ant python wrapper.

From the release notes:

Properties (name/value pairs) files are more consumable for human administrators than a mix of XML and other formats spread across multiple configuration directories.

Kudos to IBM for recognizing that.

It is still not clear though how hierarchical relationships between configuration objects will be supported.

Back in WAS 6 world, I've been using a simple jython script that converts python named parameters into wsadmin format. This is an example of a resource described in this format:

 WASConfig.DataSource(parent="testJDBCProvider", name="testDS", jndiName="jdbc/testDS",
                              description="Test DataSource", propertySet=dict(
                              resourceProperties=[
                                  dict(name="dbName", value="testDB", type="java.lang.String" ),
                                  dict(name="connectionAttribute",value="", type="java.lang.String")
                               ]))

I think that a slightly more streamlined python-based format will be superior to properties.

Using JACL might have been a valid option just a few years back (when WAS just came out) given the uncertainty surrounding the Jython project. Today, however, jython is clearly alive and well; an alpha version supporting Python 2.5 was announced recently. Therefore there is really no point in using JACL any longer, except may be for shops with a large collection of existing JACL scripts. JACL syntax is quite arcane compared with Python and the language is clearly not as widely used.

IBM confirmed this view by releasing JACL to Jython converter a couple years back.

Unfortunately, up until recently, jython was not officially supported in another IBM product, WebSphere Portal, which comes with wpsript tool for managing pages, deployable modules and other portal artifacts.

But since portal scripting relies on wsadmin's shell, jython is in fact fully supported by the product, it's just not documented. All that you need to do to switch to jython is to invoke wsadmin with "-lang jython" and "-wsadmin_classpath " followed by the list of portal jars (you can copy the classpath from SCRPATH variable definition in wpscript.sh).

As an example, I put together a simple Jython script for cleaning up a page hierarchy. Removing pages before applying an XMLAccess script with page definitions allows to start portal configuration from a clean "known" state. Very often, especially in a development environment, an application's page hierarchy gets polluted with various "test" pages created by developers. The script gets rid of them.

In WebSphere Portal 6.1 Jython is finally made a first-class citizen. The product's documentation proclaims that JACL support will be phased out and that jython is the way to go. Surprisingly, though, all examples still use good old JACL. I assume it's just a matter of time before they convert.

What is not clear though is how this product is different from AntHill, Buildforge, Bamboo, TeamCity, Gauntlet and the likes. The field is certainly becoming crowded - and I haven't even mentioned numerous open-source contenders. All these products seem to be doing the same thing - organizing your build scripts, interfacing with version control, running builds on a distributed "build server farm", collecting statistics, publishing reports and providing UI for all these functions.

All these features are important and useful. Ironically, however, what build servers don't do is automatically building or deploying your software. You still have to write Ant or Maven scripts, define and manage configuration parameters (using properties, XML, environment variables), deal with different environments (if I'm not mistaken, AntHill is the only product that has an explicit concept of an environment). For a complex project this could be a lot of work. Granted, every project is somewhat unique (if not, just use the default Maven configuration and you're done), so this could be a tough nut to crack. It should however be possible for a build server to have enough intelligence to infer how to build a project directly from the code base.

• Convenient way of sharing build results and artifacts.
• Ability to run builds without having to mess with command line parameters.
• Tight integration with version control.
• History and statistics.
• Accountability and audit (e.g., who ran what build).

In most instances build servers, such as Hudson, Continuum or LuntBuild are used mostly for continuous integration and development builds. But they can be used for much more than that. A build server provides a generic framework for performing any kind of deployment or operations-related activity, including deployment of an application into acceptance and production environments, configuring of an app server (e.g., setting up data sources and JMS destinations) and even app server monitoring.

Several things make it possible. Build servers have flexible scheduling capabilities allowing to schedule any kind of repetitive activity, such as "pinging" a server every 15 minutes. Integration with version control makes it easier to manage the scripts. Very often, scripts for administration or configuration tasks are created in an ad-hoc fashion with multiple versions scattered around different servers and directories. A build server is able to check out a specific version of a script from a version control system and execute it. In other words, a build server helps treating admin scripts same way with business applications. Additionally, build servers can publish logs. This is powerful, since it lets anyone (who has permissions) see results of configuration activities or monitoring scripts. Finally, build servers do a great job of notifying all interested parties via e-mail, RSS, IM and other means of communication.

Security could be another important benefit of leveraging a build server outside of development builds. Usually, access to production environments is tightly controlled. In many organizations, developers have no access to production whatsoever. Build server can help enforce this policy yet give developers visibility into the details of production deployment. This can be done by creating a separate build configuration just for production deployments.

All the benefits that I mentioned in the beginning of this post also apply to non-build activities. For example, the ability to see history of all production deployments can be very valuable.

Not all build servers are created equal, especially open source ones. Security, for one, could be supported and implemented very differently in different products. Support for various version control features, such as labels and branches, also varies. In other words, careful evaluation is required before embarking on an effort of expanding the build server beyond development. But more importantly, this effort requires buy-in from operations and support organization and other groups that will become users of the build server.

As an example, on my current job we've been able to implement a number of interesting non-build functions using our build server (we use LuntBuild, which has worked great for us):

• Deployment of pre-built EAR/WAR files into test and production environments.
• Configuration of application servers. In our case, most of the app server resources are shared, so we run the configuration script separately from application deployments. In fact, we treat configuration as a separate "unit of release" with its own lifecycle.
• Monitoring of application servers. We have a script which tests health of application servers using JMX.
• Log scanning. We've implemented a custom log scanning tool which "greps" logs for various patterns. The build server sends notification if a pattern is found in the logs.

Of course, many of these functions can be accomplished using various commercial system management and monitoring products. These products however can get expensive and very often organizations simply don't have enough licenses to cover all environments. So a combination of custom scripts with a build server can provide a viable alternative.