dpbuddy is implemented as a set of custom tasks for the popular build tool, Apache Ant.

Here is a quick example of dpbuddy in action:

<dp:copy cleanDirectories="true">
    <dpFileset prefix="/apps/services" 
               dir="services" includes="**/*.wsdl **/*.xsd"/>
</dp:copy>

This Ant task will remove remote directories if they exist, reproduce the local directory tree (all folders under "services") on the device and upload the necessary files based on the "includes" pattern.

dpbuddy is completely free; it can be downloaded from the dpbuddy product page

dpbuddy provides many cool features, including:

• Response from the device is presented in a human-readable form as opposed to raw SOAP/XML messages. dpbuddy makes it easy to understand error and status messages.

• Powerful remote "copy" command that automatically reproduces local directory tree on the device.

• Tight integration with Ant. Ant variables can be used inside deployment policies and configuration files.

• Easy-to-use alternative to deployment policies based on XPath.

• Ability to remotely "tail" device logs. It is even possible to automatically get new log messages similarly to Unix "tail -f" command. "tail" task can also check for error patterns.

• "Export" based on naming patterns. You don't need to know types ("classes") of DataPower objects; simply specify a regexp pattern and dpbuddy will export all objects matching this pattern.

• Support for self-signed certificates. No need to add DataPower certificates to the JDK store.

• Support for arbitrary SOMA requests. You can use Ant variables inside a request.

• Parsing of all commands on the client. In case of XML errors, DataPower returns cryptic "internal error" message. The actual error then has to be extracted from the device logs. dpbuddy on the other hand validates management XML commands on the client and displays error messages right away.

Go to dpbuddy product page to learn more.

“Class Not Found” exceptions could be quite tricky to troubleshoot because of the complexity of Java Web applications and application servers they run on. An average web application nowadays comes bundled with dozens of jar file (and probably thousands of classes). An average application server’s classpath is many pages long. Not to mention separately deployed libraries containing jar files shared by a group of applications. There should be little surprise that it is quite common for all these different jars and classloaders to clash with each other, get out of sync or become otherwise corrupt and mis-configured.

The list below represents a subset of all the possible causes of “ClassNotFoundException”. Hopefully this list could serve as a starting point for attacking the problem. The list was inspired by A List of Dumb Things to Check.

• To start, determine a type of the offending class. Is it a an application class, a third-party library class, a class provided by the application server or a JDK class? Determine the jar file that should contain the class. Determine where that jar should be located on the file system. Is it part of application installation, application server installation or some shared library installation? You may need to search for the class within multiple jars. Here is the command to do it (source): find . <del>name *.jar -print -exec jar -tvf {} ; | awk '/YOURSEARCHSTRING/ || /jar/ {print} ' (note-it won’t search within EAR and WAR files)
• Does the jar that’s supposed to contain the class exist on the file system?
• Are you able to “unjar” the jar using jar -xvf? Does the jar indeed contain the package and class in question?
• Check the version of the jar if you can’t find the class there. To determine the version, look at the jar’s MANIFEST.MF. Usually (but, unfortunately, not always) you will find some version information there. You can also compare the file size with the “baseline”.
• Does the account that the application server’s JVM was started with have read access to the jar? An application server usually runs under some sort of a system account. The jar might have been copied to the file system using a personal account from a different group.
• Have all application jars been updated during deployment? Are all the jars (including shared libraries) at the right version? Manual deployment process is quite common, so missing to update a jar is always a possibility.
• Is the class in question packaged with the application (e.g., under WEB-INF/lib) and being loaded by one of the parent classloaders? Most application servers utilize a classloader hierarchy where WAR file’s classloader is a child of EAR classloader which in turn is a child of the system (JVM) classloader. Parent classloaders can’t go down to request a class from a child classloader. The problem occurs if, for example, some jars were moved to a shared library but they still depend on classes packaged with the application.
  In order to diagnose this situation, you’ll need to have a good understanding of your application server’s classloader hierarchy. Here is the information for WebSphere and here is the WebLogic documentation on classloaders. Here is another article covering WebSphere classloaders in depth.
• Is any of the jars packaged with the application also present on any of the parent classloader’s classpath? Running different versions of the same jar or library can cause all kinds of issues, including ClassNotFoundException. Some app servers allow overriding default classloader behavior so that the jars packaged with the application are loaded first. This could fix the problem.
• If the jar with the class in question is part of a shared library (as opposed to packaged with the application), check if this library was made available to the application via the classloader configuration. For example, WebSphere configuration involves setting up a separate classloader for the library and explicitly associating it with the application.
• Is the version and patch level of the application server correct? Does it match your development environment? Look at the detailed version information for all the different components of your app servers and also get a list of installed patches. E.g., for WebSphere run versionInfo -long command.
• Is the application server running under the right JDK? E.g., check if the server startup script relies on JAVA_HOME and see which JDK the JAVA_HOME points to.
• If the application runs in a cluster, does the problem occur on all nodes or just on some? Are you trying to troubleshoot the problem on the right node?
• If the classname is driven from a string, either in java source or some other file, have you spelled the class name correctly? (Steve Loughran)

Once again, this is by no means a complete list. If anybody wants to contribute, please add a comment below and I’ll update the post.

Unfortunately, your simple build, while being a good starting point, does not accomplish much. Contrary to what many developers think, the purpose of an automated build is not to automate production of executable code (be it a WAR file or an "exe"). Its purpose is to verify correctness of the code and to discover as many problems and defects as possible as quickly as possible.

It is common knowledge that it is much less costly to fix a defect during construction than during testing phase:

According to the chart above (source: Six Sigma and IBM Systems Sciences Institute) it is two times more costly to fix a bug during testing than during implementation. I think this difference is actually much higher. Our short-term (working) memory is extremely volatile. According to some studies, the short term memory begins decaying after eighteen seconds. The cost of "context switching" for the brain is very high. In most organizations testing cycle takes at least a few weeks. This means that the bug that you just introduced will not be discovered for another few weeks at the earliest. When (or if!) it is finally discovered, most likely you'll be working on something entirely different. It will take at least a few hours for you just to recall all the details associated with the bug.

So the fact that your code compiles serves as a very weak indicator of code quality (although catching compilation problems early is important too, especially for large teams with high check-in volume). Automated testing must be done as part of every build. Most developers implement some automated testing using XUnit. In the majority of cases, these tests do not run against a deployed application, e.g., they do not hit a Web server. This kind of testing is useful, but it has its limitations. The main limitation is that we are not testing the application from the standpoint of its end users. For example, we're not testing AJAX logic running in a browser. Also, we're not testing the functionality that is dependent on an application server. Mock object frameworks help to a degree, but emulating application server's behavior could take some effort. Not to mention the fact that the "emulated" app server won't account for quirks of your "real" application server. In many cases there are subtle differences in app servers behavior, which is very often caused by differences in how classloader hierarchy is implemented. Reproducing these nuances using mock frameworks or even an embeddable servlet container, such as jetty, is impossible.

The bottom line is that your automated build has to be able to deploy your application and run tests against it. Using a browser-based testing tool such as Selenium will allow you to test your application as if it was used by end users, including testing of all your fancy AJAX features. Automating application deployments and testing does take some effort. Developing a comprehensive automated test suite could be a daunting tasks. But it is certainly possible and well worth it.

• Support for Ant targets.
• PAnt is now bundled with Ant Jython tasks that previously were available separately.

You can follow this guide to get started with PAnt.

Notable changes in this release:

• Support for positional (as opposed to named) arguments, e.g., “pant.echo(‘message’)”.
• Support for lists to express repeating elements.
• Support for “ant_” prefix to avoid conflicts with python keywords.

More information is available from PAnt project page

Please subscribe to our feed or follow us on twitter to continue receiving updates about PAnt – new version is coming shortly.

Luckily, we can get pretty close to this vision with a few very simple (and free) tools.

In my previous post I explained how to enable dynamic class reloading for WebSphere Application Server and avoid having to deploy your Java changes altogether. But what about changes to JSPs and other non-Java resources? How can we synchronize the directory used by the application server with the development workspace?

Turns out, there is an Eclipse plugin that does exactly that. It’s Filesync plugin developed by Andrei Loskutov.

As the name implies, the plugin automatically synchronizes workspace directories with external directories by doing one-way copy of changed files. It allows to specify multiple directory pairs and also to define include/exclude patterns and even use variable substitution.

To enable automatic updates of JSPs in the deployed application directory all you need to do is to define a folder pair that links web root in your workspace with the location of the exploded WAS directory in WAS (usually located under profile_root/installedApps/cell_name/app_name.ear/app_name.war).

With WAS you need to watch for static “<%@ include %>” directives in your JSPs. WAS will not reload included files unless you also update including JSP. A workaround here is to turn everything into “jsp:include” actions or use JSTL’s “c:import”. There might be a slight performance penalty for doing that but improved productivity is well worth it.

You can use Filesync plugin to synchronize your class files as well. This provides an alternative to the resource link-based approach that I described in the previous post. I still like using resource links better because they can be defined using Eclipse variables which makes it easier to share the configuration within a team. As far as I can tell, with Filesync you have to use absolute paths.

Here’s how the filesync configuration screen looks like:

Another good use of Filesync is to pull jar files from an external directory. Projects typically have a repository-like location where all third-party jars are checked-in (or it could be a full-blown Maven repository). You can easily add an external jar to your classpath in Eclipse. But how to put it under “WEB-INF/lib” where it needs to end up for the application server? With filesync it can be done easily by adding yet another folder pair.

In short, Filesync allows you to assemble your application “on the fly” without having to run an external build process. It also completely eliminates the need to explicitly update deployed applications.

This is unfortunate because actually WAS 6.1 has good support for dynamic reloading. With dynamic reloading turned on, WAS monitors changes on the file system and automatically reloads the module (i.e., all classes loaded by the module’s classloader) when it detects a change. The reloading is almost instantaneous for simple modules. For complex modules with a lot of classes or initialization logic the reloading step could take a little bit of time but it is still faster than redeploying an entire application (you should check out Java Rebel if you want a truly instantaneous deployment).

With dynamic reloading all we need to do in order to make our changes available to the server is to update class files in the location where the deployed application resides. This is especially straightforward for web application and classes under WEB-INF/classes since WAS always explodes web application archives during deployment. In case of jar files (say the ones under WEB-INF/lib) the situation is a more complicated.

Unfortunately, the location of the deployed application is usually different from the workspace where a developer makes changes. By default, deployed binaries are located under profile_root/installedApps/cell_name. While this location can be changed, the directory structure will still be somewhat different from how code is organized in the workspace.

We could write a simple Ant script to copy changes, but this again introduces a special “pseudo-deployment” step. It would be nice if we could simply make a change in Eclipse, save it and let dynamic reloading kick in without any extra steps.

Turns out that it is quite possible to make WAS and Eclipse behave this way.

First, let’s configure WAS:

• Log in to WAS admin console and make sure that “Run in development mode” is checked for your sever. This is the default for standalone installations.
• Deploy your application to WAS using WAS admin console.
• For convenience, you may want to specify a non-standard location for application binaries during installation to shorten the path, e.g., “was_installed_apps”. This step is optional.
• Go to “Enterprise applications/your_app/Class loading and update detection”.
• Make sure that “reload classes” is checked.
• Set reload interval to some reasonable number, say “3”. By default it’s set to “0” which means “never”. IBM recommends 3 seconds as an optimal interval, although I’ve been using 1 second without any issues (for relatively small modules though).
• Stop and start the application.

Now let’s configure Eclipse. We will have to create a resource link pointing to the deployed application and configure the project to compile classes to the deployed location.

• Go to “Java Build Path” of the project. Click on “Browse” next to “Default output folder”.
• Click “Create New Folder…”, “Advanced”, check “Link to folder in the file system”.
• Click on “Browse” and locate the root of the exploded WAR file in the deployed application location. For example, for application “HelloWorldWeb” the path will be “profile_root/installedApps/cell_name/HelloWorldWeb.ear/HelloWorldWeb.war”. Give the link a meaningful name, e.g., “deployment”. Note: if you share .project and .classpath files with other developers, use Eclipse variables instead of the absolute path.
• Click OK. This will create a resource link that you can use to specify the output folder.
• Change the output folder to point to “project_name/link_name/WEB_INF/classes”, e.g., “HelloWorldWeb/deployment/WEB-INF/classes”. Click OK.
• Eclipse will recompile your project.
• From this point forward any class change will trigger dynamic reloading on the server.
• The resource link is also available in your package explorer, so you can browse and edit the deployed files. You need to be careful if you want to edit JSPs or other files that way as they will be overridden by the next full re-deployment.

This techniques takes care of class files only. Dynamic reloading of JSP files is a different story.

Note: This has been tested only with Eclipse 3.4 and WAS 6.1 and on modules with a relatively small code base. I’d be curious to know how effective this approach is for large modules.

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you’d like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you’re looking for help with any of these tasks, please let us know.

This works fine most of the time, however, in some cases there could be a need for a dynamic list of attributes. For example a task can pass attribute values to some external tool that has its own set of parameters that you don’t want to hardcode in Ant. Or you may simply like the flexibility of using dynamic attributes as opposed to predefined setters.

public void maybeConfigure() throws BuildException {
}

RuntimeConfigurable configurator= getRuntimeConfigurableWrapper();
Map attributes=configurator.getAttributeMap();
String attr1=(String)attributes.get("attr1");

It is, in fact, quite easy to deal with this situation in Jython and suppress the annoying stack trace:

import sys
from com.ibm.ws.scripting import ScriptingException
...
    try:
        AdminTask....
    except ScriptingException:
        # note that we can't use "as" because of python 2.1 version, have to use sys
        print "Error:\n"+str(sys.exc_info()[1])

NT was obviously a high-intensity, almost a death march kind of project and so the builds had to be churned out at a quick pace:

...the number of builds grew from a couple to a half dozen some week…

This may not sound like much, but since NT was getting quite big and complex, it kept the guys in the build lab busy. The builds were so critical that at some point the technical lead of the project, Dave Cutler, had to take over the build lab. This, however, did not improve the way builds were done. One of the members of the build team remembers:

He is not giving us the list, he’s basically saying, ‘Go to this directory and sync this file.’ He’s saying, ‘Pick up this file, do this, do that’.

We have all these cowboy developers, just slinging code like crazy, calling out: “We need another build!”

We’d think we were all done for the day, then test the build and it wouldn’t boot. We’d run around looking for the programmer whose code broke it.

I don’t think this situation was unique to Microsoft back then. But I also think that the attitude toward CM and development process automation has changed over the last 16 years. Today, automated builds is pretty much the norm for all but the smallest projects. Continuous integration and automated testing is becoming widespread. There is a dizzying array of build systems, build servers, version control systems and other CM and development tools.

There is a long way to go however. Implementing solid build/deployment and release management automation is still hard. Most large projects end up having to dedicate multiple highly skilled people to solving this problem. Home-grown script-based automation is still pretty much the state of the art. This is going to change. The tools will become more intelligent and advanced. I hope it won’t take another 16 years.