Some application servers require that location of the development workspace has to be different from the location of the deployed application. For example, you can easily point Tomcat to the root of your Web application using “docBase” of the “Context” element. But you’re out of luck with WebSphere Application Server (WAS). You have to go through a separate application update process (using admin console or Rational Application Developer tooling) to synchronize your deployed application with the workspace. In my view, this update (a.k.a. “deployment”) step should never be required in a local development environment. It is one thing to have to deploy to a test or a production environment that consists of multiple servers that are segregated from the machine hosting the build artifacts. But in a situation when both the code and the application server are sitting on the same machine, the deployment step is redundant. We should be able to simply tell the app server where the code is and it can then do whatever is needed to load the code into JVM.

Luckily, we can get pretty close to this vision with a few very simple (and free) tools.

In my previous post I explained how to enable dynamic class reloading for WebSphere Application Server and avoid having to deploy your Java changes altogether. But what about changes to JSPs and other non-Java resources? How can we synchronize the directory used by the application server with the development workspace?

Turns out, there is an Eclipse plugin that does exactly that. It’s Filesync plugin developed by Andrei Loskutov.

As the name implies, the plugin automatically synchronizes workspace directories with external directories by doing one-way copy of changed files. It allows to specify multiple directory pairs and also to define include/exclude patterns and even use variable substitution.

To enable automatic updates of JSPs in the deployed application directory all you need to do is to define a folder pair that links web root in your workspace with the location of the exploded WAS directory in WAS (usually located under profile_root/installedApps/cell_name/app_name.ear/app_name.war).

With WAS you need to watch for static “<%@ include %>” directives in your JSPs. WAS will not reload included files unless you also update including JSP. A workaround here is to turn everything into “jsp:include” actions or use JSTL’s “c:import”. There might be a slight performance penalty for doing that but improved productivity is well worth it.

You can use Filesync plugin to synchronize your class files as well. This provides an alternative to the resource link-based approach that I described in the previous post. I still like using resource links better because they can be defined using Eclipse variables which makes it easier to share the configuration within a team. As far as I can tell, with Filesync you have to use absolute paths.

Here’s how the filesync configuration screen looks like:

Another good use of Filesync is to pull jar files from an external directory. Projects typically have a repository-like location where all third-party jars are checked-in (or it could be a full-blown Maven repository). You can easily add an external jar to your classpath in Eclipse. But how to put it under “WEB-INF/lib” where it needs to end up for the application server? With filesync it can be done easily by adding yet another folder pair.

In short, Filesync allows you to assemble your application “on the fly” without having to run an external build process. It also completely eliminates the need to explicitly update deployed applications.

Any developer wants to see the code changes instantaneously reflected in the application server.
However, when using WebSphere Application Server (WAS), developers usually have to go through the process of deploying an application to the server. Even thought the deployment support is integrated into Rational Application Developer (RAD) or Eclipse WTP, it still introduces delays and impedes productivity. Not to mention that Eclipse WTP does not actually support WAS 6.1 runtimes, only 6.0.

This is unfortunate because actually WAS 6.1 has good support for dynamic reloading. With dynamic reloading turned on, WAS monitors changes on the file system and automatically reloads the module (i.e., all classes loaded by the module’s classloader) when it detects a change. The reloading is almost instantaneous for simple modules. For complex modules with a lot of classes or initialization logic the reloading step could take a little bit of time but it is still faster than redeploying an entire application (you should check out Java Rebel if you want a truly instantaneous deployment).

With dynamic reloading all we need to do in order to make our changes available to the server is to update class files in the location where the deployed application resides. This is especially straightforward for web application and classes under WEB-INF/classes since WAS always explodes web application archives during deployment. In case of jar files (say the ones under WEB-INF/lib) the situation is a more complicated.

Unfortunately, the location of the deployed application is usually different from the workspace where a developer makes changes. By default, deployed binaries are located under profile_root/installedApps/cell_name. While this location can be changed, the directory structure will still be somewhat different from how code is organized in the workspace.

We could write a simple Ant script to copy changes, but this again introduces a special “pseudo-deployment” step. It would be nice if we could simply make a change in Eclipse, save it and let dynamic reloading kick in without any extra steps.

Turns out that it is quite possible to make WAS and Eclipse behave this way.

First, let’s configure WAS:

• Log in to WAS admin console and make sure that “Run in development mode” is checked for your sever. This is the default for standalone installations.
• Deploy your application to WAS using WAS admin console.
• For convenience, you may want to specify a non-standard location for application binaries during installation to shorten the path, e.g., “was_installed_apps”. This step is optional.
• Go to “Enterprise applications/your_app/Class loading and update detection”.
• Make sure that “reload classes” is checked.
• Set reload interval to some reasonable number, say “3”. By default it’s set to “0” which means “never”. IBM recommends 3 seconds as an optimal interval, although I’ve been using 1 second without any issues (for relatively small modules though).
• Stop and start the application.

Now let’s configure Eclipse. We will have to create a resource link pointing to the deployed application and configure the project to compile classes to the deployed location.

• Go to “Java Build Path” of the project. Click on “Browse” next to “Default output folder”.
• Click “Create New Folder…”, “Advanced”, check “Link to folder in the file system”.
• Click on “Browse” and locate the root of the exploded WAR file in the deployed application location. For example, for application “HelloWorldWeb” the path will be “profile_root/installedApps/cell_name/HelloWorldWeb.ear/HelloWorldWeb.war”. Give the link a meaningful name, e.g., “deployment”. Note: if you share .project and .classpath files with other developers, use Eclipse variables instead of the absolute path.
• Click OK. This will create a resource link that you can use to specify the output folder.
• Change the output folder to point to “project_name/link_name/WEB_INF/classes”, e.g., “HelloWorldWeb/deployment/WEB-INF/classes”. Click OK.
• Eclipse will recompile your project.
• From this point forward any class change will trigger dynamic reloading on the server.
• The resource link is also available in your package explorer, so you can browse and edit the deployed files. You need to be careful if you want to edit JSPs or other files that way as they will be overridden by the next full re-deployment.

This techniques takes care of class files only. Dynamic reloading of JSP files is a different story.

Note: This has been tested only with Eclipse 3.4 and WAS 6.1 and on modules with a relatively small code base. I’d be curious to know how effective this approach is for large modules.

Setting maximum heap size and changing other JVM parameters is a fairly common administration task. JVM configuration might be changing quite often during application development, usually as a result of performance testing.

Typically JVM parameters have to be updated for all application servers in a cell or at least for application servers that belong to a particular cluster.

Conveniently, WebSphere Application server supports updating JVM parameters using its administration APIs, specifically, AdminConfig object. This is illustrated by the script below.

import sys

"""
Change JVM heap size for all application servers in a cell
"""
# New heap size is passed as a parameter to the script
max_heap_size=sys.argv[0]

# Get the string with config ids of all serves
server_confids=AdminConfig.list("Server")
# Iterate over all servers - config ids are separated by \n 
for server_confid in server_confids.split("\n"):
    server_confid=server_confid.strip()
    # obtain the type - types are APPLICATION_SERVER, DEPLOYMENT_MANAGER, NODE_AGENT, WEB_SERVER
    server_type=AdminConfig.showAttribute(server_confid, "serverType")
    # we're changing the heap size for application servers - we want to exclude node agents, etc.
    if server_type == "APPLICATION_SERVER":
        server_name=AdminConfig.showAttribute(server_confid, "name")
        # this is the query to get JavaVirtualMachine configuration object for a particular server
        jvm_path="/Server:%s/JavaProcessDef:/JavaVirtualMachine:/" % server_name
        jvm_confid=AdminConfig.getid(jvm_path)
        # "modify" accepts  a list of lists - each list contains name and value (odd choice I must say, why not use tuples?)
        AdminConfig.modify(jvm_confid, [["maximumHeapSize", max_heap_size]])

# Commit our changes to the repository        
AdminConfig.save()

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.

Ant uses reflection to pass data from XML to the Java class that implement an Ant task. For every attribute in XML, you have to define a setter in the Java task’s class.

This works fine most of the time, however, in some cases there could be a need for a dynamic list of attributes. For example a task can pass attribute values to some external tool that has its own set of parameters that you don’t want to hardcode in Ant. Or you may simply like the flexibility of using dynamic attributes as opposed to predefined setters.

In order to implement dynamic attributes, first you need to override “maybeConfigure” method in your Ant task and have it do nothing:

public void maybeConfigure() throws BuildException {
}

Then in your “execute” method you can access the map of attributes (that represents all attributes set in XML) as follows:

RuntimeConfigurable configurator= getRuntimeConfigurableWrapper();
Map attributes=configurator.getAttributeMap();
String attr1=(String)attributes.get("attr1");

There are many instances when it is necessary to find members (servers) that constitute a cluster. For example, to script starting and stopping an application, we need to know names of servers and nodes where the application is installed. Or we may want to restart all members of a cluster after some configuration change.

As always, this is easy to do when you know what you're doing. A query-like activity in WAS environment usually heavily relies on AdminConfig object. Our example is no exception:

import sys

"""
List application servers that belong to a particular cluster
Cluster name is passed as a parameter to the script
"""
cluster_name=sys.argv[0]

# Get configID of the cluster
cluster_conf_id = AdminConfig.getid("/ServerCluster:"+cluster_name )
if not cluster_conf_id:
    raise "Cluster %s does not exist!" % cluster_name

# Get confids of the cluster members
member_conf_ids = AdminConfig.showAttribute(cluster_conf_id, "members")
# AdminConfig returns the list in [], get rid of the brackets
member_conf_ids = member_conf_ids[1:-1]
print "Cluster %s has following members:" % cluster_name  
# split by space
for member_conf_id in member_conf_ids.split():
    # Obtain server name and node name
    member_name=AdminConfig.showAttribute(member_conf_id, "memberName")
    node_name=AdminConfig.showAttribute(member_conf_id, "nodeName")
    print node_name+"/"+member_name

Now we can do a lot of useful things with node_name and member_name variables; for example, we can try to get an MBean of this server and check its state. Or we can attempt to restart it. I will cover this in one of the future posts.

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.

It is often desirable to "push" configuration changes to nodes in the cell without having to wait for the periodic node sync process to kick in. This becomes a requirement if you want to run some kind of automated testing as part of your configuration/deployment process (which is a very good practice). For example, you may have a build process which runs HTTPUnit tests immediately after the change.

In this case, you really don't want to rely on any artificial "sleeps" and delays in your script, so triggering synchronization programmatically is the best option.

This is actually quite simple to accomplish, as illustrated by the following script:

"""
Sync configuration changes with nodes
"""

# Commit changes to the configuration repository 
AdminConfig.save()

# Obtain deployment manager MBean
dm=AdminControl.queryNames("type=DeploymentManager,*")

# "syncActiveNodes can only be run on the deployment manager's MBean, 
# it will fail in standalone environment
if dm:
    print "Synchronizing configuration repository with nodes. Please wait..."
    # Force sync with all currently active nodes
    nodes=AdminControl.invoke(dm, "syncActiveNodes", "true")
    print "The following nodes have been synchronized: "+str(nodes)
else:
    print "Standalone server, no nodes to sync"

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.

AdminContol and AdminConfig are two key scripting objects that any WAS administrator must know by heart.

The objects are available from wsadmin WAS administration tool.

AdminConfig provides an interface to the WAS configuration repository available under profile_root/config. All AdminConfig services are provided by the Deployment Manager (assuming you're connecting remotely); there is no dependency on node agents.

AdminControl provides an interface to JMX managed beans (MBeans) in your WAS environment. In general, MBeans are only available for the resources that are running. For example, if an application server is down, the corresponding MBean will not be available.

AdminConfig and AdminContol could complement each other nicely, especially when you need to find out the status of "things" in a WAS cell. To illustrate the point, here is a simple script that prints the status of all application servers within a cell:

"""
List all application servers in a cell and print their status (up or down)
"""

# Get the string with config ids of all serves
server_confids=AdminConfig.list("Server")
#Iterate over all servers - config ids are separated by \n
for server_confid in server_confids.split("\n"):
    server_confid=server_confid.strip()
    # obtain the type - types are APPLICATION_SERVER, DEPLOYMENT_MANAGER, NODE_AGENT, WEB_SERVER
    server_type=AdminConfig.showAttribute(server_confid, "serverType")

    # we are only interested in application servers
    if server_type == "APPLICATION_SERVER":
        # obtain the name of the server
        server_name=AdminConfig.showAttribute(server_confid, "name")
        print "Checking the state of the server %s of type %s" % (server_name, server_type) 

        #Now we can try to get the mbean of the server. We could've used AdminConfig.getObjectName() as
        #a shortcut, but for the sake of the example we'll use AdminControl explicitly

        # Query to search for mbeans - note the wildcard at the end to match the rest of mbean parameters
        # Note that we're simplifying a little bit since we're ignoring node names -
        # server names may not be unique within cell
        server_query="type=Server,name=%s,*" % server_name
        server_mbean=AdminControl.queryNames( server_query )
        # AdminContol returns None if there is no running mbean for this server
        if server_mbean:
            print "Server is up"
        else:
            print "Server is down!"

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.

In one of the previous posts I complained that wsadmin tool (which is the main WebSphere Application Server administration tool) still relies on Jython 2.1, which is quite old.

The issue became critical when I realized that jython automatically re-compiles classes compiled with a different jython version. In my case, I was using Jython 2.2 for my Ant scripts and Jython 2.1 for wsadmin scripts. Some of the code was shared. This led to the situation of concurrent builds stepping on each other by dynamically re-compiling classes with different jython versions. The error looked something like that:

File "<string>", line 5, in ?
java.lang.ArrayIndexOutOfBoundsException: -4
at org.python.core.imp.createFromPyClass(Unknown Source)

Bugs like that are always fun to troubleshoot.

Going back to 2.1 was not an option for me since I use closures and “new classes” in quite a few places. So I tried putting jython 2.2.1 on wsadmin classpath and it worked without a hitch with both “thin” wsadmin client and a full-blown version. All my wsadmin scripts work without a problem. I suspect though that it may not fly with IBM support, so take this advise with a grain of salt.

Intuitively we all understand that automating IT operations, including builds, deployments, configurations, upgrades and so on is a good thing. We all know that humans make mistakes and mistakes can be costly when they affect a large group of people (e.g., a large user community) or otherwise result in lost revenue to a business.

But how error-prone human actions really are? Note that we’re not talking about normal error rate in software development or other creative fields. Clearly, it is very difficult if not impossible to eliminate errors from occurring when the task can’t be formalized. On the other hand, a large area of IT is related to operations and maintenance and it involves mostly predictive and repetitive tasks. There are many tools, from simple scripts to super-expensive enterprise products that deal with automating these types of tasks. Knowing the probability of human error would help us estimate potential benefits from these tools and, consequently, assess the return on investment.

The classic formula for calculating human reliability can be found here. Without going too much into the math, empirically we can ascertain the following:

• Every action performed by a human has a probability of error. It is never zero.
• Most tasks (at least, in IT) consist of multiple steps (actions). E.g., a change may have to be made on multiple severs.
• The likelihood of error goes up proportionally to the number of steps.

So it should not come as a surprise that the probability of error could be quite high for complex task. According to the data published on Ray Panko’s website, 28 experienced users on average had 33% error rate in a task involving 14 command-line-based steps. Another interesting tidbit from the same site is 50% error rate in following a checklist. It is unfortunate that the details of these studies are not documented on the site.

Of course, many of these errors can be caught and corrected via testing. It is common knowledge that every change has to be accompanied by some verification or “smoke” testing.

But some changes are impossible or very expensive to validate. Imagine having to change JVM maximum heap size to prevent an application running out of memory. Imagine also that this is a high volume application that runs on four different servers. Imagine further that one server out of four was not updated by mistake. You are not going to find out about it until the application starts crashing on that server under load – and this will be the worst time for dealing with this issue. Now, what if a parameter that had to be changed was some obscure garbage collection setting that was going to improve application’s performance. Users will be experiencing intermittent performance issues but there will be nothing explicitly pointing to the offending server. Discovering the root cause of the problem could take quite a while. The bottom line is that some errors can only be discovered by users at which point the cost of fixing them is going to be substantial.

I think that we tend to overestimate the reliability of human actions and underestimate the cost of fixing errors. After all, how hard could it be for an experience administrator to run a few commands? And it could be very easy but it still does not make the actions of this administrator any more reliable.

The bottom line that almost any investment in IT automation is well worth it. Unfortunately, this view is not uniformly accepted. Many organizations still live in stone age when it comes to automation, but that’s a subject for another post.

I’ve long been a fan of XML Appliances. Looks like IBM customers like the appliance idea as well. IBM said that DataPower was one of the top-selling products and also announced that DataPower-lution is one of their strategic directions for 2009. Basically, more and more edge functions will be moving into the appliance. And why not use the same device for XML acceleration, load balancing, crypto acceleration, caching, perhaps even WebSEAL replacement (it’s just a fancy reverse proxy after all). We’ll see how this vision plays out.

In a related news, there is finally a DataPower book and it’s 960 pages long. And this is before IBM started adding all these edge functions to the device :).

Using AdminTask in wsadmin often results in getting ugly stack traces. In fact, AdminTask always produces a full java stack trace even when the error is fairly innocuous (e.g., a resource name was mistyped). The stack trace in this situation is pretty useless; it could actually confuse operations staff as it might be interpreted as a problem in IBM code.

It is, in fact, quite easy to deal with this situation in Jython and suppress the annoying stack trace:

import sys
from com.ibm.ws.scripting import ScriptingException
...
try:
AdminTask....
except ScriptingException:
# note that we can't use "as" because of python 2.1 version, have to use sys
print "Error:n"+str(sys.exc_info()[1])

I finally registered on twitter. User name is aananiev. I’m planning to twit mostly on the topics related to this blog, so feel free to follow if you like the blog.

Long and unwieldy stack traces is a common occurrence when dealing with Java EE application servers. Here is an example. Many (if not all) of these products re-throw the same exception multiple times which complicates things even further. Figuring out the root cause of an exception is a major undertaking.

Of course, most of the trace is useless when using proprietary products since it points to classes that you don’t have the source code for. And not only you – level 1 of support most likely can’t get to the source code either. As a result, 90% of the trace has little to none immediate value.

As a rule, the more complex the product, the longer the stack trace. Makes sense, right? You’ve got more layers and components, each layer thinks that it is its duty to dump the whole thing to the log and re-throw.

May be we should start using stack traces as a code complexity metric. It would be much more telling than cyclomatic complexity.

I also think that there is a correlation between an average length of a stack trace and an average consulting rate that users of the product pay for development and support. So may be at the end of the day, developers and administrators should not grumble about it to much and I should just shut up.

I’ve started playing with WebSphere Application Sever 7.0 (the latest and greatest) and to my surprise discovered that it still uses Jython/Python 2.1 as a scripting language for its wsadmin tool.

Jython 2.2 has been around for quite a while and, from my experience, is very stable. So it’s odd that IBM choose not to upgrade it. The difference between 2.1 and 2.2 is quite significant, the biggest selling point of 2.2 is new style classes with a unified type model. Python 2.2 also supports properties. I don’t believe there is closures support in 2.1 either.

Why is it important? Well, using Java for WAS administration is hard; the API is obscure and poorly documented. This makes Jython the only game in town (with JACL deprecated some time back). So being able to use a modern version of Jython is highly desirable.

I’m still hoping that IBM might upgrade jython as part of one the minor upgrades; WAS 8.0 is probably a long time away.

Quick tips for those wishing to use thin wsadmin client with process server/WESB: add com.ibm.soacore.runtime_6.1.0.jar to the classpath.

Normally, you need the following jars for the client to work: com.ibm.ws.admin.client_6.1.0.jar, com.ibm.ws.security.crypto_6.1.0.jar com.ibm.ws.webservices.thinclient_6.1.0.jar.

However, if you want to use any of the SCA commands available in WESB, the extra jar is needed. Otherwise, you’ll be getting “class not found” for SCACommandException. The jar is available under wesb_root/plugins.

I don’t think it’s documented in infocenter.