We're pleased to announce the release of our new product, DataPower Buddy (dpbuddy). "dpbuddy" is a tool for automating administration of IBM WebSphere DataPower appliances. The tool supports export/import, file transfer, backup and many other functions.
dpbuddy is implemented as a set of custom tasks for the popular build tool, Apache Ant.
Here is a quick example of dpbuddy in action:
<dp:copy cleanDirectories="true">
<dpFileset prefix="/apps/services"
dir="services" includes="**/*.wsdl **/*.xsd"/>
</dp:copy>
This Ant task will remove remote directories if they exist, reproduce the local directory tree (all folders under "services") on the device and upload the necessary files based on the "includes" pattern.
dpbuddy is completely free; it can be downloaded from the dpbuddy product page
dpbuddy provides many cool features, including:
Go to dpbuddy product page to learn more.
PAnt 2.0.1 contains an important bug fix that was preventing using certain Ant tasks from python. Specifically, the bug affected all tasks that utilize addConfigured method to handle nested elements. This included the "manifest" task and several others.
You can download PAnt 2.0.1 from the PAnt project page
Application management, including deployment, has remained mostly unchanged since WAS 5.
WAS 7, however, introduced a new concept -- "Business Level Applications" (BLA). It would be more accurate to use the term "composite application" because this is what BLAs are -- it's a composition of Java EE applications (WARs or EARs) or other "assets".
Unfortunately, the term "composite application" is already used in a different context by IBM.
The word "asset" is another confusing and terribly overused term. In WAS7 it's a "catch all" word for any kind of artifact, including jar, zip, and, with the introduction of OSGi support, an OSGi bundle (you need to have a beta OSGI feature pack installed to be able to use OSGi).
BLAs are created by combining applications and assets together either using admin console or scripting. BLA configuration is saved under $profile_root/config/cells/$cell_name/blas. Assets are saved under $profile_root/config/cells/$cell_name/assets.
I can see two use cases for BLAs:
AdminTask.startBLA and AdminTask.startBLA provide an easy way to restart several "regular" applications.AdminTask.importAsset command. You can then reference the jar file using "SharedLibRelationship" option of AdminApp install/update commands. This will automatically create a classloader for this library (note--this might require server restart, at least for me it did). Being able to use regular administration APIs for managing shared libraries is certainly a welcome feature.Unfortunately, BLAs also have limitations:
To summarize, the BLA idea has some promise but its current implementation is too limited to be truly useful. I'm sure IBM will be evolving BLA in future WAS releases.
You can find more information about BLA in this article and in this webcast
If you're a WebSphere administrator or if you're trying to automate some tasks around the IBM WebSphere line of products, you'll inevitably be using WAS scripting (a.k.a. "wsadmin"). For the record, it is also possible to use Java and JMX-based APIs for automation, but it's is more complex and less documented. So "wsadmin" scripting tool is the primary tool of trade for any WebSphere administrator.
There are several things that you need to do in order to be successful with wsadmin:
-javaoption "-Dpython.path=your_path"). Do not forget to put python libraries (the ones that come with jython) on the python path. I prefer using thin administration client as this is more flexible. I also use 2.2.1 version of jython since jython 2.1 that comes with wsadmin is just too old and misses a lot of important language features.This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.
Most people assume that in order to run WAS admin scripts, you need to have full WAS installation available on the client machine. This is not correct. Starting with WAS 6.1, it is possible to use the so-called administration thin client instead.
The thin client has many benefits:
You'll need to make sure that you update your client jars whenever the server installation is upgraded.
Oh, and if you're using WebSphere ESB/Process Server and want to utilize some of WESB-specific admin tasks (available from AdminTask object), you're out of luck; the thin client will not support those.
This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.
You deploy a new version of your application into production environment, hit the application’s URL and get a 500 error with a long error stack and nasty “java.lang.ClassNotFoundException” in bold at the top.
“Class Not Found” exceptions could be quite tricky to troubleshoot because of the complexity of Java Web applications and application servers they run on. An average web application nowadays comes bundled with dozens of jar file (and probably thousands of classes). An average application server’s classpath is many pages long. Not to mention separately deployed libraries containing jar files shared by a group of applications. There should be little surprise that it is quite common for all these different jars and classloaders to clash with each other, get out of sync or become otherwise corrupt and mis-configured.
The list below represents a subset of all the possible causes of “ClassNotFoundException”. Hopefully this list could serve as a starting point for attacking the problem. The list was inspired by A List of Dumb Things to Check.
find . <del>name *.jar -print -exec jar -tvf {} ; | awk '/YOURSEARCHSTRING/ || /jar/ {print} ' (note-it won’t search within EAR and WAR files)jar -xvf? Does the jar indeed contain the package and class in question?versionInfo -long command.Once again, this is by no means a complete list. If anybody wants to contribute, please add a comment below and I’ll update the post.
You're part of a development team that just started working on a brand new Java EE application and you're asked to put together a build script for this app. "Nothing can be easier" you think and you quickly put together a simple Ant script or a Maven POM file. Your build compiles Java code, runs JUnit tests and creates a WAR file for your app. Your job is done and you move on to more exciting and important tasks.
Unfortunately, your simple build, while being a good starting point, does not accomplish much. Contrary to what many developers think, the purpose of an automated build is not to automate production of executable code (be it a WAR file or an "exe"). Its purpose is to verify correctness of the code and to discover as many problems and defects as possible as quickly as possible.
It is common knowledge that it is much less costly to fix a defect during construction than during testing phase:

According to the chart above (source: Six Sigma and IBM Systems Sciences Institute) it is two times more costly to fix a bug during testing than during implementation. I think this difference is actually much higher. Our short-term (working) memory is extremely volatile. According to some studies, the short term memory begins decaying after eighteen seconds. The cost of "context switching" for the brain is very high. In most organizations testing cycle takes at least a few weeks. This means that the bug that you just introduced will not be discovered for another few weeks at the earliest. When (or if!) it is finally discovered, most likely you'll be working on something entirely different. It will take at least a few hours for you just to recall all the details associated with the bug.
So the fact that your code compiles serves as a very weak indicator of code quality (although catching compilation problems early is important too, especially for large teams with high check-in volume). Automated testing must be done as part of every build. Most developers implement some automated testing using XUnit. In the majority of cases, these tests do not run against a deployed application, e.g., they do not hit a Web server. This kind of testing is useful, but it has its limitations. The main limitation is that we are not testing the application from the standpoint of its end users. For example, we're not testing AJAX logic running in a browser. Also, we're not testing the functionality that is dependent on an application server. Mock object frameworks help to a degree, but emulating application server's behavior could take some effort. Not to mention the fact that the "emulated" app server won't account for quirks of your "real" application server. In many cases there are subtle differences in app servers behavior, which is very often caused by differences in how classloader hierarchy is implemented. Reproducing these nuances using mock frameworks or even an embeddable servlet container, such as jetty, is impossible.
The bottom line is that your automated build has to be able to deploy your application and run tests against it. Using a browser-based testing tool such as Selenium will allow you to test your application as if it was used by end users, including testing of all your fancy AJAX features. Automating application deployments and testing does take some effort. Developing a comprehensive automated test suite could be a daunting tasks. But it is certainly possible and well worth it.
We've just updated PAnt documentation to reflect changes implemented in PAnt 2.0. There are two significat changes:
You can follow this guide to get started with PAnt.
We just added How to Get Started with PAnt page explaining PAnt setup process. The rest of PAnt documentation should get updated later this week.
We've just released a new version of PAnt, a tool for developing build scripts in python. The key new feature of PAnt 2.0 is the ability to create Ant targets directly from python. This allows for completely getting rid of Ant XML files and for developing build scripts entirely in Python.
Here is an example:
@target(unless="prop.name", depends=ptarget1)
def p_target2():
"""Python project ptarget2"""
ant.echo("echo ptarget2")
You can find more details about creating targets from PAnt here.
This is a major update to our popular python Ant wrapper.
Notable changes in this release:
More information is available from PAnt project page
Please subscribe to our feed or follow us on twitter to continue receiving updates about PAnt – new version is coming shortly.
Some application servers require that location of the development workspace has to be different from the location of the deployed application. For example, you can easily point Tomcat to the root of your Web application using “docBase” of the “Context” element. But you’re out of luck with WebSphere Application Server (WAS). You have to go through a separate application update process (using admin console or Rational Application Developer tooling) to synchronize your deployed application with the workspace. In my view, this update (a.k.a. “deployment”) step should never be required in a local development environment. It is one thing to have to deploy to a test or a production environment that consists of multiple servers that are segregated from the machine hosting the build artifacts. But in a situation when both the code and the application server are sitting on the same machine, the deployment step is redundant. We should be able to simply tell the app server where the code is and it can then do whatever is needed to load the code into JVM.
Luckily, we can get pretty close to this vision with a few very simple (and free) tools.
In my previous post I explained how to enable dynamic class reloading for WebSphere Application Server and avoid having to deploy your Java changes altogether. But what about changes to JSPs and other non-Java resources? How can we synchronize the directory used by the application server with the development workspace?
Turns out, there is an Eclipse plugin that does exactly that. It’s Filesync plugin developed by Andrei Loskutov.
As the name implies, the plugin automatically synchronizes workspace directories with external directories by doing one-way copy of changed files. It allows to specify multiple directory pairs and also to define include/exclude patterns and even use variable substitution.
To enable automatic updates of JSPs in the deployed application directory all you need to do is to define a folder pair that links web root in your workspace with the location of the exploded WAS directory in WAS (usually located under profile_root/installedApps/cell_name/app_name.ear/app_name.war).
With WAS you need to watch for static “<%@ include %>” directives in your JSPs. WAS will not reload included files unless you also update including JSP. A workaround here is to turn everything into “jsp:include” actions or use JSTL’s “c:import”. There might be a slight performance penalty for doing that but improved productivity is well worth it.
You can use Filesync plugin to synchronize your class files as well. This provides an alternative to the resource link-based approach that I described in the previous post. I still like using resource links better because they can be defined using Eclipse variables which makes it easier to share the configuration within a team. As far as I can tell, with Filesync you have to use absolute paths.
Here’s how the filesync configuration screen looks like:

Another good use of Filesync is to pull jar files from an external directory. Projects typically have a repository-like location where all third-party jars are checked-in (or it could be a full-blown Maven repository). You can easily add an external jar to your classpath in Eclipse. But how to put it under “WEB-INF/lib” where it needs to end up for the application server? With filesync it can be done easily by adding yet another folder pair.
In short, Filesync allows you to assemble your application “on the fly” without having to run an external build process. It also completely eliminates the need to explicitly update deployed applications.
Any developer wants to see the code changes instantaneously reflected in the application server.
However, when using WebSphere Application Server (WAS), developers usually have to go through the process of deploying an application to the server. Even thought the deployment support is integrated into Rational Application Developer (RAD) or Eclipse WTP, it still introduces delays and impedes productivity. Not to mention that Eclipse WTP does not actually support WAS 6.1 runtimes, only 6.0.
This is unfortunate because actually WAS 6.1 has good support for dynamic reloading. With dynamic reloading turned on, WAS monitors changes on the file system and automatically reloads the module (i.e., all classes loaded by the module’s classloader) when it detects a change. The reloading is almost instantaneous for simple modules. For complex modules with a lot of classes or initialization logic the reloading step could take a little bit of time but it is still faster than redeploying an entire application (you should check out Java Rebel if you want a truly instantaneous deployment).
With dynamic reloading all we need to do in order to make our changes available to the server is to update class files in the location where the deployed application resides. This is especially straightforward for web application and classes under WEB-INF/classes since WAS always explodes web application archives during deployment. In case of jar files (say the ones under WEB-INF/lib) the situation is a more complicated.
Unfortunately, the location of the deployed application is usually different from the workspace where a developer makes changes. By default, deployed binaries are located under profile_root/installedApps/cell_name. While this location can be changed, the directory structure will still be somewhat different from how code is organized in the workspace.
We could write a simple Ant script to copy changes, but this again introduces a special “pseudo-deployment” step. It would be nice if we could simply make a change in Eclipse, save it and let dynamic reloading kick in without any extra steps.
Turns out that it is quite possible to make WAS and Eclipse behave this way.
First, let’s configure WAS:
Now let’s configure Eclipse. We will have to create a resource link pointing to the deployed application and configure the project to compile classes to the deployed location.
This techniques takes care of class files only. Dynamic reloading of JSP files is a different story.
Note: This has been tested only with Eclipse 3.4 and WAS 6.1 and on modules with a relatively small code base. I’d be curious to know how effective this approach is for large modules.
Setting maximum heap size and changing other JVM parameters is a fairly common administration task. JVM configuration might be changing quite often during application development, usually as a result of performance testing.
Typically JVM parameters have to be updated for all application servers in a cell or at least for application servers that belong to a particular cluster.
Conveniently, WebSphere Application Server (WAS) supports updating JVM parameters using its administration APIs, specifically, AdminConfig object. This is illustrated by the script below.
import sys
"""
Change JVM heap size for all application servers in a cell
"""
# New heap size is passed as a parameter to the script
max_heap_size=sys.argv[0]
# Get the string with config ids of all serves
server_confids=AdminConfig.list("Server")
# Iterate over all servers - config ids are separated by \n
for server_confid in server_confids.split("\n"):
server_confid=server_confid.strip()
# obtain the type - types are APPLICATION_SERVER, DEPLOYMENT_MANAGER, NODE_AGENT, WEB_SERVER
server_type=AdminConfig.showAttribute(server_confid, "serverType")
# we're changing the heap size for application servers - we want to exclude node agents, etc.
if server_type == "APPLICATION_SERVER":
server_name=AdminConfig.showAttribute(server_confid, "name")
# this is the query to get JavaVirtualMachine configuration object for a particular server
jvm_path="/Server:%s/JavaProcessDef:/JavaVirtualMachine:/" % server_name
jvm_confid=AdminConfig.getid(jvm_path)
# "modify" accepts a list of lists - each list contains name and value (odd choice I must say, why not use tuples?)
AdminConfig.modify(jvm_confid, [["maximumHeapSize", max_heap_size]])
# Commit our changes to the repository
AdminConfig.save()
This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.
Ant uses reflection to pass data from XML to the Java class that implement an Ant task. For every attribute in XML, you have to define a setter in the Java task’s class.
This works fine most of the time, however, in some cases there could be a need for a dynamic list of attributes. For example a task can pass attribute values to some external tool that has its own set of parameters that you don’t want to hardcode in Ant. Or you may simply like the flexibility of using dynamic attributes as opposed to predefined setters.
In order to implement dynamic attributes, first you need to override “maybeConfigure” method in your Ant task and have it do nothing:
public void maybeConfigure() throws BuildException {
}
Then in your “execute” method you can access the map of attributes (that represents all attributes set in XML) as follows:
RuntimeConfigurable configurator= getRuntimeConfigurableWrapper();
Map attributes=configurator.getAttributeMap();
String attr1=(String)attributes.get("attr1");