DataPower Buddy 2.3 Release

You can find DPBuddy 2.3 announcement here.

Security Hardening of WebSphere Application Server Installations

It is a known fact that an out of the box a WAS installation with security enabled is not entirely secure. There is a number of steps that has to be taken in order to "harden" the installation. Most of the steps are documented in the WebSphere hardening guide, which should be closely studied and followed by any WAS administrator or developer. The hardening guide, however, is a little light on the specifics regarding how to secure WAS at the OS level. For example, hardening should normally include changing file and directory permissions to restrict access to sensitive configuration files.

Why bother hardening the installation at the OS level? This helps defend against external threats, i.e., a server hosting the installation is compromised and against internal threats, i.e., an unauthorized access from inside.

Here are some of the things you can do to secure your installation on a Unix/Linux platform:

• If you installed WAS as a non-root account, make sure that this account is not part of the "users" group. Create a separate group for all administrators, say "wasadmins". If you're doing this the post-install, make sure that you changed the ownership of all WAS files and profiles. To do that, bring all WAS processes down and run chgrp -R wasadmins $WAS_INSTALL_ROOT.
• Refrain from running WAS using root. There is no reason to do it on Unix/Linux, unless you absolutely have to use low ports. Yes, you can use sudo to tighten down access to root, but it is safer just to stay away from it. You can actually install WAS using root so it can be properly registered with the OS, but you can later change the installation to be owned by a non-root account.
• Following the principle of least privilege, remove read access from "others". Why do you need to do it? There are many sensitive files that are part of WAS install, including files containing encoded passwords (see below). It's better to err on a side of caution and disallow general read access altogether. We should revoke execute access as well. On the other hand, we may want to grant write/execute to all members of wasadmins groups, assuming the membership in this group is going to be tightly controlled:
  chmod -R 770 $WAS_INSTALL_ROOT
• You may want to allow access to logs to "others" so that developers can view logs. Since we just revoked read permissions from the root WAS install directory and from all profile directories, you can't just "chmod" the "logs" directory. You need to redirect logs to some other directory which is not under "$WAS_INSTLALL_ROOT" or profile roots. In order to do it, change LOG_ROOT WAS variable. You can do it from the console (Environment/WebSphere variables). By default, "LOG_ROOT" is defined at a node level. If the location is going to be the same on all nodes (which it should), just create LOG_ROOT at the cell level. You'll need to restart WAS servers after this. You can redirect logs to, say, /var/log/was. Make sure that "wasadmin" system account has write permissions to this directory.
• Do not allow login to the system account used to run WAS. You can read this guide explaining how to do it for various login mechanisms. You also need to disable "sudo su wasadmin" and only allow sudo for the specific commands/scripts that are needed to run WAS. You will need to add something like this to the sudoers file:
  %wasadmins ALL=(wasadmin) WAS_INSTLALL_ROOT/bin/, PROFILE_ROOT/bin/
  This configuration allows all member of "wasadmins" group to run commands from "bin" using "sudo -u", e.g., sudo -u wasadmin ./startServer.sh server1.
  You may relax this a little bit by allowing sudo access without having to enter password every time:
  %wasadmins ALL=(wasadmin) NOPASSWD: WAS_INSTLALL_ROOT/bin/, PROFILE_ROOT/bin/
  It's a bit of a pain having to run all commands via sudo (although, of course, it can be scripted), but it gives you complete audit trail of who did what in the environment. By the way, by default sudo logs via syslog, but you can redirect sudo's log into its own file by adding this directive in sudoers:
  Defaults logfile=PATH_TO_LOG_FILE
• Secure "sensitive" files. As any WAS administrator knows, passwords stored in WAS are not really encrypted, they are just encoded, so they can be decoded pretty easily (there is a way to plug in custom encryption, but that's a different story). Therefore, it is imperative to properly protect files containing encoded passwords. To do that, we're going to revoke any access to these files from "wasadmins" group. That way, only the "wasadmin" system account (which we can't even login into) or root can read these files. Encoded passwords are stored in "security.xml" and "wim/wimconfig.xml" (if you have LDAP configured). All these files reside under "$PROFILE_ROOT/config/cells/$CELL_NAME". "fileRegistry.xml" also contains passwords, although these passwords are properly encrypted.
  You may also want to protect keystores (key.p12). Note that in addition to the cell key store, each node has its own keystore. You don't need to worry as much about trust stores (trust.p12), in fact it might be convenient to keep group-level access to trust stores so that administrators can modify them outside of WAS (e.g., by adding a signer cert using ikeyman). While you're on it, make sure that you don't use the default password (WebAS) for any of the keystores/truststores.
  To revoke group access to any of the files mentioned above, just run "chmod 600" on them, e.g., chmod 600 security.xml. Note that you'll have to repeat it for each profile, including deployment manager and application server profiles (unless you have a standalone install).
  Node synchronization built into WAS fully supports changing permissions on config files. The changes will not disappear after node sync (provided you updated the DM profile).
• If your environment has a DMZ utilizing IHS, make sure that you don't have WAS installation on the DMZ boxes. It is convenient to have node agents running on the IHS servers, so you can manage IHS from admin console, but, unfortunately, it is a security risk. Deployment manager pushes all configuration files, including "security.xml" to all nodes, and we certainly don't want this file sitting in the DMZ (even with "600" permissions).
• Repeat all changes on every server containing WAS installation. You may want to script them or, better yet, use a tool like Puppet to automate OS-level configuration changes across multiple machines. For example, see this guide on how to centralize management of the sudoers file.

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you'd like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you're looking for help with any of these tasks, please let us know.

WebSphere Application Server Tutorial for Developers — WAS in 5 Minutes

If you're developing applications for WAS and you're new to it, this is what you need to know:

• How to view the logs. Logs can be found under install_root/profiles/profile_name/logs/server_name. The default profile name is AppSrv01 and the default server name is server1. Example:/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1. SystemOut.log is the file containing everything that was logged to standard out. Logs can also be viewed from the admin console by navigating to Troubleshooting/Logging and Tracing/server_name/Runtime.
• How to deploy an application. In admin console, navigate to Applications/Application Types/WebSphere enterprise applications, click on "Install new application", select "Fast path", accept all the defaults except that on "step 2" make sure that you targeted correct servers (if you have multiple servers/clusters in your environment). Note you can deploy a WAR file directly, you don't have to build an EAR. In this case, make sure that you set a context root on "step 4" screen.
• How to change context root of a Web application. Go to Applications/Application Types/WebSphere enterprise applications/application_name/Context Root For Web Modules in the console. Re-start the application after the change.
• How to change the order of classloaders. If you're getting a ClassNotFoundException when you're starting the app, changing the order of classloaders is the first thing you may want to try. Go to Applications/Application Types/WebSphere enterprise applications/application_name/Manage Modules/module_name and make the appropriate selection in the "Class loader order" drop-down (this assumes you're doing it for a WAR module).
• How to enable dynamic class reloading. If you need to frequently update your deployed application (e.g., you use a local WAS installation for development), enabling dynamic reloading could be a huge time saver. Go to your application in the console, "Class loading and update detection", set "Override class reloading settings ..." and set polling interval to 2 seconds. See this post for more details on how to configure your development environment to support class reloading.
• How to find a host name and a port of the server. Go to Server/Server Types/WebSphere application servers. You'll find the host name in the Host Name column. To find a port, click on your server, and expand Ports. WC_defaulthost is the HTTP port and WC_defaulthost_secure is the HTTPS port.
• How to start/stop a server. If you're dealing with the "Network Deployment" type of installation (multiple application servers running under the control of the "deployment manager"), your can start/stop a server from the console. Otherwise you have to do it from command line. Go to install_root/bin and run ./startServer.sh server_name, e.g., ./startServer.sh server1 (this assumes that your installation has only one profile defined, otherwise you may need to "cd" to the profile_name/bin directory). Make sure that you run all commands using the appropriate system account. To stop the server, run ./stopServer.sh server_name -username user_name -password password. user_name and password is the credentials of an admin account, typically the same one you use to login to the console.
• How to browse JMS messages. Go to Buses/Your bus name/Destinations/Your destination/Queue points/Your queue point/Runtime/Messages.

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you'd like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you're looking for help with any of these tasks, please let us know.

Tips for Developing JMS Client for WebSphere Application Server

Following are some tips for developing a JMS client for WAS.

• First, we need to put the right jar files on the client's classpath. Following jars are required: com.ibm.ws.sib.client.thin.jms_<WAS version>.jar, com.ibm.ws.ejb.thinclient_<WAS version>.jar. If you're using Oracle/Sun JDK, you'll also need com.ibm.ws.orb_<WAS version>.jar as IBM libraries rely on IBM ORB implementation. IBM JDK has it built-in but Oracle's obviously does not. These jars are available from the "runtimes" directory of your WAS installation. It is a good idea to make sure that the version of your client jars matches the version of your target WAS installation, including the fixpack level. You may want to update your client jars every time a new fixpack is installed.

• If you're not using DNS, you'll need to add the host name of the WAS server to the host file on your client machine, otherwise you're bound to get a cryptic ORB error message. You can find WAS host name by going to Server/Server types/ WebSphere Application Servers in admin console.
  By the way, most ORB error messages are cryptic; it is very rare to see the root cause of the problem in the message. The way to deal with it is to enable ORB tracing. Add -Dcom.ibm.CORBA.Debug=true to your JDK parameters; other debug/tracing parameters are explained here.

• Another frequent source of errors is a missing bootstrap property in the connection factory configuration. It is specified in the "provider endpoint" field and has a form of <host>:<SIB_Port>:BootstrapBasicMessaging. If this is not provided, WAS will attempt to bootstrap the messaging engine using localhost.

• Most JMS examples use JNDI lookup in order to get a queue object. It is more efficient to use session.crerateQueue() method. Note that in this case you need to provide the name of the JMS destination as opposed to the queue JNDI name. This is the "Queue name" parameter on the queue configuration screen in WAS admin console.

• Finally, not all developers are aware that WAS admin console has a built-in JMS queue/topic browser. It takes a few steps to get to it from the console: Buses/Your bus name/Destinations/Your destination/Queue points/Your queue point/Runtime/Messages.

DataPower Deployment Policies and XPath

Deployment policy in IBM DataPower appliance is a very nice concept. Deployment policies allow for "tweaking" device and domain configuration for different environments. Let's face it, there are always differences between environments. Sometimes, these differences are small, such as different back-end hosts, and in other cases these differences could be significant, such as different security policies. It is commendable that IBM recognized this fact and implemented a mechanism for dealing with this very common problem directly in their product.

Deployment policies do a decent job of dealing with differences between environments. Deployment policies support changing, adding and deleting configuration, so it is possible to implement fairly complex transformations.

However, dealing with deployment policies could be confusing. Deployment policy's match rules utilize xpath, but the syntax of the rules is not pure xpath (note -- I'm using 3.7 firmware. The syntax might be different in 3.8). Consider this simple deployment policy match rule:

The part before "?" looks like xpath. But what schema is this xpath based on? There is no "protocol" element in the DataPower XML management schema. The part after "?" that uses name-value parameters is even more odd. Why use this instead of proper xpath? After all, DataPower has an XML-processing engine with full xpath support, so it would certainly be more logical to rely on XML standards.

The bottom line is that while deployment policies are useful, they have limitations. They have to be developed using Deployment Policy builder in WebGUI. They can only be applied to configuration elements supported in WebGUI. For example, creating a deployment policy that updates RemoteEndpointPort of a Web Services Proxy proves to be a non-trivial task.

This is why we added support for "plain" xpath-based overrides in our DPBuddy DataPower management tool. Instead of dealing with the obscure syntax of deployment policies, developers can simply look at the configuration export file and specify an xpath expression against this file. "dp:setConfig" task of DPBuddy will update the matching element of this expression with the new value. For example, to update RemoteEndpointPort, one can define the following task:

<dp:setConfig>
    <configFile file="dpconfigs/config-wsproxy.xml">
        <override xpath="//RemoteEndpointPort" value="${endpoint.port}"/>
    </configFile>
</dp:setConfig>

It is not possible to add or remove configuration using "dp:setConfig"; deployment policies still have to be used for this kind of config changes. However, from our experience, majority of environment differences can be dealt with the xpath-based override mechanism.

MyArch, Inc. offers paid support for dpbuddy, including implementing custom enhancements. If you're interested, please contact us as info at myarch.com.

WebSphere 7 Business Level Applications

Application management, including deployment, has remained mostly unchanged since WAS 5.

WAS 7, however, introduced a new concept -- "Business Level Applications" (BLA). It would be more accurate to use the term "composite application" because this is what BLAs are -- it's a composition of Java EE applications (WARs or EARs) or other "assets".

Unfortunately, the term "composite application" is already used in a different context by IBM.

The word "asset" is another confusing and terribly overused term. In WAS7 it's a "catch all" word for any kind of artifact, including jar, zip, and, with the introduction of OSGi support, an OSGi bundle (you need to have a beta OSGI feature pack installed to be able to use OSGi).

BLAs are created by combining applications and assets together either using admin console or scripting. BLA configuration is saved under $profile_root/config/cells/$cell_name/blas. Assets are saved under $profile_root/config/cells/$cell_name/assets.

I can see two use cases for BLAs:

• BLAs can be used to start and stop multiple Java EE applications at once. This might be useful if operationally several applications have to be treated together or if there are run-time dependencies between applications. This is especially the case if scripting is used; AdminTask.startBLA and AdminTask.startBLA provide an easy way to restart several "regular" applications.

• BLAs can be used to manage shared libraries. There is no way to deploy "traditional" shared libraries ("Environment/Shared Libraries") using Deployment Manager; you have to use scp or ftp to copy binaries to target servers. With BLA you can simply import "assets" (e.g., a jar file) using AdminTask.importAsset command. You can then reference the jar file using "SharedLibRelationship" option of AdminApp install/update commands. This will automatically create a classloader for this library (note--this might require server restart, at least for me it did). Being able to use regular administration APIs for managing shared libraries is certainly a welcome feature.

Unfortunately, BLAs also have limitations:

• I was not able to find a way to create an asset consisting of multiple jars. This severely limits the usefulness of utilizing BLAs for managing shared libraries since a library will almost always consist of multiple jars. It is possible to import jars individually, but this is tedious and difficult to manage.

• It is not possible to deploy multiple Java EE applications using BLAs. You still have to deploy applications individually.

• There is no support for version information in BLA assets. This is unfortunate. I wish IBM though about integrating BLA assets with Maven repositories and supporting true dependency management (e.g., version resolution, transitive dependencies, etc.). As of right now the only way to import multiple version of the same asset is to append a version number to the asset's file name.

To summarize, the BLA idea has some promise but its current implementation is too limited to be truly useful. I'm sure IBM will be evolving BLA in future WAS releases.

You can find more information about BLA in this article and in this webcast

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you'd like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you're looking for help with any of these tasks, please let us know.

WebSphere Adminstration: Getting Started with WebSphere Application Server Scripting

If you're a WebSphere administrator or if you're trying to automate some tasks around the IBM WebSphere line of products, you'll inevitably be using WAS scripting (a.k.a. "wsadmin"). For the record, it is also possible to use Java and JMX-based APIs for automation, but it's is more complex and less documented. So "wsadmin" scripting tool is the primary tool of trade for any WebSphere administrator.

There are several things that you need to do in order to be successful with wsadmin:

• Learn Python. Wsadmin uses Python as a language (JACL was deprecated a long time ago and should not be used) and relies on jython. Jython provides full implementation of Python language sans some libraries. Needless to say that good knowledge of Python is essential for developing wsadmin scripts. Most WebSphere admins have Java background. In my opinion (and speaking from personal experience), knowing Java does more harm than good for a Python developer. Java people usually speak Python "with accent" and have a hard time properly utilizing all python capabilities such as closures, list comprehensions and even modules and packages. It's no wonder, the two languages are very different. For example, in Java everything is a class and in most cases a class resides in its own file. In Python a module can encompass multiple classes and classes themselves are completely optional. Learning how to properly use modules and classes takes some mental shift.

• Invest in designing your scripts. I'm not even sure if "scripting" is the right word to use in this context. Scripts are usually relatively simple. Wsadmin scripts are anything but. A seemingly simple task of application deployment may require many lines of code if you want to do it right. Unless you're really trying to do something unsophisticated (e.g., setting heap size), you'll be developing a full-blown python application. Therefore, you'll need to think about its design. Python has excellent support for modules. You'll have at least one module in your application. Do not put any logic into the "main" file that you'll be calling from wsadmin. Instead, define appropriate functions and classes in your modules and call them from the main file.

• Invest in setting up your development environment. Notepad or vi are not going to cut it. Remember, Python lacks compiler, so the only way to find out if your code is even syntactically correct is to execute it. That is, unless you use an IDE, such as PyDev. The latest version of PyDev is very good at detecting syntax errors and many other issues, such as uninitialized variables. It also has very decent (for a dynamic language) code completion.
  You will also need to setup your wsadmin shell script to supply python path (using -javaoption "-Dpython.path=your_path"). Do not forget to put python libraries (the ones that come with jython) on the python path. I prefer using thin administration client as this is more flexible. I also use 2.2.1 version of jython since jython 2.1 that comes with wsadmin is just too old and misses a lot of important language features.
  Another practical piece of advice is to never use "wsadmin -f" to test your scripts. Wsadmin takes eternity to come up, so running it every time you want to run your scripts is simply counterproductive. Instead, load it once and then run your script using Python "execfile" function. Don't forget to call reload function for all modules that are part of your applications, otherwise you won't see the changes until your re-start wsadmin.

• Help function is your best friend, so learn how to use it. I'm talking about the "help" function available for all key wsadmin modules (AdminConfig, AdminControl, etc). Wsadmin APIs, in my opinion, are poorly organized and difficult to learn. Ten thousand options available for AdminApp.install function is an example of that. Also, "lists of lists" data structures used by AdminTask and AdminApp are truly baffling. In any event, I usually run a separate wsadmin window that I use solely for consulting "help". This is much faster than searching for the same information in InfoCenter.

• Finally, you will need a testing environment. You can do some development using a standalone WAS installation but eventually you'll need to test your code against a Network Deployment environment with deployment manager and, ideally, multiple servers and multiple nodes. In other words, your test environment should resemble the target environments where you're going to run your scripts. There are many things that are impossible to test with just a standalone profile, one such example is node synchronization.

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you'd like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you're looking for help with any of these tasks, please let us know.

WebSphere Administration: Using Thin Administration Client

Most people assume that in order to run WAS admin scripts, you need to have full WAS installation available on the client machine. This is not correct. Starting with WAS 6.1, it is possible to use the so-called administration thin client instead.

The thin client has many benefits:

• The client consists of only two jars approximately 33MB in size, so you can set it up anywhere. With the traditional wsadmin client, you have to get WAS installed on your machine just so you can run wsadmin scripts. This is especially tedious if you have to deal with multiple versions of WebSphere products (and most large organizations have WAS versions in parallel). With the thin client you can just copy the two jars from an existing WAS installation onto a client machine. The only catch is that IBM JDK (or, more precisely, a JDK containing IBM security provider -- on Sun and HP-UX WAS actually comes with the Sun JDK) is still required. You can copy it from an existing install as well (I hope it doesn't violate any licensing terms).

• Since the client's size is so small, you can add it to your version control or to your Maven repository. This makes the build more repeatable for developers (your build does include deploying to the application server and running tests against it, doesn't it?).

• The shell script to run the thin client could be easily customized. For example, you may want to add some Java classes to wsadmin classpath. This will make it possible to invoke these classes from your wsadmin scripts. Or you may want to use a more modern jython version for your scripts. In fact IBM does't supply a shell script with the thin client, they just provide an example .

You'll need to make sure that you update your client jars whenever the server installation is upgraded.

Oh, and if you're using WebSphere ESB/Process Server and want to utilize some of WESB-specific admin tasks (available from AdminTask object), you're out of luck; the thin client will not support those.

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you'd like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you're looking for help with any of these tasks, please let us know.

ClassNotFoundException: A List of Dumb Things to Check

You deploy a new version of your application into production environment, hit the application’s URL and get a 500 error with a long error stack and nasty “java.lang.ClassNotFoundException” in bold at the top.

“Class Not Found” exceptions could be quite tricky to troubleshoot because of the complexity of Java Web applications and application servers they run on. An average web application nowadays comes bundled with dozens of jar file (and probably thousands of classes). An average application server’s classpath is many pages long. Not to mention separately deployed libraries containing jar files shared by a group of applications. There should be little surprise that it is quite common for all these different jars and classloaders to clash with each other, get out of sync or become otherwise corrupt and mis-configured.

The list below represents a subset of all the possible causes of “ClassNotFoundException”. Hopefully this list could serve as a starting point for attacking the problem. The list was inspired by A List of Dumb Things to Check.

• To start, determine a type of the offending class. Is it a an application class, a third-party library class, a class provided by the application server or a JDK class? Determine the jar file that should contain the class. Determine where that jar should be located on the file system. Is it part of application installation, application server installation or some shared library installation? You may need to search for the class within multiple jars. Here is the command to do it (source): find . <del>name *.jar -print -exec jar -tvf {} ; | awk '/YOURSEARCHSTRING/ || /jar/ {print} ' (note-it won’t search within EAR and WAR files)
• Does the jar that’s supposed to contain the class exist on the file system?
• Are you able to “unjar” the jar using jar -xvf? Does the jar indeed contain the package and class in question?
• Check the version of the jar if you can’t find the class there. To determine the version, look at the jar’s MANIFEST.MF. Usually (but, unfortunately, not always) you will find some version information there. You can also compare the file size with the “baseline”.
• Does the account that the application server’s JVM was started with have read access to the jar? An application server usually runs under some sort of a system account. The jar might have been copied to the file system using a personal account from a different group.
• Have all application jars been updated during deployment? Are all the jars (including shared libraries) at the right version? Manual deployment process is quite common, so missing to update a jar is always a possibility.
• Is the class in question packaged with the application (e.g., under WEB-INF/lib) and being loaded by one of the parent classloaders? Most application servers utilize a classloader hierarchy where WAR file’s classloader is a child of EAR classloader which in turn is a child of the system (JVM) classloader. Parent classloaders can’t go down to request a class from a child classloader. The problem occurs if, for example, some jars were moved to a shared library but they still depend on classes packaged with the application.
  In order to diagnose this situation, you’ll need to have a good understanding of your application server’s classloader hierarchy. Here is the information for WebSphere and here is the WebLogic documentation on classloaders. Here is another article covering WebSphere classloaders in depth.
• Is any of the jars packaged with the application also present on any of the parent classloader’s classpath? Running different versions of the same jar or library can cause all kinds of issues, including ClassNotFoundException. Some app servers allow overriding default classloader behavior so that the jars packaged with the application are loaded first. This could fix the problem.
• If the jar with the class in question is part of a shared library (as opposed to packaged with the application), check if this library was made available to the application via the classloader configuration. For example, WebSphere configuration involves setting up a separate classloader for the library and explicitly associating it with the application.
• Is the version and patch level of the application server correct? Does it match your development environment? Look at the detailed version information for all the different components of your app servers and also get a list of installed patches. E.g., for WebSphere run versionInfo -long command.
• Is the application server running under the right JDK? E.g., check if the server startup script relies on JAVA_HOME and see which JDK the JAVA_HOME points to.
• If the application runs in a cluster, does the problem occur on all nodes or just on some? Are you trying to troubleshoot the problem on the right node?
• If the classname is driven from a string, either in java source or some other file, have you spelled the class name correctly? (Steve Loughran)

Once again, this is by no means a complete list. If anybody wants to contribute, please add a comment below and I’ll update the post.

Eliminate the Need to Redeploy Your Web Files

Some application servers require that location of the development workspace has to be different from the location of the deployed application. For example, you can easily point Tomcat to the root of your Web application using “docBase” of the “Context” element. But you’re out of luck with WebSphere Application Server (WAS). You have to go through a separate application update process (using admin console or Rational Application Developer tooling) to synchronize your deployed application with the workspace. In my view, this update (a.k.a. “deployment”) step should never be required in a local development environment. It is one thing to have to deploy to a test or a production environment that consists of multiple servers that are segregated from the machine hosting the build artifacts. But in a situation when both the code and the application server are sitting on the same machine, the deployment step is redundant. We should be able to simply tell the app server where the code is and it can then do whatever is needed to load the code into JVM.

Luckily, we can get pretty close to this vision with a few very simple (and free) tools.

In my previous post I explained how to enable dynamic class reloading for WebSphere Application Server and avoid having to deploy your Java changes altogether. But what about changes to JSPs and other non-Java resources? How can we synchronize the directory used by the application server with the development workspace?

Turns out, there is an Eclipse plugin that does exactly that. It’s Filesync plugin developed by Andrei Loskutov.

As the name implies, the plugin automatically synchronizes workspace directories with external directories by doing one-way copy of changed files. It allows to specify multiple directory pairs and also to define include/exclude patterns and even use variable substitution.

To enable automatic updates of JSPs in the deployed application directory all you need to do is to define a folder pair that links web root in your workspace with the location of the exploded WAS directory in WAS (usually located under profile_root/installedApps/cell_name/app_name.ear/app_name.war).

With WAS you need to watch for static “<%@ include %>” directives in your JSPs. WAS will not reload included files unless you also update including JSP. A workaround here is to turn everything into “jsp:include” actions or use JSTL’s “c:import”. There might be a slight performance penalty for doing that but improved productivity is well worth it.

You can use Filesync plugin to synchronize your class files as well. This provides an alternative to the resource link-based approach that I described in the previous post. I still like using resource links better because they can be defined using Eclipse variables which makes it easier to share the configuration within a team. As far as I can tell, with Filesync you have to use absolute paths.

Here’s how the filesync configuration screen looks like:

Another good use of Filesync is to pull jar files from an external directory. Projects typically have a repository-like location where all third-party jars are checked-in (or it could be a full-blown Maven repository). You can easily add an external jar to your classpath in Eclipse. But how to put it under “WEB-INF/lib” where it needs to end up for the application server? With filesync it can be done easily by adding yet another folder pair.

In short, Filesync allows you to assemble your application “on the fly” without having to run an external build process. It also completely eliminates the need to explicitly update deployed applications.

Instantly Redeploy Your Classes to WebSphere Application Server

Any developer wants to see the code changes instantaneously reflected in the application server.
However, when using WebSphere Application Server (WAS), developers usually have to go through the process of deploying an application to the server. Even though the deployment support is integrated into Rational Application Developer (RAD) or Eclipse WTP, it still introduces delays and impedes productivity. Not to mention that Eclipse WTP does not actually support WAS 6.1 runtimes, only 6.0.

This is unfortunate because actually WAS 6.1 has good support for dynamic reloading. With dynamic reloading turned on, WAS monitors changes on the file system and automatically reloads the module (i.e., all classes loaded by the module’s classloader) when it detects a change. The reloading is almost instantaneous for simple modules. For complex modules with a lot of classes or initialization logic the reloading step could take a little bit of time but it is still faster than redeploying an entire application (you should check out Java Rebel if you want a truly instantaneous deployment).

With dynamic reloading all we need to do in order to make our changes available to the server is to update class files in the location where the deployed application resides. This is especially straightforward for web application and classes under WEB-INF/classes since WAS always explodes web application archives during deployment. In case of jar files (say the ones under WEB-INF/lib) the situation is a more complicated.

Unfortunately, the location of the deployed application is usually different from the workspace where a developer makes changes. By default, deployed binaries are located under profile_root/installedApps/cell_name. While this location can be changed, the directory structure will still be somewhat different from how code is organized in the workspace.

We could write a simple Ant script to copy changes, but this again introduces a special “pseudo-deployment” step. It would be nice if we could simply make a change in Eclipse, save it and let dynamic reloading kick in without any extra steps.

Turns out that it is quite possible to make WAS and Eclipse behave this way.

First, let’s configure WAS:

• Log in to WAS admin console and make sure that “Run in development mode” is checked for your sever. This is the default for standalone installations.
• Deploy your application to WAS using WAS admin console.
• For convenience, you may want to specify a non-standard location for application binaries during installation to shorten the path, e.g., “was_installed_apps”. This step is optional.
• Go to “Enterprise applications/your_app/Class loading and update detection”.
• Make sure that “reload classes” is checked.
• Set reload interval to some reasonable number, say “3”. By default it’s set to “0” which means “never”. IBM recommends 3 seconds as an optimal interval, although I’ve been using 1 second without any issues (for relatively small modules though).
• Stop and start the application.

Now let’s configure Eclipse. We will have to create a resource link pointing to the deployed application and configure the project to compile classes to the deployed location.

• Go to “Java Build Path” of the project. Click on “Browse” next to “Default output folder”.
• Click “Create New Folder…”, “Advanced”, check “Link to folder in the file system”.
• Click on “Browse” and locate the root of the exploded WAR file in the deployed application location. For example, for application “HelloWorldWeb” the path will be “profile_root/installedApps/cell_name/HelloWorldWeb.ear/HelloWorldWeb.war”. Give the link a meaningful name, e.g., “deployment”. Note: if you share .project and .classpath files with other developers, use Eclipse variables instead of the absolute path.
• Click OK. This will create a resource link that you can use to specify the output folder.
• Change the output folder to point to “project_name/link_name/WEB_INF/classes”, e.g., “HelloWorldWeb/deployment/WEB-INF/classes”. Click OK.
• Eclipse will recompile your project.
• From this point forward any class change will trigger dynamic reloading on the server.
• The resource link is also available in your package explorer, so you can browse and edit the deployed files. You need to be careful if you want to edit JSPs or other files that way as they will be overridden by the next full re-deployment.

This techniques takes care of class files only. Dynamic reloading of JSP files is a different story.

Note: This has been tested only with Eclipse 3.4 and WAS 6.1 and on modules with a relatively small code base. I’d be curious to know how effective this approach is for large modules.

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you’d like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you’re looking for help with any of these tasks, please let us know.

WebSphere Administration: Setting Heap Size

Setting maximum heap size and changing other JVM parameters is a fairly common administration task. JVM configuration might be changing quite often during application development, usually as a result of performance testing.

Typically JVM parameters have to be updated for all application servers in a cell or at least for application servers that belong to a particular cluster.

Conveniently, WebSphere Application Server (WAS) supports updating JVM parameters using its administration APIs, specifically, AdminConfig object. This is illustrated by the script below.

import sys

"""
Change JVM heap size for all application servers in a cell
"""
# New heap size is passed as a parameter to the script
max_heap_size=sys.argv[0]

# Get the string with config ids of all serves
server_confids=AdminConfig.list("Server")
# Iterate over all servers - config ids are separated by \n 
for server_confid in server_confids.split("\n"):
    server_confid=server_confid.strip()
    # obtain the type - types are APPLICATION_SERVER, DEPLOYMENT_MANAGER, NODE_AGENT, WEB_SERVER
    server_type=AdminConfig.showAttribute(server_confid, "serverType")
    # we're changing the heap size for application servers - we want to exclude node agents, etc.
    if server_type == "APPLICATION_SERVER":
        server_name=AdminConfig.showAttribute(server_confid, "name")
        # this is the query to get JavaVirtualMachine configuration object for a particular server
        jvm_path="/Server:%s/JavaProcessDef:/JavaVirtualMachine:/" % server_name
        jvm_confid=AdminConfig.getid(jvm_path)
        # "modify" accepts  a list of lists - each list contains name and value (odd choice I must say, why not use tuples?)
        AdminConfig.modify(jvm_confid, [["maximumHeapSize", max_heap_size]])

# Commit our changes to the repository        
AdminConfig.save()

This post is part of the series on WebSphere Application Server administration. Please subscribe to our blog if you'd like to receive updates.

Note: We offer professional services in the area of WebSphere architecture, implementation and operations. If you're looking for help with any of these tasks, please let us know.

WebSphere Administration: Finding Cluster Members

There are many instances when it is necessary to find members (servers) that constitute a cluster. For example, to script starting and stopping an application, we need to know names of servers and nodes where the application is installed. Or we may want to restart all members of a cluster after some configuration change.

As always, this is easy to do when you know what you're doing. A query-like activity in WAS environment usually heavily relies on AdminConfig object. Our example is no exception:

import sys

"""
List application servers that belong to a particular cluster
Cluster name is passed as a parameter to the script
"""
cluster_name=sys.argv[0]

# Get configID of the cluster
cluster_conf_id = AdminConfig.getid("/ServerCluster:"+cluster_name )
if not cluster_conf_id:
    raise "Cluster %s does not exist!" % cluster_name

# Get confids of the cluster members
member_conf_ids = AdminConfig.showAttribute(cluster_conf_id, "members")
# AdminConfig returns the list in [], get rid of the brackets
member_conf_ids = member_conf_ids[1:-1]
print "Cluster %s has following members:" % cluster_name  
# split by space
for member_conf_id in member_conf_ids.split():
    # Obtain server name and node name
    member_name=AdminConfig.showAttribute(member_conf_id, "memberName")
    node_name=AdminConfig.showAttribute(member_conf_id, "nodeName")
    print node_name+"/"+member_name

Now we can do a lot of useful things with node_name and member_name variables; for example, we can try to get an MBean of this server and check its state. Or we can attempt to restart it. I will cover this in one of the future posts.

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.

WebSphere Administration: How to Force Node Synchronization

It is often desirable to "push" configuration changes to nodes in the cell without having to wait for the periodic node sync process to kick in. This becomes a requirement if you want to run some kind of automated testing as part of your configuration/deployment process (which is a very good practice). For example, you may have a build process which runs HTTPUnit tests immediately after the change.

In this case, you really don't want to rely on any artificial "sleeps" and delays in your script, so triggering synchronization programmatically is the best option.

This is actually quite simple to accomplish, as illustrated by the following script:

"""
Sync configuration changes with nodes
"""

# Commit changes to the configuration repository 
AdminConfig.save()

# Obtain deployment manager MBean
dm=AdminControl.queryNames("type=DeploymentManager,*")

# "syncActiveNodes" can only be run on the deployment manager's MBean, 
# it will fail in standalone environment
if dm:
    print "Synchronizing configuration repository with nodes. Please wait..."
    # Force sync with all currently active nodes
    nodes=AdminControl.invoke(dm, "syncActiveNodes", "true")
    print "The following nodes have been synchronized: "+str(nodes)
else:
    print "Standalone server, no nodes to sync"

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.

WebSphere Administration: AdminControl vs AdminConfig

AdminContol and AdminConfig are two key scripting objects that any WAS administrator must know by heart.

The objects are available from wsadmin WAS administration tool.

AdminConfig provides an interface to the WAS configuration repository available under profile_root/config. All AdminConfig services are provided by the Deployment Manager (assuming you're connecting remotely); there is no dependency on node agents.

AdminControl provides an interface to JMX managed beans (MBeans) in your WAS environment. In general, MBeans are only available for the resources that are running. For example, if an application server is down, the corresponding MBean will not be available.

AdminConfig and AdminContol could complement each other nicely, especially when you need to find out the status of "things" in a WAS cell. To illustrate the point, here is a simple script that prints the status of all application servers within a cell:

"""
List all application servers in a cell and print their status (up or down)
"""

# Get the string with config ids of all serves
server_confids=AdminConfig.list("Server")
#Iterate over all servers - config ids are separated by \n
for server_confid in server_confids.split("\n"):
    server_confid=server_confid.strip()
    # obtain the type - types are APPLICATION_SERVER, DEPLOYMENT_MANAGER, NODE_AGENT, WEB_SERVER
    server_type=AdminConfig.showAttribute(server_confid, "serverType")

    # we are only interested in application servers
    if server_type == "APPLICATION_SERVER":
        # obtain the name of the server
        server_name=AdminConfig.showAttribute(server_confid, "name")
        print "Checking the state of the server %s of type %s" % (server_name, server_type) 

        #Now we can try to get the mbean of the server. We could've used AdminConfig.getObjectName() as
        #a shortcut, but for the sake of the example we'll use AdminControl explicitly

        # Query to search for mbeans - note the wildcard at the end to match the rest of mbean parameters
        # Note that we're simplifying a little bit since we're ignoring node names -
        # server names may not be unique within cell
        server_query="type=Server,name=%s,*" % server_name
        server_mbean=AdminControl.queryNames( server_query )
        # AdminContol returns None if there is no running mbean for this server
        if server_mbean:
            print "Server is up"
        else:
            print "Server is down!"

This post is part of the series on WebSphere Application Server administration. Please subscribe to this blog if you'd like to receive updates.