Update on DPBuddy Release 3.0

DPBuddy 3.0 is almost ready to go, a beta version should become available in March this year.
We're going to deliver pretty much all the features that were announced in the roadmap and more:

• Task for quiescing domains and objects. This task is able to wait for objects to go into the target state. This should be useful for fully automating "quiesce-import-unquiesce" deployment cycle.
• XPath-based transformation of DataPower configuration (or any XML file for that matter). This allows for creating environment-specific transformations of any complexity. Adding, deleting and updating elements is supported. Setting text in attributes and elements (similar to the current <a href="/datapower-deployment-policies-and-xpath"override feature of setConfig task) now supports expressions. E.g., if you wanted to add a prefix to the current server value in a load-balancing group, you could do this: <setText xpath="//LBGroupMembers/Server" expression="dev.+#currentValue"/>. All these features are supported for both import (including importing zip files) and copy tasks.
• Task for verifying operational state of objects. E.g., you may want to verify that all objects/services are in "UP" state after import.
• Integration of DataPower-related environment variables (e.g., a port number) with the prefix-based device definition, so all environment-specific properties could be defined in one place: dev.backend.port=9080, prod.backend.port=80.
• Task for downloading multiple files from the device
• Task for deleting DataPower objects, services and domains.
• Task for secure backup (execute secure backup and download the resulting file).
• Downloading of log files from the device.
• Improved error reporting for import and other tasks.

Stay tuned for more updates.

Using Eclipse for DataPower Development

If you're using Eclipse for developing IBM WebSphere DataPower artifacts (xslt, schemas, etc.) you can easily configure DPBuddy to automatically copy your artifacts to the target device and do various other chores. Since DPBuddy relies on Apache Ant, all you need to do is to define an Ant file with DPBuddy tasks and configure the Ant file as a builder for your project.

Here are the steps:

(more...)

Using Auto-complete in Eclipse Ant Editor for DPBuddy Tasks

Many developers utilize Eclipse for editing Ant files. Ant editor in Eclipse is fairly powerful, this article provides a good overview of its capabilities.

One of the most useful features is auto-complete for Ant tasks. Just press Ctrl-space and you'll see all attributes and nested elements supported by the task.

Auto-complete comes very handy when working with DPBuddy Ant tasks (and for developing WebSphere DataPower-related artifacts in general).

Just type '<dp:' (assuming that this is the prefix you assigned to DPBuddy's antlib) and then Ctrl-space and you'll see the list of all DPBuddy's tasks. Pressing Ctrl-space anywhere after the name of the task shows the list of attributes or nested elements.

Unfortunately, auto-complete won't display help for custom tasks. Apparently description of Ant tasks is embedded inside Eclipse and does not seem to be extendable.

DataPower Buddy 2.3.2 Bug Fix Release

Release 2.3.2 of dpbuddy fixes a single but significant issue preventing copying files (using dp:copy command) to the hard disk array. RAID is mounted under local: filesystem on the device; the name of the mount point can be found under Administration/Storage Devices/Hard Disk Array in WebGUI.

SOMA "CreateDir" action errors out when it is used to re-create the RAID mount directory. In this release of dpbuddy logic was added to ignore the error.

The updated DPBuddy package can be downloaded from dpbuddy download page.

Managing DataPower Using Maven

Please refer to this article.

DPBuddy Roadmap for Release 3.0

We're currently working on the next version of dpbuddy, our popular tool for managing IBM WebSphere DataPower appliances.
We're planning to implement the following new features:

• Tasks for quiescing domains and objects. The task will be able to wait for domain or object to go into the quiesced state..
• DPBuddy's xpath-based alternative to deployment policy can be used with import and even with copying files. Currently this feature is only supported with setConfig task, whereas majority of dpbuddy users rely on import. In other words, you'll be able to apply a simple xpath expression to make environment-specific changes to your configuraiton or files.
• Integration of DataPower-related environment variables (e.g., a port number) with the prefix-based device definition.
• Task for downloading multiple files from the device
• Task for deleting configuration and domains.
• Task for secure backup (execute secure backup and download the resulting file).
• Automated downloading of the log file from the device in case of errors.
• Tasks for various device management "actions".

What else would you like to see in the new version? Please comment on this post or in our google group.

DataPower Buddy 2.3 Release

You can find DPBuddy 2.3 announcement here.

DataPower Secure Backup with DPBuddy

Secure backup is the only way to backup the entire device, including keys and certificates. Therefore, it is desirable to run it on a regular basis.

You can easily do it with dpbuddy using its support for "do-action" command. Secure backup is simply one of the actions with a few additional parameters.

Here's how it could be expressed using Ant and dpbuddy. Note that we're using "local://" filesystem as a destination; in reality you probably want to use "ftp:" instead.

<target name="secure-backup" description="Backup the entire device">
    <tstamp/>
    <dp:action>
        <SecureBackup>
            <cert>test-cert</cert>
            <destination>local://sbackup-${DSTAMP}</destination>
        </SecureBackup>
    </dp:action>
</target>

Why Pay for DPBuddy Support?

You can find information on our support services here.

Keeping Multiple DataPower Devices in Sync

Most DataPower installations in production contain two DataPower devices with identical or similar configuration.
It is actually quite easy to keep the two devices in sync using Apache Ant and DPBuddy administration tool.
DPBuddy supports a notion of an environment prefix. The prefix groups together connectivity properties for each device:

dp.username=dpbuddy
dp.password=123 
dp.domain=deployTest
prod1.dp.xmlmgm.url=https://dp1.prod
prod2.dp.xmlmgm.url=https://dp2.prod

The above property list defines dp.username, dp.password and dp.domain properties that are common to all environments and two XML management URLs for our DataPower cluster.

Then the Ant target for importing configuration into both devices could look like this:

<parallel>
    <dp:import envPrefix="prod1" file="${import.file}" overwriteFiles="true" overwriteObjects="true" />
    <dp:import envPrefix="prod2" file="${import.file}" overwriteFiles="true" overwriteObjects="true" />
</parallel>

Note that for speed we chose to execute both imports in parallel; it is perfectly save to do it with dpbuddy as all of its tasks are thread-safe.

If your "dp:import" tasks is more complex (e.g. because of use of deployment policies), you may want to create an Ant macro for each task to avoid duplicating parameters.

MyArch, Inc. offers paid support for dpbuddy, including implementing custom enhancements. If you're interested, please contact us as info at myarch.com.

DataPower Buddy Google Group

There is now a Google group for Data Power buddy: http://groups.google.com/a/myarch.com/group/dpbuddy.
DPBuddy users can use this group to discuss questions and issues related to the dpbuddy tool.

DataPower Buddy 2.2.2 Release

DataPower Buddy 2.2.2 is available from this link. This release contains several bug fixes, mostly around "import" Ant task.

MyArch, Inc. offers paid support for dpbuddy, including implementing custom enhancements. If you're interested, please contact us as info at myarch.com.

DataPower Deployment Policies and XPath

Deployment policy in IBM DataPower appliance is a very nice concept. Deployment policies allow for "tweaking" device and domain configuration for different environments. Let's face it, there are always differences between environments. Sometimes, these differences are small, such as different back-end hosts, and in other cases these differences could be significant, such as different security policies. It is commendable that IBM recognized this fact and implemented a mechanism for dealing with this very common problem directly in their product.

Deployment policies do a decent job of dealing with differences between environments. Deployment policies support changing, adding and deleting configuration, so it is possible to implement fairly complex transformations.

However, dealing with deployment policies could be confusing. Deployment policy's match rules utilize xpath, but the syntax of the rules is not pure xpath (note -- I'm using 3.7 firmware. The syntax might be different in 3.8). Consider this simple deployment policy match rule:

*/*/protocol/http?Name=personSreviceHTTP&Property=LocalPort

The part before "?" looks like xpath. But what schema is this xpath based on? There is no "protocol" element in the DataPower XML management schema. The part after "?" that uses name-value parameters is even more odd. Why use this instead of proper xpath? After all, DataPower has an XML-processing engine with full xpath support, so it would certainly be more logical to rely on XML standards.

The bottom line is that while deployment policies are useful, they have limitations. They have to be developed using Deployment Policy builder in WebGUI. They can only be applied to configuration elements supported in WebGUI. For example, creating a deployment policy that updates RemoteEndpointPort of a Web Services Proxy proves to be a non-trivial task.

This is why we added support for "plain" xpath-based overrides in our DPBuddy DataPower management tool. Instead of dealing with the obscure syntax of deployment policies, developers can simply look at the configuration export file and specify an xpath expression against this file. "dp:setConfig" task of DPBuddy will update the matching element of this expression with the new value. For example, to update RemoteEndpointPort, one can define the following task:

<dp:setConfig>
    <configFile file="dpconfigs/config-wsproxy.xml">
        <override xpath="//RemoteEndpointPort" value="${endpoint.port}"/>
    </configFile>
</dp:setConfig>

It is not possible to add or remove configuration using "dp:setConfig"; deployment policies still have to be used for this kind of config changes. However, from our experience, majority of environment differences can be dealt with the xpath-based override mechanism.

MyArch, Inc. offers paid support for dpbuddy, including implementing custom enhancements. If you're interested, please contact us as info at myarch.com.

DPBuddy — Tool For DataPower Administrators and Developers

We're pleased to announce the release of our new product, DataPower Buddy (dpbuddy). "dpbuddy" a free command-line tool for automating administration, management and deployment of IBM WebSphere DataPower appliances. The tool supports export/import, file transfer, backups and many other functions.

dpbuddy is implemented as a set of custom tasks for the popular build tool, Apache Ant.

Here is a quick example of dpbuddy in action:

<dp:copy cleanDirectories="true">
    <dpFileset prefix="/apps/services" 
               dir="services" includes="**/*.wsdl **/*.xsd"/>
</dp:copy>

This Ant task will remove remote directories if they exist, reproduce the local directory tree (all folders under "services") on the device and upload the necessary files based on the "includes" pattern.

dpbuddy is completely free; it can be downloaded from the dpbuddy product page

dpbuddy provides many cool features, including:

• Response from the device is presented in a human-readable form as opposed to raw SOAP/XML messages. dpbuddy makes it easy to understand error and status messages.

• Powerful remote "copy" command that automatically reproduces local directory tree on the device.

• Tight integration with Ant. Ant variables can be used inside deployment policies and configuration files.

• Easy-to-use alternative to deployment policies based on XPath.

• Ability to remotely "tail" device logs. It is even possible to automatically get new log messages similarly to Unix "tail -f" command. "tail" task can also check for error patterns.

• "Export" based on naming patterns. You don't need to know types ("classes") of DataPower objects; simply specify a regexp pattern and dpbuddy will export all objects matching this pattern.

• Support for self-signed certificates. No need to add DataPower certificates to the JDK store.

• Support for arbitrary SOMA requests. You can use Ant variables inside a request.

• Parsing of all commands on the client. In case of XML errors, DataPower returns cryptic "internal error" message. The actual error then has to be extracted from the device logs. dpbuddy on the other hand validates management XML commands on the client and displays error messages right away.

Go to dpbuddy product page to learn more.