Deployment policies do a decent job of dealing with differences between environments. Deployment policies support changing, adding and deleting configuration, so it is possible to implement fairly complex transformations.

However, dealing with deployment policies could be confusing. Deployment policy's match rules utilize xpath, but the syntax of the rules is not pure xpath (note -- I'm using 3.7 firmware. The syntax might be different in 3.8). Consider this simple deployment policy match rule:

The part before "?" looks like xpath. But what schema is this xpath based on? There is no "protocol" element in the DataPower XML management schema. The part after "?" that uses name-value parameters is even more odd. Why use this instead of proper xpath? After all, DataPower has an XML-processing engine with full xpath support, so it would certainly be more logical to rely on XML standards.

The bottom line is that while deployment policies are useful, they have limitations. They have to be developed using Deployment Policy builder in WebGUI. They can only be applied to configuration elements supported in WebGUI. For example, creating a deployment policy that updates RemoteEndpointPort of a Web Services Proxy proves to be a non-trivial task.

This is why we added support for "plain" xpath-based overrides in our DPBuddy DataPower management tool. Instead of dealing with the obscure syntax of deployment policies, developers can simply look at the configuration export file and specify an xpath expression against this file. "dp:setConfig" task of DPBuddy will update the matching element of this expression with the new value. For example, to update RemoteEndpointPort, one can define the following task:

<dp:setConfig>
    <configFile file="dpconfigs/config-wsproxy.xml">
        <override xpath="//RemoteEndpointPort" value="${endpoint.port}"/>
    </configFile>
</dp:setConfig>

It is not possible to add or remove configuration using "dp:setConfig"; deployment policies still have to be used for this kind of config changes. However, from our experience, majority of environment differences can be dealt with the xpath-based override mechanism.

MyArch, Inc. offers paid support for dpbuddy, including implementing custom enhancements. If you're interested, please contact us as info at myarch.com.

dpbuddy is implemented as a set of custom tasks for the popular build tool, Apache Ant.

Here is a quick example of dpbuddy in action:

<dp:copy cleanDirectories="true">
    <dpFileset prefix="/apps/services" 
               dir="services" includes="**/*.wsdl **/*.xsd"/>
</dp:copy>

This Ant task will remove remote directories if they exist, reproduce the local directory tree (all folders under "services") on the device and upload the necessary files based on the "includes" pattern.

dpbuddy is completely free; it can be downloaded from the dpbuddy product page

dpbuddy provides many cool features, including:

• Response from the device is presented in a human-readable form as opposed to raw SOAP/XML messages. dpbuddy makes it easy to understand error and status messages.

• Powerful remote "copy" command that automatically reproduces local directory tree on the device.

• Tight integration with Ant. Ant variables can be used inside deployment policies and configuration files.

• Easy-to-use alternative to deployment policies based on XPath.

• Ability to remotely "tail" device logs. It is even possible to automatically get new log messages similarly to Unix "tail -f" command. "tail" task can also check for error patterns.

• "Export" based on naming patterns. You don't need to know types ("classes") of DataPower objects; simply specify a regexp pattern and dpbuddy will export all objects matching this pattern.

• Support for self-signed certificates. No need to add DataPower certificates to the JDK store.

• Support for arbitrary SOMA requests. You can use Ant variables inside a request.

• Parsing of all commands on the client. In case of XML errors, DataPower returns cryptic "internal error" message. The actual error then has to be extracted from the device logs. dpbuddy on the other hand validates management XML commands on the client and displays error messages right away.

Go to dpbuddy product page to learn more.

In a related news, there is finally a DataPower book and it's 960 pages long. And this is before IBM started adding all these edge functions to the device :).

When implementing Web services Java developers tend to think in terms of Java classes that XML documents map to. Using XSLT (or even Schema) for implementing part of their processing logic is not on their list because the common thinking is that it is too expensive to do it in Java.

With XML appliances the situation is exactly the opposite. XSLT all of a sudden becomes one of the best performing part of an application (although, I would imagine that using Java hardware acceleration such as the one provided by Azul might once again change that). This could be a serious "paradigm shift" for many developers and architects.

Another obstacle to more effective usage of appliances could simply be the lack of XSLT skills. XSLT is essentially a functional language and so it comes with a learning curve attached, especially for complex transformations. It is important to have a good knowledge of XSLT to understand what kind of work can be "outsourced" to an appliance. Not that many developers have this knowledge today, but perhaps it will change with more widespread use of XML appliances.

How fast are XML appliances? I’m not aware of published benchmarks; however, I did have a chance to conduct an informal performance testing of DataPower XI50 for a client and the results were quite impressive; for example we saw little to no overhead validating medium-size XML schemas.

Clearly, offloading as much XML processing as possible to an appliance could be a performance booster. So what kind of processing could be done on the device? Here are my recommendations based on the capabilities of the DataPower appliance; the situation could be slightly different with devices from other vendors.

• Validation. Turning schema validation on is a no-brainer, especially for complex schemas that can choke software parsers. In addition to schema validation, XSLT can be used to validate rules that are not grammar-based. You can use Schematron to generate XSLT that validates these rules. Combination of Schema and Schematron provides very powerful validation mechanism. The beauty of this approach is that failed XML documents never reach the application thus reducing resource consumption of the application server.
• Splitting. It is possible to define XPath-rules that in combination with XSLT will split complex documents into multiple fragments that can then be sent to different back-end services for processing.
• Filtering. Sometimes an application only needs a subset of elements provided in a message. Eliminating unnecessary tags will reduce the application’s XML processing overhead.
• Flattening. Complex XML documents with deep nesting can be “flattened” using XSLT to eliminate some of the nesting. This could make application XML parsing quicker.
• "Canonicanization". In many cases, XML schema may allow for different representation of the same piece of data to accommodate needs of different clients. It could be allowing for SSN with and without dashes or supporting different name formats (e.g., "firstName, lastName" vs. "fullName"). XSLT logic on the device could implement the rules to clean up incoming messages and transform them into one strict canonical representation so that the application only needs to implement one way of parsing data.

"Shredding"” could also be very effective for high-throughput XML processing. Shredding is the process of breaking up large documents into smaller fragments and sending them off to the application for concurrent processing. Shredding helps dealing with the documents consisting of a large number of repetitive groups. Each group/fragment can be processed independently without waiting for the completion of the processing of the entire document. Note that shredding is different from "splitting" which allows for processing of different logical parts (different types) of the document independently (as opposed to repetitive groups). Shredding is the most useful for processing large bathes of data of the same type.

Unfortunately, shredding does not seem to be supported by DataPower, although the appliance does support streaming XML processing which would’ve worked very nicely in conjunction with shredding logic. Perhaps devices from other vendors do provide this support.

The bottom line is that just plugging the appliance in and using it in a gateway mode is not necessarily going to improve XML processing performance by much. Web services (and any other application components that interface via XML such as XML-based batch file processing components) have to be designed with the appliance in mind. In the course of the design effort, developers need to identify pre-processing or post-processing logic that can be implemented in XSLT as opposed to making it all application’s responsibility. Anything that uses XSLT can then go to the appliance to improve the processing speed and throughput.

So how long before appliances begin supporting JBI and SCA components? The trick here of course is to make sure that an appliance is still a robust low-maintenance device that can't be brought down by a rogue application thread. But if Layer7 was able to figure it out for Java in general (I wonder if they use a general Sun JDK or some kind of a special JDK build), there is no reason why an appliance can't start supporting more advanced specifications such as JBI/SCA.