Automating DataPower Firmware Upgrades with DPBuddy

Posted on 01/29/2017 ,

Please follow this link.

How to Deal with Generated DataPower Policies

Posted on 07/25/2016 ,

Please follow this link.

How to Manage and Remotely Tail DataPower Logs

Posted on 07/25/2016 ,

Please follow this link.

DataPower Buddy Release 3.3

Posted on 06/08/2016 ,

We're pleased to announce the availability of DataPower Buddy 3.3.

New Tasks/Commands and Notable New Feature

  • Support for encrypted properties and configuration settings. This can be used for encrypting DataPower account passwords. It can also be used for encrypting any sensitive data inside DataPower configuration/export files or inside any files that are copied to DataPower using DPBuddy's copy command.
  • PasswordAlias task/command to create DataPower password aliases. The password can be stored encrypted.
  • New format for configuration settings and properties/variables based on the open-source HOCON (Human-Optimized Config Object Notation) library. HOCON is a less restrictive JSON superset geared towards defining configuration settings. HOCON allows for comments, it does not require string quotes, it supports variable substitution and includes. For more details please refer to HOCON documentation. HOCON provides many benefits over the property/prefix-based format introduced in earlier DPBuddy releases. The properties format, however, is still fully supported, migration to HOCON is completely optional.
  • Support for DataPower firmware 7.5
  • Secure restore task/command. The command performs restoration of a secure backup and optionally waits for the completion of the secure restore process. This command can be used for automated sync of multiple appliances in production or for managing disaster recovery appliances.
  • Restore task/command. This command restores multiple domains (or all of the domains in an appliance) form the backup taken by the backup command. This command can also be used for appliance syncing and for DR.
  • RestartDP task/command. This command restarts the appliance and optionally waits for the completion of the restart process.
  • License command that prints DPBuddy's license information.

Configuration Transformation-Related Changes

  • Support for repeaters (loops). Repeaters allow for generating XML fragments in a loop based on an array or an object defined in a HOCON configuration file. For example, you may need to change the definition of a Load Balancer Group depending on the target environment (each environment can have a different number of LBG members). You can define your LBG members in the configuration file and then apply "update" action with "repeat" attribute.
  • "if" expressions are now supported at the "transform" container level (previously "if" was only supported at the action level).
  • "verbose" setting is now supported at the "transform" container and at the individual transformation action levels.
  • Improved logging of transformed element names.
  • Support for local Ant properties in Groovy expressions.
  • "antProject" is now part of the scope in all Groovy expressions.
  • Passwords are now automatically masked in all transformation logs (including verbose mode).

Changes to Existing Tasks/Commands

  • Import, copy and restore commands now stream files to DataPower, which speeds up file transfer and allows for supporting large files. Previously, DPBuddy loaded files in memory prior to import/copy.
  • Export task/command now supports sorting of the exported XML configuration file. DPBuddy will sort all DataPower configuration objects alphabetically according to their names and types. This feature can be used to make tracking of DataPower configuration changes easier since DataPower sometimes re-orders objects in the exported configuration.
  • Import task/command can now re-run import automatically in case of import errors. This feature should be enabled when importing the sorted configuration file created by the export command. DataPower requires configuration objects to be defined prior to being referenced at import. With the sorted export file, this is no longer the case, hence the import may need to be run twice.
  • Export command now automatically removes "export-manifest" from the exported xml files (can be controlled by the "defaultTransorm" option).
  • Import now supports "failOnError" option/attribute.
  • More efficient mkdir task/command -- a directory tree is now created in a single request.
  • assertState now prints error code/error description for objects in the "down" state.
  • Improved CLI help.
  • For DPBuddy CLI, config files can now reside in "/etc" directory.

To upgrade to this release, you can simply download and un-archive the distribution and point your DPBUDDY_HOME environment variable to the new location. If you're using DPBuddy from Apache Ant, you will also need to add <pathelement location="${dpbuddy.home}/conf"/> to the DPBuddy library's "taskdef" in your Ant files, otherwise, you will see verbose logging output in the console.

You can download DPBuddy 3.3 from this page.

DataPower Buddy Release 3.3 Beta

Posted on 03/15/2016 ,

We're pleased to announce the availability of DataPower Buddy 3.3 Beta.

This release introduces support for defining configuration properties/variables using HOCON (Human-Optimized Config Object Notation) format. HOCON is a superset of JSON, it is quite flexible (e.g., it supports comments, includes, substitutions) and it is more readable than raw JSON. HOCON provides a powerful alternative to defining environment-specific properties using prefix-based notation. The prefix-based mechanism, however, is still fully supported, so the use of HOCON is completely optional.

Other notable features of this release include:

  • Support for restore/import of multiple domains (or all of the domains). This could be useful for keeping multiple production appliances in sync.
  • Support for secure restore. This can also be used to maintain a DataPower cluster in sync or in a DR situation.
  • Support for appliance reboot/restart. Both secure restore and restart/reboot commands can optionally wait for the appliance to come back online.
  • Password encryption inside configuration files using open-source Jascrypt tool.
  • Under the hood, DPBuddy now streams files to the appliance during copy/import/restore, so these commands are now performed much faster and with lower memory requirements.
  • "Add" and "update" configuration transformation functions now support repeaters (loops). This can be used to generate environment-specific load-balancing group configuration with variable number of back-end servers.
  • Many minor changes and bug fixes. For example, passwords are now automatically masked when environment transformations run in verbose mode.
  • CLI help has been improved to make the use of CLI easier.

This release could also provide support for firmware 7.5; this feature will be finalized once 7.5 becomes available.

The general availability of DPBuddy 3.3 is expected in April 2016. Meanwhile, please let us know if you're interested in evaluating the beta version.

DataPower Buddy Roadmap

Posted on 12/08/2015 ,

The focus of the next DPBuddy release is on more powerful configuration transformations, including supporting transformations directly from DPBuddy CLI.

We're also planning on implementing native plugins for Maven and Gradle.

What else would you like to see in the upcoming versions of the product? Please let us know.

DPBuddy Release 3.2.4 (Improved Auditing)

Posted on 10/12/2015 ,

We're pleased to announce that DataPower Buddy 3.2.4 is now available. The focus of this release is on improved audit and logging.

DPBuddy now generates an audit log file in JSON format, in addition to the XML format supported in earlier releases. This file can be easily tailed, analyzed with jq and/or uploaded to an enterprise SIEM tool. DPBuddy now uses logback framework for auditing and logging. This provides a lot of flexibility in configuring log file location, rollover policies, appenders and other parameters.

Other new features include:

  • DPBuddy now captures import failures in the audit log.
  • backup command now supports the new option/attribute, "failIfNoDomain". If set to "false", "backup" will not fail if the target domain does not exist.
  • Better error handling. A root cause of an error now reported automatically, without having to run the tool in verbose mode.
  • Bug fixes.

To upgrade to this release, you can simply download and un-archive the distribution and point your DPBUDDY_HOME environment variable to the new location. If you're using DPBuddy from Apache Ant, you will also need to add <pathelement location="${dpbuddy.home}/conf"/> to the DPBuddy library's "taskdef" in your Ant files, otherwise, you will see verbose logging output in the console.

You can download DPBuddy 3.2.4 from this page.

DataPower Buddy Release 3.2.3 (Firmware 7.2)

Posted on 06/28/2015 ,

We're pleased to announce that DPBuddy 3.2.3 is now available. This release provides the support for DataPower firmware 7.2 and bug fixes.

An up-to-date version of Java 7 is required when using DPBuddy with firmware 7.2, otherwise you may encounter SSL-related error when trying to connect to DataPower. This is due to the bug in earlier versions of openjdk.

Other new features include:

  • "quiet" in "delConfig" task now suppresses all deletion errors, including the ones caused by an object being referenced by another object. This is to provide a workaround for the bug in firmware v7.2 which causes DataPower to retain references from objects that have been deleted.
  • "resetDomain", "restartDomain" and "wsrrSynchronize" now support "domain" and other common attributes/options; these attributes/options were ignored in earlier versions.

You can download DPBuddy 3.2.3 trial from this page.

DataPower Buddy Release 3.2.2

Posted on 05/27/2015 ,

We're pleased to announce that DPBuddy 3.2.2 is now available. The main feature of this release is the ability to create domains with a single command, including automatic deletion of a domain:

dpbuddy createDomain -domain domain_name -delete -maxCheckpoints 5

Other new features include:

  • Full support for the latest versions of DataPower firmware, including 7.1.0.4
  • copy now supports "excludes" attribute; "includes" defaults to "include all" similar to the standard Ant copy command.
  • All transformations now support "verbose" attribute so you can see the list of changes performed by transformations without having to run DPBuddy/Ant in "verbose" mode.
  • Support for DataPower deployment variables in the import command.
  • Cleanup of the logic used by environment-specific property resolution mechanism.
  • "overwriteFiles" attribute of the import command now defaults to "false" for XML import files.

The release also contains several important bug fixes.

You can download DPBuddy 3.2.2 trial from this page.

DataPower Buddy Release 3.2

Posted on 03/18/2015 ,

We're pleased to announce that DPBuddy 3.2 is now available. The main feature of this release is the ability to execute any DPBuddy task or command against multiple devices and/or domains. Simply define the URLs of your devices using environment properties and supply a list of environment prefixes to a command or an Ant task:

dpbuddy import -file services.zip -env "dev,test"

DPBuddy will execute the command against all the devices in the list, and, in case of any errors, it would optionally rollback either just the failed domains or all the devices/domains in the list.

You can find more details in the DPBuddy documentation.

Other new features include:

  • Full support for the latest versions of DataPower firmware, including versions 7.0.0.5 and 7.1.0.3
  • Support for remotely executing DataPower CLI commands and configuration scripts. DPBuddy can upload the script to the device and execute it there. The scripts can contain Ant/Groovy variables.
  • "Copy", "action" and several other commands now support auto-rollback in case of an error.
  • New execTarget task allowing for automatic execution of any set of Ant targets against multiple devices/domains. Similar to other DPBuddy tasks, "execTask" supports auto-rollback.
  • tailLog command used for remotely tailing DataPower logs now displays domain, log category and object type.
  • Deployment policy used by "import" and other commands can now be direct output of "export". Previously, the exported file had to be manually cleaned up prior to using it as part of "import".
  • Support for a global "ignore list" of object classes and names. This list is used by state check commands to exclude those objects from the check that are supposed to be in the "down" state.
  • Better error handling in the "copy" command.
  • Support for TLS 1.2.
  • DPBuddy CLI can now be run under Cygwin.
  • New ping command.
  • Enhanced troubleshooting of HTTP connections between DataPower and DPBuddy.

The release also contains several important bug fixes.

You can download DPBuddy 3.2 trial from this page.

DataPower Buddy Release 3.2-Beta

Posted on 12/10/2014 ,

We're pleased to announce that DPBuddy 3.2-beta is now available. The main feature of this release is the ability to execute any DPBuddy task or command against multiple devices and/or domains. Simply define the URLs of your devices using environment properties and supply a list of environment prefixes to a command or an Ant task:

dpbuddy import -file services.zip -env "dev,test"

DPBuddy will execute the command against all the devices in the list, and, in case of any errors, it would optionally rollback either just the failed domains or all the devices/domains in the list.

You can find more details in the DPBuddy documentation.

Other new features include:

The release also contains several important bug fixes.

The general availability of DPBuddy 3.2 is expected in January 2015. Meanwhile, please let us know if you're interested in becoming part of our beta program.

DPBuddy Updates for June, 2014

Posted on 07/01/2014 ,

Updated Online Documentation

We've converted our documentation to the online HTML format, which is much easier to navigate than the old PDF file. Future releases of DPBuddy will be bundled with this documentation instead of PDF.

Support for DataPower Firmware Version 7

We ran our full test suite against the recently released DataPower Version 7 and did not see any issues. If you're planning to migrate to version 7 in the future, all your DPBuddy scripts and commands should continue working.

Support for GatewayScript

Support for JavaScript (a.k.a. GatewayScript) is the most noticeable new feature in DataPower firmware Version 7. GatewayScript can be used as part of any processing policy as an alternative to the XSLT-based rules.

Please see this page for the tips on developing and deploying GatewayScript files.

DPBuddy Updates for May, 2014

Posted on 05/28/2014 ,

New and Updated Guides and Tutorials

  • Getting started with DPBuddy CLI. As this tutorial shows, getting started with DPBuddy CLI is very easy and only takes a few minutes. Download DPBuddy, follow this guide and you'll be able to run your first automated deployment in no time.
  • Automating testing of DataPower services using SoapUI. This post is a must read if you're looking to implement automated testing of your services.
  • How to run DPBuddy tasks from Jenkins. Jenkins provides a nice UI along with many other important features such as job management, log retention and security. If you're looking to implement continuous integration, automated (or even continuous) delivery of your DataPower artifacts, or if you're simply running all of your Ant/DPBuddy scripts from command line, follow this guide to create your Jenkins jobs to deploy to DataPower.

DPBuddy Development Update

We're currently working on the release 3.1.1. This release will have some minor changes and a few bug fixes. It will also contain an updated HTML-based User Guide.

We're also planning to implement the support for DataPower firmware version 7.0 as soon as it becomes available. This will include supporting DataPower GatewayScript and other new features.

In the meantime, you can always download DPBuddy 3.1 from this link and give it a try.

If you have any questions, suggestions or feature requests, please comment on this post. We appreciate your feedback.

Automating DataPower Build/Deployment with Jenkins and DPBuddy

Posted on 05/13/2014 ,

You can easily automate your DataPower deployments with Jenkins and DPBuddy. DPBuddy comes with a comprehensive set of Ant tasks, so you can simply create an Ant script defining your build/deployment process and then invoke the script from Jenkins using the "Invoke Ant" build step.
You can find the details in this tutorial.

Automating Testing of DataPower Services

Posted on 05/08/2014 ,

Any automated build/deployment process should be accompanied by automated testing. This is required for automated (let alone for continuous) delivery of any application. Services residing in DataPower appliances should also be covered by automated testing. This can be achieved by invoking SoapUI test cases as part of an automated deployment, after all the services have been imported. This article explains how to do it.