Installation and Configuration ============================== .. _dpbuddyInstallation: DPBuddy Installation and Configuration -------------------------------------- .. note:: You can skip these installation steps and use :ref:`DPBuddy docker image ` instead. DPBuddy requires Java version 8 or better. All JVM types are supported, including Oracle, IBM and OpenJDK. DPBuddy is pure Java, so it runs on any OS where Java is available. In some cases, DPBuddy may require a large heap size so a 64-bit JDK is recommended. All the latest versions of Java SE or JDK are fully supported. We recommend installing the most recent long term support version of Java from `Oracle `_. Both DPBuddy and Apache Ant will first check if the ``JAVA_HOME`` environment variable is defined. If it is defined, this Java installation will be used. If ``JAVA_HOME`` is not defined, then java executable must be added to the path. To verify, run the ``java -version`` command. Install DPBuddy by following these steps: * `Download DPBuddy `_ and unzip/untar the content of the file into a directory, e.g., ``c:\dpbuddy`` or ``/opt/dpbuddy``: * Define the ``DPBUDDY_HOME`` environment variable and point it to the directory containing the DPBuddy installation, e.g., ``c:\dpbuddy\dpbuddy-3.5`` or ``/opt/dpbuddy/dpbuddy-3.5``. * Add ``%DPBUDDY_HOME%\bin`` or ``$DPBUDDY_HOME/bin`` to your PATH environment variable. On Linux, make sure that ``$DPBUDDY_HOME/bin/dpbuddy`` has the "execute" permission, run the appropriate "chmod" command if it does not. * Open a command line/terminal window and type ``dpbuddy -h``. You should see a help screen containing the list of commands and options supported by the tool. On Linux, you can add these lines to your .bash_profile: .. code-block:: bash export DPBUDDY_HOME=/opt/dpbuddy/dpbuddy-3.5 export PATH=$DPBUDDY_HOME/bin:$PATH If you already have a previous version of DPBuddy installed, you will need to update your ``taskdef`` Ant tag to point to the location of the new version. DPBuddy does not require root or administrator privileges, it can run under a regular user account. When running DPBuddy under a non-administrative account on Windows you may see the warning "Could not open/create prefs root node Software\JavaSoft\Prefs ...". This warning can be ignored. This can also be fixed by running DPBuddy as administrator once (after that you can continue running it as user) or by manually creating necessary keys in the Windows registry as explained `here `_. You can also follow our :ref:`Quick Start Guide ` to complete your DPBuddy setup. .. _dpbuddyDocker: Using DPBuddy Docker Image -------------------------- DPBuddy docker image contains the latest versions of DPBuddy, Java 8, Apache Ant and Gradle. Use of DPBuddy docker image is fully supported in all environments, including production. Pull the image from the DPBuddy's docker repository: .. code-block:: bash docker pull myarch/dpbuddy:3.5 DPBuddy docker image exposes two volumes, one for `DPBuddy logs `_ and another one for your scripts, configurations, exports, etc. You need to create these volumes and provide them to docker to preserve the files between container runs or image updates. Creating volumes is not required for Docker on macOS. .. code-block:: bash docker volume create dpbuddy-logs docker volume create dpbuddy-work Run the container: .. code-block:: bash docker run -it --rm --name dpbuddy -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5 Note the ``--rm`` option which means your container will be destroyed upon exit, so make sure that you preserve all your work in volumes. Alternatively, you can first create the container and then run it using `docker start command `_ .. code-block:: bash docker create -it --name dpbuddy -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5 docker start -i dpbuddy .. note:: Run docker with ``sudo`` if you get any errors attempting to execute docker commands. On macOS you can simply point to a directory under `one of the shared root directories `_: .. code-block:: bash docker run -it --rm --name dpbuddy -v $PWD/dpbuddy-docker/vols/work:/opt/dpbuddy/src -v $PWD/dpbuddy-docker/vols/logs:/var/log/dpbuddy myarch/dpbuddy:3.5 You can run the following command from the docker's shell to verify that DPBuddy is up and running: .. code-block:: bash dpbuddy -h Next, you will need to create your ``dpbuddy.conf`` file. You can use the ``dpbuddy.starter.conf`` file in ``/opt/dpbuddy/conf`` as the starting point. Copy it to the ``/opt/dpbuddy/src`` first (to make sure it's not lost after container restart): .. code-block:: bash cp /opt/dpbuddy/conf/dpbuddy.starter.conf /opt/dpbuddy/src/dpbuddy.conf Then follow :ref:`quick start guide ` to modify it. DPBuddy docker image predefines several DPBuddy environment variables. * :ref:`DPBUDDY_CONFIG `. It points to ``/opt/dpbuddy/src/dpbuddy.conf``. * :ref:`DP_MASTER_PASSWORD `. You may want to override it if you intend to use DPBuddy's encryption/decryption capabilities. * DPBUDDY_HOME is set to ``/opt/dpbuddy``. You need to provide a new environment variable value to the ``docker create`` or ``docker run`` commands using ``-e`` option. E.g., to provide your master password: .. code-block:: bash docker run -it --rm --name dpbuddy -e DP_MASTER_PASSWORD=my-secure-password -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5 You can also provide any DPBuddy command on the docker command line, in which case the command will be executed and then the container will be terminated: .. code-block:: bash docker run -it --rm --name dpbuddy -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5 dpbuddy connect .. _upgrading from dpbuddy 3.0: Upgrading DPBuddy ----------------- To upgrade DPBuddy, simply un-archive the distribution into a new folder and change your ``DPBUDDY_HOME`` environment variable to point to the new location. The DPBuddy 3.5 distribution contains Apache Ant 1.10 jar files. To avoid classloader conflicts, exclude Ant jars from the taskdef used to define DPBuddy tasks. Additionally, DPBuddy starting with version 3.2.4 uses `logback `_ logging framework and relies on `logback.xml `_ configuration file. "logback.xml" is located in "$DPBUDDY_HOME/conf" directory. In Ant, this directory needs to be explicitly added to the classpath of the DPBuddy's "taskdef". The resulting "taskdef" should be defined as following: .. code-block:: xml All DPBuddy 3.5 tasks are backward compatible with 3.0, 3.1, 3.3 and 3.4. .. _configuring_dp: Configuring DataPower to Work with DPBuddy ------------------------------------------ You need to make sure that you have XML Management enabled in all devices that will be managed by DPBuddy. You can do this from DataPower UI (Network/Management/XML Management or Objects/Device Management/XML Management interface in newer UI in version 10 of DataPower). DPBuddy uses the "old" DataPower v2004 message format for file transfer (the 2004 schema is more efficient, since it allows for uploading multiple files at once). This means that the "SOAP Configuration Management (v2004)" checkbox must be checked on the XML management screen. In newer firmware UI it is set by default, you can see it by expanding "Enabled Services". "SOAP Management URI" and "SOAP Configuration Management" services must be enabled as well. Be careful performing these changes on the appliances residing in a production DMZ -- make sure that the management ports are not available to the outside world. If you're using a DataPower docker container, make sure to map the XML Management's port, e.g., by adding `-p 5550:5550` to your docker command. DPBuddy does not use SSH, so SSH can be disabled. We also recommend enabling internal logging. DataPower does not return error messages to the client. If a DPBuddy command fails and you get an "Internal Error" message from the device, you will need to look for the root cause in the DataPower system log in the default domain. DataPower captures errors only if internal logging is enabled. You can enable it by navigating to "Troubleshooting", then to "Logging." Set "Enable Internal Logging" and click on "Set Log Level." We recommend setting the level to "Warning", since anything more detailed (e.g., "Information") results in needlessly verbose messages. DPBuddy needs an account to use to login to DataPower. DPBuddy will work with any "Group-Defined" account that has read/write access to the target domain(s). In general, DPBuddy does not need to access the "default" domain. However, certain features of DPBuddy's commands, such as updating domain's comments during import will not work if the write access to the default domain is restricted. All domain-level commands also require access to the default domain. Create a privileged account if you're going to use any of these commands. If you're creating a new account that will be used by DPBuddy, make sure that the password for the account has been reset or disable initial password change. Before running DPBuddy, make sure that you're able to login to WebGUI using this account. Note that by default DataPower uses a self-signed certificate for SSL. By default, DPBuddy ignores certificate validation errors. You can change this behavior by setting ``dp.trust.all.certs`` to ``false`` as described in :ref:`Common Attributes/Options and Properties `. If you choose this option, you will need to manually add the DataPower certificate to your JVM's keystore. You also need to make sure that your DataPower appliance has sufficient internal space. For exports and backups DataPower first generates export files in the internal filesystem. If you're trying to export large configuration files and there is not enough space, you will get an error. You can check the available temporary space by navigating to "Filesystem information" in DataPower UI. It is highly recommended (although not absolutely required) to place all machines hosting DPBuddy and all DataPower appliances in the same time zone. DataPower relies on ZIP format for export/import; this means that file timestamps have no time zone information. This may lead to various issues if an export/import archive needs to be modified during your build/deployment process. .. _install_config_dpbuddy_conf: Creating DPBuddy Configuration File ----------------------------------- All DataPower connection parameters and other configuration properties have to be defined in the :ref:`dpbuddy.conf ` config file. You can follow the :ref:`quick start guide ` to bootstrap its creation. Once you created ``dpbuddy.conf``, you should keep it under version control. ``dpbuddy.conf`` is extremely flexible and should be used for all your environment and configuration settings. To use ``dpbuddy.conf`` from CLI, define :ref:`DPBUDDY_CONFIG ` environment variable. .. _configuring_ant: Configuring Apache Ant to Work with DPBuddy ------------------------------------------- Using DPBuddy from Apache Ant provides more flexibility and better integration with various build/deployment and release management tools than DPBuddy's CLI. It is also more efficient since there is no need to start a JVM for every command. DPBuddy requires Apache Ant 1.8 or above. If you don't have Apache Ant installed on your system, download and install it as documented at `http://ant.apache.org/manual/install.html `_. Note that on many Linux systems you can also install Ant using a package manager, e.g., you can use ``apt-get -u install ant`` on Ubuntu. We also recommend running Ant with the ``-e`` option, as it provides a much cleaner output. You can set this option globally by `setting the ANT_ARGS `_ environment variable to ``-e``. DPBuddy tasks are packaged in a single Antlib. You may wish to familiarize yourself with the `Ant documentation of the Antlib feature `_, especially the use of namespaces in Ant, before configuring DPBuddy tasks in your build file. To declare DPBuddy tasks, add "taskdef" to your build file and point it to the DPBuddy installation: .. code-block:: xml You must also declare the namespace used by DPBuddy tasks: .. code-block:: xml The namespace's URI must match the URI used in taskdef. You must use ``antlib:com.myarch.dpbuddy`` as the URI. You can also copy all the jar files that come with DPBuddy to ``${user.home}/.ant/lib`` directory. If you choose this option, you don't need to provide an explicit ``taskdef``. Please consult the `Ant manual `_ for more details. .. _verifying_connectivity: Verifying Connectivity with DataPower ------------------------------------- To verify DPBuddy's connectivity with DataPower, run the following command: .. code-block:: bash dpbuddy connect -env .. _advanced_configuration: Increasing Memory for Large Downloads ------------------------------------- If you're planning to use DPBuddy to download or upload large binary files (bigger than 30MB), we recommend increasing the maximum heap size for the JVM used to run DPBuddy. DPBuddy CLI supports the ``DPBUDDY_OPTS`` environment variable, which can be used to pass custom options to the JVM. By default, this variable is set to ``-Xms256m -Xmx2096m``, meaning that the maximum heap size is set to 2GB. To change the heap size for Ant, set the `ANT_OPTS `_ environment variable. Set the heap size to 4GB or higher. Also, make sure to use a 64-bit Java. .. _proxy: HTTPS Proxy Configuration ------------------------- If you're going to run DPBuddy behind a proxy, add the following Java system properties to the ``DPBUDDY_OPTS`` or ``ANT_OPTS`` variables: * ``https.proxyHost``: The host name of the proxy server * ``https.proxyPort``: The port number of the proxy server; defaults to 443 * ``https.nonProxyHosts``: A list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by ',' or '|'. The patterns may start or end with a '*' for wildcards. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy. Note that you must use the "https" prefix. For example: .. code-block:: bash export DPBUDDY_OPTS="-Dhttps.proxyHost= -Dhttps.proxyPort=" .. _dpbuddy_license: DPBuddy License --------------- All DPBuddy downloads are bundled with a 45-day evaluation license. You do not need to request or install the evaluation license; it is generated automatically when you run DPBuddy for the first time. Once the evaluation license is expired, you'll need to purchase the license from MyArch or request an extension of your trial license. Please send your request to `license at myarch.com `_ or submit it at `http://myarch.com/dpbuddy-pricing `_. To install a permanent license that you received from us, simply save the license file and set ``DPBUDDY_LICENSE_FILE`` environment variable to point to the license file (must be an absolute path to the file, e.g. ``export DPBUDDY_LICENSE_FILE=/opt/dpbuddy/dpbuddy-license``. You can also save the license file in ``$DPBUDDY_HOME/conf`` (``%DPBUDDY_HOME%\conf`` on Windows) or, if you're using our docker image, under ``/opt/dpbuddy/src``. In this case, you don't need to specify any environment variables. Alternative, you can add the property ``dpbuddy.license.file`` to your configuration file or Ant build file. You can also provide the ``license`` option or attribute to a DPBuddy command or task. .. _upgrading_from_dpbuddy_2_3: Upgrading from DPBuddy 2.3 -------------------------- DPBuddy 3 is mostly backward compatible with DPBuddy 2.3 with a few exceptions: * The namespace for the DPBuddy antlib has changed to ``com.myarch.dpbuddy``. Make sure to update it in the ``project`` tag and in the ``taskdef`` of your build file, e.g., ``xmlns:dp="antlib:com.myarch.dpbuddy"``. * The ``request`` task was renamed to :ref:`somaRequest`. * The ``setConfig`` task was deprecated; use :ref:`import` instead. * :ref:`modifyConfig` no longer supports the nested ``configFile`` element. You can replace it with the new :ref:`transform element `. * ``toDir`` is now a required attribute of the :ref:`copy`. You can also provide ``toDir`` at the nested ``dpFileset`` level. * :ref:`Environment prefix mechanism ` has been substantially enhanced; you may need to change the way your properties are named to take advantage of the hierarchical prefixes.