Installation
============

.. _dpbuddyInstallation:

DPBuddy Installation
--------------------

.. note::

  You can skip these installation steps and use :ref:`DPBuddy docker image <dpbuddyDocker>` instead.

DPBuddy requires Java SE Runtime Environment 8 or better. All JVM types are supported, including Oracle, IBM and OpenJDK. DPBuddy is pure Java, so it runs on any OS where Java is available. In some cases, DPBuddy may require a large heap size so a 64-bit JRE is recommended.

All the latest versions of Java SE or JDK are fully supported.

Both DPBuddy and Apache Ant will first check if the ``JAVA_HOME`` environment variable is defined. If it is defined, this Java installation will be used. If ``JAVA_HOME`` is not defined, then java executable must be added to the path. To verify, run the ``java -version`` command.

Install DPBuddy by following these simple steps:

* `Download DPBuddy <https://myarch.com/download-dpbuddy>`_ and unzip/untar the content of the file into a directory, e.g., ``c:\dpbuddy`` or ``/opt/dpbuddy``:

* Define the ``DPBUDDY_HOME`` environment variable and point it to the directory containing the DPBuddy installation, e.g., ``c:\dpbuddy\dpbuddy-3.5`` or ``/opt/dpbuddy/dpbuddy-3.5``.

* Add ``%DPBUDDY_HOME%\bin`` or ``$DPBUDDY_HOME/bin`` to your PATH environment variable. On Linux, make sure that ``$DPBUDDY_HOME/bin/dpbuddy`` has the "execute" permission, run the appropriate "chmod" command if it does not.

* Open a command line/terminal window and type ``dpbuddy -h``. You should see a help screen containing the list of commands and options supported by the tool.

On Linux, you can add these lines to your .bash_profile:

.. code-block:: bash

  export DPBUDDY_HOME=/opt/dpbuddy/dpbuddy-3.5
  export PATH=$DPBUDDY_HOME/bin:$PATH

If you already have a previous version of DPBuddy installed, you will need to update your ``taskdef`` Ant tag to point to the location of the new version.

DPBuddy does not require root or administrator privileges, it can run under a regular user account.

When running DPBuddy under a non-administrative account on Windows you may see the warning "Could not open/create prefs root node Software\JavaSoft\Prefs ...". This warning can be ignored. This can also be fixed by running DPBuddy as administrator once (after that you can continue running it as user) or by manually creating necessary keys in the Windows registry as explained `here <http://stackoverflow.com/questions/16428098/groovy-shell-warning-could-not-open-create-prefs-root-node>`_.

You can also follow our :ref:`Quick Start Guide <quickStart>` to complete your DPBuddy setup.

.. _dpbuddyDocker:

Using DPBuddy Docker Image
--------------------------

DPBuddy docker image contains the latest versions of DPBuddy, Java 8, Apache Ant and Gradle. Use of DPBuddy docker image is fully supported in all environments, including production.

Pull the image from the DPBuddy's docker repository:

.. code-block:: bash

  docker pull myarch/dpbuddy:3.5

DPBuddy docker image exposes two volumes, one for `DPBuddy logs <logging>`_ and another one for your scripts, configurations, exports, etc. You need to create these volumes and provide them to docker to preserve the files between container runs or image updates.

Creating volumes is not required for Docker on macOS.

.. code-block:: bash

  docker volume create dpbuddy-logs
  docker volume create dpbuddy-work

Run the container:

.. code-block:: bash

  docker run -it --rm --name dpbuddy  -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5

Note the ``--rm`` option which means your container will be destroyed upon exit, so make sure that you preserve all your work in volumes.

Alternatively, you can first create the container and then run it using `docker start command <https://docs.docker.com/engine/reference/commandline/start/>`_

.. code-block:: bash

  docker create -it --name dpbuddy -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5
  docker start -i dpbuddy

.. note::

  Run docker with ``sudo`` if you get any errors attempting to execute docker commands.

On macOS you can simply point to a directory under `one of the shared root directories <https://docs.docker.com/docker-for-mac/osxfs/#access-control>`_:

.. code-block:: bash

  docker run -it --rm --name dpbuddy  -v $PWD/dpbuddy-docker/vols/work:/opt/dpbuddy/src -v $PWD/dpbuddy-docker/vols/logs:/var/log/dpbuddy myarch/dpbuddy:3.5

You can run the following command from the docker's shell to verify that DPBuddy is up and running:

.. code-block:: bash

  dpbuddy -h


Next, you will need to create your ``dpbuddy.conf`` file. You can use the ``dpbuddy.starter.conf`` file in ``/opt/dpbuddy/conf`` as the starting point.

Copy it to the ``/opt/dpbuddy/src`` first (to make sure it's not lost after container restart):

.. code-block:: bash

  cp /opt/dpbuddy/conf/dpbuddy.starter.conf /opt/dpbuddy/src/dpbuddy.conf

Then follow :ref:`quick start guide <dpbuddy_conf_quick_start>` to modify it.

DPBuddy docker image predefines several DPBuddy environment variables.

* :ref:`DPBUDDY_CONFIG <confFileCli>`. It points to ``/opt/dpbuddy/src/dpbuddy.conf``.
* :ref:`DP_MASTER_PASSWORD <encrypting_props>`. You may want to override it if you intend to use DPBuddy's encryption/decryption capabilities.
* DPBUDDY_HOME is set to ``/opt/dpbuddy``.

You need to provide a new environment variable value to the ``docker create`` or ``docker run`` commands using ``-e`` option.

E.g., to provide your master password:

.. code-block:: bash

  docker run -it --rm --name dpbuddy -e DP_MASTER_PASSWORD=my-secure-password -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5

You can also provide any DPBuddy command on the docker command line, in which case the command will be executed and then the container will be terminated:

.. code-block:: bash

  docker run -it --rm --name dpbuddy  -v dpbuddy-work:/opt/dpbuddy/src -v dpbuddy-logs:/var/log/dpbuddy myarch/dpbuddy:3.5 dpbuddy connect


.. _upgrading from dpbuddy 3.0:

Upgrading DPBuddy
-----------------

To upgrade DPBuddy, simply un-archive the distribution into a new folder and change your ``DPBUDDY_HOME`` environment variable to point to the new location.

The DPBuddy 3.5 distribution contains Apache Ant 1.10 jar files. To avoid classloader conflicts, exclude Ant jars from the taskdef used to define DPBuddy tasks. Additionally, DPBuddy starting with version 3.2.4 uses `logback <http://logback.qos.ch/>`_ logging framework and relies on `logback.xml <http://logback.qos.ch/manual/configuration.html>`_ configuration file. "logback.xml" is located in "$DPBUDDY_HOME/conf" directory. In Ant, this directory needs to be explicitly added to the classpath of the DPBuddy's "taskdef". The resulting "taskdef" should be defined as following:

.. code-block:: xml

    <taskdef uri="antlib:com.myarch.dpbuddy">
        <classpath>
            <pathelement location="${dpbuddy.home}/conf"/>
            <fileset dir="${dpbuddy.home}/lib" excludes="ant-*"/>
        </classpath>
    </taskdef>

All DPBuddy 3.5 tasks are backward compatible with 3.0, 3.1, 3.3 and 3.4.

.. _configuring_dp:

Configuring DataPower to Work with DPBuddy
------------------------------------------

You need to make sure that you have XML Management enabled in all devices that will be managed by DPBuddy. You can do this from DataPower WebGUI (Network/Management/XML Management).

DPBuddy does not use SSH, so SSH can be disabled.

DPBuddy uses the "old" DataPower v2004 message format for file transfer (the 2004 schema is more efficient, since it allows for uploading multiple files at once). This means that the "SOAP Configuration Management (v2004)" checkbox must be checked on the XML management screen.

Be careful performing these changes on the appliances residing in a production DMZ -- make sure that the management ports are not available to the outside world.

We also recommend enabling internal logging as documented in the DataPower `Info Center <http://pic.dhe.ibm.com/infocenter/wsdatap/v6r0m0/index.jsp?topic=%2Fcom.ibm.dp.xs.doc%2Flogging_internal.html&resultof%3D%2522%2569%256e%2574%2565%2572%256e%2561%256c%2522%2520%2522%2569%256e%2574%2565%2572%256e%2522%2520%2522%256c%256f%2567%2567%2569%256e%2567%2522%2520%2522%256c%256f%2567%2522%2520>`_. DataPower does not return error messages to the client. If a DPBuddy command fails and you get an "Internal Error" message from the device, you will need to look for the root cause in the DataPower system log in the default domain. DataPower captures the error only if internal logging is enabled. You can enable it by navigating to "Troubleshooting" from the control panel, then to "Logging."  Set "Enable Internal Logging" and click on "Set Log Level." We recommend setting the level to "Warning", since anything more detailed (e.g., "Information") results in needlessly verbose messages.

If you're creating a new account that will be used by DPBuddy, make sure that the password for the account has been reset. Before running DPBuddy, make sure that you're able to login to WebGUI using this account.

DPBuddy will work with any "Group-Defined" account that has read/write access to the target domain(s). In general, DPBuddy does not need to access the "default" domain. However, certain features of DPBuddy's tasks, such as updating domain's comments during import will not work if the write access to the default domain is restricted. :ref:`quiesceDomain` tasks also require access to the default domain.

Note that by default DataPower uses a self-signed certificate for SSL. By default, DPBuddy ignores certificate validation errors. You can change this behavior by setting ``dp.trust.all.certs`` to ``false`` as described in :ref:`Common Attributes/Options and Properties <connection_options>`. If you choose this option, you will need to manually add the DataPower certificate to your JVM's keystore.

You also need to make sure that your DataPower appliance has sufficient internal space. For exports and backups DataPower first generates export files in the internal filesystem. If you're trying to export large configuration files and there is not enough space, you will get an error. You can checkOld the available temporary space by navigating to "Status/System/Filesystem information" in WebGUI.

It is highly recommended (although not absolutely required) to place all machines hosting DPBuddy and all DataPower appliances in the same time zone. DataPower relies on ZIP format for export/import; this means that file timestamps have no time zone information. This may lead to various issues if an export/import archive needs to be modified during your build/deployment process.

.. _install_config_dpbuddy_conf:

Creating DPBuddy Configuration File
-----------------------------------
All DataPower connection parameters and other configuration properties have to be defined in the :ref:`dpbuddy.conf <hoconConfig>` config file. You can follow the :ref:`quick start guide <dpbuddy_conf_quick_start>` to bootstrap its creation.

Once you created ``dpbuddy.conf``, you should keep it under version control.

``dpbuddy.conf`` is extremely flexible and should be used for all your environment and configuration settings.

To use ``dpbuddy.conf`` from CLI, define :ref:`DPBUDDY_CONFIG <confFileCli>` environment variable.

.. _configuring_ant:

Configuring Apache Ant to Work with DPBuddy
-------------------------------------------

Using DPBuddy from Apache Ant provides more flexibility and better integration with various build/deployment and release management tools than DPBuddy's CLI. It is also more efficient since there is no need to start a JVM for every command.

DPBuddy requires Apache Ant 1.8 or above.

If you don't have Apache Ant installed on your system, download and install it as documented at `http://ant.apache.org/manual/install.html <http://ant.apache.org/manual/install.html>`_. Note that on many Linux systems you can also install Ant using a package manager, e.g., you can use ``apt-get -u install ant`` on Ubuntu. We also recommend running Ant with the ``-e`` option, as it provides a much cleaner output. You can set this option globally by `setting the ANT_ARGS <http://ant.apache.org/manual/running.html>`_ environment variable to ``-e``.

DPBuddy tasks are packaged in a single Antlib. You may wish to familiarize yourself with the `Ant documentation of the Antlib feature <https://ant.apache.org/manual/Types/antlib.html>`_, especially the use of namespaces in Ant, before configuring DPBuddy tasks in your build file.

To declare DPBuddy tasks, add "taskdef" to your build file and point it to the DPBuddy installation:

.. code-block:: xml

   <property name="dpbuddy.home" location="your location" />
   <taskdef uri="antlib:com.myarch.dpbuddy">
        <classpath>
            <!-- Add conf directory to the classpath -->
            <pathelement location="${dpbuddy.home}/conf"/>
            <!-- We need to exclude Ant jars packages with DPBuddy -->
            <fileset dir="${dpbuddy.home}/lib" excludes="ant-*"/>
        </classpath>
    </classpath>

You must also declare the namespace used by DPBuddy tasks:

.. code-block:: xml

   <project name="your project" xmlns:dp="antlib:com.myarch.dpbuddy" >

The namespace's URI must match the URI used in taskdef. You must use ``antlib:com.myarch.dpbuddy`` as the URI.

You can also copy all the jar files that come with DPBuddy to ``${user.home}/.ant/lib`` directory. If you choose this option, you don't need to provide an explicit ``taskdef``. Please consult the `Ant manual <http://ant.apache.org/manual/running.html>`_ for more details.

.. _verifying_connectivity:

Verifying Connectivity with DataPower
-------------------------------------

To verify DPBuddy's connectivity with DataPower, run the following command:

.. code-block:: bash

   dpbuddy connect -env <your environment from dpbuddy.conf>


.. _advanced_configuration:

Advanced Configuration
----------------------

If you're planning to use DPBuddy to download or upload large binary files (bigger than 30MB), we recommend increasing the maximum heap size for the JVM used to run DPBuddy.

DPBuddy CLI supports the ``DPBUDDY_OPTS`` environment variable, which can be used to pass custom options to the JVM. By default, this variable is set to ``-Xms256m -Xmx1024m``, meaning that the maximum heap size is set to 1GB.

To change the heap size for Ant, set the `ANT_OPTS <http://ant.apache.org/manual/running.html#options/>`_ environment variable.

``-Xmx1648m`` is the recommended setting for both Ant and DPBuddy CLI.

If you're going to run DPBuddy behind a proxy, add the following Java system properties to the ``DPBUDDY_OPTS`` or ``ANT_OPTS`` variables: ``-Dhttps.proxyHost=<your proxy host> -Dhttps.proxyPort=<your proxy port>`` (note "https" prefix), or use `Ant's setProxy task <https://ant.apache.org/manual/proxy.html>`_.

.. _dpbuddy_license:

DPBuddy License
---------------

All DPBuddy downloads are bundled with a 45-day evaluation license. You do not need to request or install the evaluation license; it is generated automatically when you run DPBuddy for the first time.

Once the evaluation license is expired, you'll need to purchase the license from MyArch or request an extension of your trial license. Please send your request to `license at myarch.com <mailto:license@myarch.com>`_ or submit it at `http://myarch.com/dpbuddy-pricing <http://myarch.com/dpbuddy-pricing>`_.

To install a permanent license that you received from us, simply save the license file and set ``DPBUDDY_LICENSE_FILE`` environment variable to point to the license file (must be an absolute path to the file, e.g. ``export DPBUDDY_LICENSE_FILE=/opt/dpbuddy/dpbuddy-license``.

You can also save the license file in ``$DPBUDDY_HOME/conf`` (``%DPBUDDY_HOME%\conf`` on Windows) or, if you're using our docker image, under ``/opt/dpbuddy/src``. In this case, you don't need to specify any environment variables.

Alternative, you can add the property ``dpbuddy.license.file`` to your configuration file or Ant build file.  You can also provide the ``license`` option or attribute to a DPBuddy command or task.

.. _upgrading_from_dpbuddy_2_3:

Upgrading from DPBuddy 2.3
--------------------------

DPBuddy 3 is mostly backward compatible with DPBuddy 2.3 with a few exceptions:

* The namespace for the DPBuddy antlib has changed to ``com.myarch.dpbuddy``. Make sure to update it in the ``project`` tag and in the ``taskdef`` of your build file, e.g., ``xmlns:dp="antlib:com.myarch.dpbuddy"``.

* The ``request`` task was renamed to :ref:`somaRequest`.

* The ``setConfig`` task was deprecated; use :ref:`import` instead.

* :ref:`modifyConfig` no longer supports the nested ``configFile`` element. You can replace it with the new :ref:`transform element <transform_elt>`.

* ``toDir`` is now a required attribute of the :ref:`copy`. You can also provide ``toDir`` at the nested ``dpFileset`` level.

* :ref:`Environment prefix mechanism <managing_env_vars>` has been substantially enhanced; you may need to change the way your properties are named to take advantage of the hierarchical prefixes.