Integration with Logstash and Elasticsearch

DPBuddy allows you to collect and analyze DataPower logs and various metrics and statuses (e.g., memory, CPU, objects up/down) using the popular open-source Elasticsearch/Logstash/Kibana (ELK) stack.

To enable log collection, DPBuddy provides plugins for Logstash.

DPBuddy supports the syslog format as well as SOAP/XML supported by DataPower. You can use the “push” approach by creating log target in DataPower pointing to the machine running Logstash with the syslog or HTTP listeners (configured as Logstash input plugins). Alternatively, you can periodically “pull” log files from DataPower using dpdownload plugin, in which case no special DataPower configuration is required.

DPBuddy also comes with a number of predefined queries for Kibana to make it easier to analyze DataPower logs and metrics. You can also easily create your own queries and dashboards using powerful Kibana visualization capabilities.

Before you can start analyzing your logs in ELK, you need to install the plugins provided by DPBuddy.

DPBuddy’s Logstash Plugins Installation

DPBuddy plugins are regular Logstash plugins, thus you can use all the plugin management commands that come with Logstash.

Use logstash-plugin install Logstash command to install DPBuddy plugins.

Before installing DPBuddy plugins, you will need to add MyArch plugin repository to Logstash.

To enable this repository, modify the file “Gemfile” located in the root of your Logstash installation (e.g., “/opt/logstash-5.2.0/Gemfile”). Usually, this file defines only the default rubygems repository at the top:

source "https://rubygems.org"

Insert the following line right after the line above:

source "http://nexus.myarch.com/repository/rubygems/"

Now you can run logstash-plugin install command to install a plugin, e.g.:

bin/logstash-plugin install logstash-filter-datapowerlog

Elastisearch/Kibana Setup

We recommend saving DataPower logs into its own set of indexes to distinguish from other data that can be stored in Elastisearch.

Logstash will create the Elastisearch index automatically when you start sending data to Elasticsearch. It will also rotate indexes based on the template provided to the index setting of the “elasticsearch” plugin.

We recommend creating an index template for indexes that will be used for DataPower logs and metrics as following:

PUT _template/dp-log
{
    "template": "dp-log*",
    "mappings": {
        "_default_": {
            "properties": {
                "tech": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "event_type": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "host": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "node": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "type": {
                    "type": "string"
                }
            }
        }
    }
}

Hint

You can run all Elasticsearch APIs from the Dev tools console in Kibana.

To start viewing data in Kibana, go to “Management/Index Patterns”, add and select the index “dp*”. You can then go back to “Discover” in Kibana and start viewing your logs and metrics.

You can easily create custom queries in Kibana using its intuitive UI.

You can also download and import our canned DataPower-related searches from MyArch website. Once you downloaded the JSON file with the queries, import it into your Kibana instance from the “Management/Kibana/Saved Objects” screen. You can then modify these searches as needed.

Logstash DataPower Log Plugin

DPBuddy provides a Logstash filter plugin that parses DataPower log records and generates JSON for consumption in Elasiticsearch or any other log management tool supported by Logstash via one of its output plugins.

The filter plugin supports DataPower syslog and SOAP/XML log formats.

The plugin extracts all the key data elements from DataPower log entries, including DataPower object/class, domain, transaction ID, global transaction ID (gtid), DataPower host name, level, category and others. These data elements are available in Elasticsearch/Kibana as fields that can be searched on or used in visualizations.

Normally, these fields are embedded in syslog messages using DataPower-proprietary format, so the plugin saves you a lot of time by not having to parse the syslog messages using regexps or a similar labor-intensive approach.

An output of the plugin may look like the following in JSON format:

{
    "@timestamp": "2017-07-05T18:21:25.754Z",
    "appName": "app-services",
    "category": "xmlfirewall",
    "client_host": "172.16.253.1",
    "dp": {
        "class": "xmlfirewall",
        "domain": "e2e-test",
        "object": "testFirewall"
    },
    "environment": "dev",
    "event_type": "LOG_ENTRY",
    "global_tran_id": 89024,
    "host": "172.16.253.15",
    "level": "info",
    "message": "rule (testFirewall_request): selected via match 'testFirewall' from processing policy 'testFirewall'",
    "message_id": "0x80e000b4",
    "src_log_format": "syslog",
    "tags": [
        "datapower",
        "datapower_log"
    ],
    "tech": "DataPower",
    "tran_id": 89024,
    "tran_type": "request",
    "type": "dp.log"
}

The plugin integrates with the dpbuddy config file (dpbuddy.conf). It reads DataPower environment/device configuration from that file and it can optionally add some information from the environment definition (e.g., an environment name or other metadata) as fields to the Elasticsearch events. This allows you to tag log entries with the information about the DataPower, domains or services the logs originated from.

The plugin supports the following log message formats, the format can be provided using the “format” setting of the plugin:

• syslog. Requires using Logstash syslog input and the syslog log target type on DataPower.
• xml. Requires using Logstash http input and the SOAP log target type on DataPower. The target format must be set to “XML” and the timestamp format must be set to “syslog”.
• file. Requires using Logstash file input and the “file” target type on DataPower. The log format must be “syslog” and the timestamp format must be set to “zulu”. Log files could be mounted on a shared NFS filesystem or you can use dpdownload plugin to periodically download log files from DataPower.
• auto. DPBuddy will automatically determine the format of a log message based on the metadata generated by Logstash.

Hint

For syslog and xml formats, the machine running Logstash must be visible to DataPower and its ports must be open so that DataPower can send traffic to it.

You also need to ensure accurate clock setting/NTP in all DataPower devices.

To start using the plugin, create appropriate log targets (unless you want to use “dpdownload” plugin, see below) in DataPower and then install the plugin in Logstash:

bin/logstash-plugin install logstash-filter-datapowerlog

Next, create a Logstash pipeline that includes a DataPower input (syslog or http or file), the DataPower log filter and an output (e.g., Elasticsearch).

Here is an example of the syslog pipeline:

input {
    # Input to handle syslog log target on DataPower. Define the target and point it to the the Logstash server. This input handles both UDP and TCP syslog
    syslog {
        port => 5514
        tags => [datapower, datapower_log]
    }
}

filter {
    if "datapower_log" in [tags] {
        # DataPower Log plugin provided by DPBuddy. The plugin will convert syslog messages coming from DataPower to proper JSON
        datapowerlog {
          conf_file => "${LS_DP_PLUGIN_HOME}/../conf/dpbuddy.conf"
        }
    }
}

output {
    # You can dump the output to the console for testing purposes
    stdout {
        codec => json_lines
    }
    # Save the output of the filter plugin in Elasticsearch
    if "datapower_log" in [tags] {
        elasticsearch {
            # Name of the index to store DataPower logs
            # Use patterns so that the new index is created daily
            index => "dp-log-%{+YYYY.MM.dd}"
            # Update these values to specify where Elasticsearch instances are located
            hosts => ["127.0.0.1:9200","127.0.0.2:9200"]
        }
    }
}

SOAP/XML/HTTP pipeline can be defined in a similar way (only the input and the filter are shown):

input {
  # Input to handle HTTP/SOAP log messages (point the DataPower log target to <logstash host>:6080)
  http {
      port => 6080
      tags => [datapower, datapower_log]
  }
}

filter {
    if "datapower_log" in [tags] {
        # DataPower Log plugin provided by DPBuddy. The plugin will convert SOAP/XML coming from DataPower to proper JSON
        datapowerlog {}
     }
}

The DataPower log plugin creates a number of fields, including log category, log level, transaction id, DataPower domain, DataPower object class/name. It also correctly sets the “@timestamp” field based on the content of a SOAP message or a log file.

You can run your pipeline as usual:

bin/logstash -f syslog-dp-input.conf

Plugin Configuration Options

Name	Description	Required
add_tag	If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax. You must provide tags in the array format, e.g., add_tag => [ "foo_%{somefield}", "taggedy_tag"]	No
conf_file	Path to the dpbuddy config file (dpbuddy.conf). The plugin will find a matching entry in dpbuddy.conf and use “environment” property of that entry as the value for the “env” field. Additionally, all fields with the key “analyticsInfo” will be copied into the Elasticsearch event. The plugin uses the value of the “host” field to search for an environment definition in dpbuddy.conf. The host must match either “dpUrl” or “host” in the environment.	No
format	“auto”, “syslog”, “xml” or “file”. Defaults to auto.	No
infer_host_domain	This setting applies to the “file” format. “dpdownload” plugin can automatically create directory structure in the format host/domain. The logs will be stored under the appropriate domain. If “infer_host_domain” is set to true, the plugin will use the parent directory of the log file as the domain name (unless the domain name is already present in the log entry) and its parent as the host name.	No
preserve_fields	If set to true, preserve all event fields. This will include http header fields set by the http input plugin, original (unparsed) text of the log message, various syslog fields. Defaults to false. In this case, the plugin preserves only the “tags” fields, all other fields are dropped.	No

When dealing with the syslog format and when the log target is defined at a domain level, the plugin allows for using the “Local identifier” field in DataPower to specify the host alias and the domain in the format “hostalias–domainname”. Note the use of two dashes as the separator.

For example, when encountering the identifier “dp7-qa–e2e-test”, DPBuddy will add “dp7-qa” to the array of aliases and it will use “e2e-test” as the domain name.

Note that the “Local identifier” field must not contain underscores.

Below is an example of using dpbuddy.conf for providing additional metadata for log entries (when “conf_file” is set):

// dev environment
dev: {
    dpUrl: dev-dp
    // this value will be used in the "env" field of the log entry
    environment: "dp-dev"
    dpDomain: default
    // host can be used for matching the host from the log entry if it is different from dpUri
    host: "172.16.253.15"
    // we can add any fields to the log entry, e.g., the appName
    analyticsInfo: {appName: fin-services}
}

Logstash DataPower File Download Plugin

DPBuddy provides a Logstash input plugin that periodically downloads log files from DataPower using the DPBuddy’s download command. The plugin downloads files only if they’re changed on DataPower. The plugin should be used in conjunction with the Logstash file plugin. The file plugin detects changes in the downloaded files (including log rotation) and “tails” them. It creates events containing unparsed message and other fields. DataPower log plugin will then parse the message field and create DataPower-specific fields, such as DataPower object’s class/name.

To install the plugin (make sure that MyArch plugin repository was added to Logstash before running this command):

bin/logstash-plugin install logstash-filter-dpdownload

Then create your pipeline, e.g.:

input {

    dpdownload {
        interval => 60
        device => "dp1,dp2"
        conf_file => "./conf/dpbuddy.conf"
        # Download default log files, including rotated files, excluding XML logs
        # We recommend creating custom file target instead of directly using the default log
        include => "logtemp:/default-log(\..*|$)"
        # save to dplogs, append the host name
        to_dir => "/var/dplogs"

    }

    file {
        # read all updated files in the dplogs__<host name> directory, the path must be absolute
        path => "/var/dplogs*/*"
        start_position => beginning
        tags => [datapower, datapower_log]
    }

}

filter {
    # Parse log messages, create DataPower-specific fields
    datapowerlog {}
}

Plugin Configuration Options

Most of the configuration options of the plugin are the same with the download command. To be consistent with other Logstash plugins, the underscore naming convention is used for option names.

Name	Description	Required
conf_file	Path to the DPBuddy configuration file. The “device” option must point to environment names from that file.	Yes
create_host_dir	If set to true, create subdirectory under ‘to_dir’ with the name matching DataPower host name. If the files are downloaded from the “default” domain, the domain subdirectories will be created automatically. DataPower log plugin utilizes this naming convention to determine the host name and the environment of the DataPower device the log file(s) was downloaded from. This is also convenient when multiple devices were specified in the “device”. Defaults to true.	No
device	Environment (device) name from the DPBuddy configuration file. The device definition includes DataPower host or url, username/password. You can provide multiple environments/devices separated by comma. In this case, the plugin will be downloading files from each of the devices.	Yes
domain	DataPower domain to download the logs from. Defaults to default.	No
download_if_changed	If set to true, download only the files that are newer on DataPower than the ones that exist locally. Defaults to true.	No
include	Regular expression pattern defining what files to download. The match is done against the entire path of a file, including the filesystem (e.g., local:/). Note that the filesystem is separated by a single slash, i.e., the pattern local://.*xsd will not match anything.	Yes
interval	Interval to run the the download command. Value is in seconds. The interval should be determined based on the volume of messages in your log files Defaults to 120 seconds.	No
to_dir	Local directory where to save files.	Yes

You can use “dpdownload” plugin to download default logs. Use the following pattern to achieve that:

include => "logtemp:/default-log(\..*|$)"

Custom log files (created using custom log targets) can be downloaded as following:

include => "logstore:/e2e-test/.*log"

Saving DPBuddy Audit Log in Elasticsearch

DPBuddy generates detailed audit log in JSON format. It can be beneficial to save this log in Elasticsearch, this can greatly facilitate troubleshooting or any kind of log analysis.

This can be done using Logstash by creating a very simple pipeline with the “file” input. You can also make it part of your main pipeline dealing with DataPower logs:

input {
    # Input to handle syslog log target on DataPower. Define the target and point it to the the Logstash server. This input handles both UDP and TCP syslog
    syslog {
        port => 5514
        tags => [datapower, datapower_log]
    }

    # Pick up DPBuddy audit log too
    file {

        path => "/Users/aananiev/.dpbuddy/log/dpbuddy-audit.log"
        start_position => beginning
        codec => json
        tags => [datapower, datapower_audit]
    }
}

filter {
    if "datapower_log" in [tags] {
        # DataPower Log plugin provided by DPBuddy. The plugin will convert syslog events coming from DataPower to proper JSON
        datapowerlog {
            conf_file => "${LS_DP_PLUGIN_HOME}/../conf/dpbuddy.conf"
        }
    }
}

output {
    # You can dump the output to the console for testing purposes
    stdout {
        codec => json_lines
    }
    # Save the output of the filter plugin in Elasticsearch
    if "datapower_log" in [tags] {
        elasticsearch {
            # What index do we want to use to store the data?
            # Use patterns so that the new index is created daily
            index => "dp-log-%{+YYYY.MM.dd}"
            # Update to specify where Elasticsearch is located
            hosts => ["127.0.0.1:9200","127.0.0.2:9200"]
        }

    }
    # save audit file in a different index
    if "datapower_audit" in [tags] {
        elasticsearch {
            # What index do we want to use to store the data?
            # Use patterns so that the new index is created daily
            index => "dp-audit-%{+YYYY.MM.dd}"
            # Update to specify where Elasticsearch is located
            hosts => ["127.0.0.1:9200","127.0.0.2:9200"]
        }
    }
}

Another option could be to use the filebeat shipper instead of Logstash.