.. highlight:: xml .. index:: logstash .. _logstash_main: Integration with Logstash and Elasticsearch =========================================== DPBuddy allows you to collect and analyze DataPower logs and various metrics and statuses (e.g., memory, CPU, objects up/down) using the popular `open-source Elasticsearch/Logstash/Kibana (ELK) `_ stack. To enable log collection, DPBuddy provides plugins for `Logstash `_. DPBuddy supports the syslog format as well as SOAP/XML supported by DataPower. You can use the "push" approach by creating log target in DataPower pointing to the machine running Logstash with the syslog or HTTP listeners (configured as Logstash input plugins). Alternatively, you can periodically "pull" log files from DataPower using :ref:`dpdownload plugin `, in which case no special DataPower configuration is required. DPBuddy also comes with a number of predefined queries for Kibana to make it easier to analyze DataPower logs and metrics. You can also easily create your own queries and dashboards using powerful `Kibana visualization capabilities `_. Before you can start analyzing your logs in ELK, you need to install the plugins provided by DPBuddy. .. _logstash_plugin_install: DPBuddy's Logstash Plugins Installation --------------------------------------- DPBuddy plugins are regular Logstash plugins, thus you can use all the `plugin management commands `_ that come with Logstash. Use `logstash-plugin install `_ Logstash command to install DPBuddy plugins. Before installing DPBuddy plugins, you will need to add MyArch plugin repository to Logstash. To enable this repository, modify the file "Gemfile" located in the root of your Logstash installation (e.g., "/opt/logstash-5.2.0/Gemfile"). Usually, this file defines only the default rubygems repository at the top: .. code-block:: ruby source "https://rubygems.org" Insert the following line right after the line above: .. code-block:: ruby source "http://nexus.myarch.com/repository/rubygems/" Now you can run ``logstash-plugin install`` command to install a plugin, e.g.: .. code-block:: bash bin/logstash-plugin install logstash-filter-datapowerlog .. _logstash_elk_setup: Elastisearch/Kibana Setup ------------------------- We recommend saving DataPower logs into its own set of indexes to distinguish from other data that can be stored in Elastisearch. Logstash will create the Elastisearch index automatically when you start sending data to Elasticsearch. It will also rotate indexes based on the template provided to the `index setting `_ of the "elasticsearch" plugin. We recommend creating an index template for indexes that will be used for DataPower logs and metrics as following: .. code-block:: js PUT _template/dp-log { "template": "dp-log*", "mappings": { "_default_": { "properties": { "tech": { "type": "string", "index": "not_analyzed" }, "event_type": { "type": "string", "index": "not_analyzed" }, "host": { "type": "string", "index": "not_analyzed" }, "node": { "type": "string", "index": "not_analyzed" }, "type": { "type": "string" } } } } } .. hint:: You can run all Elasticsearch APIs from the Dev tools console in Kibana. To start viewing data in Kibana, go to "Management/Index Patterns", add and select the index "dp*". You can then go back to "Discover" in Kibana and start viewing your logs and metrics. You can easily create custom queries in Kibana using its intuitive UI. You can also download and import our canned DataPower-related searches from `MyArch website `_. Once you downloaded the JSON file with the queries, import it into your Kibana instance from the "Management/Kibana/Saved Objects" screen. You can then modify these searches as needed. .. _logstash_logfilter: Logstash DataPower Log Plugin ------------------------------- DPBuddy provides a `Logstash filter plugin `_ that parses DataPower log records and generates JSON for consumption in Elasiticsearch or any other log management tool supported by Logstash via one of its `output plugins `_. The filter plugin supports DataPower syslog and `SOAP/XML `_ log formats. The plugin extracts all the key data elements from DataPower log entries, including DataPower object/class, domain, transaction ID, global transaction ID (gtid), DataPower host name, level, category and others. These data elements are available in Elasticsearch/Kibana as fields that can be searched on or used in visualizations. Normally, these fields are embedded in syslog messages using DataPower-proprietary format, so the plugin saves you a lot of time by not having to parse the syslog messages using regexps or a similar labor-intensive approach. An output of the plugin may look like the following in JSON format: .. code-block:: js { "@timestamp": "2017-07-05T18:21:25.754Z", "appName": "app-services", "category": "xmlfirewall", "client_host": "172.16.253.1", "dp": { "class": "xmlfirewall", "domain": "e2e-test", "object": "testFirewall" }, "environment": "dev", "event_type": "LOG_ENTRY", "global_tran_id": 89024, "host": "172.16.253.15", "level": "info", "message": "rule (testFirewall_request): selected via match 'testFirewall' from processing policy 'testFirewall'", "message_id": "0x80e000b4", "src_log_format": "syslog", "tags": [ "datapower", "datapower_log" ], "tech": "DataPower", "tran_id": 89024, "tran_type": "request", "type": "dp.log" } The plugin integrates with the :ref:`dpbuddy config file (dpbuddy.conf) `. It reads DataPower environment/device configuration from that file and it can optionally add some information from the environment definition (e.g., an environment name or other metadata) as fields to the Elasticsearch events. This allows you to tag log entries with the information about the DataPower, domains or services the logs originated from. The plugin supports the following log message formats, the format can be provided using the "format" setting of the plugin: * syslog. Requires using `Logstash syslog `_ input and the syslog log target type on DataPower. * xml. Requires using `Logstash http `_ input and the SOAP log target type on DataPower. The target format must be set to "XML" and the timestamp format must be set to "syslog". * file. Requires using `Logstash file `_ input and the "file" target type on DataPower. The log format must be "syslog" and the timestamp format must be set to "zulu". Log files could be mounted on a shared NFS filesystem or you can use :ref:`dpdownload ` plugin to periodically download log files from DataPower. * auto. DPBuddy will automatically determine the format of a log message based on the metadata generated by Logstash. .. hint:: For syslog and xml formats, the machine running Logstash must be visible to DataPower and its ports must be open so that DataPower can send traffic to it. You also need to ensure accurate clock setting/NTP in all DataPower devices. To start using the plugin, create appropriate log targets (unless you want to use "dpdownload" plugin, see below) in DataPower and then install the plugin in Logstash: .. code-block:: bash bin/logstash-plugin install logstash-filter-datapowerlog Next, create a `Logstash pipeline `_ that includes a DataPower input (syslog or http or file), the DataPower log filter and an output (e.g., Elasticsearch). Here is an example of the syslog pipeline: .. code-block:: ruby input { # Input to handle syslog log target on DataPower. Define the target and point it to the the Logstash server. This input handles both UDP and TCP syslog syslog { port => 5514 tags => [datapower, datapower_log] } } filter { if "datapower_log" in [tags] { # DataPower Log plugin provided by DPBuddy. The plugin will convert syslog messages coming from DataPower to proper JSON datapowerlog { conf_file => "${LS_DP_PLUGIN_HOME}/../conf/dpbuddy.conf" } } } output { # You can dump the output to the console for testing purposes stdout { codec => json_lines } # Save the output of the filter plugin in Elasticsearch if "datapower_log" in [tags] { elasticsearch { # Name of the index to store DataPower logs # Use patterns so that the new index is created daily index => "dp-log-%{+YYYY.MM.dd}" # Update these values to specify where Elasticsearch instances are located hosts => ["127.0.0.1:9200","127.0.0.2:9200"] } } } SOAP/XML/HTTP pipeline can be defined in a similar way (only the input and the filter are shown): .. code-block:: ruby input { # Input to handle HTTP/SOAP log messages (point the DataPower log target to :6080) http { port => 6080 tags => [datapower, datapower_log] } } filter { if "datapower_log" in [tags] { # DataPower Log plugin provided by DPBuddy. The plugin will convert SOAP/XML coming from DataPower to proper JSON datapowerlog {} } } The DataPower log plugin creates a number of fields, including log category, log level, transaction id, DataPower domain, DataPower object class/name. It also correctly sets the "@timestamp" field based on the content of a SOAP message or a log file. You can run your pipeline as usual: .. code-block:: bash bin/logstash -f syslog-dp-input.conf Plugin Configuration Options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: :widths: 20 80 8 :header-rows: 1 * - Name - Description - Required * - add_tag - If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax. You must provide tags in the array format, e.g., ``add_tag => [ "foo_%{somefield}", "taggedy_tag"]`` - No * - conf_file - Path to the :ref:`dpbuddy config file (dpbuddy.conf) `. The plugin will find a matching entry in dpbuddy.conf and use "environment" property of that entry as the value for the "env" field. Additionally, all fields with the key "analyticsInfo" will be copied into the Elasticsearch event. The plugin uses the value of the "host" field to search for an environment definition in dpbuddy.conf. The host must match either "dpUrl" or "host" in the environment. - No * - format - "auto", "syslog", "xml" or "file". Defaults to ``auto``. - No * - infer_host_domain - This setting applies to the "file" format. "dpdownload" plugin can automatically create directory structure in the format host/domain. The logs will be stored under the appropriate domain. If "infer_host_domain" is set to ``true``, the plugin will use the parent directory of the log file as the domain name (unless the domain name is already present in the log entry) and its parent as the host name. - No * - preserve_fields - If set to ``true``, preserve all event fields. This will include http header fields set by the http input plugin, original (unparsed) text of the log message, various syslog fields. Defaults to ``false``. In this case, the plugin preserves only the "tags" fields, all other fields are dropped. - No When dealing with the syslog format and when the log target is defined at a domain level, the plugin allows for using the "Local identifier" field in DataPower to specify the host alias and the domain in the format "hostalias--domainname". Note the use of two dashes as the separator. For example, when encountering the identifier "dp7-qa--e2e-test", DPBuddy will add "dp7-qa" to the array of aliases and it will use "e2e-test" as the domain name. Note that the "Local identifier" field must not contain underscores. Below is an example of using dpbuddy.conf for providing additional metadata for log entries (when "conf_file" is set): .. code-block:: js // dev environment dev: { dpUrl: dev-dp // this value will be used in the "env" field of the log entry environment: "dp-dev" dpDomain: default // host can be used for matching the host from the log entry if it is different from dpUri host: "172.16.253.15" // we can add any fields to the log entry, e.g., the appName analyticsInfo: {appName: fin-services} } .. _logstash_dpdownload: Logstash DataPower File Download Plugin --------------------------------------- DPBuddy provides a `Logstash input plugin `_ that periodically downloads log files from DataPower using the DPBuddy's :ref:`download command `. The plugin downloads files only if they're changed on DataPower. The plugin should be used in conjunction with the `Logstash file plugin `_. The file plugin detects changes in the downloaded files (including log rotation) and "tails" them. It creates events containing unparsed message and other fields. :ref:`DataPower log plugin ` will then parse the message field and create DataPower-specific fields, such as DataPower object's class/name. To install the plugin (make sure that MyArch plugin repository was added to Logstash before running this command): .. code-block:: bash bin/logstash-plugin install logstash-filter-dpdownload Then create your pipeline, e.g.: .. code-block:: ruby input { dpdownload { interval => 60 device => "dp1,dp2" conf_file => "./conf/dpbuddy.conf" # Download default log files, including rotated files, excluding XML logs # We recommend creating custom file target instead of directly using the default log include => "logtemp:/default-log(\..*|$)" # save to dplogs, append the host name to_dir => "/var/dplogs" } file { # read all updated files in the dplogs__ directory, the path must be absolute path => "/var/dplogs*/*" start_position => beginning tags => [datapower, datapower_log] } } filter { # Parse log messages, create DataPower-specific fields datapowerlog {} } Plugin Configuration Options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Most of the configuration options of the plugin are the same with the :ref:`download command `. To be consistent with other Logstash plugins, the underscore naming convention is used for option names. .. list-table:: :widths: 20 80 8 :header-rows: 1 * - Name - Description - Required * - conf_file - Path to the :ref:`DPBuddy configuration file `. The "device" option must point to environment names from that file. - Yes * - create_host_dir - If set to ``true``, create subdirectory under 'to_dir' with the name matching DataPower host name. If the files are downloaded from the "default" domain, the domain subdirectories will be created automatically. :ref:`DataPower log plugin ` utilizes this naming convention to determine the host name and the environment of the DataPower device the log file(s) was downloaded from. This is also convenient when multiple devices were specified in the "device". Defaults to ``true``. - No * - device - :ref:`Environment (device) ` name from the :ref:`DPBuddy configuration file `. The device definition includes DataPower host or url, username/password. You can provide multiple environments/devices separated by comma. In this case, the plugin will be downloading files from each of the devices. - Yes * - domain - DataPower domain to download the logs from. Defaults to ``default``. - No * - download_if_changed - If set to ``true``, download only the files that are newer on DataPower than the ones that exist locally. Defaults to ``true``. - No * - include - Regular expression pattern defining what files to download. The match is done against the entire path of a file, including the filesystem (e.g., ``local:/``). Note that the filesystem is separated by a single slash, i.e., the pattern ``local://.*xsd`` will not match anything. - Yes * - interval - Interval to run the the download command. Value is in seconds. The interval should be determined based on the volume of messages in your log files Defaults to 120 seconds. - No * - to_dir - Local directory where to save files. - Yes You can use "dpdownload" plugin to download default logs. Use the following pattern to achieve that: .. code-block:: ruby include => "logtemp:/default-log(\..*|$)" Custom log files (created using custom log targets) can be downloaded as following: .. code-block:: ruby include => "logstore:/e2e-test/.*log" .. _logstash_dpauditlog: Saving DPBuddy Audit Log in Elasticsearch ----------------------------------------- DPBuddy generates detailed :ref:`audit log in JSON format `. It can be beneficial to save this log in Elasticsearch, this can greatly facilitate troubleshooting or any kind of log analysis. This can be done using Logstash by creating a very simple pipeline with the "file" input. You can also make it part of your main pipeline dealing with DataPower logs: .. code-block:: ruby input { # Input to handle syslog log target on DataPower. Define the target and point it to the the Logstash server. This input handles both UDP and TCP syslog syslog { port => 5514 tags => [datapower, datapower_log] } # Pick up DPBuddy audit log too file { path => "/Users/aananiev/.dpbuddy/log/dpbuddy-audit.log" start_position => beginning codec => json tags => [datapower, datapower_audit] } } filter { if "datapower_log" in [tags] { # DataPower Log plugin provided by DPBuddy. The plugin will convert syslog events coming from DataPower to proper JSON datapowerlog { conf_file => "${LS_DP_PLUGIN_HOME}/../conf/dpbuddy.conf" } } } output { # You can dump the output to the console for testing purposes stdout { codec => json_lines } # Save the output of the filter plugin in Elasticsearch if "datapower_log" in [tags] { elasticsearch { # What index do we want to use to store the data? # Use patterns so that the new index is created daily index => "dp-log-%{+YYYY.MM.dd}" # Update to specify where Elasticsearch is located hosts => ["127.0.0.1:9200","127.0.0.2:9200"] } } # save audit file in a different index if "datapower_audit" in [tags] { elasticsearch { # What index do we want to use to store the data? # Use patterns so that the new index is created daily index => "dp-audit-%{+YYYY.MM.dd}" # Update to specify where Elasticsearch is located hosts => ["127.0.0.1:9200","127.0.0.2:9200"] } } } Another option could be to use the `filebeat shipper `_ instead of Logstash. .. ``Automating Creating DataPower Log Targets`` You can use DPBuddy to automate creation of log targets in your appliances. Create the _common_event_model: ``Event Fields Reference`` -------------------------- ``Common Event Fields`` ^^^^^^^^^^^^^^^^^^^^^^^ These fields are available in all plugins provided by MyArch: ``Log Fields`` ^^^^^^^^^^^^^^ These fields are set for events produced from log entries (mention orig_message)