Configure Datadog metrics collection for Consul on Kubernetes

This page describes the processes for integrating Datadog metrics collection in your Consul on Kubernetes deployment. The Helm chart includes automated configuration options to simplify the integration process.

Datadog Metrics Integration Methods

• DogstatsD
• Datadog Checks: Official Consul Integration
• Openmetrics Prometheus

Users should choose one integration method from the three described below that best suites the intent for metrics collection. DogStatsD, Consul Integration, and Openmetrics Prometheus methods of integration are mutually exclusive.

Reasoning: The consul-k8s helm chart automated configuration implements Datadog's Consul Integration method using the use_prometheus_endpoint configuration parameter. DogstatsD, Consul Integration, and Openmetrics Prometheus Metrics by design share the same metric name syntax for collection, and would therefore cause a conflict. The consul.py integration source code, as well as the consul-k8s helm chart prohibit the enablement of more that one integration at a time.

DogstatsD

This method of implementation leverages the hashicorp/go-metrics DogstatsD client library to manage metrics collection. Metrics are aggregated and sent via UDP or UDS transports to a Datadog Agent that runs on the same Kube Node as the Consul servers.

Enabling this method of metrics collection allows Consul to control the delivery of metrics traffic directly to a Datadog agent rather than a Datadog agent attempting to reach Consul and scrape the /v1/agent/metrics API endpoint.

This is accomplished by updating each server agent's configuration telemetry stanza.

Helm Chart Configuration

metrics:
  enabled: true
  enableAgentMetrics: true
  datadog:
    enabled: true
    namespace: "datadog"
    dogstatsd:
      enabled: true
      socketTransportType: "UDS"
      dogstatsdAddr: "/var/run/datadog/dsd.socket"

{
  "telemetry": {
    "dogstatsd_addr": "unix:///var/run/datadog/dsd.socket"
  }
}

metrics:
  enabled: true
  enableAgentMetrics: true
  datadog:
    enabled: true
    namespace: "datadog"
    dogstatsd:
      enabled: true
      socketTransportType: "UDP"
      # Set `dogstatsdPort` to `0` (default) to omit port number append to address.
      dogstatsdPort: 0
      dogstatsdAddr: "datadog-agent.datadog.svc.cluster.local"

{
  "telemetry": {
    "dogstatsd_addr": "datadog-agent.datadog.svc.cluster.local"
  }
}

metrics:
  enabled: true
  enableAgentMetrics: true
  datadog:
    enabled: true
    namespace: "datadog"
    dogstatsd:
      enabled: true
      socketTransportType: "UDP"
      dogstatsdPort: 8125
      dogstatsdAddr: "172.20.180.10"

{
  "telemetry": {
    "dogstatsd_addr": "172.20.180.10:8125",
  }
}

UDS/UDP Advantages and Disadvantages

This integration method accomplishes metrics collection by leveraging either Unix Domain Sockets (UDS) or User Datagram Protocol (UDP) transport. Practitioners who manage their Kubernetes infrastructure and/or service-mesh should take into account the implications outlined in the tables below.

UDS

Packet Transport: Unix Domain Socket File

UDP

Packet Transport:

• Kubernetes Service IP:Port
• Container Host Port

Verifying DogstatsD Metric Collection

To confirm you're Datadog agent is receiving traffic, the status subcommand can be ran from the Datadog Agent expecting to receive DogstatsD traffic from Consul.

There should be an increase in either UDP or UDS traffic packet counts from the resultant output after the configuration has been properly established.

# Example: UDP Packet and Metric Packet Traffic Increase
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 5,908
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 636,872
  Udp Packet Reading Errors: 0
  Udp Packets: 3,300
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0
  Unterminated Metric Errors: 0

# Example: UDS Packet and Metric Packet Traffic Increase
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 30,523
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 124,635
  Udp Packet Reading Errors: 0
  Udp Packets: 731
  Uds Bytes: 2,957,433
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 11,563
  Unterminated Metric Errors: 0

Traffic verification can also be accomplished using the netstat command line utility from a consul-server expected to be submitting metrics data to Datadog.

$ netstat -nup | grep "172.28.13.12:8125.*ESTABLISHED
udp        0      0 127.0.0.1:53874         127.0.0.1:8125          ESTABLISHED 23176/consul

$ netstat -x
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ]         DGRAM      CONNECTED     15952473
unix  2      [ ]         DGRAM                    15652537 @9d10c

UDS provides the additional capability for verification by sending a test metrics packet to the Unix Socket configured.

$ echo -n "custom.metric.name:1|c" | nc -U -u -w1 /var/run/datadog/dsd.socket
Bound on /tmp/nc-IjJkoG/recv.sock

Use Case

DogstatsD integration provides full-scope metrics collection from Consul, and minimizes access control configuration requirements as traffic flow is outbound (toward the Datadog Agent) as opposed to inbound (toward the /v1/agent/metrics/ API endpoint).

Metrics Data Collected

• Full list of metrics sent via DogstatsD consists of those listed in the Agent Telemetry documentation.

Datadog Checks: Official Consul Integration

The Datadog Agent package includes official third-party integrations for built-in availability upon agent deployment.

The Consul Integration Datadog checks provided some additional metric verification checks that leverage Consul's built-in feature-set, and help monitor Consul during normal operation beyond that of Consul's available metrics.

See the below table for an outline of the features added by the official integration.

Helm Chart Configuration

global:
  tls:
    enabled: true
    enableAutoEncrypt: true
  acls:
    manageSystemACLs: true
  metrics:
    enabled: true
    enableAgentMetrics: true
    datadog:
      enabled: true
      namespace: "datadog"

"ad.datadoghq.com/consul.checks": |
  {
    "consul": {
    "init_config": {},
    "instances": [
        {
          "url": "https://consul-server.consul.svc:8501",
          "tls_cert": "/etc/datadog-agent/conf.d/consul.d/certs/tls.crt",
          "tls_private_key": "/etc/datadog-agent/conf.d/consul.d/certs/tls.key",
          "tls_ca_cert": "/etc/datadog-agent/conf.d/consul.d/ca/tls.crt",
          "use_prometheus_endpoint": true,
          "acl_token": "ENC[k8s_secret@consul/consul-datadog-agent-metrics-acl-token/token]",
          "new_leader_checks": true,
          "network_latency_checks": true,
          "catalog_checks": true,
          "auth_type": "basic"
        }
      ]
    }
  }

Additional Integration Checks Performed

Use Case

This integration is primarily for basic Consul monitoring with focus on the service discovery.

Metrics Data Collected

The list of Consul's Prometheus metrics scraped and mapped by this method are listed in the latest metrics.py of the integration source code.

To understand how Consul Latency metrics are calculated, review the Consul Network Coordinates documentation.

Review the Datadog Documentation for the full description of Metrics data collected.

Openmetrics Prometheus

For Datadog agents at or above v6.5.0, OpenMetrics and Prometheus checks are available to scrape Kubernetes application Prometheus endpoints.

This method implements the collection via Openmetrics as that is fully supported for Prometheus text format and is accomplished using pod annotations as demonstrated below.

Helm Chart Configuration

global:
  tls:
    enabled: true
    enableAutoEncrypt: true
  acls:
    manageSystemACLs: true
  metrics:
    enabled: true
    enableAgentMetrics: true
    datadog:
      enabled: true
      namespace: "datadog"
      openMetricsPrometheus:
        enabled: true

ad.datadoghq.com/consul.checks: |
  {
    "openmetrics": {
      "init_config": {},
      "instances": [
        {
          "openmetrics_endpoint": "https://consul-server.consul.svc:8501/v1/agent/metrics?format=prometheus",
          "tls_cert": "/etc/datadog-agent/conf.d/consul.d/certs/tls.crt",
          "tls_private_key": "/etc/datadog-agent/conf.d/consul.d/certs/tls.key",
          "tls_ca_cert": "/etc/datadog-agent/conf.d/consul.d/ca/tls.crt",
          "headers": {
            "X-Consul-Token": "ENC[k8s_secret@consul/consul-datadog-agent-metrics-acl-token/token]"
          },
          "namespace": "consul",
          "metrics": [ ".*" ]
        }
      ]
    }
  }

Use Case

This method of integration is useful for Prometheus-enabled scrapes with further customization of the collected data.

By default, all metrics pulled using this method scrape Consul metrics using the /v1/agent/metrics?format=prometheus API query, and are considered to be custom metrics.

Use of this method maps to Datadog as described in Mapping Prometheus Metrics to Datadog Metrics. The following table summarizing how these metrics map to each other.

Metrics Data Collected

The integration, by default, uses a wildcard (".*") to collect all metrics emitted from the /v1/agent/metrics endpoint.

Please refer to the Agent Telemetry documentation for a full list and desription of the metrics data collected.