Observe Consul service mesh traffic
In this tutorial, you will configure and use Consul to observe traffic within your service mesh. This enables you to quickly understand how services interact with each other and effectively debug your services' traffic.
To enable Consul's observability features, you will:
- Enable additional Consul features using Helm or the Consul K8S CLI
- Deploy monitoring suite (Grafana, Loki, Prometheus)
- Generate traffic in the HashiCups UI
- Explore the Consul UI metrics visualization
- Explore the monitoring suite
Prerequisites
The tutorial assumes that you have completed the previous tutorials in this getting started collection.
For this tutorial, you will need:
Enable Consul telemetry features
You will now enable additional Consul features in your Kubernetes cluster using the official Consul Helm chart or the consul-k8s
CLI.
Review the additional configuration details. These configuration items enable metrics features for Consul agents, the Consul UI, and the Envoy proxy sidecars.
Update Consul in your Kubernetes cluster with Consul K8S CLI. Confirm the run by entering y
.
Refer to the Consul K8S CLI documentation to learn more about additional settings.
In addition to updating your Consul environment, you will also configure the Envoy proxy sidecars in your service mesh to expose a Prometheus metrics endpoint. This allows Prometheus to scrape metrics from the Envoy sidecar proxies.
The ProxyDefaults
configuration entry lets you configure global defaults across all services for Consul service mesh proxy configurations. The proxy/proxy-defaults.yaml
file configures envoy_prometheus_bind_addr
to expose a Prometheus metrics endpoint.
Modify the proxy defaults.
Review the Consul Proxy Defaults documentation to learn more.
Restart sidecar proxies
You need to restart your sidecar proxies to retrieve the updated proxy defaults configuration. To do so, redeploy your HashiCups services.
First, delete your existing HashiCups services.
Then, redeploy the HashiCups application.
Confirm that your proxy defaults updated your Envoy proxy’s configuration. You can view your Envoy sidecar proxy’s configuration by going to port 19000
from a service that has a sidecar proxy.
Open http://localhost:19000/config_dump in your browser to find the Envoy configuration. Search for 20200
, the value you defined in the ProxyDefaults
. You should find two different stanzas that reference this port. One of them is included next for reference.
This confirms that Consul has configured Envoy to publish Prometheus metrics.
Deploy observability suite
The monitoring suite you deploy in this tutorial uses Grafana for visualization, Prometheus for metrics, and Loki for logs.
Deploy the observability suite. This adds and installs the respective Helm charts for Grafana, Prometheus, and Loki.
Tip
The observability suite deployment will take about 1-2 minutes to complete.
Check the pods to make sure the monitoring suite is up and running successfully. Your complete output will include Grafana, Prometheus services, Loki services, and HashiCups microservices.
Generate traffic in the demo application
Since you deployed HashiCups, start generating some traffic so that you will have some metrics to look at in the Consul UI and Grafana.
Output the Consul API Gateway value to your terminal and paste it in your browser to access the HashiCups UI through the API Gateway. Click through some components within the HashiCups UI to generate traffic for visualization in the next steps.
Explore the Consul UI metrics visualization
Consul collects metrics and provides a built-in UI to help visualize topology between interdependent services. This can be helpful when configuring new services or troubleshooting existing connections.
Output the Consul URL value to your terminal and paste it in your browser.
Output the token value to your terminal and copy the value to your clipboard. You will use this ACL token to authenticate in the Consul UI.
On the left navigation pane, click the Services tab to review your deployed services. Click the nginx service to view the metrics details.
Explore the monitoring suite
Consul service mesh lets you expose metrics from the Envoy proxies and your services for third-party monitoring tools like Prometheus and Grafana to provide visualization for layer 7 application traffic.
Open a separate terminal window and expose the Grafana UI with kubectl port-forward
using the grafana
service name as the target.
Navigate to the custom HashiCups dashboard in Grafana at http://localhost:3000/d/hashicups/hashicups. Notice this dashboard includes both the Envoy proxy's metrics for each HashiCups services and the individual HashiCups service logs.
Consul integrates with a wide variety of providers to enhance your complete service mesh observability experience. Check out this page to learn more about Consul and observability.
Clean up
Destroy the Terraform resources to clean up your environment. Confirm the destroy operation by inputting yes
.
Note: Due to race conditions with the various cloud resources created in this tutorial, it may be necessary to run the destroy
operation twice to ensure all resources have been properly removed.
Next steps
In this tutorial, you enabled Consul monitoring features and deployed a monitoring suite into your Kubernetes cluster. After deploying the monitoring suite, you used the monitoring suite to explore the metrics and logs of the demo application, HashiCups, in your service mesh.
You have now completed the getting started collection!
Check out the following resources to learn more about running Consul on Kubernetes: