Monitor your application health with distributed checks
In modern distributed systems, health checks play a significant role in the maintenance of complex deployments. Health checks provide a way to monitor and verify the status of your service fleet and can be used to automatically react to failures or excessive load. Consul uses distributed health checks to maintain an updated catalog of all services and nodes registered within the datacenter and to periodically verify services' behavior.
The Consul DNS service provides a standard interface to query the Consul catalog. Using Consul FQDN for service configuration, instead of IPs, you can simplify application configuration. One further simplification provided by Consul is the load-balancing feature provided across multiple instances of the same service. This is completely transparent to the downstream service instances and does not require extra configuration. Consul DNS service prevents unhealthy service instances to be resolved and simplifies the configuration process for your applications that do not have to change configuration in case of service failure.
Consul DNS service is based on the service definition files that the different Consul agents provide and uses a distributed health check system to ensure reliability of DNS results.
In this tutorial you will learn how to write a service definition to register your service in Consul and how to create health checks to make sure your service is available only when fully functional.
Specifically, you will:
- Review the ACL permissions required to register a Consul service
- Generate a token for a Consul service using the
service-identity
policy - Create a service configuration for the
hashicups-api
service - Reload Consul configuration to include the newly configured service
- Verify service registration
- Verify the load-balancing feature provided by Consul DNS
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you have a fully deployed Consul datacenter with an instance of the HashiCups application, composed by four services, NGINX, Frontend, API, and Database, deployed and registered in Consul catalog.
In the Consul datacenter, there is an extra node that you want to use to host a second instance of the API service for HashiCups, that you want to include in the datacenter.
By the end of this tutorial, you will have configured the API service to be part of Consul catalog.
This tutorial will focus specifically on service and health checks definitions to register a service in Consul catalog.
Once the service is registered in Consul, it will automatically appear in Consul DNS results and Consul will automatically load balance traffic across the different instances of the service.
All operations in this tutorial, after the scenario deployment, will be performed from an additional node, a bastion host, that will be deployed as part of the scenario. The tutorial provides instructions on how to connect to the bastion host at the end of the deployment.
Prerequisites
This tutorial assumes you are already familiar with Consul service discovery and its core functionalities. If you are new to Consul refer to refer to the Consul Getting Started tutorials collection.
If you want to follow along with this tutorial and you do not already have the required infrastructure in place, the following steps guide you through the process to deploy a demo application and a configured Consul datacenter on AWS automatically using Terraform.
To create a Consul deployment on AWS using terraform, you need the following:
Clone GitHub repository
Clone the GitHub repository containing the configuration files and resources.
Enter the directory that contains the configuration files for this tutorial.
Create infrastructure
With these Terraform configuration files, you are ready to deploy your infrastructure.
Issue the terraform init
command from your working directory to download the
necessary providers and initialize the backend.
Then, deploy the resources. Confirm the run by entering yes
.
Tip
The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization or learn more about the Raft protocol in a fun and interactive way.
After the deployment is complete, Terraform returns a list of outputs you can use to interact with the newly created environment.
The Terraform outputs provide useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:
- The
ip_bastion
provides IP address of the bastion host you use to run the rest of the commands in this tutorial. - The
remote_ops
lists the bastion host IP, which you can use access the bastion host. - The
retry_join
output lists Consul'sretry_join
configuration parameter. The next tutorial uses this information to generate Consul server and client configuration. - The
ui_consul
output lists the Consul UI address. The Consul UI is not currently running. You will use the Consul UI in a later tutorial to verify that Consul started correctly. - The
ui_grafana
output lists the Grafana UI address. You will use this address in a future tutorial. - The
ui_hashicups
output lists the HashiCups UI address. You can open this address in a web browser to verify the HashiCups demo application is running properly.
List AWS instances
The scenario deploys seven virtual machines.
After deployment, six virtual machines, consul_server[0]
, database[0]
, frontend[0]
, api[0]
, api[1]
, and nginx[0]
are configured in a Consul datacenter with service discovery. Node api[1]
does not have a service registered but is already a member of Consul datacenter.
The remaining node, bastion
is used to perform the tutorial steps.
Login into the bastion host VM
Login to the bastion host using ssh
.
Configure CLI to interact with Consul
Configure your bastion host to communicate with your Consul environment using the two dynamically generated environment variable files.
That will produce no output.
After loading the needed variables, verify you can connect to your Consul datacenter.
That will produce an output similar to the following.
Use consul catalog services
to query Consul catalog for the services present in the datacenter.
Every service for the HashiCups application is tagged with the inst_0
tag to indicate they are the first instance of the service.
Create ACL token for service registration
When ACLs are enabled, service registration requires a token with write permission over the service name you want to register. In this case service name is hashicups-api
.
Service identities let you to quickly construct policies for services, rather than manually creating identical polices for each service instance and for their companion sidecar proxy. Generate a new token for hashicups-api-1
using service-identity
.
That will produce an output similar to the following.
Set your newly generated token as the CONSUL_AGENT_TOKEN
environment variable. You will use this variable later in the tutorial to generate the service definition file.
Review service configuration for Consul services
Consul allows multiple ways to register a service in the service catalog:
- Creating a service definition file in the agent's configuration directory
- Using the
consul services register
CLI command - Calling the
/agent/service/register
HTTP API endpoint
Each of these methods will persist the registration in the Consul data folder.
To make sure your service registration survives data folder corruption, in this tutorial you will configure a service by placing a service definition file inside the agent's configuration directory.
Each service configuration is composed by two parts, service definitions and health checks definitions.
Service definition
The service definition requires the following parameters to be defined:
name
- the service name to register. Multiple instances of the same service share the same service name.id
- the service id. Multiple instances of the same service require a unambiguous id to be registered.tags
- [optional] - tags to assign to the service instance. Useful for blue-green deployment, canary deployment or to identify services inside Consul datacenter.port
- the port your service is exposing.token
- a token to be used during service registration. This is the token you created in the previous section.
Below an example configuration for the service definition.
Read more on service definitions at Services configuration reference
Checks definition.
Consul provides distributed monitoring for your services when health checks for the service are configured. Health checks configurations are nested in the service block. They can be defined using the following parameters:
id
- unique string value that specifies an ID for the check.name
- required string value that specifies the name of the check.service_id
- specifies the ID of a service instance to associate with a service check.interval
- specifies how frequently to run the check.timeout
- specifies how long unsuccessful requests take to end with a timeout.
The other parameter required to define the check is the type. Consul supports multiple check types, but for this tutorial you will use the TCP and HTTP check types.
A tcp check establishes connections to the specified IPs or hosts. If the check successfully establishes a connection, the service status is reported as success
.
If the IP or host does not accept the connection, the service status is reported as critical
.
An example of tcp check for the hashicups-api service, listening on port 8081
is the following:
HTTP checks send an HTTP request to the URL specified in the definition and report the service health based on the HTTP response code.
An example of tcp check for the hashicups-api service, which exposes an health
endpoint to test service status, is the following:
Create service configuration for HashiCups API service
The HashiCups API service is composed by three different services, public-api
, product-api
, and payments
, listening respectively on port 8081
, 9090
, and 8080
. The only service that requires external access is the public-api
on port 8081
.
Create service configuration file.
Tip
To simplify tests, all services already present in the Consul catalog have a tag assigned, inst_0
. The new instance uses a different tag, inst_1
.
Note
The service definition file can be adapted for Consul service mesh by adding a `connect` section in the file. Learn more on this on Securely connect your services with Consul service meshCopy the configuration file on the hashicups-api-1
node.
Login to hashicups-api-1
from the bastion host.
Verify the VM contains the files required for Consul configuration.
That will produce an output similar to the following.
Reload Consul to apply the service configuration.
Start hashicups-api
service.
The output of the command will show some Docker output while the Docker images that compose the services are being pulled on the node.
To continue with the tutorial, exit the ssh session to return to the bastion host.
Verify service registration
The service will now start appearing among Consul DNS results when a query is performed. Use the inst_1
tag in the results to recognize the new instance or in the query to filter the DNS results. Consul DNS service can be queried using all interfaces provided by Consul.
API interface and UI can also be used to retrieve information about the health checks configured for the service.
Use the consul catalog
command to query the Consul catalog and show all available services.
Notice that the hashicups-api
instance presents two different tags, inst_0
and inst_1
, that indicate that both instances are registered within Consul.
Verify Consul load balancing functionalities
When multiple instances of a service are defined, Consul DNS will automatically provide basic round-robin load balancing capabilities.
To test it, you will make 100 requests to the Consul DNS service asking to resolve the hashicups-api.service.dc1.consul
name and count the different results.
That will produce an output similar to the following.
Notice that the request was balanced across the two different instances of the hashicups-api
service.
Destroy the infrastructure
Now that the tutorial is complete, clean up the infrastructure you created.
From the ./self-managed/infrastruture/aws
folder of the repository, use terraform
to destroy the infrastructure.
Next steps
In this tutorial you learned how to register a service in a Consul datacenter by writing a service definition with associated health checks. You created a token with the necessary permissions to register the service, generated a definition for the hashicups-api
service including health checks for all the different components of the service, and registered the service in Consul catalog by reloading the Consul agent on the hashicups-api-1
node. You then used all Consul interfaces, CLI, API, DNS, and UI, to verify the service got registered correctly and learned how to identify the different instances of the service using the inst_1
tag. Finally, you tested Consul DNS load-balancing capabilities by performing multiple queries for the same service and verifying the number of request was roughly even across the two instances.
For more information about the topics covered in this tutorial, refer to the following resources:
To learn more about other features provided by Consul DNS service to: