Connect external services to Consul with terminating gateways
Consul terminating gateways give your services the ability to communicate securely with applications and services outside of your service mesh. Common use cases include managed services (such as Amazon RDS), legacy services running on an unsupported OS, and hybrid applications transitioning towards the service mesh that require integration testing or are required to maintain compatibility with other legacy systems. This extends the ability to securely connect your applications on any runtime, any cloud, or on-premises environments with Consul.
Terminating gateways provide service mesh extendability to these scenarios that would otherwise require alternate security and policy enforcement mechanisms. They terminate service mesh mTLS connections, enforce intentions, and forward requests to the appropriate destination.
In this tutorial, you will deploy a Consul terminating gateway that provides secure communication with a managed AWS RDS instance. You will connect the front end of the HashiCups demo application to the HashiCups backend database on AWS RDS. In the process you will see how this feature provides you with simplified and secure communication to services external to your service mesh.
Note
Consul v1.19 introduces the Registration
CRD for Consul on Kubernetes to simplify the process to register external services. We recommend using the Registration
CRD to register external services directly with Consul instead of using Terminating Gateways. For more information, refer to Register services running on external nodes to Consul on Kubernetes.
Scenario overview
HashiCups is a coffee shop demo application. It has a microservices architecture and uses Consul service mesh to securely connect the services. At the beginning of this tutorial, you will use Terraform to deploy the HashiCups microservices, a self-managed Consul cluster on AWS EKS, and a managed AWS RDS instance that contains the HashiCups database.
In this tutorial, you will:
- Deploy the following resources with Terraform:
- Elastic Kubernetes Service (EKS) cluster
- A self-managed Consul datacenter on EKS
- A managed AWS RDS instance
- HashiCups demo application on EKS
- Perform the following Consul procedures:
- Explore the demo application (broken state)
- Review and enable the terminating gateway feature
- Register the AWS RDS instance as a Consul service
- Configure and link the AWS RDS service to the terminating gateway
- Update the service mesh applications to communicate with AWS RDS
- Explore the demo application (working state)
Prerequisites
The tutorial assumes that you are familiar with Consul and its core functionality. If you are new to Consul, refer to the Consul Getting Started tutorials collection.
For this tutorial, you will need:
- An AWS account configured for use with Terraform
- terraform >= 1.0
- consul >= 1.17.0
- consul-k8s >= 1.2.1
- helm >= 3.0
- git >= 2.0
- kubectl > 1.24
Clone GitHub repository
Clone the GitHub repository containing the configuration files and resources.
Change into the directory that contains the complete configuration files for this tutorial.
Review repository contents
This repository contains Terraform configuration to spin up the initial infrastructure and all files to deploy Consul, the demo application, and the observability suite resources.
The eks
directory contains the following Terraform configuration files:
aws-vpc.tf
defines the AWS VPC resourcesaws-lambda.tf
defines the AWS Lambda resourcesaws-rds.tf
defines the AWS RDS resourceseks-cluster.tf
defines Amazon EKS cluster deployment resourceseks-consul.tf
defines the self-managed Consul deploymenteks-hashicups-with-consul.tf
defines the HashiCups resourcesproviders.tf
defines AWS and Kubernetes provider definitions for Terraformvariables.tf
defines variables you can use to customize the tutorial
The directory also contains the following subdirectories:
api-gw
contains the Kubernetes configuration files for the Consul API gatewayhashicups
contains the Kubernetes configuration files for HashiCupsconfig
contains the custom Consul ACL configuration file, AWS Lambda database initialization function, and terminating gateway configuration fileshelm
contains the Helm charts for Consul
Deploy infrastructure and demo application
With these Terraform configuration files, you are ready to deploy your infrastructure. Initialize your Terraform configuration to download the necessary providers and modules.
Then, deploy the resources. Confirm the run by entering yes.
The Terraform deployment could take up to 15 minutes to complete.
Connect to your infrastructure
Now that you have deployed the Kubernetes cluster, configure kubectl
to interact with it.
Configure your CLI to interact with Consul datacenter
In this section, you will set environment variables in your terminal so your Consul CLI can interact with your Consul datacenter. The Consul CLI reads these environment variables for behavior defaults and will reference these values when you run consul
commands.
Set the Consul destination address.
Retrieve the ACL bootstrap token from the respective Kubernetes secret and set it as an environment variable.
Export the Consul CA certificate to use for TLS communication with Consul.
Set the server name to use as the SNI host for connecting to Consul via TLS.
Run the consul catalog services
CLI command to print all known services in your Consul catalog.
Explore the demo application (broken state)
In this section, you will visit your demo application to explore the HashiCups UI.
Retrieve the Consul API gateway public DNS address.
Open the Consul API gateway's URL in your browser and explore the HashiCups UI. Notice that HashiCups is in a broken state and unable to retrieve coffees from the backend product database. This behavior is expected since the HashiCups frontend services within the Consul service mesh cannot securely communicate with the external backend product database (on AWS RDS) by default.
Enable Consul terminating gateway
Consul terminating gateways are egress proxies that provide your service mesh applications connectivity to external destinations by terminating mTLS connections, enforcing Consul intentions, and forwarding requests to appropriate destination services.
In this section, you will review the parameters that enable this feature and update your Consul installation to apply the new configuration.
Review the Consul values file
Review the highlighted lines in the values file below to see the parameters that enable terminating gateways.
Refer to the Consul metrics for Kubernetes documentation and official Helm chart values to learn more about metrics configuration options and details.
Update Consul in your Kubernetes cluster with Consul K8S CLI to deploy a terminating gateway. Confirm the run by entering y
.
Refer to the Consul K8S CLI documentation to learn more about additional settings.
The Consul update could take up to 5 minutes to complete.
Review the official Helm chart values to learn more about these settings.
Verify the Consul terminating gateway successfully deployed in your environment.
Register the AWS RDS instance as a Consul service
To reach external services through the Consul terminating gateway, the external service must be registered in the Consul catalog.
In this section, you will create the Consul service configuration file for your AWS RDS instance and register it in the Consul catalog with the name managed-aws-rds
.
Retrieve the AWS RDS private DNS address and set it as an environment variable.
Create a custom Consul service configuration file for managed-aws-rds
with envsubst
. This will fill all placeholders with your unique AWS RDS private DNS address.
Review your unique Consul service configuration file.
Refer to the Consul services configuration documentation to learn more about Consul service configuration options and details.
Register managed-aws-rds
as a service in Consul.
Apply the Consul service defaults for the external managed-aws-rds
service. The configuration in service-defaults.yaml
creates a virtual service in Consul, which allows the services within your service mesh to communicate with the external service using Consul DNS.
Confirm that Consul can successfully resolve your external service.
Notice that managed-aws-rds.virtual.consul
resolves to a multicast IP address, which facilitates one-to-many communication for all instances of the managed-aws-rds
Consul service.
Configure and link the AWS RDS service to the terminating gateway
To begin routing traffic to external services, you must configure the Consul security systems and link your external service to the terminating gateway.
In this section, you will configure Consul ACLs and intentions to allow secure communication between the services within your mesh, the terminating gateway, and the external managed-aws-rds
service. You will then link your external managed-aws-rds
service to the terminating gateway.
First, create a custom ACL policy that allows the terminating gateway to communicate with the managed-aws-rds
service.
Example output:
Review the Consul ACL Policies documentation to learn more.
Retrieve the terminating gateway ACL role ID and set it as an environment variable.
Attach your custom ACL policy to the terminating gateway role.
Example output:
Create an intention that allows communication from the product-api
service to the managed-aws-rds
service. This intention allows traffic to flow from the HashiCups frontend services to the HashiCups backend database service.
Deploy the terminating gateway for the managed-aws-rds
by applying the terminating-gateway.yaml
CRD to your cluster.
Update your service mesh applications to communicate with external services
To configure your service mesh applications, the respective Consul virtual address must be configured into the relevant application connection parameter.
In this section, you will deploy the product-api
service mesh application to communicate with the external managed-aws-rds
virtual service.
Open your config/products-api
deployment configuration file and review the ConfigMap
section of the deployment.
Notice how the database connection string for this HashiCups service includes the managed-aws-rds.virtual.consul
Consul DNS address.
Note
The placement of the external service's virtual address is unique to the application. This scenario uses PostgreSQL as the HashiCups database.
Deploy the product-api
service.
Explore demo application (working state)
Open the HashiCup's URL in your browser and refresh the HashiCups UI.
Notice that the HashiCups UI functions correctly. You have successfully connected the HashiCups frontend to the external HashiCups backend database on AWS RDS using Consul terminating gateway.
Clean up resources
Destroy the Terraform resources to clean up your environment. Confirm the destroy operation by inputting yes
.
Note
Due to race conditions with the cloud resources in this tutorial, you may need to run the destroy
operation twice to remove all the resources.
Next steps
In this tutorial, you deployed and configured a Consul terminating gateway to extend secure communication to services outside of your Consul service mesh. This integration offers increased simplicity for network, security, and policy uniformity. It also provides reduced operational overhead and ease of adoption for legacy services.
For more information about the topics covered in this tutorial, refer to the following resources: