Vault as Consul service mesh certification authority
You can use Vault's PKI Secrets Engine to generate and renew dynamic TLS X.509 certificates. Aside from being useful to secure Consul's RPC and Raft communication, the PKI secrets engine provides support serving as a Certificate Authority for Consul service mesh.
One of the benefits of using a service mesh is creating, generating, and rotating TLS certificates for services in the service mesh. Using Vault as the CA for your service mesh provides unified management for TLS certificates.
This tutorial configures a Consul datacenter to use Vault as the Consul service mesh CA for new and existing Consul datacenters.
Prerequisites
To test the commands provided by this tutorial you need a Consul datacenter configured with ACL, TLS encryption, and Gossip encryption enabled. You can use the following configuration file as an example to set up a single node Consul datacenter.
More details on how to securely configure your consul datacenter are available in the following tutorials:
- Vault: the tutorial assumes you already have a running Vault 0.10.3+ cluster in your network. You can use a local Vault dev agent or an existing Vault deployment.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Security Warning
This tutorial is not for production use. By default, the lab will install an insecure configuration of Consul.
Configure Vault ACL
To interact with the PKI Secrets Engine endpoints, you have to generate a Vault token, giving the appropriate permissions to Consul to generate the certificates.
Create Vault policies
The following Vault policies apply to Consul 1.12 and later. For use with earlier Consul versions, refer to the Vault CA provider documentation and select your version from the version dropdown.
If you want to manually create and tune the PKI secret engines to be used as CA, you can create them in Vault before enabling the integration.
In this example, the RootPKIPath
is connect_root
and the IntermediatePKIPath
is connect_dc1_inter
. Update these values to reflect your environment.
First, enable the PKI secrets engine for the root CA.
Next, enable the PKI secrets engine for the intermediate CA.
Finally, create a policy in Vault for the integration.
The following Vault policy allows Consul to use pre-existing PKI paths in Vault. Consul is granted read-only access to the PKI mount points and the root CA, but is granted full control of the intermediate or leaf CA for Consul service mesh clients.
Save the configuration in a file named vault-policy-connect-ca.hcl
and create the policy with the Vault CLI.
Create Vault token
Once the policy is created, create a Vault token to use for the integration.
Setup your environment
If you use a remote deployment, you can configure a local Consul binary to interact with the datacenter. Set the CONSUL_HTTP_ADDR
variable on your local machine or jump host to the IP address of a server.
The same applies to the ACL token to access Consul data. You can export the token with the CONSUL_HTTP_TOKEN
variable.
Configure Consul service mesh CA
Once Vault is configured and the necessary policies and tokens are created, configure Consul to use Vault PKI secrets engine as Consul service mesh CA.
The suggested approach for using Vault as Consul service mesh CA is to configure it directly when deploying a new Consul datacenter.
Configure Consul servers
Configure the service mesh CA using the connect
stanza in the Consul configuration file.
Note
The leaf_cert_ttl
defines the TTL for the certificates issued by the CA for Consul service mesh. The default value is 72h
but using the parameter you can tune the lease duration to certificates. The shortest TTL value is 1h
.
Start Consul servers
After the configuration you can start the Consul servers making sure to include the connect
configuration.
Verify Consul service mesh CA configuration
Verify the CA settings by querying Consul for the CA configuration using either the Consul CLI or the REST API.
Configure Consul clients
Once you configured Consul service mesh for the servers, you can configure the clients.
The configuration for the clients does not require to specify the provider but only to enable the service mesh feature.
Note
The client configuration is not tied to the CA provider in use by your Consul datacenter. This means that if you are changing the CA for an existing Consul datacenter, you will not need to apply any change to the already running clients.
Verify certificates
Once Vault is configured to act as CA for Consul service mesh you can verify the certificates being used in your service mesh.
Check root and intermediate certificates
Use the /agent/connect/ca/roots
endpoint to retrieve the list of currently trusted root certificates.
When a datacenter is initialized, this will only list one trusted root. As in the example above, multiple roots may appear as part of certificate rotation or if the CA got migrated from Consul to Vault.
Check leaf certificates for services
Use the /agent/connect/ca/leaf/<service>
endpoint to retrieve the leaf certificates for the Consul service mesh services.
For example, to retrieve the certificate for the service web
you can use the following.
Certificate rotation and API caching
The /agent/connect/ca/leaf/<service>
endpoint has a caching mechanism that might return stale results for service certificates. You can verify the result's age by inspecting the header data using the -verbose
option with the curl
command.
You can verify that the result is served by the cache using the x-cache
header that, in this example shows a value of HIT
.
In case you notice that the age
header is bigger than the value defined for leaf_cert_ttl
you can invalidate the cache by pointing at a different Consul node.
Change the CONSUL_HTTP_ADDR
variable on your local machine or jump host to the IP address of a different server.
Re-issue the curl
command to query the API endpoint. Leave the -verbose
parameter set so to verify the cache headers.
This time you can verify that the x-cache
value is MISS,
meaning that the result is not coming from the cache, and you can verify from the output that the certificates returned are different from the ones returned by the cached request.
Certificate rotation in logs
You can verify the rotation is successfully happening also from Consul logs.
Server agents
When intermediate certificates for the service mesh get rotated, you will get that signaled on the leader's log.
Client agents
Client agents will get log lines for the rotation of intermediate certificates and the rotation of the service leaf certificate. The log level for those events are DEBUG
, so you have to set up that verbosity level to troubleshoot the certificate rotation.
Intermediate certificate rotation:
Leaf certificate rotation:
Next Steps
In this tutorial, you configured Consul service mesh's certification Authority using the PKI secrets engine in Vault. You configured a root certificate for Consul service mesh backed by Vault, and generated leaf certificates for services deployed to Consul, signed by this root certificate.
Continue learning about the options of securing Consul by taking additional tutorials on Learn. The Secure Consul Gossip Communication with Vault tutorial teaches you to secure gossip protocol communication inside your Consul clusters. The Generate mTLS Certificates for Consul using Vault tutorial helps you implement mutual TLS for the overall Consul cluster.
Learn more about ACL token administration on Vault with the Generate Consul Tokens with HashiCorp Vault tutorial.