Vault as the Secrets Backend - Systems Integration
Overview
Integrating Vault with Consul on Kubernetes includes a one-time setup on Vault and setting up the secrets backend for each Consul datacenter via Helm.
Complete the following steps once:
- Enabling Vault KV Secrets Engine - Version 2 to store arbitrary secrets
- Enabling Vault PKI Engine if you are choosing to store and manage either Consul Server TLS credentials or Service Mesh and Consul client TLS credentials
Repeat the following steps for each datacenter in the cluster:
- Installing the Vault Injector within the Consul datacenter installation
- Configuring a Kubernetes Auth Method in Vault to authenticate and authorize operations from the Consul datacenter
- Enable Vault as the Secrets Backend in the Consul datacenter
Please read Run Vault on Kubernetes if instructions on setting up a Vault cluster are needed.
Vault KV Secrets Engine - Version 2
The following secrets can be stored in Vault KV secrets engine, which is meant to handle arbitrary secrets:
- ACL Bootstrap token (
global.acls.bootstrapToken
) - ACL Partition token (
global.acls.partitionToken
) - ACL Replication token (
global.acls.replicationToken
) - Gossip encryption key (
global.gossipEncryption
) - Enterprise license (
global.enterpriseLicense
) - Snapshot Agent config (
client.snapshotAgent.configSecret
)
In order to store any of these secrets, we must enable the Vault KV secrets engine - Version 2.
Vault PKI Engine
The Vault PKI Engine must be enabled in order to leverage Vault for issuing Consul Server TLS certificates. More details for configuring the PKI Engine is found in Bootstrapping the PKI Engine under the Server TLS section.
Set Environment Variables
Before installing the Vault Injector and configuring the Vault Kubernetes Auth Method, some environment variables need to be set to better ensure consistent mapping between Vault and Consul on Kubernetes.
DATACENTER
We recommend using the value for
global.datacenter
in your Consul Helm values file for this variable.VAULT_AUTH_METHOD_NAME
We recommend using a concatenation of a
kubernetes-
prefix (to denote the auth method type) with theDATACENTER
environment variable for this variable.VAULT_SERVER_HOST
We recommend using the external IP address of your Vault cluster for this variable.
If Vault is installed in a Kubernetes cluster, get the external IP or DNS name of the Vault server load balancer.
On EKS, you can get the hostname of the Vault server's load balancer with the following command:If Vault is not running on Kubernetes, utilize the
api_addr
as defined in the Vault High Availability Parameters configuration:VAULT_AUTH_METHOD_NAME
We recommend connecting to port 8200 of the Vault server.
If your vault installation is current exposed using SSL, this address will need to use
https
instead ofhttp
. You will also need to setup theVAULT_CACERT
environment variable.VAULT_TOKEN We recommend using your allocated Vault token as the value for this variable. If running Vault in dev mode, this can be set to to
root
.
Install Vault Injector in Consul k8s cluster
A minimal valid installation of Vault Kubernetes must include the Agent Injector which is utilized for accessing secrets from Vault. Vault servers could be deployed external to Vault on Kubernetes with the injector.externalVaultAddr
value in the Vault Helm Configuration.
Issue the Helm install
command to install the Vault agent injector using the HashiCorp Vault Helm chart.
Configure the Kubernetes Auth Method in Vault
Ensure that the Vault Kubernetes Auth method is enabled.
After enabling the Kubernetes auth method, in Vault, ensure that you have configured the Kubernetes Auth method properly as described in Kubernetes Auth Method Configuration.
First, while targeting your Consul cluster, get the externally reachable address of the Consul Kubernetes cluster.
Next, you will configure the Vault Kubernetes Auth Method for the datacenter. You will need to provide it with:
token_reviewer_jwt
- this a JWT token from the Consul datacenter cluster that the Vault Kubernetes Auth Method will use to query the Consul datacenter Kubernetes API when services in the Consul datacenter request data from Vault.kubernetes_host
- this is the URL of the Consul datacenter's Kubernetes API that Vault will query to authenticate the service account of an incoming request from a Consul data center kubernetes service.kubernetes_ca_cert
- this is the CA certification that is currently being used by the Consul datacenter Kubernetes cluster.
Update Vault Helm chart
Finally, you will configure the Consul on Kubernetes helm chart for the datacenter to expect to receive the following values (if you have configured them) to be retrieved from Vault:
- ACL Bootstrap token (
global.acls.bootstrapToken
) - ACL Partition token (
global.acls.partitionToken
) - ACL Replication token (
global.acls.replicationToken
) - Enterprise license (
global.enterpriseLicense
) - Gossip encryption key (
global.gossipEncryption
) - Snapshot Agent config (
client.snapshotAgent.configSecret
) - TLS CA certificates (
global.tls.caCert
) - Server TLS certificates (
server.serverCert
)
Next Steps
As a next step, please proceed to Vault integration with Consul on Kubernetes' Data Integration.
Troubleshooting
The Vault integration with Consul on Kubernetes makes use of the Vault Agent Injectors. Kubernetes annotations are added to the deployments of the Consul components which cause the Vault Agent Injector to be added as an init-container that will then attach Vault secrets to Consul's pods at startup. Additionally the Vault Agent sidecar is added to the Consul component pods which is responsible for synchronizing and reissuing secrets at runtime. As a result of these additional sidecar containers the typical location for logging is expanded in the Consul components.
As a general rule the best way to troubleshoot startup issues for your Consul installation when using the Vault integration
is to establish if the vault-agent-init
container has completed or not via kubectl logs -f <your-consul-component> -c vault-agent-int
and checking to see if the secrets have completed rendering.
- If the secrets are not properly rendered the underlying problem will be logged in
vault-agent-init
init-container and generally is related to the Vault Kube Auth Role not having the correct policies for the specific secret e.g.global.secretsBackend.vault.consulServerRole
not having the correct policies for TLS. - If the secrets are rendered and the
vault-agent-init
container has completed AND the Consul component has not becomeReady
, this generally points to an issue with Consul being unable to utilize the Vault secret. This can occur if, for example, the Vault Role created for the PKI engine does not have the correctalt_names
or otherwise is not properly configured. The best logs for this circumstance are the Consul container logs:kubectl logs -f <your-consul-component> -c consul
.