Kubernetes with HCP Vault Dedicated
Managing access to secrets is a critical process for organizations that run workloads on Kubernetes. HCP Vault Dedicated can manage secrets for Kubernetes pods from outside the Kubernetes cluster.
A Vault client (a user, an application, Kubernetes pod, etc.) must authenticate with Vault before it can read secrets managed by Vault.
In this tutorial, you will setup:
- Your local environment to support Vault Dedicated.
- Start a Kubernetes cluster using minikube.
- Authenticate with Vault Dedicated using:
- Vault token
- Kubernetes auth method
- Kubernetes auth method via the Vault Agent injector service
- Read secrets stored in Vault Dedicated.
Running Vault in Kubernetes
Running self hosted Vault in a Kubernetes cluster is explored in the Vault Installation to minikube via Helm and Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar tutorials.
Prerequisites
- HCP Vault Dedicated dev tier or higher cluster available
- Vault
- kubectl
- Helm
- minikube installed and configured
- ngrok installed and configured with an auth token
This tutorial was tested against the following specific versions:
- HCP Vault Dedicated 1.12.3
- Vault CLI 1.13
- Kubectl 1.26
- Helm 3.112
- minikube 1.29
- Kubernetes 1.26
Personas
The scenario described in this tutorial involves two personas:
admin
with privileged permissions to manage Vault and Kubernetesapps
reads the secrets from Vault (client)
Lab setup
Vault setup
Note
If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.
Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
Warning
Organizations running HCP Vault Dedicated should consider disabling public access to the Vault cluster. Pods running in Kubernetes can connect to the private Vault Dedicated address using connectivity options with a supported public cloud provider such as using an AWS transit gateway or an Azure VNet peering connection.
Open a new terminal and set the
VAULT_ADDR
environment variable to the copied address.Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the
VAULT_TOKEN
environment variable.Set the
VAULT_NAMESPACE
environment variable toadmin
.The
admin
namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.Note
For these tasks, you can use HCP Vault Dedicated's admin token. However, it is recommended that admin tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.
Enable the KV secret engine.
Create a secret at path
secret/exampleapp/config
with ausername
andpassword
.The Vault Dedicated server is ready.
minikube setup
minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes cluster locally inside Virtual Machines (VM) on your system.
Start a Kubernetes cluster.
The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.
Verify the status of the minikube cluster.
Create a Kubernetes service account named
exampleapp
with a service account token.Example output:
minikube is ready.
Token auth method authentication
(Persona: admin)
A pod can authenticate directly with a Vault Dedicated cluster using a token. In this example, the admin token is used for simplicity but any token can be supplied in the pod configuration.
Tokens can be created in Vault using a auth method and attaching a policy. Policies should follow the concept of least privilege to ensure the pod only has access to the secrets required.
Create a pod named
exampleapp-token
that sets theVAULT_ADDR
,VAULT_NAMESPACE
, andVAULT_TOKEN
environment variables.Note
This pod configuration includes examples of how to set environment variables that your application may require to interact with Vault. In this tutorial, you are interacting with the pod using the
kubectl
command-line interface so the environment variables in subsequent commands are also interpreted from your local shell session.Example output:
Get all the pods in the default namespace.
Wait until the
exampleapp-token
pod reports that is running and ready (1/1
).Update the containers apt repositories.
Install cURL in the container.
Note
cURL is used to simulate an application retrieving secrets from the HCP Vault Dedicated cluster.
Verify connectivity from the Ubuntu container to the Vault Dedicated cluster.
(Persona: apps)
Example output:
Read the
secret/exampleapp/config
secret created in the lab setup section.The API returns the secret defined at the path
secret/data/exampleapp/config
.Example output:
Stop the
exampleapp-token
pod.
Kubernetes auth method authentication
Rather than generating and assigning a token though the token auth method, pods can authenticate directly with Vault using a Kubernetes service account and an associated service account token. Vault uses the service account to validate the identity of the pod using the Kubernetes API.
In this scenario, Vault requires access to the Kubernetes API. In production environments, this would be done by connecting your HCP HVN to a transit gateway or peering connection with the network where Kubernetes is running.
For the purposes of this tutorial, you will configure ngrok to expose the Kubernetes API.
Kubernetes configuration
(Persona: admin)
Create a Kubernetes service account for Vault to authenticate with the Kubernetes, a service account token for the service account, and a role to allow Vault to access the appropriate API.
Example output:
Retrieve the
vault-auth
secret and store it as an environment variable.Review the secret.
Example output:
The secret includes the public key
ca.crt
andtoken
as base64 encoded strings.Decode the
ca.crt
certificate and store it as an environment variable.Decode the
token
and store it as an environment variable.You have collected the necessary information for the
vault-auth
service account to configure the Kubernetes auth method.
Configure networking
(Persona: admin)
In another terminal, start a proxy to expose the Kubernetes API.
Leave this tab open with the proxy running.
In another terminal, start ngrok and create a tunnel to the proxy listening on port
8001
.Warning
ngrok is used to expose the Kubernetes API to HCP Vault Dedicated. Using
--scheme=http
exposes the API without encryption to avoid TLS certificate errors.For production workloads, use a private peering or transit gateway connection with trusted certificates.
Example output:
Copy the ngrok forwarding address.
Return to the terminal where you set the
VAULT_ADDR
environment variable and set an environment variable for the ngrok forwarding address.
Configure Vault
(Persona: admin)
Enable the Kubernetes auth method.
Configure the Kubernetes auth method.
Example output:
Create a Vault policy that permits read access to
secret/exampleapp/config
Example output:
Create a role for the Kubernetes auth method and include the
exampleapp-read
Vault policy.Example output:
Authenticate and read secret
(Persona: apps)
Retrieve the
exampleapp
service account token and store it as an environment variable.Create a pod named
exampleapp-auth
that sets theEXAMPLEAPP_AUTH
,VAULT_ADDR
, andVAULT_NAMESPACE
environment variables.VAULT_TOKEN
is not required because the pod will authenticate with Vault using the Kubernetes service account defined in the auth method configuration to retrieve a token.Note
This pod configuration includes examples of how to set environment variables that your application may require to interact with Vault. In this tutorial, you are interacting with the pod using the
kubectl
command-line interface so the environment variables in subsequent commands are also interpreted from your local shell session.Example output:
Get all the pods in the default namespace.
Wait until the
exampleapp-auth
pod reports that is running and ready (1/1
).Update the containers apt repositories.
Install cURL in the container.
Note
cURL is used to simulate an application retrieving secrets from the HCP Vault Dedicated cluster.
Verify connectivity from the Ubuntu container to the Vault Dedicated cluster.
Example output:
Use the
exampleapp
Vault role to authenticate and store the generated Vault token as an environment variable.Read the
secret/exampleapp/config
secret created in the lab setup section using thePOD_TOKEN
.The API returns the secret defined at the path
secret/data/exampleapp/config
because the API call used the token obtained using the Kubernetes auth method with theexampleapp-read
policy attached.Example output:
Stop the
exampleapp-auth
pod.
Vault Agent injector service
While a pod can access Vault directly, there are times when network restrictions do not allow one or more pods to connect to external services such as Vault Dedicated.
When requirements limit egress network traffic, one option to consider is using the Vault Agent injector service. The injector service enables the authentication and secret retrieval on behalf of the applications. The Vault Agent pod, however, will require an exception to permit egress communications to Vault Dedicated.
In this section, you will install the Vault Helm chart to run the injector service, and configure the pod with the necessary annotations to work with the injector service.
Install Vault Agent
(Persona: admin)
Add the HashiCorp Helm repository.
Update all the repositories to ensure
helm
is aware of the latest versions.Install the Vault agent and configure with the Vault Dedicated public address.
Note
The Vault Helm chart also creates a service account named
vault
. In the previous section you created a dedicated service account for Vault so thevault
account will not be used in this tutorial.The
global.externalVaultAddr
is assigned the address of Vault Dedicated cluster.Get all the pods in the default namespace.
The Vault Agent Injector pod is deployed in the default namespace.
Wait until the
vault-agent-injector
pod reports that it is running and ready (1/1
).
Inject secrets into the pod
(Persona: apps)
The Vault Agent Injector only modifies a pod or deployment if it has a specific set of annotations. Refer to the Vault documentation for a complete list of available annotations.
Review the pod configuration.
These annotations define a partial structure of the deployment schema and are prefixed with
vault.hashicorp.com
.agent-inject
enables the Vault Agent Injector servicerole
specifies the Vault Kubernetes authentication roleagent-inject-secret-FILEPATH
prefixes the path of the file,credentials.txt
written to the/vault/secrets
directory. The value is the path to the secret defined in Vault.namespace
defines the Vault Dedicated namespace to be used. In this tutorial you are using the defaultadmin
namespace.
Create the
exampleapp-secret-agent
pod.Example output:
Get all the pods in the default namespace.
Wait until the
exampleapp-secret-agent
pod reports that it is running and ready (2/2
).Display the secrets written to the file
/vault/secrets/secret-credentials.txt
on theexampleapp-secret-agent
pod.The result displays the unformatted secret data present on the container.
Formatting data
A template can be applied to structure this data to meet the needs of the application.
The application in this pod still retrieves the secrets directly, but now that the injector service is deployed and capable of retrieving secrets for the application, future updates can remove that application logic.
Cleanup
Delete the
exampleapp-secret-agent
pod.Remove the Vault agent pod.
Stop minikube.
Delete minikube.
Warning
This will remove all existing minikube configurations.
Return to the terminal where the proxy is running and press ctrl-c to stop the proxy.
Return to the terminal where ngrok is running and press ctrl-c to stop ngrok.
Next steps
You deployed Vault Dedicated external to a Kubernetes cluster and deployed pods that leveraged it as a secrets store. You configured a pod to retrieve secrets using the token auth method, using the Kubernetes auth method, and finally through the Vault injector service.
Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, exploring how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.