Vault with AWS KMS external key store (XKS) via PKCS#11 and XKS proxy
Appropriate Vault Enterprise license required
Note: AWS xks-proxy
is used in this document as a sample implementation.
Vault's KMIP Secrets Engine can be used as an external key store for the AWS KMS External Key Store (XKS) protocol using the AWS xks-proxy
along
with the Vault PKCS#11 Provider.
Overview
This is tested as working with Vault 1.11.0 Enterprise (and later) with Advanced Data Protection (KMIP support).
Prerequisites:
- A server capable of running XKS Proxy on port 443, which is exposed to the Internet or a VPC endpoint. This can be the same as the Vault server.
- A valid DNS entry with a valid TLS certificate for XKS Proxy.
libvault-pkcs11.so
downloaded from releases.hashicorp.com for your platform and available on the XKS Proxy server.- Vault Enterprise with the KMIP Secrets Engine available and with TCP port 5696 accessible to where XKS Proxy will be running.
There are 3 parts to this setup:
- Vault KMIP Secrets Engine standard setup. (There is nothing specific to XKS in this setup.)
- Vault PKCS#11 setup to tell the PKCS#11 provider (
libvault-pkcs11.so
) how to talk to the Vault KMIP Secrets Engine. (There is nothing specific to XKS in this setup.) - XKS Proxy setup.
Important: XKS has a strict 250 ms latency requirement. In order to serve requests with this latency, we recommend hosting Vault and the XKS proxy as close as possible to the desired KMS region.
Vault setup
On the Vault server, we need to setup the KMIP Secrets Engine:
Start the KMIP Secrets Engine and listener:
Create a KMIP scope to contain the AES keys that will be accessible. The KMIP scope is essentially an isolated namespace. Here is an example creating one called
my-service
(which will be used throughout this document).Create a KMIP role that has access to the scope:
Create TLS credentials (a certificate, key, and CA bundle) for the KMIP role:
Note: This command will output the credentials in plaintext.
The response from the
credential/generate
endpoint is JSON. The.data.certificate
entry contains a bundle of the TLS client key and certificate we will use to connect to KMIP with fromxks-proxy
. The.data.ca_chain[]
entries contain the CA bundle to verify the KMIP server's certificate. Save these to, e.g.,cert.pem
andca.pem
:
XKS proxy setup
The rest of the steps take place on the XKS Proxy server.
For this example, We will use an HTTPS proxy service like ngrok to forward connections to the XKS proxy. This helps to quickly setup a valid domain and TLS endpoint for testing.
Start
ngrok
:This will output a domain that can be used to configure KMS later, such as
https://example.ngrok.io
.Copy the
libvault-pkcs11.so
binary to the server, such as/usr/local/lib
(should be same as in the TOML config file below), andchmod
it so that it is executable.Copy the TLS certificate bundle (e.g.,
/etc/kmip/cert.pem
) and CA bundle (e.g.,/etc/kmip/ca.pem
) to thexks-proxy
server (doesn't matter where, as long as thexks-proxy
process has access to it) from the Vault setup.Create a
configuration/settings_vault.toml
file for the XKS to Vault PKCS#11 configuration, and set theXKS_PROXY_SETTINGS_TOML
environment variable to point to the file location.The important settings to change:
[[external_key_stores]]
:- change URI path prefix to anything you like
- choose random access ID
- choose random secret key
- set which key labels are accessible to XKS (
xks_key_id_set
)
[pkcs11]
: set thePKCS11_HSM_MODULE
to the location of thelibvault-pkcs11.so
(or.dylib
) file downloaded from releases.hashicorp.com.
Note: vault-pkcs11-provider
versions of 0.1.0–0.1.2 require the last two lines to be changed to can_generate_iv = true
and is_zero_iv_required = true
.
Create a file,
/etc/vault-pkcs11.hcl
with the following contents:This file is used by
libvault-pkcs11.so
to know how to find and communicate with the KMIP server. See the Vault docs for all available parameters and their usage.If you want to view the Vault logs (helpful when trying to find error messages), you can specify the
VAULT_LOG_FILE
(default is stdout) andVAULT_LOG_LEVEL
(default isINFO
). We'd recommend settingVAULT_LOG_FILE
to something like/tmp/vault.log
or/var/log/vault.log
. Other useful log levels areWARN
(quieter) andTRACE
(very verbose, could possibly contain sensitive information, like raw network packets).Create an AES-256 key in KMIP, for example, using
pkcs11-tool
(usually installed with the OpenSC package). See the Vault docs for the full setup.
Enable XKS in the AWS CLI
Create the KMS custom key store with the appropriate parameters to point to your XKS proxy (in this example, through
ngrok
).Tell KMS to connect to the key store.
Wait for the
ConnectionState
of your custom key store to beCONNECTED
. This can take a few minutes.Create a KMS key associated with the XKS key ID (
abc123
in this example):Encrypt some data with this key:
Decypt the resulting ciphertext:
Optionally, clean up your key and key store with:
(The
aws kms delete-custom-key-store
command will not succeed until all keys in the key store have been disabled and deleted.)