Data tokenization with transform secrets engine

40min
|
Enterprise
Vault

Deployment

Vault Enterprise feature

Transform secrets engine requires a Vault Enterprise Advanced Data Protection (ADP) license.

What is data tokenization?

Data tokenization process replaces sensitive data with unique values (tokens) that are unrelated to the original value in any algorithmic sense. Therefore, those tokens cannot risk exposing the plaintext satisfying the PCI-DSS guidance.

Vault's transform secrets engine has a data transformation method to tokenize sensitive data stored outside of Vault.

Tokenization versus FPE

When encrypting sensitive data, preservation of the original data format or length may be required to meet certain industry standards such as HIPAA or PCI. One method to fulfill this requirement is to use format preserving encryption (FPE).

However, there are organizations that care more about the irreversibility of the tokenized data and not so much about preserving the original data format. Therefore, the transform secrets engine's FPE transformation may not meet the governance, risk and compliance (GRC) strategy they are looking for due to the use of reversible cryptography to perform FPE.

Characteristics of the tokenization transformation:

Non-reversible identification: Protect data pursuant to requirements for data irreversibility (PCI-DSS, GDPR, etc.)
Integrated Metadata: Supports metadata for identifying data type and purpose
Extreme scale and performance: Support for performantly managing billions of tokens across clouds as well as on-premise

Prerequisites

To perform the tasks described in this tutorial, you need to have Vault Enterprise with the Advanced Data Protection module or HCP Vault Dedicated Plus tier cluster.

Access to a Vault Enterprise license with the ADP module to run Vault in dev mode or a HCP Vault Dedicated Plus tier cluster.
Vault binary installed
Docker installed
jq installed

Policy requirements

For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

If you are not familiar with policies, refer to the policies tutorial.

Lab setup

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

Type vault status to verify your connectivity to the Vault cluster.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
...snipped...

The Vault Dedicated server is ready.

Setup the transform secrets engine

Create a role named mobile-pay which is attached to a transformation named credit-card. The tokenized value will have a fixed maximum time-to-live (TTL) of 24 hours.

Components relationship diagram

Enable the transform secrets engine at transform/.

$ vault secrets enable transform
Success! Enabled the transform secrets engine at: transform/

Create a role named mobile-pay with a transformation named credit-card.

$ vault write transform/role/mobile-pay transformations=credit-card
Success! Data written to: transform/role/mobile-pay

Create a transformation named credit-card which sets the generated token's time-to-live (TTL) to 24 hours.
```
$ vault write transform/transformations/tokenization/credit-card \
  allowed_roles=mobile-pay \
  max_ttl=24h
```
Example output:
```
Success! Data written to: transform/transformations/tokenization/credit-card
```
The max_ttl is an optional parameter which allows you to control how long the token should stay valid.
You can set the allowed_roles parameter to a wildcard (*) to allow all roles or with globs at the end for pattern matching (e.g. mobile-*).

Display details about the credit-card transformation.

$ vault read transform/transformations/tokenization/credit-card

Key                 Value
---                 -----
allowed_roles       [mobile-pay]
deletion_allowed    false
mapping_mode        default
max_ttl             24h
stores              [builtin/internal]
templates           <nil>
type                tokenization

The type is set to tokenization.

Note

Unless called out with an Example output block, API calls do not produce output.

Enable transform secrets engine using /sys/mounts endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
     --request POST \
     --data '{"type":"transform"}' \
     $VAULT_ADDR/v1/sys/mounts/transform

Create a role named mobile-pay with a transformation named credit-card.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
     --request POST \
     --data '{"transformations": ["credit-card"]}' \
     $VAULT_ADDR/v1/transform/role/mobile-pay

Create a API request payload for the credit-card tokenization transformation.
```
$ tee payload.json <<EOF
{
  "max_ttl": "24h",
  "allowed_roles": [ "mobile-pay" ]
}
EOF
```
The max_ttl is an optional parameter which allows you to control how long the token should stay valid.
You can set the allowed_roles parameter to a wildcard (*) to allow all roles or with globs at the end for pattern matching (e.g. mobile-*).

Create a transformation named credit-card.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/tokenization/credit-card

List existing tokenization transformations to verify that credit-card was created successfully.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
     --request LIST \
    $VAULT_ADDR/v1/transform/transformations/tokenization \
    | jq ".data"

Example output:

{
  "keys": [
    "credit-card",
  ]
}

Read the credit-card transformation configuration.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
     $VAULT_ADDR/v1/transform/transformations/tokenization/credit-card \
     | jq ".data"

Example output:

{
  "allowed_roles": [
    "mobile-pay"
  ],
  "deletion_allowed": false,
  "mapping_mode": "default",
  "max_ttl": 86400,
  "stores": [
    "builtin/internal"
  ],
  "templates": null,
  "type": "tokenization"
}

The type is set to tokenization.

Tokenize secrets

The Vault client applications must have the following in their policy to perform tokenization transformation using the transform secrets engine enabled at transform/.

Required Client Policy

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/mobile-pay" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/mobile-pay" {
   capabilities = [ "update" ]
}

# To validate the token
path "transform/validate/mobile-pay" {
   capabilities = [ "update" ]
}

# To retrieve the metadata belong to the token
path "transform/metadata/mobile-pay" {
   capabilities = [ "update" ]
}

# To check and see if the secret is tokenized
path "transform/tokenized/mobile-pay" {
   capabilities = [ "update" ]
}

Encode data using the mobile-pay role and store it in a variable.

$ TOKEN_VALUE=$(vault write transform/encode/mobile-pay value=1111-2222-3333-4444 \
    transformation=credit-card \
    ttl=8h \
    metadata="Organization=HashiCorp" \
    metadata="Purpose=Travel" \
    metadata="Type=AMEX" -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $TOKEN_VALUE

The ttl value is an optional parameter. Remember that the max_ttl was set to 24 hours when you created the credit-card transformation. You can overwrite that value to make the token's TTL to be shorter.

In addition, you can set optional metadata about the data.

Example output:

encoded_value: Q4tYgFXHxUbU5XZoQsxusAzxhNyVWmCFWxAT8SGkHoYB3VkrmQGXVR

Retrieve the metadata of the token.

$ vault write transform/metadata/mobile-pay value=$TOKEN_VALUE transformation=credit-card

Key                Value
---                -----
expiration_time    2021-03-13 06:49:58.041608 +0000 UTC
metadata           map[Organization:HashiCorp Purpose:Travel Type:AMEX]

Notice the expiration_time value. Since you have overwritten the max_ttl, the ttl is set to 8 hours.

Validate the token value.

$ vault write transform/validate/mobile-pay value=$TOKEN_VALUE transformation=credit-card

Key      Value
---      -----
valid    true

Validate that the credit card number has been tokenized already.

$ vault write transform/tokenized/mobile-pay value=1111-2222-3333-4444 transformation=credit-card

Key          Value
---          -----
tokenized    true

Retrieve the original plaintext credit card value.

$ vault write transform/decode/mobile-pay transformation=credit-card value=$TOKEN_VALUE

Key              Value
---              -----
decoded_value    1111-2222-3333-4444

Create an HTTP request payload with a value and metadata.
```
$ tee payload.json <<EOF
{
  "value": "1111-2222-3333-4444",
  "transformation": "credit-card",
  "ttl": "8h",
  "metadata": {
       "Organization": "HashiCorp",
        "Purpose": "Travel",
        "Type": "AMEX"
  }
}
EOF
```
The ttl value is an optional parameter. Remember that the max_ttl was set to 24 hours when you created the credit-card transformation. You can overwrite that value to make the token's TTL to be shorter.
In addition, you can set optional metadata about the data.

Encode a value with the mobile-pay role and store it in a variable.

$ TOKEN_VALUE=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/mobile-pay | jq -r '.data | .encoded_value') \
    && echo encoded_value: $TOKEN_VALUE

Example output:

encoded_value: 9UrVvkyFti3hHaT1cWqs5Hch14hKL2B8bMtgRbqYjyepgCgK6Hy5zXtrHLzqJs93C41DRGnf

Retrieve the metadata of the token.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"value": "'"$TOKEN_VALUE"'", "transformation": "credit-card"}' \
    $VAULT_ADDR/v1/transform/metadata/mobile-pay | jq ".data"

Example output:

{
  "expiration_time": "2021-03-13 06:49:58.041608 +0000 UTC",
  "metadata": {
    "Organization": "HashiCorp",
    "Purpose": "Travel",
    "Type": "AMEX"
  }
}

Since you have overwritten the max_ttl, the ttl is set to 8 hours.

Validate the token value.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"value": "'"$TOKEN_VALUE"'", "transformation": "credit-card"}' \
    $VAULT_ADDR/v1/transform/validate/mobile-pay | jq ".data"

Example output:

{
  "valid": true
}

Validate that the credit card number has been tokenized already.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"value": "1111-2222-3333-4444", "transformation": "credit-card"}' \
    $VAULT_ADDR/v1/transform/tokenized/mobile-pay | jq ".data"

Example output:

{
  "tokenized": true
}

Retrieve the original plaintext credit card value.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"value": "'"$TOKEN_VALUE"'", "transformation": "credit-card"}' \
    $VAULT_ADDR/v1/transform/decode/mobile-pay | jq ".data"

Example output:

{
  "decoded_value": "1111-2222-3333-4444"
}

Convergent tokenization

Requirement

This feature requires Vault 1.11.0 or later.

If you encode the same value multiple times, it returns a different encoded value each time.

Example:

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 transformation=credit-card
Key              Value
---              -----
encoded_value    Q4tYgFXHxURKc5zaVDRhi3QT35htvADKtnBpBazpoBZPG373FP1mXs

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 transformation=credit-card
Key              Value
---              -----
encoded_value    Q4tYgFXHxUVpMnf4SSVQZWm1YhAG2eSGvS3dBeDNmm7fkPLnw3JV54

In some use cases, you may want to have the same encoded value for a given input so that you can query your database to count the number of entries for a given secret.

Key derivation is supported to allow the same key to be used for multiple purposes by deriving a new key based on a user-supplied context value. In this mode, convergent encryption can optionally be supported, which allows the same input values to produce the same ciphertext.

Update the mobile-pay role with a convergent transformation named credit-card-convergent.

$ vault write transform/role/mobile-pay transformations="credit-card, credit-card-convergent"
Success! Data written to: transform/role/mobile-pay

Create a transformation named credit-card-convergent which sets the enables the convergent encryption. When you define a transformation, set convergent=true.

$ vault write transform/transformations/tokenization/credit-card-convergent \
     allowed_roles="*" \
     convergent=true

Example output:

Success! Data written to: transform/transformations/tokenization/credit-card-convergent

Encode a value using the credit-card-convergent transformation.

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
     transformation=credit-card-convergent

Example output:

Key              Value
---              -----
encoded_value    DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x

Run the command again.

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
     transformation=credit-card-convergent

Example output:

It returns the same encrypted value.

Key              Value
---              -----
encoded_value    DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x

Create an API request payload.

$ tee payload-convergent.json <<EOF
{
   "convergent": true,
   "allowed_roles": [ "*" ]
}
EOF

NOTE: Set the convergent parameter to "true" to enable convergent encryption.

Create a transformation named credit-card-convergent.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload-convergent.json \
    $VAULT_ADDR/v1/transform/transformations/tokenization/credit-card-convergent

Add the credit-card-convergent transformation to the mobile-pay role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"transformations": ["credit-card","credit-card-convergent"]}' \
    $VAULT_ADDR/v1/transform/role/mobile-pay

Create an API request payload to encode a value using the credit-card-convergent transformation.

$ tee payload.json <<EOF
{
  "value": "5555-6666-7777-8888",
  "transformation": "credit-card-convergent"
}
EOF

Encode the value in the payload.json file using the mobile-pay role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/mobile-pay | jq ".data"

Example output:

{
  "encoded_value": "DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x"
}

Run the command again.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/mobile-pay | jq ".data"

Example output:

It returns the same encrypted value.

{
  "encoded_value": "DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x"
}

Lookup token

When the transformation is configured with convergent encryption, you can look up the tokenized value (token).

Encode the value using the credit-card-convergent transformation with time-to-live (TTL) of 8 hours.

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
     transformation=credit-card-convergent ttl=8h

Example output:

Key              Value
---              -----
encoded_value    MQEgMY8jwXYJNb9p177dYCttwRUcctAeo89yVmMPiLotzfe3sx2btpBGNLijo2KA9cy4hAnTfeV78D5rJwyBdC5j6tMEZgH

The encoded value (token) is longer than the one without a TTL.

Look up the token for card number "5555-6666-7777-8888".

$ vault write transform/tokens/mobile-pay value=5555-6666-7777-8888 \
    transformation=credit-card-convergent

Example output:

Key       Value
---       -----
tokens    [DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x]

Look up with expiration of "any".

$ vault write transform/tokens/mobile-pay value=5555-6666-7777-8888 \
    transformation=credit-card-convergent expiration="any"

Example output:

Key       Value
---       -----
tokens    [DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x MQEgMY8jwXYJNb9p177dYCttwRUcctAeo89yVmMPiLotzfe3sx2btpBGNLijo2KA9cy4hAnTfeV78D5rJwyBdC5j6tMEZgH]

This returns two space-separated tokens. In absence of the expiration parameter, the command returns token with no expiration. When the expiration is set to "any", it returns tokens with any expiration.

Look up tokens that have an expiration between a given range using min_expiration and min_expiration which are RFC3339 formatted time and date.

$ vault write transform/tokens/mobile-pay value=5555-6666-7777-8888 \
    transformation=credit-card-convergent \
    min_expiration=$(echo $(date -v-60M "+%Y-%m-%dT%H:%M:%S+00:00")) \
    max_expiration=$(echo $(date -v+10d "+%Y-%m-%dT%H:%M:%S+00:00"))

Example output:

Key       Value
---       -----
tokens    [MQEgMY8jwXYJNb9p177dYCttwRUcctAeo89yVmMPiLotzfe3sx2btpBGNLijo2KA9cy4hAnTfeV78D5rJwyBdC5j6tMEZgH]

Any token that expires within the provided date range is displayed.

Create an HTTP request payload.

$ tee payload-input-3.json <<EOF
{
  "value": "5555-6666-7777-8888",
  "transformation": "credit-card-convergent",
  "ttl": "8h"
}
EOF

Encode the value using the credit-card-convergent transformation with time-to-live (TTL) of 8 hours using the mobile-pay role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload-input-3.json \
    $VAULT_ADDR/v1/transform/encode/mobile-pay | jq ".data"

Example output:

{
  "encoded_value": "MQEgMY8jwXMzbK94WCG1SsWvrAP6JBUNBLyLM45Kb4h42La4zHj15N6ZG3Q3aaQqPvDhf6wXCm6B3L4L4zx19yaPiskLbNm"
}

Notice that the encoded value (token) is longer than the one without TTL.

Look up the token for a card number, "5555-6666-7777-8888".

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"value": "5555-6666-7777-8888", "transformation": "credit-card-convergent"}' \
    $VAULT_ADDR/v1/transform/tokens/mobile-pay | jq ".data"

Example output:

{
  "tokens": [
    "DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x"
  ]
}

Look up with expiration of "any".

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"expiration": "any", "value": "5555-6666-7777-8888", "transformation": "credit-card-convergent"}' \
    $VAULT_ADDR/v1/transform/tokens/mobile-pay | jq ".data"

Example output:

{
  "tokens": [
    "DaCJhefr1oRUoY1vCtgJ8HixeazFhWHzFsNoNGeDWnyGGe76Xp8Er4UyW76xiPyQ2Eh2jGztnb3x",
    "MQEgMY8jwXMzbK94WCG1SsWvrAP6JBUNBLyLM45Kb4h42La4zHj15N6ZG3Q3aaQqPvDhf6wXCm6B3L4L4zx19yaPiskLbNm"
  ]
}

Create an API request payload to look up tokens that are expire between a given range using min_expiration and min_expiration which are RFC3339 formatted time and date.

$ tee payload-range.json <<EOF
{
  "value": "5555-6666-7777-8888",
  "transformation": "credit-card-convergent",
  "min_expiration": "$(echo $(date -v-60M "+%Y-%m-%dT%H:%M:%S+00:00"))",
  "max_expiration": "$(echo $(date -v+10d "+%Y-%m-%dT%H:%M:%S+00:00"))"
}
EOF

Look up the tokens that expire within the given range.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload-range.json \
    $VAULT_ADDR/v1/transform/tokens/mobile-pay | jq ".data"

Example output:

The request returns tokens that expire within the given expiration range.

{
  "tokens": [
    "MQEgMY8jwXJKzBtYixUTKERLehxpHX8YohswL8RYgiHozcJxfvhSd38BjHikRh4hFNHPs1aq3DxJGSGa7MKmcwhRvY8z6fK"
  ]
}

Key rotation

Note

The automatic key rotation requires Vault Enterprise 1.12.0 or later.

Rotating keys regularly limits the amount of information produced by a key if that key ever becomes compromised. In this section, you are going to enable automatic key rotation for your tokenization keys.

Read the key information for credit-card transformation.

$ vault read transform/tokenization/keys/credit-card

Key                       Value
---                       -----
auto_rotate_period        0s
latest_version            1
min_available_version     0
min_decryption_version    1
name                      credit-card
type                      aes256-gcm96

Notice that the latest_version is 1.

Rotate the key for credit-card transformation.

$ vault write -force transform/tokenization/keys/credit-card/rotate
Success! Data written to: transform/tokenization/keys/credit-card/rotate

Read the key information again.

$ vault read transform/tokenization/keys/credit-card

Key                       Value
---                       -----
auto_rotate_period        0s
latest_version            2
min_available_version     0
min_decryption_version    1
name                      credit-card
type                      aes256-gcm96

The latest_version is now 2.

Configure the key to be automatically rotated every 90 days to reduce operational overhead.
Note
The minimum permitted value for the auto_rotate_period is 1 hour.
```
$ vault write transform/tokenization/keys/credit-card/config \
   auto_rotate_period=90d
```
Example output:
```
Success! Data written to: transform/tokenization/keys/credit-card/config
```

Verify the configuration.

$ vault read transform/tokenization/keys/credit-card

Example output:

Key                       Value
---                       -----
auto_rotate_period        2160h
latest_version            1
min_available_version     0
min_decryption_version    1
name                      credit-card
type                      aes256-gcm96

If the key becomes compromised, you can rotate the key using the transform/tokenization/keys/<transformation_name>/rotate, and then set the min_decryption_version to the latest key version so that the older (possibly compromised) key will not be able to decrypt the data.

Test the auto-rotation

Because the minimum rotation period you can set is 1 hour, you will need to come back later to see that the key is rotated.

Set the rotation period to 1 hour.

$ vault write transform/tokenization/keys/credit-card/config \
   auto_rotate_period=1h

Encrypt another value for testing.

$ export TOKEN_VALUE_2=$(vault write -format=json transform/encode/mobile-pay \
   value=1234-5678-9012-3456 transformation=credit-card ttl=8h \
   | jq -r ".data.encoded_value")

The value, 1234-5678-9012-3456 is encoded with credit-card key version of 2, and the returned encoded value is stored in the TOKEN_VALUE_2 environment variable.

You can make sure that the environment variable holds the encoded value.

$ echo $TOKEN_VALUE_2
eRwUjS2L9e7fjTv7gFa3Kaf1LakuWc387XdECkbWBBLu6AiMCMWyG8mr1cPBkrSj6keC4DV69

Wait for at least 1 hour to see the key has been rotated.

$ vault read transform/tokenization/keys/credit-card

Key                       Value
---                       -----
auto_rotate_period        2160h
latest_version            3
min_available_version     0
min_decryption_version    1
name                      credit-card
type                      aes256-gcm96

You can test that the data encoded by version 1 of the key can be still decoded because min_decryption_version is 1.
```
$ vault write transform/decode/mobile-pay transformation=credit-card value=$TOKEN_VALUE
```
Also, you should be able to decode TOKEN_VALUE_2.
```
$ vault write transform/decode/mobile-pay transformation=credit-card value=$TOKEN_VALUE_2
```
If you change the min_decryption_version to 2, you will be able to decode TOKEN_VALUE_2 but not TOKEN_VALUE.

Read the key information for the credit-card transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card | jq ".data"

Example output:

{
   "auto_rotate_period": 0,
   "latest_version": 1,
   "min_available_version": 0,
   "min_decryption_version": 1,
   "name": "credit-card",
   "type": "aes256-gcm96"
}

The latest_version is 1.

Rotate the key for the credit-card transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card/rotate

Read the key information again.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card | jq ".data"

Example output:

{
   "auto_rotate_period": 0,
   "latest_version": 2,
   "min_available_version": 0,
   "min_decryption_version": 1,
   "name": "credit-card",
   "type": "aes256-gcm96"
}

The latest_version is now 2.

Configure the key to be automatically rotated every 90 days to reduce operational overhead.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{ "auto_rotate_period": "90d" }' \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card/config

Note

The minimum permitted value for the auto_rotate_period is 1 hour.

Verify the configuration.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card | jq ".data"

Example output:

{
   "auto_rotate_period": 7776000,
   "latest_version": 2,
   "min_available_version": 0,
   "min_decryption_version": 1,
   "name": "credit-card",
   "type": "aes256-gcm96"
}

The auto_rotate_period shows the time in seconds before the key is auto rotated.

If the key gets compromised, you can rotate the key using the transform/tokenization/keys/<transformation_name>/rotate, and then set the min_decryption_version to the latest key version so that the older (possibly compromised) key will not be able to decrypt the data.

Test the auto-rotation

You will now set the minimum rotation period to 1 hour. To complete this section, you will need to come back later to see that the key is rotated.

Set the rotation period to 1 hour.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{ "auto_rotate_period": "1h" }' \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card/config

Create an API request payload.

$ tee payload.json <<EOF
{
  "value": "1234-5678-9012-3456",
  "transformation": "credit-card",
  "ttl": "8h"
}
EOF

Encode the value with the mobile-pay role.

$ TOKEN_VALUE_2=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/mobile-pay | jq -r '.data | .encoded_value') \
    && echo encoded_value: $TOKEN_VALUE_2

Example output:

encoded_value: eRwUjS2L9dis8xUQwgDpvGYdPb5ht1poFyFCeeYL9oN2afLGDbwhmbzvc4Lc6QJnQ2ZmJ28ZA

The value, 1234-5678-9012-3456 is encoded with credit-card key version of 2 and stored in the TOKEN_VALUE_2 environment variable.

Wait for at least 1 hour to see the key has been rotated.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    $VAULT_ADDR/v1/transform/tokenization/keys/credit-card | jq ".data"

Example output:

{
   "auto_rotate_period": 3600,
   "latest_version": 3,
   "min_available_version": 0,
   "min_decryption_version": 1,
   "name": "credit-card",
   "type": "aes256-gcm96"
}

To test that the data encoded by version 1 of the key can be still decoded because min_decryption_version is 1, create a request payload for TOKEN_VALUE.
```
$ tee payload-my-token.json <<EOF
{
   "value": "$TOKEN_VALUE",
   "transformation": "credit-card"
}
EOF
```

Decode the value.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload-my-token.json \
    $VAULT_ADDR/v1/transform/decode/mobile-pay | jq ".data"

You should also be able to decode TOKEN_VALUE_2. Create an API request payload for TOKEN_VALUE_2.

$ tee payload-my-token-2.json <<EOF
{
   "value": "$TOKEN_VALUE_2",
   "transformation": "credit-card"
}
EOF

Decode the value.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload-my-token-2.json \
    $VAULT_ADDR/v1/transform/decode/mobile-pay | jq ".data"

If you change the min_decryption_version to 2, you will be able to decode TOKEN_VALUE_2 but not TOKEN_VALUE.

Setup external token storage

Vault Dedicated does not currently support external key storage.

Bring your own key (BYOK)

Note

This section is provided to help you understand the process generate and use a data key without sending data to Vault. A complete end-to-end scenario cannot be replicated in the tutorial.

When your use case requires an external key, users of Vault version 1.12.0 or greater can use BYOK functionality to import an existing encryption key that was generated outside Vault.

The target key for import can originate from an HSM or other external source, and must be prepared according to its origin before you can import it.

Note

Tokenization transformations with imported keys do not currently support convergent tokenization.

The example shown here will use a 256-bit AES key, referred to as the target key. To successfully import the target key, you must perform the following operations to prepare it.

Generate an ephemeral 256-bit AES key.
Wrap the target key using the ephemeral AES key with AES-KWP.
Wrap the AES key under the Vault wrapping key using RSAES-OAEP with MGF1 and either SHA-1, SHA-224, SHA-256, SHA-384, or SHA-512.
Delete the ephemeral AES key.
Append the wrapped target key to the wrapped AES key.
Base64 encode the result.

A specific code example for preparing and wrapping the key for import is beyond the scope of this tutorial. For more details about wrapping the key for import including instructions for wrapping key from an HSM, refer to the key wrapping guide.

Before you can wrap the key for import, you must read the wrapping key from Vault so that it can be used to prepare your key.

$ vault read -field=public_key transform/wrapping_key
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAttRvwFiW9kPtTutUeC3s
Q3PXXrCOuHrUlAqJxH/nNj7HpuERSJg+HW70b16YCU7mo+lVhpD+bMW3WVzov3v3
jXu6LSZYPZjlO8cEUtQQfydpWVC98cxjn8WCKrIeI+J211/xQQIsOT8sF9MYe11G
Pz/VGdEj+BKel7HWWD1nJ4i0Uo10iN76r0E5GioN8E5wK8L/11k/puEB3EcWhyPp
6aQ8PUbFqU8cmChd2Jg+Cj0k81sp72vOPkDJ7oAekYeR/iIYUjUINlz2Yob/qt25
AuF1t+TSLcI+RHzTetpL6/oHnbKdyM95MNq4XlELun4x2xOa4Q2EVkczwnb+X1kO
0d3FL2dAmMVxCXQnMdd/e2rJx8HxQZX02R4IfojmvfD5x/yICYrqm6bmv9RzTCxE
myQMVUYZsEfXgCKffW0vmEZOFIKvLkI5iaaHqMPk3SCGpMnniybEFt7dmXAnswpE
mtxf+5/gm+aTBOUmEWy0WYb+hHK0RXnUC9ScRWBd+eOMONEm+N7nUUS4e57+EnjH
OrjshTEU/B2BO8DvfYyuYyFzBiWzCJywj2eOaQxYdg8Y05YgDrE1GVEOceErDxT8
eoP/MQgUvVOAsD1KIAT5WmmNOCZ0HHAHjOKDdZ1FWtasNBjGJI7g5a1oCHuwl1p/
CZ2i8Rd7XOEnWZ+ld7l+qS0CAwEAAQ==
-----END PUBLIC KEY-----

The output is the (4096-bit RSA) wrapping key.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request GET --silent \
    $VAULT_ADDR/v1/transform/wrapping_key | jq

Example output:

{
  "request_id": "aa998396-098f-391b-213e-572fd0882608",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "public_key": "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAttRvwFiW9kPtTutUeC3s\nQ3PXXrCOuHrUlAqJxH/nNj7HpuERSJg+HW70b16YCU7mo+lVhpD+bMW3WVzov3v3\njXu6LSZYPZjlO8cEUtQQfydpWVC98cxjn8WCKrIeI+J211/xQQIsOT8sF9MYe11G\nPz/VGdEj+BKel7HWWD1nJ4i0Uo10iN76r0E5GioN8E5wK8L/11k/puEB3EcWhyPp\n6aQ8PUbFqU8cmChd2Jg+Cj0k81sp72vOPkDJ7oAekYeR/iIYUjUINlz2Yob/qt25\nAuF1t+TSLcI+RHzTetpL6/oHnbKdyM95MNq4XlELun4x2xOa4Q2EVkczwnb+X1kO\n0d3FL2dAmMVxCXQnMdd/e2rJx8HxQZX02R4IfojmvfD5x/yICYrqm6bmv9RzTCxE\nmyQMVUYZsEfXgCKffW0vmEZOFIKvLkI5iaaHqMPk3SCGpMnniybEFt7dmXAnswpE\nmtxf+5/gm+aTBOUmEWy0WYb+hHK0RXnUC9ScRWBd+eOMONEm+N7nUUS4e57+EnjH\nOrjshTEU/B2BO8DvfYyuYyFzBiWzCJywj2eOaQxYdg8Y05YgDrE1GVEOceErDxT8\neoP/MQgUvVOAsD1KIAT5WmmNOCZ0HHAHjOKDdZ1FWtasNBjGJI7g5a1oCHuwl1p/\nCZ2i8Rd7XOEnWZ+ld7l+qS0CAwEAAQ==\n-----END PUBLIC KEY-----\n"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

The output is a JSON object with a data.public_key field containing the (4096-bit RSA) wrapping key.

Use the wrapping key value at step 3 in the previously detailed preparation steps. Once you have prepared and base64 encoded the ciphertext, export the value to the environment variable IMPORT_CIPHERTEXT.

Example:

$ export IMPORT_CIPHERTEXT=ceXhQrVMuf70i2qL3DvQu/0AFhkPXAV6JyzbPdTs9A/Twjd8PGs/2XV3VhBgvhb4Fr1xWnVmIUKwxgP+emBlIqwpmoJsnkVNJSMpXP0YG+MkvheB9ATlfGXTlf6RLt7OaOtSSBxeVZQBtuWuVnbatTQiXhhC91J49V4+n1JiDjs0tRpz8hUxuzyedkjXWv8Mn0gD4nHV1GgrxLvNGPrrk2Y3xcZO4MoO3Lp447BjPTXwSwmR2rSOeW9+MsuYPfjx2dKC/uJRr6GM2wyyzaRu+5N+goNDaPzTAcNhutb/isf9wAOC3v8SGYXkF9919/ioxZKejWDscFLKBU/6O3+n7Zgf+NLJP/5qyjFOR6F/OnyXpEaOcD+5zpW0yppo/SqThS14E3jDIDmvwxiDBY0SHMBebLYx/t6Y29P+kAP3dPBXeHejHTZ0kJlQuKNcu2Ge1EkTFvR4vScvFlG4OcZIDzxwVxbS6IR83m/xeFyQhuMc/PntDvIJBnUb+AREtZCtNOcwh2Qei8iHfSttCNzROLRZ4ETaBUEKBBWKJztscDfKFOAGKiJIZeze+nVYK7sf94PMJF1r3ET75IyvwCXUEQoPtacg8nyUx6UVpp4BP6xzeOcEPKr1GPEj1kjcj43+rhozzzrs5AgTknQAcorpUwHFUezQtKWyEh/wastp6elX7V0IBL8YmKmWmbIz2hS+uH9TspQM9BnJL/ex18qgmpwIhfR40hzo

Create a new transformation role named legacy-system to use for the transformation that you will import the key into.

$ vault write transform/role/legacy-system transformations=application-form
Success! Data written to: transform/role/legacy-system

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"transformations": ["application-form"]}' \
    $VAULT_ADDR/v1/transform/role/legacy-system

Note

This command returns no output.

Import the key into the application-form transformation. Imported keys do not support rotation by default, so include the allow_rotation parameter and set its value to true so that you can also try rotating the imported key. Add the allowed_roles parameter and specify the legacy-system role.

$ vault write transform/transformations/tokenization/application-form/import \
    ciphertext=$IMPORT_CIPHERTEXT allowed_roles=legacy-system allow_rotation=true
Success! Data written to: transform/transformations/tokenization/application-form/import

Create a JSON payload containing ciphertext value and any additional parameters.

$ cat > payload.json << EOF
{
  "ciphertext": "ceXhQrVMuf70i2qL3DvQu/0AFhkPXAV6JyzbPdTs9A/Twjd8PGs/2XV3VhBgvhb4Fr1xWnVmIUKwxgP+emBlIqwpmoJsnkVNJSMpXP0YG+MkvheB9ATlfGXTlf6RLt7OaOtSSBxeVZQBtuWuVnbatTQiXhhC91J49V4+n1JiDjs0tRpz8hUxuzyedkjXWv8Mn0gD4nHV1GgrxLvNGPrrk2Y3xcZO4MoO3Lp447BjPTXwSwmR2rSOeW9+MsuYPfjx2dKC/uJRr6GM2wyyzaRu+5N+goNDaPzTAcNhutb/isf9wAOC3v8SGYXkF9919/ioxZKejWDscFLKBU/6O3+n7Zgf+NLJP/5qyjFOR6F/OnyXpEaOcD+5zpW0yppo/SqThS14E3jDIDmvwxiDBY0SHMBebLYx/t6Y29P+kAP3dPBXeHejHTZ0kJlQuKNcu2Ge1EkTFvR4vScvFlG4OcZIDzxwVxbS6IR83m/xeFyQhuMc/PntDvIJBnUb+AREtZCtNOcwh2Qei8iHfSttCNzROLRZ4ETaBUEKBBWKJztscDfKFOAGKiJIZeze+nVYK7sf94PMJF1r3ET75IyvwCXUEQoPtacg8nyUx6UVpp4BP6xzeOcEPKr1GPEj1kjcj43+rhozzzrs5AgTknQAcorpUwHFUezQtKWyEh/wastp6elX7V0IBL8YmKmWmbIz2hS+uH9TspQM9BnJL/ex18qgmpwIhfR40hzo", "allowed_roles": "legacy-system", "allow_rotation": true
}
EOF

Import the key.

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/transformations/tokenization/application-form/import

Note

This command returns no output.

Try using the newly imported key to encode a value and some metadata.

$ vault write transform/encode/legacy-system value=887345 \
     transformation=application-form \
     ttl=8h \
     metadata="Organization=ColoTown" \
     metadata="Purpose=Server-Cage" \
     metadata="Type=PIN"
Key              Value
---              -----
encoded_value    eRwUjS2L9e5mNKLD9Rqt6c49QfsWiqbCEWjtaM334NStmZRG4K7parh6UXJqhYpMVLT9Fe81q

Create a JSON payload.

$ cat > payload.json << EOF
{
   "value": "887345",
   "metadata": "organization=ColoTown,purpose=Server-Cage,type=PIN",
   "transformation": "application-form"
}
EOF

Encode the values.

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/legacy-system | jq

Example output:

{
  "request_id": "08754cc7-90a3-e68a-2824-0a35db768ccd",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "encoded_value": "Q4tYgFXHxUXeL61wuEeFMMAeho1gXj8yw5VtNKNUY4NRdBCkvY9FRB"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Note

The response payload may return a tweak along with the encoded value if the tweak_source for the specified transformation is set to generated. The resource owner should properly store this tweak, which must be supplied back when decrypting the encoded value.

The imported key is working, and the encoded value returned by the application-form transformation is using the imported key.

Note

To import subsequent versions of the key, you must use the import_version API endpoint.

Review the key information.

$ vault read transform/tokenization/keys/application-form
Key                       Value
---                       -----
auto_rotate_period        0s
latest_version            1
min_available_version     0
min_decryption_version    1
name                      application-form
type                      aes256-gcm96

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request GET \
    $VAULT_ADDR/v1/transform/tokenization/keys/application-form | jq

Example output:

{
  "request_id": "93d63de8-c83d-3190-3e6a-ec33ecdf8478",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "auto_rotate_period": 0,
    "latest_version": 1,
    "min_available_version": 0,
    "min_decryption_version": 1,
    "name": "application-form",
    "type": "aes256-gcm96"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

The key's latest_version is currently 1..

Rotate the key.

Note

Once an imported key is rotated within Vault, it will no longer support importing key material with the import_version endpoint.

$ vault write -force transform/tokenization/keys/application-form/rotate
Success! Data written to: transform/tokenization/keys/application-form/rotate

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    $VAULT_ADDR/v1/transform/tokenization/keys/application-form/rotate

Note

This command returns no output.

Check the key information once more.

$ vault read transform/tokenization/keys/application-form
Key                       Value
---                       -----
auto_rotate_period        0s
latest_version            2
min_available_version     0
min_decryption_version    1
name                      application-form
type                      aes256-gcm96

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request GET \
    $VAULT_ADDR/v1/transform/tokenization/keys/application-form | jq

Example output:

{
  "request_id": "a01d18ea-8219-0187-12ce-d0012dfbeb18",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "auto_rotate_period": 0,
    "latest_version": 2,
    "min_available_version": 0,
    "min_decryption_version": 1,
    "name": "application-form",
    "type": "aes256-gcm96"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

The key's latest_version is currently 2., and you can no longer import external versions of the key as it is now internally maintained by Vault.

Clean up

Unset the VAULT_TOKEN environment variable.
```
$ unset VAULT_TOKEN
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```
Unset the IMPORT_CIPHERTEXT environment variable.
```
$ unset IMPORT_CIPHERTEXT
```
Stop and remove the Postgres container.
```
$ docker rm postgres --force
```
Stop the Vault Enterprise dev mode container.
```
$ docker rm vault-enterprise --force
```

Summary

Transformation secrets engine introduced tokenization transformation feature which replaces sensitive data with unique value (token) that are unrelated to the original value in any algorithmic sense. This can help organizations to meet certain industry standards.

If retaining the original data format is important, refer to the Transform Secrets Engine to learn about the format preserving encryption (FPE) transformation.

Help and Reference

Transform secrets engine

KMIP secrets engine

This tutorial also appears in:

5 tutorials

Data encryption
Vault provides Encryption as a Service (EaaS) to enable security teams to fortify data during transit and at rest. So even if an intrusion occurs, your data is encrypted and the attacker would never get a hold of the raw data.
- Vault
6 tutorials

Advanced data protection
Advanced Data Protection is a Vault Enterprise module focused on securely connect, control, and integrate advanced encryption keys. Learn features that provide advanced data protection.
- Vault