Google Cloud Platform workload identity federation
Nomad Workload Identities uniquely identify each instance of a workload running in a Nomad cluster. Google Cloud Platform Workload Identity Federation grants access to Google Cloud Platform services such as Google Cloud Storage via third party identity providers such as Nomad Workload Identities.
In this tutorial, you will setup Nomad Workload Identity as an identity provider for Google Cloud Platform's Workload Identity Federation. You will test identity federation by running a sample batch job that uploads a file to a private Google Cloud Storage bucket.
Prerequisites
To follow this tutorial, you need:
A Google Cloud Platform Project
A Parent Zone in Google Cloud DNS. Nomad will create a subdomain under this parent domain.
HashiCorp Terraform configured with your GCP authentication method.
Create infrastructure with Terraform
The hashicorp-education/learn-nomad-workload-identity-federation repository contains sample Terraform for creating the Google Cloud Platform infrastructure required by this tutorial.
Clone the repository:
Change to the gcp
subdirectory of the repository:
Initialize terraform:
This will initialize the required Terraform providers.
Now, look at the variables.tf
file, which contains all of the input
variables you must provide.
region
- The GCP region to deploy to.zone
- The Google Compute Engine zone to create compute instances in.project
- The GCP project to use.parent_zone_name
- The parent domain for the HTTPS certificate. This must already exist.domain
- The domain for Nomad cluster. This must be a child domain of the preexisting parent zone. Terraform will use the domain to create a TLS certificate. If the parent zone wereexample.com
then this variable could benomad.example.com
.
Tip
In production the domain
is an important choice: it must match the
oidc_issuer
configured on
your Nomad Server agents.
Create a file called tutorial.auto.tfvars
and paste in the following code,
which defines the Terraform variable values. Replace the values with your own.
Then use Terraform apply to create the infrastructure.
Review the infrastructure plan and respond yes
to
apply it. Once Terraform creates the infrastructure, it will template the domain
into a Nomad Agent
configuration file named agent.hcl
and provide other outputs you will use
in the next step.
Run Nomad agent
Terraform created a Google Compute Engine (GCE) instance and installed the Nomad binary on it. For this tutorial you will manually run the Nomad agent as a single node cluster. See Set up a Nomad cluster on GCP for instructions on deploying a production Nomad cluster to GCP.
Open a terminal on the GCE instance through GCP's Console or gcloud
CLI. Copy the
agent.hcl
file that Terraform created onto the GCE instance. If you are
using the gcloud
CLI, Terraform's output will include helpful configuration
commands:
Once you've started a terminal session on the GCE instance and uploaded agent.hcl
, run the Nomad Agent as root.
Notice that Nomad warns you that you didn't configure TLS and are running in bootstrap mode. You can ignore these warnings for this tutorial. Nomad will also confirm that the node is registered.
Run NGINX proxy
Nomad's Workload Identities are JSON Web Tokens (JWT) signed by a public key that the Nomad Server Agent created. Google must get that public key from Nomad to confirm that a JWT is a valid Nomad Workload Identity.
Nomad exposes public keys as a RFC 7517
JWKS HTTP endpoint:
/.well-known/jwks.json
.
Nomad's HTTP API is for local network access only and not intended to be
directly exposed to the Internet.
However Google requires the JWKS endpoint to be secured with a trusted public
TLS Certificate, so the endpoint must also be available on a public domain. In
this tutorial the domain corresponds to the domain
Terraform variable above
and should be a subdomain of the existing parent zone defined by the
parent_zone_name
Terraform variable.
Terraform configured a load balancer with the proper domain name and certificate, but the load balancer is not configured to route directly to the Nomad Agent. Instead you will follow the best practice of running a proxy between the load balancer and Nomad. For this tutorial you will use NGINX. While a proxy is not strictly necessary, an NGINX load balancer will let you:
- only expose the
/.well-known/jwks.json
endpoint. - allow unauthenticated access to
/.well-known/jwks.json
if ACLs are enabled. - use the Task API to avoid having to handle Nomad mTLS certificates if mTLS is enabled.
- expose Nomad's UI through an NGINX proxy as well. See the Configure NGINX reverse proxy for Nomad's web UI tutorial for details.
In a new terminal, copy the proxy.nomad.hcl
file to the GCE instance and run
the NGINX proxy. To upload the job with gcloud
in the gcp/
subdirectory of
the repo run:
Once uploaded, open a terminal on the instance to run the job:
Run the job on the instance:
Google Federated Workload Identity can now read Nomad's JWKS endpoint to
validate Nomad Workload Identity JWTs. You can inspect the JWKS endpoint from
your browser or with curl
and jq
in a terminal:
Test the federation
To test that Google Federated Workload Identity accepts Nomad Workload
Identities, you will upload a test file to a GCS bucket created by Terraform.
The Nomad job uses variables to specify parameters such as the GCP project and
GCS bucket. Terraform created a Nomad variable file gcs.nomadvars.hcl
to
set the variables for the infrastructure created in the tutorial.
Upload the gcs.nomadvars.hcl
and gcs.nomad.hcl
files to the GCE instance
and run the GCS job with Nomad:
Once uploaded, open a terminal on the instance to run the job:
Run the job with the variable file:
The job uses a large Google Cloud SDK image which may take a couple minutes to download. You may use Nomad to check the status of the workload:
In a few seconds the job creates a test.txt
file in the GCS bucket that Terraform created. Read the file with the
gcloud
CLI.
Open the gcs.nomad.hcl
file and notice how the job uses its workload
identity to authenticate with Google and upload to GCS. If you make changes to
the template
and re-run the job, your changes should be reflected in the
test.txt
file in GCS.
When you are done exploring run terraform destroy
to cleanup all
of the created infrastructure.
Next Steps
In this tutorial, you configured Nomad as a federated identity provider for Google Cloud Platform.
The following resources are recommended for learning more:
Google Cloud's Workload Identity Federation documentation covers all of the workflows and infrastructure supported by Nomad. In Google's documentation "Workload identity pool provider" refers to the role Nomad fulfills.
Nomad's Workload Identity documentation covers how the details of Nomad's federated identity support for workloads.
Nomad's Cluster Setup and Transport Security tutorials cover how to set up production clusters. This tutorial only used a single VM to run Nomad and the sample workload.