WAN Federation Between VMs and Kubernetes Through Mesh Gateways
1.8.0+: This feature is available in Consul versions 1.8.0 and higher
This topic requires familiarity with Mesh Gateways and WAN Federation Via Mesh Gateways.
This page describes how to federate Consul clusters separately deployed in VM and Kubernetes runtimes. Refer to Multi-Cluster Overview for more information, including Kubernetes networking requirements.
Consul datacenters running on non-kubernetes platforms like VMs or bare metal can be federated with Kubernetes datacenters.
Kubernetes as the Primary
One Consul datacenter must be the primary. If your primary datacenter is running on Kubernetes, use the Helm config from the Primary Datacenter section to install Consul.
Once installed on Kubernetes, and with the ProxyDefaults
resource created,
you'll need to export the following information from the primary Kubernetes cluster:
- Certificate authority cert and key (in order to create SSL certs for VMs)
- External addresses of Kubernetes mesh gateways
- Replication ACL token
- Gossip encryption key
The following sections detail how to export this data.
Certificates
Retrieve the certificate authority cert:
And the certificate authority signing key:
With the
consul-agent-ca.pem
andconsul-agent-ca-key.pem
files you can create certificates for your servers and clients running on VMs that share the same certificate authority as your Kubernetes servers.You can use the
consul tls
commands to generate those certificates:Note the
-node
option in the above command. This should be same as the node name of the Consul Agent. This is a requirement for Consul Federation to work. Alternatively, if you plan to use the same certificate and key pair on all your Consul server nodes, or you don't know the nodename in advance, use-node "*"
instead. Not satisfying this requirement would result in the following error in the Consul Server logs:[ERROR] agent.server.rpc: TLS handshake failed: conn=from= error="remote error: tls: bad certificate"
See the help for output of
consul tls cert create -h
to see more options for generating server certificates.These certificates can be used in your server config file:
server.hclFor clients, you can generate TLS certs with:
Or use the auto_encrypt feature.
Mesh Gateway Addresses
Retrieve the WAN addresses of the mesh gateways:
In this example, the addresses are the same because both mesh gateway pods are fronted by the same Kubernetes load balancer.
These addresses will be used in the server config for the primary_gateways
setting:
Replication ACL Token
If ACLs are enabled, you'll also need the replication ACL token:
This token will be used in the server config for the replication token.
NOTE: You'll also need to set up additional ACL tokens as needed by the ACL system. See tutorial Secure Consul with Access Control Lists (ACLs) for more information.
Gossip Encryption Key
If gossip encryption is enabled, you'll need the key as well. The command to retrieve the key will depend on which Kubernetes secret you've stored it in.
This key will be used in server and client configs for the encrypt
setting:
Final Configuration
A final example server config file might look like:
Kubernetes as the Secondary
If you're running your primary datacenter on VMs then you'll need to manually construct the Federation Secret in order to federate Kubernetes clusters as secondaries. In addition, primary clusters must be able to make requests to the Kubernetes API URLs of secondary clusters when ACLs are enabled.
Your VM cluster must be running mesh gateways, and have mesh gateway WAN federation enabled. See WAN Federation via Mesh Gateways.
You'll need:
The root certificate authority cert placed in
consul-agent-ca.pem
.The root certificate authority key placed in
consul-agent-ca-key.pem
.The IP addresses of the mesh gateways running in your VM datacenter. These must be routable from the Kubernetes cluster.
If ACLs are enabled you must create an ACL replication token with the following rules:
This token is used for ACL replication and for automatic ACL management in Kubernetes.
If you're running Consul Enterprise you'll need the rules:
If ACLs are enabled you must also modify the anonymous token policy to have the following permissions:
With Consul Enterprise, use:
These permissions are needed to allow cross-datacenter requests. To make a cross-dc request the sidecar proxy in the originating DC needs to know about the services running in the remote DC. To do so, it needs an ACL token that allows it to look up the services in the remote DC. The way tokens are created in Kubernetes, the sidecar proxies have local ACL tokens–i.e tokens that are only valid in the local DC. When a request goes from one DC to another, if the request has a local token, it is stripped from the request because the remote DC won't be able to validate it. When the request lands in the other DC, it has no ACL token and so will be subject to the anonymous token policy. This is why the anonymous token policy must be configured to allow read access to all services. When the Kubernetes DC is the primary, this is handled automatically, but when the primary DC is on VMs, this must be configured manually.
To configure the anonymous token policy, first create a policy with the above rules, then attach it to the anonymous token. For example using the CLI:
If gossip encryption is enabled, you'll need the key.
With that data ready, you can create the Kubernetes federation secret:
If ACLs are enabled, you must next determine the Kubernetes API URL for the secondary cluster. The API URL of the must be specified in the config files for all secondary clusters because secondary clusters need to create global Consul ACL tokens (tokens that are valid in all datacenters) and these tokens can only be created by the primary datacenter. By setting the API URL, the secondary cluster will configure a Consul auth method in the primary cluster so that components in the secondary cluster can use their Kubernetes ServiceAccount tokens to retrieve global Consul ACL tokens from the primary.
To determine the Kubernetes API URL, first get the cluster name in your kubeconfig:
Then get the API URL:
You'll use this URL when setting global.federation.k8sAuthMethodHost
.
Then use the following Helm config file:
Notes:
- You must fill out the
server.extraConfig
section with the IPs of your mesh gateways running on VMs. - Set
global.federation.k8sAuthMethodHost
to the Kubernetes API URL of this cluster (includinghttps://
). global.federation.primaryDatacenter
should be set to the name of your primary datacenter.
With your config file ready to go, follow our Installation Guide to install Consul on your secondary cluster(s).
After installation, if you're using consul-helm 0.30.0+, create the
ProxyDefaults
resource
to allow traffic between datacenters.
Next Steps
In both cases (Kubernetes as primary or secondary), after installation, follow the Verifying Federation section to verify that federation is working as expected.