Upgrading Consul on Kubernetes components
This topic describes considerations and strategies for upgrading Consul deployments running on Kubernetes clusters. In addition to upgrading the version of Consul, you may need to update your Helm chart or the release version of the Helm chart.
Version-specific upgrade requirements
As of Consul v1.14.0, Kubernetes deployments use Consul Dataplane instead of client agents. If you upgrade Consul from a version that uses client agents to a version that uses dataplanes, you must follow specific steps to update your Helm chart and remove client agents from the existing deployment. Refer to Upgrading to Consul Dataplane for more information.
The v1.0.0 release of the Consul on Kubernetes Helm chart also introduced a change to the externalServers[].hosts
parameter. Previously, you were able to enter a provider lookup as a string in this field. Now, you must include exec=
at the start of a string containing a provider lookup. Otherwise, the string is treated as a DNS name. Refer to the go-netaddrs
library and command line tool for more information.
Upgrade types
We recommend updating Consul on Kubernetes when:
- You change your Helm configuration
- A new Helm chart is released
- You want to upgrade your Consul version
Helm configuration changes
If you make a change to your Helm values file, you need to perform a helm upgrade
for those changes to take effect.
For example, if you installed Consul with connectInject.enabled: false
and you want to change its value to true
:
Determine your current installed chart version.
In this example, version
0.40.0
(fromconsul-k8s:0.40.0
) is being used, and Consul on Kubernetes is installed in theconsul
namespace.Perform a
helm upgrade
:Before performing the upgrade, be sure you read the other sections on this page, continuing at Determine scope of changes.
Note: You should always set the --version
flag when upgrading Helm. Otherwise, Helm uses the most up-to-date version in its local cache, which may result in an unintended upgrade.
Upgrade Helm chart version
You may wish to upgrade your Helm chart version to take advantage of new features and bug fixes, or because you want to upgrade your Consul version and it requires a certain Helm chart version.
Update your local Helm repository cache:
List all available versions. Here you can observe that the latest version of
0.40.0
.To determine which version you have installed, issue the following command:
In this example, version
0.39.0
(fromconsul-k8s:0.39.0
) is being used.
If you want to upgrade to the latest 0.40.0
version, use the following procedure:
Check the changelog for any breaking changes from that version and any versions in between: CHANGELOG.md.
Upgrade by performing a
helm upgrade
with the--version
flag:Before performing the upgrade, be sure you've read the other sections on this page, continuing at Determine scope of changes.
Upgrade Consul version
If a new version of Consul is released, you need to perform a Helm upgrade to update to the new version. Before you upgrade to a new version:
Read the Upgrading Consul documentation.
Read any specific instructions for the version you want to upgrade to, as well as the Consul changelog for that version.
Read our Compatibility Matrix to ensure your current Helm chart version supports this Consul version. If it does not, you may need to also upgrade your Helm chart version at the same time.
Set
global.image
in yourvalues.yaml
to the desired version:values.yamlDetermine the version of your existing Helm installation. In this example, version
0.39.0
(fromconsul-k8s:0.39.0
) is being used.Perform a
helm upgrade
:Before performing the upgrade, be sure you have read the other sections on this page, continuing at Determine scope of changes.
Note: You should always set the --version
flag when upgrading Helm. Otherwise, Helm uses the most up-to-date version in its local cache, which may result in an unintended upgrade.
Determine scope of changes
Before upgrading, it is important to understand the changes that affect your cluster. For example, you need to take more care if your upgrade results in the Consul server StatefulSet being redeployed.
There is no built-in functionality in Helm that shows what a helm upgrade changes. There is, however, a Helm plugin helm-diff that can be used.
Install
helm-diff
with:If you are updating your
values.yaml
file, do so now.Take the same
helm upgrade
command you were planning to issue but performhelm diff upgrade
instead ofhelm upgrade
:This command prints out the manifests that will be updated and their diffs.
To see only updated objects, add
| grep "has changed"
:Take specific note if
consul-server, StatefulSet
is listed, as it means your Consul server statefulset will be redeployed.If your Consul server statefulset needs to be redeployed, follow the same pattern for upgrades as on other platforms by redeploying servers one by one. Refer tp Upgrading Consul for more information.
If neither the server statefulset is not being redeployed, then you can continue with the Helm upgrade without any specific sequence to follow.
Upgrade Consul servers
Initiate the server upgrade:
Change the
global.image
value to the desired Consul version.Set the
server.updatePartition
value to the number of server replicas. By default, there are 3 servers, so you would set this value to3
.Set the
updateStrategy
for clients toOnDelete
.values.yamlThe
updatePartition
value controls how many instances of the server cluster are updated. Only instances with an index greater than theupdatePartition
value are updated (zero-indexed). Therefore, by setting it equal to replicas, updates should not occur immediately.Next, perform the upgrade:
This command does not cause the servers to redeploy, although the resource is updated. If everything is stable, decrease the
updatePartition
value by one and performinghelm upgrade
again. This will cause the first Consul server to be stopped and restarted with the new image.Wait until the Consul server cluster is healthy again (30s to a few minutes). This can be confirmed by issuing
consul members
on one of the previous servers, and ensuring that all servers are listed and arealive
.Decrease
updatePartition
by one and upgrade again. Continue untilupdatePartition
is0
. At this point, you may remove theupdatePartition
configuration. Your server upgrade is complete.
Upgrading to Consul Dataplane
In earlier versions, Consul on Kubernetes used client agents in its deployments. As of v1.14.0, Consul uses Consul Dataplane in Kubernetes deployments instead of client agents.
If you upgrade Consul from a version that uses client agents to a version the uses dataplanes, complete the following steps to upgrade your deployment safely and without downtime.
If ACLs are enabled, you must first upgrade to consul-k8s 0.49.8 or above. These versions expose the setting
connectInject.prepareDataplanesUpgrade
which is required for no-downtime upgrades when ACLs are enabled.Set
connectInject.prepareDataplanesUpgrade
totrue
and then perform the upgrade to 0.49.8 or above (whichever is the latest in the 0.49.x series)Consul dataplanes disables Consul clients by default, but during an upgrade you need to ensure Consul clients continue to run. Edit your Helm chart configuration and set the
client.enabled
field totrue
and specify an action for Consul to take during the upgrade process in theclient.updateStrategy
field:Follow our recommended procedures to upgrade servers on Kubernetes deployments to upgrade Helm values for the new version of Consul. The latest version of consul-k8s components may be in a CrashLoopBackoff state during the performance of the server upgrade from versions <1.14.x until all Consul servers are on versions >=1.14.x. Components in CrashLoopBackoff will not negatively affect the cluster because older versioned components will still be operating. Once all servers have been fully upgraded, the latest consul-k8s components will automatically restore from CrashLoopBackoff and older component versions will be spun down.
Run
kubectl rollout restart
to restart your service mesh applications. Restarting service mesh application causes Kubernetes to re-inject them with the webhook for dataplanes.Restart all gateways in your service mesh.
Now that all services and gateways are using Consul dataplanes, disable client agents in your Helm chart by deleting the
client
stanza or settingclient.enabled
tofalse
and running aconsul-k8s
or Helm upgrade.If ACLs are enabled, outdated ACL tokens will persist a result of the upgrade. You can manually delete the tokens to declutter your Consul environment.
Outdated connect-injector tokens have the following description:
token created via login: {"component":"connect-injector"}
. Do not delete the tokens that have a description wherepod
is a key, for exampletoken created via login: {"component":"connect-injector","pod":"default/consul-connect-injector-576b65747c-9547x"}
). The dataplane-enabled connect inject pods use these tokens.You can also review the creation date for the tokens and only delete the injector tokens created before your upgrade, but do not delete all old tokens without considering if they are still in use. Some tokens, such as the server tokens, are still necessary.
Configuring TLS on an existing cluster
If you already have a Consul cluster deployed on Kubernetes and would like to turn on TLS for internal Consul communication, refer to Configuring TLS on an Existing Cluster.