Migrate Alternative Worker Images to Agents: v202302-1
Summary
Terraform Enterprise v202302-1 introduced the agent run pipeline mode which
changes the container image used to perform Terraform runs. The previous run
pipeline mode is now referred to as the legacy run pipeline mode.
If you currently use the default image, you do not need to take action. Terraform Enterprise will automatically migrate you to the new default agent image when you adopt v202302-1 or later.
If you currently use a legacy custom
image Terraform
Enterprise will continue using your legacy custom image until you migrate to an
agent custom image and enable the agent run pipeline mode. Before adopting
v202305-1, you must build an agent custom
image. If your
legacy custom image also executes custom
scripts,
you must update migrate your customization agent
hooks instead.
Terraform Enterprise will stop supporting legacy custom images in v202305-1.
Continue reading to determine if this change impacts you and what action to take.
What are alternative worker images?
An alternative worker image is the custom image Terraform Enterprise uses to perform Terraform runs.
What is changing and why?
We are replacing three components of the run pipeline, terraform-build-worker,
terraform-build-manager, and tfe-rabbitmq with local Terraform Cloud/Enterprise
agents. If you use an alternative worker image to execute custom logic within
the run lifecycle, you will need to migrate that logic to an agent custom image.
Like Terraform runs that use alternative worker images, runs that use custom agents will still execute in isolated, short-lived Docker containers. This change does not affect the Terraform Runs API and UI.
This change is part of an ongoing effort to refresh the architecture of Terraform Enterprise, improve performance and reliability of runs, and support new application-level features. Some immediate benefits of this change include:
- The run pipeline contains fewer components, making it easier to understand and debug.
- Local runs and remote agent runs can now use the same base images, and the same pre- and post- run hooks.
- Runs are more evenly distributed across Active/Active nodes by eliminating a redundant queue.
- You can now access previously inaccessible run logs (Sentinel, cost
estimation, and plan-export-worker logs) via Docker logs and any tool that can
interact with it. This includes the
docker logscommand, the Docker API, and TFE log forwarding.
In v202302-1, Terraform Enterprise installations using the default worker image (not an alternative worker image) will automatically use the new run pipeline.
Installations that use an alternative worker image will continue to use the legacy run pipeline when you upgrade to v202302-1. You must migrate your Terraform Enterprise installation to use an agent custom image before adopting v202305-1, which requires rebuilding your alternative worker image as an agent custom image.
Once you rebuild your image, you must manually switch to the new run pipeline by setting the run_pipeline_mode = "agent". You can revert back to the legacy run pipeline at any time before you adopt v202305-1 by setting run_pipeline_mode = "legacy". The legacy option will become unavailable in v202305-1 and later.
Check if you use an alternative worker image
By default, Terraform Enterprise uses the standard image included in a release. You can verify whether your pipeline uses an alternative worker image by reviewing your settings.json file or your admin console.
In the admin console's Settings tab, check if Provide the location of a custom image is selected. If it is, your pipeline uses the alternative worker image specified in the Custom image tag field.
You can also review your settings.json file to check if your pipeline uses alternative worker image. If your file includes the custom_image_tag field, your pipeline uses the worker image specified.
Migrate to a custom agent image
To migrate to a custom agent image, first review the customizations in your worker image. Then build your new agent image, add your customizations, and test the new, customized agent image.
Step 1: Evaluate customizations in your current image
Before making any changes, you must evaluate your current image and familiarize yourself with all of the implemented customizations.
Consider the following:
- Which custom tools does your image include?
- Does the image use the initialize and/or finalize hook(s)?
- If yes, what do your initialize and/or finalize script(s) do?
Step 2: Build your new custom agent image
Starting with v202302-1, the Docker container that Terraform Cloud and Enterprise uses for Terraform
operations is based on the hashicorp/tfc-agent Docker
image. Like the legacy worker image, the agent image supports customizations to perform
custom logic, but there are several implementation differences between
worker images and agent images to consider.
The following tables describe the differences between the legacy alternative worker image and the new custom agent image.
Use an image
| Alternative worker image | Custom agent image | |
|---|---|---|
| Building a custom image | Documentation | Documentation |
| Executing custom scripts | Documentation | Documentation |
settings.json parameter | Set the custom_image_tag field to your custom worker image | Set the custom_agent_image_tag field to your custom agent image. |
| Image location | The image must be located on the host. | If Terraform Enterprise cannot find the image on the host, it will try to pull it from the specified location. |
Customize an image
| Alternative Worker Image | Custom Agent Image | |
|---|---|---|
| Base image | The base image must be ubuntu:bionic or an official RHEL7 image (e.g., registry.access.redhat.com/ubi7/ubi-minimal). | The base image must be the hashicorp/tfc-agent. |
| Image location | The image must exist on the Terraform Enterprise host. You can add it by running docker pull from a local registry or any other similar method. | We recommend that the image exist locally, but if it does not, the tfe-task-worker Docker driver will pull the image from the specified source. |
| Required software | The image must contain the software packages defined here. | N/A |
| Certificates | The /usr/local/share/ca-certificates directory must contain all necessary PEM-encoded CA certificates. Each file added to this directory must end with the .crt extension. Terraform Enterprise will not automatically add the CA certificates configured in the CA Bundle settings to the image at runtime. | The /usr/local/share/ca-certificates directory must contain all necessary PEM-encoded CA certificates. Each file added to this directory must end with the .crt extension. Terraform Enterprise will not automatically add the CA certificates configured in the CA Bundle settings to the image at runtime. This will be similar to the certificate config in this Dockerfile example. |
| Terraform binary | The image cannot include Terraform. Terraform Enterprise installs Terraform at runtime. | The image cannot include Terraform. Terraform Enterprise installs Terraform at runtime. |
| Rhel 7 | In addition to the packages that yum installs, curl installs the envdir tool Python port because it is a dependency of the Terraform Build Worker service. If the alternative worker image does not include envdir, Terraform runs will fail within Terraform Enterprise. | N/A |
| Run before plans and applies | The same initialize script runs before a terraform init during both plans and applies. | You can have separate hooks that run before and after plans and applies: terraform-pre-plan terraform-pre-apply. |
| Run after plans and applies | The same finalize script runs after terraform plan and terraform apply complete. | You can have separate hooks that run before and after plans and applies: terraform-post-plan terraform-post-apply |
| Configuration | Ensure your worker image contains an executable shell script at: /usr/local/bin/init_custom_worker.sh /usr/local/bin/finalize_custom_worker.sh | Store hooks in a hooks directory, and name them according to their purpose. For example: ~/.tfc-agent/hooks/terraform-pre-plan ~/.tfc-agent/hooks/terraform-post-plan ~/.tfc-agent/hooks/terraform-pre-apply ~/.tfc-agent/hooks/terraform-post-apply |
| Non-zero exit codes | If the script exits with a non-zero exit code, the Terraform Enterprise run will immediately fail with an error. | If a terraform apply does not complete successfully, the hook will still run. If a hook exits with a non-zero exit code, the Terraform run will fail immediately. |
stdout/stderr | The build worker logs whether or not a custom script has executed but does not log the script's standard output and standard error. If the custom script exits non-zero, then its standard output and standard error print to the UI logs. | The standard output and standard error from the hook will print alongside the Terraform run output in the Terraform Cloud/Enterprise user interface, but not in the UI logs. |
| File locations | You cannot customize the name or location of the script. | The hook name must match one of the supported hooks. You cannot customize or change these names. Because of this, you can only configure one hook of each type for each agent. For example, you could create a pre-plan and pre-apply hook, but you cannot create two pre-plan hooks. |
| Script permissions | The script must have execute permissions. | Each hook must have the execute permission set. |
| Execution timeout | The execution of the script does not have a timeout. It is up to the Terraform Enterprise administrator to ensure scripts execute in a timely fashion. | Each hook has a 60 second timeout. If a hook times out the Terraform run will fail immediately. |
| Execution context | The execution of the script is not sandboxed. The script executes in the same container where terraform runs, and both can access the same set of environment variables. | Like with alternative worker images, the execution of the script is not sandboxed. The script executes in the same container where terraform runs, and both can access the same set of environment variables. |
| Reading and writing variables | Users can "export" environment variables for subsequent Terraform commands (plan, apply, etc) to use, by writing their environment variable value to a file /env/FOO where FOO is the name of the environment variable. | You cannot persist environment variables from the custom script to subsequent Terraform commands (plan, apply, etc) like you could with build workers. There is no concept of an /env directory in agents. Custom scripts run as a child process, so environment variables from the export command will not persist after the custom script finishes executing. |
| Running in Docker | Example Dockerfile snippet: ADD init_custom_worker.sh /usr/local/bin/init_custom_worker.sh | When running tfc-agent using Docker, you must build a new Docker image containing the hooks. To configure hooks, follow these instructions. |
| Execute/running as a binary | Scripts always run in the worker container. To execute, initialize and finalize scripts and all the commands they invoke, ensure your worker image contains an executable shell script at /usr/local/bin/<init or finalize>_custom_worker.sh. | If you want to run agents outside of containers, you must create your own remote agent pool. To run hooks as a binary, you need to change the permissions to the hook scripts. |
Step 3: Test your custom agent image
We strongly recommend that you test your custom agent image before applying it to your production installation of Terraform Enterprise.
Test on Terraform Enterprise v202301 or earlier
Terraform Enterprise v202301 and earlier use worker images, but you can still test your agent image by configuring a test workspace to use a remote agent. You will need to install your agent on the host of your choice, and configure your workspace to use that agent. Follow the process described here for more details.
Test on Terraform Enterprise v202302 or later
To test in v202302 or later, install a test installation of Terraform Enterprise, define a custom_agent_image_tag, and set run_pipeline_mode = "agent" in your test installation’s settings.json.
Step 4: Adopt the new image
After building and testing the new custom agent image, adopt the new image in your production Terraform Enterprise installation (v202302 or later) by defining a custom_agent_image_tag, and setting run_pipeline_mode = "agent" in your settings.json file.
Frequently Asked Questions
Q. I currently use an alternative worker image, but I no longer need the customizations. What should I do?
A. If you want to use the default agent image going forward instead of a customized image, please remove your alternative worker image from use before you upgrade to v202302-1. If you want to remove the alternative worker image after upgrading to v202302-1 or later, set run_pipeline_mode = "agent" in your settings.json file without specifying a custom agent image.
Q. I already use a custom image for my remote agents. Can I re-use that image for my local agent image?
A. Yes. If your remote agent image accomplishes everything you want, you can specify that image as your custom agent image.
Q. I use an alternative worker image, when do I need to complete my migration to a custom agent image?
A. Terraform Enterprise will stop supporting workers in v202305-1. To maintain your customizations, you must complete your migration to a custom agent image before upgrading to v202305-1 or later.
Q. Can I switch back to the legacy run pipeline?
A. Yes, until v202305-1 you can switch between workers and agents. To switch back to the worker run pipeline, set run_pipeline_mode = "legacy" in settings.json. We will remove legacy mode in v202305-1.
Q. I use an alternative worker image to dynamically manage run credentials. What is the best way to accomplish this with a custom agent image?
A. You can dynamically manage credentials using workspace variables and/or agent hooks. Upcoming enhancements to Terraform Enterprise will make this easier. Please reach out to your HashiCorp account representative for more information.
Q. I have a custom workflow on my alternative worker image that does not seem possible on a custom agent image. What should I do?
A. Please reach out to your HashiCorp account representative for further guidance.