Terraform Enterprise Consolidated Services Architecture

Terraform Enterprise is adopting a simplified architecture where all server services are consolidated into a single container called terraform-enterprise. This change improves how you install and operate Terraform Enterprise, and includes an immediate security benefit of running containers as a non-root user. This architecture is the default configuration in the v202309-1 release, and you can disable it using the consolidated_services_enabled setting until v202401-1, when we will remove the setting.

Changes

When consolidated_services_enabled is enabled:

Terraform Enterprise services run inside the terraform-enterprise container.
Containers run as a non-root user.

When log-forwarding is not enabled, all service logs are logged to STDOUT, are in json-file format and include a new component attribute that specifies which Terraform Enterprise service emitted the log.

The log below is an example of a log from the atlas component and is in escaped JSON format.

{"component":"atlas","log":"2023-09-14 19:04:42 [INFO] [d6e4e6db-06b6-4297-b1ff-ae310ebee25e] [dd.service=atlas
dd.trace_id=4001067029455957720 dd.span_id=0] {\"method\":\"GET\",\"path\":\"/_health_check\",\"format\":\"html\",
\"status\":200,\"allocations\":503,\"duration\":1.26,\"view\":0.32,\"db\":0.0,\"dd\":{\"trace_id\":
\"4001067029455957720\",\"span_id\":\"0\",\"env\":\"\",\"service\":\"atlas\",\"version\":\"\"},\"ddsource\":
[\"ruby\"],\"uuid\":\"d6e4e6db-06b6-4297-b1ff-ae310ebee25e\",\"remote_ip\":\"127.0.0.1\",\"request_id\":
\"d6e4e6db-06b6-4297-b1ff-ae310ebee25e\",\"user_agent\":\"Load Balancer Agent\",\"user\":null,\"auth_source\":null}"}

The log below is an example of a log from the sidekiq component and is a JSON formatted log.

{"component":"sidekiq","log":"2023-09-14 19:04:19 [INFO] msg=Worker finish worker=AgentStatusWorker"}

The log below is an example of an Audit Log from the atlas component and is in JSON escaped format.

{"component":"atlas","log":"2023-09-14 19:59:36 [INFO] [9787b874-565a-4cd1-9146-f9e89a6286f4] [dd.service=atlas
dd.trace_id=1839403271971964936 dd.span_id=0] [Audit Log] {\"resource\":\"workspace\",\"action\":\"create\",
\"resource_id\":\"ws-8RoTSc9iow6JE6Nt\",\"organization\":\"banana\",\"organization_id\":\"org-DPfZszgSorjbaF9M\",
\"actor\":\"manage\",\"timestamp\":\"2023-09-14T19:59:36Z\",\"meta\":{\"project_id\":\"prj-f54CoqSE8X9sXd5F\"},
\"actor_ip\":\"24.17.65.143\"}"}

The following components emit log messages in escaped JSON format:

If using the log-forwarding feature, your service logs are sent to your configured log aggregation service and are in the format that the fluent-bit plugin uses.

In previous releases, the container_name attribute contained the name of the Terraform Enterprise service that emitted the log. In the v202309-1 release and beyond, the container_name attribute is terraform-enterprise, and the component metadata attribute logs the name of the service responsible for emitting that log. For example, prior to v202309-1 your log metadata resembles:

  ```
  service: terraform_enterprise
  container_name: tfe-atlas
  component: n/a
  ```

After v202309-1, the log metadata resembles:

```
service: terraform_enterprise
container_name: terraform_enterprise
component: atlas
```

What hasn’t changed?

Terraform runs continue to execute in isolated, short-lived containers.

How can I disable consolidated services?

Disable the consolidated_services_enabled setting.

replicatedctl app-config set consolidated_services_enabled --value 0

Restart Terraform Enterprise.

replicatedctl app stop

replicatedctl app status

replicatedctl app start

Frequently Asked Questions (F.A.Q)

What should I test to verify if I'm impacted by this change?

We advise users to evaluate the impact this will have on your monitoring and log forwarding implementation.

All server services are now included in a single container. If you are monitoring container metrics, please note that you will have fewer containers reporting information. Run containers are not impacted by this change, they remain separate and short-lived.

Service logs have been consolidated into a single log stream.

Where should I seek help with issues?

Contact HashiCorp support for help with any issues.

Will this always be an optional architecture?

No. In v202401-1, consolidated services will become the default and only option.