Signed SSH certificates
The signed SSH certificates is the simplest and most powerful in terms of setup complexity and in terms of being platform agnostic. By leveraging Vault's powerful CA capabilities and functionality built into OpenSSH, clients can SSH into target hosts using their own local SSH keys.
In this section, the term "client" refers to the person or machine performing the SSH operation. The "host" refers to the target machine. If this is confusing, substitute "client" with "user".
This page will show a quick start for this secrets engine. For detailed documentation
on every path, use vault path-help
after mounting the secrets engine.
Client key signing
Before a client can request their SSH key be signed, the Vault SSH secrets engine must be configured. Usually a Vault administrator or security team performs these steps. It is also possible to automate these actions using a configuration management tool like Chef, Puppet, Ansible, or Salt.
Signing key & role configuration
The following steps are performed in advance by a Vault administrator, security team, or configuration management tooling.
Mount the secrets engine. Like all secrets engines in Vault, the SSH secrets engine must be mounted before use.
This enables the SSH secrets engine at the path "ssh-client-signer". It is possible to mount the same secrets engine multiple times using different
-path
arguments. The name "ssh-client-signer" is not special - it can be any name, but this documentation will assume "ssh-client-signer".Configure Vault with a CA for signing client keys using the
/config/ca
endpoint. If you do not have an internal CA, Vault can generate a keypair for you.If you already have a keypair, specify the public and private key parts as part of the payload:
Regardless of whether it is generated or uploaded, the client signer public key is accessible via the API at the
/public_key
endpoint.Add the public key to all target host's SSH configuration. This process can be manual or automated using a configuration management tool. The public key is accessible via the API and does not require authentication.
Add the path where the public key contents are stored to the SSH configuration file as the
TrustedUserCAKeys
option.Restart the SSH service to pick up the changes.
Create a named Vault role for signing client keys.
IMPORTANT NOTE: Prior to Vault-1.9, if
"allowed_extensions"
is either empty or not specified in the role, Vault will assume permissive defaults: any user assigned to the role may specify any arbitrary extension values as part of the certificate request to the Vault server. This may have significant impact on third-party systems that rely on anextensions
field for security-critical information. In those cases, consider using a template to specify default extensions, and explicitly setting"allowed_extensions"
to an arbitrary, non-empty string if the field is empty or not set.Because of the way some SSH certificate features are implemented, options are passed as a map. The following example adds the
permit-pty
extension to the certificate, and allows the user to specify their own values forpermit-pty
andpermit-port-forwarding
when requesting the certificate.
Client SSH authentication
The following steps are performed by the client (user) that wants to authenticate to machines managed by Vault. These commands are usually run from the client's local workstation.
Locate or generate the SSH public key. Usually this is
~/.ssh/id_rsa.pub
. If you do not have an SSH keypair, generate one:Ask Vault to sign your public key. This file usually ends in
.pub
and the contents begin withssh-rsa ...
.The result will include the serial and the signed key. This signed key is another public key.
To customize the signing options, use a JSON payload:
Save the resulting signed, public key to disk. Limit permissions as needed.
If you are saving the certificate directly beside your SSH keypair, suffix the name with
-cert.pub
(~/.ssh/id_rsa-cert.pub
). With this naming scheme, OpenSSH will automatically use it during authentication.(Optional) View enabled extensions, principals, and metadata of the signed key.
SSH into the host machine using the signed key. You must supply both the signed public key from Vault and the corresponding private key as authentication to the SSH call.
Host key signing
For an added layers of security, we recommend enabling host key signing. This is used in conjunction with client key signing to provide an additional integrity layer. When enabled, the SSH agent will verify the target host is valid and trusted before attempting to SSH. This will reduce the probability of a user accidentally SSHing into an unmanaged or malicious machine.
Signing key configuration
Mount the secrets engine. For the most security, mount at a different path from the client signer.
Configure Vault with a CA for signing host keys using the
/config/ca
endpoint. If you do not have an internal CA, Vault can generate a keypair for you.If you already have a keypair, specify the public and private key parts as part of the payload:
Regardless of whether it is generated or uploaded, the host signer public key is accessible via the API at the
/public_key
endpoint.Extend host key certificate TTLs.
Create a role for signing host keys. Be sure to fill in the list of allowed domains, set
allow_bare_domains
, or both.Sign the host's SSH public key.
Set the resulting signed certificate as
HostCertificate
in the SSH configuration on the host machine.Set permissions on the certificate to be
0640
:Add host key and host certificate to the SSH configuration file.
Restart the SSH service to pick up the changes.
Client-Side host verification
Retrieve the host signing CA public key to validate the host signature of target machines.
Add the resulting public key to the
known_hosts
file with authority.SSH into target machines as usual.
Troubleshooting
When initially configuring this type of key signing, enable VERBOSE
SSH
logging to help annotate any errors in the log.
Restart SSH after making these changes.
By default, SSH logs to /var/log/auth.log
, but so do many other things. To
extract just the SSH logs, use the following:
If you are unable to make a connection to the host, the SSH server logs may provide guidance and insights.
Name is not a listed principal
If the auth.log
displays the following messages:
The certificate does not permit the username as a listed principal for
authenticating to the system. This is most likely due to an OpenSSH bug (see
known issues for more information). This bug does not respect
the allowed_users
option value of "*". Here are ways to work around this
issue:
Set
default_user
in the role. If you are always authenticating as the same user, set thedefault_user
in the role to the username you are SSHing into the target machine:Set
valid_principals
during signing. In situations where multiple users may be authenticating to SSH via Vault, set the list of valid principles during key signing to include the current username:
No prompt after login
If you do not see a prompt after authenticating to the host machine, the signed
certificate may not have the permit-pty
extension. There are two ways to add
this extension to the signed certificate.
As part of the role creation
As part of the signing operation itself:
No port forwarding
If port forwarding from the guest to the host is not working, the signed
certificate may not have the permit-port-forwarding
extension. Add the
extension as part of the role creation or signing process to enable port
forwarding. See no prompt after login for examples.
No x11 forwarding
If X11 forwarding from the guest to the host is not working, the signed
certificate may not have the permit-X11-forwarding
extension. Add the
extension as part of the role creation or signing process to enable X11
forwarding. See no prompt after login for examples.
No agent forwarding
If agent forwarding from the guest to the host is not working, the signed
certificate may not have the permit-agent-forwarding
extension. Add the
extension as part of the role creation or signing process to enable agent
forwarding. See no prompt after login for examples.
Known issues
On SELinux-enforcing systems, you may need to adjust related types so that the SSH daemon is able to read it. For example, adjust the signed host certificate to be an
sshd_key_t
type.On some versions of SSH, you may get the following error:
This is a bug introduced in OpenSSH version 7.2 and fixed in 7.5. See OpenSSH bug 2617 for details.
On some versions of SSH, you may get the following error on target host:
Fix is to add below line to /etc/ssh/sshd_config
The ssh-rsa algorithm is no longer supported in OpenSSH 8.2
API
The SSH secrets engine has a full HTTP API. Please see the SSH secrets engine API for more details.