Certificates and Trust Management

The components of Charmed Kubernetes need to be able to communicate securely over the network. This is accomplished using TLS and public-key encryption with a chain of trust up to a shared root Certificate Authority (CA). However, when the cluster is being brought up or a new unit is being added, the chain of trust and certificates required must be bootstrapped into the machines somehow.

Juju relations

All communication between Juju units and the Juju controller happens over TLS-encrypted websockets. The certificate for the TLS connection to the controller is added as explicitly trusted to each machine as part of the bootstrap process using a combination of cloud-init and SSH.

With this secure channel, Juju charms can communicate with each other using relation data. The data published by one unit is sent to the controller, which then makes it available for all other units on the same relation. The data for each relation is scoped by ID and is only visible to units participating in the specific relation on which it is set. However, it is worth noting that relation data is stored on the controller machine in MongoDB,so for truly sensitive information, proper secret storage engines, such as Vault, and encryption-at-rest should be used.

Managing certificates

Unfortunately, the Juju controller does not provide any mechanisms for generating or distributing additional certificates to be used by the applications on the machines, so they must be managed by the charms via the secure relation data channel. This is done using the tls-certificates interface protocol and a relation to an application providing a Certificate Authority. (This CA could be either a root CA, or an intermediary CA authorised by some other root CA to issue certificates.)

When the relation is established, the root CA’s certificate is sent via the relation and installed as trusted. Then, certificate requests can be issued over the relation and new certificates created by the CA and returned over the relation.

Because all units with a relation to the CA have a chain of trust up to it (or its root CA), they will automatically trust a certificate generated by the CA without requiring anything to be communicated from the unit which holds the certificate. On the other hand, for the certificates to be trusted externally (such as by clients) or by applications without a relation to the CA, the CA will have to be configured as an intermediary CA with a chain of trust up to a globally trusted CA (such as Comodo or DigiCert).

The certificate information will also be included in the Kubernetes config file that Charmed Kubernetes generates to be used by kubelet, so that communications between the local machine and Kubernetes will be secured.

Server certificates

Each service that will be connected to will need a server certificate identifying and validating it to any clients that wish to connect.

The primary address at which the service can be reached will be its common name; ideally, this will be a fully-qualified domain name (FQDN) which will not change, but for internal service communication, it is often just the ingress address for the unit. Any additional names or addresses by which the service can be reached will be its subject alternative names (SANs).

Charmed Kubernetes will manage the server certificates automatically, including generating the certificate with the proper CN and SANs. However, the kubernetes-control-plane charm also supports an extra_sans option which can be used to provide additional names to be added to the SANs list.

Client certificates

In order to provide for two-way security, some services require that clients identify themselves via a client certificate. These are more or less the same as server certificates, but are presented by a client to the service they are connecting to so that the service can validate that client’s identity. Client certificates can only be used to identify a client and will be rejected by clients if presented by a service they are connecting to.

Certificate Authorities for Charmed Kubernetes

Charmed Kubernetes can use a CA provided by any charm which provides a tls-certificates endpoint. The two current recommendations are EasyRSA and Vault.

EasyRSA

By default, the Charmed Kubernetes bundle includes the EasyRSA charm. This is a relatively simple charm which uses OpenVPN’s easy-rsa to provide a CA and sign certificates. This is lightweight and works out of the box without any additional configuration, but it cannot act as an intermediary CA and does not support HA.

Vault

For production systems, it is recommended to replace EasyRSA with the Vault charm. This uses HashiCorp’s Vault to provide either a root or intermediate CA. It can also be deployed HA, as well as provide a secure secrets store which can be used to enable encryption-at-rest for Charmed Kubernetes. However, it requires a database to store its data, and (depending on configuration) some manual steps will be required after deployment as well as after any reboot to unseal Vault so that the secrets, such as certificates and signing keys, can be accessed.

See the operations documentation for details on how to deploy Vault as a CA.