Security problem
Even minimal operating system images that are used in deployment can contain lots of applications and complex environment which is, in practice, hard to verify even automatically.
Various mechanisms can be leveraged to be sure that semi-automatically produced images are compliant with the company policies, such as the Security & Compliance Automation Ansible Roles created by the DevSecOps team of the Deutsche Telekom IT GmbH.
However, the question on how to verify an OS image pulled by the Kubernetes orchestrator or the OpenStack Glance image service component is hard to answer straightforwardly.
Security control proposal
Kubernetes deployments
When Docker operating system images are used with Kubernetes, Kubernetes orchestrator automatically uses Docker software support verification of Docker images by leveraging Docker Content Trust for signature verification of the images that contain a digital signature. However, it’s still not clear how this can be transparently used in orchestrator system like the Kubernetes. In the time of this writing, it’s still an open issue in a Kubernetes Github about the ability to verify OS image signatures.
The most straightforward solution for this problem can be open source software Connaisseur. Connaisseur is an admission controller for Kubernetes that enables Image Signature Verification over the images pulled and deployed by the orchestrator. It leverages existing Docker Content Trust and Notary solutions, which make it not suitable for other types of container images, e.g. Kata Containers.
The alternative approach is to use cryptographic hash instead of a tag when defining OS image for suitable for a deployment. Image definition then changes from image:tag to image@sha256:a23c412....
In that way, we can be sure, that deployed image is the same as we do expect. Problems with this approach are, that we always have to know the suitable SHA256 hash and we can’t leverage just a 3rd party signing key for our deployments.
Sign and verify OS image signatures in the OpenStack
Several features related to image signing are now available in the OpenStack.
OpenStack since Mitaka release supports operating system image signature verification. the image service can verify these signed images, and, to provide a full chain of trust, the compute service has the option to perform image signature verification prior to image boot. The signature verification has to be enabled in the OpenStack Nova component configuration. The whole process can be found in the OpenStack Operations Guide.
The high-level steps are:
- Obtain the operating system image file
- Sign image using the certificate
- Upload certificate to the Castellan OpenStack key manager
- Upload image and the signature to OpenStack
- Enable Glance signature verification in the configuration
- Signature is verified during the instance boot