Auditing of container images

Security problem

A common problem with every software stack that reach production usage is how to make sure that operating system and its components that are always up-to-date or at least updated in a short time after new publicly known vulnerabilities are found. With the virtual machine and the container deployment, it should be easier to redeploy upgraded operating system images given that the system is designed properly.
However, as the reality shows, this is not always the case, hence we propose the following recommendations to harden the production deployments.

Security control proposal

1. Don’t use vulnerable images

Images with critical CVEs should NOT be allowed for deployment. These Images must be blocked from deployment by the Registry. Exact time between the date when the vulnerability is published and time of upgrade deployment depends on the severity of a vulnerability. Of course, the best is to have preventive measures in places as soon as possible and have deployment fixed as soon as there is a patch available.

Deutsche Telekom requirements for containers and orchestrators (PSA) can be found on Deutsche Telekom PSA-portal webpage.

Since there is no concept requirement for tagging within the document, we define tagging scheme as follows:

  • If you are using containers in a production environment (such as Kubernetes),Use immutable tags. This ensures that your deployment is not affected if a new revision inadvertently breaks existing functionality.
  • If you are using containers for development, use rolling tags. This ensures that you are always using the latest version. Rolling tags also make it easier to use a specific version of a development tool (such as bitnami/3-debian-9 for Redmine software version 3 for Debian Linux version 9).

For more information on a concept of immutable and rolling tags, please see Understand Bitnami’s Rolling Tags for Container Images article.

2. Running Containers must be updated frequently

Running containers must be reprovisioned with the latest stable release at least every 14 days with updated images. As this is hard to enforce technically, each application or project is responsible for running their own containers and should have a process in place to make sure the images are up to date.

Based on how image building and deployment process is defined, we do recommend to use the latest stable base image for each project, updated at least every month.
Please note that the latest refers to a specific version of an image, not the actual “latest” tag, which should not be used in production.

3. Image vulnerabilities must be reported and documented, risk management for unfixed vulnerabilities must be provided

We recommend to have a process on how to obtain a specific security exception in place.
It’s understandable, that there is not always a fix for a vulnerability, so you may have to allow it to be deployed into production. However, proper risk and possible impact evaluation have to be made accepting the risk of using unpatched system in production.
That’s why vulnerability scanning and compliance assessment on a production systems are critical as well.

Potential negative impacts of strict image auditing

Potential negative impacts of image auditing that must be considered:

  • Vulnerability scanners like Clair can have many false positives.
  • It’s hard to this with latest images being prohibited in production.
  • Building images from scratch is difficult for some projects as they can be just “provided”.
  • There is no easy way to scan images during the runtime.
  • It’s not easy to prohibit or untrust the out-of-date images from usage.

Practical implementation

There is a plenty of tools that can be leveraged for image and infrastructure definition scanning:

  • DefectDojo for tracking, aggregating and managing vulnerabilities
  • Krane - Kubernetes RBAC static analysis & visualization tool
  • Helm plugin for Snyk - checking images in Helm charts for vulnerabilities