Introduction

When delivering a service, we want to be swift, have short time to market, be secure and compliant.

CI/CD focus on tooling, DevOps focus on culture. Agile or Scrum are focused on organizing and managing the work to be done.

The main pillars

There are 6 main pillars of the CI/CD security context: IaC, Development and Build, Automated Testing, Deployment, Runtime and Collaboration. Within the IaC pillar, we discuss how to prepare infrastucture where code runs, how network, storage and compute are dealt here. Development and Build deals with programming and compiling, where unit testing is also in place. Automated Testing consits of Integration and E2E testing. Deployment deals with Artefact and Configuration Management and deployment. Runtime pillar includes orcherstration, monitoring/Ops incident management and loggin. Finally, Collaboration pillar deals with Issue/Bug Tracking, ChatOps and Documentation.

Security Tooling in a CI/CD Context

Security Tooling deals with the whole lifecycle withing CI and CD. In the CI lifecycle there are 7 stages: Analyse, Design, Code, Unit Test, Integration, Integration Test and Release. CD lifecycle containts the following stages: Verify, Pack and Ship, where underlying stages are Build Infrastructure, Deploye Stage, E2E Test, Deploy Prod and Observe

CI Dilemmas & How we approached it

Dilemma: Analysis & Design

Within the Anylisis&Design stage, we define the dilemma where some design decissions have fundamtental mismatch with general DT Group Security requirements and we do propose the solution for it.

Some Design decisions can have fundamental mismatch with general DT Group Security requirements

Proposed solution: Analysis & Design

Proposed solution for dilemma in Analysis&Design - discussions to be raised if any design mismatch.

  • Discussions are open (in Slack channels or via Gitlab Issues) and accessible/transparent
  • Since designs and architecture are part of the repository (documentation), adding or changing architectures/design goes through a review process. We expect senior engineers to be aware of Group Policies and raise discussions in case something is not properly done/designed.

Dilemma: Code

Typical problems within the Code stage: dependencies, exception handling, code smells.

Typical problems with code are (not limited to):

  • Code Smells
  • Dependencies (inherited vulnerabilities)
  • Exception handling

Proposed solution: Code

Proposed solution: using SonarQube, Limiting and use of tools.

  • We use SonarQube to identify and report Security Issues
  • Linting and use of tools to identify vulnerable libraries
  • Proposal:
    • Expand/formalize existing Security Rules
    • Create specific rules to match Group Policies
    • Specially if the rules match policies such as usage ofdeprecated methods, functions or standards (e.g. crypto algorithms)

Dilemma: Unit Testing

Input validation is perhaps the lowest hanging fruit for security at this level – and that can be dealt with at unit test level. Encoding and escaping are also on the list.

Input validation is perhaps the lowest hanging fruit for security at this level – and that can be dealt with at unit test level. Encoding and escaping are also on the list.

Proposed solution: Unit Testing

Proposal: Reference materials and snippets to raise awareness and Highlight use of language specific, battle tested tools: coverage, pytest, go test,etc.

  • This is not centrally managed, every squad is responsible to develop their own unit tests related to security (e.g. input validation, etc.)
  • Proposal:
    • Reference materials and snippets to raise awareness
    • Highlight use of language specific, battle tested tools: coverage, pytest, go test,etc. (community effort)

STATUS: There are several projects in Pan-Net with same level of testing doing suitable tests: idp-api,safes, pdns-zoner

Dilemma: Integration Test

Within the Integration Test stage, the most common problem here is security versus versioning and feature of dependencies.

Perhaps the most common problem here is security versus versioning and feature of dependencies. Teams should be aware that they are using third-party libraries and those libraries are safe to be used (don’t contain known vulnerabilities)

Proposed solution: Integration Test

Proposal for the Integration Test dilemma would be to create baseline framework with examples.

  • This is not centrally managed, every squad is responsible to develop their security test framework.
  • Proposal: create a baseline framework with examples

STATUS: Currently not widespread in Pan-Net

CD Dilemmas & How we approached it

Dilemma: Binary Verification

Make sure binarries signature are verified or at least do an integrity check. Binary signing can be done in several ways and will depend on the tools available in the environment. The dilemma here lies in low verification of binary signatures and security checks.

Proposed solution: Binary Verification

Make sure binaries are signed or at least do an integrity check STATUS: Currently not widespread in Pan-Net

Solution is to verify the binary signatures and integrity checks.

Dilemma: Bin Packing

A binary is most often not alone in an environment. The environment usually contains runtime dependencies and other tooling or OS which could increase the attack surface.

Dilemma lies in runtime dependencies which could increase the attack surface.

Proposed solution: Bin Packing

  • Make sure images have a static validation process against known vulnerabilities after built
  • Whenever possible use distroless-like images

STATUS: Docker images scan on build time (using anchore or clair) and distroless images are readily available.

The solution is to make sure images have a static validation process against unknown vulnerabilities after check and to use dostroless-like images.

Dilemma: Shipping artifact

The most common artifact is image, specially docker images. They may be safe right after built, but later in time they can become vulnerable (because a vulnerability was discovered later).

Shipiing Artifact dillema is the vulnerability of docker images.

Proposed solution: Shipping Artifact

Continuous Security on artifacts is a must: they must be periodically tested, scanned, and if vulnerabilities are found, projects/developers must be warned.

STATUS: No central solution ATM (PoC X-Ray via Jfrog being done)

The proposed solution would be to periodically test artifacts.

Dilemma: IaC

IaC dilemma deals with hard coded values in the infrastructure environment.

When building infrastructure sometimes engineers will hard code values (access keys, ssh private keys) or not fulfill corporate policies (e.g. use encrypted S3 buckets).It may also be related to missing network control mechanisms (e.g. security groups).

Proposed solution: IaC

The proposed solution for this would be to implement compliance check before pushing/applying pipeline.

Compliance check on existing IaC code before pushing / applying (in-line pipeline) should match specified compliance norms.

STATUS: Not widespread in Pan-Net

PROPOSAL: Use static analysis tools for IaC code (e.g. checkov, terraform-compliance);example 1, 2.

Dilemma: Deployment (stage)

Deployment stage is sometimes not close to the production environment.

This environment is known to be “as close as possible to production environment”.

Proposed solution: Deployment (stage)

The proposed solution is to make sure the stage environment is correctly parametrized and ready to be deployed in production. Also we need to check if all paswords and sensitive information are actually stored outside manifests, if secrets are hardcoded somewhere and if there’re tests for offensive and intrusive security.

This is the ideal place for a full check up on some compliance requirements:

  • Make sure this stage is properly parametrized and ready to be deployed into production (there should be zero
  • difference between stage and production, apart from some names/strings)
  • Are all password and sensitive information stored outside manifests in a secure storage (e.g. vault)?
  • Are there any hard-coded secret somewhere?
  • Offensive security and more intrusive security tests (like penetration testing)

Dilemma: Deploy (Prod)

Deployment in Production stage’s dilemma is obvious: This environment is considered “too important to play with”.

This environment is considered “too important to play with” (both on data it holds as well as its stability / availability).

Proposed solution: Deploy (Prod)

As a solution, we need to perform compliance and ready-only tasks.

Perform compliance and read-only tasks:

  • Are all password and sensitive information stored outside manifests in a secure storage?
  • Are there any hard-coded secret somewhere?
  • Monitor critical security components (e.g. SSL certificates)

Updated on 08 Jul 2020