Security problem

Observability means that we are able to examine various metrics about the system and find security problems which we missed before going into production.
No matter how thorough your testing and pentesting process is, real world attacks are unpredictable and hard to prevent. The attacks that were unknown today might be uncovered tomorrow. With good observability you might be able to capture an attack before they are announced or, at least, detect anomalies that will lead to uncovering of an ongoing attack.

Security control proposal

Along with your application, deploy software to monitor and gather information from running systems. The gathered information must enable you to clearly describe what is going on in your application, so your team is able to uncover and reproduce the steps attacker already done. Automated alarms should be set to monitor for unusual activity.

Monitoring should also make sure to automatically check for newer versions of the libraries and code used by your application. In case newer versions of the library/code is available, you should get notification and upgrade as soon as possible. Tools that regularly check deployed instances for a configuration can further enhance the resilience of an application. Resources like memory or CPU usage can grow over time, so it’s a good practice to monitor them and periodically evaluate if the values are up-to-date with an existing requirement for a given project.

Example tools that can help you to observe the Kubernetes deployments:

• Popeye Kubernetes Cluster Sanitizer - detects K8S misconfigurations, able to identify deprecated API resources.
• Falco - native runtime security tool. Detect any behavior that involves making Linux system calls.