As we discussed in a previous article, enterprise workloads are increasingly being deployed to and orchestrated by Kubernetes. Since many Kubernetes based workloads require access to secured resources, and authentication to those resources requires the use of secrets, properly managing secrets has become a critical aspect of enterprise security and agility. Kubernetes natively offers options to handle secrets but utilizing them correctly represents some real challenges. In this article, you will learn about some of the challenges of properly managing secrets in Kubernetes. Along the way, we’ll describe several real-world attacks due to improper secrets management and what current industry best-practices exist to prevent them.

Challenges

Kubernetes has transformed how organizations manage applications, but secrets management remains a persistent challenge for many teams. In a typical Kubernetes cluster, secrets like API keys, tokens, and database credentials are often critical for operations, but improper management can expose sensitive data and increase the risk of security breaches.

A key problem teams face is how to securely manage secrets across environments and within source control. It’s not uncommon to see secrets stored directly in Git repositories, especially in early-stage development. This convenience, however, carries significant risk: anyone with access to that repository could potentially misuse those secrets. Similarly, storing secrets directly in Kubernetes without appropriate safeguards can make them vulnerable to compromise because Kubernetes secrets are stored as base64 encoded text - which is not a form of encryption.

Another challenge is secrets rotation. When secrets need to be updated—whether due to a breach or simply following best practices for periodic rotation—the process can be cumbersome. Ensuring all pods in a cluster get the updated secret without downtime is complex and often requires specialized tooling.

Real World Consequences of Mismanaging Secrets

The consequences of poor secrets management can be severe. Here are a few real-world examples that illustrate how weaknesses in managing Kubernetes secrets can lead to serious issues. Note that these are the types of incidents we help our customers prevent.

1. 1. Credentials Exposed through Source Control - In one incident, a development team accidentally committed Kubernetes manifests containing production database credentials to a public Git repository. Attackers found these credentials, gaining access to the production database and compromising sensitive customer data.
2. 2. Misconfigured Kubernetes Secrets - Another case involved a company storing API keys in Kubernetes secrets without proper access control. The secrets were accessible to more pods than necessary, due to overly permissive Role-Based Access Control (RBAC) settings. An attacker exploited a vulnerability in one of the pods to access these secrets, allowing them to escalate privileges and move laterally within the cluster.
3. 3. Secrets Stored Unencrypted in etcd - Kubernetes stores secrets in etcd, which, by default, is unencrypted. In a notable breach, attackers gained access to etcd and were able to retrieve all secrets stored in the cluster, including database credentials and private keys. Encrypting etcd at rest and using an external secrets management solution could have significantly reduced the impact of this attack.
4. 4. Lack of Secrets Rotation Leading to Credential Reuse - A company experienced a data breach when an old, compromised credential was reused. The secret had not been rotated for years, making it an easy target for attackers who found it through other means. Automated secrets rotation using tools like External Secrets Operator or Hashicorp Vault could have ensured that stale credentials were replaced regularly, reducing the risk of this kind of attack.

Now that you have an idea of what some attack vectors look like and how serious the consequences can be, let’s dive into some best practices that can be used to mitigate risk.

Best Practices

The resources and security posture of each organization are different but the need to maintain confidentiality, integrity and availability of sensitive information is universal. While there are numerous solutions and techniques in use to address this problem, we recommend the following best practices:

1. 1. Shift Left - Never commit plain-text credentials to source control. Add automated checks such as pre-commit hooks, pull/merge request scans, and static code analysis tools such as SonarQube or Trufflehog to flag and prevent this. Git Secrets can be used to prevent secrets being added during commits. If secrets directly in source code are desired, considering using a tool like SOPS which would allow for committing of the cyphertext which can be decrypted later with a key. The key must be managed outside of source control though and not committed. SOPS supports common enterprise key vault solutions and cloud native ones like AWS Secrets Manager.
2. 2. Role Based Access Control (RBAC) - Secrets access at the Kubernetes API level is governed by RBAC. Preventing excessive or unintended access is paramount to ensuring proper control of secrets. Apply least privilege principles with properly configured service accounts to ensure narrowly scoped access.
3. 3. Monitoring - Access to secrets should be continually monitored through audit logging such that all access is logged and tracked. Kubernetes has native support for audit logging. Most popular SIEM tools support ingesting Kubernetes audit logs as well. Here is a useful code snippet that can be used to list any roles within a cluster that have access to secrets.
4. 4. Rotation - Secrets rotation is the process of periodically updating secrets, such as passwords, API keys, and certificates, to reduce the risk of credential compromise. In Kubernetes, this can be achieved through automated solutions that help maintain security without service interruption. External Secrets Operator is a proven offering in this space. It provides an abstraction layer that allows you to integrate secrets from cloud secrets stores like AWS Secrets Manager or Azure Key Vault without having to manage them directly within Kubernetes manifests. This means you can keep your source of truth outside of Kubernetes while still leveraging Kubernetes-native access controls. Hashicorp Vault can also be leveraged to deploy a secrets store in on-prem, hybrid, or cloud environments. It has native integration points into Kubernetes.
5. 5. Encryption - By default, Kubernetes does not encrypt secrets in etcd. Encrypting etcd at rest is an essential step to secure Kubernetes secrets from unauthorized access. Here are the basic steps to enable encryption for etcd in your Kubernetes cluster.

How BridgePhase Simplifies Secrets Management

At BridgePhase we build and operate hardened Kubernetes platforms. Our work on Big Bang, which is a DevSecOps platform utilized throughout the DoD, leverages our expertise in Kubernetes secrets management. Big Bang supports out of the box secrets management capabilities. We have implemented the techniques described above to deliver a platform that is utilized in numerous mission environments where security is critical. Our solutions provide flexible, compliance focused ways to meet the unique challenges of highly secure environments.

Closing Remarks

Managing secrets in Kubernetes is a complex but vital aspect of enterprise security. Solutions like Vault, the External Secrets Operator, and SOPS provide different approaches that facilitate various workflows - whether dynamic secret management, GitOps, or cloud-native integrations. Whichever solution you choose, prioritizing the secure handling, rotation, and storage of secrets should be a core part of your Kubernetes strategy. Lastly, secrets management is only one aspect of a comprehensive DevSecOps implementation. Follow us on LinkedIn to be notified of future blog posts where we will discuss how to implement organizational processes and governance that make up the other pillars of secure software development and operations.