Every application needs configuration to communicate with other systems. A web API may require a database password, an authentication service may need a client secret, and a background worker may need access to an external storage account.
In Kubernetes, developers often use environment variables or configuration files to provide these values to containers. This works for ordinary application settings, but sensitive information requires additional protection.
Kubernetes provides a resource called Secret for storing confidential information such as passwords, tokens, certificates, and private keys. It separates sensitive configuration from application images and allows workloads to access only the information they need.
However, creating a Kubernetes Secret does not automatically make its contents secure. Secret values are Base64-encoded in their serialized representation, which is not encryption. Depending on cluster configuration, Secret data may also be stored unencrypted in etcd.
A poorly configured Secret can therefore expose credentials through deployment files, excessive permissions, application logs, or backups.
This article explains how Kubernetes Secrets work, how to provide them to applications, and how to manage access, encryption, and rotation without unnecessarily exposing sensitive configuration.
Understand What Kubernetes Secrets Actually Protect
A Kubernetes Secret is an API resource designed to hold confidential information.
For example, an application might need a database username and password:
apiVersion: v1
kind: Secret
metadata:
name: database-secret
namespace: production
type: Opaque
stringData:
username: appuser
password: replace-with-a-secure-password
The stringData field allows developers to provide ordinary text values when creating a Secret. Kubernetes converts these values into the Secret's data representation when storing the resource.
The important point is that Base64 encoding does not provide confidentiality. Anyone who can retrieve the Secret through the Kubernetes API may be able to decode its values.
For example, the following command retrieves a Secret:
kubectl get secret database-secret -n production -o yaml
The returned data is Base64-encoded, but it is not encrypted simply because it appears encoded.
This distinction matters when reviewing Kubernetes manifests, deployment pipelines, and cluster permissions.
Secrets should be treated as sensitive credentials throughout their lifecycle. They need protection when created, stored, accessed, mounted into containers, backed up, and eventually rotated or deleted.
Create Secrets Without Committing Credentials to Source Control
One of the most common mistakes is placing passwords directly inside a Secret YAML file and committing that file to a Git repository.
Although the file uses a Kubernetes Secret resource, its contents are still available to anyone who can access the repository.
For example, this approach is risky:
apiVersion: v1
kind: Secret
metadata:
name: database-secret
type: Opaque
stringData:
username: appuser
password: MyRealDatabasePassword
Even if the password is removed from the latest version of the file, it may remain in Git history.
A safer approach is to create the Secret from a protected file or provide its values through a controlled deployment process.
For example:
kubectl create secret generic database-secret \
--namespace production \
--from-file=username=./credentials/db-username.txt \
--from-file=password=./credentials/db-password.txt
The credential files should be stored securely, excluded from source control, and protected through appropriate filesystem permissions.
You can also create a Secret from environment-specific deployment tooling or an external secret manager.
For automated environments, avoid placing sensitive values directly in command-line arguments when possible. Command history, process information, build logs, and debugging output can expose values that were intended to remain private.
The objective is to ensure that credentials do not become part of the application repository or ordinary deployment logs.
Provide Secrets to Containers as Environment Variables
Kubernetes supports providing Secret values to containers through environment variables.
For example, an application can receive a database password using secretKeyRef:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-api
spec:
replicas: 2
selector:
matchLabels:
app: my-api
template:
metadata:
labels:
app: my-api
spec:
containers:
- name: my-api
image: my-api:latest
env:
- name: Database__Username
valueFrom:
secretKeyRef:
name: database-secret
key: username
- name: Database__Password
valueFrom:
secretKeyRef:
name: database-secret
key: password
The application receives the values through its environment.
For an ASP.NET Core application, the configuration system can read environment variables using the double-underscore naming convention.
For example, Database__Password maps to the Database:Password configuration key.
The application can retrieve the value through dependency injection:
var builder = WebApplication.CreateBuilder(args);
var connectionString =
builder.Configuration.GetConnectionString("DefaultConnection");
var databasePassword =
builder.Configuration["Database:Password"];
var app = builder.Build();
app.MapGet("/", () => "Application is running.");
app.Run();
In a real application, construct the database connection through the appropriate configuration or connection-string provider. Avoid printing the password or complete connection string in application logs.
Environment variables are convenient, but they have security considerations. They may be exposed through diagnostic output, crash dumps, application debugging tools, or accidental logging.
They also remain part of the container's process environment for the lifetime of that process.
For applications handling highly sensitive credentials, mounting Secret values as files may provide better control over access and rotation.
Mount Secrets as Files When Appropriate
Kubernetes can mount Secret values into a container as files.
This is useful when an application expects a certificate, private key, configuration file, or other credential in file format.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-api
spec:
replicas: 2
selector:
matchLabels:
app: my-api
template:
metadata:
labels:
app: my-api
spec:
containers:
- name: my-api
image: my-api:latest
volumeMounts:
- name: database-credentials
mountPath: /app/secrets
readOnly: true
volumes:
- name: database-credentials
secret:
secretName: database-secret
The Secret's keys are made available as files under the specified directory.
For example, the application can read the username and password from:
/app/secrets/username
/app/secrets/password
The volume mount is read-only, which helps prevent the application from modifying the mounted Secret files.
By default, Secret volumes are backed by the node's memory-backed filesystem mechanism rather than ordinary persistent storage. However, this should not be treated as a replacement for node security, access control, or encryption at rest.
Also, mounting a Secret does not mean every process inside the container should be able to access it. File ownership, permissions, container users, and application architecture still matter.
A useful practice is to mount each Secret only into the containers that require it rather than exposing every credential to every container in a Pod.
Restrict Access With Kubernetes RBAC
Protecting Secret values requires controlling who can retrieve them through the Kubernetes API.
Kubernetes Role-Based Access Control, commonly called RBAC, allows administrators to define which users and service accounts can perform specific operations on resources.
For example, a service account that only needs to retrieve one particular Secret should not automatically receive permission to read every Secret in the namespace.
A Role can restrict access to a named Secret:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: database-secret-reader
namespace: production
rules:
- apiGroups:
- ""
resources:
- secrets
resourceNames:
- database-secret
verbs:
- get
The corresponding RoleBinding connects the Role to a service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: database-secret-reader-binding
namespace: production
subjects:
- kind: ServiceAccount
name: my-api-service-account
namespace: production
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: database-secret-reader
This gives the specified service account permission to retrieve the named Secret.
There is an important security detail here. Permission to list or watch Secrets can expose their contents, not just their names. Those permissions should not be granted casually.
Also, anyone who can create workloads in a namespace may be able to create a Pod that mounts Secrets from that namespace. This means restricting direct Secret access alone is not enough. Permissions to create or modify Deployments, Pods, and other workload resources must also be considered.
Use separate service accounts for applications with different responsibilities, and grant only the permissions they actually require.
Enable Encryption at Rest for Secrets
Kubernetes stores API resources in etcd. By default, Secret data is not necessarily encrypted at rest.
This means that someone who obtains access to the underlying etcd data or its backups may be able to retrieve sensitive information, depending on the cluster configuration.
Kubernetes supports encryption at rest for API resources, including Secrets.
Cluster administrators can configure the API server to encrypt Secret data before storing it in etcd.
For production environments, using a managed Key Management Service can provide additional protection by keeping encryption keys outside the cluster.
The encryption process should also include a key rotation strategy. Encryption configuration must be managed carefully because losing the required keys can make stored data unreadable.
Enabling encryption for new writes is not always enough. Existing Secret data may need to be rewritten so that it is stored using the configured encryption provider.
For managed Kubernetes services, check the provider's supported encryption capabilities and configuration requirements.
Encryption at rest protects stored data, but it does not replace RBAC. An authorized user who can retrieve a Secret through the Kubernetes API can still receive its decrypted value.
Both controls are necessary because they address different risks.
Use External Secret Managers for Sensitive Production Credentials
For larger applications, managing every credential directly through Kubernetes manifests and manually created Secret resources can become difficult.
External secret managers provide another approach.
Examples include cloud key management and secret storage services, as well as dedicated secret management platforms.
An external secret manager can store credentials centrally and provide controlled access to applications or deployment systems.
Kubernetes integrations can retrieve values from these systems and make them available to workloads. One option is the Secrets Store CSI Driver, which supports mounting secrets from external providers into Pods.
This approach can reduce the need to store long-lived credentials directly in application repositories or deployment configuration.
For example, an application might retrieve a database password from a managed secret store when its Pod starts.
The exact configuration depends on the cloud provider, authentication mechanism, and integration being used.
Workload identity is often preferable to storing another long-lived credential inside the Pod just to access the external secret manager.
The application should authenticate using an identity assigned to its workload, with permission limited to the required secret.
External secret managers also introduce operational considerations. Teams must plan for provider availability, authentication failures, synchronization behavior, and credential rotation.
The integration should be tested before it becomes a critical dependency for application startup.
Plan Secret Rotation Without Breaking Applications
Credentials should not remain valid indefinitely.
Database passwords, API tokens, certificates, and other sensitive values should have a rotation process that allows them to be replaced without unnecessarily interrupting application availability.
A common problem occurs when a Secret is updated but the application continues using the old value.
Kubernetes handles Secret updates differently depending on how the Secret is consumed.
Secret values injected as environment variables do not automatically change inside an already running container. The application normally needs to be restarted to receive the updated environment.
Secret volumes can receive updated values eventually, but an application that reads a file only once during startup may continue using the old value until it reloads the file or restarts.
A rotation process should therefore include both the credential update and the application refresh strategy.
For example, a controlled rollout can restart the Deployment after updating the Secret:
kubectl rollout restart deployment/my-api \
-n production
Monitor the rollout:
kubectl rollout status deployment/my-api \
-n production
For database credentials, consider a rotation strategy that allows both the old and new credentials to work temporarily. This gives the application time to restart and establish new connections before the old credential is disabled.
The rotation process should also include validation. Confirm that new Pods can authenticate successfully before removing the previous credential.
Common Mistakes When Managing Kubernetes Secrets
Mistake | Why it creates problems | Better approach |
|---|
Assuming Base64 encoding is encryption | Encoded values can be decoded easily | Enable encryption at rest |
Committing Secret manifests containing real credentials | Git history can retain sensitive values | Use secure deployment workflows |
Granting broad Secret permissions | Users or workloads may retrieve unrelated credentials | Apply least-privilege RBAC |
Exposing all Secrets through environment variables | Environment values can appear in diagnostics and logs | Consider restricted file mounts |
Using the same service account everywhere | Unrelated workloads receive unnecessary access | Create dedicated service accounts |
Updating a Secret without refreshing the application | Existing processes may continue using old values | Plan application reload or restart |
Logging connection strings or tokens | Credentials may enter centralized logging systems | Redact sensitive configuration |
Ignoring backup security | Secret data may be exposed through etcd backups | Encrypt and restrict backup access |
Best Practices for Production Kubernetes Secrets
Secure Secret management should be part of application deployment and cluster administration rather than an isolated configuration task.
Use Kubernetes Secrets for confidential values and ConfigMaps for non-sensitive configuration.
Never treat Base64 encoding as encryption.
Avoid committing real credentials to Git repositories.
Restrict Secret access through least-privilege RBAC.
Review permissions to create or modify workloads, not only direct Secret permissions.
Enable encryption at rest for Secret data.
Use external secret managers when centralized credential management is required.
Prefer workload identity over storing additional long-lived access credentials.
Mount sensitive files only into containers that need them.
Avoid exposing credentials through logs, debugging output, and diagnostic tools.
Define a rotation process that includes application refresh and validation.
Protect etcd backups and encryption keys with appropriate access controls.
These practices help reduce accidental exposure and make credential management more consistent across environments.
Summary
Kubernetes Secrets provide a standard way to manage passwords, tokens, certificates, and other confidential application configuration. However, creating a Secret is only the beginning of securing that information.
Base64 encoding does not encrypt Secret values, and access to the Kubernetes API or etcd can expose sensitive data if the cluster is not configured correctly.
A secure implementation combines carefully controlled Secret creation, least-privilege RBAC, encryption at rest, restricted workload access, and a reliable rotation process.
For applications with more demanding security requirements, integrating an external secret manager can provide centralized storage and stronger credential lifecycle management.
The most important principle is to treat credentials as sensitive throughout their entire lifecycle. Protecting a Secret means controlling who can create it, who can retrieve it, which containers can use it, where it is stored, and how it is replaced when it needs to change.
Join the conversation! Your thoughts help the community grow.