Language:English VersionChinese Version

Environment variables are the default secret management solution for most applications, and they are inadequate for most production systems. They leak into process listings, get committed to version control in .env files, appear in crash reports and debug logs, and cannot be rotated without redeployment. Yet many teams running serious production workloads still manage secrets this way, primarily because better alternatives feel complex to adopt.

This article surveys the real options — HashiCorp Vault, Mozilla SOPS, cloud-native secret managers, and the emerging sealed secrets pattern — with enough implementation detail to make a genuine comparison. The goal is to give you a clear decision framework rather than a list of tools.

Why Environment Variables Fail at Scale

The specific failure modes of environment variables are worth naming precisely before discussing alternatives:

  • Process listing exposure: On Linux, /proc/<pid>/environ is readable by users with the appropriate permissions. In shared hosting or multi-tenant environments, environment variables from one process can potentially be read by others.
  • Log leakage: Exception handlers, debugging middleware, and monitoring agents frequently dump process environment as part of error context. One misconfigured error tracker and your database password is in your logging system.
  • .env file incidents: The number of times .env files have been committed to public GitHub repositories and crawled by credential scanners is not trivial. GitGuardian’s 2025 report found over 10 million new secrets committed to public repositories in 2024.
  • No rotation without redeployment: Rotating a database password means updating the environment variable everywhere it’s used — every Kubernetes pod, every Lambda function, every EC2 instance — and redeploying. This friction means rotation happens infrequently, which means compromised credentials stay valid longer.
  • No audit trail: There is no built-in record of who accessed which secret, when, and from which system. This is a compliance problem for any regulated workload.

The Four-Tier Secret Management Landscape

Secret management solutions sit at four broad tiers of sophistication:

  1. Encrypted files in version control (SOPS): Secrets are encrypted and stored alongside code. Simple, auditable, works without external infrastructure.
  2. Cloud-native secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault): Managed services with IAM-based access control, automatic rotation, and audit logging. Best choice if you’re already committed to a cloud provider.
  3. Self-hosted secret manager (HashiCorp Vault): Full-featured, provider-agnostic, with dynamic secrets, PKI management, and rich access policies. High operational overhead.
  4. GitOps-native sealed secrets (Sealed Secrets, External Secrets Operator): Kubernetes-native patterns that integrate secret management into GitOps workflows.

SOPS: Encrypted Secrets in Version Control

Mozilla SOPS (Secrets OPerationS) encrypts secret files using AWS KMS, GCP KMS, Azure Key Vault, or age keys. The encrypted file lives in your git repository. Decryption happens at deployment time using cloud IAM credentials or an age private key.

The key insight: SOPS encrypts only the values in a YAML/JSON/ENV file, not the keys. This means diffs are readable and meaningful in code review — you can see that a key was added or removed without seeing its value.

# Install SOPS
brew install sops   # macOS
# or: apt install sops / download from GitHub releases

# Create an age key (simplest setup for small teams)
age-keygen -o ~/.config/sops/age/keys.txt
# Created: /Users/you/.config/sops/age/keys.txt
# Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

# .sops.yaml — configure which files use which keys
creation_rules:
  - path_regex: secrets/.*\.yaml$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
  - path_regex: secrets/prod/.*\.yaml$
    # Production uses AWS KMS for better key management
    kms: arn:aws:kms:us-east-1:123456789:key/mrk-abc123

# Encrypt a secrets file
cat > secrets/database.yaml <<EOF
database_url: postgres://user:password@host:5432/db
api_key: sk-live-abc123def456
jwt_secret: very-secret-signing-key
EOF

sops --encrypt --in-place secrets/database.yaml

# The encrypted file is safe to commit — values are encrypted, keys are visible
# Decrypt at deployment time (requires age key in environment)
sops --decrypt secrets/database.yaml | envsubst

SOPS shines for teams that want secrets in version control with the same Git workflows as code — PRs, review, history. The limitation is that it requires the decryption key to be present wherever deployment happens, and there’s no built-in access audit at the secret level (only at the KMS key level).

AWS Secrets Manager: The Right Default for AWS Shops

If your infrastructure runs on AWS, Secrets Manager is the pragmatic choice. It handles automatic rotation for RDS databases natively, integrates with IAM for fine-grained access control, and costs $0.40/secret/month — cheap enough that cost is not a factor for typical applications.

# Store a secret
aws secretsmanager create-secret \
  --name "prod/myapp/database" \
  --secret-string '{"username":"myapp","password":"s3cr3t","host":"db.example.com"}'

# Enable automatic rotation for an RDS secret
aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/database" \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123:function:SecretsManagerRDSRotation \
  --rotation-rules AutomaticallyAfterDays=30

# Retrieve in application code — no environment variable needed
import boto3
import json

def get_db_credentials() -> dict:
    client = boto3.client('secretsmanager', region_name='us-east-1')
    response = client.get_secret_value(SecretId='prod/myapp/database')
    return json.loads(response['SecretString'])

# For Kubernetes: use External Secrets Operator to sync to K8s Secrets
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
  - secretKey: DATABASE_URL
    remoteRef:
      key: prod/myapp/database
      property: connection_string

The External Secrets Operator pattern is important: it syncs cloud secret manager values into Kubernetes Secrets automatically, rotating them on a schedule. Your pods reference a Kubernetes Secret; the operator handles keeping it current.

HashiCorp Vault: When You Need Provider Independence

Vault is warranted when you need capabilities beyond what cloud-native managers provide: dynamic secrets (credentials generated on-demand with a TTL, automatically revoked after use), multi-cloud secret federation, PKI certificate management, or KMIP for database encryption key management.

The feature that most teams don’t fully use is dynamic secrets. Instead of storing a database password that gets rotated periodically, Vault generates a unique database credential for each service instance with a configurable TTL:

# Configure Vault database secrets engine for PostgreSQL
vault secrets enable database

vault write database/config/postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="app-role" \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/app_db" \
    username="vault_admin" \
    password="vault_admin_password"

vault write database/roles/app-role \
    db_name=postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# Each service request generates a unique credential valid for 1 hour
vault read database/creds/app-role
# Key                Value
# ---                -----
# lease_id           database/creds/app-role/Hyu1...
# lease_duration     1h
# password           A1a-generated-unique-password
# username           v-app-role-xyz123abc

With dynamic secrets, a compromised credential is automatically invalid after its TTL. There’s no password to rotate because every credential is temporary by design. This is a fundamentally stronger security posture than periodic rotation of static passwords.

The operational cost of Vault is real: you need a highly-available Vault cluster (minimum 3 nodes), regular unsealing procedures, monitoring, and Vault-specific operational expertise. For most teams that don’t require dynamic secrets or multi-cloud support, cloud-native managers are lower cost for equivalent static secret management.

Kubernetes Sealed Secrets for GitOps

Sealed Secrets (from Bitnami/VMware) solves the specific problem of storing Kubernetes Secrets safely in Git. It uses a controller running in the cluster that holds a private key. You encrypt your secret with the corresponding public key into a SealedSecret custom resource, which is safe to commit. Only the cluster’s controller can decrypt it.

# Install Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Get the cluster's public key
kubeseal --fetch-cert > pub-cert.pem

# Create a Kubernetes Secret and seal it
kubectl create secret generic my-app-secret \
  --from-literal=api_key=sk-live-abc123 \
  --dry-run=client -o yaml | \
  kubeseal --cert pub-cert.pem --format yaml > sealed-secret.yaml

# sealed-secret.yaml is safe to commit to git
# Apply it to the cluster — controller decrypts and creates the real Secret
kubectl apply -f sealed-secret.yaml

Sealed Secrets is the right choice for pure GitOps workflows where you want all cluster state (including secrets) defined in Git. The limitation is that the sealed secret is tied to a specific cluster’s private key — you can’t use the same sealed secret in multiple clusters without re-sealing for each.

A Decision Matrix

Scenario Recommended Approach
Small team, single cloud (AWS) AWS Secrets Manager + External Secrets Operator
GitOps workflow, Kubernetes Sealed Secrets or External Secrets Operator
Multi-cloud or on-premises HashiCorp Vault
Secrets need to live in Git SOPS with age or KMS
High-security, dynamic credentials needed HashiCorp Vault dynamic secrets
Maximum simplicity, small project SOPS with age key

Migration Path from Environment Variables

Moving an existing application off environment variables is a phased process:

  1. Audit: Find every secret in your environment variables. env | grep -iE 'key|secret|password|token|credential' is a useful starting point.
  2. Classify: Separate secrets (need rotation, audit trail) from configuration (database host, log level) — not everything in environment variables is a secret.
  3. Choose your store: Based on the matrix above.
  4. Abstract retrieval: Add a secret retrieval layer to your application that reads from the new store. Keep the env var fallback during transition for local development.
  5. Rotate immediately after migration: The first thing to do after migrating a secret to a proper store is rotate it. Assume the environment variable version was compromised.

Conclusion

Environment variables are a convenience, not a security control. The solutions exist, they are mature, and most of them are not as operationally complex as their reputations suggest. AWS Secrets Manager requires almost no operational overhead for AWS users. SOPS requires only a key file and a one-time configuration. Sealed Secrets adds a Helm chart and a CLI tool to your existing Kubernetes workflow.

The cost of a secrets incident — credential compromise, data breach, compliance violation — consistently exceeds the investment in proper secret management by orders of magnitude. Pick the right tool for your infrastructure tier and migrate your highest-value secrets first. The operational patterns are well-understood; the risk of inaction is not hypothetical.

By Michael Sun

Founder and Editor-in-Chief of NovVista. Software engineer with hands-on experience in cloud infrastructure, full-stack development, and DevOps. Writes about AI tools, developer workflows, server architecture, and the practical side of technology. Based in China.

Leave a Reply

Your email address will not be published. Required fields are marked *