Skip to content

Module 07 — Secrets Management

Type 7 · Build-&-Operate — operate Vault and SOPS for runtime secret delivery behind a least-privilege read; the deliverable is the working secrets backend, not a credential in a config file. (Secondary: Decision/ADR — Vault vs SOPS vs cloud KMS.) Go to the hands-on lab →

Last reviewed: 2026-06

[Track 08 — Cryptography, PKI & Secrets] — Secrets in environment variables, config files, and source code are not secrets — they are liabilities waiting to be found.

Difficulty: Intermediate  ·  Estimated time: ~5–7 hrs (study + lab)  ·  Prerequisites: Foundations

In 60 seconds

The most careful TLS config is irrelevant if the database password sits in a .env committed to git. Secrets management solves the distribution and lifecycle problems that make credentials leak: centralised encrypted storage, policy-based access, audit logging, and rotation. Vault's dynamic secrets go further — the app gets a fresh, short-lived credential per connection and never holds a long-lived one; SOPS handles the secrets-that-must-live-in-git case by encrypting values while leaving keys readable. Uber 2016 (static AWS keys in a repo → 57M-user S3 bucket) is what these tools exist to prevent.

Why this matters

In 2016, attackers found AWS access keys hardcoded in a private GitHub repository belonging to an Uber engineer; those long-lived keys unlocked an S3 bucket holding personal data on 57 million riders and drivers (breaches.cloud — Uber incident analysis). The keys were static, broadly scoped, and stored alongside source — three failures secrets management exists to prevent. This is the gap between "we have good cryptography" and "our credentials are actually secure": the most careful TLS configuration is irrelevant if the database password is in a .env file committed to git, or if every developer has a copy of the production API key in their shell history. Secrets management tools — HashiCorp Vault and Mozilla SOPS are the open-source standard — solve the distribution and lifecycle problems that make secrets insecure: centralised storage, access control, audit logging, automatic rotation, and encryption at rest and in transit.

Objective

Use Vault's dev server to store, retrieve, and rotate a database credential, demonstrate the audit log, and use SOPS with an age key to encrypt a YAML configuration file — covering both the runtime secrets broker pattern and the secrets-in-git pattern.

The core idea

Vault's security model separates secrets storage from secrets access. Secrets are stored encrypted in Vault's backend (an encrypted database, a cloud KMS, or a hardware HSM). Access is controlled by policies — Vault tokens or identity-linked roles grant access to specific paths, not to all secrets. Every read, write, and rotation is written to the audit log. This combination — encrypted storage, policy-based access, comprehensive audit trail — is why Vault is the production standard for secrets management in cloud-native environments.

The mental model

The goal is to shrink what a leaked secret is worth, not just to hide it better. A static key in a vault is still a static key — steal it once and you own it indefinitely. A dynamic, short-lived, scoped credential is worth almost nothing the moment after it's issued. Reframe "where do I store the secret?" as "how short-lived and narrowly-scoped can I make it?"

The critical insight is that the secret itself never needs to leave Vault in most architectures. Dynamic secrets take this further: instead of storing a long-lived database password, Vault creates an ephemeral database credential on demand, with a short lease, and revokes it when the lease expires:

sequenceDiagram
    participant App as Corp app
    participant V as Vault
    participant DB as Database
    App->>V: authenticate (token / role)
    V->>DB: create short-lived user
    V->>App: ephemeral credential + lease
    App->>DB: connect with credential
    Note over V,DB: lease expires → Vault revokes the user

An application that uses dynamic secrets never holds a long-lived credential — if the application is compromised, the attacker gets a credential that expires in minutes. This is the opposite of a hardcoded password: the application gets a fresh, expiring credential every time it connects. Had Uber's S3 access been a short-lived dynamic credential rather than a static key checked into a repo, the 2016 leak would have yielded a credential already expired by the time it was found.

The gotcha

"We deployed Vault, so secrets are solved" is the trap — secrets management is a discipline, not a deployment. The leak vector that survives Vault is secret sprawl: copies accumulating in CI variables, developer shells, backups, and old config. Rotation, access auditing, and detection (module 08) are the ongoing work; the tool is just the place the canonical copy lives.

SOPS (Secrets Operations) solves a different problem: secrets that need to live in a git repository, such as Kubernetes secrets, Helm values files, or Terraform variable files. SOPS encrypts the values (not the keys) of a YAML, JSON, or dotenv file, using a key from Vault, AWS KMS, GCP KMS, or a local age key. The encrypted file can be committed to git — the keys are readable (so a reviewer can see what is being configured), the values are encrypted (so a compromised git clone doesn't yield live credentials). This is the "GitOps-safe secrets" pattern.

The common mistake organisations make is treating secrets management as a one-time deployment problem — "we set up Vault, now we're done." The ongoing discipline is secret rotation: Vault's leases automatically revoke dynamic secrets, but static secrets (API keys, TLS certificates, service account passwords) require an explicit rotation process. Secret sprawl — the gradual accumulation of copies of a secret across environments, developer machines, CI/CD systems, and backups — is the primary way secrets leak even in organisations that use Vault. The remediation is a combination of access auditing (who has retrieved this secret, and from where?), automatic rotation (so copies that do exist become stale quickly), and detection (module 08 scans for secrets that escaped the vault).

Go deeper: Vault vs SOPS vs cloud KMS — they solve different problems

These aren't competitors to rank; they target different deployment shapes. Vault is a runtime broker — best when you want dynamic, short-lived, audited credentials, at the cost of running and unsealing a stateful service. SOPS+age keeps secrets in the repo so GitOps stays git-only, encrypting values while leaving keys reviewable — but it gives you no rotation or audit trail of its own. Cloud KMS hands you managed envelope encryption at the cost of vendor lock-in. The decision follows your deployment model, and module 12 turns exactly this trade-off into an ADR.

AI caveat

An AI drafts a Vault policy HCL quickly, but policy bugs are silent — an over-broad path or an extra capability grants access nobody intended. Verify the generated capability list against the Vault policy docs and test it live: confirm a token with that policy can read but genuinely cannot write or delete.

Learn (~4 hrs)

HashiCorp Vault - Vault documentation — Getting Started — work through the "Your first secret" and "Dynamic secrets" tutorials (~1 hr); these two tutorials cover the core workflow. - Vault architecture documentation — read the storage backend, seal/unseal, and audit device sections; understand why Vault needs an unseal key before explaining it to others.

SOPS - mozilla/sops README (GitHub) — read the usage section and the "Encryption Protocol" section to understand how SOPS encrypts values while preserving key structure. - age encryption tool — read the README; age is the modern, simple alternative to GPG for SOPS key management.

Secrets management's real-world failure — the why (~15 min) - breaches.cloud — Uber 2014 & 2016 incidents — the 2016 breach: AWS access keys hardcoded in a private GitHub repo unlocked an S3 bucket with 57M users' data. Read it for the chain of failures — static keys, broad scope, secret-in-source — that Vault dynamic secrets and access policies exist to break.

Secrets management principles - CISA Alert AA22-137A — Weak Security Controls (Credentials section) — real-world context for why hardcoded credentials are exploited routinely; read the credentials section.

Key concepts

  • Vault: encrypted storage + policy-based access + audit log. Never store secrets in environment variables or config files committed to source control.
  • Dynamic secrets: Vault creates ephemeral credentials on demand; revokes them automatically on lease expiry. No long-lived credentials in the application.
  • SOPS: encrypts values (not keys) in structured files; git-safe secrets for Kubernetes/Terraform/Helm workflows.
  • Secret rotation: Vault automates dynamic secret revocation; static secrets require an explicit rotation process.
  • Secret sprawl: the accumulation of secret copies is the primary leak vector — audit access logs and rotate frequently.
  • Uber 2016 (static AWS keys hardcoded in a private GitHub repo → 57M-user S3 bucket) is the canonical case for dynamic, short-lived, scoped credentials over long-lived secrets in source.

AI acceleration

Ask an AI to generate a Vault policy HCL file that grants read access to secret/corp/app/* but not write or delete access. Then verify the policy against the Vault policy documentation — does the capability list match the intended permissions? Apply the policy and confirm a token with that policy can read but not write the secret.

Check yourself

  • Had Uber's S3 access been a Vault dynamic secret instead of a static key in a repo, why would the 2016 leak have been far less damaging?
  • SOPS lets you commit secrets to a git repo. What does it encrypt, what does it leave readable, and why is that split deliberate?
  • An org runs Vault but still suffers a credential leak. Name the most likely mechanism and the three controls that address it.

Comments

Sign in with GitHub to comment. Choose the type: Feedback (errors or suggestions on this page) · Hints (help for fellow learners — no spoilers) · General (anything else).