Zero Trust, Secrets & Supply Chain

A vulnerable image-parsing library lets an attacker run code inside one search pod. Now replay the default setup. From that pod, the attacker calls order with the header X-Account-Id: acc-42 and reads Ana's orders, because "only the gateway can reach order". They read the environment variables and find DATABASE_URL with the one password every service shares. They find PSP_API_KEY, because the payment key was put in a ConfigMap "temporarily". They open a connection to https://attacker.example and upload everything, because the private subnet has a NAT route and nothing restricts egress. Every step was possible because something trusted the network.

This page is about the architecture that removes that trust — Zero Trust — and about two places where trust hides in every system: secrets and the software supply chain.

From perimeter to Zero Trust

The perimeter model ("castle and moat") protects the edge — firewall, VPN — and trusts everything inside. Once an attacker is inside, through a stolen laptop, a phished password or a vulnerable dependency, they move from machine to machine: lateral movement.

Zero Trust removes the idea of a trusted network. Its motto is "never trust, always verify": every request — from the Internet, from the office, from the pod next door — is authenticated, authorized and encrypted on its own merits¹. Google built its corporate network this way in the BeyondCorp project after 2011²; NIST standardized the architecture in SP 800-207¹.

flowchart LR
    subgraph perimeter ["Perimeter model"]
        fw["Firewall / VPN"] -->|"inside = trusted"| int["search, order, payment, DB<br/>(trust each other)"]
    end
    subgraph zt ["Zero Trust model"]
        any["Any request<br/>(user or workload)"] -->|"identity + context"| pep["Policy Enforcement Point<br/>(gateway, mesh proxy)"]
        pep -->|"ask"| pdp["Policy Decision Point<br/>(policies, identity, device, risk)"]
        pdp -->|"allow, least privilege"| res["Resource"]
        pdp -->|"deny"| rej["Rejected + logged"]
    end

NIST names the moving parts: a Policy Enforcement Point (PEP) sits in the path of every request and blocks or forwards it; a Policy Decision Point (PDP) decides from identity and context. In BoxOffice, the API gateway is the PEP for users, and the mesh proxy next to each service is the PEP for workloads (Service mesh).

Principles

The compromised search pod, again

The attack still started — Zero Trust does not prevent the first compromise. It makes the compromised pod a dead end.

Workload identity: no more long-lived keys

Zero Trust needs every workload to have an identity it can prove without a static secret. Three common forms:

The third is a small change with a big effect: GitHub Actions can request an OIDC token that says "I am workflow release.yaml of this repository on branch main", and the cloud trusts that claim for a role valid only for the job's duration³:

permissions:
  contents: read
  id-token: write          # allow the job to request an OIDC token
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/boxoffice-deploy
          aws-region: sa-east-1
      - run: aws eks update-kubeconfig --name boxoffice-prod

No AWS key is stored in GitHub; the role's trust policy accepts only tokens whose sub matches repo:<org>/<repo>:ref:refs/heads/main. A forked repository or another branch cannot assume it.

Secrets management

A secret is any value that grants access: the payment provider API key, database passwords, the signing key of the identity provider, webhook HMAC secrets, registry tokens. Secrets must not live where things get copied, versioned or logged: source code, container images, plain ConfigMaps, CI logs.

kubectl get secret psp-api-key -n boxoffice -o jsonpath='{.data.PSP_API_KEY}' | base64 -d

Where secrets should live

The External Secrets Operator (ESO) bridges both worlds: the source of truth is the secrets manager; ESO syncs values into native Secrets that pods read as usual.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: psp-api-key
  namespace: boxoffice
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    name: psp-api-key            # Kubernetes Secret created and kept in sync by ESO
  data:
    - secretKey: PSP_API_KEY
      remoteRef:
        key: prod/boxoffice/payment/psp-api-key

The life of the payment API key

The payment provider lets a merchant hold two active keys at once — the standard trick for rotation without downtime. The lifecycle of BoxOffice's most valuable secret:

sequenceDiagram
    autonumber
    actor Ops as Payments on-call
    participant PSP as Payment provider
    participant SM as Secrets Manager
    participant ESO as External Secrets Operator
    participant K8s as Secret psp-api-key
    participant Pay as payment pods
    Ops->>PSP: create key 2 (key 1 still active)
    Ops->>SM: put key 2 as a new version of prod/boxoffice/payment/psp-api-key
    ESO->>SM: refresh (every 1 h, or forced)
    SM-->>ESO: version 2
    ESO->>K8s: update Secret
    K8s-->>Pay: mounted file changes → app reloads (or rolling restart)
    Pay->>PSP: charges now signed with key 2
    Ops->>Ops: confirm in metrics: zero calls with key 1 for 24 h
    Ops->>PSP: revoke key 1

Notice what the design gives you: only the payment service account may read that secret (IAM); every read is in the audit log; the key never appears in Git, images or CI logs; and if it leaks, the same procedure runs in minutes instead of days.

Dynamic secrets

Even better than rotating a shared secret is not having one. Dynamic secrets (Vault's database engine, cloud IAM database authentication) create a unique, short-lived credential per requester:

$ vault read database/creds/order
Key                Value
lease_id           database/creds/order/8sTt3lV0b2yL
lease_duration     1h
username           v-k8s-order-x7Qa2
password           A1a-4kd0s8f7GQpz

Each order pod gets its own database user that expires in an hour. A credential stolen from a pod stops working by itself, and the database logs show exactly which pod did what.

Encryption at rest with envelope encryption

Databases, object storage and backups are encrypted at rest with envelope encryption: each object is encrypted with a data key; the data key is itself encrypted with a key-encryption key that never leaves a KMS (Key Management Service) or HSM (hardware security module).

flowchart LR
    D["order data"] -->|"AES-256-GCM with data key"| E["ciphertext"]
    DK["data key (plaintext, in memory only)"] -->|"encrypt via KMS API"| EDK["encrypted data key<br/>(stored next to the ciphertext)"]
    KMS["KMS master key<br/>(never leaves the KMS/HSM)"] -.-> EDK

Rotating or revoking the master key in KMS controls access to all the data without re-encrypting terabytes; every decrypt call is authorized by IAM and logged.

Keep secrets out of Git

Scan before they land: a pre-commit hook and a CI job with a secret scanner (gitleaks, trufflehog, GitHub secret scanning with push protection) block commits that look like keys. If a secret is ever committed, rotate it — deleting the commit does not delete clones, forks or caches.

Supply-chain security

BoxOffice's order image contains a few hundred libraries written by strangers, built by a CI system, stored in a registry, and pulled by the cluster. Each step is an attack path: a malicious or compromised dependency (the 2024 xz-utils backdoor), a vulnerable one (Log4Shell, 2021), a compromised build system (SolarWinds, 2020), typosquatting (spring-boot-strater) and dependency confusion (a public package with the name of your internal one). Supply-chain security answers three questions: what is inside?, is it vulnerable?, did it come from our pipeline, unmodified?

flowchart LR
    src["Source<br/>(reviewed PR)"] --> build["CI build<br/>(isolated runner)"]
    build --> sbom["SBOM<br/>(syft → CycloneDX)"]
    build --> scan["Scan<br/>(trivy / grype)"]
    scan -->|"no critical CVEs"| sign["Sign + provenance<br/>(cosign keyless, SLSA)"]
    sbom --> sign
    sign --> reg[("Registry<br/>image@sha256 + attestations")]
    reg --> adm{"Admission policy<br/>(Kyverno)"}
    adm -->|"signature valid"| run["Pod runs"]
    adm -->|"unsigned / wrong signer"| deny["Rejected"]

What is inside: the SBOM

An SBOM (Software Bill of Materials) is the ingredient list of an artifact: every package, version and license, in a standard format — CycloneDX (OWASP) or SPDX (Linux Foundation, ISO/IEC 5962)⁵. When the next Log4Shell is announced, the question "which of our images contain this version?" becomes a query instead of a week of panic.

Is it vulnerable: scanning

Scanners (Trivy, Grype, Dependabot/Renovate for upgrades) match the SBOM against vulnerability databases (CVE identifiers, GitHub advisories). Gate the pipeline on a policy — e.g. "no critical vulnerability with a fix available" — not on zero findings, which never happens.

Did it come from us: signing and provenance

Sigstore cosign signs container images. In keyless mode, the CI job proves its identity with its OIDC token, a short-lived certificate is issued by Sigstore's CA (Fulcio), and the signature is recorded in a public transparency log (Rekor) — no long-lived signing key to protect⁶. SLSA (Supply-chain Levels for Software Artifacts) defines levels of build integrity; its provenance attestation records which source, workflow and builder produced the artifact⁷.

The BoxOffice release job, trimmed to the security steps (tool installation omitted):

permissions:
  contents: read
  packages: write
  id-token: write                      # keyless signing uses the job's OIDC identity
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - id: build
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: registry.example.com/boxoffice/order:${{ github.sha }}
      - name: SBOM
        run: syft registry.example.com/boxoffice/order@${{ steps.build.outputs.digest }} -o cyclonedx-json > sbom.cdx.json
      - name: Scan (fail on fixable critical CVEs)
        run: trivy image --severity CRITICAL --ignore-unfixed --exit-code 1 registry.example.com/boxoffice/order@${{ steps.build.outputs.digest }}
      - name: Sign and attest
        run: |
          IMAGE=registry.example.com/boxoffice/order@${{ steps.build.outputs.digest }}
          cosign sign --yes "$IMAGE"
          cosign attest --yes --type cyclonedx --predicate sbom.cdx.json "$IMAGE"

permissions:
  contents: read
  packages: write          # push to ghcr.io
  id-token: write          # keyless signing of the attestations (Sigstore)
  attestations: write      # store the provenance and SBOM attestations
  security-events: write   # upload the Trivy report to the Security tab
  - name: Gate - no CRITICAL vulnerability that has a fix
    uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
    with:
      image-ref: boxoffice-${{ matrix.service }}:ci
      format: table
      severity: CRITICAL
      ignore-unfixed: true
      exit-code: "1"
  - name: Attest build provenance (who built it, from which commit, how)
    uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
    with:
      subject-name: ${{ env.IMAGE }}
      subject-digest: ${{ steps.push.outputs.digest }}
      push-to-registry: true
  - name: Attest the SBOM
    uses: actions/attest-sbom@c604332985a26aa8cf1bdc465b92731239ec6b9e # v4.1.0
    with:
      subject-name: ${{ env.IMAGE }}
      subject-digest: ${{ steps.push.outputs.digest }}
      sbom-path: sbom.spdx.json
      push-to-registry: true

The last link is verification at deploy time. An admission controller (Kyverno, Sigstore policy-controller, OPA Gatekeeper) rejects any pod whose image was not signed by our release workflow:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-boxoffice-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: signed-by-release-workflow
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["boxoffice"]
      verifyImages:
        - imageReferences: ["registry.example.com/boxoffice/*"]
          mutateDigest: true             # pin the tag to the verified digest
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/repo-classes/pma271/.github/workflows/release.yaml@refs/heads/main"

Other habits that close common gaps: pin by digest (image@sha256:…, actions by commit SHA), use minimal or distroless base images, run containers as non-root with a read-only filesystem, use a private package proxy with scoped names to prevent dependency confusion, and follow the NIST Secure Software Development Framework (SSDF) as the checklist for the process⁸.

Try it. Open the ai-assistant scenario. The assistant calls tools (catalog, order) on the user's behalf: list every identity in the path (user, assistant, tool registry, services) and what each one should be allowed to do. Where would you put the PEP for a tool call? Which secrets does the LLM gateway hold? (The AI systems class goes deeper into the safety mesh.)

A defense checklist for BoxOffice

How it connects

flowchart LR
    PER["perimeter model:<br/>inside = trusted"] -->|"replaced by"| ZT["Zero Trust:<br/>never trust, always verify"]
    ZT --> PEP["PEP: gateway for users,<br/>mesh proxy for workloads"]
    PEP --> MESH["mTLS + authorization policy<br/>→ Service mesh"]
    ZT --> WI["workload identity:<br/>SPIFFE, cloud roles, CI OIDC"]
    WI --> SEC["secrets manager, dynamic secrets,<br/>dual-key rotation"]
    SEC --> KMS["envelope encryption<br/>with a KMS"]
    ZT --> SC["supply chain: SBOM, scan,<br/>cosign, SLSA"]
    SC --> ADM["admission policy at deploy<br/>→ Cloud-native"]
    ZT --> DET["denied calls logged and alerted<br/>→ Observability"]
    DET --> IR["leaked-key runbook<br/>→ incident response"]

Palette components: apigw and mesh (the enforcement points) · payment (the key only payment may read) · logs · alerting (audit and denied-call alerts) · llmgw · toolreg (identities in an agent's tool calls). Scenarios: ai-assistant, payment-system. Continues in: Service mesh, Identity, Cloud-native — delivery, Observability — incident response, AI systems — safety & evals, Hands-on 11 — supply chain, Hands-on 13 — security.

Key takeaways