Zero Trust, mTLS & Secrets

Picture the store running in Kubernetes. A vulnerable JSON library lets an attacker execute code inside one product-service pod. From there, in a classic setup, the attacker can call account-service directly with the header id-account: <any id> and read every customer, connect to Postgres with the password found in the pod's environment variables, and reach any other pod in the cluster — because "inside the cluster" means "trusted". Nothing the gateway does helps: the attacker is already behind it. This page is about limiting what an attacker can do after one component is compromised.

From perimeter security to Zero Trust

Traditional perimeter security (the "castle and moat" model) trusts everything inside the network and blocks everything outside. The problem: once an attacker is inside — via a compromised credential, a vulnerable dependency, or an insider — they can move from machine to machine. This is called lateral movement.

Zero Trust removes the idea of a trusted network. Its principle is "never trust, always verify": every request is authenticated and authorized on its own merits, regardless of where it comes from. Google popularised the model with its BeyondCorp project¹, and NIST standardised it in SP 800-207².

flowchart LR
    subgraph perimeter ["Perimeter model"]
        fw["Firewall"] -->|"trusted zone"| int["Internal services<br>(trust each other implicitly)"]
    end
    subgraph zt ["Zero Trust model"]
        any["Any request<br>(internal or external)"] -->|"identity + context"| pep["Enforcement point<br>(gateway, sidecar)"]
        pep -->|"ask"| pdp["Policy decision"]
        pdp -->|"allow, least privilege"| res["Resource"]
        pdp -->|"deny"| rej["Rejected"]
    end

NIST names the two moving parts: a Policy Enforcement Point (PEP) sits in the path of the request and blocks or forwards it, and a Policy Decision Point (PDP) decides, from identity and context, whether the request is allowed. In the store, the gateway is the PEP for external traffic; inside the cluster, a service-mesh sidecar next to each service plays the same role for internal calls.

Zero Trust principles

Example: the compromised pod, again

Service-to-service identity: mTLS

How does product-service know that a call really comes from order-service? An IP address proves nothing — pods are recreated with new IPs, and an attacker can send from any pod. The answer is a cryptographic identity per workload.

• TLS (Transport Layer Security) — the protocol behind HTTPS. The server presents an X.509 certificate (its public key plus its name, signed by a Certificate Authority, CA); the client verifies it. The channel is encrypted, but the server does not know who the client is.
• mTLS (mutual TLS) — the server also asks for the client's certificate. Both sides present certificates and verify each other: the channel is encrypted and both identities are proven.

sequenceDiagram
    autonumber
    participant Order as order-service
    participant CA as SPIRE (workload CA)
    participant Prod as product-service

    Order->>CA: attest workload, request SVID
    CA-->>Order: X.509 SVID: spiffe://cluster.local/ns/store/sa/order-service
    Prod->>CA: attest workload, request SVID
    CA-->>Prod: X.509 SVID
    Order->>Prod: ClientHello
    Prod-->>Order: ServerHello + product cert + CertificateRequest
    Order->>Order: verify product cert against trust bundle
    Order->>Prod: order cert + CertificateVerify (proof of private key)
    Prod->>Prod: verify order cert, read SPIFFE ID
    Note over Order,Prod: encrypted channel + mutual identity

Issuing and rotating certificates by hand for dozens of services does not scale. SPIFFE³ (Secure Production Identity Framework for Everyone) standardises a workload identity — a URI such as spiffe://cluster.local/ns/store/sa/order-service — and the SVID (SPIFFE Verifiable Identity Document), a short-lived X.509 certificate carrying it. SPIRE is its reference implementation: it checks (attests) which workload is asking, typically from its Kubernetes service account, and issues certificates that live for about an hour by default and rotate automatically.

Service meshes (Istio, Linkerd) inject a sidecar proxy next to each pod that performs mTLS transparently — no application code changes. Istio derives the identity from the pod's Kubernetes service account in the same SPIFFE format.

mTLS authenticates; a policy authorizes

mTLS proves who is calling; it does not decide whether the caller may call. That is an authorization policy, evaluated by the sidecar on every request. With Istio⁴, allowing only order-service to read products looks like this:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: product-allow-order
  namespace: store
spec:
  selector:
    matchLabels:
      app: product-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/store/sa/order-service"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/products/*"]

Once a workload has at least one ALLOW policy, requests that match none of its rules are denied.

Network segmentation with NetworkPolicy

A Kubernetes NetworkPolicy⁵ is a firewall rule for pods: it selects pods by label and lists which other pods (or namespaces, or IP ranges) may connect to them, on which ports. It works at layers 3/4 (IP and port), so it complements mTLS: even if an attacker steals a certificate, the network refuses connections that were never declared.

The recommended pattern is default deny, then allow explicitly. First, deny all ingress to every pod in the namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: store
spec:
  podSelector: {}        # all pods in the namespace
  policyTypes:
    - Ingress            # no ingress rules listed = nothing allowed

Then open each legitimate flow. This policy allows only gateway-service pods to reach order-service on port 8080:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: order-service-ingress
  namespace: store
spec:
  podSelector:
    matchLabels:
      app: order-service
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: gateway-service
      ports:
        - port: 8080

Policies are additive: a pod selected by any policy accepts only the traffic that at least one policy allows. A podSelector inside from matches pods in the same namespace; to allow another namespace, add a namespaceSelector.

Secrets management

A secret is any value that grants access: database passwords, the JWT_SECRET_KEY of Hands-on 3, API keys, the Docker Hub token of Hands-on 4, private keys. Secrets must not live in source code, container images, or plain configuration — anything that gets copied, versioned, or logged.

echo c3RvcmU= | base64 -d        # prints: store
kubectl get secret postgres-secrets -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d

Minimum hygiene for native Secrets: enable encryption at rest for etcd⁷, restrict get/list on Secrets with RBAC, and keep secret manifests out of Git (or commit them only encrypted, e.g. with Sealed Secrets or SOPS). Beyond that, use an external secrets manager:

Tools such as the External Secrets Operator synchronise values from a cloud manager or Vault into native Secrets, so applications keep reading them the usual way.

HashiCorp Vault introduces dynamic secrets⁸: instead of storing one long-lived database password shared by every replica, Vault creates a short-lived, unique database user for each requester, with a lease (for example, one hour). When the lease expires, Vault drops the user. A password stolen from a compromised pod stops working on its own, and the audit log shows exactly which pod used which credential.

The Vault Agent Injector pattern adds a sidecar that authenticates to Vault with the pod's service account and writes the secret into a file — here /vault/secrets/db — which the application reads; the credential never appears in the pod spec, a ConfigMap, or an environment variable⁹:

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "order-service"
  vault.hashicorp.com/agent-inject-secret-db: "database/creds/order-service"

Key takeaways

• Zero Trust drops the "trusted internal network": every request, internal or external, carries an identity and is checked against a policy (PEP + PDP).
• Design for assume breach: limit what a compromised pod can reach (NetworkPolicy), call (authorization policy) and read (short-lived secrets).
• mTLS gives each workload a cryptographic identity; SPIFFE/SPIRE or a service mesh issue and rotate those certificates automatically.
• NetworkPolicy should start from default deny, and it only works with a CNI that enforces it.
• A Kubernetes Secret is base64, not encryption; enable encryption at rest, restrict RBAC, keep secrets out of Git, and prefer a secrets manager with short-lived credentials.