Service Mesh

BoxOffice now runs nine services in Kubernetes. Each team has re-implemented the same things in its own way: order has a 2-second timeout to payment, inventory has none; catalog retries three times, search retries forever; nobody encrypts traffic inside the cluster because "it's internal"; and when security asks "which services can call payment?", the honest answer is "any pod that knows the URL". A service mesh moves these cross-cutting concerns — encryption, identity, authorization, timeouts, retries, traffic splitting, telemetry — out of the application code and into a uniform infrastructure layer.

What a service mesh is

A service mesh is a dedicated infrastructure layer that handles service-to-service communication¹. It has two parts:

• Data plane — proxies that sit in the path of every request between services. They terminate and originate TLS, enforce policies, retry, time out, balance load and emit metrics and traces.
• Control plane — the brain that configures the proxies: it distributes routing rules and policies, and acts as a certificate authority that issues each workload its identity.

flowchart TB
    subgraph cp ["Control plane (istiod / linkerd-control-plane)"]
        CFG["config + policy"]
        CA["certificate authority"]
    end
    subgraph dp ["Data plane"]
        subgraph po ["pod: order"]
            O["order app"] --- PO["proxy"]
        end
        subgraph pp ["pod: payment"]
            PP["proxy"] --- P["payment app"]
        end
    end
    PO <-->|"mTLS"| PP
    CFG -.->|"routes, policies"| PO & PP
    CA -.->|"certificates (rotated)"| PO & PP

The application still calls http://payment:8080/charges in plain HTTP. The proxy on the way out upgrades it to mutual TLS, the proxy on the way in verifies the caller's identity, checks the policy and hands plain HTTP to payment. Neither application changed a line.

What you get

The mesh is the natural home of three topics of the course: reliability patterns (see Reliability), observability of every hop (see Observability) and Zero Trust between services (Zero Trust).

Data-plane architectures: sidecar, ambient, eBPF

Where does the proxy run? Three designs are in production today²³.

Identity and mTLS

How does payment know that a call really comes from order? IP addresses prove nothing — pods are recreated with new IPs, and the attacker in the compromised search pod can send from any IP it likes. The mesh gives each workload a cryptographic identity.

• TLS (Transport Layer Security) — the server proves its identity with an X.509 certificate signed by a certificate authority; the channel is encrypted.
• mTLS (mutual TLS) — the client also presents a certificate. Both sides are authenticated.
• SPIFFE (Secure Production Identity Framework For Everyone)⁴ — a standard name for workload identities, spiffe://<trust-domain>/<path>. Istio uses spiffe://cluster.local/ns/<namespace>/sa/<service-account>: the identity is the Kubernetes service account.
• SVID (SPIFFE Verifiable Identity Document) — the short-lived X.509 certificate that carries that identity, rotated automatically (Istio's default lifetime is 24 hours; SPIRE's default is one hour).

sequenceDiagram
    autonumber
    participant O as order (proxy)
    participant CA as Mesh CA (istiod)
    participant P as payment (proxy)
    O->>CA: CSR + proof of service account token
    CA-->>O: SVID spiffe://cluster.local/ns/boxoffice/sa/order (24 h)
    P->>CA: CSR + proof of service account token
    CA-->>P: SVID spiffe://cluster.local/ns/boxoffice/sa/payment
    O->>P: ClientHello
    P-->>O: payment certificate + CertificateRequest
    O->>O: verify payment's SVID against the trust bundle
    O->>P: order certificate + proof of private key
    P->>P: verify order's SVID, read its SPIFFE ID
    P->>P: authorization policy: may sa/order POST /charges? → allow
    Note over O,P: encrypted channel, both identities proven, policy applied

Turn on strict mTLS

By default Istio accepts both plain-text and mTLS (permissive mode) so that services can migrate gradually. Once every workload has a proxy, make it strict:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: boxoffice
spec:
  mtls:
    mode: STRICT        # reject any plain-text connection to pods in this namespace

mTLS authenticates; a policy authorizes

A valid certificate says who is calling — not whether they may. That is an authorization policy, evaluated by the proxy on every request. Only order may charge cards:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payment-allow-order
  namespace: boxoffice
spec:
  selector:
    matchLabels:
      app: payment
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/boxoffice/sa/order"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charges", "/refunds"]

# API versions change between Linkerd releases: check yours
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: payment-http
  namespace: boxoffice
spec:
  podSelector:
    matchLabels:
      app: payment
  port: 8080
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata:
  name: order-only
  namespace: boxoffice
spec:
  identities:
    - "order.boxoffice.serviceaccount.identity.linkerd.cluster.local"
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: payment-allow-order
  namespace: boxoffice
spec:
  targetRef:
    group: policy.linkerd.io
    kind: Server
    name: payment-http
  requiredAuthenticationRefs:
    - group: policy.linkerd.io
      kind: MeshTLSAuthentication
      name: order-only

Traffic policy

The second big job of a mesh is shaping traffic: timeouts, retries, circuit breaking and splitting.

Timeouts and retries per route

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payment
  namespace: boxoffice
spec:
  hosts: ["payment"]
  http:
    - route:
        - destination:
            host: payment
            subset: v1
          weight: 90
        - destination:
            host: payment
            subset: v2        # canary: 10% of the traffic
          weight: 10
      timeout: 3s             # total budget for the call
      retries:
        attempts: 2
        perTryTimeout: 1s
        retryOn: connect-failure,refused-stream,unavailable

Three decisions are encoded here: a canary sends 10% of the calls to v2; the whole call has a 3-second budget; and retries happen only on failures where the request certainly did not reach the application (connect-failure, refused-stream, unavailable). Retrying a POST /charges after a timeout could charge the customer twice — that is what idempotency keys are for (see Compute — integrations).

Circuit breaking with outlier detection

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payment
  namespace: boxoffice
spec:
  host: payment
  subsets:
    - name: v1
      labels: { version: v1 }
    - name: v2
      labels: { version: v2 }
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 200          # bulkhead: max concurrent requests
        maxRequestsPerConnection: 50
    outlierDetection:
      consecutive5xxErrors: 5          # eject a replica after 5 errors in a row
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50           # never eject more than half the replicas

Outlier detection removes a misbehaving replica from the load-balancing pool for a while — a per-instance circuit breaker. The connection pool limits are a bulkhead: when payment is slow, at most 200 requests wait on it, instead of every thread of order.

Try it. Open the cascading-failure scenario: a slow third-party dependency with no timeouts exhausts the caller's threads, and unrelated requests start failing. Turn on the timeout and the circuit breaker, run again, and compare the error rate and p99 of the other routes. In BoxOffice, the mesh VirtualService timeout and DestinationRule limits are one way to apply exactly that fix without touching the code.

Gateway API for the mesh (GAMMA)

The Kubernetes Gateway API started as the successor of Ingress for north–south traffic (see Edge). The GAMMA initiative extends the same HTTPRoute resource to east–west traffic inside the mesh, so a canary looks the same in Istio, Linkerd or Cilium:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-canary
  namespace: boxoffice
spec:
  parentRefs:
    - group: ""
      kind: Service           # attach to a Service = mesh (east-west) routing
      name: payment
      port: 8080
  rules:
    - backendRefs:
        - name: payment-v1
          port: 8080
          weight: 90
        - name: payment-v2
          port: 8080
          weight: 10

Mesh policy and NetworkPolicy: two layers

A Kubernetes NetworkPolicy is a firewall for pods at L3/L4 (IP and port), enforced by the CNI plugin (Calico, Cilium)⁵. A mesh authorization policy works at L7 with cryptographic identity. They are complementary layers of defense in depth:

Start with default deny at the network layer, then open only the edges of the call graph:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: boxoffice
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payment-from-order
  namespace: boxoffice
spec:
  podSelector:
    matchLabels:
      app: payment
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: order
      ports:
        - port: 8080

Costs and when not to use a mesh

A mesh is powerful and expensive to run. It adds a control plane to operate and upgrade, proxies that consume CPU and memory, a few milliseconds per hop, a new class of failures ("the proxy rejected it"), and a steep learning curve for debugging.

How it connects

flowchart LR
    CP["control plane<br/>config + CA"] --> DP["data plane:<br/>sidecar, ambient, eBPF"]
    CP --> SVID["SPIFFE identity (SVID)<br/>from the service account"]
    SVID --> MTLS["strict mTLS"]
    MTLS --> AUTHZ["authorization policy<br/>only order → payment"]
    AUTHZ --> NP["NetworkPolicy L3/L4<br/>default deny + egress"]
    AUTHZ --> ZT["PEP for workloads<br/>→ Zero Trust"]
    DP --> TP["timeouts, retries,<br/>outlier detection"]
    TP --> REL["resilience patterns<br/>→ Reliability"]
    DP --> CAN["HTTPRoute weights (GAMMA)<br/>→ canary in Cloud-native"]
    DP --> TEL["golden signals per hop<br/>→ Observability"]

Palette components: mesh · app (the services behind each proxy) · ingress (Gateway API, north–south) · metrics · tracing (telemetry of every hop). Scenarios: cascading-failure, retry-storm. Continues in: Zero Trust, Reliability — patterns, Observability — signals, Edge — load balancing in Kubernetes, Cloud-native — delivery (canary).

Key takeaways