9. Reliability

Reliability is a critical aspect of modern software systems, ensuring that services remain available, performant, and trustworthy for users. Central to managing reliability are three key concepts: Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs). This document provides a comprehensive overview of these concepts, their definitions, interrelationships, and practical applications.

1. Core Definitions

2. In-Depth Breakdown

SLI – The Raw Measurement

• What it is: A quantitative metric pulled from logs, monitoring systems, or instrumentation.
• Examples:
  • Availability: % of successful HTTP requests (e.g., 200 OK responses)
  • Latency: 95th percentile request latency < 200 ms
  • Error rate: Errors per second / Total requests per second
  • Throughput: Requests per second (RPS)
  • Durability: % of data writes acknowledged by 3+ replicas
• Key properties:
  • Objective (not subjective)
  • Directly measurable
  • Relevant to user experience
• How it's collected:
  • Prometheus exporters, OpenTelemetry, Cloud monitoring, application logs, synthetic probes.

SLO – The Target You Commit To

• What it is: A specific, time-bound target for an SLI.
• Format: SLI ≥ threshold over a rolling window (e.g., 28 days).
• Examples:
  • Availability SLO: 99.9% of requests return 200 OK in a 28-day window.
  • Latency SLO: 95th percentile latency ≤ 150 ms over 28 days.
  • Error Budget: Allowed error = 100% – 99.9% = 0.1% → ~43 minutes downtime/month.
• Error Budget = 1 – SLO target
  • Drives reliability decisions: If budget is consumed → stop features, focus on stability.

SLA – The Contractual Commitment

• What it is: A legal/business agreement with penalties (credits, refunds) if SLOs are breached.
• Structure:
  • List of SLOs
  • Measurement period
  • Exclusions (maintenance, force majeure)
  • Remedies (service credits)
• Example: > "Provider guarantees 99.95% monthly uptime. If below, customer receives 10% credit for that month."
• SLA ≠ SLO: SLA is external, SLO is internal target (often stricter than SLA to provide buffer).

3. How SLI → SLO → SLA Connect (The Reliability Stack)

flowchart TD
    A("SLI: Measurable Metric<br><small>e.g., % successful requests</small>") --> B("SLO: Target Value<br><small>Internal target (e.g., 99.99% uptime)</small>")
    B --> C("SLA: Formal Agreement<br><small>External contract (e.g., 99.95% uptime)</small>")
    C --> D[Customer Expectations]
    B --> E[Error Budget Management]
    A --> F[Monitoring & Alerting]

Flow:

1. Choose SLIs → What matters to users?
2. Set SLOs → What can we realistically achieve?
3. Define SLA → What do we promise customers? (usually looser than SLO)

Best Practice: SLO > SLA → gives error budget buffer to avoid penalties.

4. Roles & Responsibilities

5. Real-World Example (Web Service)

→ Team monitors SLI daily, uses error budget to decide:

"We have 10 min left this month → no new deploys until stabilized."

6. Key Best Practices

7. Summary Table

Final Takeaway

SLI is what you measure
SLO is what you promise internally
SLA is what you promise externally (with money on the line)

They form a hierarchy of reliability:

SLI (data) → SLO (target) → SLA (contract)

Used together, they enable data-driven reliability engineering — the foundation of modern SRE practices¹.