Skip to content

2. RED Dashboard

Data you cannot read in ten seconds during an incident is not observability, it is storage. This step builds one dashboard for every BoxOffice service, provisioned as code, organized around two methods from the signals class:

Method For Signals Question it answers
RED (Tom Wilkie) request-driven services Rate, Errors, Duration "Are my users being served well?"
USE (Brendan Gregg) resources Utilization, Saturation, Errors "Is a resource the bottleneck?"

RED says that something is wrong for users; USE says where the capacity ran out. The top row of the dashboard is RED per service; the bottom row is USE for the JVMs and their connection pools.

Dashboards as code

Grafana loads dashboards from files listed by a provider. Both files live in your repository, so the dashboard is reviewed, versioned and identical on every laptop:

infra/grafana/dashboards.yaml
apiVersion: 1

providers:
  - name: BoxOffice
    folder: BoxOffice
    type: file
    options:
      path: /otel-lgtm/boxoffice-dashboards

You saved this provider in step 1; its volumes mount it into Grafana's provisioning folder and infra/grafana/dashboards/ at the path it points to. Save the dashboard below as infra/grafana/dashboards/boxoffice-red.json and restart: docker compose restart otel-lgtm. It appears under Dashboards → BoxOffice → BoxOffice — RED.

infra/grafana/dashboards/boxoffice-red.json (click to expand)
infra/grafana/dashboards/boxoffice-red.json
{
  "uid": "boxoffice-red",
  "title": "BoxOffice — RED",
  "tags": [
    "boxoffice"
  ],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "10s",
  "time": {
    "from": "now-15m",
    "to": "now"
  },
  "templating": {
    "list": [
      {
        "name": "service",
        "label": "service",
        "type": "query",
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "query": {
          "query": "label_values(http_server_request_duration_seconds_count, service_name)",
          "refId": "service"
        },
        "definition": "label_values(http_server_request_duration_seconds_count, service_name)",
        "multi": true,
        "includeAll": true,
        "current": {
          "text": "All",
          "value": "$__all"
        },
        "refresh": 2
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "type": "row",
      "title": "RED: Rate, Errors, Duration",
      "gridPos": {
        "x": 0,
        "y": 0,
        "w": 24,
        "h": 1
      },
      "collapsed": false,
      "panels": []
    },
    {
      "id": 2,
      "type": "timeseries",
      "title": "Rate (requests/s)",
      "description": "R: how much work arrives",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 0,
        "y": 1,
        "w": 8,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "reqps"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\"}[$__rate_interval]))",
          "legendFormat": "{{service_name}}"
        }
      ]
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "Errors (% of requests answered 5xx)",
      "description": "E: the share of that work that fails",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 8,
        "y": 1,
        "w": 8,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "percent"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "100 * sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\", http_response_status_code=~\"5..\"}[$__rate_interval])) / sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\"}[$__rate_interval]))",
          "legendFormat": "{{service_name}}"
        }
      ]
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "Duration p95 (s)",
      "description": "D: how long the slowest 5% wait",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 16,
        "y": 1,
        "w": 8,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "s"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "histogram_quantile(0.95, sum by (service_name, le) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\"}[$__rate_interval])))",
          "legendFormat": "{{service_name}}"
        }
      ]
    },
    {
      "id": 5,
      "type": "timeseries",
      "title": "Duration p50 / p95 / p99 (all selected services)",
      "description": "The gap between p50 and p99 is the tail",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 0,
        "y": 9,
        "w": 12,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "s"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\"}[$__rate_interval])))",
          "legendFormat": "p50"
        },
        {
          "refId": "B",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\"}[$__rate_interval])))",
          "legendFormat": "p95"
        },
        {
          "refId": "C",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\"}[$__rate_interval])))",
          "legendFormat": "p99"
        }
      ]
    },
    {
      "id": 6,
      "type": "table",
      "title": "Slowest routes (p95 over 5 min)",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 12,
        "y": 9,
        "w": 12,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "s"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "instant": true,
          "format": "table",
          "expr": "topk(10, histogram_quantile(0.95, sum by (service_name, http_request_method, http_route, le) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\"}[5m]))))"
        }
      ],
      "transformations": [
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true
            }
          }
        }
      ]
    },
    {
      "id": 7,
      "type": "row",
      "title": "USE: Utilization, Saturation (JVM and pools)",
      "gridPos": {
        "x": 0,
        "y": 17,
        "w": 24,
        "h": 1
      },
      "collapsed": false,
      "panels": []
    },
    {
      "id": 8,
      "type": "timeseries",
      "title": "CPU utilization (of the container's CPUs)",
      "description": "",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 0,
        "y": 18,
        "w": 8,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "avg by (service_name) (jvm_cpu_recent_utilization_ratio{service_name=~\"$service\"})",
          "legendFormat": "{{service_name}}"
        }
      ]
    },
    {
      "id": 9,
      "type": "timeseries",
      "title": "Heap used",
      "description": "",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 8,
        "y": 18,
        "w": 8,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "bytes"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "sum by (service_name) (jvm_memory_used_bytes{service_name=~\"$service\", jvm_memory_type=\"heap\"})",
          "legendFormat": "{{service_name}}"
        }
      ]
    },
    {
      "id": 10,
      "type": "timeseries",
      "title": "DB connections in use / waiting (saturation)",
      "description": "Waiting > 0 means requests queue for a connection: the pool is saturated",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "x": 16,
        "y": 18,
        "w": 8,
        "h": 8
      },
      "fieldConfig": {
        "defaults": {
          "unit": "short"
        },
        "overrides": []
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "sum by (service_name) (db_client_connections_usage{service_name=~\"$service\", state=\"used\"})",
          "legendFormat": "{{service_name}} used"
        },
        {
          "refId": "B",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "sum by (service_name) (db_client_connections_max{service_name=~\"$service\"})",
          "legendFormat": "{{service_name}} max"
        },
        {
          "refId": "C",
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "sum by (service_name) (db_client_connections_pending_requests{service_name=~\"$service\"})",
          "legendFormat": "{{service_name}} waiting"
        }
      ]
    }
  ]
}

Reading the panels

Every panel is one PromQL query over the histogram the agent exports. Learn the four shapes; every RED dashboard you will ever see is a variation of them.

Panel Query shape Read it as
Rate sum by (service_name) (rate(…_count[$__rate_interval])) requests per second, per service
Errors rate(…_count{status=~"5.."}) / rate(…_count) the share of requests that fail (not the count)
Duration p95 histogram_quantile(0.95, sum by (service_name, le) (rate(…_bucket[…]))) 95% of requests were faster than this
Slowest routes topk(10, histogram_quantile(0.95, sum by (…, http_route, le) …)) where to look first
DB connections db_client_connections_usage{state="used"} vs _max, _pending_requests the pool is saturated when waiting > 0

Three habits make the difference:

  • Aggregate before the quantile. histogram_quantile must receive sum by (le) of the bucket rates: you can add buckets across instances, you cannot average percentiles (the average of three p95s is not the p95 of anything).
  • Errors as a ratio. 50 errors per second is a disaster at 60 req/s and noise at 60 000 req/s.
  • Percentiles from buckets are estimates. histogram_quantile interpolates inside a bucket; with the agent's default bucket boundaries (…, 0.25, 0.5, 0.75, 1, 2.5 s…), a p99 of "0.6 s" means "somewhere between 0.5 and 0.75 s".

Exemplars: from a spike to the trace that caused it

The otel-lgtm Prometheus stores exemplars: a few sample trace ids attached to histogram buckets. In the Duration panel, turn on Exemplars in the query options; the diamonds on the chart are real requests, and clicking one opens its trace in Tempo. This is the shortest path from "p99 went up" to "this SQL statement is slow".

Exercise: watch a bottleneck appear

  1. Run the staircase stress test of lab 2 against the edge — it raises the arrival rate in steps up to PEAK and comes back down: k6 run -e BASE_URL=http://localhost -e PEAK=600 load/stress.js.
  2. Watch Duration p95, CPU utilization and DB connections side by side. The latency curve bends where the first resource saturates. When waiting leaves zero, requests queue for a connection (Little's Law: with the pool full, extra concurrency becomes waiting time); when a service's CPU nears 100% of its container's CPU, they queue for the processor. In a test run of this lab the gateway (limited to one CPU) saturated first: about 95% CPU at about 320 req/s, its p95 above 2 s and 504s after its 5 s timeout, while the catalog pool never had a request waiting.
  3. Open an exemplar at the top of the curve: the trace shows where the time went — a long gap before the SQL span is time spent waiting for a connection, not running SQL; a long gateway span around a short catalog span is time spent queued in the gateway.

Write in your results which resource saturated first and the request rate at which it happened.

Next: SLO and burn-rate alert