4. Measure It

Observability is not free: every span is allocated, every metric point aggregated, every batch sent over the network. This step puts a number on that cost and on the benefit — how long it takes to know that users are hurting. As always, run each experiment twice and report the second run; numbers in "expected shape" boxes are illustrative.

The k6 script

import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL || 'http://localhost'; // the edge (lab 3)

export const options = {
  scenarios: {
    catalog: {
      executor: 'constant-arrival-rate', // open model: the arrival rate does not depend on latency
      rate: Number(__ENV.RATE || 200),
      timeUnit: '1s',
      duration: __ENV.DURATION || '2m',
      preAllocatedVUs: 50,
      maxVUs: 400,
    },
  },
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const id = 1 + Math.floor(Math.random() * 200000);
  const res = http.get(`${BASE_URL}/catalog/events/${id}`, { tags: { name: 'GET /catalog/events/{id}' } });
  check(res, { 'status 200': (r) => r.status === 200 });
}

A. Agent overhead

Same image, same load, agent on and off:

Run docker stats in a second terminal while k6 runs (the numbers after the test are idle numbers). The docker compose logs lines show the startup time with the agent off and on (the logs of a container are gone once it is recreated, so read them before switching).

B. Sampling

Keeping 100% of the traces is right for a lab and wrong for 2 million fans. Sample 10% at the root and let every downstream service follow the root's decision (parentbased_…), so that a trace is either complete or absent:

The x-otel anchor already reads both settings from the environment (${OTEL_TRACES_SAMPLER:-parentbased_always_on}), so no file changes: recreate the services with the two variables set and run the same load.

OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 docker compose up -d --wait
k6 run load/catalog-otel.js
docker compose up -d --wait # afterwards: back to 100% (the defaults of the anchor)

Count the traces stored per minute in Tempo (Explore → Tempo → Search, service gateway, last 5 minutes) and compare p95 with experiment A. Note that metrics are not sampled: the RED dashboard still counts every request — that is why rates and percentiles come from metrics, and examples (the why) from traces.

C. Cardinality

Every distinct combination of label values is a separate time series, stored and queried separately. In Grafana Explore (Prometheus), run:

count by (service_name) ({service_namespace="boxoffice"})
count by (__name__) ({service_name="catalog"})

The first gives the series per service; the second shows which metrics are the most expensive. Now imagine adding a label user_id to the HTTP histogram: with 2 million fans and 14 buckets, how many series would catalog create? (Answer it in your results: this is why the agent records http_route, the template, and never the raw URL.)

D. Time to detect

Follow "Fire it" in step 3 and record the three times.

Results

Copy this table into results/lab8.md and fill it in.

Below the table, answer in a paragraph each: is the agent's overhead worth it for BoxOffice? Which sampling rate would you choose for the flash sale, and why? Does your measured time to detect match the prediction of step 3?

Back to the overview Quiz