Hands-on 6 — Observability

A request is slow. Where?

• Without data from inside each service, you can only guess.
• You need metrics over time: request rates, latencies, error rates, JVM memory.
• And they must be collected from every service, in one place.

The three pillars

Architecture

gateway, auth, account ← scrape ← Prometheus → Grafana

The observability layer sits alongside the trusted layer; clients still enter through the load balancer on port 80.

The steps

1. Microservices — add the Actuator and Prometheus registry dependencies, expose the endpoint.
2. Docker Compose — add the prometheus and grafana services.
3. Prometheus — prometheus.yml with one scrape job per service.
4. Grafana — provision Prometheus as the datasource, import a dashboard.
5. Run — rebuild the stack and check that every target is UP.

1 · Dependencies (each service)

In the pom.xml of gateway-service, auth-service and account-service:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <scope>runtime</scope>
</dependency>

Actuator: production-ready endpoints (health, metrics). Micrometer registry: exports them in the Prometheus text format.

1 · Expose the endpoint

# gateway-service application.yaml
management:
  endpoints:
    web:
      base-path: /gateway/actuator
      exposure:
        include: ['prometheus']

• Each service uses its own context prefix: no conflicts, no internal ports exposed.
• The gateway also sets spring.cloud.gateway.metrics.enabled: true.

3 · Prometheus scrapes (pull)

setup/prometheus/prometheus.yml — one job per service:

scrape_configs:
  - job_name: 'GatewayMetrics'
    metrics_path: '/gateway/actuator/prometheus'
    scrape_interval: 1s
    static_configs:
      - targets:
        - gateway:8080
        labels:
          application: 'Gateway Application'

Same for AuthMetrics (auth:8080) and AccountMetrics (account:8080).

The fields that matter

4 · Grafana datasource

setup/grafana/provisioning/datasources/datasources.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Loaded automatically on startup — no manual UI setup. Open http://localhost:3000, login admin / admin.

4 · Dashboards

• Import a pre-built one: Dashboards → Import, ID 4701 (JVM (Micrometer)) or 12900 (SpringBoot APM).
• Select the provisioned Prometheus datasource.
• If panels stay empty: Dashboard → Edit → Settings → Variables, create DS_PROMETHEUS, type Constant, value prometheus (the uid in datasources.yml).
• Then watch request rates, JVM memory, response latencies and error rates.

More dashboards at grafana.com/grafana/dashboards.

5 · Run and verify

docker compose up -d --build
docker compose ps -a

• Every container, including prometheus and grafana, should be running.
• Prometheus UI http://localhost:9090 → Status → Targets: gateway, auth and account all UP.
• Grafana: import a dashboard and see the panels populate.

Common pitfalls

• prometheus does not start → check that setup/prometheus/prometheus.yml exists and is valid YAML.
• Targets must use the Docker hostnames, not localhost (inside a container, that is the container itself).
• Dashboard empty → the DS_PROMETHEUS variable does not point to the prometheus datasource.
• Only prometheus is in exposure.include: other Actuator endpoints stay closed.

Takeaways

1. Observability has three pillars: logging, monitoring, tracing — this hands-on does monitoring.
2. Actuator + Micrometer turn each Spring Boot service into a metrics endpoint.
3. Each service exposes /<context>/actuator/prometheus under its own prefix.
4. Prometheus pulls metrics on a schedule (scrape_interval), one job per service.
5. Grafana reads Prometheus through a provisioned datasource and pre-built dashboards.
6. Verify with Status → Targets: every service must be UP.