Overview
In the team project you design, build, measure, break and defend one system, from the first estimate to a load-tested deployment on Kubernetes. It is the course in miniature: every class adds a piece, every lab gives you a working example to adapt, and every milestone asks for evidence β a diagram, a passing check, a k6 report, a dashboard β not promises.
Goals
By the end of the project your team will have:
- Designed a system from requirements to a reviewed design doc with estimates, SLOs and ADRs.
- Built its core path as services that run with one command (Docker Compose) and on Kubernetes.
- Scaled it with caching, replication and asynchronous processing, and protected it with resilience patterns.
- Delivered it continuously (CI/CD with tests, image scan and a performance gate).
- Proved it meets a target SLO with a load-test campaign, and that it degrades gracefully in a chaos game day.
- Priced it (cost report with unit economics) and defended it in a final presentation.
Team rules
- 2 to 3 students per team. Teams are registered by week 3 (form on Blackboard) with the chosen system option; changing option after week 5 needs the instructor's approval.
- One GitHub repository per team (monorepo recommended:
services/,deploy/,load/,chaos/,docs/), with every member as a collaborator and the instructor and teaching assistant added. - Documentation on GitHub Pages (MkDocs Material or Zensical), one section per milestone; the documentation template is a good start.
- Ownership table on the home page: each member owns at least one service and at least one cross-cutting area (e.g. observability, CI/CD, load testing, chaos, cost). Owners lead; everybody must understand everything.
- Every member commits with their own GitHub identity (
git config user.name/user.email). Work done in pairs is marked withCo-authored-by:trailers. - Milestones are tags:
m1β¦m9on the commit to be graded, plus a short entry in the docs ("what we delivered, evidence, what is missing").
Individual grades inside a team
The team grade is, by default, the same for all members. The instructor may adjust an individual grade when the evidence (commit history, ownership table, milestone check-ins, the final oral questions, and a short confidential peer evaluation at the end) shows that a member did not contribute or cannot explain the system. The oral check rule applies to every member.
System options
Pick one option. Each one exercises the whole palette, but stresses a different part of it. Every option has two scales:
- Design target β the real-world scale, used in the design doc, the estimates and the cost report (you will not run it).
- Test target β a scaled-down load that your deployment (laptop with Docker Compose or kind/minikube, or a small cloud account) must sustain within the SLOs in the load-test campaign. If your environment cannot reach it, show the bottleneck with numbers and agree a new target with the instructor before week 14.
The course's running example. A stadium show goes on sale at 10:00; 2 M fans arrive for 60 k seats. Fans browse the catalog, enter a waiting room, hold seats for a few minutes, pay through an external provider and receive their tickets by e-mail.
Core requirements: event catalog and search; waiting room with fair admission; seat hold with expiry; order + payment (provider stub with latency and failures); ticket e-mail; no double booking β ever.
| SLO / constraint | Design target | Test target |
|---|---|---|
| Catalog read latency | p99 < 200 ms at 200 k req/s | p99 < 200 ms at 1,000 req/s |
| Seat hold latency | p99 < 500 ms | p99 < 500 ms at 100 holds/s |
| Double bookings | 0 | 0 (checked by a consistency query after every test) |
| Availability during the on-sale | 99.9% | error rate < 0.1% (excluding 429 from the waiting room) |
| Ticket e-mail | p95 < 60 s after payment | p95 < 30 s, consumer lag < 10 s |
Playground: flash-sale Β· Case study: ticketing
One-to-one and small-group chat (up to 100 members) with online presence and message history, used on mobile and web.
Core requirements: WebSocket (or SSE) connections behind a load balancer; message fan-out through pub/sub across instances; per-conversation ordering; at-least-once delivery with client-side de-duplication; history pagination; push notification for offline users.
| SLO / constraint | Design target | Test target |
|---|---|---|
| End-to-end delivery (sender β online receiver) | p99 < 300 ms | p99 < 500 ms |
| Concurrent connections | 10 M | 5,000 |
| Messages | 1 M msg/s peak | 500 msg/s |
| Message loss | 0 acknowledged messages lost | 0 (checked by sequence numbers) |
| History read | p99 < 200 ms | p99 < 200 ms at 200 req/s |
Drivers send their location every few seconds; riders request a ride and get matched with a nearby available driver.
Core requirements: location ingestion (write-heavy); geo index (geohash, H3 or PostGIS) of available drivers; matching with a driver lock (one driver, one ride); trip state machine; pricing stub; notifications to both sides.
| SLO / constraint | Design target | Test target |
|---|---|---|
| Location update ingestion | p99 < 100 ms at 1 M updates/s | p99 < 200 ms at 1,250 updates/s (5,000 drivers every 4 s) |
| Nearby-drivers query | p99 < 150 ms | p99 < 200 ms at 100 req/s |
| Match (request β driver assigned) | p95 < 3 s | p95 < 3 s at 20 rides/s |
| Double assignment of a driver | 0 | 0 (checked after every test) |
| Location freshness | < 10 s | < 10 s |
Playground: ride-sharing Β· Case study: ride-sharing
Users post short texts with images and follow other users; the home timeline shows recent posts of the people they follow β including a few celebrities with millions of followers.
Core requirements: posts with media in object storage; follow graph; home timeline with a hybrid fan-out (push for normal users, pull for celebrities); timeline cache; likes counter; notifications.
| SLO / constraint | Design target | Test target |
|---|---|---|
| Home timeline read | p99 < 200 ms at 300 k req/s | p99 < 200 ms at 1,000 req/s |
| Post visible to followers | p95 < 5 s | p95 < 5 s, including one account with 100 k followers |
| Post write | p99 < 300 ms | p99 < 300 ms at 50 posts/s |
| Availability (reads) | 99.95% | error rate < 0.1% |
Creators upload videos; the platform transcodes them into several resolutions; viewers watch them with adaptive streaming (HLS) served from a CDN-like edge.
Core requirements: resumable/multipart upload to object storage (MinIO locally) with pre-signed URLs; transcoding jobs on a queue with workers (FFmpeg); HLS manifests and segments served through a cache (Nginx as the "CDN"); video metadata and search; view counter.
| SLO / constraint | Design target | Test target |
|---|---|---|
| Upload acknowledged (after the transfer) | p99 < 1 s | p99 < 1 s |
| Transcoding done | p95 < 2Γ video duration | p95 < 2Γ duration for 20 parallel uploads of 30 s clips |
| Playback start (manifest + first segment) | p99 < 300 ms | p99 < 300 ms at 500 req/s |
| Edge (cache) hit ratio for segments | > 95% | > 90% |
Playground: video-streaming Β· Case study: video streaming
A support assistant for BoxOffice (or another domain agreed with the instructor): it answers questions with RAG over the help center and the event catalog, and uses tools to look up or cancel orders β safely.
Core requirements: LLM gateway (LiteLLM) with model routing, fallback and per-user quotas; ingestion pipeline (chunking, embeddings, pgvector); orchestrator with a bounded agent loop and tool calls to existing services; guardrails (prompt injection, PII, output validation); semantic cache; token and cost metrics; an evaluation set.
| SLO / constraint | Design target | Test target |
|---|---|---|
| Time to first token (TTFT) | p95 < 1.5 s | p95 < 2 s with a stub LLM (fixed latency distribution) at 20 concurrent chats; measured separately with the real model at 1 req/s |
| Answer quality | β₯ 90% on the evaluation set | β₯ 80% on a 30-question evaluation set |
| Unsafe tool actions | 0 | 0 on a 20-prompt injection test set |
| Cost per conversation | < US$ 0.01 | measured and reported (tokens Γ price) |
Playground: ai-assistant Β· Case study: AI assistant
Other systems
A different system (e.g. food delivery, collaborative editor, payments ledger) is welcome if it has the same breadth. Send a one-page proposal with requirements and target SLOs by week 3; it needs the instructor's approval.
Mandatory components
Whatever the option, the system must have the components below (palette ids in parentheses β see the component catalog). The milestone where each one is first required is in the last column.
| Area | Requirement | Milestone |
|---|---|---|
| Edge | An entry point: API gateway (apigw) or Ingress (ingress) with rate limiting on at least one route | M2 |
| Identity | Authentication (auth) with JWT or OIDC; at least one protected route | M2 |
| Services | At least three services of your own (app/worker), each with its own data | M2 |
| Data | A SQL database (sql) with migrations; a justified second store if needed (nosql, objstore, vectordb) | M2 |
| Caching | A cache (cache) on a hot read path, with a measured hit ratio | M3 |
| Replication | A read replica or a connection pool in front of the database, justified by a measurement | M3 |
| Messaging | A broker (kafka, queue or pubsub) with a transactional outbox or an idempotent consumer, and a DLQ | M4 |
| Resilience | Timeout + retry with backoff and jitter + circuit breaker on at least one slow/external dependency (payment, thirdparty, email or llmgw) | M4 |
| Observability | OpenTelemetry β metrics, logs and traces (grafana/otel-lgtm is enough); a RED dashboard; one SLO burn-rate alert | M4 |
| Platform | Kubernetes (kind/minikube or cloud) with probes, requests/limits and an HPA on at least one service | M5 |
| Delivery | GitHub Actions: build, tests, image build, image scan, SBOM, and a k6 performance gate | M5 |
| Proof | k6 load-test campaign against the test target; chaos game day with at least three experiments | M6βM7 |
| Cost | Cost report for the design target and for the environment you actually used | M8 |
The hands-on labs give you a working version of almost every row: reuse them (and say so).
Deliverables
All deliverables live in the team repository and its GitHub Pages site:
- Design doc + ADRs β design doc (M1, kept up to date until the end).
- Running system β code,
compose.yaml, Kubernetes manifests (or Helm/Kustomize), CI/CD workflows (milestones M2βM5). - Load-test campaign report β plan, k6 scripts, raw results, analysis (load-test campaign, M6).
- Game day report β experiment plans, run sheet, findings and action items (game day, M7).
- Cost report β design-target and actual costs, unit economics (cost report, M8).
- Final presentation β slides, live demo, 3-minute video (presentation, M9).
Grading
| Phase | Deliverable | Page | Weight |
|---|---|---|---|
| Design | M1 β Design doc & estimates | design doc | 10% |
| Build | M2 β Core path running | milestones | 10% |
| M3 β Scale & cache | milestones | 10% | |
| M4 β Async & resilience | milestones | 10% | |
| M5 β Kubernetes + CI/CD | milestones | 10% | |
| Prove | M6 β Load-test campaign | load-test campaign | 15% |
| M7 β Chaos game day | game day | 10% | |
| Communicate | M8 β Cost report | cost report | 10% |
| M9 β Final presentation | presentation | 10% | |
| Documentation quality (whole site) | this page | 5% |
Each deliverable receives a concept grade (I, D, C, B, A, with +) using the rubric on its page and the common level definitions; the project grade is the weighted average of the numeric equivalents. How the project grade enters the final grade is on the course page.
Documentation quality rubric
| Criterion | I | D | C | B | A |
|---|---|---|---|---|---|
| Structure | No site | Pages missing or broken | One section per milestone, navigable | Clear home page with architecture, ownership and links to every deliverable | A reviewer finds any evidence in two clicks |
| Reproducibility | Cannot run | Instructions incomplete | README runs the system locally | One command per environment (Compose, K8s) + seeded test data | A new person reproduces a load test from scratch in 30 min |
| Accuracy | Contradicts the code | Outdated in several places | Matches the code at the final tag | Diagrams and numbers updated after every milestone | Changelog of design changes linked to ADRs |
| AI usage | Not declared | Vague | Declared per deliverable | Declared with what was verified | Includes lessons on where AI helped or misled |
Timeline
Weeks are relative to the start of the semester; the dates are on the planning page. Each milestone has a 10-minute check-in in class during its week (demo + questions to any member) and is graded on the tag pushed by the deadline.
| Week | Milestone | What is due |
|---|---|---|
| 3 | Team registration | Team, system option, repository link |
| 5 | M1 Design doc & estimates | Design doc v1, estimates, SLOs, 3+ ADRs, test target agreed |
| 7 | M2 Core path running | The main use case end to end with Docker Compose; gateway + auth; smoke test |
| 9 | M3 Scale & cache | Cache + replica/pool, before/after k6 numbers |
| 12 | M4 Async & resilience | Broker + outbox/idempotency + DLQ; resilience patterns; observability stack |
| 14 | M5 Kubernetes + CI/CD | Deployment on K8s with probes + HPA; CI/CD with scan, SBOM and performance gate |
| 16 | M6 Load-test campaign | Campaign report against the test target |
| 17 | M7 Chaos game day | Game day run in class; report with action items |
| 18 | M8 Cost report | Cost report and final design doc update |
| 19β20 | M9 Final presentation | Presentation, live demo, video; final tag final |
gantt
title Project timeline (relative weeks)
dateFormat X
axisFormat W%s
section Design
M1 Design doc :m1, 3, 5
section Build
M2 Core path :m2, 5, 7
M3 Scale & cache :m3, 7, 9
M4 Async & resilience :m4, 9, 12
M5 K8s + CI/CD :m5, 12, 14
section Prove
M6 Load-test campaign :m6, 14, 16
M7 Game day :m7, 16, 17
section Communicate
M8 Cost report :m8, 17, 18
M9 Presentation :m9, 18, 20 Use of AI
The AI policy of the exercises applies: AI tools are allowed, their use is declared per deliverable, measurements must come from your own runs, and every member must be able to explain and change any part of the system β including the parts written with AI. The final presentation includes individual questions.