How the Internet Works

At 10:00 a fan in São Paulo taps Buy on a phone connected to the home Wi-Fi. A few milliseconds later a pod with the address 10.20.40.17 charges the card through a payment provider that accepts requests from exactly one address in the world. Between the two there are a dozen routers, at least three address translations and four route tables — and none of them knows anything about BoxOffice. This page explains that path from the beginning: what an IP address is, why some addresses are public and others private, how a block of addresses is cut into subnets, and how NAT lets millions of private devices share a few public addresses. The next page builds the BoxOffice network in the cloud on top of these ideas, and the rest of the class secures it.

Layers: who decides what

A request is not one thing on the wire. It is an HTTP message inside a TCP segment, inside an IP packet, inside a Wi-Fi or Ethernet frame. Each layer has its own address and answers a different question:

flowchart LR
    subgraph F["Wi-Fi frame (MAC → MAC): rewritten at every hop"]
        subgraph P["IP packet: src 192.168.0.12 → dst 203.0.113.10"]
            subgraph T["TCP segment: port 51514 → 443"]
                H["HTTP: POST /orders (inside TLS)"]
            end
        end
    end

Two facts follow from the picture and explain most of this page:

• Routers look only at the IP header. They never read the URL; the destination IP decides the path. The frame around the packet is thrown away and rebuilt at every hop.
• Ports belong to the transport layer, but NAT devices and firewalls read them too: that is how one public address is shared, and how a security group allows 5432 and refuses 22.

How a packet crosses the Internet

The Internet is a network of networks: tens of thousands of autonomous systems (AS) — carriers, universities, cloud providers, large companies — each with its own routers and its own blocks of addresses. Each AS announces its blocks to its neighbours with BGP (Border Gateway Protocol)¹: "send me the packets for 203.0.113.0/24". The announcements propagate until every core router has a route to about a million IPv4 prefixes.

No router knows the whole path. Each one does the same thing for every packet:

1. Read the destination IP.
2. Find the most specific route that contains it in its table (longest prefix match: a route to /24 beats a route to /16, which beats the default route 0.0.0.0/0).
3. Send the packet to the next hop of that route, and decrement the packet's TTL (a hop counter that kills packets caught in loops).

Press play to follow one packet from the fan's phone to BoxOffice (the addresses are documentation examples²):

sequenceDiagram
    autonumber
    participant P as Phone 192.168.0.12
    participant R as Home router (NAT)
    participant C as Carrier (AS of the ISP)
    participant X as Transit / exchange point
    participant E as Cloud provider edge
    participant G as Internet gateway (VPC)
    participant L as Load balancer 10.20.0.10
    P->>R: dst 203.0.113.10 is not in 192.168.0.0/24 → default gateway
    R->>C: NAT: src becomes 198.51.100.7 (the home's public IP)
    C->>X: longest match for 203.0.113.10: route learned with BGP
    X->>E: next hop: the network that announced 203.0.113.0/24
    E->>G: inside the provider: the VPC that owns 203.0.113.10
    G->>L: 1:1 NAT: dst 203.0.113.10 becomes 10.20.0.10
    L-->>P: the reply takes the same translations backwards

Run traceroute (Linux, macOS) or tracert (Windows) to any site and each output line is one router of this list, with its round-trip time. A typical path has 10 to 20 hops; the time is dominated by distance (light in fibre covers about 200 km per millisecond), not by the number of routers — which is why the edge puts CDNs close to the fans.

IP addresses and CIDR

An IPv4 address is a 32-bit number, written as four decimal octets: 10.20.40.17 is 00001010.00010100.00101000.00010001. Every address has two parts:

• the network part (the first n bits) says which network the host is on — it is what routers match;
• the host part (the other 32 − n bits) says which host inside that network.

CIDR notation (Classless Inter-Domain Routing)³ writes the split after a slash: 10.20.0.0/16 is "the 2³²⁻¹⁶ = 65 536 addresses that start with the same 16 bits as 10.20.0.0". Each extra bit in the prefix halves the block: a /17 is half a /16, a /24 has 256 addresses, a /32 is a single host. The same split is sometimes written as a mask: /16 is 255.255.0.0. In a classic network the first address of a block names the network and the last one is the broadcast, so a /24 has 254 usable hosts; AWS reserves 5 addresses in every subnet.

Try it:

1. The explorer starts with 10.20.40.17/19. The host bits are not zero, so the explorer tells you the block that contains this address: 10.20.32.0/19, the private app subnet of BoxOffice in AZ a. Drag the prefix slider from /19 to /24 and watch the block shrink by half at every step.
2. Type 10.20.0.0/16, choose Split into 4 subnets and test the address 10.20.40.17: in which of the four /18 blocks does it fall?
3. Click the examples: 192.168.0.12 (your home), 100.64.7.9 (your mobile carrier's NAT), 169.254.169.254 (the cloud metadata service), 8.8.8.8 (public). The kind of address line says which ones the Internet can route.

Public and private addresses

Thirty-two bits give 2³² ≈ 4.3 billion addresses — fewer than the phones alive today, without counting servers, cars and light bulbs. The registries handed out the last free IPv4 blocks between 2011 and 2019. The Internet survived with two ideas:

• Some blocks are private (RFC 1918)⁴: any network may use them internally, as many times as it wants, and no one routes them on the Internet.
• NAT (next section) lets many private hosts share one public address, unique in the world and announced with BGP.

The sizes are computed from the prefixes:

A public address is what makes a host reachable from the Internet: someone announces its block, so packets to it find a route. A private address is reachable only inside its own network and through whatever translates or routes into it. In BoxOffice, only two things have public addresses — the load balancer (to receive fans) and the NAT gateway (to call the payment provider from a fixed address). Every service, pod and database has only a private one.

NAT: many private hosts behind one public address

NAT (Network Address Translation) rewrites addresses in the IP header as packets cross a device. The most common form, source NAT (also called PAT or masquerading), is what your home router, your mobile carrier and the cloud's NAT gateway do: they replace the private source of every outgoing packet with their own public address and a free port, and write the pair down in a translation table.

sequenceDiagram
    autonumber
    participant P as Phone 192.168.0.12
    participant N as Home router (NAT) public 198.51.100.7
    participant S as Server 203.0.113.10
    participant A as Someone on the Internet
    P->>N: src 192.168.0.12:51514 → dst 203.0.113.10:443
    Note over N: table: 192.168.0.12:51514 ⇄ 198.51.100.7:40001
    N->>S: src 198.51.100.7:40001 → dst 203.0.113.10:443
    S-->>N: reply to 198.51.100.7:40001
    Note over N: found in the table → back to 192.168.0.12:51514
    N-->>P: reply delivered
    A->>N: a new connection to 198.51.100.7:22
    Note over N: no entry for port 22 → dropped

Four consequences show up again and again in system design:

1. Connections start only from inside. An outside host can reply, but it cannot open a new connection: there is no table entry to tell the NAT where to send it. This is why pods in a private subnet can call the payment provider while nobody on the Internet can call them.
2. The server sees the NAT, not the client. Thousands of fans behind one carrier-grade NAT arrive at BoxOffice with the same IP. A rate limit per IP would block a whole neighbourhood at 10:00 — the reason the edge limits per API key or user and treats per-IP limits as a coarse defence.
3. A NAT gives a stable egress address. The pods move between nodes and their IPs change, but everything they send to the Internet leaves with the NAT's fixed public IP — the one BoxOffice registers in the payment provider's allow-list (cloud networking).
4. Destination NAT publishes a private service. The reverse rewrite — "packets to my public :443 go to this private host" — is what a load balancer, a cloud internet gateway (1:1 NAT for public IPs) and Docker's ports: "80:80" do.

A NAT also has a capacity: each connection to the same destination needs its own public port, and a public address has at most 65 535 of them. An AWS NAT gateway supports up to 55 000 simultaneous connections to each unique destination⁶ — a real limit when hundreds of pods call the same payment API during an on-sale.

Subnets and the gateway

A host does not route packets the way the Internet core does. Its network configuration has only three numbers — its address, its prefix and its default gateway (plus a DNS server) — and it makes one decision for every packet:

flowchart TD
    A["packet to dst"] --> B{"is dst inside<br/>my subnet (address/prefix)?"}
    B -->|"yes"| C["find dst's MAC (ARP)<br/>and send it directly"]
    B -->|"no"| D["send it to the default gateway<br/>(the router of the subnet)"]
    D --> E["the router looks up its route table:<br/>longest prefix match → next hop"]
    E --> F["…until a router delivers it<br/>inside dst's subnet"]

A subnet is therefore a block of addresses whose hosts talk to each other directly; anything else goes through a router. Designs cut the address space into subnets for three reasons:

• Placement: in the cloud a subnet lives in one availability zone, so a subnet per zone lets you spread replicas and lose a zone without losing a service.
• Routing policy: each subnet has a route table, and the route table is what makes it public (a route to an internet gateway), private (a route to a NAT) or isolated (no route out).
• Blast radius and rules: firewalls and flow logs are attached per subnet; a mistake in one tier does not open the others.

A route table is a list of destination block → target. Every table in a VPC has the local route for the VPC's own block; the rest decides the tier:

When a packet to 10.20.8.10 leaves a private subnet, both routes "contain" it (0.0.0.0/0 contains everything), and the longest prefix wins: /16 beats /0, so it stays inside the VPC.

The packet walk: the BoxOffice VPC

Now put the pieces together. The simulator below is a small version of the BoxOffice network (one availability zone): a public subnet with the load balancer and the NAT gateway, a private subnet with the order and payment pods, and an isolated subnet with PostgreSQL. Each step shows where the packet is, its source and destination (the fields that just changed are highlighted), the route table or security group consulted, and the row that matched.

Try it:

1. A fan opens the site. Count the address rewrites: the home router changes the source (private → public), the internet gateway changes the destination (public → the load balancer's private address). Then the load balancer opens a second connection, inside the VPC, to order.
2. payment calls the provider. The pod's private address 10.20.40.17 is rewritten twice on the way out (NAT, then internet gateway); the provider sees only 203.0.113.20 and finds it in its allow-list. Follow the reply back through the NAT table.
3. A fan tries a pod directly. The packet never reaches BoxOffice: no Internet router has a route to a private address.
4. Break the network. Untick route 0.0.0.0/0 → NAT and rerun scenario 2; untick sg-data accepts 5432 and run order queries PostgreSQL; untick the load balancer has a public IP and run scenario 1. Each switch is a real misconfiguration that has taken sites down.
5. Scanners. Try port 5432 on the NAT's IP and on the load balancer's IP: one is dropped by the NAT (no translation), the other by a security group.

# three identical replicas; no "ports:" — only the edge is reachable from outside
catalog-1:
  <<: *catalog
  hostname: catalog-1
catalog-2:
  <<: *catalog
  hostname: catalog-2
catalog-3:
  <<: *catalog
  hostname: catalog-3

edge:
  image: nginx:1.30-alpine
  ports:
    - "80:80"
  volumes:
    - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
  depends_on:
    - catalog-1
    - catalog-2
    - catalog-3

resolver  127.0.0.11  valid=10s  ipv6=off;

upstream catalog {
    zone catalog 64k;                 # shared memory, required by "resolve"
    # least_conn;                     # try it: pick the replica with fewest active requests
    server catalog-1:8082 resolve max_fails=3 fail_timeout=10s;
    server catalog-2:8082 resolve max_fails=3 fail_timeout=10s;
    server catalog-3:8082 resolve max_fails=3 fail_timeout=10s;

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: boxoffice
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080   # NodePort of the Ingress controller (Traefik)
        hostPort: 80           # http://localhost
        protocol: TCP
  - role: worker
  - role: worker

IPv6 in one paragraph

IPv6 addresses have 128 bits (2¹²⁸ ≈ 3.4 × 10³⁸), written in hexadecimal groups such as 2001:db8:20::17. There are enough for every device to have a globally routable address, so IPv6 networks normally use no NAT — which means that with IPv6 a pod could be reachable from the Internet, and only the route tables and firewalls (security groups) decide whether it is. Clouds and carriers run dual-stack (IPv4 and IPv6 side by side), and public IPv4 has become a priced resource: since February 2024 AWS charges for every public IPv4 address, in use or not⁷. The design ideas of this page — subnets, route tables, longest prefix match, firewalls — are the same in IPv6.

How it connects

flowchart LR
    L["layers: frame, packet,<br/>segment, request"] --> IP["IP address:<br/>network + host bits (CIDR)"]
    IP --> PP["public vs private<br/>(RFC 1918, CGNAT)"]
    PP --> NAT["NAT: shared public address,<br/>stable egress IP"]
    NAT --> RL["per-IP limits are unfair<br/>→ Edge: rate limiting"]
    IP --> SUB["subnets + default gateway"]
    SUB --> RT["route tables:<br/>longest prefix match"]
    RT --> VPC["public / private / isolated<br/>→ Cloud networking"]
    BGP["BGP: networks announce prefixes"] --> RT
    PP --> ZT["private ≠ trusted<br/>→ Zero Trust"]
    VPC --> SG["security groups<br/>→ Cloud networking"]

Palette components: vpc · subnet · nat · lb (the only public address that receives traffic) · dns (the name that leads to it). Continues in: Cloud networking (the BoxOffice VPC, security groups, endpoints), Edge — the request's journey and rate limiting, Zero Trust, Hands-on 3 and Hands-on 10.

Key takeaways