Cloud Networking

On a developer laptop, BoxOffice lives in one Docker network: every container can reach every other container, and only Nginx publishes a port. In the cloud, the equivalent decision is far bigger and much harder to change later. Which machines get a public address? Can the database reach the Internet — and can the Internet reach the database? How does the payment service call the provider, which only accepts requests from allow-listed IP addresses? How does an engineer debug a pod without opening SSH to the world? This page builds the BoxOffice network in a cloud provider, one component at a time.

The examples use AWS names, the most common in job descriptions; the tabs at the end map them to Google Cloud and Azure. The ideas are the same everywhere¹. This page assumes you know what an IP address, a CIDR block, a private address and NAT are — if not, read How the Internet works first: it has the same BoxOffice network as an interactive packet walk.

The VPC: your own private network in the cloud

A VPC (Virtual Private Cloud) is a logically isolated network inside a cloud region. You choose its address range, divide it into subnets, and decide every route in and out. Nothing enters or leaves a VPC unless you add a component and a route that allows it.

CIDR planning

Addresses are written in CIDR notation (Classless Inter-Domain Routing): 10.20.0.0/16 means "the first 16 bits are fixed", leaving 32 − 16 = 16 bits for hosts, i.e. 2¹⁶ = 65 536 addresses. Each extra bit in the prefix halves the block.

The VPC should use a private range (RFC 1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)². Two rules save a lot of pain later:

1. Do not overlap with other networks you may ever connect to — the office, another VPC, a partner. Two networks with overlapping ranges cannot be routed to each other (peering is refused).
2. Leave room: a VPC range is hard to change. Plan for more AZs and more subnet tiers than you need today.

AWS reserves 5 addresses in every subnet (network address, VPC router, DNS, one reserved for future use, broadcast)¹. The table below is computed, not typed:

Check the plan in the address explorer: split 10.20.0.0/16 into 8 subnets. The three app subnets are the /19 blocks 2, 3 and 4; the public /24s and the data /22s all fit in the first /19 (10.20.0.0 to 10.20.31.255); blocks 5 to 8 — half of the VPC — are free. Test 10.20.17.4: which tier would own it?

Public, private and isolated subnets

A subnet is not "public" because of a checkbox. It is public because of its route table: it has a route 0.0.0.0/0 → Internet Gateway. Everything else follows from routes.

Every route table also has an implicit local route (10.20.0.0/16 → local): all subnets of the VPC can reach each other at the routing level. Isolation between tiers is therefore the job of security groups and NACLs, not of routes.

The simulator walks one packet through these three tiers. It starts with payment calling the provider: follow the route-table lookup (0.0.0.0/0 → NAT), the two rewrites of the source address and the reply coming back through the NAT table. Then untick route 0.0.0.0/0 → NAT — the misconfiguration behind many "the payment provider is down" incidents — and run it again; and run PostgreSQL tries to download an update to see what an isolated subnet means.

Internet Gateway and NAT gateway

• Internet Gateway (IGW) — attached to the VPC; lets resources with public IPs send and receive Internet traffic. It is horizontally scaled and highly available by design.
• NAT gateway — lives in a public subnet with a fixed public address (an Elastic IP). Private resources send outbound traffic to it; it rewrites the source address to its own and keeps a translation table, so the replies find their way back. Connections can only be started from inside.

sequenceDiagram
    autonumber
    actor U as Fan (mobile)
    participant IGW as Internet Gateway
    participant ALB as Load balancer (public subnet)
    participant PAY as payment pod 10.20.40.17 (private subnet)
    participant NAT as NAT gateway 15.228.x.x (public subnet)
    participant PSP as Payment provider
    U->>IGW: HTTPS checkout.boxoffice.com
    IGW->>ALB: to the ALB's public address
    ALB->>PAY: 10.20.40.17:8080 (via gateway → order)
    PAY->>NAT: src 10.20.40.17 → dst PSP:443 (route 0.0.0.0/0 → NAT)
    NAT->>PSP: src 15.228.x.x (Elastic IP, allow-listed)
    PSP-->>NAT: 200 charge authorized
    NAT-->>PAY: translated back to 10.20.40.17
    PAY-->>U: order confirmed (through ALB)

The provider sees one stable source address — the NAT gateway's Elastic IP — which BoxOffice registers in the provider's allow-list. Pods can be rescheduled to any node; the egress address never changes.

Cost: the surprise line on the bill

NAT gateways charge per hour and per GB processed. Traffic that does not need to go to the Internet — pulling images from the registry, writing to object storage, reading secrets — still goes through NAT if nothing else is configured. The fix is a VPC endpoint:

• Gateway endpoints (S3, DynamoDB on AWS) — a route-table entry; free.
• Interface endpoints (PrivateLink) — a private IP inside your subnet that reaches a cloud service (Secrets Manager, ECR, KMS…) or a partner's service without crossing the Internet; billed per hour and per GB, usually cheaper than NAT for heavy traffic.

Security groups and NACLs

Routes decide where packets can go; firewalls decide whether they are allowed. AWS has two, at different levels:

Treat them as layers of the same defense in depth⁴. The most useful feature is SG references: "allow port 5432 from the security group of the app nodes" instead of "from 10.20.32.0/19". The rule follows the workloads as they scale, move or change IP.

flowchart LR
    subgraph sgalb ["sg-alb"]
        ALB["ALB :443"]
    end
    subgraph sgapp ["sg-app"]
        N["EKS nodes :8080"]
    end
    subgraph sgdb ["sg-db"]
        DB[("RDS :5432")]
    end
    I(["Internet 0.0.0.0/0"]) -->|"443"| ALB
    ALB -->|"8080 from sg-alb"| N
    N -->|"5432 from sg-app"| DB

The BoxOffice VPC

Putting it together (two of the three AZs drawn):

flowchart TB
    NET(["Internet"]) --- IGW["Internet Gateway"]
    subgraph vpc ["VPC 10.20.0.0/16 — sa-east-1"]
        IGW
        subgraph aza ["AZ a"]
            subgraph puba ["public 10.20.0.0/24"]
                ALBa["ALB node"]
                NATa["NAT gw + EIP"]
            end
            subgraph appa ["private app 10.20.32.0/19"]
                Ka["EKS nodes<br/>gateway · order · payment · …"]
            end
            subgraph dataa ["isolated data 10.20.8.0/22"]
                RDSa[("PostgreSQL primary")]
                Ra[("Redis")]
            end
        end
        subgraph azb ["AZ b"]
            subgraph pubb ["public 10.20.1.0/24"]
                ALBb["ALB node"]
                NATb["NAT gw + EIP"]
            end
            subgraph appb ["private app 10.20.64.0/19"]
                Kb["EKS nodes"]
            end
            subgraph datab ["isolated data 10.20.12.0/22"]
                RDSb[("PostgreSQL standby")]
                Rb[("Redis replica")]
            end
        end
        EP["VPC endpoints<br/>S3 · ECR · Secrets Manager"]
    end
    IGW --> ALBa & ALBb
    ALBa --> Ka
    ALBb --> Kb
    Ka --> RDSa & Ra
    Kb --> RDSa
    NATa <-.-|"egress 0.0.0.0/0 → IGW"| Ka
    NATb <-.-|"egress 0.0.0.0/0 → IGW"| Kb
    Ka & Kb -.-> EP
    RDSa -.->|"sync replication"| RDSb

As code

Networks are declared with infrastructure as code (IaC) so they can be reviewed, versioned and recreated. A trimmed Terraform/OpenTofu sketch of one AZ (the real module loops over the AZs):

resource "aws_vpc" "boxoffice" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  tags = { Name = "boxoffice-prod" }
}

resource "aws_subnet" "public_a" {
  vpc_id            = aws_vpc.boxoffice.id
  cidr_block        = "10.20.0.0/24"
  availability_zone = "sa-east-1a"
}

resource "aws_subnet" "app_a" {
  vpc_id            = aws_vpc.boxoffice.id
  cidr_block        = "10.20.32.0/19"
  availability_zone = "sa-east-1a"
}

resource "aws_subnet" "data_a" {
  vpc_id            = aws_vpc.boxoffice.id
  cidr_block        = "10.20.8.0/22"
  availability_zone = "sa-east-1a"
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.boxoffice.id
}

resource "aws_eip" "nat_a" {
  domain = "vpc"
}

resource "aws_nat_gateway" "a" {
  subnet_id     = aws_subnet.public_a.id # NAT lives in the PUBLIC subnet
  allocation_id = aws_eip.nat_a.id       # fixed egress IP for the PSP allow-list
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.boxoffice.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

resource "aws_route_table" "app_a" {
  vpc_id = aws_vpc.boxoffice.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.a.id # egress only, same AZ
  }
}

# data subnets get a route table with no 0.0.0.0/0 route at all

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "app_a" {
  subnet_id      = aws_subnet.app_a.id
  route_table_id = aws_route_table.app_a.id
}

resource "aws_security_group" "app" {
  name   = "sg-app"
  vpc_id = aws_vpc.boxoffice.id
}

resource "aws_security_group" "db" {
  name   = "sg-db"
  vpc_id = aws_vpc.boxoffice.id

  ingress {
    description     = "PostgreSQL only from the app nodes"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id] # reference, not a CIDR
  }
}

Read it as a diagram: the only 0.0.0.0/0 → igw route is in the public table; the app table sends the Internet to NAT; the data subnets have no default route; the database accepts 5432 only from members of sg-app.

Try it. Open the payment-system scenario. The payment service calls an external provider. Increase the load and inject chaos on the provider: note that every call crosses the network boundary (in the real deployment, through the NAT gateway). Ask yourself which failures belong to the network (NAT in one AZ, egress IP not allow-listed) and which to the provider. Rebuild the design in the full playground with vpc, subnet and nat components to make the boundaries visible.

Private connectivity: reaching other networks

BoxOffice soon needs to talk to networks it does not own: the office, the data team's VPC, a partner, a second region.

Getting engineers in: bastion vs session manager

The old answer is a bastion host: a small VM in a public subnet with SSH open, from which engineers jump into the private network. It is an attack surface (port 22 on the Internet) and a key-management problem. The modern answer is an agent-based session service (AWS Systems Manager Session Manager, GCP IAP TCP forwarding, Azure Bastion): the engineer authenticates with the company identity provider and MFA, the session is opened by an agent from the inside, no inbound port is open, and every command is logged. For Kubernetes, kubectl through the cluster's authenticated API plays the same role. This is Zero Trust applied to people (Zero Trust).

DNS inside the VPC

Services find each other by name. The VPC provides a resolver (in AWS, at the VPC base address + 2); private hosted zones give internal names like db.boxoffice.internal that resolve only inside the VPC. Inside Kubernetes, CoreDNS resolves payment.boxoffice.svc.cluster.local. Interface endpoints rely on private DNS so that secretsmanager.sa-east-1.amazonaws.com resolves to the private endpoint IP instead of the public one.

The same ideas in other clouds

The vocabulary changes between providers, the building blocks do not³. The biggest real difference: a Google Cloud VPC is global, while AWS and Azure networks are regional.

Trade-offs at a glance

How it connects

flowchart LR
    VPC["VPC + CIDR plan"] --> SUB["public / private / isolated<br/>subnets, one per AZ"]
    SUB --> RT["route tables:<br/>IGW, NAT or none"]
    RT --> NAT["NAT gateway per AZ<br/>stable egress IP"]
    NAT --> PSP["provider allow-list<br/>→ Compute integrations"]
    SUB --> SG["security groups + NACLs"]
    SG --> NOID["not an identity: NetworkPolicy,<br/>mesh policy → Service mesh"]
    SUB --> AZ["AZ and region outages<br/>→ Reliability"]
    VPC --> CON["peering, transit, PrivateLink,<br/>VPN, session manager"]
    CON --> ZT["access for people<br/>→ Zero Trust"]
    VPC --> IAC["Terraform / OpenTofu<br/>→ Cloud-native"]

Palette components: vpc · subnet · nat · vpn · lb (the ALB in the public subnets) · payment (reached through the NAT). Scenarios: payment-system, multi-region, scale-out. Continues in: Service mesh, Zero Trust, Compute — integrations, Reliability — disaster recovery, Cloud-native — delivery (infrastructure as code).

Key takeaways