Cloud Networking: VPCs, Subnets, Security Groups, and Why NAT Exists
The Layer Where Nothing Is Where You Left It
There is a particular kind of afternoon that almost everyone who works with cloud infrastructure has lost at least once. A database is running. An application is running. They are, according to the console, in the same network. The application cannot reach the database. Every guide found by searching says to check the security groups, so the security groups get checked, and they look fine, and the application still cannot reach the database, and by four o'clock somebody has opened a rule to the entire internet just to see whether that changes anything.
This happens because cloud networking is the one layer with no physical object to point at. A server is a computer, and you can hold that idea. A disk is a disk. But there is no cable in a virtual private cloud, no switch you can trace a wire out of, nothing you can look at to see where the packet stopped. All of it is configuration, and configuration that is invisible tends to get copied rather than understood.
The first lesson in this series introduced the shape of the arrangement: a private network, a public subnet with the load balancer in it, a private subnet with everything else, and a NAT gateway letting connections out but not in. That was the map. This lesson is the machinery underneath it - what a virtual network actually is when there is no wire, how addresses get carved up and why that decision is one of the few you cannot take back, what genuinely makes a subnet public, what NAT is really for, and how the two different firewalls work and why one of them behaves in a way that surprises people.
The examples use AWS names because they are the ones you are most likely to meet, but almost none of this is AWS-specific. Azure calls the network a virtual network and the firewall a network security group; Google calls the network a VPC and the firewall a firewall rule. The concepts underneath survive the rename.
A Virtual Network Has No Wires
Start with the thing that is genuinely strange, because everything else follows from it. Your server has an address, say 10.0.1.47. That address does not exist on any physical network. No switch anywhere has learned it. If you unplugged every cable in the data centre and read the traffic on them, you would not find a packet addressed to 10.0.1.47.
What actually happens is that the hypervisor on the physical host intercepts the packet your server sends, wraps it inside another packet addressed to the physical host where the destination virtual machine is currently running, and puts that on the real network. A mapping service - a distributed database of which virtual address currently lives on which physical machine - tells it where to send it. At the far end, another hypervisor unwraps it and hands the original packet to the destination as though it had arrived over a wire. Your network is a lookup table with an encapsulation format.
This is not trivia. It explains several things that are otherwise confusing. It explains how you and a thousand other customers can all use 10.0.0.0/16 without any of you colliding, because your packets are only ever interpreted in the context of your own network's identifier. It explains why isolation between two virtual private clouds is genuinely strong rather than a naming convention: there is no route for the mapping service to even look up. And it explains why you can create an entire network topology in eight seconds and destroy it in four, which is the property that makes the whole infrastructure-as-code approach from the previous lesson possible at all. You are not cabling anything. You are writing rows into a routing database.
It also explains a class of limits that look arbitrary until you know this. Traffic between instances is capped per instance rather than per switch, jumbo frames have a specific supported size because the encapsulation header has to fit, and you cannot run your own layer-2 protocols or send broadcast traffic. There is no layer 2. There is a lookup table pretending to be one.
One region, many zones
A virtual private cloud lives in exactly one region and spans every availability zone in it. The second lesson in this series covered what an availability zone is: a physically separate facility, close enough for fast links but far enough that one flood does not take out both. The VPC sits across all of them at once.
A subnet does not. A subnet lives in exactly one availability zone, and that single fact is the reason you end up with more subnets than you expected. You do not create a public and a private subnet. You create a public and a private subnet in each zone you intend to survive the loss of. If you have three zones and two tiers, you have six subnets, and each of them is a slice of your address range pinned to one building.
CIDR, Properly This Time
Lesson 1 gave the working rule: 10.0.0.0/16 means the first 16 bits are fixed and the rest are yours, bigger number means smaller range. That rule is correct and it will get you through most days. Here is what is underneath it, because the sizing decisions later in this section do not make sense without it.
An IPv4 address is 32 bits, which humans write as four decimal numbers so they can say them out loud. The number after the slash is the prefix length: how many of those 32 bits identify the network, leaving the remainder to identify hosts inside it.
10.0.1.0/24 written out in the form the machines actually use
10 . 0 . 1 . 0
00001010 00000000 00000001 00000000
|------- network: 24 bits -------|-- host --|
8 bits left
So every address from 00000000 to 11111111 in that last octet
belongs to this subnet: 10.0.1.0 through 10.0.1.255
Widen it to /16 and you free up eight more bits:
00001010 00000000 00000001 00000000
|--- network: 16 bits ---|---- host: 16 bits ----|
10.0.0.0 through 10.0.255.255
The arithmetic is always the same: 2 ^ (32 - prefix) = total addressesPrefix Total addresses Usable in an AWS subnet Rough purpose
/28 16 11 the smallest AWS allows
/24 256 251 a comfortable app tier
/22 1,024 1,019 a large tier
/20 4,096 4,091 a generous per-zone slice
/16 65,536 - a whole VPC
/8 16,777,216 - the entire 10.x private rangeThe gap between the second and third columns is the first thing that catches people. A /28 does not give you 16 addresses to use. AWS reserves five in every single subnet you create, and it is worth knowing which five, because the arithmetic never adds up until you do.
In a subnet 10.0.1.0/24, these five are never yours:
10.0.1.0 the network address itself
10.0.1.1 the VPC router
10.0.1.2 the DNS resolver for this VPC
10.0.1.3 reserved for future use
10.0.1.255 the broadcast address
(nothing broadcasts here, but the address is
still held back so the maths matches convention)
256 - 5 = 251 usableThis is a real operational problem at the small end and people hit it during incidents rather than during planning. A /28 subnet has eleven usable addresses. A load balancer wants several per zone. Every container task on Fargate takes one. Every RDS instance takes one. Teams have had autoscaling silently stop scaling, in the middle of a traffic spike, because the subnet ran out of addresses, and the error surfaces as a launch failure rather than anything mentioning networks.
Two decisions you do not get to revisit
Almost everything in cloud infrastructure is reversible. You picked the wrong instance type, so you change it. You put the load balancer in the wrong subnet, so you move it. Networking has two exceptions, and they are worth spending real thought on because they will outlive your involvement with the system.
The first: you cannot shrink a VPC's address range after creating it. You can add extra ranges alongside the original one, up to a quota, but the primary block is fixed for the life of the VPC, and the only way to change it is to build a new VPC and migrate everything into it. Since private addresses cost nothing and consume nothing until an interface actually uses one, there is no reason to be frugal. Take a /16. It gives you 65,536 addresses and it costs the same as a /24 that gives you 251.
The second, and the more expensive mistake: two networks with overlapping address ranges can never be directly connected. Not peered, not joined over a transit gateway, not reached across a VPN to an office. Routing has no way to decide which of two identical destinations a packet is for, so the connection is simply refused at creation time.
This sounds abstract until you meet it. It is extremely common for a company to have several teams, each of whom independently accepted a console default of 10.0.0.0/16, or copied the same starter Terraform module. For years this is harmless, because the networks never touch. Then two products need to talk to each other, or the company acquires another company, or somebody wants a VPN from the office, and the answer is that one side has to be rebuilt from scratch. The fix costs weeks. The prevention costs one afternoon and a document.
An address plan, which is the entire prevention:
10.0.0.0/16 production, eu-central-1
10.1.0.0/16 staging, eu-central-1
10.2.0.0/16 development, eu-central-1
10.10.0.0/16 production, us-east-1
10.20.0.0/16 the office network, over the VPN
10.90.0.0/16 reserved: acquisitions, partners, anything unplanned
And then, inside production, carving one /16 into six /20 subnets
across three availability zones:
10.0.0.0/20 public, eu-central-1a 10.0.0.1 - 10.0.15.254
10.0.16.0/20 private, eu-central-1a 10.0.16.1 - 10.0.31.254
10.0.32.0/20 public, eu-central-1b 10.0.32.1 - 10.0.47.254
10.0.48.0/20 private, eu-central-1b 10.0.48.1 - 10.0.63.254
10.0.64.0/20 public, eu-central-1c 10.0.64.1 - 10.0.79.254
10.0.80.0/20 private, eu-central-1c 10.0.80.1 - 10.0.95.254
10.0.96.0 onward: still free, which is the point of taking a /16
The private ranges you are choosing from are set by RFC 1918, and there are exactly three: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. The 10 range is the largest by an enormous margin, which is why cloud plans almost always live there. If you ever need to know why your home router hands out 192.168.1.x while your work laptop gets 10.something, that document from 1996 is the entire answer.
What Actually Makes a Subnet Public
Here is a claim that sounds pedantic and turns out to be the single most useful thing in this lesson: there is no such thing as a public subnet. There is no property on a subnet called public. Nothing in the data model records it. The word is a description that people apply to a subnet based on one line in a table somewhere else.
Every subnet is associated with exactly one route table. A route table is a list of destinations and where to send traffic for each. When a packet leaves an interface, its destination address is compared against every route in that table, and the most specific match wins - the one with the longest prefix, because a longer prefix means a narrower and therefore more deliberate statement about where that traffic goes.
The route table attached to the public subnets
Destination Target Meaning
10.0.0.0/16 local anything inside this VPC, hand it
straight over; cannot be deleted
0.0.0.0/0 igw-0a1b2c3d everything else, send to the
internet gateway
The route table attached to the private subnets
Destination Target Meaning
10.0.0.0/16 local same as above, always present
0.0.0.0/0 nat-04f5e6d7 everything else, send to the NAT
gateway, which will let it out but
will not let anyone in
That is the whole difference. One word in one cell.Two things in there deserve attention. The local route for 10.0.0.0/16 is created with the VPC, cannot be removed, and cannot be overridden by anything less specific. This is why every instance in a VPC can route to every other instance in it by default, across subnets and across availability zones, with no configuration at all. Routing is not where your isolation between tiers comes from. That job belongs to the firewalls, which are the next section.
And 0.0.0.0/0 is a prefix length of zero, meaning zero bits are fixed, meaning it matches every address in existence. It is the default route: the answer to "I have no more specific idea where this goes." Because longest match wins, adding a route for 52.94.0.0/16 pointing somewhere else would take precedence over it for exactly that range and leave everything else alone. That mechanism is how VPC endpoints, peering and on-premises links get inserted into an existing network without disturbing anything.
A useful consequence: to make a private subnet public you add one route, and to make a public subnet private you delete one. There is no migration, no rebuild, and no need to move anything. That is also the reason this is worth getting right in code rather than in the console, since a single-row difference between two environments is exactly the kind of drift nobody notices until it matters.
Why NAT Exists, Which Is Two Questions
Lesson 1 described the NAT gateway in one line: it lets connections out and lets the replies back in, while refusing to let anyone outside start a conversation. That is what it does. It is not why it exists, and the difference matters because people routinely treat NAT as a security control when security was never the design goal.
Network address translation was invented for arithmetic. IPv4 has 32 bits, so there are 4,294,967,296 addresses in total, which was an unimaginable number in 1981 and a visibly insufficient one by the early nineties. The response was to reserve a few ranges for private use, let every organisation on earth reuse them internally, and put a device at the boundary that rewrites the source address of outgoing packets to one genuinely public address that the organisation does own. A thousand machines, one public address. That is the whole idea, and it bought the internet roughly thirty extra years.
To do the rewriting, that device has to keep a table, because when a reply comes back to the single public address it needs to know which of the thousand internal machines the reply belongs to. It answers that by also rewriting the source port, giving each conversation a unique number to come back on.
The translation table, which is all a NAT gateway really is
Outbound: your app server calls an external API
what the server sends 10.0.16.32:51234 -> 93.184.216.34:443
what leaves the gateway 52.28.7.19:40001 -> 93.184.216.34:443
^^^^^^^^^^^^^^^^
source rewritten, and the pair
(40001, 93.184.216.34:443) is written
into the table alongside 10.0.16.32:51234
Inbound: the reply arrives
what arrives 93.184.216.34:443 -> 52.28.7.19:40001
table lookup on 40001 found: 10.0.16.32:51234
what gets delivered 93.184.216.34:443 -> 10.0.16.32:51234
Inbound: somebody on the internet tries to start a conversation
what arrives 203.0.113.9:6666 -> 52.28.7.19:40001
table lookup on 40001 the entry is for 93.184.216.34, not this
sender, so there is nothing to map it to
what gets delivered nothing. the packet is dropped.Read that last block again, because it is the point. The inbound packet is not refused by a rule. Nobody wrote a policy saying to reject it. It is dropped because the gateway genuinely does not know which internal machine it could possibly be for. The one-way behaviour is not a feature that was designed; it is an unavoidable consequence of solving an addressing problem with a lookup table.
Do not treat NAT as your inbound protection. It is an addressing mechanism whose side effect resembles a firewall, and the resemblance breaks down as soon as anything else is involved: a port forward, a peering connection, a VPN, or a compromised machine inside the subnet that opens the outbound connection itself and then does whatever it likes over it. The security group is doing the actual policy work. NAT is doing arithmetic.
There is a neat proof that these really are two separate concerns rather than one. IPv6 has 128 bits and therefore no scarcity problem at all, so there is nothing for translation to solve, and AWS does not offer a NAT gateway for it. What it offers instead is an egress-only internet gateway: a device that provides exactly the one-way property, outbound allowed and inbound refused, while performing no address translation whatsoever. Same behaviour, none of the rewriting. The two reasons come apart cleanly once the scarcity is gone.
On cost, lessons 2 and 5 in this series have already made the case at length and it will not be repeated here, beyond the shape of it: a NAT gateway bills by the hour whether or not anything uses it, bills again per gigabyte it processes, and you need one per availability zone if you want the zone-failure protection you built the multiple subnets for in the first place. Three zones means three of them, running continuously.
A large share of the traffic paying that per-gigabyte charge is going to the cloud provider's own services, which is faintly absurd - a private instance reaching S3 goes out through the NAT gateway, across the public internet path, and back into the same provider. A gateway VPC endpoint fixes this by adding a route so that S3 and DynamoDB traffic never leaves the VPC, and it is free. It is one of very few changes that reduces both your bill and your exposure at the same time. Interface endpoints, which cover most other services, do charge per hour, so those are a judgement call rather than an automatic yes.
The Two Firewalls, and Why One of Them Surprises You
There are two independent filtering layers in a VPC, and every packet has to satisfy both. They work differently enough that knowing which is which turns an afternoon of guessing into a two-minute check.
Security groups
A security group attaches to a network interface, not to a subnet. This is the first thing that makes it unlike the firewall you may be picturing. A traditional firewall sits at a boundary and everything crossing that boundary is subject to it. A security group travels with the machine, which means two servers sitting in the same subnet can have entirely different rules, and moving a server to another subnet changes nothing about its firewall.
It contains only allow rules. There is no deny. The absence of a matching allow is the denial, and the default for a new group is to allow nothing inbound and everything outbound. And it is stateful, which is the property that matters most in practice: if a connection is allowed in, its replies are allowed back out automatically, no matter what the outbound rules say. You never write a return rule.
The genuinely valuable feature, and the one most tutorials skip in favour of typing IP ranges, is that the source of a rule does not have to be an address range. It can be another security group. Read that rule as "anything carrying this badge may enter" rather than "anything from this address may enter."
A three-tier application, written the way that keeps working
sg-alb the load balancer
inbound 443 from 0.0.0.0/0 the public, as intended
inbound 80 from 0.0.0.0/0 only to redirect to 443
outbound 8080 to sg-app
sg-app the application servers
inbound 8080 from sg-alb <- a group, not an address
outbound 5432 to sg-db
outbound 443 to 0.0.0.0/0 calling external APIs
sg-db the database
inbound 5432 from sg-app <- a group, not an address
outbound (nothing configured; replies still work, because stateful)
Now scale the application tier from two servers to fifty.
The new servers launch with sg-app attached.
The database rule already permits them. Nothing to update, and
nothing to forget to update.Compare that with the version written using address ranges. The database allows 10.0.16.0/20, which is the whole private subnet in one zone, so you also need 10.0.48.0/20 and 10.0.80.0/20 for the others. Now the database is reachable from anything at all that happens to be in those subnets, including a debugging container somebody started last month and forgot. The group-referencing version says what you actually mean: the database accepts connections from the application, and from nothing else, regardless of where anything is sitting.
The single most common serious mistake in this area is 0.0.0.0/0 on port 22, or 3389 for Windows. Automated scanners find a newly opened SSH port in minutes, not days, and from there it is a credential-guessing problem rather than a network problem. The modern answer is not a better bastion host, it is to have no inbound rule at all: Session Manager reaches the instance through an agent that dials outbound, so you get a shell on a machine whose inbound rules are completely empty.
Network ACLs, and the stateless trap
The second layer sits at the subnet boundary. A network ACL is evaluated for every packet entering or leaving the subnet, its rules are numbered and evaluated in order until one matches, and unlike a security group it can express an explicit deny.
It is also stateless, and that word is where the afternoon goes. Stateless means the ACL keeps no record of connections. It sees each packet in isolation, with no idea whether it is a reply to something. So allowing a request in does not allow the response out. You have to write that rule yourself, and to write it you need to know which port the response will be sent to.
What a security group needs to accept HTTPS:
inbound allow tcp/443 from 0.0.0.0/0
Done. Replies are allowed out because the group is stateful.
What a network ACL needs, for exactly the same traffic:
INBOUND
100 allow tcp/443 from 0.0.0.0/0 the request
* deny all from 0.0.0.0/0 implicit, unremovable
OUTBOUND
100 allow tcp/1024-65535 to 0.0.0.0/0 the reply <- this one
* deny all to 0.0.0.0/0 implicit, unremovable
Why that range: the client picked a random high-numbered port to
receive the answer on, so the reply is addressed there rather than
to 443. Linux picks from 32768-60999 by default, but you are not
filtering your own clients, so the safe range to permit is the wide
one. Miss this rule and the symptom is a connection that opens and
then hangs, which reads like an application problem and is not.
The honest guidance about network ACLs is that most architectures should leave them alone. The default one permits all traffic in both directions, and that is a reasonable place for it to stay, because a subnet-wide stateless filter is a blunt instrument and every rule you add to it is a rule you have to remember exists at three in the morning. Put your policy in security groups, where it is stateful, attached to the thing it describes, and can reference other groups.
There is one job they do that security groups genuinely cannot, and it is the reason to keep them in mind: an explicit deny. If a specific address range is scanning you, or a compliance rule requires a hard block at the network edge, a security group has no way to express that at all, because it only knows how to allow. The ACL is where a deny lives. Use it for that, and resist using it for anything else.
When Traffic Does Not Flow: Check in This Order
Back to the lost afternoon from the opening. The reason it takes an afternoon is almost never that the problem is hard. It is that there are seven plausible causes and people check them in the order they happen to think of them, usually starting with the one they most recently read about. Checking them in a fixed order, outermost to innermost, turns it into a short exercise. Each rung either passes or gives you your answer.
# 1. Is there a route to the destination at all?
# Different VPC, on-premises, or internet: check before anything else.
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-0a1b2c3d
# 2. Network ACL, and remember it is stateless: check BOTH directions.
# If the default has been modified, this is a strong suspect.
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=subnet-0a1b2c3d
# 3. The DESTINATION's security group: is there an inbound rule for
# this port from this source? Usually the answer.
aws ec2 describe-security-groups --group-ids sg-0db1234
# 4. The SOURCE's security group outbound rules. Default allows
# everything, so this only bites when somebody tightened it.
# 5. The host's own firewall, which the cloud console cannot see.
sudo iptables -L -n # or: sudo ufw status verbose
# 6. Is the process even listening, and on which interface?
ss -tlnp | grep 5432
# 127.0.0.1:5432 -> listening on loopback only. Nothing outside
# the machine can ever reach it. Not a network
# problem, a config problem. This is number one
# on the list of things mistaken for firewalls.
# 0.0.0.0:5432 -> listening on all interfaces, as intended.
# 7. DNS: did the name resolve to the private address or the public one?
dig +short db.internal.example.com
# A private address means you are going straight across the VPC.
# A public one means you are leaving and coming back, which will
# hit different rules, cost egress, and fail from a private subnet.Before working through that list by hand, try VPC Reachability Analyzer. You give it a source and a destination and it evaluates the entire path statically - route tables, ACLs, security groups, gateways - without sending a single packet, and tells you which specific component blocked it. It answers rungs one to four in about thirty seconds. It cannot see inside the instance, so rungs five and six are still yours, but it removes most of the search space.
What to Take From This
The through-line of this lesson is that cloud networking looks arbitrary until you know which layer each piece belongs to, and then it stops. A virtual network is a lookup table, not a wire. A subnet is a slice of addresses pinned to one building. Public is not a property of a subnet, it is a description of its route table. NAT is arithmetic that happens to behave like a door. A security group travels with the machine and remembers your connections; a network ACL guards the subnet and remembers nothing.
Two things are worth doing something about rather than just knowing. Write down an address plan before you create your second VPC, because overlapping ranges are the one mistake here with a cost measured in weeks. And write your security group rules as references to other groups rather than address ranges, because that is the version that still says something true after the architecture has changed twice.
The next lesson in this roadmap is identity and access: least privilege, what a role actually is as distinct from a user, and how cloud accounts really get breached, which turns out to have very little to do with anything in this lesson and a great deal to do with a credential somebody committed to a repository.
Further reading
- AWS: VPC User GuideThe reference for everything named in this lesson, including the subnet sizing rules and the five reserved addresses.
- RFC 1918: Address Allocation for Private InternetsThe 1996 document that defined 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, and the reason your address plan starts with a 10.
- RFC 2663: NAT Terminology and ConsiderationsWhere the address-scarcity motivation is set out directly, including a section on why NAT should not be treated as a security mechanism.
- AWS: Compare Security Groups and Network ACLsThe official stateful-versus-stateless table, and the ephemeral port guidance behind the ACL rules above.
- AWS: VPC Reachability AnalyzerThe tool that replaces the first four rungs of the debugging ladder with a single query.
- AWS: Gateway Endpoints for S3 and DynamoDBThe free route-table change that keeps S3 traffic off your NAT gateway, mentioned in the cost note above.
- AWS: Egress-Only Internet GatewaysThe IPv6 device that provides NAT's one-way behaviour without any address translation, which is the clearest evidence the two purposes are separate.