IPsec: Internet Protocol Security

What IPsec is

IPsec (Internet Protocol Security), defined in RFC 4301, protects IP packets at the network layer rather than the transport or application layer — the layer where TLS and SSH operate. Because it works below transport, it protects everything riding on top of IP without any application needing to know IPsec exists: TCP, UDP, and every protocol built on them all get protected automatically. This is exactly why IPsec is the standard choice for VPNs: a site-to-site VPN between two offices, or a remote-access VPN for an individual laptop, both rely on IPsec to make an untrusted network (the public internet) carry traffic as if it were on a private, trusted link.

IKE and security associations

Before any traffic is protected, two peers need to agree on keys and algorithms — the job of IKEv2 (Internet Key Exchange version 2, RFC 7296). The result of a successful negotiation is a Security Association (SA): an agreed set of algorithms, keys, and parameters both sides store and use to process matching traffic. IKEv2 negotiates in two stages: first an IKE SA, an encrypted and authenticated control channel built with a Diffie-Hellman exchange (the same primitive from the fundamentals lesson); then, inside that already-protected channel, the peers authenticate each other and negotiate one or more Child SAs — the SAs that actually protect real traffic. This two-stage design is why IKEv2 needs only two round trips to go from nothing to a fully negotiated tunnel, versus IKEv1's considerably longer exchange.

AH and ESP

IPsec defines two protocols for actually protecting packets. AH (Authentication Header, IP protocol 51) authenticates the entire packet, including the outer IP header, but provides no encryption — it proves a packet wasn't altered and came from the expected peer, but anyone can still read its contents. ESP (Encapsulating Security Payload, IP protocol 50) encrypts the payload and optionally authenticates it too. In practice, ESP is used almost universally: confidentiality is usually a requirement anyway, and AH's biggest practical weakness is that its integrity check covers the outer IP header, which breaks the instant a NAT device rewrites that header in transit — a problem that is solvable for ESP (below) but not for AH.

Transport vs. tunnel mode

IPsec can protect a packet in one of two modes. Transport mode protects only the original payload, leaving the original IP header intact and visible — suited to protecting traffic directly between two hosts that are themselves the IPsec peers. Tunnel mode instead wraps the entire original IP packet, header included, inside a brand-new IP packet with its own new header between the two IPsec gateways. This is exactly what a site-to-site VPN needs: the two original hosts never need to know IPsec is involved at all, because their original packet, headers and all, travels as cargo inside a new packet exchanged by the gateways standing in for them.

Interactive IPsec tunnel establishment

Step through two site-to-site VPN gateways building an IPsec tunnel: the IKE_SA_INIT exchange that creates an encrypted control channel, the IKE_AUTH exchange that authenticates both sides and negotiates a Child SA, and the ESP tunnel-mode encapsulation that actually carries traffic between the two private networks.

NAT traversal and practical considerations

ESP has no port numbers of its own, which is a problem the moment a NAT device sits between two IPsec peers: NAT relies on port numbers to multiplex multiple internal hosts onto one public address, and plain ESP gives it nothing to rewrite. NAT-T (NAT Traversal) solves this by detecting a NAT in the path during IKE negotiation and, when one is found, wrapping ESP packets inside UDP on port 4500 — giving the NAT device the port number it needs, at the cost of a small per-packet overhead. This is specifically why ESP, not AH, is the practical default: AH's integrity check over the outer header has no equivalent fix, since NAT must rewrite that header for NAT-T to work at all.

IKEv1 vs. IKEv2: modes and phases

The lesson so far has described IKEv2. Its predecessor, IKEv1, negotiates in two phases, and Phase 1 can run as one of two different modes — a design IKEv2 deliberately replaced.

Phase 1 establishes the ISAKMP SA, the protected control channel equivalent to IKEv2's IKE SA:

• Main Mode takes 6 messages (3 round trips). The first two negotiate the ISAKMP SA's own parameters (encryption, hash, authentication method, DH group); the next two perform the Diffie-Hellman exchange and nonces; the last two exchange identity and authentication — and those last two are already encrypted under keys derived from the DH exchange. That's identity protection: nothing identifying either peer is ever visible to an eavesdropper.
• Aggressive Mode compresses the same job into 3 messages. It sends the SA proposal, DH exchange, nonce, and identity together in the opening messages — identity included, in the clear, before any encryption exists. It's faster, but gives up identity protection, and with pre-shared-key authentication specifically, that cleartext exchange exposes enough material to make an offline dictionary attack against a weak PSK realistic. This is the standard reason to prefer Main Mode over Aggressive Mode whenever both are available.

Phase 2 uses Quick Mode to negotiate the actual IPsec (Child) SAs that protect real traffic — entirely inside the already-encrypted Phase 1 SA. One Phase 1 SA can run Quick Mode repeatedly, to rekey or to negotiate additional Child SAs, without repeating the expensive Phase 1 exchange each time.

| | Purpose | Round trips | Identity protected? | | --- | --- | --- | --- | | IKEv1 Main Mode | Establish the ISAKMP SA (Phase 1) | 3 | Yes | | IKEv1 Aggressive Mode | Establish the ISAKMP SA (Phase 1), faster | 1.5 | No | | IKEv1 Quick Mode | Negotiate the IPsec SA (Phase 2), inside Phase 1 | 1.5, already protected | N/A — already inside a secured SA | | IKEv2 IKE_SA_INIT + IKE_AUTH | Both phases, collapsed into one exchange | 2 total | Yes, always |

IKEv2 replaces this entire two-phase, three-mode structure with just two exchanges, IKE_SA_INIT and IKE_AUTH (covered earlier in this lesson), which always provide identity protection — IKE_AUTH only ever happens after IKE_SA_INIT's encrypted channel already exists — in a fixed two round trips, with no mode to choose between. Rekeying or adding a Child SA later reuses the same generic CREATE_CHILD_SA exchange rather than a separate mode like Quick Mode.

Inspect IPsec evidence

On a representative network device, show crypto isakmp sa (or the equivalent IKE SA command) confirms Phase 1 (the IKE SA) is established between the expected peers, and show crypto ipsec sa confirms Phase 2 (the Child SA) shows the expected encapsulation, encryption, and traffic counters actually incrementing — a tunnel showing zero packets encrypted is a tunnel carrying no traffic, whatever its status says.

Guided IPsec mode practice

Work through why both site-to-site and remote-access VPNs use tunnel mode, even though a remote-access VPN is technically host-to-gateway rather than gateway-to-gateway. The deciding factor isn't how many hosts are involved — it's whether the IPsec peer is acting on behalf of other addressing that needs to be preserved. A remote worker's laptop connecting to a VPN concentrator needs to appear on the internal network with an internal address, exactly like a branch office does in a site-to-site tunnel: tunnel mode's new outer header carries the original (internal) packet, headers included, across the untrusted public path. Transport mode is reserved for the much narrower case where the two IPsec peers are the two original endpoints exchanging traffic directly, with no gateway standing in for anyone else.

Troubleshoot IPsec

1. Confirm Phase 1 (IKE SA) proposals match on both peers first — a mismatched encryption algorithm, hash, or Diffie-Hellman group here fails the negotiation before any traffic is even discussed.
2. If Phase 1 succeeds but Phase 2 (Child SA) fails, check the traffic selectors (proxy IDs): both sides must agree on exactly which subnets the tunnel protects, not just that a tunnel should exist.
3. Verify the authentication method and credentials match exactly — an identical pre-shared key on both sides, or certificates that both chain to a mutually trusted CA.
4. If a NAT device sits between the peers, confirm NAT-T was actually negotiated (both sides must detect the NAT during IKE) — a tunnel that reports "up" but passes no traffic often means ESP packets are arriving unencapsulated and being silently dropped.
5. Distinguish "tunnel up, zero traffic" from "tunnel won't come up at all": the first points at routing or ACLs (packets never reach the tunnel), the second points at IKE or Child SA negotiation itself.
6. Confirm routing actually sends traffic for the protected subnets toward the tunnel interface or policy — a perfectly negotiated SA carries nothing if nothing is routed into it.

Knowledge check and summary

Pro deep dive