Because IPsec is implemented at the IP layer, it can provide protection to any IP protocol including TCP and UDP. The security services that IPsec provides are:

Access control

The ability to establish an IPsec communication is controlled by a policy -- refusal to negotiate security parameters will prevent communication.

Data origin authentication

The recipient of a packet can be sure that it comes from the sender it appears to come from.

Message integrity

An attacker cannot modify a packet and have it accepted.

Replay protection

An attacker cannot resend a previously sent packet and have it accepted.

Confidentiality

An attacker cannot read intercepted data.

IPsec is made up of three protocols, each of which is defined as a framework that defines packet layouts and field sizes and is suitable for use by multiple cryptographic algorithms. The protocols themselves do not define specific cryptographic algorithms to use, although every implementation is required to support a specified set of algorithms. The protocols that make up IPsec are:

• The Authentication Header (AH)

• The Encapsulating Security Payload (ESP)

• The Internet Security Association Key Management Protocol (ISAKMP)

The Encapsulating Security Payload (ESP) protocol provides confidentiality (encryption) and limited protection against traffic flow analysis. ESP also includes some of the services normally provided by AH. Both AH and ESP rely on the availability of shared keys, and neither one has a way to move them from one machine to another. Generating these keys is handled by the third IPsec protocol, the ISAKMP.

ISAKMP is also a framework protocol; it doesn't by itself define the algorithms that are used to generate the keys for AH and ESP. The Internet Key Exchange (IKE) protocol uses the ISAKMP framework with specific key exchange algorithms to set up cryptographic keys for AH and ESP. This layering may seem confusing and overly complicated, but the separation of ISAKMP from IKE means that the same basic IPsec framework can be used with multiple different key exchange algorithms (including plain old manual key exchange). The standardization of IKE allows different people to implement the same key exchange algorithms and be guaranteed interoperability. The Linux FreeS/WAN project has an implementation of IKE called Pluto.

In IPv6 the AH and ESP protocols can be used simultaneously, with an IPv6 feature called header chaining, to provide authentication modes that ESP alone cannot provide. When they are used in this way it is recommended that ESP be wrapped by the additional AH header. In IPv4, it's not possible to use them both at once (you can have only one header at a time).

IPsec provides two operating modes for AH and ESP, transport and tunnel. In transport mode, AH or ESP occur immediately after the IP header and encapsulate the remainder of the original IP packet. Transport mode works only between individual hosts; the packet must be interpreted by the host that receives it. Transport is used to protect host-to-host communications. Hosts can use it to protect all of their traffic to other cooperating hosts, or they can use it much the way TLS is used, as a protection layer around specific protocols.

In tunnel mode, the entire original packet is encapsulated in a new packet, and a new IP header is generated. IPsec uses the term security gateway for any device that can operate in tunnel mode. This term applies to all devices that can take IP packets and convert them to and from the IPsec protocols, whether they are hosts or dedicated routers. Because the whole IP packet is included, the recipient can forward packets to a final destination after processing. Tunnel mode is used when two security gateways or a gateway and a host communicate, and it is what allows you to build a virtual private network using IPsec.

The AH and ESP protocols each contain a 32-bit value that is called the Security Parameter Index (SPI). This is an identifier that is used to distinguish between different conversations going to the same destination. Every IPsec implementation is required to be able to independently track security parameters for the combination of SPI, destination IP address, and the security protocol that is being used (either AH or ESP). This combination of parameters is called a Security Association (SA). It is the responsibility of the specific ISAKMP key management protocol to negotiate and set the cryptographic parameters, including the SPI, for each Security Association.

An SA is effectively the collection of the cryptographic keys and parameters for use by either AH or ESP:

AH

The cryptographic keys and algorithm identifiers used for integrity protection and anti-replay parameters

ESP

The cryptographic keys and algorithm identifiers used for encryption, integrity protection, and anti-replay parameters