The TCP three-way handshake is the foundation of reliable internet communication. Before any HTTP request, database query, or file transfer proceeds, client and server establish a synchronized session through an exchange of SYN, SYN-ACK, and ACK packets. Network engineers troubleshoot connectivity failures by examining this handshake in packet captures daily. Understanding sequence numbers, window sizes, state machine transitions, and failure modes separates engineers who guess from those who diagnose with evidence.
TCP provides byte-stream delivery with ordering, retransmission, congestion control, and flow control. The handshake negotiates initial sequence numbers that every subsequent byte references, enabling duplicate detection and in-order delivery. Both sides also exchange maximum segment size options during the handshake, establishing the largest payload each will accept per segment. MSS negotiation prevents fragmentation on the path and is frequently overlooked when diagnosing throughput problems on VPN tunnels with reduced MTU.
Handshake Mechanics
The client sends a SYN packet with its initial sequence number — ISN — chosen randomly for security. The SYN flag consumes one sequence number even though it carries no data. The server responds with SYN-ACK, acknowledging the client ISN plus one and offering its own ISN. The acknowledgment number field confirms receipt of the client's SYN. The client completes with an ACK acknowledging the server ISN plus one. After this exchange, both sides enter ESTABLISHED state and may transmit data.
Simultaneous open — both sides sending SYN at the same time — is handled by the protocol using SYN-ACK responses that converge to ESTABLISHED without a separate handshake path. This matters for peer-to-peer applications but rarely appears in client-server troubleshooting. The key diagnostic skill is recognizing which leg of the handshake failed: SYN without SYN-ACK indicates server unreachable or filtered; SYN-ACK without final ACK indicates client-side filtering or asymmetric routing returning responses via a path the client does not recognize.
State Machine and Timers
TCP connections progress through defined states: CLOSED, LISTEN, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT, CLOSE-WAIT, TIME-WAIT, and others during graceful teardown. Servers in LISTEN state accept incoming SYNs; clients transition from CLOSED to SYN-SENT on connect. Half-open connections occur when one side believes the session is established while the other does not — often from crashed hosts or middleboxes that dropped state without sending RST.
Retransmission timers govern SYN retries when no SYN-ACK arrives. Initial timeout is typically one second, doubling with each retry up to a platform-specific maximum attempt count. Linux defaults to five SYN retransmissions over roughly one hundred twenty seven seconds before abandoning the connection attempt. Users perceive this as a hung browser while the stack silently retries. Tuning SYN retransmission is rarely advisable; fix the underlying reachability or filtering problem instead.
Failure Modes and Security
SYN flood attacks send massive volumes of SYN packets without completing handshakes, exhausting server connection tables and memory allocated for half-open sessions. SYN cookies allow servers under attack to encode connection state in the SYN-ACK sequence number rather than allocating memory until the final ACK arrives. Rate limiting SYN packets per source IP at firewalls and load balancers provides additional protection. Cloud DDoS services absorb volumetric SYN floods before they reach origin infrastructure.
Firewall rules blocking SYN but allowing established connections create confusing symptoms — existing sessions work while new connections fail. Stateful firewalls track connection state and must see the three-way handshake to create entries. Asymmetric routing where SYN traverses one firewall and SYN-ACK returns through another breaks stateful inspection because the second firewall never saw the initial SYN. Capture packets on both sides of suspected asymmetric paths to confirm whether handshake packets traverse consistent routes.
Performance Optimizations
TCP Fast Open allows data in the SYN packet on subsequent connections after an initial TFO cookie exchange, shaving one round trip from latency-sensitive protocols. TFO requires client, server, and middlebox support — many corporate proxies disable it. TLS 1.3 reduces handshake round trips independently through session resumption and zero-round-trip early data modes with careful replay protection. Combining TLS 1.3 session tickets with TFO materially improves web application perceived responsiveness for returning visitors.
Selective acknowledgment and window scaling negotiated during handshake enable high-bandwidth connections over high-latency paths. Window scaling multiplies the sixteen-bit window field by a scale factor up to fourteen, supporting multi-megabyte receive windows necessary for filling long fat pipes. Missing window scaling on either side caps throughput regardless of link speed — verify both endpoints negotiate scaling in the SYN and SYN-ACK options during performance investigations.
Diagnostic Workflow
Capture traffic at client, server, and intermediate firewalls simultaneously when possible. Filter for tcp.flags.syn and the relevant host and port. Count retransmitted SYNs to quantify timeout behavior. Examine whether SYN-ACK arrives with correct acknowledgment numbers. Check for ICMP unreachable messages between SYN attempts indicating routing failures or ACL denials with ICMP feedback. Follow TCP stream in Wireshark after successful handshake to separate connection establishment issues from application-layer problems.
Document normal handshake timing for critical applications — typically one to two round-trip times on LAN, tens of milliseconds across regions. Alert on connection establishment time SLIs exceeding baselines. The three-way handshake is simple in diagram form but rich in operational detail; mastering its behavior and failure signatures resolves a disproportionate share of network connectivity tickets in every operations team.
Load Balancer and NAT Interactions
Load balancers terminate TCP from clients and originate new TCP connections to backend servers, making the handshake visible twice in end-to-end captures — client-to-LB and LB-to-server. Asymmetric routing through load balancers breaks SYN cookie and stateful inspection when return traffic bypasses the LB. Ensure both directions traverse the same LB instance or use connectionless direct server return designs with appropriate ARP and routing configuration.
NAT devices translate addresses during handshake, rewriting source IP and port in SYN packets. Return SYN-ACK must traverse the same NAT binding or the client receives responses it cannot associate with its outbound SYN. NAT table exhaustion during SYN flood conditions drops new connection attempts even when backend servers are healthy — monitor NAT table utilization on internet-facing firewalls as part of capacity planning.
TCP timestamp options negotiated during handshake enable round-trip time measurement and protect against sequence number wrap on high-bandwidth long-fat pipes. PAWS — Protect Against Wrapped Sequence numbers — rejects segments with stale timestamps. Missing timestamp support on either end disables PAWS, creating theoretical vulnerability on very high speed long distance links that modern networks rarely encounter but capture analysis should note when troubleshooting sequence-related anomalies.