TCP connection in slow motion

I’ve always been curious about the netstat output: what is the meaning of the different TCP connection states? How the connection transit from a state to another? I am also working on a different post on TCP errors, so I need to understand better the different TCP connection phases.

I then decide to “film” a TCP connection between two computers in slow motion. The idea is to show how the state of the TCP connection changes in the two machines and what packets transit between the two hosts.

Setup

To perform the experiment I have used a laptop (thinkpad) and a raspberry pi (raspberrypi) connected through my home’s wifi network.

A server application runs on the thinkpad and listens on port 9090. The server is just a [netcat](https://linux.die.net/man/1/nc) in listening mode:

nc -l 9090

A client application runs on the rapsberrypi and connects to the server on the thinkpad. Again the client is a netcat instance:

nc 192.168.1.108 9090

Note that the server and the client applications run on top of a Linux OS. The operative system manages the TCP connection and offers to the applications primitives to listen for connection, accept incoming connections, create connection, send and receive data and finally close connections. In the post, I will refer to the server and the client both to indicate the application and/or the OS.

The packets between the client and the server are captured and visualized with Wireshark.

The connection status is monitored in both machines with a repeated netstat command with output filtered on port 9090 and visualization of the timers (-o flag):

watch -n 1 "netstat -nao | grep \"9090\|Proto Rec\""

To slow down the packets between the two machines I have used Traffic Control. Here the bash script that setups TC:

#!/bin/bash

interface=wlp5s0
port=9090
delay=3s

# clean all the queuing disciplines
sudo tc qdisc del dev $interface root

# adds a queuing discipline with id 1:0 to the outbound traffic (root/egress)
# the discipline is PRIO https://linux.die.net/man/8/tc-prio 
# the prio class is setup to send all the packets to band 0
sudo tc qdisc add dev $interface root handle 1:0 prio priomap 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

# adds a queuing discipline to the class 2 of the prio discipline we defined before
# the discipline with id 20:0 will use net emulator to delay packets
sudo tc qdisc add dev $interface parent 1:2 handle 20:0 netem delay $delay

# adds a filter to the class 0 of the prio discipline we defined before
# the classifier (u32) is a generic filter based on packet properties 
# the match is on packet with port $port
# the selected packets are sent to class 2 of the prio discipline
sudo tc filter add dev $interface parent 1:0 protocol ip u32 match ip sport $port 0xffff flowid 1:2
sudo tc filter add dev $interface parent 1:0 protocol ip u32 match ip dport $port 0xffff flowid 1:2

# To remove the delay
# sudo tc -p qdisc ls dev wlp5s0
# sudo tc qdisc del dev wlp5s0 root

Screen layout

Top left: Wireshark, all the packets between the two hosts will be visible there.

Top right: the TCP status diagram for both machines. The blue colour is used for the server, the red colour for the client. The green colour is used to indicate that both server and client are in the same state.

Bottom left - first console: the console that runs the server on the thinkpad machine.

Bottom left - second console: the connection status for the server. A repeated run of the netstat command on the thinkpad machine.

Bottom right - first console: the console that runs the client on the raspberrypi machine.

Bottom right - second console: the connection status for the client. A repeated run of the netstat command on the rapsberrypi machine.

Storyline

Server and client are not running. Both side of the connection are in state CLOSED (there is no connection).

The server application is launched and starts listening on port 9090. The connection on the server side is now in status LISTEN. Netstat shows a process listening on port 9090. No packets exchanged or activity on the client side.

TCP connection establishment

The client application is launched and initiates a connection toward the server. The client sends a SYN packet: the 3-way handshake has begun. The connection status on the client side is now SYN_SENT.

The server receives the SYN packet and moves to status SYN_RECEIVED. The server then sends back a SYN-ACK packet.

The client receives the SYN-ACK packet and moves the connection status to ESTABLISHED. The client sends back an ACK packet.

The server receives the ACK packet and moves the connection status to ESTABLISHED.

The server application has been notified of the presence of the incoming connection and will manage it in a separate thread. At this point, there are two sockets open on the server side: one listing for new connections and another for the established connection to the client.

The 3-way handshake is now completed and the connection established.

In the Wireshark screen, you can notice some packets visualized in light grey, those are packets that have been re-sent because an ack was not received in time (packets delayed by traffic controller).

Data communication

The server sends a “Hello” message to the client. The client receives it, print it in the console and sends back an ACK packet.

The client sends a “Bye” message to the server. The server receives it, print it in the console and sends back an ACK packet.

Notice the flag PSH, it indicates that the received data should be pushed to the application.

TCP connection termination

The connection can be terminated by the server or by the client, in this case, the server initializes the process.

The server application is terminated, this indicates to the OS that the socket can be closed. The server sends a FIN packet to the client and sets its connection status to FIN_WAIT1. The four-way handshake has begun.

The client receives the FIN packet, sets the connection status to CLOSE_WAIT and sends back an ACK packet.

The server receives the ACK packet from the client and moves to FIN_WAIT2.

The client process is terminated and the connection can be closed. The client sends a FIN packet to the server.

The server receives the FIN packet and replies with the last ACK packet moving to TIME_WAIT status. This status is used to receive and discard packets that have been delayed in the network.