TCP Congestion Control

Upgrading a link from 1 Gbps to 10 Gbps usually does nothing at all for a single transfer. Not a little — nothing. The sender was never waiting for the link; it was waiting for acknowledgements.

A TCP sender may have at most one window of unacknowledged data outstanding at a time. That window divided by the round trip time is the entire throughput of the flow. Bandwidth appears nowhere in that expression. It enters only indirectly, as the ceiling on how large the window is allowed to grow before the path starts dropping packets — and there are three other things that cap the window long before the link does.

The panel below is a packet-conservation model of one flow crossing one bottleneck. The sender injects packets while its window has room, the bottleneck serves them at exactly the link rate and tail-drops what will not fit, and acknowledgements return one propagation delay after service. Every number underneath is measured from that, not computed from a formula. Start by raising bottleneck bandwidth and watching the hero number refuse to move.

Three things to do, in order. One: drag bandwidth from 100 Mbps to 10 Gbps. Throughput follows, because nothing else is binding yet — so now set loss unrelated to congestion to 0.01% and drag bandwidth up again. From 25 Mbps onwards the hero number sticks at about 24 Mbps for the rest of the drag: a hundredfold more link, from 100 Mbps to 10 Gbps, moves it by nothing at all while utilisation falls from 24% to 0.24%. Two: put the loss back to none and set the receive window to 64 KB instead. The same shape for a completely different reason — the number is now 6.55 Mbps and it too ignores the whole bandwidth slider. The log will tell you which limit is binding in each case. Three: put bandwidth back at 100 Mbps, RTT at 20 ms, and drag the bottleneck buffer from 20 ms to 500 ms. Throughput stays pinned at 100 Mbps for the entire drag while the measured RTT goes past a quarter of a second. That is bufferbloat, and you just built it.

The window is the whole story

TCP is a sliding window protocol, so the sender's rate is not a rate at all. It is a quantity divided by a delay: at most one window of unacknowledged bytes may be outstanding, and each byte is released only when its acknowledgement arrives, one round trip after it was sent.

So throughput = window / RTT, and the window in force is the smallest of three separate limits:

• cwnd, the congestion window — the sender's own estimate of what the path will carry, which is what a congestion control algorithm computes.
• rwnd, the receive window — how much the receiver has promised to buffer.
• the send buffer — you cannot have unacknowledged data outstanding that you have not got memory to retransmit.

The window that fills the pipe exactly is the bandwidth-delay product: bandwidth times round trip time. At 10 Gbps and 100 ms that is 125 MB in flight, or about 83,000 packets. Below it you underrun the link; above it the excess is not in flight at all, it is sitting in a queue, adding delay and nothing else. That is the single most useful line in this lesson: the bytes above the BDP buy you latency, not throughput.

The receive window is where this bites people first, and it is the reason for experiment two. The window field in the TCP header is 16 bits, so without the window scale option a receiver can never advertise more than 65,535 bytes. RFC 7323 adds a scale factor, capped at a shift of 14, which raises the ceiling to 1 GiB — but it is negotiated in the SYN, so if either end has scaling disabled, or a middlebox strips the option, you are back to 64 KB. Over an 80 ms path, 64 KB per round trip is 6.55 Mbps, and that is what you get on a 10 Gbps link. The simulation reproduces that number exactly, because it is not an approximation; it is arithmetic.

Linux ships net.ipv4.tcp_rmem with a default of 131072 bytes, which the kernel documentation notes "results in initial window of 65535", and grows it by auto-tuning up to tcp_rmem[2] as the path demands. The footgun is in the same paragraph: calling setsockopt(SO_RCVBUF) disables auto-tuning for that socket. A well-meaning line of code that sets an 8 MB receive buffer is fine; the one that sets 256 KB has silently capped every long-haul transfer the process will ever make at 256 KB / RTT.

Loss-based control needs loss

Set the algorithm to Reno with a clean path and watch the first panel. The window climbs one packet per round trip, crosses the top of the buffer, loses a packet, and halves. This is AIMD, and RFC 5681 states it in two lines: in congestion avoidance the window grows by SMSS*SMSS/cwnd per acknowledgement, which is one segment per RTT; on the third duplicate ACK, ssthresh becomes max(FlightSize/2, 2*SMSS).

Read that as a control loop and the problem is obvious. Reno's only sensor is a dropped packet. It has no way to know the pipe is full until it has already overfilled it, so it must overfill it, repeatedly, forever. The sawtooth is not a defect; it is the algorithm searching for a boundary it can only find by crossing.

That search costs more the larger the pipe gets. RFC 3649 gives the steady-state response function for standard TCP as an average window of roughly 1.2/sqrt(p) segments at loss rate p, and then works the example: to hold 10 Gbps with 1500-byte packets on a 100 ms path, a sender needs an average window of 83,333 segments, which requires at most one congestion event per five billion packets — the RFC puts it at one event every hour and forty minutes. No real path is that clean. This is why experiment one works. Select Reno, turn on a hundredth of a percent of loss, and the window settles at 122 packets against the 120 that 1.2/sqrt(0.0001) predicts: the model is landing on the response function, not being told to. That is 18 Mbps over an 80 ms path whether the link underneath it is 100 Mbps or 10 Gbps. CUBIC, the default in the panel, sits above the standard-TCP curve by design and settles around 161 packets, or 24 Mbps — a better constant in front of the same brick wall.

CUBIC attacks exactly that, and it is what your Linux machine is running: it has been the kernel default since 2.6.19 in November 2006, when commit 597811e switched net/ipv4/Kconfig over from BIC, and DEFAULT_CUBIC is still the default choice there today. RFC 9438 replaces the linear increase with W_cubic(t) = C(t − K)³ + W_max, where W_max is the window at the last congestion event, C is 0.4, and K is the time needed to climb back to it. The window returns to its previous operating point quickly, flattens as it approaches it — that is the concave region, where CUBIC is being careful near a level it knows caused loss — and then accelerates past it to probe for more. Its decrease is gentler too: it multiplies the window by beta_cubic = 0.7 where Reno multiplies by 0.5, so it gives up 30% rather than 50% on each event.

Both properties are visible in the log. Switch between Reno and CUBIC on a 1 Gbps, 100 ms path with 0.01% loss: same path, same loss rate, and CUBIC delivers roughly 60% more — 23 Mbps against 14. But note what did not change. CUBIC is still loss-based, still cannot distinguish a full buffer from a flaky transceiver, and still reduces on both. Set the loss slider to 0.1% and both algorithms are on the floor.

Bufferbloat: the buffer is the operating point

Experiment three is the one that changes how people design networks. A loss-based sender grows its window until the path drops something. The path drops something when the bottleneck buffer is full. Therefore the sender's steady-state window is not the bandwidth-delay product — it is the bandwidth-delay product plus the entire bottleneck buffer, minus whatever the last multiplicative decrease took off.

Whoever sized that buffer chose your latency. Drag the buffer slider and watch the warm band in the first panel grow while the calm band stays exactly the same size. The window rides on top of the warm band because that is where the drops are. Every one of those packets is queued, not in flight, and contributes delay and nothing else. Throughput does not move, because throughput was already the link rate — there is nothing above 100% to win.

Two consequences follow, and the simulation will show you both. The first: the delay is paid by everything sharing that queue. A 300 ms standing queue on your upload does not slow your upload down at all; it adds 300 ms to every DNS lookup, every SSH keystroke and every video call behind the same link. This is why "my connection is fine, I ran a speed test" and "video calls are unusable" are consistent statements. A speed test measures the quantity bufferbloat does not damage.

The second: shrinking the buffer is not a free fix. Select Reno, set the path to 100 Mbps and 20 ms, and walk the buffer down. At 6 ms Reno holds 96% utilisation while CUBIC on the same path holds 100%; at the 2 ms minimum Reno is down to 81% and even CUBIC manages only 92%. When a decrease takes the window below the BDP the link idles until the window climbs back, and that gives you the buffer you need in one line. The window at the drop is BDP + buffer; after the cut it is beta × (BDP + buffer); requiring that to still be at least the BDP gives buffer ≥ BDP × (1 − beta) / beta. For Reno's beta of 0.5 that is one whole BDP, which is where the old "buffer = one BDP" rule of thumb comes from; for CUBIC's 0.7 it is 0.43. The rule is that arithmetic, not a law of nature, and it was derived for one flow rather than for a thousand — with many flows desynchronised the requirement falls by roughly the square root of their number.

The real fix is not a buffer size, because there is no size that is right for both 10 Mbps and 10 Gbps, both 2 ms and 300 ms. It is to make the queue signal congestion by time rather than by overflowing. Tick AQM at the bottleneck and drag the buffer across its whole range: the measured RTT stops responding to it entirely. CoDel watches how long packets have been sitting — RFC 8289 sets its default target sojourn time at 5 ms and its interval at 100 ms — and signals once the standing queue exceeds the target for longer than the interval, at a rate that increases as the inverse square root of how many times it has had to signal.

With ECN, that signal costs nothing. RFC 3168 lets the router set the CE codepoint instead of discarding the packet; the receiver echoes it, and the sender reduces the window exactly as it would for a drop, no more than once per window — but the packet arrives, and no retransmission is needed. The log distinguishes the two cases so you can watch a congestion event happen with zero packets lost.

Where a model-based sender changes the answer, and where it does not

BBR asks a different question. Instead of inferring the operating point from failures, it measures the path directly. The comment at the top of the Linux implementation is the whole algorithm: bottleneck_bandwidth = windowed_max(delivered/elapsed, 10 round trips), min_rtt = windowed_min(rtt, 10 seconds), pacing_rate = pacing_gain × bottleneck_bandwidth, cwnd = max(cwnd_gain × bottleneck_bandwidth × min_rtt, 4), with cwnd_gain of 2.

Look at what is absent. Neither the buffer size nor the loss rate appears anywhere in that expression, and the source says so explicitly: "the core algorithm does not react directly to packet losses or delays." Read the rest of that sentence before you take it as an absolute — it continues "although BBR may adjust the size of next send per ACK when loss is observed, or adjust the sending rate if it estimates there is a traffic policer, in order to keep the drop rate reasonable." Loss is a guard rail at the edges, not a term in the window. That is why BBR's line in the first panel is flat at roughly twice the BDP whatever you do to the buffer, and why its throughput barely notices the loss slider. Every ten seconds without a new RTT minimum it enters ProbeRTT and clamps the window to four packets for 200 ms, deliberately emptying every queue on the path so its delay estimate cannot drift upwards — you can see those notches in the RTT trace.

Now find its boundary, because it has a sharp one. Select BBR, set the path to 100 Mbps and 20 ms, and walk the buffer down from 40 ms. At 40 ms BBR drops nothing at all. At 10 ms it takes over five hundred congestion events and discards more than thirty thousand packets in forty seconds, and its throughput does not change by a megabit. A window of two BDPs cannot fit in a buffer holding less than one BDP, so BBR overruns it continuously and does not care, because loss does not enter the window calculation. Everything else queued behind it pays for the retransmissions. Fairness against loss-based flows sharing that bottleneck is the well-known complaint about this version of the algorithm — BBRv1 is what is modelled here and what tcp_bbr.c implements — and the later BBRv2 and v3 revisions add an explicit loss and ECN response precisely to address it.

And BBR does not repeal arithmetic. Set the receive window to 64 KB with BBR selected on a 10 Gbps path: 6.2 Mbps, against CUBIC's 6.55 on the same path, and 0.06% of the link. A model of the path is no help at all when the binding constraint is a promise the receiver made.

Checking it on a real system

The single most useful command is ss -ti against the live socket while the transfer is running. It prints the sender's actual state:

• cwnd: and ssthresh: — the congestion window in segments. Multiply by mss: and divide by rtt: to get the ceiling this flow can reach. If that number matches your measured throughput, bandwidth is not your problem.
• rtt: is smoothed RTT and variance; minrtt: is the lowest ever seen. The gap between them is your standing queue. A socket showing minrtt:12 and rtt:190 is telling you 178 ms of bufferbloat, in one line.
• rcv_space: is the receiver's auto-tuned buffer, and wscale: shows the negotiated send and receive shifts. A wscale:0,0 on a long path is your whole diagnosis.
• delivery_rate, pacing_rate and send — when delivery_rate sits far below send, the flow is not application-limited.
• retrans: and bytes_retrans: — divide by bytes_sent for the loss rate to put into the response function above.

To reproduce any point in the simulation on real hardware, impose the path with netem on the sender: tc qdisc add dev eth0 root netem delay 40ms loss 0.01%, then run iperf3 -c host -t 40 single-stream. Compare against iperf3 -P 10: if ten streams get ten times the throughput of one, the bottleneck is per-flow window, not capacity, and the whole diagnosis is settled in one command. Switch algorithms without recompiling anything using sysctl -w net.ipv4.tcp_congestion_control=bbr, or per-socket with setsockopt(TCP_CONGESTION).

For bufferbloat specifically, never measure it with an idle ping. Saturate the link and ping at the same time; the difference between loaded and unloaded latency is the number. Then put fq_codel or cake on the egress qdisc, shaped a few percent below the real line rate so the queue you control is the one that fills, and measure again. Moving the queue from a device you cannot configure to one you can is the entire fix.

Next: sizing the window against the pipe, the ramp that decides how a short transfer performs in slow start, why small writes stall on Nagle and delayed ACK, and the marking scheme that makes congestion signalling free, ECN.