Networking / transport / tcp / socket options
The window scale is fixed in the SYN, and SO_RCVBUF is what fixes it low
That window scaling is on by default, so the 65,535-byte ceiling is gone and the receive window is whatever the buffer allows. Three things are wrong. The shift count is chosen once, in tcp_select_initial_window during the SYN exchange, as clamp(ilog2(space) - 15, 0, 14) where space is max(tcp_rmem[2], rmem_max) clamped by window_clamp — nothing after the handshake can raise it, so raising tcp_rmem on a running server does not affect a single existing connection. Calling setsockopt(SO_RCVBUF) before connect is worse than doing nothing: tcp_connect_init pins window_clamp to that buffer, which lowers the shift, and __sock_set_rcvbuf sets SOCK_RCVBUF_LOCK, which makes tcp_rcvbuf_grow return immediately so autotuning never runs again on that socket. And the advertised window is not the buffer: tcp_win_from_space halves it at the default scaling_ratio of 128/256, and __tcp_select_window clamps it to rcv_ssthresh, which starts at the initial window and grows by 2*advmss per segment — the receive window has its own slow start, running underneath the congestion window's.
The TCP header's window field is 16 bits, so the window scale option carries a left-shift count that multiplies it; Linux computes that shift once, in tcp_select_initial_window, from the larger of net.ipv4.tcp_rmem[2] and net.core.rmem_max clamped by window_clamp, and it cannot change for the life of the connection — which is why an application that calls setsockopt(SO_RCVBUF) before connecting both pins window_clamp to that buffer and sets SOCK_RCVBUF_LOCK, permanently capping the window and switching receive-buffer autotuning off.
Where this is already explained
- Your bandwidth-delay product used the wrong bandwidth and the wrong delay
That the bandwidth-delay product is arithmetic you do once — the speed your link is sold at, times the number your ping prints — and that setting the socket buffer to the answer fills the pipe. Both inputs are usually wrong. The bandwidth is the narrowest hop on the path, not the link you pay for, and sizing to your own link rate on a path with a slower hop is worse than leaving the default alone. The delay has to be the unloaded minimum RTT, because throughput measured during a transfer multiplied by the RTT measured during the same transfer always returns the window you already had — so the rule certifies whatever queue you are already carrying instead of correcting it.
- TCP Congestion Control
That throughput is set by the link's bandwidth, so a slow transfer means buying more of it. A single flow is bounded by window/RTT, and the window is capped first by the receive window, then by the loss rate a loss-based controller has to operate against — upgrading 1 Gbps to 10 Gbps moves none of those, and oversized buffers along the path make the window grow far past the bandwidth-delay product, inflating latency while throughput stays flat.
- Slow start is not slow, and your keep-alive connection is not warm
That slow start is a slow warm-up phase you can wait out, that it ends early in any real transfer, and that a long-lived keep-alive connection stays warm. Growth is exponential, so slow start is the fastest thing TCP does; but its cost is counted in round trips rather than in seconds, which means a 1 MB response costs the same number of round trips on a 10 Mbps link and a 10 Gbps one, and for anything under a few megabytes slow start is not a phase of the transfer, it is the whole transfer. Linux then leaves slow start well below the bandwidth-delay product because CUBIC's HyStart exits on a delay rise once cwnd reaches 16, and net.ipv4.tcp_slow_start_after_idle defaults to 1, which halves the window once per retransmission timeout of idleness down to the initial 10 segments — so a connection used once a second is cold on every request.
3 published lessons depend on this concept, which is what moves it up the writing queue. Nothing is hidden behind this page — it has not been written.