Replication Slots: WAL Kept for a Consumer That Left

A replication slot's job is to retain write-ahead log for a consumer that is not there. That is not a failure mode, it is the feature: a slot exists so that a standby which has been offline for an hour can reconnect and carry on. Which means a standby that has been offline for a month is being served exactly as designed, and the disk fills.

The write-ahead log, or WAL, is the sequential record of every change, and Postgres normally deletes segments of it as soon as a checkpoint has flushed the corresponding data pages. A slot interrupts that. Each slot records a restart_lsn — the oldest log sequence number, or LSN, that its consumer might still need — and the documentation is blunt about the consequence: WAL from that position onward "won't be automatically removed during checkpoints". The consumer advances restart_lsn by confirming what it has received. A consumer that has stopped confirms nothing, so restart_lsn stops moving, and every checkpoint from then on removes zero bytes while the server keeps writing.

The panel runs 72 hours from the moment a consumer stops. The slider that carries the lesson is hours since the consumer stopped; everything else sets the conditions. Start by moving it and watching pg_wal grow in a straight line, then look at what each checkpoint decided and why.

At 8 MB/s — an unremarkable rate for a busy OLTP primary — the slot accumulates 675 GB a day. Twelve hours after the consumer stopped, pg_wal holds 338 GB in 21,600 segments, and all 144 checkpoints in between ran successfully and deleted nothing. The server is not degraded, and nothing is logged above LOG level. The first sign is the disk.

Why the checkpoint is not the thing that is broken

Every five minutes the checkpointer works out the oldest WAL segment anything still needs and deletes or recycles everything before it. Four things can hold that position back: a slot's restart_lsn, wal_keep_size, an archiver that has not finished archiving, and the requirements of crash recovery itself. Only the first has no upper bound by default.

wal_keep_size defaults to 0 and is a fixed quantity when set — "keep this many megabytes beyond what is needed", so the worst case is exactly that many megabytes. A slot is the opposite shape. It says "keep everything since this position", and the position is set by someone else. The difference is the whole reason slots exist: before them, a standby that fell behind by more than wal_keep_size was simply broken, and people set wal_keep_size to absurd values to avoid it. Slots replaced a guess with a guarantee, and a guarantee with no ceiling is an unbounded resource commitment.

Notice what the decision log does not say. There is no failed checkpoint, no error, no warning. The checkpointer is behaving correctly, the slot is behaving correctly, and pg_wal grows at exactly the rate the application generates WAL. The only two places the situation is visible before the disk fills are the size of the directory and one row in pg_replication_slots.

That is also why the instinct to delete files from pg_wal is so dangerous, and why it is the first suggestion in every thread on the subject. Those files are not logs in the diagnostic sense; they are the only durable record of committed transactions that have not yet been written back to the data files. Deleting one that the server still needs turns a full disk into an unrecoverable database. The supported way to remove them is to remove the reason they are being kept, and then let a checkpoint run.

Set WAL generated to 64 MB/s, which is a heavy but perfectly ordinary write load, and the arithmetic stops being comfortable: 5.3 TB a day. A 500 GB WAL volume fills in 2.2 hours. There is no monitoring interval that turns that into a business-hours problem, which is the argument for the cap.

The cap, and what it costs

Put the rate back to 8 MB/s and drag max_slot_wal_keep_size to 100 GB. The bar chart changes shape: pg_wal climbs for 3.6 hours, and then falls back to 1 GB and stays there for the rest of the 72 hours. The disk is safe.

Read the wal_status line at hour 4. The slot is lost, and invalidation_reason is wal_removed. That is a permanent state. A physical standby now needs a fresh pg_basebackup — hours of transfer and a full re-copy of the database. A logical subscriber needs the initial data copy of every table in the publication repeated, and until it finishes, the subscriber holds an inconsistent, partially-updated copy. The setting did not prevent the failure. It chose which failure you get.

The four wal_status values are the states of that choice, and they are worth knowing before an incident rather than during one. reserved means the retained files fit inside max_wal_size and everything is normal. extended means max_wal_size has been exceeded and the files are being kept anyway — this is the state that fills disks, and it produces no log line. unreserved means the slot has fallen past the cap and its segments are due for removal at the next checkpoint; a consumer that is still connected and catching up can get back to extended from here, which is the one recoverable warning you get. lost means the segments are gone.

The column that lets you act before unreserved is safe_wal_size: how many more bytes can be written before this slot is in danger. It is NULL when max_slot_wal_keep_size is -1, which is the default, which means the single most useful early-warning number in the system is switched off unless you have opted into the cap. Setting the cap is therefore two decisions at once: an upper bound on the disk, and a number you can alert on.

PostgreSQL 18 adds the other kind of bound. idle_replication_slot_timeout invalidates a slot that has had no replication connection for longer than the configured duration, with invalidation_reason = 'idle_timeout'. It defaults to 0, which disables it. It is a better fit than a size cap for the common case — a forgotten slot from a decommissioned consumer — because it triggers on the thing that is actually wrong. It does not help with a consumer that is connected and simply not keeping up, which is the case the size cap covers. Note the timing detail in the documentation: the invalidation happens at a checkpoint, so with the default checkpoint_timeout the slot can outlive the timeout by up to five minutes.

The half nobody expects: the vacuum horizon

Set what the slot is for to "physical standby, hot_standby_feedback on" and read the rows vacuum may not remove readout. It says: every table, every database.

A slot can carry two transaction ids as well as an LSN. pg_replication_slots.xmin is documented as "the oldest transaction that this slot needs the database to retain. VACUUM cannot remove tuples deleted by any later transaction." catalog_xmin is the same sentence restricted to catalog tuples. In the source these are two separate inputs to the horizon calculation: the slot's xmin holds back the horizon used for ordinary tables and shared catalogs, while catalog_xmin holds back only the catalog horizon. So which of your tables stop being vacuumable depends on what kind of slot it is, and the three cases are genuinely different:

• A physical slot without hot_standby_feedback holds no transaction id at all. It is a disk problem and nothing more.
• A physical slot with hot_standby_feedback = on stores the standby's feedback xmin in the slot, and vacuum then cannot remove any row version deleted after that point, in any table, in any database. This is the configuration people turn on to stop ERROR: canceling statement due to conflict with recovery on the standby, and it moves the cost to the primary as unremovable dead rows.
• A logical slot holds catalog_xmin, because decoding an old transaction requires the catalog as it was when that transaction ran. Your application tables vacuum normally. pg_class, pg_attribute and friends do not, in every database, and catalog bloat shows up as planning time rather than as anything obviously storage-shaped.

The middle case has a detail worth stating precisely, because it is the difference between a standby outage that heals and one that does not. Without a slot, the standby's feedback xmin lives in the walsender process's own entry in the process array, so the moment the connection drops the primary stops holding anything. With a slot, the feedback is written into the slot, and the slot survives the disconnection — that being the entire point. A standby that crashes at 02:00 with feedback enabled and a slot is still holding the primary's vacuum horizon at 09:00.

Set the transaction rate to 8,000 and drag the hours slider. At 7 hours the horizon is 201 million transaction ids old, past autovacuum_freeze_max_age, and the problem changes category: Postgres begins launching anti-wraparound autovacuums that also cannot advance relfrozenxid, because the rows they need to freeze are protected by the same slot. That path ends with the server refusing new write transactions — see transaction ID wraparound. The disk filling is usually the first thing you notice, but it is not always the first thing that will take you down.

One quieter consequence in the same family: while the horizon is held, vacuum also cannot mark pages all-visible, so index-only scans across the whole cluster silently start visiting the heap. A forgotten slot degrades query plans, grows tables, and fills a disk, and all three symptoms arrive without anything in the log tying them together.

Checking it on a real system

One query answers almost all of it. Run it before you look at anything else, including during an unrelated disk alert:

SELECT slot_name, slot_type, database, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained,
       pg_size_pretty(safe_wal_size) AS headroom,
       age(xmin) AS xmin_age, age(catalog_xmin) AS catalog_xmin_age,
       now() - inactive_since AS inactive_for,
       invalidation_reason
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

active = false with a large retained is the whole diagnosis. inactive_since and invalidation_reason were added in PostgreSQL 17, and inactive_since in particular replaces the guesswork about when a consumer went away — note that it stops updating once the slot is invalid, so it records the beginning of the outage rather than its length. If xmin_age or catalog_xmin_age is in the tens of millions, this slot is also your bloat and your wraparound clock.

Compare the retained size against the actual directory, because there are other reasons for WAL to accumulate and confusing them wastes the outage:

SELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();

SELECT archived_count, last_archived_wal, failed_count,
       last_failed_wal, last_failed_time
FROM pg_stat_archiver;

A large pg_wal with no lagging slot and a rising failed_count is a broken archive_command, not a slot — the retention mechanism is different and so is the fix. A large pg_wal with neither is usually a max_wal_size that is simply set large, or a burst of writes between checkpoints, and it will come back down on its own.

When it is a slot, the decision is binary and worth making deliberately. Either the consumer comes back — in which case buy time by moving WAL to another volume, or add disk, and let it catch up — or it does not, in which case drop the slot:

SELECT pg_drop_replication_slot('cdc_prod');
CHECKPOINT;

The space is released by the checkpoint, not by the drop. If the slot is still marked active the drop will fail, and you have to terminate the walsender holding it with pg_terminate_backend() first — look for it in pg_stat_activity where backend_type = 'walsender'. And be certain, because dropping a slot is not reversible: recreating it with the same name gives you a new restart_lsn at the current position, and the consumer's next request for an old segment returns ERROR: requested WAL segment ... has already been removed.

What to configure before the next one. Set max_slot_wal_keep_size to something you can actually afford to lose — a value that leaves real headroom on the volume — and alert on safe_wal_size falling, which only exists once you have set the cap. On PostgreSQL 18, set idle_replication_slot_timeout to a few hours so that abandoned slots clean themselves up. Alert on max(age(coalesce(xmin, catalog_xmin))) across pg_replication_slots, not only on WAL size, because the two failure modes have different clocks. And treat a slot as an object with an owner: every slot should map to a named consumer someone is responsible for, because the ones that cause incidents are always the ones created for a migration in 2023 that nobody remembers.

Next, the deadline this turns into if the horizon stays put: transaction ID wraparound. And for the throughput side of vacuum, which is a different problem with identical symptoms, autovacuum tuning.