Delta Lake Optimistic Concurrency

Delta Lake advertises that multiple writers on multiple clusters can modify the same table at once. Then two MERGE jobs that touch completely different rows fail against each other, and the advice you find is "add a retry loop." The retry loop is not the answer. The conflict checker is answering a narrower question than you think it is.

A Delta write is optimistic: read the current snapshot, write new Parquet files, then try to append one JSON commit to the log. If somebody else committed the version you were aiming for, your transaction re-runs a conflict check against their commit before retrying. The check that produces ConcurrentAppendException asks exactly one thing: did the winner add any file that I should have read?

And "should have read" is decided by partition predicates only. Not by your join key. Not by row identity. Not by the min/max statistics sitting in the log. Move the controls below and read the reason the checker gives.

Start with the defaults: unpartitioned table, condition s.id = t.id, a concurrent MERGE landing in a different country and rewriting rows your batch never touches. It commits — row-level concurrency compared the row keys and found them disjoint. Now switch the layout to PARTITIONED BY (event_date), the one change every blog post recommends, and the identical transaction fails. Partitioning switched off the thing that was protecting you, and your condition still contributes no partition predicate to replace it.

What the checker actually compares

Every Delta write goes through three phases: read the snapshot, stage new files, then validate-and-commit. Validation is the only place conflicts exist. It runs against each commit that landed between the version you read and the version you are trying to write, one at a time, in order.

The append check in ConflictChecker takes the winner's AddFile actions and filters them by your transaction's read predicates. If even one file survives the filter, you get ConcurrentAppendException. It stops at the first match — the exception names one partition because the code literally takes head(1).

The filter is the part nobody expects. When your transaction records a read, Delta splits the predicates into two buckets: those that reference only partition columns, and everything else. Only the first bucket reaches the conflict check. A predicate on id is stored, and then ignored. So is a min/max range that would have made the answer obvious — the checker does no data skipping at all, which the Delta project has an open issue to change.

One line in OptimisticTransaction.scala settles the whole question:

val partitionPredicate: Expression =
  partitionPredicates.reduceLeftOption(And).getOrElse(Literal.TrueLiteral)

An empty list of partition predicates does not mean "filter nothing out." It collapses to TRUE, which matches every file the winner wrote. Contributing no partition predicate and contributing a predicate that matches everything are the same thing to this code.

Two consequences follow, and they are the whole lesson:

• An equality between two tables is not a filter. s.event_date = t.event_date mentions the partition column but constrains nothing on its own — the value comes from the source at runtime. Select condition c1 in the panel with the table partitioned by date: the predicate count stays at zero and you still conflict. Only t.event_date = '2026-08-12', a literal against the target, becomes a usable partition predicate.
• On an unpartitioned table there is nothing to filter with. The code short-circuits: no partition columns means the first candidate file is returned unconditionally. Any concurrent modification anywhere in the table conflicts with any transaction that read the table. This is why the error so often reads Files were added to the root of the table by a concurrent update — "the root of the table" is what Delta prints when there are no partition values to name.

That second point is why "just partition the table" became folklore. It works for the reason the docs give: partitioning by the columns your conditions filter on makes the two file sets provably disjoint at directory granularity. Note that this is Hive-style partitioning — directories on object storage — and not Spark's in-memory partitioning, which shares the word and nothing else.

Serializable is the opposite of a fix

The most common wrong turn is reaching for delta.isolationLevel = 'Serializable', because stronger sounds safer. Tick the box in the panel with the winning commit set to INSERT INTO: a scenario that could not conflict now does.

Delta's default is WriteSerializable, and the concession it makes is precise. When a commit only adds files and never read the table — Delta marks it isBlindAppend: true — those files are excluded from everyone else's append check. A blind append cannot invalidate anybody's read, because the serial order can simply be rewritten to put the append last. Under Serializable, that reordering is not allowed, so blind appends become candidates like everything else and your MERGE starts colliding with plain inserts.

The flag is computed, not declared: a transaction is a blind append only if it added no read predicates and read no files. An INSERT INTO … SELECT whose subquery reads the target table is not a blind append, and neither is a MERGE, however simple. This is the same read-set-versus-write-set reasoning as Postgres isolation levels, with one structural difference: Postgres detects conflicts against live transaction state, while Delta detects them against committed JSON files in the transaction log, after both sides have already done all their work.

Notice also who fails. The exception is always raised by the transaction that read the data — your MERGE — never by the INSERT that appended it. Retry logic belongs on the MERGE. Wrapping the ingest job in retries changes nothing.

Row-level concurrency, and the trap in enabling it

Since Databricks Runtime 14.3 LTS, conflict detection can drop from partition granularity to row granularity. Two MERGEs that modify different rows of the same file no longer conflict. It is not a setting you turn on; it activates when three conditions hold at once:

• Databricks Runtime 14.3 LTS or above,
• deletion vectors enabled on the table, and
• the table is not partitioned.

That third condition is the trap, and the panel is built to make you walk into it. Partitioning is the classic remedy for ConcurrentAppendException. Row-level concurrency is the modern one. They are mutually exclusive. A table partitioned by event_date gets partition-granularity conflict detection and nothing finer, no matter which runtime you are on — you can see it in the error itself, which grows a PARTITIONED_TABLE_WITHOUT_MERGE_SOURCE sub-condition when row-level resolution was attempted and declined.

Row-level detection has its own boundary. Set the winning commit to OPTIMIZE … ZORDER BY on the unpartitioned table with deletion vectors on, and the conflict comes back regardless of where your keys are. Plain bin-packing OPTIMIZE on a deletion-vector table is invisible to concurrent writers, because it changes no rows; Z-ordering shuffles rows between files, so every concurrent writer's file set moves under it. If your nightly ZORDER job overlaps your streaming MERGE window, that is your conflict, and no amount of predicate tightening will help.

Databricks also declines row-level resolution for conditions it cannot evaluate — struct, array and map comparisons, non-deterministic expressions, correlated subqueries — reporting PREDICATES_NEED_REWRITE, and it gives up under load with ALLOTTED_TIME_EXCEEDED rather than blocking the commit. The guarantee is best-effort narrowing, not an absence of conflicts. Liquid clustering is the way out of the partitioning fork: it gives you data locality on chosen keys while leaving the table unpartitioned, so row-level concurrency stays available.

The other exceptions, and what each one means

ConcurrentAppendException is one of six, and reading the class name tells you which side lost:

• ConcurrentAppendException — the winner added files matching your read set. Narrow your read predicate.
• ConcurrentDeleteReadException — the winner removed a file you had already read. Typical of a MERGE racing an OPTIMIZE. Not fixable by predicates; the file is gone.
• ConcurrentDeleteDeleteException — you and the winner both removed the same file. Two compactions on the same range.
• MetadataChangedException — someone changed schema, table properties or protocol. This can fail every concurrent write, including blind inserts. Schedule DDL away from your write windows.
• ConcurrentTransactionException — two streaming queries share a checkpoint location. This is a bug in your job configuration, not a tuning problem.
• ProtocolChangedException — the table's reader or writer version moved, or several writers tried to create the same table at once.

Retries are still legitimate — Delta's own commit loop retries internally, and a conflict is by definition a signal that the world moved. What retries cannot do is fix a workload whose conflict probability is near one. If twenty notebooks MERGE into one unpartitioned table with join-key-only conditions, every one of them re-reads and re-writes its files on each attempt, and the job gets slower in proportion to how much work it wasted. Cut the conflict rate first; retry the residue.

Checking it yourself

Run DESCRIBE HISTORY my_table and read three columns on the commit that beat you. operation tells you which side it was. isBlindAppend tells you whether WriteSerializable would have excluded it — if that is false on something you assumed was a plain insert, look for a subquery reading the target. operationParameters.predicate shows the predicate Delta actually extracted from your statement; if it is [] or mentions no partition column, your condition contributed nothing to conflict avoidance.

On Databricks the exception carries SQLSTATE 2D521 and an error sub-condition. WHOLE_TABLE_READ means your transaction tainted the entire table. ROW_LEVEL_CHANGES means row-level detection ran and found a genuine overlap — that one is real contention, and the answer is to partition the work, not the table. In open-source Delta the equivalent evidence is the driver log line Partition predicate is matching a file changed by the winning transaction, which prints both the predicate and the offending path.

Next: the layout decision underneath all of this, Z-ordering and per-file statistics, and the log structure the whole protocol rests on, Delta's transaction log. For the version of this problem in a system with real locks, see Postgres MVCC and vacuum.