Interactive simulations of engineering internals
Understand the mechanism. Then hand it to AI.
AI is a sword: decisive in a ninja’s hands, a danger to an amateur’s — the blade is the same. What differs is what the hand already knows. Every lesson here is a working model of a real system you can operate, so that what you bring to AI is deep.
Pick a system
Or follow the reading order →LLM internals
Attention, KV cache, batching, quantisation. Why inference costs what it costs.
Security
TLS, auth, browsers and the transport underneath — the parts people assume are simpler than they are.
Agentic Security
What an agent can be tricked into, and where the only real enforcement points are.
Postgres
MVCC, vacuum, locking, indexes. Why the table does not shrink and the query does not use the index.
RAG
Chunking, ranking, fusion, reranking. Why the retriever returns the wrong thing.
Kafka
Consumer groups, rebalancing, lag. Why the cluster is healthy and you are still behind.
Databricks
Delta Lake internals — the transaction log, file layout, clustering, deletion vectors.
Kubernetes
Scheduling, cgroups, limits, eviction. Why the pod is throttled at 30% CPU.
Spark
Shuffle, skew, AQE, the optimizer. Why the job sits at 99% for an hour.
Graph RAG
Indexing and querying knowledge graphs for retrieval, and where the fan-out explodes.
Networking
TCP, congestion control, windows, buffers. Why a fast link is slow.
What a lesson is
A mechanism you operate
Not a diagram of the thing — the thing. Move a parameter and watch the system react, including into the states where the usual advice stops working.
Built around one wrong belief
Every lesson names the specific misconception it exists to delete. The expensive bugs are not in what engineers find hard; they are in what engineers are sure they already understand.
Checked against the source
Claims are verified against the project’s code, RFCs and papers, and every number in the prose is produced by the simulation on the page. Lessons that turn out wrong come down.
Try one now · Spark / execution / shuffle · move the control
Even distribution. Every core finishes together — the only state in which adding machines helps.
A stage ends when its slowest task ends.
Easy to nod at, hard to believe until you have moved the slider yourself. That gap — between agreeing with a sentence and having a model of it — is what every lesson here is for.
Read the full shuffle lessonRecently published
All 101 →Kernel Enforcement Cannot See Intent
That putting Tetragon or Falco under an agent contains it, because eBPF sees everything and blocks in-kernel at microsecond latency. Coverage really is that good and the block really does hold. The gap is semantic: the hook sees an outbound connection to an allowed address, not that a prompt injection caused it or that the bytes are rows from a table the agent was never meant to query. Because a legitimate agent's paths, hosts and processes drift prompt to prompt, a learned allowlist is either loose enough to contain the attacker's primitives too or tight enough to break the agent — and the settings in between shrink to nothing as that drift grows. The designs that work concede this by construction: they use eBPF to see and to redirect, and put the decision at a layer that has the nouns — an L7 proxy, a per-agent identity, a network policy, an admission rule. The cost of not knowing that is a control that is trusted, correct, and produces no security.
The Gateway Reports On Its Own Denominator
That transparent interception means the gateway governs the agent's model traffic, because the words 'transparent' and 'no source changes' make coverage sound like a property of eBPF. It is a property of your hook set. A cgroup/connect4 program sees TCP over IPv4 from the processes in its scope, opening new connections — not IPv6, not a subprocess outside the scope, not a connection that was already established, and never a unix socket. Each of those is a live model call that never reaches the proxy. The failure is invisible because the gateway's own dashboard is computed over the calls it handled: a gateway governing 59.5% of the fleet reports 100% compliance, and the missing calls are not counted as violations, they are simply absent. On top of that, reading the request at all requires the payload to be readable, so a hosted model over TLS means terminating your own agents' connections with a certificate you issue; the proxy sits in the data path with a latency and a failure mode; and a gateway that rewrites prompts is your own infrastructure writing into the model's context.
A nonce Is Not a Second state Parameter
That nonce is a second, redundant anti-CSRF token: state already proves the callback belongs to this browser, so the nonce is decoration you can leave unverified in the authorization code flow. state is checked at the redirect, against the browser session. nonce is checked after the token exchange, against a claim inside the ID Token. In an authorization code injection attack the attacker starts their own flow and swaps in a stolen code, so their own state comes back and matches by construction — the state check passes, and the only thing that says the ID Token describes someone else is the nonce claim the client was told it could skip.
A speed test cannot see bufferbloat, and a smaller buffer is not the fix
That bufferbloat is a hardware defect in somebody else's router, that a speed test will show it, and that the cure is a smaller buffer or a faster link. The queue is built by your own bulk flows, so a speed test — which measures exactly the quantity bufferbloat does not damage — reports the link as healthy while a video call on the same link is unusable. Shrinking the buffer trades latency for throughput and there is no single size that is right at two different link rates. And selecting fq_codel changes nothing at all unless you also shape a few percent below line rate, because until you do, the queue forms in the modem downstream of every qdisc you configured.
Agent Memory Poisoning
That an injection is a per-session event you recover from by starting a new chat. The write-back step is the bug: the agent summarises its own compromised turn into the memory store, and from then on retrieval — not the attacker — supplies the payload. Because the same loop keeps adding near-duplicate records around whatever query triggered it, the retrieval probability rises with every hit instead of decaying, and the record that reinfects you was authored by your own agent, so every provenance check that looks at 'where did this text come from' reads clean.
Approval Fatigue as a Decaying Control
That human-in-the-loop is a control with a fixed strength you can put in a threat model. It is a rate, and the rate is set by the volume of benign prompts, not by the reviewer's diligence. Every standard — OWASP, the MCP specification — recommends it, and none of them price the decay. Two things then finish the job: the prompt shows a truncated action, so the approver reviews a label rather than the arguments; and 'always allow' converts one tired decision into a permanent grant that no later prompt will revisit. Chrome's SSL interstitial, a far starker warning shown far less often, still had a 70.2% click-through rate.