Databases · storage · distributed systems
See database and distributed systems from the inside
Storage engines, query execution, streaming systems and the protocols that keep data correct. Follow a guided path from first principles to the behavior that matters in production.
Choose a path
01 / Pick where to begin
Choose a learning path
25 guides across 3 paths01 / Storage · MVCC · query execution
Foundations
0 of 9 runs completeHow does Postgres read old rows without locking everything up?
How Postgres actually decides which row version you see, why vacuum exists and what freezing has to do with transaction wraparound.
What really happens inside a MySQL index lookup?
How InnoDB stores rows in clustered leaves, descends separators, splits full pages, follows secondary keys, and trades read speed for write and cache cost.
How does DuckDB turn SQL into vectorized pipelines so quickly?
Why DuckDB is so fast: how SQL becomes a physical plan that gets sliced into vectorized pipelines operating on column chunks.
What do databases mean when they promise consistency?
Why these four words are not interchangeable, which anomaly each model forbids and where Spanner, Cockroach, Postgres and DynamoDB actually sit on the lattice.
Why do RocksDB LSM trees stay write-fast even when compaction gets messy?
The shared LSM skeleton behind both engines: how writes stay fast through the WAL and memtables while compaction quietly reshapes levels.
How does Pinot turn one SQL query into coordinated distributed stages?
A source-backed journey from broker planning and leaf scans through mailbox exchange, local hash joins, partial aggregation, root output, and stage-level diagnosis.
How does Spark choose between broadcast speed and shuffle pain?
How Spark picks between broadcast and shuffle, why sort merge joins dominate at scale and what AQE does when data skew shows up.
How does RAG dig up the right context?
How RAG digs up the right context: chunking, embeddings, approximate nearest neighbor search, reranking and grounded answer generation.
How do you search 100 GB of text without scanning it?
Build full-text search from raw bytes: analyze terms, invert postings, test phrase positions, size and bulk-load shards, merge BM25 results, and maintain immutable segments.
02 / Consistency · replication · failure handling
Coordination
0 of 8 runs completeWhy do Kafka consumers stop for a rebalance and reshuffle?
Why consumers stop, what the coordinator actually does during a rebalance and how cooperative revocation makes it less painful.
How do Cassandra nodes discover peers and detect failures without a leader?
How seeds bootstrap discovery, generation and version tuples reject stale knowledge, peer exchanges spread state, and local failure detectors judge reachability.
How can distributed systems order events when clocks disagree?
How happens before becomes logical time, how vectors expose concurrency, and how uncertainty aware clocks support safe real time ordering.
How does Jepsen find distributed system bugs under real faults?
How Jepsen prepares a real cluster, schedules concurrent operations, injects network and process faults, records honest histories, and checks linearizability.
When does a retried operation really happen only once?
How delivery guarantees differ from effect guarantees, how idempotency patterns absorb retries, and where Kafka exactly-once semantics stop.
How does Flink keep duplicate writes out of the picture?
How Flink cuts a barrier through a live stream, snapshots state at every operator and replays cleanly after a crash.
How can you check a distributed protocol across every ugly interleaving?
A high-school-friendly path from state machines and invariants through PlusCal, TLA+ model checking, stale-leader counterexamples, epoch fencing and refinement.
How do databases merge a record that carries only a few changed fields?
A cross-database guide to overwrite, increment and append semantics through MongoDB, DynamoDB, PostgreSQL, Cassandra, Redis Active-Active and retry guards.
03 / On-disk structures · logs · data formats
Storage & formats
0 of 8 runs completeHow does an LLM-maintained wiki compound knowledge over time?
What happens when an LLM maintains its own wiki instead of re-deriving the same answers on every query.
How do realtime workers keep ClickHouse ingestion fast and safe?
How worker pools read ordered stream lanes, batch and shape rows, insert blocks into ClickHouse, and use part pressure plus retries to keep offsets safe.
How does Pinot keep stream rows queryable while it builds immutable segments?
A code-backed tour from Kafka fetch and mutable indexing through winner election, split commit, seven pauseless stages, and failure-specific repair.
How does Pinot keep one logical row while old versions remain inside segments?
A code-backed tour of full upsert record selection, valid and queryable bitmaps, deletes, out-of-order policy, compaction, recovery, TTL, and consistent query views.
How does Iceberg time travel without copying the whole table?
How Iceberg gets time travel and branching without ever rewriting a data file, just by being clever with metadata pointers.
How does Git turn files into immutable history under the hood?
An engineering-level tour of canonical object bytes, loose-object fanout, index-to-tree assembly, commit and ref resolution, reachability, structural sharing, packfiles and delta storage.
How does Arrow make columnar data zero-copy across every language?
The full Arrow IPC file format from columnar memory layout through validity bitmaps, buffer alignment, record batch serialization, footer random access and the zero-copy mmap read path.
How does the Parquet file format organize columns and skip data?
The full Parquet file format from columnar vs row storage through row groups, page internals, dictionary/RLE/delta encodings, Dremel repetition and definition levels and predicate pushdown.
About this collection
Data infrastructure is easiest to understand when the hidden state becomes visible. A database query is not a single operation: it crosses planners, indexes, memory structures and storage layers. A distributed write is also more than a network call. It moves through replication rules, ordering guarantees and recovery paths that decide what users eventually observe. This collection turns those internal paths into systems you can inspect one transition at a time.
Start with PostgreSQL MVCC for a practical model of versions and visibility, then follow the path that matches the system you are working on. The guides cover storage formats, stream processing, query execution, consensus and failure testing. Each one keeps the mechanism connected to the engineering question it answers, so you can compare tradeoffs without reducing correctness to a slogan. Use the collection when you are designing a data path, preparing for a systems interview or debugging behavior that crosses several components.