How DuckDB query execution pipelines work
Follow SQL through planning, pipeline boundaries, parallel scans, DataChunks, vectorized operators, blocking sinks, and result delivery.
The whole story in 7 lines
DuckDB streams column batches through every locally answerable step, then waits at blocking state until a complete answer can safely...
- The physical plan turns SQL semantics into concrete operators whose dependencies determine the later execution graph.
- A pipeline breaker ends streaming, materializes state through a sink, and creates a new source for downstream work.
- Scans emit bounded DataChunks made of column vectors, cardinality, and vector-format metadata.
- Vectorized operators process many rows per call and can filter with selection vectors instead of copying full rows.
- Sink, combine, and finalize merge thread-local state into the materialized result required by the next pipeline.
- The finalize barrier prevents incomplete thread-local groups from escaping into the downstream result source.
- The result pipeline reads finalized state in chunks, applies remaining operators, and emits bounded result batches.
Setup
Our grouped sales query is already scanning data, and batches are moving toward an aggregate. Then they pile up while the result side stays empty. The puzzle is why the query can start working before it can start returning final rows.
Call the uninterrupted part of that work a pipeline: a source produces batches, operators transform them, and a sink receives them. This name does not yet tell us why the flow stopped. It only gives us a boundary to investigate.
The moving object is a DataChunk, a collection of column vectors that represents many row positions at once. DuckDB passes this batch between physical operators instead of making one operator call for every row.
Each operator performs one job on that batch. A scan supplies values, a filter chooses lanes, and a projection computes new vectors. Several such jobs can keep the same DataChunk moving without owning the whole query result.
The sink is where a pipeline hands over what it produced. Sometimes that handoff is merely another streaming step. Sometimes the sink must retain state whose answer is still changing. Our waiting aggregate is that second kind, but we have not yet earned the reason for a new pipeline.
We will keep this stopped query in view while its SQL becomes a plan, its columns become DataChunks, and its partial groups become one finalized table. First we need to see which physical jobs DuckDB chose before any batch was allowed to move.
SQL Becomes a Plan
The stopped batches came from this actual request: regional totals for 2025 sales of at least 40. The SQL says what answer we want, but it does not name the loops, state, or handoffs that can produce it.
DuckDB first parses those clauses into a syntax tree. At this point `region` and `amount` are still names from the query text. The parser has shaped the request without proving that those columns exist or deciding their types.
The binder now consults the catalog, connects each name to a real table column, resolves expression types, and extracts the aggregate. The same tree has become specific enough for later planning to reason about actual values rather than unresolved words.
Because the `year = 2025` and `amount ≥ 40` predicates only need scan columns, the optimizer can move that filter toward the table and discard rows early. This is still planning: the tree changes so less data will travel when execution finally begins.
The abstract request has now become concrete physical operators: scan, filter, projection, and hash group-by. That is real progress, but the neat bottom-to-top tree also invites a dangerous picture—that every operator can run as one continuous conveyor.
The tree tells us which job depends on which earlier job. It does not say that all of them can stream at once. To find where our batches stopped, we must ask what happens when a DataChunk climbs from the scan and reaches the hash aggregate.
Pipelines Form at Blockers
★ If you remember one thing · A blocking operator cuts one physical plan into separate pipelines that can run with different sources and sinks.
Take the completed operator tree and imagine one DataChunk entering at the scan. The question is no longer what the jobs are. It is how far that batch can travel before a later job needs information from batches that have not arrived yet.
Scan, filter, and projection can form Pipeline 1 because each call can hand its current output onward. A qualifying sales lane does not need to wait for the last table morsel before its projected values reach the group-by sink.
At hash group-by, however, EU’s total is still provisional while another chunk may contain more EU rows. The next operator needs stable groups, not a parade of changing guesses. What creates a new DuckDB execution pipeline?
The blocking operator ends Pipeline 1 as a sink and keeps consuming until its grouped state is complete. Only then can that materialized state act as the source of Pipeline 2. The break exists because the meaning of the data changed from partial input to stable groups.
Use the Blocker control to compare Hash Agg with Window. The stored state changes from grouped buckets to ordered window state, but in both cases downstream work waits because the selected operator needs more than the current chunk.
DuckDB can schedule whichever pipeline is ready, but splitting the tree has left one practical question unanswered: what a scan task actually hands to the next operator. We will follow the first morsel from column storage into that moving object.
Scans Emit DataChunks
Pipeline 1 is ready, so a worker begins at the column storage shown on the left. Our query needs `region`, `amount`, and `year`. It does not need a complete row object containing every column in the orders table.
The worker claims a morsel—a manageable slice of those column segments—rather than reserving the whole table. Other workers can claim different morsels, which gives DuckDB parallel scan tasks without making them duplicate the same slice.
The scan cursors read the selected segments together so position zero in each chosen column still describes the same logical row. Columnar storage saves unnecessary reads, but the next operator also needs those separate columns to travel as one aligned batch.
DuckDB assembles the aligned slices into a DataChunk: one vector per selected column plus a shared cardinality. This drawing uses eight lanes so we can inspect them. DuckDB’s documented standard vector size currently defaults to 2,048 tuples.
Vectors can use different formats. `amount` may be flat, repeated `year` constant, and dictionary data can retain child values plus selection indexes. The DataChunk preserves one logical row count across those representations.
We have found the object that crosses a streaming pipeline boundary: aligned column vectors with one cardinality, not a queue of row objects. The next puzzle is how a filter can discard lanes without copying every surviving row into a fresh batch.
Operators Run on Vectors
The scan hands us the eight illustrative `amount` values from the previous DataChunk. Each horizontal position is a logical row lane, while the values still live in a column vector rather than eight separate row objects.
The filter evaluates `amount ≥ 40` across the vector and produces one keep-or-reject decision per lane. It has learned which positions survive, but it has not yet copied their complete rows anywhere.
Those decisions compress into the selection indexes `0, 2, 3, 5, 7`. Downstream code can use those indexes as a view over the original vectors, so rejected lanes disappear logically while the underlying column data can stay where it is.
Projection now needs only the surviving `amount` values. If DuckDB wants to avoid copying filtered rows, which structure should tell projection where those values already live?
Projection follows those indexes, computes the new values, and hands only active lanes to the group-by sink. The sink’s buckets now accumulate keys and values from this chunk, but their totals remain provisional because other workers still have chunks in flight.
DuckDB repeats this vectorized loop for each DataChunk, which explains how work stays batched up to the sink. It still does not explain how several workers can build aggregate state concurrently and end with exactly one total per region.
Sink, Combine, Finalize
The projected lanes from Pipeline 1 now arrive as chunks on the left. Hash group-by cannot pass them through as final rows, because each region’s count or sum may still be increased by a later chunk from another worker.
Worker T1 therefore updates its own hash table instead of locking one global table for every lane. T2 and T3 can do the same with their chunks. Local ownership keeps the hot sink work parallel, even though none of those tables is yet the query’s answer.
As input continues, the three local tables disagree in an ordinary way: one has AP, another has more EU, and a third has more US. That disagreement is not a race or an error. It is the incomplete state we deliberately created to avoid constant cross-thread contention.
After every input task finishes, combine merges equal keys across the local tables. EU’s separate partial counts become one EU state, and the same happens for US and AP. The operator can finally describe the whole input instead of one worker’s share.
Finalize turns that merged sink state into the readable grouped rows shown on the right. This is the exact moment the aggregate stops being an unfinished destination for Pipeline 1 and becomes a trustworthy source for downstream work.
We have explained why DuckDB waits, but not whether the wait is merely a scheduling convenience. The next stage puts the finalize barrier under your control: release the source after one local table and see whether the client can still receive the right groups.
Test the Finalize Barrier
The three worker-local tables on the left are the unfinished states we just combined. Release the Finalize gate early, then restore it: the routes reveal whether the client receives one worker’s partial groups or the complete merged result.
Result Pipeline Streams Out
With the Finalize gate safe, the left table contains the complete answer: EU 510, US 330, and AP 240. This materialized state is no longer accepting partial updates, so a new source can read it without chasing moving totals.
A source cursor drains those stable groups back into column vectors. The pipeline has restarted from a different source, but its unit of work is familiar: bounded DataChunks still carry aligned columns rather than one callback per row.
As that cursor reads the three stable rows, DuckDB aligns each `region` key with its `total` value in a fresh DataChunk. No ordering or pruning is needed here. This downstream pipeline is only re-batching finalized groups for delivery.
Those aligned `region` and `total` vectors now form an outgoing result chunk. Nothing has reverted to a row-at-a-time engine: the client-facing path still uses the same columnar batch idea that connected scan, filter, and projection.
The client can now receive correct result batches while this final pipeline drains. The opening silence was meaningful: DuckDB began useful upstream work early, yet withheld final groups until the blocking state could safely become a source.
Once the last output chunk leaves, the whole query is complete. We can now connect the two ideas that seemed to conflict: vectorized pipelines keep local work moving, while explicit barriers keep global answers correct.
Recap
We began with SQL that asked for regional sales totals. Parsing shaped the request, binding connected names and types, optimization moved useful work earlier, and the physical plan finally named the operators that could execute it.
That plan looked like one conveyor until the hash aggregate exposed the flaw. Scan, filter, and projection could stream current chunks, but final regional totals depended on unseen chunks, so the aggregate ended one pipeline and later became another pipeline’s source.
Inside the first pipeline, workers claimed separate morsels from `region`, `amount`, and `year`. The scan aligned those column slices into DataChunks, letting many logical rows travel together while the engine remained columnar.
The amount predicate became a lane mask and then compact selection indexes. Projection followed those indexes into the existing vectors, computed new values for surviving lanes, and handed batched aggregate input onward without reconstructing full rows.
At the sink, each worker updated a private hash table so concurrent chunks did not fight over every group. After all input finished, combine merged equal keys and finalize turned the complete aggregate state into readable grouped rows.
The Finalize gate proved why that wait matters. Releasing one local table produced missing and undercounted groups. Holding the source until all three locals merged produced the complete result. The pipeline boundary is therefore part of correctness, not presentation.
With stable groups available, the downstream source read EU, US, and AP into aligned key and total vectors, and the resulting chunks reached the client. The engine stayed batched even though the source of those batches changed across the break.
The opening pause now makes sense. DuckDB does not choose between continuous speed and blocking correctness. It draws the boundary where each is valid. DataChunks keep local operator work moving, and finalized materialized state lets the next pipeline begin with a trustworthy answer.
The whole story in 7 lines
DuckDB streams column batches through every locally answerable step, then waits at blocking state until a complete answer can safely...
- The physical plan turns SQL semantics into concrete operators whose dependencies determine the later execution graph.
- A pipeline breaker ends streaming, materializes state through a sink, and creates a new source for downstream work.
- Scans emit bounded DataChunks made of column vectors, cardinality, and vector-format metadata.
- Vectorized operators process many rows per call and can filter with selection vectors instead of copying full rows.
- Sink, combine, and finalize merge thread-local state into the materialized result required by the next pipeline.
- The finalize barrier prevents incomplete thread-local groups from escaping into the downstream result source.
- The result pipeline reads finalized state in chunks, applies remaining operators, and emits bounded result batches.