Apache Parquet File Format: Row Groups, Pages and Footer
Apache Parquet is an open-source, column-oriented file format that uses row groups, pages and footer statistics to compress data and skip unnecessary reads.
The whole story in 7 lines
Parquet aligns fields into byte ranges, describes them with metadata and lets readers prove which ranges need no I/O.
- Columnar storage aligns analytical projections with contiguous field bytes instead of complete records.
- The footer maps row groups, column chunks and pages to byte locations and available statistics.
- Page headers describe encoded bodies, while optional dictionaries and codecs make each page independently decodable.
- Plain, dictionary, run-length, bit-packing and delta encodings exploit different value patterns and can layer.
- Definition and repetition levels reconstruct nulls and repeated lists from flat column streams.
- Projection and min/max proofs let readers reject unrelated fields and impossible row groups before decoding.
- Writer-chosen row-group boundaries change min/max ranges and therefore how much a later predicate can skip.
Setup
Our 2 TB people file contains far more than SELECT name, age WHERE age > 30 needs. The reader must locate useful data and reject impossible regions before decoding them.
Parquet first cuts the table into horizontal batches called row groups. Each group covers a different set of rows, so its metadata can describe that region independently.
Inside every row group, each field gets its own column chunk. Our query can follow the age and name chunks without crossing id, city or email bytes.
A column chunk is still too large to handle as one block, so Parquet divides it into pages. Each page carries encoded values and has its own compression boundary.
The missing piece is a map. Parquet writes one at the file’s end: a footer containing schema, chunk locations and statistics that guide the reader toward possible matches.
Those four objects let us follow the query instead of touring the format. We begin with the physical choice that makes selective reading possible: placing values from the same column together.
Row vs Column Storage
★ If you remember one thing · For an age-only query, row storage touches every field while Parquet can read one quarter of this four-column example.
We can see the storage choice using four people and four fields. The row file keeps each person together, while Parquet places all four age values beside one another.
A row file is not badly organized. It is organized for a different request. Fetching one complete person is convenient because that person’s id, name, age and city are adjacent.
Our analytical query wants every age but none of the other fields. Both files contain the answer, yet one layout makes the reader cross unwanted values. Which one?
The row file touches all four fields because each age is trapped inside a complete record. Parquet follows one contiguous age column, so this example reads one quarter of the displayed values.
Switch the Query control between SELECT age and SELECT *. The Parquet read grows from one column chunk to all four, while the row file already touches every field.
So columnar storage does not make every query smaller. It makes the query’s requested fields line up with physical byte ranges. Our age query can avoid other columns, but it still needs their exact locations.
File Anatomy
To find those ranges, the reader treats the Parquet file as an address space. The PAR1 markers frame one byte sequence whose map lives near the end.
The data area contains consecutive row groups. Each group owns a horizontal batch of records, and the writer chooses where one batch ends and the next begins.
Within each row group, the four fields separate into four column chunks. Two row groups therefore create eight chunks, each occupying a known byte range in the file.
Each chunk then breaks into pages, where encoding and compression happen. Optional page indexes can describe smaller ranges, but the file remains a sequence of bytes rather than nested folders.
The footer turns that byte sequence into something navigable. It records the schema and row-group metadata, including chunk offsets, encodings, codecs and available min, max and null counts.
Because the footer length sits beside the final PAR1 marker, a reader can fetch the tail first. The returned offsets lead directly to the age chunks instead of forcing a scan from byte zero.
Our query now knows which chunk bytes to request. Addresses alone cannot produce values, though. The reader still has to open those pages and reverse the transformations stored inside them.
Page Internals
The same footer lookup works for any projected field. We open the city chunk because repeated city names make the page machinery easier to see than the age values did.
A page begins with a Thrift-encoded header. Before touching the body, the reader learns its type, value count, compressed size, uncompressed size and the encoding needed to interpret it.
This chunk starts with an optional dictionary page containing Austin, Boston, Chicago and Denver once. Later data pages can refer to those strings with small integer IDs.
The Data Page V1 shown here lays three streams back to back: repetition levels, definition levels and encoded values. Flat required columns omit the unnecessary level streams, leaving only the values.
For this Data Page V1, the encoded body—including levels and values—is compressed and decompressed together. Data Page V2 differs: its level streams stay uncompressed, while its value section may be compressed.
For the V1 page shown, the reader decompresses the whole encoded body before separating its streams. One instruction still matters: the encoding named in metadata determines how compact bytes become the original city values.
Encodings
A reader needs more than ‘compressed values’ because different patterns shrink in different ways. The page metadata names an encoding, and the reader later applies its exact inverse.
PLAIN is the dependable baseline. Fixed-width numbers use their specified byte width, while a byte-array string stores its length followed by the string bytes. Repeated strings remain repeated.
Our city values repeat, so dictionary encoding writes each unique city once. The data page then carries IDs such as 0, 1, 0 instead of repeating Austin and Boston.
Those dictionary IDs are small integers, which creates another opportunity. The RLE and bit-packing hybrid collapses equal runs and packs bounded integers using only their required bits.
Ordered numbers have a different pattern. Delta encoding keeps the first value and records changes between neighbors, so 1000, 1001, 1003 becomes 1000 followed by 1 and 2.
Encodings can therefore layer rather than compete. A dictionary removes repeated strings, then the RLE and bit-packing hybrid compresses the resulting IDs. Metadata preserves the decoding recipe.
All of these tricks assume we already know where one logical value belongs. A repeated contacts list breaks that simple picture because a flat phone column no longer shows person boundaries or missing phones.
Repetition & Definition Levels
Suppose Alice has two contact records and Bob has one contact with no phone. If Parquet writes only a flat phone stream, the original grouping seems to disappear.
The schema supplies the missing path: person, then a repeated contacts list, then an optional phone. It also tells the reader how deep a present phone can be.
A definition level records how much of that path exists. Here D=2 reaches a real phone value, while D=1 says the contact exists but its optional phone does not.
Definition alone cannot separate Alice from Bob. The repetition level marks boundaries: R=0 starts a new person, while R=1 starts another contact for the current person.
The flat stream now carries three aligned entries. Alice begins at R=0, continues at R=1, and Bob begins at the next R=0 with D=1 and no phone value.
Those two level streams let the reader rebuild the branches that flattening removed. Because their maximum values come from the schema, RLE and bit-packing can store them compactly beside the leaf values.
So Parquet can keep phone values columnar without forgetting lists or nulls. Our original age query needs a different kind of help: it wants to avoid decoding irrelevant pages altogether.
Predicate Pushdown
We return to the query that opened the lesson: SELECT name, age WHERE age > 30. The reader has a footer map, but it still needs proof that some mapped regions are irrelevant.
Projection settles the first part. The footer points to name and age chunks, so id, city and email remain outside the planned byte ranges.
The age statistics now test the predicate before decoding. If a row group’s maximum age is 29, every value fails age > 30, so that group cannot contribute a row.
We have not opened a data page yet. Which metadata is enough to prove that an entire row group cannot contain a matching age?
Now the range proof becomes physical I/O. Row groups ending below 30 are skipped, while the group reaching 45 remains. An optional page index can narrow that surviving work further.
That finally explains how most of the file stays asleep. Yet one choice was made before the query arrived: the writer decided which rows share each min and max. That boundary can change the proof.
Row-Group Boundary Lab
To test that writer choice, this frame introduces eight illustrative sorted ages from 18 through 47. Drag their row-group boundary and watch each group’s min, max and READ or SKIP verdict recompute for age > 30.
Recap
We can now answer the opening mystery without calling Parquet magic. The query avoided work because several physical choices and metadata proofs cooperated in a particular order.
First, columnar storage placed all ages together. That changed the query from ‘cross every record’ into ‘follow the age byte ranges.’
The footer then supplied those ranges. Row groups divided records, column chunks divided fields, and pages created smaller units inside each chunk.
At a chosen range, page headers described what followed. Optional dictionaries and data pages held the streams, while a codec compressed each page body independently.
Encodings made those streams smaller by exploiting their actual patterns. Dictionary IDs could themselves become runs or packed integers, so the transformations could layer.
Repetition and definition levels repaired the part flattening seemed to destroy. They preserved person boundaries, repeated contacts and missing phones beside one flat value stream.
At read time, projection rejected unwanted fields and min/max statistics rejected impossible ranges. The engine could make both decisions before decoding their values.
The boundary lab added the writer’s side of that bargain. Moving the row-group split changed the same rows’ min and max, which changed how much the later query could safely skip.
Parquet’s advantage is therefore not one compression trick. Its layout gives queries useful byte boundaries, and its metadata lets readers prove which of those boundaries need no I/O.
The whole story in 7 lines
Parquet aligns fields into byte ranges, describes them with metadata and lets readers prove which ranges need no I/O.
- Columnar storage aligns analytical projections with contiguous field bytes instead of complete records.
- The footer maps row groups, column chunks and pages to byte locations and available statistics.
- Page headers describe encoded bodies, while optional dictionaries and codecs make each page independently decodable.
- Plain, dictionary, run-length, bit-packing and delta encodings exploit different value patterns and can layer.
- Definition and repetition levels reconstruct nulls and repeated lists from flat column streams.
- Projection and min/max proofs let readers reject unrelated fields and impossible row groups before decoding.
- Writer-chosen row-group boundaries change min/max ranges and therefore how much a later predicate can skip.