How the Apache Arrow IPC file format works
Explore Arrow columnar buffers, record batch messages, aligned IPC bodies, file footers, random access, memory mapping, and zero-copy reconstruction.
The whole story in 6 lines
Follow columnar arrays into aligned buffers, record-batch messages, footer metadata, random access, and a zero-copy Arrow read path.
- Column-major buffers keep values of one field contiguous, reducing unrelated memory traffic during projected scans.
- An Arrow IPC file uses leading and trailing magic bytes, encapsulated messages, and a footer that indexes stored blocks.
- Arrow arrays separate validity, optional offsets, and value data so fixed-width and variable-width types share one model.
- RecordBatch metadata describes each buffer, while the message body stores aligned buffers in a contiguous layout.
- The footer repeats the schema and indexes dictionary and record-batch blocks, enabling direct access without a full scan.
- A compatible memory-mapped reader can create Arrow array views over aligned buffers without decoding every value.
Setup
Welcome. Apache Arrow is a language-independent columnar memory format. A Schema defines the structure of your data, listing every field by name and data type. Arrow schemas are strongly typed: an INT64 column is always 8 bytes per value, a UTF8 column stores variable-length strings. Let us learn the key terms you will see in every stage ahead.
A Record Batch is a chunk of rows stored in columnar layout. Each batch carries a slice of the table as a set of equal-length column arrays. Batches are the unit of data transfer. A file holds one or more of them.
A Buffer is one continuous block of bytes. Each column uses buffers for its values and may add buffers for null markers or string offsets. We will unpack those validity and offset structures later. For now, think of buffers as the physical pieces that carry Arrow data.
Arrow uses FlatBuffers for metadata such as schemas, record batches, and the file footer. A reader can follow that structured metadata directly from mapped bytes. That is the first hint of Arrow's zero-copy design, because the description of the data does not need a separate decoding pass.
Our journey follows six questions, from columnar memory through the zero-copy read path. We will meet validity bits and buffer alignment when their jobs become visible inside the file. Now let us begin with the most fundamental idea: why columnar layout changes analytical work.
Row vs Columnar Memory
★ If you remember one thing · Columnar layout lets a projected query read one contiguous value buffer instead of stepping through every field in every row.
Here is a small table with four columns: id (INT32), name (UTF8), price (FLOAT64), and qty (INT32). On the left, data is stored row by row. Each row is a struct with all four fields packed together. On the right, the same data is stored column by column. Each column is a contiguous array.
In row-major layout, a single row occupies a variable number of bytes because the name field has variable length. Accessing the price of row N requires skipping over every field that comes before it. There is no fixed stride.
The stage has reached the decision that determines its next state. Which query must touch every displayed Arrow column buffer?
Watch what happens when we compute SUM(price). In the row store, the CPU loads cache lines that contain id, name, and qty bytes it will never use. Those wasted bytes evict useful data from cache. The scan touches every byte in the table to read one column.
Switch the Scan Target control through every choice. Compare the downstream outcome while the earlier input and system boundary remain fixed.
That is the core tradeoff. Columnar layout sacrifices random row access for massive throughput on analytical scans. Arrow locks this decision into its memory format so that every library in every language gets the same cache-friendly layout. Next, let us open an Arrow IPC file and see how it organizes these columnar arrays on disk.
IPC File Anatomy
An Arrow IPC file organizes columnar buffers for durable interchange. It starts and ends with the ARROW1 magic plus padding, allowing a reader to verify the container before interpreting messages and footer blocks.
Immediately after the opening magic comes the Schema message. This is a Flatbuffer-encoded description of every field: name, data type, nullability, and any nested children. The schema is written once and tells any reader the exact structure of every record batch that follows.
After the schema come one or more Record Batch messages. Each message has two parts: a Flatbuffer metadata header describing buffer offsets, lengths, and null counts, followed by the raw buffer body. The body is the actual column data.
Optionally, the file can include Dictionary Batch messages before or between record batches. Dictionary batches define lookup tables for dictionary-encoded columns. The actual record batch data then stores integer indices instead of full values. This is how Arrow handles categorical strings efficiently.
At the very end, just before the closing magic bytes, sits the Footer. The footer is another Flatbuffer message. It contains a copy of the schema plus a Block index: an array of (offset, metadataLength, bodyLength) triples, one per record batch. This index is what makes random access possible.
A reader can start with the final ten bytes, which contain the footer length and closing ARROW1 magic. It then reads the footer and follows one Block entry directly to a record batch. The reader does not scan earlier batches first. Next, let us zoom into the buffer layout of a single column.
Buffer Layout
We just saw that record batches contain raw buffer bodies. Now let us zoom in and see what those buffers actually look like for a single column. The buffer layout depends on the data type.
Every nullable column starts with a Validity Bitmap. This is a bit array packed 8 rows per byte. Bit position 0 is the least significant bit. A 1 bit means the value is present. A 0 bit means null. For 1000 rows, the validity bitmap is only 125 bytes plus padding to the next 64-byte boundary.
For a fixed-width type like INT64, the data buffer is a flat array of 8-byte values. Row N lives at offset N * 8. There is no length prefix, no delimiter, no framing of any kind. The CPU can load values directly into SIMD registers. A null row still occupies 8 bytes in the data buffer. The validity bitmap is the only place that records nullness.
For a variable-length type like UTF8, the layout adds an Offsets buffer between validity and data. The offsets buffer is an array of INT32 values. offset[i] is the byte position where string. I starts in the data buffer. offset[i+1] minus offset[i] is the byte length of string i. So N strings require N+1 offset entries.
The UTF8 data buffer stores all string bytes packed end to end with no delimiters. Reading string i means: check validity bit i, read offset[i] and offset[i+1], then slice data[offset[i]..offset[i+1]]. This avoids the overhead of per-string length prefixes that formats like Protobuf or Thrift use.
IPC buffers obey alignment and padding rules, with eight-byte alignment as the minimum for record-batch bodies. Arrow recommends 64-byte alignment when an implementation can provide it because that suits wide SIMD access. The hatched tail in the picture is padding, not another value. Next, let us pack several buffers into one record batch message.
Record Batch Message
We now know what a single column looks like in memory. But a record batch contains multiple columns. How does Arrow pack them into one contiguous message that can be written or sent as a single unit. The answer is a two-part message: metadata header plus body.
The metadata header is a Flatbuffer message. It contains the number of rows in the batch, and for each column, a FieldNode with the column length and null count. After the field nodes comes a list of Buffer descriptors. Each descriptor is a (offset, length) pair pointing into the body. The header is a map of the body.
The body is where the actual array bytes live. Column buffers are flattened in schema order. A nullable variable-width array contributes validity, offsets, and data buffers, while a primitive array usually contributes validity and values. Padding keeps every body buffer at least eight-byte aligned.
The buffer descriptors in the metadata use absolute byte offsets from the start of the body. This means a reader can seek to any individual buffer without parsing the buffers that came before it. Reading column 47 out of 200 is a single seek plus a read of exactly the bytes described by that buffer descriptor.
The payoff is the complete IPC envelope. It starts with the four-byte 0xFFFFFFFF continuation marker and a four-byte little-endian metadata length. FlatBuffer metadata follows, then padding to an eight-byte boundary and the body. The entire serialized message remains a multiple of eight bytes.
This design means writing a record batch is a sequential write of metadata plus body. Reading it is: parse the small metadata header, then use the buffer descriptors to access column data directly without any further parsing. The body bytes are the final in-memory representation. Next, let us look at how the footer enables random access to any batch in the file.
Footer & Random Access
We now understand how each record batch is structured. But a file can hold hundreds of batches. How does a reader find batch N without scanning every byte before it. That is the job of the footer.
The footer is a FlatBuffer message near the end of the file, before its length and the closing ARROW1 magic. It repeats the schema, so the reader does not need to seek back to the beginning. It also carries separate Block arrays for record and dictionary batches.
Each Block carries an offset, metadata length, and body length. The offset gives the message position in the file. The other two values describe the message metadata and body extents. Together they tell the reader exactly where one batch begins and how much data belongs to it.
The stage has reached the decision that determines its next state. What lets an Arrow file reader jump directly to a record batch?
This end-first design means a writer can stream record batches sequentially, appending each one. And only write the footer at the end when it knows all the offsets. A reader can open a multi-gigabyte file and access batch 500 out of 1000 with exactly three reads: the tail magic, the footer, and the target batch. No scanning.
The footer is what turns a flat byte sequence into a random-access columnar database file. Without it, Arrow IPC would only support sequential streaming. With it, any batch and any column within that batch can be accessed in constant time. Now let us see the ultimate payoff: the zero-copy read path.
Zero-Copy Read Path
Everything we have learned so far exists for one reason: to make reading columnar data as cheap as casting a pointer. This stage shows the zero-copy read path and why it is fundamentally different from traditional deserialization.
A read-and-reconstruct path copies file bytes into a userspace buffer, then allocates another representation while decoding values. The repeated byte tapes show that extra movement. Exact traffic depends on the reader and codec, but each extra pass consumes memory bandwidth before computation begins.
A compatible reader can memory-map the file into its virtual address space. Pages still arrive from storage through the operating system, but mmap avoids a separate read copy into another userspace byte buffer. The same file-backed pages become addressable on demand.
Because Arrow buffers already use their final in-memory representation, a compatible reader can avoid reconstructing every value. It reads metadata to locate the buffers, then creates array views over them. A read-and-copy path adds allocation and copying before computation can begin.
Here is the contrast. A compatible Arrow reader can examine the metadata and construct array views over the mapped body buffers. Encoded or compressed storage formats may need decompression and decoding first. Zero-copy describes avoiding reconstruction of Arrow body data, not a promise that disks or operating systems never move bytes.
That is the full payoff. Columnar layout keeps related values together, while aligned buffers support efficient access. FlatBuffer metadata locates those buffers and the footer locates each batch. With compatible uncompressed buffers, mmap can expose the serialized body through array views without copying every value. Now let us step back and connect the whole picture.
Recap
We have walked through the entire Apache Arrow IPC file format, from the fundamental choice of columnar layout to the zero-copy read path that it enables. Let us connect all six concepts into one picture.
We started with columnar memory layout: the decision to store each column as a contiguous array instead of packing rows together. This single choice is what makes SIMD processing, cache-line efficiency, and fixed-stride access possible.
We then opened the file and saw its anatomy: ARROW1 magic bytes, a schema message, record batch messages, optional dictionary batches, a footer, and closing magic. The format is self-describing and designed for random access from the end.
We zoomed into one column. Validity bitmaps encode nulls as bits, fixed-width types use flat value arrays, and variable-width types add offsets. IPC guarantees aligned buffers, with 64-byte alignment recommended where practical.
We saw how a record batch message packs multiple columns. FlatBuffer metadata carries buffer descriptors, while the contiguous body places each buffer at an aligned offset. The metadata is a compact map of those body bytes.
We examined the footer and its Block index. Three numbers per batch: offset, metadata length, body length. The reader starts from the last 10 bytes, reads the footer, and can seek to any batch in constant time.
Finally, we saw the compatible zero-copy read path. Memory-map the file, read metadata, locate a buffer, and construct an array view over those body bytes. The metadata is examined, but the values do not need reconstruction into another buffer.
Together, these six ideas make Apache Arrow the lingua franca of columnar data. Columnar layout for throughput. Typed buffers for safety. Alignment for SIMD, which completes this part of the mechanism. Flatbuffers for zero-copy metadata. A footer for random access. And mmap for zero-copy data, which completes this part of the mechanism. That is the complete Arrow file format.
The whole story in 6 lines
Follow columnar arrays into aligned buffers, record-batch messages, footer metadata, random access, and a zero-copy Arrow read path.
- Column-major buffers keep values of one field contiguous, reducing unrelated memory traffic during projected scans.
- An Arrow IPC file uses leading and trailing magic bytes, encapsulated messages, and a footer that indexes stored blocks.
- Arrow arrays separate validity, optional offsets, and value data so fixed-width and variable-width types share one model.
- RecordBatch metadata describes each buffer, while the message body stores aligned buffers in a contiguous layout.
- The footer repeats the schema and indexes dictionary and record-batch blocks, enabling direct access without a full scan.
- A compatible memory-mapped reader can create Arrow array views over aligned buffers without decoding every value.