The future of column-oriented data processing with Arrow and Parquet

A session at Strata + Hadoop World in London 2016

Thursday 2nd June, 2016

2:55pm to 3:35pm (GMT)

In pursuit of speed and efficiency, big data processing is continuing its logical evolution toward columnar execution. Julien Le Dem offers a glimpse into the future of column-oriented data processing with Arrow and Parquet.

A number of key big data technologies have or will soon have in-memory columnar capabilities. This includes Kudu, Ibis, Drill and many others. Modern CPUs will achieve higher throughput using SIMD instructions and vectorization on Apache Arrow’s columnar in-memory representation. Similarly Apache Parquet will provide storage and I/O optimized columnar data access using statistics and appropriate encodings. For interoperability, row-based encodings (CSV, Thrift, Avro) combined with general-purpose compression algorithms (GZip, LZO, Snappy) are common but inefficient. Julien explains why the Arrow and Parquet Apache projects define standard columnar representations that allow interoperability without the usual cost of serialization.

This solid foundation for a shared columnar representation across the big data ecosystem promises great things for the future. Julien discusses the future of columnar data processing and the hardware trends it can take advantage of. Arrow-based interconnection between the various big data tools (SQL, UDFs, machine learning, big data frameworks, etc.) will allow using them together seamlessly and efficiently without overhead. When collocated on the same processing node, read-only shared memory and IPC avoid communication overhead; when remote, scatter-gather I/O sends the memory representation directly to the socket, avoiding serialization costs; and soon RDMA will allow exposing data remotely.

As in-memory processing becomes more popular, the traditional tiering of RAM as working space and HDD as persistent storage is outdated. More tiers, like SSDs and nonvolatile memory, are now available that provide much higher data density, achieving a latency close to RAM at a fraction of the cost. Execution engines can take advantage of more granular tiering and avoid the traditional spilling to disk which impacts performance by an order of magnitude when the working dataset does not fit in main memory.

About the speaker

This person is speaking at this event.
Julien Le Dem

@DremioHQ Architect, formerly tech lead at @Twitter, @Yahoo, Kelkoo, @ApacheParquet co-author and VP, Apache Pig and @ApacheArrow PMC, Kitesurfer, parent

Sign in to add slides, notes or videos to this session

Tell your friends!


Time 2:55pm3:35pm GMT

Date Thu 2nd June 2016


ExCeL London

Short URL


Official session page


View the schedule


See something wrong?

Report an issue with this session