Apache Arrow

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware.

The project is developing a multi-language collection of libraries for solving systems problems related to in-memory analytical data processing. This includes such topics as:

• Zero-copy shared memory and RPC-based data movement

• Reading and writing file formats (like CSV, Apache ORC, and Apache Parquet)

• In-memory analytics and query processing

To learn how to use Arrow refer to the documentation specific to your target environment.

Supported Environments

• C/GLib
• C++
• C#
• Go
• Java
• JavaScript
• Julia
• MATLAB
• Python
• R
• Ruby
• Rust
• Implementation Status

Cookbooks

• C++
• Python
• R

Specifications and Protocols

• Format Versioning and Stability
  • Backward Compatibility
  • Forward Compatibility
  • Long-Term Stability
  • Pre-1.0.0 Versions
• Arrow Columnar Format
  • Terminology
  • Physical Memory Layout
  • Logical Types
  • Serialization and Interprocess Communication (IPC)
  • Implementation guidelines
• Arrow Flight RPC
  • RPC Methods
  • Authentication
  • Error Handling
  • External Resources
  • Protocol Buffer Definitions
• Integration Testing
  • Running integration tests
  • JSON test data format
• The Arrow C data interface
  • Rationale
  • Data type description – format strings
  • Structure definitions
  • Example use case
  • C producer examples
  • Why two distinct structures?
  • Updating this specification
  • Inspiration
• The Arrow C stream interface
  • Semantics
  • Structure definition
  • C consumer example
• Other Data Structures
  • Tensor (Multi-dimensional Array)
  • Sparse Tensor

Development

• Contributing to Apache Arrow
  • Code of Conduct
  • Join the mailing lists
  • Report bugs and propose features
  • Improve documentation
  • Contribute code
  • Experimental repositories
  • Guidance for specific features
• New Contributor’s Guide
  • Why contribute to Arrow?
  • Quick Reference
  • Full Table of Contents
• Reviewing contributions
  • Principles
  • Guidelines
  • Social aspects
• C++ Development
  • Building Arrow C++
  • Development Guidelines
  • Developing on Windows
  • Conventions
  • Fuzzing Arrow C++
• Python Development
  • Coding Style
  • Unit Testing
  • Building on Linux and MacOS
  • Building on Windows
• Continuous Integration
  • Continuous Integration
  • Running Docker Builds
  • Daily Development using Archery
  • Packaging and Testing with Crossbow
• Benchmarks
  • Setup
  • Running the benchmark suite
  • Comparison
  • Regression detection
  • Scripting
• Building the Documentation
  • Prerequisites
  • Building
  • Building with Docker
  • Building a single directory for dev purposes without all the pre-requisites
• Arrow Compute IR (Intermediate Representation)

next

C/GLib docs