Lance

Lance is an open-source lakehouse format for multimodal AI, developed under the lance-format organization and released under Apache-2.0. It combines a file format, table format, and catalog spec so teams can build a lakehouse on object storage for vector search, full-text search, feature engineering, and model training. The project describes a VLDB 2025 paper behind the format, and the latest listed release is v13.0.0.

What It Is

Lance is a columnar data format and table layer aimed at AI workloads. Data is written to Lance datasets (for example converted from Parquet via PyArrow), then read back through Python, Pandas, DuckDB and other engines. The core is implemented in Rust, with Python (PyO3) and Java (JNI) bindings.

Search and Random Access

Vector similarity search, BM25 full-text search, and SQL-style filters can run on the same dataset, accelerated by secondary indexes defined in the specification. The README shows building an IVF_PQ vector index and querying nearest neighbors. The project states that random access is 100x faster than Parquet or Iceberg, a vendor-published claim.

Multimodal Data and Data Evolution

Images, video, audio, text, and embeddings can be stored alongside tabular data, with blob encoding and lazy loading for large binaries. Columns can be added with backfilled values by writing new files rather than rewriting the table, which suits ML feature engineering. Versioning includes ACID transactions, time travel, tags, and branches.

Ecosystem and Compatibility

Lance integrates with Apache Arrow, Pandas, Polars, DuckDB, Spark, Ray, PyTorch, Trino, and Flink, and with catalogs such as Apache Polaris, Unity Catalog, Apache Gravitino, and Hive Metastore. The docs state that stable data_storage_version values are a long-term compatibility contract, while the next alias is unstable and only for experimentation.