ingestr

ingestr is a command-line app from Bruin Data Limited that ingests data from a source into a destination using simple command-line flags, with no code required. It supports incremental loading and ships as a single-command install, and it also offers a Python SDK for sending rows, generators, and DataFrames to a destination.

What It Is

ingestr is a data ingestion CLI. You give it a source URI, a source table, a destination URI, and a destination table, and it copies the data across. The docs say it removes the need to manage a backend or write ingestion code. It covers databases, message queues, files, and many SaaS and API platforms as sources, with a smaller set of destinations such as BigQuery, Snowflake, Postgres, DuckDB, and Databricks.

How It Works

A typical run pulls a table such as public.some_data from Postgres and loads it into a BigQuery schema and table. Incremental loading supports append, merge, or delete+insert strategies. The documentation also covers change data capture (CDC) for sources like Postgres, MySQL, SQL Server, MongoDB, PlanetScale, and Vitess, along with reverse ETL and data masking. The documentation includes tutorials such as loading Stripe data to Postgres and CDC from Postgres to DuckDB.

Python SDK and Bruin Integration

Installing the pip package with the sdk extra lets Python code call ingestr.ingest with lists of rows, DataFrames, or generators, or use it as a context manager for push-style ingestion. Data is sent to the ingestr binary as Arrow IPC streams. Within the Bruin platform, ingestr is used as an asset type in pipeline definitions.

License

The repository license is the Functional Source License 1.1, which allows internal production use, development, testing, education, and research but prohibits offering a competing commercial ingestion product. Each version converts to Apache 2.0 two years after release.