pyrosome

pyrosome stores polars data frames on disk as versioned, append-only datasets, and reads them back as lazy frames.

>>> from pathlib import Path
>>> import polars as pl
>>> from pyrosome import ParquetDal
>>> dal = ParquetDal(Path("data"))
>>> weather = pl.DataFrame({"city": ["Oslo", "Lima"], "temp": [4, 19]})
>>> dal.write(weather, "weather").as_posix()
'data/weather/v0'
>>> dal.append(pl.DataFrame({"city": ["Pune"], "temp": [31]}), "weather").as_posix()
'data/weather/v0/1.parquet'
>>> dal.read("weather").filter(pl.col("temp") > 10).collect()["city"].to_list()
['Lima', 'Pune']

• Versioned: every write creates a new version. Old versions stay on disk until you remove them.

• Append-only: append adds new files and never rewrites existing ones.

• Safe with several processes: concurrent writes and appends never overwrite each other, and readers never see half-written data.

• Lazy: read returns a polars.LazyFrame, so filters and column selections are pushed down into the file scan.

Every example in these docs is run as a test, so they stay in step with the code.

• Quickstart
  • Install
  • Create a data access layer
  • Write data
  • Read data
  • Append data
  • Write a new version
  • Use CSV instead of parquet
• Concepts
  • On-disk layout
  • Keys
  • Versions
  • Safety with several processes
  • Limitations
• Recipes
  • Keep a snapshot of a full refresh
  • Ingest data in batches
  • Query without loading everything
  • Filter by column values
  • Group profiles into a dataset
  • Read data that may not exist yet
  • Combine several profiles
  • Compact many small appends
  • Change the schema
  • Read an older version
  • Roll back to an older version
  • Remove old versions
  • Write from several processes
  • Exchange data as CSV
  • Recover from a crashed writer
• API reference
  • Data access
  • Filtering reads
  • Datasets
  • Internals