Skip to content

Conversation

@Chowdhury-Anik
Copy link

@Chowdhury-Anik Chowdhury-Anik commented Nov 8, 2025

This commit adds four data sink methods for Polars LazyFrame:

  • sink_parquet: Write LazyFrame to Parquet format
  • sink_csv: Write LazyFrame to CSV format
  • sink_ipc: Write LazyFrame to IPC/Feather format
  • sink_ndjson: Write LazyFrame to NDJSON format

These sinks allow users to write LazyFrames directly without needing to call .collect() first, improving performance for large datasets.

Fixes #791

Changes

How I tested this

Notes

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

This commit adds four data sink methods for Polars LazyFrame:
- sink_parquet: Write LazyFrame to Parquet format
- sink_csv: Write LazyFrame to CSV format
- sink_ipc: Write LazyFrame to IPC/Feather format
- sink_ndjson: Write LazyFrame to NDJSON format

These sinks allow users to write LazyFrames directly without needing to call .collect() first, improving performance for large datasets.

Fixes apache#791
Copy link
Contributor

@elijahbenizzy elijahbenizzy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good -- maybe add a test?

Copy link
Contributor

@skrawcz skrawcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just need some tests please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add data source sinks for Polars Lazyframe implementation

3 participants