Skip to content

Commit 0ea0e79

Browse files
Add note on programmatically creating lambdas when lazily saving a PartionedDataset (#5158)
* Add note advising caution when using loops to create lambdas Signed-off-by: chrisschopp <[email protected]> * Add description of change to RELEASE.md Signed-off-by: chrisschopp <[email protected]> --------- Signed-off-by: chrisschopp <[email protected]> Co-authored-by: Ankita Katiyar <[email protected]>
1 parent 37358c3 commit 0ea0e79

File tree

2 files changed

+7
-0
lines changed

2 files changed

+7
-0
lines changed

RELEASE.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,13 @@
55

66
## Bug fixes and other changes
77

8+
## Documentation changes
9+
* Added a note on programmatically creating lambdas when lazily saving a `PartionedDataset`.
10+
811
## Community contributions
912
Many thanks to the following Kedroids for contributing PRs to this release:
1013
* [Aseem Sangalay](https://github.com/aseemsangalay)
14+
* [Chris Schopp](https://github.com/chrisschopp)
1115

1216
# Release 1.0.0
1317
## Major features and improvements

docs/catalog-data/partitioned_and_incremental_datasets.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,9 @@ new_partitioned_dataset:
252252
save_lazily: False
253253
```
254254

255+
!!! note
256+
If creating lambdas in a list/dictionary comprehension or a for loop, be cautious when referencing variables defined outside the scope of the lambdas. See the [Python Programming FAQ](https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result) for an explanation of how this can result in unexpected values being returned from the lambdas.
257+
255258
## Incremental datasets
256259

257260
[IncrementalDataset](https://docs.kedro.org/projects/kedro-datasets/en/feature-8.0/api/kedro_datasets/partitions.IncrementalDataset/) is a subclass of `PartitionedDataset`, which stores the information about the last processed partition in the so-called `checkpoint`. `IncrementalDataset` addresses the use case when partitions have to be processed incrementally, that is, each subsequent pipeline run should process just the partitions which were not processed by the previous runs.

0 commit comments

Comments
 (0)