fix: streamline annotation layer with lazy streaming #276
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aksg87
force-pushed
the
fix/annotation-generator-align
branch
2 times, most recently
from
54ebb97 to
aefe83a
Compare
Verify annotate_documents uses 'yield from' to properly delegate to generators, ensuring correct document attribution across batches. Co-authored-by: Vayoa <[email protected]>
Stream documents lazily and emit incrementally to reduce memory from O(documents) to O(batch_size). Improve code clarity with better naming (keep_last_doc, _emit_docs_iter) and removed verbose comments.
The test was not passing through the real batches, which prevented the lazy document capture from running. Updated mock to use side_effect to pass through the iterable while still allowing inspection of call args.
aksg87
force-pushed
the
fix/annotation-generator-align
branch
from
aefe83a to
f738897
Compare
This was referenced Nov 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes generator delegation bug causing incorrect document attribution. Changed
yieldtoyield fromand streamlined code for clarity. This issue was also comprehensively reproduced and documented in #260 by @vayoa.Changes
yield fromto properly delegate to generator helperskeep_last_doc,_emit_docs_iter), remove verbose commentsInvalidDocumentErrorandInternalErrorexception classesTesting
Fixes #260 #269