Optimize evaluation #173
Optimize evaluation #173
Conversation
Co-authored-by: rhoadesScholar <[email protected]>
Co-authored-by: rhoadesScholar <[email protected]>
Co-authored-by: rhoadesScholar <[email protected]>
…x function fix(tests): Import os module for environment variable access in test file
There was a problem hiding this comment.
Pull Request Overview
This PR optimizes the evaluation code for instance segmentation scoring by introducing a new vectorized iou_matrix function that replaces the previous nested loop approach for computing IoU (Intersection over Union) between predicted and ground truth instances.
Key changes:
- Introduces a new
iou_matrixfunction using NumPy'sbincountfor efficient IoU computation - Replaces
skimage.measure.labelwithfastremap.renumberfor faster instance relabeling - Adds parallel query support to KDTree operations in Hausdorff distance computation
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
tests/test_iou_matrix.py |
Comprehensive test suite for the new iou_matrix function covering edge cases (empty arrays, non-square matrices, ratio cutoffs) and validation against naive reference implementation |
src/cellmap_segmentation_challenge/evaluate.py |
Implements optimized iou_matrix function, refactors score_instance to use vectorized IoU computation, adds parallel workers to KDTree queries, and simplifies instance matching with fastremap.remap |
pyproject.toml |
Adds fastremap dependency for optimized array operations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Assumes IDs are sequential starting at 1 (0 is background). | ||
| Returns float32 array of shape (num_gt_ids, num_pred_ids). |
There was a problem hiding this comment.
The documentation states "Assumes IDs are sequential starting at 1" but the implementation actually uses g.max() to determine the number of instances. This means the function expects IDs from 1 to max_id without gaps. If there are non-sequential IDs (e.g., 1, 2, 5), the implementation will create a matrix with empty rows/columns. Consider clarifying the documentation to state "Assumes IDs range from 1 to max(ID) with 0 as background" or handle non-sequential IDs explicitly.
| Assumes IDs are sequential starting at 1 (0 is background). | |
| Returns float32 array of shape (num_gt_ids, num_pred_ids). | |
| Assumes IDs range from 1 to max(ID) (0 is background). If IDs are non-sequential (e.g., 1, 2, 5), the output matrix will contain empty rows/columns for missing IDs. | |
| Returns float32 array of shape (max(gt) + 1, max(pred) + 1), where rows/columns for missing IDs will be empty. |
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
…_matrix test fix(fetch_data): Ensure ThreadPoolExecutor shuts down after fetching data
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
…s-coverage Add unit tests for utils, security, crops, and loss modules
- Added "connected-components-3d" to pyproject.toml dependencies. - Refactored evaluate.py to use cc3d for connected components. - Introduced comprehensive tests for IoU matrix and Hausdorff distance calculations in test_evaluate_metrics.py. - Cleaned up unused imports and optimized existing functions.
- Update package_submission function to log successful crop packaging and raise an error if no crops are packaged. - Refactor test dependencies and streamline test cases in test_all.py, test_crops.py, test_loss.py, and test_security.py. - Modify train_config.py to adjust input and target array shapes and reduce iterations per epoch.
…le calls in tests
No description provided.