Version 0.7.0

Release Notes - v0.7.0

Release Date: December 1, 2025

🎉 What's New

Delta Lake Instrumentation 🚀

This release introduces comprehensive Delta Lake monitoring and instrumentation capabilities:

• Delta Lake Table Monitoring: New spark.dataflint.instrument.deltalake configuration flag to enable Delta Lake-specific instrumentation
• Delta Lake Scan Page: New dedicated UI page showing Delta Lake scan operations and metrics
• Full Table Scan Detection: Automatic alerts for full table scans on Delta Lake tables to help identify performance issues
• Z-Order Cache Tracking: Monitor Z-Order optimization cache usage in table properties
• Delta Log Integration: Direct integration with Delta Lake's cached snapshots for improved performance monitoring

Enhanced UI & User Experience 📊

Alerts Tab Improvements

• Grouped Alerts: Alerts are now organized by alert type for better visibility and navigation
• Search Functionality: New description search bar to quickly find specific alerts
• Spill Selector: New UI component to identify and navigate to operations with data spills
• Duration/Alert Navigation: Improved button logic for advancing through alerts by duration or index

SQL Flow Enhancements

• Subquery Differentiation: Better visual differentiation for subqueries in the SQL execution plan
• Union Support: Improved stage identification algorithm for UNION operations and missing nodes with same-stage neighbors
• SQL Text Display: Enhanced SQL text rendering and display

JDBC Support

• JDBC Scan Detection: Better support for JDBC scan operations with dedicated parsing and visualization
• JDBC Examples: New comprehensive JDBC example demonstrating monitoring capabilities

Telemetry & Analytics 📈

• Scarf Pixel Integration: Optional telemetry to help monitor OSS usage patterns
  • Can be disabled with spark.dataflint.telemetry.enabled=false flag

Technical Improvements 🔧

Core Enhancements

• Delta Lake Reflection Utils: New utility classes for Delta Lake introspection and monitoring
• Delta Table Path Parser: Robust parsing of Delta Lake table paths and identifiers
• Improved Metrics Processing: Enhanced metric processors for better performance data collection

Bug Fixes

• Fixed bytesToHumanReadableSize utility to handle comma-separated values correctly
• Fixed read parser unit tests
• Improved central snapshot deployment configuration
• Better handling of missing nodes in stage identification

Build & CI/CD

• Updated CI/CD workflows for improved reliability
• Enhanced build configuration for Spark 3.x and 4.x compatibility
• Improved artifact publishing process

📝 Full Changelog

Full Changelog: v0.6.1...v0.7.0

Features

• Delta Lake instrumentation and monitoring (#multiple commits)
• Alert tab grouping by alert type (bce9c5f)
• Spill selector component (0051e5c)
• Description search bar (e9f1a36)
• Scarf pixel telemetry integration (71d8287)
• SQL text improvements (#39)
• Subquery UI differentiation (4ec35fb)
• JDBC scan support (3e2060f)
• Full scan table alerts for Delta Lake (b8709cc)
• Z-Order cache tracking (8718974)

Bug Fixes & Improvements

• Improved stage identification algorithm for unions (d9e2902)
• Fixed duration/alert navigation logic (f7b8c16)
• Fixed read parser unit tests (b49efb7)
• Fixed bytesToHumanReadableSize comma handling (c7e3695)
• Improved Delta Lake listener implementation (def2b01)
• Refactored listener architecture (3edef71)
• Enhanced Delta Log integration (120a910, 1bc376c)
• Added more supported SQL plan nodes (6e5b64e)

CI/CD & Build

• CI improvements (#38)
• Fixed central snapshot deployment (5e9a966)
• Updated README and documentation (18810dc, #36)

🙏 Contributors

Special thanks to:

• @menishmueli - Core development and features
• @cxzl25 - SQL text improvements and CI enhancements
• @daniel Aronovich - Documentation updates

📚 Documentation

For detailed usage instructions, see the README.

For Delta Lake instrumentation setup:

spark.conf.set("spark.dataflint.instrument.deltalake", "true")

To disable telemetry:

spark.conf.set("spark.dataflint.telemetry.enabled", "false")

New Contributors

• @cxzl25 made their first contribution in #38