[Bug] Regression in Redshift Spectrum Querying Iceberg Table #6502
Description
Search before asking
- I searched in the issues and found nothing similar.
Paimon version
master branch d0b66f272ca5cdfb4f8f588b37b2461f1c34e06c
Compute Engine
Flink
Minimal reproduce step
When using Paimon tables with Iceberg compatibility enabled, querying via AWS Redshift Spectrum fails with the following error:
[2025-10-30 15:05:17] [XX000] ERROR:
[2025-10-30 15:05:17] -----------------------------------------------
[2025-10-30 15:05:17] error: Error parsing table metadata.
[2025-10-30 15:05:17] code: 15003
[2025-10-30 15:05:17] context: Required field total-records missing.
[2025-10-30 15:05:17] query: 0
[2025-10-30 15:05:17] location: json_utils.hpp:56
[2025-10-30 15:05:17] process: padbmaster [pid=1073857039]
[2025-10-30 15:05:17] -----------------------------------------------
What doesn't meet your expectations?
I checked a table created last year and hasn't been updated since (don't remember the version used to create it). That table had the summary stats needed by Spectrum. It looks like one of the refactorings in the past year got rid of it.
Our use case requires querying Iceberg tables from Redshift warehouse.
cat old_paimon_iceberg_table_from_last_year_metadata.json | jq '.snapshots[0].summary'
{
"operation": "append",
"flink.operator-id": "f6dc7f4d2283f4605b127b9364e21148",
"flink.job-id": "c36bef679b5b9f6d8cb121f4a76c430c",
"flink.max-committed-checkpoint-id": "1",
"added-data-files": "18",
"added-records": "2741136",
"added-files-size": "203739326",
"changed-partition-count": "4",
"total-records": "2741136",
"total-files-size": "203739326",
"total-data-files": "18",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0"
}
cat new_paimon_iceberg_table_metadata.json | jq '.snapshots[0].summary'
{
"operation": "append"
}
While the Iceberg specification lists these fields as "optional," certain query engines like Redshift Spectrum treat some of them (particularly total-records) as required for proper metadata parsing.
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!