feat: [iceberg] CometExecRDD supports per-partition plan data, Iceberg native scan with DPP

[mbutrovich]

Continues efforts from #3295, #3297, and #3301, building on #3295's diff. I asked Claude to summarize the diff and draft the PR description for me:

Which issue does this PR close?

Closes #.

Rationale for this change

This PR continues the work from #3295, addressing two problems with Iceberg native scan:

1. Serialization overhead: All Iceberg FileScanTask data was serialized into the protobuf plan at planning time and distributed to every executor. For tables with many files, this creates significant overhead since every task receives the full plan containing all partitions' tasks.

2. No DPP support: Dynamic Partition Pruning couldn't work because partition data was serialized at planning time, before DPP subqueries execute. Now that we'e deferring some parts of plan generation, that opens the door for DPP.

What changes are included in this PR?

Partition data distribution (solves plan bloat problem)

• Planning now creates a placeholder IcebergScan with only metadata_location (for matching)
• serializePartitions() runs at execution time after DPP resolves, producing:
• commonData: shared across partitions (catalog properties, schema, pools)
• perPartitionData: array of serialized FileScanTask data, one per partition
• CometExecPartition carries only its partition's data; PlanDataInjector injects it into the operator tree at execution time
• Cache the commonData parsing to eliminate repeated allocations for PartitionData and PartitionValue

Unified RDD (replaces ZippedPartitionsRDD)

• ZippedPartitionsRDD is removed; CometExecRDD now handles all cases:
• With inputs: zips them via inputPartitions (equivalent to ZippedPartitionsBaseRDD)
• Without inputs: uses numPartitions to create partitions (standalone scan case)
• This is equivalent because:
• getPartitions creates CometExecPartition with inputPartitions array from each input RDD at that index
• compute iterates each input RDD with its corresponding partition (same as ZippedPartitionsRDD.compute)
• getDependencies returns OneToOneDependency for each input (same dependency semantics)
• getPreferredLocations uses intersection-then-union logic (duplicated from ZippedPartitionsBaseRDD)

Dynamic Partition Pruning support

• CometIcebergNativeScanExec.doPrepare() triggers DPP subquery preparation
• serializedPartitionData is lazy: waits for DPP values, then serializes only filtered partitions
• Handles SubqueryAdaptiveBroadcastExec via reflection to set InSubqueryExec.result
• Added Spark version shims for SubqueryAdaptiveBroadcastExec.index vs .indices (SPARK-46946)

CometNativeExec iterator creation

• Moved from inline closure to CometExecRDD.compute()
• findAllPlanData traverses the plan tree to collect planning data, stopping at stage boundaries
• collectSubqueries gathers ScalarSubquery expressions for registration with CometScalarSubquery
• Both are equivalent to the previous behavior but structured for the new RDD

How are these changes tested?

• Extended CometIcebergNativeSuite with DPP tests:
• runtime filtering - join with dynamic partition pruning: verifies DPP prunes partitions (3 → 1)
• runtime filtering - multiple DPP filters on two partition columns: tests multi-column DPP
• CI runs full test suite including Iceberg Java TestRuntimeFiltering tests

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.06796% with 78 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.17%. Comparing base (f09f8af) to head (9cc541a).
⚠️ Report is 918 commits behind head on main.

Files with missing lines	Patch %	Lines
...e/spark/sql/comet/CometIcebergNativeScanExec.scala	61.53%	10 Missing and 20 partials ⚠️
...n/scala/org/apache/comet/rules/CometScanRule.scala	21.73%	11 Missing and 7 partials ⚠️
.../comet/serde/operator/CometIcebergNativeScan.scala	89.79%	13 Missing and 2 partials ⚠️
...n/scala/org/apache/spark/sql/comet/operators.scala	87.67%	2 Missing and 7 partials ⚠️
...cala/org/apache/spark/sql/comet/CometExecRDD.scala	93.33%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3349      +/-   ##
============================================
+ Coverage     56.12%   60.17%   +4.05%     
- Complexity      976     1491     +515     
============================================
  Files           119      174      +55     
  Lines         11743    16348    +4605     
  Branches       2251     2713     +462     
============================================
+ Hits           6591     9838    +3247     
- Misses         4012     5129    +1117     
- Partials       1140     1381     +241     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

we should probably have a follow up issue to revisit the choice to ignore this warning

[mbutrovich]

Even though CI is green, gonna mark this as draft since I might still keep refactoring a bit to clean things up and don't want it merged early.