[Feature Request] Supports StageOnly metadataUpdate

[stevie9868]

Feature Request / Improvement

Currently in java, there is a way to set stageOnly for snapshotUpdate so that only the snapshots are created but the refs are not updated.

[stevie9868]

We have some internal used case for this feature and If people think this is a useful feature, I am willing to contribute.

[dingo4dev]

@stevie9868
Would you share more on this use case? I think it will be useful and we can add some examples and docs on this feature.

[kevinjqliu]

ah this is used for WAP
https://github.com/apache/iceberg/blob/c3d50e177ce17a45f4dd3af7593350dd6724ca8f/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java#L221-L226

And its different from writing to branches...

Is the expected operations here to commit to the main branch but not fast forward the reference?

[kevinjqliu]

Would be interesting to get WAP feature into pyiceberg

[stevie9868]

Is the expected operations here to commit to the main branch but not fast forward the reference?

Yeah that's correct, only trigger an AddSnapshotUpdate in pyiceberg but don't trigger a SetSnapshotRefUpdate when it's stageOnly defined here

Would be interesting to get WAP feature into pyiceberg

I Totally agree. Here are my thoughts to support this:

For wap.id

1. Implement stageOnly in snapshotProducer
2. Integrate stageOnly into the commit path
3. implement wap.id

For wap.branch:
It should be relatively easy, as I believe py-iceberg already support writing to branches.

Also looks like we haven't implemented cherrypick and fastforward operation in py-iceberg? Would be nice to add them as well.