Merge snapshots into 1 under transaction of multiple operations

[jim-ngoo]

Question

For my use case, I have a daily cron job that batch process and append data but I only want a single snapshot record after the whole process. I tried to do them under a single transaction but still there are multiple snapshots created. Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

[ForeverAngry]

Question

For my use case, I have a daily cron job that batch process and append data but I only want a single snapshot record after the whole process. I tried to do them under a single transaction but still there are multiple snapshots created. Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

Can you share what function you are using to do the append?

[jim-ngoo]

Question

For my use case, I have a daily cron job that batch process and append data but I only want a single snapshot record after the whole process. I tried to do them under a single transaction but still there are multiple snapshots created. Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

Can you share what function you are using to do the append?

So here's a sample code:

from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
import pyarrow as pa

if __name__ == "__main__":
    catalog = InMemoryCatalog(name="test_catalog")
    catalog.create_namespace_if_not_exists("test_ns")
    table = catalog.create_table_if_not_exists(
        "test_ns.test_table",
        schema=Schema(NestedField(field_id=1, name="a", field_type=LongType())),
    )
    with table.transaction() as transaction:
        transaction.append(pa.Table.from_pylist([{"a": 1}]))
        transaction.append(pa.Table.from_pylist([{"a": 2}]))

    for entry in table.metadata.snapshot_log:
        print(entry)

And two snapshots are created - is there a way I can do batches of append, but there will be only one snapshot created at the end?

[ForeverAngry]

Hi! There is! However, some of the approaches come to mind might require some changes to how you are looking to add data. So i can better help, do you plan only only ever writing from a dataframe to your iceberg table?

[jim-ngoo]

Hi! There is! However, some of the approaches come to mind might require some changes to how you are looking to add data. So i can better help, do you plan only only ever writing from a dataframe to your iceberg table?

yes for my use case we manipulate data using polars dataframe/ lazyframe, and then append into the iceberg table.

[jayceslesar]

#2152 related

[ForeverAngry]

@jim-ngoo an alternative could be to product the parquet files seprate from the commit, and then use the add_files function to commit them as a single snapshot.

[kevinjqliu]

Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

Im surprised that multiple snapshots are produced. I would expect only 1 snapshot per transaction.

[Fokko]

Producing multiple snapshots in a single commit is very common for PyIceberg. For example, the upsert operation can produce up to four snapshots. Technically, you can squash them all in an OVERWRITE snapshot, but then you lose a lot of information about what happened along the way. This information can be used for other clients to determine if a snapshot is conflicting with what they are trying to do.

[Fokko]

To add to the above, the upsert produces both DELETE and APPEND operations, we could of course merge consecutive APPEND operations. The question here is, should the user just call transaction.append just once, or should PyIceberg merge it under the cover? I would be leaning towards the first, since that would also produce a single Parquet file, instead of two. I don't think we should rewrite Parquet files on an append call.

[jayceslesar]

I wonder if there should just be better documentation on what snapshots are truly used for? From personal experience, they have only ever been useful to me as a user when I need to disinfect/debug data that was written to a table, and I am trusting that major iceberg implementations are reading and creating them correctly.

Additionally, when googling "how are snapshots used iceberg", there is a nice AI response (for better or for worse) but none of the linked documentation from the official iceberg website explicitly states how snapshots are used, or rather does not explain it as well as you just did in your first comment