DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam

[oznur-synnada]

Hey all,

Thanks to the support of @ohommos, his company Adyen volunteered to be the venue of our next meetup in Amsterdam. I've opened this discussion to initiate the event organization. Please feel free to share ideas - we'd love to see who's willing to participate or take the stage (event details below).

@alamb @ozankabak @v-kessler any comments or suggestions?

• Event's name: Apache DataFusion Meet-up Amsterdam
• Event's date: May 27th 2025, from 17:30 to 21:00
• Event's location: Rokin Office - Basement event space (Max 100)
• Address: Amsterdam. Rokin 49; 1012 KK, The Netherlands

[alamb]

@Dandandan is local, perhaps he knows of some locals who might be interested

[v-kessler]

Currently, the iceberg-rust implementation is still under active development and not yet ready for use with DataFusion. I expect it may take a couple more months before it’s in a usable state. For now, the only available option is the implementation from Jan Kaul.

That said, I’d be happy to attend and present Apache Iceberg and our Lakekeeper catalog to help raise awareness and motivate the DataFusion community to engage with the Iceberg integration.

What do you think about this approach?

[alamb]

I think those topics would be great personally

BTW In my mind the dream integration scenario is something like @roeap is actively working in https://github.com/delta-incubator/deltalake-datafusion/blob/0adfc493b351da6e937e16d0c68a5b20266a8db0/crates/datafusion/examples/simple_read.rs#L11-L33

(that is for delta, but he has some plans for iceberg too)

#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let ctx = SessionContext::new().enable_delta_kernel(None);

    let base = Url::from_directory_path(std::fs::canonicalize(PATH).unwrap()).unwrap();
    let path = format!("{}column_mapping/delta", base.to_string());

    // build a table and get the latest snapshot from it
    let table = Table::try_from_uri(path)?;
    ctx.register_delta("delta_table", table.location()).await?;

    let df = ctx.sql("SELECT * FROM delta_table").await?;
    let df = df.collect().await?;
    print_batches(&df)?;

    let log_provider = DeltaLogTableProvider::try_new(table.into())?;
    ctx.register_table("delta_log", Arc::new(log_provider))?;

    let df = ctx
        .sql("SELECT add['path'] as path FROM delta_log WHERE add['path'] IS NOT NULL")
        .await?;
    let df = df.collect().await?;
    print_batches(&df)?;

    Ok(())
}

[roeap]

If folks are interested I am happy to see if I can come to Amsterdam and talk about it a bit more :). Currently focussing on proving some end to end integrations and get a sense for performance. I did however start to look at the best integration points in iceberg-rust to extract the information needed and think there is an avenue to at least have a PoC.

The general idea is to make it work and have the minimal API shown above - then work with the community to make things a bit smoother in the backend ..