duckdb connections support for R

[sounkou-bioinfo]

Hi,
Thank you for this great package.

I want to ask if there were plans to support duckdb connection for the R version of the package or how one could implement this

Thank you

[bruceyyu]

@sounkou-bioinfo I definitely support your proposal!
However, the workload to integrate duckdb will be large, so it is my next milestone for the package, which may take months.

Meanwhile, I recommend you to check out the sibling of GWalker, PyGWalker, if you use Python, as it already supports duckdb computation and it is much more developed.

Regarding how developers could implement customized features for the package, I will write some instructions in the docs soon. Stay tuned! ☺️

[sounkou-bioinfo]

@bruceyyu
Great, will be looking forward forward to the developers guide
thanks !

[bruceyyu]

@sounkou-bioinfo We just had a pre-release version for GWalkR with the computation powered by DuckDB! Check it out v0.2.0-alpha and welcome to give it a shot!

[sounkou-bioinfo]

@bruceyyu great !
i will test this right away and let you know !

[BorgeJorge]

Question: If I'm understanding, GWalkR's duckdb integration means GWalkR can now use duckdb for internal operations. I have some large duckdb tables that I'd like to visualize. Is there a way to read a duckdb table directly into GWalkR without having to first convert the table to a dataframe?

[bruceyyu]

Question: If I'm understanding, GWalkR's duckdb integration means GWalkR can now use duckdb for internal operations. I have some large duckdb tables that I'd like to visualize. Is there a way to read a duckdb table directly into GWalkR without having to first convert the table to a dataframe?

@BorgeJorge Currently no, but this is a good point and I've marked it on my todo list

[gregvolny]

Can you also give us access to PostgreSQL : https://github.com/electric-sql/pglite for the same reason @bruceyyu mentioned?

[bruceyyu]

Can you also give us access to PostgreSQL : https://github.com/electric-sql/pglite for the same reason @bruceyyu mentioned?

@gregvolny I'm not so sure about your use case. Do you have a local instance of PostgreSQL to connect?

[gregvolny]

I'm not so sure about your use case. Do you have a local instance of PostgreSQL to connect?

@bruceyyu , https://github.com/electric-sql/pglite create a persistent or in memory local instance of PostgreSQL in the browser virtual file system. This local instance can synchronize data to remote PG servers using https://pglite.dev/docs/sync. One of the most important features that can be implemented in Gwalker is a direct access to data source from remote relational database server.

[bruceyyu]

@gregvolny
R supports connecting to a remote database using odbc and I guess you want GWalkR to support visualization of a table stored in a remote database?
I will try to support this feature in the next version!

[BorgeJorge]

DuckDB's PostgreSQL extension allows DuckDB to read/write to/from a PostgreSQL database, and the httpfs extension allows for reading/writing to/from remote http and S3 files, so if you do end up allowing write access to GWalkR's internal DuckDB database, maybe there's a way to let people also write remote PostgreSQL, http, and S3 files into GWalkR/DuckDB? Just a thought.

[bruceyyu]

DuckDB's PostgreSQL extension allows DuckDB to read/write to/from a PostgreSQL database, and the httpfs extension allows for reading/writing to/from remote http and S3 files, so if you do end up allowing write access to GWalkR's internal DuckDB database, maybe there's a way to let people also write remote PostgreSQL, http, and S3 files into GWalkR/DuckDB? Just a thought.

Good point. Will look into it definitely