-
Notifications
You must be signed in to change notification settings - Fork 118
Open
Description
It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy
Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.
For context, I just ran a comparison between the two operations.
Database row retrieval count: 236,170,980
to_dataframe()
time: 774 seconds
COPY TO to_csv()
time: 223 seconds
converting csv to dataframe time: 24 seconds
Metadata
Metadata
Assignees
Labels
No labels