Skip to content

Add support for postgres COPY TO/FROM command #160

@DeeeeLAN

Description

@DeeeeLAN

It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy

Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.

For context, I just ran a comparison between the two operations.

Database row retrieval count: 236,170,980
to_dataframe() time: 774 seconds
COPY TO to_csv() time: 223 seconds
converting csv to dataframe time: 24 seconds

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions