feat: collect once during display() in jupyter notebooks

[timsaucer]

Which issue does this PR close?

None

Rationale for this change

By design in a Jupyter notebook display() calls both __repr__ and _repr_html_. This currently causes collect() on DataFrames to occur twice, which can lead to double the execution time during evaluation. This PR causes collect to only happen once.

What changes are included in this PR?

If we are in a jupyter notebook, we will cache the result of a __repr__ or _repr_html_ call. When the other call happens, it will consume the cached calls. This means that for display() in a jupyter notebook the collected data will be freed.

Are there any user-facing changes?

None.

[alamb]

Makes sense to me

[kylebarron]

I don't think this is a reasonable workaround because there are many Jupyter-protocol frontends that do not support displaying HTML output. This means that repr would be broken for the IPython console, for example

[kylebarron]

By design in a Jupyter notebook display() calls both __repr__ and _repr_html_.

Ref https://discourse.jupyter.org/t/find-out-if-my-code-runs-inside-a-notebook-or-jupyter-lab/6935/8

You might be able to look in the IPython config to see what's running, but this answer is 10+ years old and might've changed https://stackoverflow.com/a/24937408

[timsaucer]

I don't think this is a reasonable workaround because there are many Jupyter-protocol frontends that do not support displaying HTML output. This means that repr would be broken for the IPython console, for example

Thanks for the feedback! I changed to check for the environment as you suggested and tested in jupyter, ipython console, and regular python console.

[kylebarron]

As mentioned in a comment on SO, that fails in jupyter console, and I verified that still fails:

[kylebarron]

I think that looks a lot more stable, even though it's kinda annoying