Thanks @edschofield for the report

Simpler reproducer (perhaps):

df = pl.DataFrame({'a': ['x','x','x'], 'b': [1,1,1], 'c': [7,8,9], 'd': [0,2,1]})
df.pivot(on='a',index='b',values='c', aggregate_function=pl.element().sort_by('d'))

Outputs:

shape: (1, 2)
┌─────┬───────────┐
│ b   ┆ x         │
│ --- ┆ ---       │
│ i64 ┆ list[i64] │
╞═════╪═══════════╡
│ 1   ┆ [7, 8, 9] │
└─────┴───────────┘

However, sorting column 'c' by column 'd' should output [7, 9, 8]

I don't mind giving this a closer look if there are no other takers, but may need some help (new to the code base).

Update.

(a) Functions like sort_by and alike reference another column ("d" in the example). This column gets reset to "" in https://github.com/pola-rs/polars/blob/main/crates/polars-lazy/src/physical_plan/exotic.rs#L7, which is called from pivot_stable() in https://github.com/pola-rs/polars/blob/main/crates/polars-lazy/src/frame/pivot.rs#L84-L88. This explains why these arguments get ignored, even when a non-existent column name is provided.

(b) In addition, the tmp_df dataframe passed to evaluate only contains the values column. Therefore, it is unable to evaluate the expression in the case the column names would have been preserved. A quick modification to the code illustrates this: we now get (polars.exceptions.ColumnNotFoundError: "d" not found, from evaluate() in impl PhysicalExpr for ColumnExpr in column.rs. To be confirmed.

I will see if I can make it work, no promises.

@MarcoGorelli if you have any insights or feedback, that would be appreciated. We can move to a draft PR if you prefer.

Small update - with the proposed approach (not production grade code), I get the expected result, at least for the sort_by MRE:

shape: (1, 2)
┌─────┬───────────┐
│ b   ┆ x         │
│ --- ┆ ---       │
│ i64 ┆ list[i64] │
╞═════╪═══════════╡
│ 1   ┆ [7, 9, 8] │
└─────┴───────────┘

Next steps: investigate top_k_by, proper code, PR.

Well done and thanks for your work on this, @kdn36! This looks promising...

Update.

In addition to (a) and (b), we have:
(c) The functions sortby() and top_k_by() are implemented differently: the first one is an explicit AExpr::SortBy and the second one is a more generic AExpr::Function variant. As part of evaluating expr inside PivotExpr, the lazyframe gets optimized with a dummy single-column dataframe based on the values dtype. The more generic Function variant wants to access all referenced columns of the original dataframe as part of this process (and fails because the dummy dataframe does not have this), while the explicit SortBy variant has what it needs without those lookups.

With a modified version, case 1 now succeeds:

baseline (group_by):
shape: (2, 3)
┌──────┬─────────────┬───────────────────────┐
│ year ┆ EU          ┆ JP                    │
│ ---  ┆ ---         ┆ ---                   │
│ i64  ┆ str         ┆ str                   │
╞══════╪═════════════╪═══════════════════════╡
│ 1970 ┆ peugeot 504 ┆ toyota corona mark ii │
│ 1971 ┆ opel 1900   ┆ toyota corona         │
└──────┴─────────────┴───────────────────────┘

case1:
shape: (2, 3)
┌──────┬─────────────┬───────────────────────┐
│ year ┆ EU          ┆ JP                    │
│ ---  ┆ ---         ┆ ---                   │
│ i64  ┆ str         ┆ str                   │
╞══════╪═════════════╪═══════════════════════╡
│ 1970 ┆ peugeot 504 ┆ toyota corona mark ii │
│ 1971 ┆ opel 1900   ┆ toyota corona         │
└──────┴─────────────┴───────────────────────┘

Case 2 still panics because of the different syntax (it explicitly references the column in pl.col('name')), that is fixable.
Case 3 panics (better than ignoring), so that needs to be converted to ColumnError, doable.

Two approaches to fix this:

1. Carry the full dataframe (schema) throughout the planning cycle.
2. Refine the logic in to_field_impl to only consider the input column when determining the result field of the Function (current modification).

I think 1) is the right thing to do, but I will need core team input. Will shift this discussion to an upcoming PR.