Skip to content

Conversation

@rluvaton
Copy link
Member

@rluvaton rluvaton commented Jun 8, 2025

Which issue does this PR close?

N/A

Rationale for this change

Actually use ignore_nulls
that was used in:

What changes are included in this PR?

forward ignore_nulls in first and last and no longer mark as unsupported

How are these changes tested?

this have the same problem as #1626, all the tests are disabled for first and last

@rluvaton rluvaton force-pushed the pass-ignore-null-to-first-and-last branch from 2b122ed to 692332f Compare June 8, 2025 15:15
@parthchandra
Copy link
Contributor

The linked PR did not add a test. Would you be able to?

@codecov-commenter
Copy link

codecov-commenter commented Jun 9, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.00%. Comparing base (f09f8af) to head (820428d).
Report is 272 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1866      +/-   ##
============================================
+ Coverage     56.12%   59.00%   +2.88%     
- Complexity      976     1141     +165     
============================================
  Files           119      130      +11     
  Lines         11743    12827    +1084     
  Branches       2251     2412     +161     
============================================
+ Hits           6591     7569     +978     
- Misses         4012     4035      +23     
- Partials       1140     1223      +83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rluvaton
Copy link
Member Author

rluvaton commented Jun 11, 2025

All the tests for first and last are disabled currently:

@andygrove
Copy link
Member

All the tests for first and last are disabled currently:

* [Re-enable tests for FIRST/LAST #1646](https://github.com/apache/datafusion-comet/issues/1646)

Yes, we should figure out a test approach for these non-deterministic functions before we start making changes to the implementation,

@parthchandra
Copy link
Contributor

@andygrove perhaps we can merge this while we wait for the tests to be made more accurate?

@andygrove
Copy link
Member

I will create a PR today to add correctness tests for first/last. We can then rebase this PR and make sure that there are no regressions.

@andygrove
Copy link
Member

@rluvaton Here is a test that fails in main and passes with your changes in this PR. Could you add this to CometAggregateSuite as part of this PR?

  test("first/last with ignore null") {
    val data = Range(0, 8192).flatMap(n => Seq((n, 1), (n, 2))).toDF("a", "b")
    withTempDir { dir =>
      val filename = s"${dir.getAbsolutePath}/first_last_ignore_null.parquet"
      data.write.parquet(filename)
      withSQLConf(CometConf.COMET_BATCH_SIZE.key -> "100") {
        spark.read.parquet(filename).createOrReplaceTempView("t1")
        for (expr <- Seq("first", "last")) {
          // deterministic query that should return one non-null value per group
          val df = spark.sql(s"SELECT a, $expr(IF(b==1,null,b)) IGNORE NULLS FROM t1 GROUP BY a ORDER BY a")
          checkSparkAnswerAndOperator(df)
        }
      }
    }
  }

@rluvaton
Copy link
Member Author

rluvaton commented Jun 19, 2025

sorry, missed your comment, added

Copy link
Contributor

@parthchandra parthchandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rluvaton

@andygrove andygrove merged commit 542f45b into apache:main Jun 23, 2025
93 of 94 checks passed
@rluvaton rluvaton deleted the pass-ignore-null-to-first-and-last branch June 23, 2025 18:41
coderfender pushed a commit to coderfender/datafusion-comet that referenced this pull request Dec 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants