[WIP] Remove COMET_SHUFFLE_FALLBACK_TO_COLUMNAR config

[andygrove]

Which issue does this PR close?

Part of #1254

Rationale for this change

The config COMET_SHUFFLE_FALLBACK_TO_COLUMNAR was added as a workaround for Spark SQL test failures and we need to remove it since users could run into the same bugs that is causing the Spark SQL failures.

What changes are included in this PR?

• Remove config
• Fix Spark SQL test failures (tracked in [EPIC] Spark SQL test failures when Comet JVM shuffle is used #1254)

How are these changes tested?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.42%. Comparing base (f09f8af) to head (dab5fbf).
Report is 240 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1736      +/-   ##
============================================
+ Coverage     56.12%   59.42%   +3.29%     
- Complexity      976     1151     +175     
============================================
  Files           119      130      +11     
  Lines         11743    12652     +909     
  Branches       2251     2370     +119     
============================================
+ Hits           6591     7518     +927     
+ Misses         4012     3928      -84     
- Partials       1140     1206      +66     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

There are now many more test failures than when this PR was first created.

[coderfender]

Working on this

[coderfender]

Working on this