Added data quality alerts #995

ghanse · 2026-01-13T16:47:10Z

Changes

This PR introduces data quality alerts for notifying end-users when DQMetrics exceed configured thresholds.

To-Do

Create an AlertConfig to store alert configuration
Create methods to store alert metadata in Delta or Lakebase tables
Create streaming and batch handlers for alerting to Slack, Microsoft Teams, and arbitrary REST API endpoints
Update the apply_checks_... methods to use the required alert handler
Add test fixtures and integration tests

Note

This PR also fixes a minor compatibility issue for has_valid_schema. Schemas are now parsed using the backwards-compatible _parse_datatype_string method instead of fromDDL.

Linked issues

Resolves #204

Tests

github-actions · 2026-01-13T17:15:26Z

❌ 390/502 passed, 37 flaky, 112 failed, 49 skipped, 1h48m44s total

❌ test_e2e_workflow_with_custom_install_folder: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.966s)

... (skipped 13942 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_with_custom_folder: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.466s)

... (skipped 13941 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_workflow_for_multiple_run_configs: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.541s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_workflow_for_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.388s)

... (skipped 13320 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_read_input_data_from_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.606s)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_read_input_data_from_table_with_schema_and_spark_options: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (729ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_read_input_data_from_workspace_file: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (872ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_read_input_data_from_workspace_file_with_spark_options: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (790ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_read_input_data_from_workspace_file_in_csv_format: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (802ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_dataframe_as_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (778ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_streaming_dataframe_in_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (701ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_batch_dataframe_to_path: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (759ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_streaming_dataframe_to_path: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (853ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_reference_dataframes: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (682ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_volume_file_missing: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (657ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_workflow_for_patterns_exclude_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.751s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_volume_file_missing_in_dir_path: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (756ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_missing_volume: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (896ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_and_load_checks_from_volume: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (755ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_volume_in_installation_when_checks_file_does_not_exist: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.952s)

... (skipped 8546 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_load_checks_from_volume_in_user_installation: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.918s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_volume_as_yaml_file: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (753ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_volume_as_json_file: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (735ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_invalid_checks_from_volume_as_yaml_file: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (718ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_workflow_for_patterns_exclude_output: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.37s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_invalid_checks_from_volume_as_json_file: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (759ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_checks_in_volume_file_as_yml: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.662s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_checks_in_volume_file_as_json: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (2.23s)

... (skipped 8546 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_and_load_checks_from_volume_with_filters: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.093s)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_workflow_for_patterns_table_checks_storage: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.957s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_read_input_data_no_input_format: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (729ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_include_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (778ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_no_pattern_match: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (801ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_no_options: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (770ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_no_matched_options: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (785ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_common_opts: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (779ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_different_opts: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (675ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_partial_opts_match: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (891ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (908ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_selected_columns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (770ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_table_non_TEST_SCHEMA_opts: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (990ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_table_with_column_selection: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (749ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (746ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_exclude_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (744ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_for_multiple_run_configs: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.524s)

... (skipped 13312 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_apply_checks_failed: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (737ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_class: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.97s)

... (skipped 13326 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_for_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.025s)

... (skipped 13320 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_table_checks_storage: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.889s)

... (skipped 13650 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_table_with_pk_detection: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (861ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profile_tables_for_patterns_with_pk_detection: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (697ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.763s)

... (skipped 13644 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_for_patterns_with_exclude_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.713s)

... (skipped 13320 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.883s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_with_custom_install_folder: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.591s)

... (skipped 14306 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_for_multiple_run_configs: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.835s)

... (skipped 13643 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_for_patterns_exclude_output: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.357s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_with_custom_install_folder: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (11.55s)

... (skipped 13941 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_for_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.824s)

... (skipped 13643 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_for_multiple_run_configs_table_checks_storage: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.517s)

... (skipped 13643 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_with_quarantine_and_metrics_for_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.068s)

... (skipped 13928 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_for_patterns_table_checks_storage: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.793s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_with_metrics: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (14.552s)

... (skipped 13927 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_for_patterns_exclude_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.213s)

... (skipped 13643 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_filter_out_all_data: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.281s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_with_quarantine_and_metrics: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.932s)

... (skipped 13926 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_for_patterns_exclude_output: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.333s)

... (skipped 13643 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_detect_pk_from_table_with_llm: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (833ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_detect_pk_from_path_with_llm: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (984ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_e2e_workflow_with_metrics: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.501s)

... (skipped 13933 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_for_patterns_table_checks_storage: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.854s)

... (skipped 13642 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_checks_to_table_with_unresolved_for_each_column: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (741ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_table_saved_from_dict_with_unresolved_for_each_column: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (747ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_from_table_with_unresolved_for_each_column: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (738ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_with_special_characters_in_table_name: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (736ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_and_load_checks_from_table_with_run_config: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (761ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_profiler_workflow_save_to_lakebase: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.169s)

... (skipped 13319 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_and_load_checks_from_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (816ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_and_load_checks_to_table_output_modes: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (692ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_load_checks_when_checks_table_does_not_exist: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (932ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_load_checks_from_table_in_user_installation: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.679s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_custom_metrics_in_workflow_for_all_run_configs: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (13.027s)

... (skipped 13927 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (731ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_only_quarantine: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (794ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_only_output: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.007s)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_user_installation: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.803s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_user_installation_output_table_provided: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.938s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_user_installation_only_output: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (2.358s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_user_installation_only_quarantine: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.644s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_user_installation_quarantine_table_provided: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.676s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_user_installation_missing_output_and_quarantine_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.729s)

... (skipped 4670 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_streaming_results_in_table: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (766ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_in_custom_folder_installation: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.563s)

... (skipped 5245 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_with_streaming_quarantine_and_metrics: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.573s)

... (skipped 13934 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_engine_without_observer_no_metrics_saved[apply_checks_and_save_in_table]: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (710ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_save_results_in_table_raises_error_for_sparkconnect: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (735ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_apply_checks_by_metadata_and_save_in_table_raises_error_for_sparkconnect[apply_checks_and_save_in_table]: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (795ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_file_delta_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (708ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_apply_checks_by_metadata_and_save_in_table_raises_error_for_sparkconnect[apply_checks_by_metadata_and_save_in_table]: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (807ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_file_parquet_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (753ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_table_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (839ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_aggregated_table_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (827ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_joined_tables_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (882ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_unioned_tables_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (890ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_mixed_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (816ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_get_tables_and_paths_from_streaming_table_based_dataframe: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (747ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_list_tables_with_exclude_patterns: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (785ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_quality_checker_workflow_with_continuous_streaming_quarantine_and_metrics: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (12.361s)

... (skipped 13927 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_list_tables: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (763ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_end_to_end_installation_and_workflow_with_upload_dependencies: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'main'. (792ms)

... (skipped 427 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_engine_without_observer_no_metrics_saved[apply_checks_by_metadata_and_save_in_table]: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (693ms)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

❌ test_compare_datasets_check_missing_records_with_partial_filter: databricks.sdk.errors.platform.BadRequest: PERMISSION_DENIED: User does not have CREATE SCHEMA on Catalog 'dqx'. (1.1s)

... (skipped 426 bytes)
ux -- Python 3.12.12 /home/runner/work/dqx/dqx/.venv/bin/python

Flaky tests:

🤪 test_apply_checks_and_save_in_tables_streaming_write (1.822s)
🤪 test_apply_checks_and_save_in_tables_for_patterns_exclude_no_tables_matching (808ms)
🤪 test_apply_checks_and_save_in_tables_multiple_tables (815ms)
🤪 test_apply_checks_and_save_in_tables_with_quarantine (749ms)
🤪 test_apply_checks_and_save_in_tables_for_patterns_with_custom_suffix (792ms)
🤪 test_apply_checks_and_save_in_tables_custom_parallelism (892ms)
🤪 test_apply_checks_and_save_in_tables_with_patterns_and_custom_functions (906ms)
🤪 test_apply_checks_and_save_in_tables_with_patterns_and_ref_df (1.009s)
🤪 test_apply_checks_and_save_in_tables_with_custom_functions (783ms)
🤪 test_apply_checks_and_save_in_tables_with_ref_df (819ms)
🤪 test_apply_checks_and_save_in_tables_for_patterns (839ms)
🤪 test_apply_checks_and_save_in_single_table (841ms)
🤪 test_apply_checks_and_save_in_tables_for_patterns_checks_in_table (856ms)
🤪 test_apply_checks_and_save_in_single_table_with_quarantine (768ms)
🤪 test_apply_checks_and_save_in_tables_for_patterns_with_quarantine (843ms)
🤪 test_apply_checks_by_metadata_and_save_in_single_table (820ms)
🤪 test_apply_checks_and_save_in_tables_for_patterns_with_exclude_patterns (793ms)
🤪 test_apply_checks_by_metadata_and_save_in_single_table_with_quarantine (820ms)
🤪 test_apply_checks_and_save_in_table_with_options (916ms)
🤪 test_apply_checks_and_save_in_table_with_different_modes (723ms)
🤪 test_apply_checks_by_metadata_and_save_in_table_with_custom_functions (963ms)
🤪 test_apply_checks_and_save_in_table_with_custom_functions (929ms)
🤪 test_apply_checks_all_row_checks_as_yaml_with_streaming (849ms)
🤪 test_apply_checks_and_save_in_table_with_ref_df (832ms)
🤪 test_apply_checks_by_metadata_and_save_in_table_with_ref_df (666ms)
🤪 test_apply_checks_and_save_in_table_streaming_write (825ms)
🤪 test_apply_checks_and_save_in_tables (696ms)
🤪 test_generate_dq_rules_ai_assisted_with_input_table (722ms)
🤪 test_profiler (15.691s)
🤪 test_foreign_key_check_on_tables (1.042s)
🤪 test_quality_checker (11.743s)
🤪 test_dataset_compare_ref_as_table_and_skip_map_col (1.062s)
🤪 test_profile_with_custom_folder (11.24s)
🤪 test_e2e_workflow (11.485s)
🤪 test_has_valid_schema_with_ref_table (794ms)
🤪 test_apply_checks_with_custom_folder (12.14s)
🤪 test_e2e_workflow (11.136s)

_{Running from acceptance #3563}

codecov · 2026-01-13T17:50:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.36%. Comparing base (c3f26b4) to head (fb92004).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #995      +/-   ##
==========================================
- Coverage   90.58%   88.36%   -2.22%     
==========================================
  Files          64       64              
  Lines        6509     6509              
==========================================
- Hits         5896     5752     -144     
- Misses        613      757     +144

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Use backwards-compatible method for schema parsing

fb92004

ghanse requested a review from a team as a code owner January 13, 2026 16:47

ghanse requested review from pratikk-databricks and removed request for a team January 13, 2026 16:47

ghanse had a problem deploying to tool January 13, 2026 16:47 — with GitHub Actions Failure

ghanse temporarily deployed to tool January 13, 2026 16:47 — with GitHub Actions Inactive

cornzyblack mentioned this pull request Jan 13, 2026

[FEATURE]: Add data quality alerts #204

Open

1 task

mwojtyczka changed the title ~~Add data quality alerts~~ Added data quality alerts Jan 14, 2026

mwojtyczka added the do-not-merge label Jan 14, 2026

Merge branch 'main' into dq_alerts

c222aa5

mwojtyczka had a problem deploying to tool January 14, 2026 18:38 — with GitHub Actions Failure

mwojtyczka temporarily deployed to tool January 14, 2026 18:38 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added data quality alerts #995

Added data quality alerts #995

Uh oh!

ghanse commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026 •

edited

Loading

Uh oh!

codecov bot commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Added data quality alerts #995

Are you sure you want to change the base?

Added data quality alerts #995

Uh oh!

Conversation

ghanse commented Jan 13, 2026

Changes

To-Do

Note

Linked issues

Tests

Uh oh!

github-actions bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 13, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Jan 13, 2026 •

edited

Loading