feature : Added Streamlit based UI for observability #558

abhishekagrawal-atlan · 2025-05-27T01:30:19Z

Changelog

Added Streamlit based UI for observability
Made it backward compatible through a flag with optional modules to ensure customers have a choice on what to use for the observability UI ( duckdb or Streamlit ).
Streamlit UI adds visualisation capabilities for metrics / traces etc.

Checklist

Additional tests added
All CI checks passed
Relevant documentation updated

Copyleft License Compliance

Have you used any code that is subject to a Copyleft license (e.g., GPL, AGPL, LGPL)?
If yes, have you modified the code in the context of this project? please share additional details.

- Added Streamlit based UI for observability - Made it backward compatible through a flag with optional modules to ensure customers have a choice on what to use for the observability UI ( duckdb or Streamlit ). - Streamlit UI adds visualisation capabilities for metrics / traces etc.

snyk-io-us · 2025-05-27T01:30:31Z

⚠️ Snyk checks are incomplete. No issues have been found so far.

⚠️ security/snyk check encountered an error. (View Details)

⚠️ license/snyk check encountered an error. (View Details)

✅ code/snyk check is complete. No issues have been found. (View Details)

github-actions · 2025-05-27T01:30:40Z

📜 Docstring Coverage Report

RESULT: PASSED (minimum: 30.0%, actual: 74.0%)

Detailed Coverage Report

======= Coverage for /home/runner/work/application-sdk/application-sdk/ ========
----------------------------------- Summary ------------------------------------
| Name                                                                              | Total | Miss | Cover | Cover% |
|-----------------------------------------------------------------------------------|-------|------|-------|--------|
| application_sdk/__init__.py                                                       |     1 |    0 |     1 |   100% |
| application_sdk/constants.py                                                      |     1 |    0 |     1 |   100% |
| application_sdk/version.py                                                        |     1 |    0 |     1 |   100% |
| application_sdk/worker.py                                                         |     4 |    0 |     4 |   100% |
| application_sdk/activities/__init__.py                                            |     9 |    0 |     9 |   100% |
| application_sdk/activities/common/__init__.py                                     |     1 |    1 |     0 |     0% |
| application_sdk/activities/common/models.py                                       |     2 |    0 |     2 |   100% |
| application_sdk/activities/common/utils.py                                        |     5 |    1 |     4 |    80% |
| application_sdk/activities/metadata_extraction/__init__.py                        |     1 |    1 |     0 |     0% |
| application_sdk/activities/metadata_extraction/rest.py                            |     1 |    1 |     0 |     0% |
| application_sdk/activities/metadata_extraction/sql.py                             |    15 |    1 |    14 |    93% |
| application_sdk/activities/query_extraction/__init__.py                           |     1 |    1 |     0 |     0% |
| application_sdk/activities/query_extraction/sql.py                                |    10 |    1 |     9 |    90% |
| application_sdk/application/__init__.py                                           |     8 |    1 |     7 |    88% |
| application_sdk/application/metadata_extraction/sql.py                            |     7 |    1 |     6 |    86% |
| application_sdk/clients/__init__.py                                               |     4 |    0 |     4 |   100% |
| application_sdk/clients/sql.py                                                    |    15 |    0 |    15 |   100% |
| application_sdk/clients/temporal.py                                               |    19 |    1 |    18 |    95% |
| application_sdk/clients/utils.py                                                  |     2 |    1 |     1 |    50% |
| application_sdk/clients/workflow.py                                               |     9 |    2 |     7 |    78% |
| application_sdk/common/__init__.py                                                |     1 |    1 |     0 |     0% |
| application_sdk/common/aws_utils.py                                               |     4 |    1 |     3 |    75% |
| application_sdk/common/dataframe_utils.py                                         |     2 |    1 |     1 |    50% |
| application_sdk/common/error_codes.py                                             |    14 |    2 |    12 |    86% |
| application_sdk/common/utils.py                                                   |    12 |    2 |    10 |    83% |
| application_sdk/docgen/__init__.py                                                |     5 |    2 |     3 |    60% |
| application_sdk/docgen/exporters/__init__.py                                      |     1 |    1 |     0 |     0% |
| application_sdk/docgen/exporters/mkdocs.py                                        |     7 |    3 |     4 |    57% |
| application_sdk/docgen/models/__init__.py                                         |     1 |    1 |     0 |     0% |
| application_sdk/docgen/models/export/__init__.py                                  |     1 |    1 |     0 |     0% |
| application_sdk/docgen/models/export/page.py                                      |     2 |    1 |     1 |    50% |
| application_sdk/docgen/models/manifest/__init__.py                                |     2 |    1 |     1 |    50% |
| application_sdk/docgen/models/manifest/customer.py                                |     3 |    1 |     2 |    67% |
| application_sdk/docgen/models/manifest/internal.py                                |     2 |    1 |     1 |    50% |
| application_sdk/docgen/models/manifest/metadata.py                                |     2 |    1 |     1 |    50% |
| application_sdk/docgen/models/manifest/page.py                                    |     2 |    1 |     1 |    50% |
| application_sdk/docgen/models/manifest/section.py                                 |     2 |    1 |     1 |    50% |
| application_sdk/docgen/parsers/__init__.py                                        |     1 |    1 |     0 |     0% |
| application_sdk/docgen/parsers/directory.py                                       |    13 |    2 |    11 |    85% |
| application_sdk/docgen/parsers/manifest.py                                        |     6 |    1 |     5 |    83% |
| application_sdk/handlers/__init__.py                                              |     6 |    1 |     5 |    83% |
| application_sdk/handlers/sql.py                                                   |    16 |    3 |    13 |    81% |
| application_sdk/inputs/__init__.py                                                |     6 |    1 |     5 |    83% |
| application_sdk/inputs/iceberg.py                                                 |     7 |    3 |     4 |    57% |
| application_sdk/inputs/json.py                                                    |     8 |    2 |     6 |    75% |
| application_sdk/inputs/objectstore.py                                             |     4 |    1 |     3 |    75% |
| application_sdk/inputs/parquet.py                                                 |     8 |    1 |     7 |    88% |
| application_sdk/inputs/secretstore.py                                             |     3 |    1 |     2 |    67% |
| application_sdk/inputs/sql_query.py                                               |    10 |    1 |     9 |    90% |
| application_sdk/inputs/statestore.py                                              |     4 |    1 |     3 |    75% |
| application_sdk/observability/__init__.py                                         |     1 |    1 |     0 |     0% |
| application_sdk/observability/logger_adaptor.py                                   |    31 |    3 |    28 |    90% |
| application_sdk/observability/metrics_adaptor.py                                  |    16 |    2 |    14 |    88% |
| application_sdk/observability/observability.py                                    |    28 |    1 |    27 |    96% |
| application_sdk/observability/streamlit_ui.py                                     |     2 |    2 |     0 |     0% |
| application_sdk/observability/traces_adaptor.py                                   |    16 |    1 |    15 |    94% |
| application_sdk/observability/decorators/observability_decorator.py               |     7 |    4 |     3 |    43% |
| application_sdk/outputs/__init__.py                                               |     9 |    0 |     9 |   100% |
| application_sdk/outputs/eventstore.py                                             |    10 |    0 |    10 |   100% |
| application_sdk/outputs/iceberg.py                                                |     5 |    1 |     4 |    80% |
| application_sdk/outputs/json.py                                                   |     8 |    1 |     7 |    88% |
| application_sdk/outputs/objectstore.py                                            |     4 |    1 |     3 |    75% |
| application_sdk/outputs/parquet.py                                                |     7 |    1 |     6 |    86% |
| application_sdk/outputs/secretstore.py                                            |     3 |    1 |     2 |    67% |
| application_sdk/outputs/statestore.py                                             |     4 |    1 |     3 |    75% |
| application_sdk/server/__init__.py                                                |     4 |    0 |     4 |   100% |
| application_sdk/server/fastapi/__init__.py                                        |    24 |    6 |    18 |    75% |
| application_sdk/server/fastapi/models.py                                          |    18 |   18 |     0 |     0% |
| application_sdk/server/fastapi/utils.py                                           |     2 |    0 |     2 |   100% |
| application_sdk/server/fastapi/middleware/logmiddleware.py                        |     4 |    4 |     0 |     0% |
| application_sdk/server/fastapi/routers/__init__.py                                |     1 |    1 |     0 |     0% |
| application_sdk/server/fastapi/routers/server.py                                  |     8 |    2 |     6 |    75% |
| application_sdk/test_utils/__init__.py                                            |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/workflow_monitoring.py                                 |     3 |    0 |     3 |   100% |
| application_sdk/test_utils/e2e/__init__.py                                        |    14 |    2 |    12 |    86% |
| application_sdk/test_utils/e2e/base.py                                            |    16 |    2 |    14 |    88% |
| application_sdk/test_utils/e2e/client.py                                          |    10 |    2 |     8 |    80% |
| application_sdk/test_utils/e2e/conftest.py                                        |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/e2e/utils.py                                           |     3 |    1 |     2 |    67% |
| application_sdk/test_utils/hypothesis/__init__.py                                 |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/__init__.py                      |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/sql_client.py                    |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/temporal.py                      |     6 |    1 |     5 |    83% |
| application_sdk/test_utils/hypothesis/strategies/clients/__init__.py              |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/clients/sql.py                   |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/common/__init__.py               |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/common/logger.py                 |     3 |    0 |     3 |   100% |
| application_sdk/test_utils/hypothesis/strategies/handlers/__init__.py             |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/handlers/sql/__init__.py         |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/handlers/sql/sql_metadata.py     |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/handlers/sql/sql_preflight.py    |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/inputs/__init__.py               |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/inputs/json_input.py             |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/outputs/__init__.py              |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/outputs/json_output.py           |     2 |    1 |     1 |    50% |
| application_sdk/test_utils/hypothesis/strategies/outputs/statestore.py            |     3 |    1 |     2 |    67% |
| application_sdk/test_utils/hypothesis/strategies/server/__init__.py               |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/hypothesis/strategies/server/fastapi/__init__.py       |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/scale_data_generator/__init__.py                       |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/scale_data_generator/config_loader.py                  |    10 |    4 |     6 |    60% |
| application_sdk/test_utils/scale_data_generator/data_generator.py                 |    10 |    3 |     7 |    70% |
| application_sdk/test_utils/scale_data_generator/driver.py                         |     3 |    3 |     0 |     0% |
| application_sdk/test_utils/scale_data_generator/output_handler/__init__.py        |     1 |    1 |     0 |     0% |
| application_sdk/test_utils/scale_data_generator/output_handler/base.py            |     7 |    3 |     4 |    57% |
| application_sdk/test_utils/scale_data_generator/output_handler/csv_handler.py     |     5 |    5 |     0 |     0% |
| application_sdk/test_utils/scale_data_generator/output_handler/json_handler.py    |     5 |    5 |     0 |     0% |
| application_sdk/test_utils/scale_data_generator/output_handler/parquet_handler.py |     6 |    6 |     0 |     0% |
| application_sdk/transformers/__init__.py                                          |     3 |    1 |     2 |    67% |
| application_sdk/transformers/atlas/__init__.py                                    |     6 |    1 |     5 |    83% |
| application_sdk/transformers/atlas/sql.py                                         |    25 |    4 |    21 |    84% |
| application_sdk/transformers/common/__init__.py                                   |     1 |    1 |     0 |     0% |
| application_sdk/transformers/common/utils.py                                      |     5 |    0 |     5 |   100% |
| application_sdk/transformers/query/__init__.py                                    |    11 |    2 |     9 |    82% |
| application_sdk/workflows/__init__.py                                             |     4 |    0 |     4 |   100% |
| application_sdk/workflows/metadata_extraction/__init__.py                         |     2 |    2 |     0 |     0% |
| application_sdk/workflows/metadata_extraction/sql.py                              |     7 |    0 |     7 |   100% |
| application_sdk/workflows/query_extraction/__init__.py                            |     2 |    2 |     0 |     0% |
| application_sdk/workflows/query_extraction/sql.py                                 |     4 |    0 |     4 |   100% |
| examples/application_custom_fastapi.py                                            |    14 |   14 |     0 |     0% |
| examples/application_fastapi.py                                                   |     9 |    9 |     0 |     0% |
| examples/application_hello_world.py                                               |     7 |    7 |     0 |     0% |
| examples/application_sql.py                                                       |     5 |    4 |     1 |    20% |
| examples/application_sql_miner.py                                                 |     5 |    4 |     1 |    20% |
| examples/application_sql_with_custom_pyatlan_transformer.py                       |    11 |    9 |     2 |    18% |
| examples/application_sql_with_custom_transformer.py                               |     9 |    8 |     1 |    11% |
| examples/application_subscriber.py                                                |    14 |   14 |     0 |     0% |
| examples/run_examples.py                                                          |     2 |    1 |     1 |    50% |
| tests/__init__.py                                                                 |     1 |    1 |     0 |     0% |
| tests/unit/__init__.py                                                            |     1 |    1 |     0 |     0% |
| tests/unit/worker.py                                                              |     5 |    5 |     0 |     0% |
| tests/unit/activities/__init__.py                                                 |     1 |    1 |     0 |     0% |
| tests/unit/clients/__init__.py                                                    |     1 |    1 |     0 |     0% |
| tests/unit/clients/test_async_sql_client.py                                       |    13 |   13 |     0 |     0% |
| tests/unit/clients/test_sql_client.py                                             |    28 |    6 |    22 |    79% |
| tests/unit/clients/test_temporal_client.py                                        |    17 |   11 |     6 |    35% |
| tests/unit/common/test_logger_adapter.py                                          |    19 |    1 |    18 |    95% |
| tests/unit/common/test_metrics_adapter.py                                         |    14 |    1 |    13 |    93% |
| tests/unit/common/test_traces_adaptor.py                                          |    10 |    1 |     9 |    90% |
| tests/unit/common/test_utils.py                                                   |    25 |    5 |    20 |    80% |
| tests/unit/docgen/parsers/test_directory_parser.py                                |    14 |    3 |    11 |    79% |
| tests/unit/docgen/parsers/test_manifest_parser.py                                 |    12 |   12 |     0 |     0% |
| tests/unit/handlers/__init__.py                                                   |     1 |    1 |     0 |     0% |
| tests/unit/handlers/sql/test_auth.py                                              |    10 |    4 |     6 |    60% |
| tests/unit/handlers/sql/test_check_schemas_and_databases.py                       |    13 |    4 |     9 |    69% |
| tests/unit/handlers/sql/test_extract_allowed_schemas.py                           |    11 |    3 |     8 |    73% |
| tests/unit/handlers/sql/test_metadata.py                                          |    27 |   10 |    17 |    63% |
| tests/unit/handlers/sql/test_preflight_check.py                                   |    16 |   15 |     1 |     6% |
| tests/unit/handlers/sql/test_prepare_metadata.py                                  |    14 |    4 |    10 |    71% |
| tests/unit/handlers/sql/test_tables_check.py                                      |     9 |    6 |     3 |    33% |
| tests/unit/handlers/sql/test_validate_filters.py                                  |    12 |    4 |     8 |    67% |
| tests/unit/inputs/test_json_input.py                                              |     4 |    4 |     0 |     0% |
| tests/unit/outputs/test_iceberg.py                                                |    11 |    4 |     7 |    64% |
| tests/unit/outputs/test_json_output.py                                            |     7 |    6 |     1 |    14% |
| tests/unit/outputs/test_objectstore.py                                            |     9 |    8 |     1 |    11% |
| tests/unit/outputs/test_output.py                                                 |    19 |    6 |    13 |    68% |
| tests/unit/outputs/test_statestore.py                                             |    11 |   11 |     0 |     0% |
| tests/unit/server/__init__.py                                                     |     1 |    1 |     0 |     0% |
| tests/unit/server/fastapi/test__init__.py                                         |    11 |    6 |     5 |    45% |
| tests/unit/server/fastapi/routers/__init__.py                                     |     1 |    1 |     0 |     0% |
| tests/unit/server/fastapi/routers/server.py                                       |     1 |    1 |     0 |     0% |
| tests/unit/transformers/__init__.py                                               |     1 |    1 |     0 |     0% |
| tests/unit/transformers/atlas/__init__.py                                         |     1 |    1 |     0 |     0% |
| tests/unit/transformers/atlas/test_column.py                                      |    17 |    6 |    11 |    65% |
| tests/unit/transformers/atlas/test_database.py                                    |     8 |    6 |     2 |    25% |
| tests/unit/transformers/atlas/test_function.py                                    |     9 |    5 |     4 |    44% |
| tests/unit/transformers/atlas/test_procedure.py                                   |     7 |    6 |     1 |    14% |
| tests/unit/transformers/atlas/test_schema.py                                      |     8 |    6 |     2 |    25% |
| tests/unit/transformers/atlas/test_table.py                                       |    13 |    6 |     7 |    54% |
| tests/unit/transformers/query/test_sql_transformer.py                             |    14 |    4 |    10 |    71% |
| tests/unit/transformers/query/test_sql_transformer_output_validation.py           |     5 |    2 |     3 |    60% |
| tests/unit/workflows/sql/__init__.py                                              |     1 |    1 |     0 |     0% |
| tests/unit/workflows/sql/test_workflow.py                                         |     9 |    4 |     5 |    56% |
|-----------------------------------------------------------------------------------|-------|------|-------|--------|
| TOTAL                                                                             |  1196 |  455 |   741 |  62.0% |
---------------- RESULT: PASSED (minimum: 30.0%, actual: 62.0%) ----------------

github-actions · 2025-05-27T01:31:18Z

📦 Trivy Vulnerability Scan Results

Schema Version	Created At	Artifact	Type
2	2025-05-30T04:00:44.545583103Z	.	filesystem

Report Summary

Target	Type	Vulnerabilities	`.`	filesystem	✅ None found

Scan Result Details

✅ No vulnerabilities found during the scan for ..

github-actions · 2025-05-27T01:31:19Z

📦 Trivy Secret Scan Results

Schema Version	Created At	Artifact	Type
2	2025-05-30T04:00:50.04391318Z	.	filesystem

Report Summary

Target	Type	Secrets	`.`	filesystem	✅ None found

Scan Result Details

✅ No secrets found during the scan for ..

github-actions · 2025-05-27T01:31:21Z

🛠 Docs available at: https://k.atlan.dev/application-sdk/streamlit_for_observability

atlan-ci · 2025-05-27T01:31:30Z

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
5231	2781	53%	0%	🟢

New Files

File	Coverage	Status
application_sdk/observability/streamlit_ui.py	0%	🟢
TOTAL	0%	🟢

Modified Files

File	Coverage	Status
application_sdk/constants.py	100%	🟢
application_sdk/observability/observability.py	46%	🟢
application_sdk/server/fastapi/init.py	61%	🟢
TOTAL	69%	🟢

updated for commit: 2eab2a8 by action🐍

github-actions · 2025-05-27T01:31:44Z

🛠 Full Test Coverage Report: https://k.atlan.dev/coverage/application-sdk/pr/558

…lit_for_observability

abhishekagrawal-atlan added 2 commits May 27, 2025 06:57

chore: added documentation

bdae0f0

abhishekagrawal-atlan changed the title ~~feature : Added streamlit based UI for observability~~ feature : Added Streamlit based UI for observability May 27, 2025

abhishekagrawal-atlan added 2 commits May 27, 2025 09:39

fix : pre-commit errors

d0203fd

Merge branch 'main' of github.com:atlanhq/application-sdk into stream…

2eab2a8

…lit_for_observability

abhishekagrawal-atlan closed this Sep 8, 2025

abhishekagrawal-atlan deleted the streamlit_for_observability branch September 8, 2025 10:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feature : Added Streamlit based UI for observability #558

feature : Added Streamlit based UI for observability #558

Uh oh!

abhishekagrawal-atlan commented May 27, 2025

Uh oh!

snyk-io-us bot commented May 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 27, 2025 •

edited

Loading

Uh oh!

atlan-ci commented May 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 27, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feature : Added Streamlit based UI for observability #558

feature : Added Streamlit based UI for observability #558

Uh oh!

Conversation

abhishekagrawal-atlan commented May 27, 2025

Changelog

Checklist

Copyleft License Compliance

Uh oh!

snyk-io-us bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Snyk checks are incomplete. No issues have been found so far.

Uh oh!

github-actions bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📜 Docstring Coverage Report

Uh oh!

github-actions bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Trivy Vulnerability Scan Results

Report Summary

Scan Result Details

Uh oh!

github-actions bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Trivy Secret Scan Results

Report Summary

Scan Result Details

Uh oh!

github-actions bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

atlan-ci commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

☂️ Python Coverage

Overall Coverage

New Files

Modified Files

Uh oh!

github-actions bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

snyk-io-us bot commented May 27, 2025 •

edited

Loading

github-actions bot commented May 27, 2025 •

edited

Loading

github-actions bot commented May 27, 2025 •

edited

Loading

github-actions bot commented May 27, 2025 •

edited

Loading

github-actions bot commented May 27, 2025 •

edited

Loading

atlan-ci commented May 27, 2025 •

edited

Loading

github-actions bot commented May 27, 2025 •

edited

Loading