* develop -> main (#283) * refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/trivy-action@0.24.0 to aquasecurity/trivy-action@0.29.0 * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * Moved scale_data_generator to tests * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: update guide Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * feat: added local server Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * feat: added supported features matrix Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * feat: agents * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * update deps * add mkdocs temporarily * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * ci: correct commit command Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * feat: interface for agents * fix: req, langgraph agent, loggers * fix: rename agent files * fix: lock dependencies * fix: pre-commit formatting * fix: langgraph agent * fix: lock dependencies * docs: added example on how to define and use langgraph agent * feat: created the langgraphagent related workflow, activities, 2. some changes for temporal.py to pass in workflow args properly * chore: examples updated * fix: remove poetry.lock * feat: tests are added for the agentic changes made to sdk * fix: gh actions to install the deps * fix: circular import problem * fix: deps install if deps are added in pyproject.toml * fix: deps duplication * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore(release): bump version (#238) * test2 * test3" " * chore: poetry lock * chore: lock updates * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- Co-authored-by: Nishchith Shetty <inishchith@gmail.com> * chore(release): bump version (#244) * feat: fastapi app added for the agent * fix: lock deps * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * feat: moved everything realted to agent workflow into the code sdk * chore: added examples, and readme to init the fastapiagentapp * fix: poetry lock * fix: failing tests, types * fix: agent type issues, validation, improve error message, llm import * fix: prefix / added in the fastapiagentapp * fix: langgraph related imports * chore: update readme for the agents * fix: error msgs, dependencies, readme * fix: api endpoint, readme example * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) * chore(release): bump version (#260) * fix: lock deps * chore(release): bump version (#264) * feat(docgen): APP-5643 add sql parsing (#237) Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore(release): bump version (#267) * fix: Make e2e test base branch dynamic (#266) * chore(release): bump version (#268) * docs: added decorator doc strings and examples (#265) * chore(release): bump version (#269) * docs: APP-5752 setup mkdocs with pydoctor (#263) * chore: cleanup * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore(release): bump version (#270) * fix(docs): APP-5272 correct docstrings for decorators (#271) * fix(docs): correct docstrings for decorators * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore: cleanup Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> * chore(release): bump version (#272) * fix: heartbeat, timeout configurable * chore(release): bump version (#275) * fix: usage of workflow_config and workflow args (#276) * chore(release): bump version (#277) * fix: langchain-openai dependency (#282) --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> Co-authored-by: Mustafa <mustafa.trunkwala@atlan.com> Co-authored-by: Abhishek Agrawal <abhishek.agrawal@atlan.com> Co-authored-by: Junaid Rahim <junaidrahim8d@gmail.com> Co-authored-by: Mrunmayi Tripathi <121743130+AtMrun@users.noreply.github.com> Co-authored-by: Mustafa <100032108+TechyMT@users.noreply.github.com> Co-authored-by: Ujala Singh <59909223+ujala-singh@users.noreply.github.com> Co-authored-by: SanilK2108 <sanil.khurana@atlan.com> Co-authored-by: ujala-singh <ujala2yz@gmail.com> Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com> Co-authored-by: Hrushikesh Dokala <hrushikesh.dokala@atlan.com> Co-authored-by: Atlan CI <62283865+atlan-ci@users.noreply.github.com> Co-authored-by: Amit Prabhu <amit.dp180@gmail.com> * fix: poetry lock --------- Signed-off-by: Junaid Rahim <junaidrahim8d@gmail.com> Co-authored-by: Nishchith Shetty <inishchith@gmail.com> Co-authored-by: Mustafa <mustafa.trunkwala@atlan.com> Co-authored-by: Abhishek Agrawal <abhishek.agrawal@atlan.com> Co-authored-by: Junaid Rahim <junaidrahim8d@gmail.com> Co-authored-by: Mrunmayi Tripathi <121743130+AtMrun@users.noreply.github.com> Co-authored-by: Mustafa <100032108+TechyMT@users.noreply.github.com> Co-authored-by: Ujala Singh <59909223+ujala-singh@users.noreply.github.com> Co-authored-by: ujala-singh <ujala2yz@gmail.com> Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com> Co-authored-by: Hrushikesh Dokala <hrushikesh.dokala@atlan.com> Co-authored-by: Atlan CI <62283865+atlan-ci@users.noreply.github.com> Co-authored-by: Amit Prabhu <amit.dp180@gmail.com>