APP-5362 : feat: integrate uvloop for improved asyncio performance #179
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TechyMT
requested review from
firecast,
inishchith and
AtMrun
as code owners
WalkthroughThe changes integrate the Changes
Sequence Diagram(s)sequenceDiagram
participant App
participant Decorators
participant Worker
participant AsyncIO
participant uvloop
App->>Decorators: Load module
Decorators->>AsyncIO: Set event loop policy\n(using uvloop.EventLoopPolicy())
AsyncIO->>uvloop: Configure event loop
App->>Worker: Initialize Worker
Worker->>AsyncIO: Set event loop policy\n(using uvloop.EventLoopPolicy())
AsyncIO->>uvloop: Async operations now use uvloop
Poem
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
📜 Docstring Coverage ReportRESULT: PASSED (minimum: 65.0%, actual: 67.6%) Detailed Coverage Report |
|
🛠 Library Docs available at: https://k.atlan.dev/application-sdk/feat/uvloop 📝 Packages Diagram: https://k.atlan.dev/application-sdk/feat/uvloop/packages.html |
📦 Example workflows test results
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (12)
application_sdk/clients/sql.py (1)
23-23: Set the event loop policy once at a central entry point
While setting the event loop policy at the library level works, it is often best practice to do so once at the main entry script to prevent confusion or potential conflicts if multiple files attempt to set different policies.examples/application_hello_world.py (1)
16-16: Multiple policy settings can lead to confusion
Consider ensuring thatasyncio.set_event_loop_policyis set in a single, central place, rather than in each example, to avoid accidental conflicts.examples/application_fastapi.py (1)
10-11: Event loop policy setting in FastAPI example
Again, consider consolidatingasyncio.set_event_loop_policy(uvloop.EventLoopPolicy())to a single main initialization to maintain consistent loop configuration across modules.examples/application_custom_fastapi.py (1)
5-6: LGTM! Consider moving uvloop initialization to application startup.The uvloop integration is correct, but for FastAPI applications, it's a good practice to initialize uvloop during application startup rather than at module level.
Consider moving the initialization to the application startup:
-asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) class MyCustomFastAPIApplication(FastAPIApplication): + def __init__(self, handler: HandlerInterface): + asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) + super().__init__(handler)Also applies to: 12-13
application_sdk/application/fastapi/routers/server.py (1)
20-20: Consider moving uvloop initialization to application startup.Setting the event loop policy at the module level could affect other modules importing this one. Since FastAPI typically uses uvicorn which already supports uvloop, consider:
- Moving the uvloop initialization to the application startup event
- Configuring uvloop through uvicorn settings instead
-import uvloop -asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) + +def on_startup(): + """Configure event loop policy during application startup.""" + import uvloop + asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) + +app.add_event_handler("startup", on_startup)Also applies to: 27-27
application_sdk/workflows/query_extraction/sql.py (1)
11-11: Monitor workflow performance after uvloop integration.While uvloop should improve the performance of
asyncio.gather()for batch processing, please:
- Monitor workflow execution times
- Compare performance metrics before and after uvloop integration
- Watch for any stability issues
Also applies to: 24-24
examples/application_subscriber.py (2)
5-5: Document uvloop usage in example code.While this example demonstrates uvloop integration, consider:
- Adding comments explaining the performance benefits
- Documenting any platform-specific considerations (e.g., Windows compatibility)
- Making uvloop optional in examples to avoid confusion
+# Note: uvloop provides better performance for asyncio operations but is optional +# and may not be available on all platforms (e.g., Windows). import uvloop asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())Also applies to: 25-25
5-25: Verify platform compatibility and add fallback mechanism.uvloop may not be available on all platforms, particularly Windows. Consider adding a fallback mechanism:
try: import uvloop asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) except ImportError: # Fallback to default event loop policy logger.warning("uvloop not available, using default event loop policy")examples/application_sql.py (1)
180-181: Consider using asyncio.Event() for graceful shutdown.Using
time.sleep(1000000)is not a recommended way to keep the script running. Consider using an asyncio event for graceful shutdown handling.- asyncio.run(application_sql(daemon=False)) + stop_event = asyncio.Event() + try: + asyncio.run(application_sql(daemon=False)) + # Wait for shutdown signal + asyncio.get_event_loop().run_until_complete(stop_event.wait()) + except KeyboardInterrupt: + stop_event.set()examples/application_sql_with_custom_transformer.py (1)
170-170: Replace hardcoded sleep with async wait.Using
time.sleep(3)blocks the event loop. Consider usingasyncio.sleep()instead.- time.sleep(3) + await asyncio.sleep(3)examples/application_sql_miner.py (2)
159-159: Replace hardcoded sleep with async wait.Using
time.sleep(3)blocks the event loop. Consider usingasyncio.sleep()instead.- time.sleep(3) + await asyncio.sleep(3)
114-114: Address the FIXME comment for Snowflake connection parameters.The FIXME comment indicates that additional Snowflake connection parameters need to be added.
Would you like me to help implement the additional Snowflake connection parameters? I can provide a comprehensive list of supported parameters and their implementation.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock
📒 Files selected for processing (22)
application_sdk/activities/common/utils.py(1 hunks)application_sdk/application/fastapi/routers/server.py(1 hunks)application_sdk/clients/sql.py(2 hunks)application_sdk/decorators/__init__.py(1 hunks)application_sdk/handlers/sql.py(2 hunks)application_sdk/inputs/sql_query.py(2 hunks)application_sdk/outputs/parquet.py(1 hunks)application_sdk/test_utils/workflow_monitoring.py(1 hunks)application_sdk/worker.py(1 hunks)application_sdk/workflows/metadata_extraction/sql.py(2 hunks)application_sdk/workflows/query_extraction/sql.py(2 hunks)examples/application_custom_fastapi.py(1 hunks)examples/application_fastapi.py(1 hunks)examples/application_hello_world.py(2 hunks)examples/application_sql.py(2 hunks)examples/application_sql_miner.py(2 hunks)examples/application_sql_with_custom_transformer.py(2 hunks)examples/application_subscriber.py(2 hunks)examples/run_examples.py(2 hunks)pyproject.toml(1 hunks)tests/unit/clients/test_async_sql_client.py(1 hunks)tests/unit/clients/test_sql_client.py(1 hunks)
✅ Files skipped from review due to trivial changes (3)
- application_sdk/workflows/metadata_extraction/sql.py
- application_sdk/handlers/sql.py
- application_sdk/inputs/sql_query.py
🧰 Additional context used
🪛 GitHub Actions: Tests
tests/unit/clients/test_sql_client.py
[error] 5-5: ModuleNotFoundError: No module named 'pandas'
tests/unit/clients/test_async_sql_client.py
[error] 5-5: ModuleNotFoundError: No module named 'pandas'
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: test
🔇 Additional comments (15)
application_sdk/clients/sql.py (1)
14-14: Use ofuvloopimport for performance boost
This import aligns well with the PR objectives of integratinguvloopfor improved asyncio performance.examples/application_hello_world.py (1)
4-4: Importinguvloopin examples
Includinguvloophere is consistent with the goal of demonstrating performance improvements in example applications.examples/application_fastapi.py (1)
4-5:uvloopimport and formatting
This import and spacing look fine. If the application runs on multiple platforms, confirm thatuvloopsupport covers your target environment(s).examples/run_examples.py (1)
7-8: LGTM! Verify uvloop compatibility with Temporal.The uvloop integration is correctly placed before any async operations. However, since this script interacts with Temporal, it's good to verify compatibility.
Also applies to: 18-19
✅ Verification successful
🌐 Web query:
Is uvloop compatible with Temporal Python SDK?💡 Result:
The uvloop is generally compatible with the Temporal Python SDK, but there are important implementation details to consider:
Compatibility Overview
Temporal's Custom Asyncio Event Loop
The Temporal Python SDK uses its ownasyncio.AbstractEventLoopimplementation to manage durable workflows ([source 1][4]). This custom loop handles workflow execution deterministically and persists state across interruptions.uvloop as a Drop-In Replacement
uvloop replaces Python’s built-in asyncio event loop but does not conflict with Temporal’s workflow-specific loop. Temporal’s SDK isolates workflow execution from your application’s general async code ([source 1][4], [source 9][9]).Activity Compatibility
For activities (non-workflow code), uvloop can improve performance when:
- Activities use
async defand run outside the workflow sandbox.- You explicitly set uvloop as the event loop for activity workers:
import uvloop uvloop.run(main_activity_worker_function())This configuration is safe because activities are stateless and do not require deterministic replay ([source 3][3], [source 6][6]).
Key Considerations
- Workflow Code: Avoid using uvloop directly in workflows, as Temporal’s SDK manages its own event loop for durability.
- Activity Code: Use uvloop freely in async activities for performance gains.
- Worker Setup: Configure uvloop at the worker level (e.g., in activity executors) rather than within workflows.
Example Configuration
import uvloop from temporalio.client import Client from temporalio.worker import Worker async def main(): client = await Client.connect("localhost:7233") # Use uvloop for activity execution worker = Worker( client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_activity], ) uvloop.run(worker.run())
Performance Impact
- uvloop can accelerate async activities by 2–4x ([source 2][2]).
- Workflow performance remains unaffected since Temporal’s custom loop prioritizes durability over raw speed ([source 9][9]).
In summary, uvloop complements the Temporal Python SDK when used judiciously in activity execution but should not interfere with workflow internals.
Citations:
- 1: https://temporal.io/blog/durable-distributed-asyncio-event-loop
- 2: https://pypi.org/project/uvloop/
- 3: https://community.temporal.io/t/invoking-coroutine-in-activity-python-sdk/1127
- 4: https://github.com/temporalio/sdk-python
- 5: https://docs.temporal.io/develop/python
- 6: https://dev.to/temporalio/python-sdk-the-release-2ji9?comments_sort=top
- 7: https://learn.temporal.io/tutorials/python/background-check/introduction/
- 8: https://youtrack.jetbrains.com/issue/PY-62467/TypeError-Task-object-is-not-callable-debugging-uvloop-with-asyncio-support-enabled
- 9: https://www.youtube.com/watch?v=jxqPWS_LRzw
- 10: https://community.temporal.io/t/python-sdk-metrics/6030
LGTM – uvloop integration confirmed with Temporal compatibility
- Import and Policy Order: The
uvloopimport at line 7 and event loop policy setting at line 18 are correctly positioned before any asynchronous operations.- Temporal Compatibility: Our web query confirms that uvloop works well with the Temporal Python SDK when used outside workflow execution. It’s a drop-in replacement for regular asynchronous activity execution without conflicting with Temporal’s custom workflow event loop.
- Usage Recommendation: Ensure that uvloop is applied only to the appropriate parts (e.g., activity workers) and not within workflow code, which Temporal manages separately.
application_sdk/test_utils/workflow_monitoring.py (1)
10-11: Verify timeout behavior with uvloop.While the uvloop integration is correct, it's important to verify that the timeout and polling behavior remains consistent with uvloop's event loop implementation.
Also applies to: 16-17
✅ Verification successful
🏁 Script executed:
#!/bin/bash # Description: Check if there are any known issues with uvloop's timeout handling # that might affect the monitoring functionality # Search for timeout-related code patterns rg -A 2 "asyncio.sleep|time.time" # Look for any existing timeout handling patterns ast-grep --pattern 'if timeout and (time.time() - start_time) > timeout:'Length of output: 3790
Timeout behavior verified with uvloop integration.
After reviewing the patterns in the timeout handling code in
application_sdk/test_utils/workflow_monitoring.py, the timeout logic usingtime.time()andawait asyncio.sleep()is implemented consistently. There is no evidence that uvloop’s event loop policy alters the intended behavior.
- The conditional check for timeout (
if timeout and (time.time() - start_time) > timeout:) is intact.- The polling mechanism using
await asyncio.sleep(polling_interval)aligns with asynchronous timing expectations regardless of the event loop used.application_sdk/activities/common/utils.py (1)
6-6: Verify Temporal's compatibility with uvloop.Setting uvloop as the event loop policy could affect Temporal's activity execution. Please verify compatibility between Temporal and uvloop.
Also applies to: 12-12
✅ Verification successful
🌐 Web query:
Does Temporal.io Python SDK support uvloop?💡 Result:
The Temporal Python SDK does not directly support uvloop integration for workflow execution due to its custom event loop implementation[4][7]. However, there are important nuances:
Key Technical Relationship
Aspect Temporal SDK uvloop Core Purpose Durable distributed workflows High-performance async I/O Event Loop Type Custom "durable" asyncio loop[4] Optimized libuv-based loop[2][6] Workflow Execution Requires SDK's own loop[4][7] Not compatible Activity Execution Can use uvloop[7][9] Supported via config
Workflows use Temporal's specialized event loop that persists execution state[4][7]. This custom loop cannot be replaced by uvloop as it implements critical durability mechanisms.
Activities (normal async functions) can leverage uvloop when:
- Configured via
activity_executorin workers[7][9]- Using separate process pools with uvloop-enabled event loops
Performance considerations:
- Temporal already optimizes workflow execution through its distributed architecture
- uvloop may still benefit activity I/O operations when properly isolated[2][6]
For optimal results:
# In worker configuration from concurrent.futures import ProcessPoolExecutor import uvloop async def main(): client = await Client.connect("localhost:7233") with ProcessPoolExecutor() as executor: worker = Worker( client, task_queue="my-queue", activities=[my_activity], activity_executor=executor # Can use uvloop here ) await worker.run()This architecture allows Temporal to manage workflows while enabling uvloop-accelerated activities through process isolation[7][9].
Citations:
- 1: https://docs.temporal.io/develop/python
- 2: https://pypi.org/project/uvloop/
- 3: https://uvloop.readthedocs.io
- 4: https://temporal.io/blog/durable-distributed-asyncio-event-loop
- 5: [feature request] add loop._run_once and loop._set_is_runnning api to easy the process of OS gui event loop integrate MagicStack/uvloop#648
- 6: https://magic.io/blog/uvloop-blazing-fast-python-networking/
- 7: https://github.com/temporalio/sdk-python
- 8: https://docs.temporal.io/cloud/support
- 9: https://www.restack.io/docs/temporal-knowledge-temporal-io-python-integration
- 10: https://learn.temporal.io/getting_started/python/first_program_in_python/
Temporal’s uvloop Compatibility: Use with Caution for Activities Only
Based on the documentation:
- Temporal workflows require a custom, durable asyncio loop and are incompatible with uvloop.
- For activities, however, Temporal supports uvloop—provided it’s configured in isolation (e.g., via an
activity_executoror separate process).Make sure that the uvloop event loop policy is applied only for activity execution and not globally, which may inadvertently affect workflows.
examples/application_sql.py (1)
34-35: LGTM! Performance improvement with uvloop.The integration of
uvloopwill enhance the performance of asyncio operations in this script.Also applies to: 50-51
application_sdk/decorators/__init__.py (1)
8-8: LGTM! Performance improvement with uvloop.The integration of
uvloopwill enhance the performance of the transform decorators and their async operations.Also applies to: 15-15
examples/application_sql_with_custom_transformer.py (1)
35-35: LGTM! Performance improvement with uvloop.The integration of
uvloopwill enhance the performance of asyncio operations in this script.Also applies to: 56-56
examples/application_sql_miner.py (1)
25-26: LGTM! Performance improvement with uvloop.The integration of
uvloopwill enhance the performance of asyncio operations in this script.Also applies to: 36-36
tests/unit/clients/test_sql_client.py (2)
7-7: LGTM! The uvloop integration looks good.The integration of uvloop aligns with the PR objective of improving asyncio performance.
Also applies to: 12-12
5-5: Fix the pipeline failure by installing pandas.The pipeline is failing due to a missing pandas dependency. This needs to be addressed.
Run the following script to verify the pandas installation:
✅ Verification successful
🏁 Script executed:
#!/bin/bash # Description: Check if pandas is installed and install it if missing. # Check if pandas is installed pip list | grep pandas || pip install pandasLength of output: 1533
Pandas Dependency Issue Resolved
The shell script output confirms that pandas (and its dependencies) has been successfully installed, which should resolve the pipeline failure issue.
🧰 Tools
🪛 GitHub Actions: Tests
[error] 5-5: ModuleNotFoundError: No module named 'pandas'
tests/unit/clients/test_async_sql_client.py (1)
7-7: LGTM! The uvloop integration looks good.The integration of uvloop aligns with the PR objective of improving asyncio performance.
Also applies to: 12-12
application_sdk/outputs/parquet.py (1)
9-9: LGTM! The uvloop integration looks good.The integration of uvloop is particularly beneficial here as it can improve the performance of chunked writing operations to parquet files.
Also applies to: 16-16
pyproject.toml (1)
38-38: Verify the uvloop version.The version constraint ^0.21.0 allows for compatible updates. Let's verify if this is the latest stable version.
✅ Verification successful
🌐 Web query:
What is the latest stable version of uvloop?💡 Result:
The latest stable version of uvloop is v0.21.0, released on October 14, 2024[1][3]. This update includes:
- Python 3.13 support[1]
- Bundled libuv upgraded to version 1.43.0[1]
- Multiple bug fixes and improvements:
- UDP error handling improvements[1]
- Cython3 compatibility updates[1]
- Enhanced debug mode activation with
-X dev[1]Key installation command:
pip install uvloopThe package supports Python versions 3.8+ and provides pre-built wheels for multiple platforms including Linux (musl/glibc), macOS (x86_64/universal2), and various architectures[3]. For Python 3.11+ users, the recommended usage pattern is through
uvloop.run(main())[3].Citations:
- 1: https://github.com/MagicStack/uvloop/releases
- 2: https://uvloop.readthedocs.io
- 3: https://pypi.org/project/uvloop/
- 4: https://readthedocs.org/projects/uvloop/
- 5: [feature request] add loop._run_once and loop._set_is_runnning api to easy the process of OS gui event loop integrate MagicStack/uvloop#648
- 6: https://www.uvicorn.org
- 7: https://sourceforge.net/projects/uvloop.mirror/
- 8: https://pypi.org/project/uvloop/0.9.1/
uvloop Version Verified: The version specified as
^0.21.0in thepyproject.tomlfile correctly refers to the latest stable version. According to the web query results, v0.21.0 is the current stable release, so no update is necessary.
Comment on lines
+11
to
12
| import uvloop | ||
| from temporalio.types import CallableType |
There was a problem hiding this comment.
Fix potential thread safety issues with uvloop initialization.
Setting the event loop policy at module level can lead to race conditions when using threading. The worker creates daemon threads that run their own event loops, which could conflict with the global policy setting.
Move the uvloop initialization into the worker thread:
-asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
class Worker:
async def start(self, daemon: bool = False, *args: Any, **kwargs: Any) -> None:
if daemon:
worker_thread = threading.Thread(
- target=lambda: asyncio.run(self.start(daemon=False)), daemon=True
+ target=lambda: self._run_worker_with_uvloop(), daemon=True
)
worker_thread.start()
return
+ def _run_worker_with_uvloop(self):
+ asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
+ asyncio.run(self.start(daemon=False))Also applies to: 18-19
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
tests/unit/common/test_event_loop.py (2)
18-32: Consider adding more comprehensive assertions.While the test correctly verifies the event loop class, consider adding assertions for:
- The event loop module name
- The loop's running state
def test_default_event_loop(self): """Test the default event loop implementation being used""" loop = asyncio.new_event_loop() loop_class = loop.__class__.__name__ + loop_module = loop.__class__.__module__ print(f"\nDefault event loop implementation: {loop_class}") - print(f"Event loop module: {loop.__class__.__module__}") + print(f"Event loop module: {loop_module}") if sys.platform == 'win32': self.assertEqual(loop_class, 'ProactorEventLoop') + self.assertEqual(loop_module, 'asyncio.windows_events') else: self.assertEqual(loop_class, 'SelectorEventLoop') + self.assertEqual(loop_module, 'asyncio.events') + + self.assertFalse(loop.is_running()) loop.close()
54-82: Enhance performance test robustness.Consider these improvements to make the performance test more reliable:
- Add assertions to verify minimum performance improvement
- Make the number of iterations configurable
- Consider environment variations
def test_event_loop_performance(self): """Compare performance between default loop and uvloop""" import time + + # Configuration + MIN_IMPROVEMENT_PERCENT = 5 + ITERATIONS = 1000 if not sys.getenv('PERF_TEST_ITERATIONS') else int(sys.getenv('PERF_TEST_ITERATIONS')) # Test with default event loop asyncio.set_event_loop_policy(asyncio.DefaultEventLoopPolicy()) loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) start_time = time.time() - loop.run_until_complete(asyncio.gather(*[self.async_operation() for _ in range(1000)])) + loop.run_until_complete(asyncio.gather(*[self.async_operation() for _ in range(ITERATIONS)])) default_time = time.time() - start_time loop.close() # Test with uvloop uvloop.install() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) start_time = time.time() - loop.run_until_complete(asyncio.gather(*[self.async_operation() for _ in range(1000)])) + loop.run_until_complete(asyncio.gather(*[self.async_operation() for _ in range(ITERATIONS)])) uvloop_time = time.time() - start_time loop.close() improvement_percent = ((default_time - uvloop_time) / default_time) * 100 print("\nPerformance comparison:") print(f"Default event loop time: {default_time:.4f} seconds") print(f"uvloop time: {uvloop_time:.4f} seconds") - print(f"Speed improvement: {((default_time - uvloop_time) / default_time) * 100:.2f}%") + print(f"Speed improvement: {improvement_percent:.2f}%") + + # Assert minimum performance improvement + self.assertGreater(improvement_percent, MIN_IMPROVEMENT_PERCENT, + f"uvloop should be at least {MIN_IMPROVEMENT_PERCENT}% faster than the default loop")🧰 Tools
🪛 Ruff (0.8.2)
78-78: f-string without any placeholders
Remove extraneous
fprefix(F541)
🪛 GitHub Actions: Pre-commit Checks
[error] 78-78: f-string without any placeholders. Remove extraneous
fprefix.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/unit/common/test_event_loop.py(1 hunks)
🧰 Additional context used
🪛 Ruff (0.8.2)
tests/unit/common/test_event_loop.py
78-78: f-string without any placeholders
Remove extraneous f prefix
(F541)
93-93: Local variable app is assigned to but never used
Remove assignment to unused variable app
(F841)
🪛 GitHub Actions: Pre-commit Checks
tests/unit/common/test_event_loop.py
[error] 78-78: f-string without any placeholders. Remove extraneous f prefix.
[error] 93-93: Local variable app is assigned to but never used. Remove assignment to unused variable app.
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: test
- GitHub Check: generate-docs
🔇 Additional comments (3)
tests/unit/common/test_event_loop.py (3)
1-17: LGTM! Well-structured test setup.The imports are appropriate, and the test class properly manages the event loop policy state through setUp and tearDown methods, ensuring test isolation.
33-48: LGTM! Comprehensive uvloop verification.The test thoroughly verifies the uvloop implementation by checking both the loop class and module, with proper resource cleanup.
111-112: LGTM! Standard unittest main block.The main block is correctly implemented for running the tests.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
tests/unit/common/test_event_loop.py (2)
6-7: Remove redundant empty line after imports.The double empty line after imports is unnecessary and should be removed to maintain consistent spacing.
import psutil import uvloop - class TestEventLoop(unittest.TestCase):
37-41: Make async_operation more configurable for various test scenarios.The fixed sleep duration of 0.1 seconds might be too long for performance testing with 1M iterations. Consider making it configurable.
async def async_operation(self): """Sample async operation for performance testing""" - await asyncio.sleep(0.1) + await asyncio.sleep(0.001) # Use smaller default sleep time return "done"Consider making it more flexible:
async def async_operation(self, sleep_time=0.001): """Sample async operation for performance testing Args: sleep_time (float): Time to sleep in seconds """ await asyncio.sleep(sleep_time) return "done"
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/unit/common/test_event_loop.py(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: test
- GitHub Check: generate-docs
🔇 Additional comments (3)
tests/unit/common/test_event_loop.py (3)
9-16: LGTM! Clean setup and teardown implementation.The setup and teardown methods correctly manage the event loop policy state, ensuring tests don't interfere with each other.
88-88: Remove unnecessary f-string prefix.The string doesn't contain any placeholders, so the
fprefix is not needed.- print(f"\nPerformance comparison:") + print("\nPerformance comparison:")
96-98: LGTM! Standard unittest main block.The main block follows the standard unittest pattern.
TechyMT
changed the title
TechyMT
commented
Feb 11, 2025
| from temporalio.types import CallableType | ||
|
|
||
| from application_sdk.clients.temporal import TemporalClient | ||
| from application_sdk.common.logger_adaptors import get_logger | ||
|
|
||
| logger = get_logger(__name__) | ||
| asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) |
There was a problem hiding this comment.
Setting the event loop policy sets the event loop at all the asyncio instances in the application sdk and in the apps, hence decided to keep it here as worker is used in all the apps
|
@OnkarVO7 to review and get back |
inishchith
pushed a commit
that referenced
this pull request
Feb 26, 2025
* feat: integrate Loguru for enhanced logging functionality * refactor: improve logger adapter structure and enhance logging level mapping * feat: enhance logging output with color formatting for better readability * refactor: streamline logger format string definition for clarity * refactor: remove unnecessary blank lines in decorators module * refactor: remove unnecessary blank lines in decorators module * feat: enhance logging messages with color formatting for improved readability * refactor: simplify AtlanLoggerAdapter initialization by removing logger name parameter * refactor: update logging message formatting to use custom tags for Run ID * feat: enhance AtlanLoggerAdapter to support severity mapping and improve OTLP logging * refactor: remove unused regex import from logger_adaptors.py * refactor: update logger adapter tests to remove unnecessary 'extra' key usage * refactor: remove unnecessary 'extra' key assertions from logger adapter tests * refactor: remove redundant assertion comment from logger adapter test * refactor: enhance logger adapter tests with improved context handling and mock setup * refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Moved scale_data_generator to tests * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: update guide Signed-off-by: Junaid Rahim <[email protected]> * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class Signed-off-by: Junaid Rahim <[email protected]> * feat: added local server Signed-off-by: Junaid Rahim <[email protected]> * feat: added supported features matrix Signed-off-by: Junaid Rahim <[email protected]> * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <[email protected]> * ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5393 correct duplicate env (#218) * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * re-added scale data generator module in pyproject.toml --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]>
inishchith
added a commit
that referenced
this pull request
Mar 3, 2025
Squashed commit of the following: commit 584fc5c Author: Atlan CI <[email protected]> Date: Fri Feb 28 16:20:32 2025 +0530 chore(release): bump version (#247) commit 0fbe68d Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 28 16:19:39 2025 +0530 APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. commit 90ccf9e Author: Atlan CI <[email protected]> Date: Fri Feb 28 15:14:49 2025 +0530 chore(release): bump version (#246) commit d789fde Author: SanilK2108 <[email protected]> Date: Fri Feb 28 15:13:54 2025 +0530 Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours commit 2add6ac Author: Atlan CI <[email protected]> Date: Fri Feb 28 13:48:20 2025 +0530 chore(release): bump version (#244) commit 2f7287c Author: SanilK2108 <[email protected]> Date: Fri Feb 28 13:47:20 2025 +0530 fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- Co-authored-by: Nishchith Shetty <[email protected]> commit c8bafe8 Author: Atlan CI <[email protected]> Date: Fri Feb 28 12:01:32 2025 +0530 chore(release): bump version (#242) commit ccf19c6 Author: Ujala Singh <[email protected]> Date: Fri Feb 28 12:00:40 2025 +0530 fix: Trivy Action to ignore the unfixed vulnerabilities (#239) commit 5d8139d Author: Atlan CI <[email protected]> Date: Fri Feb 28 11:34:14 2025 +0530 chore(release): bump version (#240) commit ca3df39 Merge: 607c18b e4bb3e5 Author: Onkar Ravgan <[email protected]> Date: Fri Feb 28 11:32:59 2025 +0530 Merge pull request #210 from atlanhq/APP-5484-e2e-test-framework feat: APP-5484 [Integration Tests] E2E test framework commit e4bb3e5 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 22:41:09 2025 +0530 update the branch to develop commit 349a920 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 21:07:33 2025 +0530 rmv whitespace commit 49a65cc Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 20:59:41 2025 +0530 add event after for commit hash commit 4507afb Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 19:43:43 2025 +0530 echo commit hashes commit e36f898 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 19:22:07 2025 +0530 test6 commit ee04d1d Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 19:08:42 2025 +0530 checkout repo commit 08403d7 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 18:52:01 2025 +0530 check commit hash commit c5bddf6 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 17:38:51 2025 +0530 test3" " commit 904b8f2 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 17:35:18 2025 +0530 test2 commit 607c18b Author: Atlan CI <[email protected]> Date: Thu Feb 27 17:29:22 2025 +0530 chore(release): bump version (#238) commit 39c6b97 Author: Junaid Rahim <[email protected]> Date: Thu Feb 27 17:25:14 2025 +0530 ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> commit 3a27d54 Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 16:40:04 2025 +0530 fix poetry 3 commit cbec63a Merge: 47107ed b62a64a Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 16:39:31 2025 +0530 Merge remote-tracking branch 'origin/develop' into APP-5484-e2e-test-framework commit 47107ed Author: Onkar Ravgan <[email protected]> Date: Thu Feb 27 16:37:02 2025 +0530 use dispatch gh commit b62a64a Author: Atlan CI <[email protected]> Date: Thu Feb 27 16:07:03 2025 +0530 chore(release): bump version (#234) commit 07f2668 Author: Atlan CI <[email protected]> Date: Thu Feb 27 16:06:20 2025 +0530 chore(release): bump version (#233) commit fa83969 Author: Atlan CI <[email protected]> Date: Thu Feb 27 16:05:37 2025 +0530 chore(release): bump version (#232) commit 36083db Author: Junaid Rahim <[email protected]> Date: Thu Feb 27 16:04:50 2025 +0530 ci: APP-5393 correct commit command (#229) Signed-off-by: Junaid Rahim <[email protected]> commit 573cb3b Author: Atlan CI <[email protected]> Date: Thu Feb 27 15:53:47 2025 +0530 chore(release): bump version (#231) commit a222055 Author: Nishchith Shetty <[email protected]> Date: Wed Feb 26 22:12:31 2025 -0800 fix: ruff errors commit cd0e90a Author: Mustafa <[email protected]> Date: Thu Feb 27 11:29:33 2025 +0530 ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py commit 56f8acc Author: Junaid Rahim <[email protected]> Date: Thu Feb 27 10:56:32 2025 +0530 ci: APP-5393 correct commit command (#226) * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * ci: correct commit command Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> commit db3c560 Author: Nishchith Shetty <[email protected]> Date: Wed Feb 26 13:16:44 2025 -0800 fix: tests commit 8eca825 Merge: 3b32057 34d5f80 Author: Nishchith Shetty <[email protected]> Date: Wed Feb 26 13:15:03 2025 -0800 Merge branch 'main' into develop commit 3b32057 Author: Nishchith Shetty <[email protected]> Date: Wed Feb 26 13:11:55 2025 -0800 fix: custom attributes commit b37c8b1 Author: Onkar Ravgan <[email protected]> Date: Wed Feb 26 20:09:10 2025 +0530 rmv mkdocs commit f1c6a7a Author: Onkar Ravgan <[email protected]> Date: Wed Feb 26 20:07:33 2025 +0530 fix poetry file commit 75c201e Merge: 3864b75 1ec326c Author: Onkar Ravgan <[email protected]> Date: Wed Feb 26 20:06:42 2025 +0530 Merge remote-tracking branch 'origin/develop' into APP-5484-e2e-test-framework commit 1ec326c Author: Junaid Rahim <[email protected]> Date: Wed Feb 26 17:00:45 2025 +0530 ci: APP-5393 correct release commit user (#225) * ci: correct release commit user Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> commit 64bfca3 Merge: 4da0c26 ceabd72 Author: Onkar Ravgan <[email protected]> Date: Wed Feb 26 13:31:10 2025 +0530 Merge pull request #223 from atlanhq/change-deps re-added scale data generator module in pyproject.toml commit ceabd72 Author: Abhishek Agrawal <[email protected]> Date: Wed Feb 26 13:26:35 2025 +0530 re-added scale data generator module in pyproject.toml commit 4da0c26 Merge: cc47138 61c8f42 Author: Onkar Ravgan <[email protected]> Date: Wed Feb 26 12:17:20 2025 +0530 Merge pull request #221 from atlanhq/APP-5635 fix(docgen): APP-5635 make mkdocs dep optional commit 61c8f42 Author: Junaid Rahim <[email protected]> Date: Wed Feb 26 12:13:11 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit f6075a9 Author: Junaid Rahim <[email protected]> Date: Wed Feb 26 12:09:20 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit 0f18275 Author: Junaid Rahim <[email protected]> Date: Wed Feb 26 11:55:49 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit 3864b75 Author: Onkar Ravgan <[email protected]> Date: Tue Feb 25 20:37:18 2025 +0530 add mkdocs temporarily commit d708e19 Author: Onkar Ravgan <[email protected]> Date: Tue Feb 25 20:33:50 2025 +0530 update deps commit 05aee0d Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 19:05:38 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit a038e0a Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 19:02:16 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit 9d5745f Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 18:57:40 2025 +0530 fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <[email protected]> commit cc47138 Author: SanilK2108 <[email protected]> Date: Tue Feb 25 16:47:04 2025 +0530 App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes commit ab468eb Author: Onkar Ravgan <[email protected]> Date: Tue Feb 25 15:45:12 2025 +0530 check api resp commit 1978143 Author: Onkar Ravgan <[email protected]> Date: Tue Feb 25 15:35:42 2025 +0530 fix poetry conflicts commit 69947a6 Merge: c1943a3 aef1071 Author: Onkar Ravgan <[email protected]> Date: Tue Feb 25 15:34:44 2025 +0530 fix conflicts commit c1943a3 Author: Onkar Ravgan <[email protected]> Date: Tue Feb 25 15:31:33 2025 +0530 add gh action to trigger workflow from sdk commit aef1071 Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 15:05:52 2025 +0530 ci: APP-5393 correct duplicate env (#218) commit 31a5233 Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 15:01:47 2025 +0530 ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <[email protected]> commit 0d99647 Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 14:23:55 2025 +0530 ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <[email protected]> commit ea13744 Author: Junaid Rahim <[email protected]> Date: Tue Feb 25 13:30:07 2025 +0530 feat: APP-5393 add semantic releases (#211) * feat: add semantic releases commit 2cba826 Author: Abhishek Agrawal <[email protected]> Date: Tue Feb 25 12:00:04 2025 +0530 APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. commit ff52c58 Merge: 58105f1 5b7308e Author: Onkar Ravgan <[email protected]> Date: Mon Feb 24 18:22:10 2025 +0530 Merge pull request #213 from atlanhq/APP-5585 fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander commit 5b7308e Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 17:48:31 2025 +0530 Made engine creation and connection under single try / Except commit 8c09f2e Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 16:40:24 2025 +0530 Removed sqlalchemy specific error handling alltogether commit 58105f1 Merge: 19d05ca 3b29b6f Author: Onkar Ravgan <[email protected]> Date: Mon Feb 24 16:16:46 2025 +0530 Merge pull request #207 from atlanhq/APP-4889 refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface commit 3b29b6f Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 16:01:30 2025 +0530 Pathed read_file and upload_file methods. commit 19d05ca Author: Junaid Rahim <[email protected]> Date: Mon Feb 24 15:42:06 2025 +0530 chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <[email protected]> commit 78b74b9 Author: Junaid Rahim <[email protected]> Date: Mon Feb 24 15:40:06 2025 +0530 feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup commit 9c94787 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 15:17:16 2025 +0530 Input will read from objectStore and Output will push to objectstore commit 2f731ea Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 14:20:39 2025 +0530 Instead of using string matches, used SQLAlchemy specific exceptions commit d5a078f Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 09:58:19 2025 +0530 Made the tests functional for testing parquet with pandas and daft both commit 28fa4a9 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 09:43:56 2025 +0530 renamed decorator from transform_pandas to transform commit 9649a8f Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 09:37:41 2025 +0530 Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. commit 97f69d0 Merge: 66c23e7 ce94a1b Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 24 09:01:43 2025 +0530 Merge branch 'develop' of github.com:atlanhq/application-sdk into APP-4889 commit ce94a1b Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 21 16:47:15 2025 +0530 APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” commit a896e7d Author: Onkar Ravgan <[email protected]> Date: Fri Feb 21 15:57:33 2025 +0530 improved code readability commit a0d71d3 Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 21 08:49:23 2025 +0530 converted exception handling into seperate function and used that in to handle exceptions. commit fbcf1bc Author: Mustafa <[email protected]> Date: Fri Feb 21 01:37:58 2025 +0530 fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization commit 9ab90ea Author: Onkar Ravgan <[email protected]> Date: Thu Feb 20 16:56:20 2025 +0530 update custom test commit 6f4a068 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 20 16:13:01 2025 +0530 fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client commit b3f3ae7 Author: Onkar Ravgan <[email protected]> Date: Wed Feb 19 20:03:09 2025 +0530 handle exceptions commit b66bb82 Author: Onkar Ravgan <[email protected]> Date: Wed Feb 19 17:56:48 2025 +0530 fix poetry commit 81b163a Author: Onkar Ravgan <[email protected]> Date: Wed Feb 19 17:13:23 2025 +0530 APP-5484 e2e test framework commit 66c23e7 Author: Abhishek Agrawal <[email protected]> Date: Wed Feb 19 11:16:58 2025 +0530 refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface commit e0efbdd Merge: d95f9f9 c9c7aa4 Author: Onkar Ravgan <[email protected]> Date: Wed Feb 19 10:48:54 2025 +0530 Merge pull request #201 from atlanhq/fix-paths fix: APP-5369 Add Cloud specific metadata for Output Bindings commit d95f9f9 Merge: 911a4fe d6fea5e Author: Onkar Ravgan <[email protected]> Date: Wed Feb 19 10:46:41 2025 +0530 Merge pull request #189 from atlanhq/stale-deps-cleanup Cleanup of stale dependencies in application-sdk commit 911a4fe Merge: 3827ff0 4c59203 Author: Onkar Ravgan <[email protected]> Date: Tue Feb 18 19:08:38 2025 +0530 Merge pull request #204 from atlanhq/feat/add-healthcheck-override-timeout feature: Allow apps to override heartbeat timeout commit c9c7aa4 Author: ujala-singh <[email protected]> Date: Tue Feb 18 17:08:33 2025 +0530 fix: Pre-Commit Check commit d6fea5e Author: Abhishek Agrawal <[email protected]> Date: Tue Feb 18 12:54:29 2025 +0530 Updated comments on pyproject.toml commit 0baac01 Author: Abhishek Agrawal <[email protected]> Date: Tue Feb 18 12:35:20 2025 +0530 Updated pyproject.toml - Moved boto3-stubs to dev-dependency commit 593e204 Author: Abhishek Agrawal <[email protected]> Date: Tue Feb 18 12:10:21 2025 +0530 fixed pre-commit checks commit 0c21ee8 Author: Abhishek Agrawal <[email protected]> Date: Tue Feb 18 12:08:04 2025 +0530 Updated pyproject.toml by adding comments commit b45ecd0 Author: Abhishek Agrawal <[email protected]> Date: Tue Feb 18 11:58:00 2025 +0530 Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy commit 01c4382 Author: ujala-singh <[email protected]> Date: Tue Feb 18 11:47:56 2025 +0530 chore: Update metadata payload commit 4c59203 Author: SanilK2108 <[email protected]> Date: Tue Feb 18 11:28:42 2025 +0530 feat: Allow apps to override heartbeat timeout commit 5f280c0 Author: ujala-singh <[email protected]> Date: Tue Feb 18 10:52:34 2025 +0530 chore: Update metadata payload commit 1c4b294 Author: ujala-singh <[email protected]> Date: Tue Feb 18 10:49:07 2025 +0530 chore: Update metadata payload commit ceeaa14 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 18:57:47 2025 +0530 Updated poetry.lock commit 89b2254 Merge: ff10413 3827ff0 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 18:56:39 2025 +0530 Merge branch 'develop' into stale-deps-cleanup commit 3827ff0 Merge: b90d60a ef27bac Author: Onkar Ravgan <[email protected]> Date: Mon Feb 17 18:19:18 2025 +0530 Merge pull request #200 from atlanhq/fix/trivy-snowflake-app Add input for skip-files commit b90d60a Merge: ceea14e c0df449 Author: Onkar Ravgan <[email protected]> Date: Mon Feb 17 18:14:04 2025 +0530 Merge pull request #181 from atlanhq/feat/loguru APP-5363 : Added Loguru for better logging commit ff10413 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 18:12:52 2025 +0530 boto3-stubs and boto3 are part of the same module commit ef27bac Author: SanilK2108 <[email protected]> Date: Mon Feb 17 18:12:14 2025 +0530 fix: required commit cff7d17 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 18:05:22 2025 +0530 made boto3-stubs as optional and as a module commit 2909774 Author: SanilK2108 <[email protected]> Date: Mon Feb 17 18:01:16 2025 +0530 debug commit 10a4aa5 Author: SanilK2108 <[email protected]> Date: Mon Feb 17 17:55:57 2025 +0530 debug commit 5ea4918 Author: SanilK2108 <[email protected]> Date: Mon Feb 17 17:54:33 2025 +0530 debug commit f526525 Author: SanilK2108 <[email protected]> Date: Mon Feb 17 17:52:56 2025 +0530 debug commit cf93bb1 Author: SanilK2108 <[email protected]> Date: Mon Feb 17 17:50:09 2025 +0530 debug commit c0df449 Author: Mustafa <[email protected]> Date: Mon Feb 17 17:43:49 2025 +0530 chore: update poetry.lock to remove unnecessary groups and markers commit baeddf6 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 17:43:38 2025 +0530 updated pyproject.toml commit 7936a15 Merge: 25c7879 ceea14e Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 17:40:27 2025 +0530 Merge branch 'develop' into stale-deps-cleanup commit 5648a53 Merge: 01385a7 ceea14e Author: Mustafa <[email protected]> Date: Mon Feb 17 17:38:43 2025 +0530 Merge branch 'develop' of https://github.com/atlanhq/application-sdk into feat/loguru commit 385c5ec Author: ujala-singh <[email protected]> Date: Mon Feb 17 17:34:49 2025 +0530 fix: Add Cloud specific metadata for Output Bindings commit 8642e7f Author: SanilK2108 <[email protected]> Date: Mon Feb 17 17:07:05 2025 +0530 fix: adding skip in the right places commit 5bedd6a Author: SanilK2108 <[email protected]> Date: Mon Feb 17 17:05:13 2025 +0530 debug commit 0115f29 Author: SanilK2108 <[email protected]> Date: Mon Feb 17 16:57:09 2025 +0530 debug commit ceea14e Merge: 2448dd9 46520c9 Author: Mrunmayi Tripathi <[email protected]> Date: Mon Feb 17 16:44:40 2025 +0530 Merge pull request #199 from atlanhq/interrogate-cleanup chore: APP-5028 cleanup interrogate commit 46520c9 Author: Junaid Rahim <[email protected]> Date: Mon Feb 17 16:26:20 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit 2448dd9 Merge: e12113a 1ef2a4b Author: Mrunmayi Tripathi <[email protected]> Date: Mon Feb 17 15:26:56 2025 +0530 Merge pull request #197 from atlanhq/APP-5028 fix: APP-5028 make docstring coverage action reusable commit 25c7879 Merge: e2a61be e12113a Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 15:20:55 2025 +0530 Merge branch 'develop' into stale-deps-cleanup commit 01385a7 Author: Mustafa <[email protected]> Date: Mon Feb 17 14:29:54 2025 +0530 No code changes made. commit e6e0ad4 Merge: 324d4e1 e12113a Author: Mustafa <[email protected]> Date: Mon Feb 17 14:25:58 2025 +0530 Merge branch 'develop' of https://github.com/atlanhq/application-sdk into feat/loguru commit e12113a Merge: d634aec a348d3a Author: Mrunmayi Tripathi <[email protected]> Date: Mon Feb 17 12:54:21 2025 +0530 Merge pull request #190 from atlanhq/APP-5320 feat(docs): APP-5320 add driver class commit d634aec Merge: 67d51ef 7f79397 Author: Mrunmayi Tripathi <[email protected]> Date: Mon Feb 17 12:49:02 2025 +0530 Merge pull request #196 from atlanhq/APP-5495 fix: APP-5495 stale PRs GitHub Action does not seem to be working commit 324d4e1 Author: Mustafa <[email protected]> Date: Mon Feb 17 12:14:35 2025 +0530 refactor: update GitHub Actions workflow to improve test coverage reporting commit 60b3dc7 Author: Mustafa <[email protected]> Date: Mon Feb 17 12:13:22 2025 +0530 refactor: enable loading of cached virtual environment in GitHub Actions workflow commit 27df9b0 Author: Mustafa <[email protected]> Date: Mon Feb 17 12:09:31 2025 +0530 refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter commit a348d3a Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 15:35:48 2025 +0530 feat: added supported features matrix Signed-off-by: Junaid Rahim <[email protected]> commit db5e3fc Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 15:07:51 2025 +0530 feat: added local server Signed-off-by: Junaid Rahim <[email protected]> commit cd0c3c3 Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 14:44:27 2025 +0530 feat(docs): add driver class Signed-off-by: Junaid Rahim <[email protected]> commit 8488f44 Author: Mustafa <[email protected]> Date: Mon Feb 17 12:03:06 2025 +0530 refactor: add print statements in logger adapter tests for debugging purposes commit d5a5215 Author: Mustafa <[email protected]> Date: Mon Feb 17 11:54:52 2025 +0530 refactor: simplify expected message formatting in logger adapter tests for improved readability commit 7f79397 Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 10:51:51 2025 +0530 Modified the cron to run daily instead of weekly commit a2f921d Author: Abhishek Agrawal <[email protected]> Date: Mon Feb 17 10:44:29 2025 +0530 APP-5495 stale prs github action does not seem to be working - Made the workflow callable commit d0936b9 Author: Mustafa <[email protected]> Date: Fri Feb 14 19:48:33 2025 +0530 refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity commit b2bf67b Author: Mustafa <[email protected]> Date: Fri Feb 14 19:38:39 2025 +0530 refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness commit 670bf75 Author: Mustafa <[email protected]> Date: Fri Feb 14 19:27:39 2025 +0530 refactor: update logging format in SQLClient for improved string interpolation commit 8a0e10e Author: Mustafa <[email protected]> Date: Fri Feb 14 15:21:28 2025 +0530 refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity commit 4e1f7d5 Author: Mustafa <[email protected]> Date: Fri Feb 14 15:16:35 2025 +0530 refactor: update logging format in SQLClient for improved string interpolation commit c1f2ce9 Author: Mustafa <[email protected]> Date: Fri Feb 14 15:04:17 2025 +0530 refactor: update logging format in FastAPIApplication for improved readability commit b8ce510 Author: Mustafa <[email protected]> Date: Fri Feb 14 14:20:07 2025 +0530 refactor: comment out condition for cached dependencies installation in GitHub Actions workflow commit c0cb33d Author: Mustafa <[email protected]> Date: Fri Feb 14 14:19:52 2025 +0530 refactor: comment out cached venv loading in GitHub Actions workflow commit 8564174 Author: Mustafa <[email protected]> Date: Fri Feb 14 14:16:52 2025 +0530 refactor: improve error logging in AtlasTransformer for better context commit f759e0f Author: Mustafa <[email protected]> Date: Fri Feb 14 14:10:02 2025 +0530 refactor: update test command to use quiet mode and short traceback format commit 7a2655e Author: Mustafa <[email protected]> Date: Fri Feb 14 14:00:45 2025 +0530 refactor: update logger adapter test to use run_id instead of workflow_run_id commit 93c3c3d Author: Mustafa <[email protected]> Date: Fri Feb 14 13:55:19 2025 +0530 refactor: update expected logger messages to use dynamic workflow and activity context variables commit e2a61be Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 14 13:47:28 2025 +0530 Updating pyproject.toml commit 4eb544d Author: Mustafa <[email protected]> Date: Fri Feb 14 13:47:00 2025 +0530 test: add unit tests for AtlanLoggerAdapter and logger functions commit d5ba82f Merge: 54919a0 67d51ef Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 14 13:40:34 2025 +0530 Merge branch 'develop' into stale-deps-cleanup commit 0350b5a Author: Mustafa <[email protected]> Date: Fri Feb 14 13:37:06 2025 +0530 refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter commit ad71d66 Author: Mustafa <[email protected]> Date: Fri Feb 14 13:15:29 2025 +0530 refactor: replace logging with loguru in logger adaptors and directory parser commit d5866c9 Merge: bb40924 67d51ef Author: Mustafa <[email protected]> Date: Fri Feb 14 12:57:58 2025 +0530 Merge branch 'develop' of https://github.com/atlanhq/application-sdk into feat/loguru commit bb40924 Author: Mustafa <[email protected]> Date: Fri Feb 14 12:23:22 2025 +0530 refactor: update expected logger messages to include detailed workflow and activity context commit 4f9ab12 Author: Mustafa <[email protected]> Date: Fri Feb 14 12:13:50 2025 +0530 refactor: simplify expected message in logger tests by removing context details commit 73831cc Author: Mustafa <[email protected]> Date: Fri Feb 14 12:01:02 2025 +0530 refactor: enhance logger test messages with formatted workflow and activity context commit 67d51ef Author: SanilK2108 <[email protected]> Date: Fri Feb 14 12:00:46 2025 +0530 APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring commit e069d22 Author: Ujala Singh <[email protected]> Date: Fri Feb 14 12:00:26 2025 +0530 APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check commit b261f18 Merge: 0a8c0c3 45ee41f Author: Mustafa <[email protected]> Date: Fri Feb 14 11:52:24 2025 +0530 Merge branch 'develop' of https://github.com/atlanhq/application-sdk into feat/loguru commit 45ee41f Author: Mustafa <[email protected]> Date: Fri Feb 14 11:41:38 2025 +0530 APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports commit 0a8c0c3 Author: Mustafa <[email protected]> Date: Fri Feb 14 11:10:15 2025 +0530 refactor: update logger format string for improved clarity commit ffd3acf Author: Mustafa <[email protected]> Date: Fri Feb 14 11:07:48 2025 +0530 refactor: remove redundant logging from SQL query fetch method commit 54919a0 Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 14 10:57:33 2025 +0530 made numpy an optional test dependency. commit 1ef4a48 Author: Abhishek Agrawal <[email protected]> Date: Fri Feb 14 10:55:06 2025 +0530 Updated pyproject.toml wrt PR comments. commit cf70201 Merge: 2f48ea5 2f6f2de Author: Mrunmayi Tripathi <[email protected]> Date: Thu Feb 13 18:26:13 2025 +0530 Merge pull request #191 from atlanhq/conventional-commits feat: APP-5391 enforce conventional commits and semantic PR title commit 82629bc Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 18:07:36 2025 +0530 added back pylint commit 80bca90 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 18:03:54 2025 +0530 Added back pydata-sphinx-theme commit 75d8dae Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 18:01:07 2025 +0530 Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" commit 2f6f2de Merge: 581325c 2f48ea5 Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 17:57:11 2025 +0530 Merge branch 'develop' into conventional-commits commit 2f48ea5 Merge: 5938b1c 8220470 Author: Mrunmayi Tripathi <[email protected]> Date: Thu Feb 13 17:55:19 2025 +0530 Merge pull request #185 from atlanhq/APP-5069 APP-5069 Added pyright pre-commit checks to application-sdk commit 11fdc69 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 17:45:05 2025 +0530 Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser commit 581325c Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 17:38:13 2025 +0530 chore: update guide Signed-off-by: Junaid Rahim <[email protected]> commit b6d69c1 Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 17:28:32 2025 +0530 chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> commit 8b8ba68 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 17:24:23 2025 +0530 Moved scale_data_generator to tests commit 76f4185 Author: Junaid Rahim <[email protected]> Date: Thu Feb 13 17:22:52 2025 +0530 feat: enforce conventional commits Signed-off-by: Junaid Rahim <[email protected]> commit 35a9e25 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 17:22:12 2025 +0530 update poetry.lock file commit 26d5305 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 17:20:02 2025 +0530 Removed otel-deps, numpy and pyyaml commit 5938b1c Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 14:20:03 2025 +0530 APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. commit afb34c0 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 14:18:26 2025 +0530 APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] commit 8220470 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 13:22:57 2025 +0530 Added a new-line at EOF for .pre-commit-config.yaml commit 1714169 Author: Abhishek Agrawal <[email protected]> Date: Thu Feb 13 12:33:11 2025 +0530 cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml commit e2633f9 Author: Abhishek Agrawal <[email protected]> Date: Wed Feb 12 11:25:15 2025 +0530 Used pyright version 1.1.393 which is the latest one. commit eefd9c2 Author: Abhishek Agrawal <[email protected]> Date: Wed Feb 12 11:16:56 2025 +0530 APP - 5069 : Added pyright pre-commit checks to application-sdk commit 9604a22 Merge: 177713f 80edec6 Author: Mustafa <[email protected]> Date: Tue Feb 11 23:42:40 2025 +0530 Merge branch 'feat/loguru' of https://github.com/atlanhq/application-sdk into feat/loguru commit 177713f Author: Mustafa <[email protected]> Date: Tue Feb 11 23:42:38 2025 +0530 refactor: update logger adapter test to use 'request_id' instead of 'extra' key commit 80edec6 Merge: 592cacf 02d0dbf Author: Mustafa <[email protected]> Date: Tue Feb 11 23:36:44 2025 +0530 Merge branch 'develop' into feat/loguru commit 592cacf Author: Mustafa <[email protected]> Date: Tue Feb 11 23:34:47 2025 +0530 refactor: enhance logger adapter tests with improved context handling and mock setup commit 9a1df4a Author: Mustafa <[email protected]> Date: Tue Feb 11 23:30:08 2025 +0530 refactor: remove redundant assertion comment from logger adapter test commit a5d8f44 Author: Mustafa <[email protected]> Date: Tue Feb 11 23:29:35 2025 +0530 refactor: remove unnecessary 'extra' key assertions from logger adapter tests commit 0aad505 Author: Mustafa <[email protected]> Date: Tue Feb 11 23:28:58 2025 +0530 refactor: update logger adapter tests to remove unnecessary 'extra' key usage commit 52e7615 Author: Mustafa <[email protected]> Date: Tue Feb 11 23:17:12 2025 +0530 refactor: remove unused regex import from logger_adaptors.py commit cfe3b88 Author: Mustafa <[email protected]> Date: Tue Feb 11 23:16:19 2025 +0530 feat: enhance AtlanLoggerAdapter to support severity mapping and improve OTLP logging commit 155f6a5 Author: Mustafa <[email protected]> Date: Tue Feb 11 16:07:28 2025 +0530 refactor: update logging message formatting to use custom tags for Run ID commit 3f3e29b Author: Mustafa <[email protected]> Date: Tue Feb 11 16:04:17 2025 +0530 refactor: simplify AtlanLoggerAdapter initialization by removing logger name parameter commit 7294cb8 Author: Mustafa <[email protected]> Date: Tue Feb 11 16:03:28 2025 +0530 feat: enhance logging messages with color formatting for improved readability commit e30180d Author: Mustafa <[email protected]> Date: Tue Feb 11 12:23:49 2025 +0530 refactor: remove unnecessary blank lines in decorators module commit 211ac5b Author: Mustafa <[email protected]> Date: Tue Feb 11 12:23:00 2025 +0530 refactor: remove unnecessary blank lines in decorators module commit 7811055 Author: Mustafa <[email protected]> Date: Tue Feb 11 09:37:08 2025 +0530 refactor: streamline logger format string definition for clarity commit 7b9304c Author: Mustafa <[email protected]> Date: Tue Feb 11 09:30:43 2025 +0530 feat: enhance logging output with color formatting for better readability commit 5c9c0ae Author: Mustafa <[email protected]> Date: Mon Feb 10 17:59:44 2025 +0530 refactor: improve logger adapter structure and enhance logging level mapping commit 11727f6 Author: Mustafa <[email protected]> Date: Mon Feb 10 17:52:00 2025 +0530 feat: integrate Loguru for enhanced logging functionality
inishchith
added a commit
that referenced
this pull request
Mar 4, 2025
* feat: integrate Loguru for enhanced logging functionality * refactor: improve logger adapter structure and enhance logging level mapping * feat: enhance logging output with color formatting for better readability * refactor: streamline logger format string definition for clarity * refactor: remove unnecessary blank lines in decorators module * refactor: remove unnecessary blank lines in decorators module * feat: enhance logging messages with color formatting for improved readability * refactor: simplify AtlanLoggerAdapter initialization by removing logger name parameter * refactor: update logging message formatting to use custom tags for Run ID * feat: enhance AtlanLoggerAdapter to support severity mapping and improve OTLP logging * refactor: remove unused regex import from logger_adaptors.py * refactor: update logger adapter tests to remove unnecessary 'extra' key usage * refactor: remove unnecessary 'extra' key assertions from logger adapter tests * refactor: remove redundant assertion comment from logger adapter test * refactor: enhance logger adapter tests with improved context handling and mock setup * refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Moved scale_data_generator to tests * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: update guide Signed-off-by: Junaid Rahim <[email protected]> * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class Signed-off-by: Junaid Rahim <[email protected]> * feat: added local server Signed-off-by: Junaid Rahim <[email protected]> * feat: added supported features matrix Signed-off-by: Junaid Rahim <[email protected]> * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <[email protected]> * ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * update deps * add mkdocs temporarily * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * ci: correct commit command Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#238) * test2 * test3" " * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- Co-authored-by: Nishchith Shetty <[email protected]> * chore(release): bump version (#244) * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Onkar Ravgan <[email protected]> Co-authored-by: Atlan CI <[email protected]>
inishchith
added a commit
that referenced
this pull request
Mar 6, 2025
* refactor: enhance logger adapter tests with improved context handling and mock setup * refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Moved scale_data_generator to tests * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: update guide Signed-off-by: Junaid Rahim <[email protected]> * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class Signed-off-by: Junaid Rahim <[email protected]> * feat: added local server Signed-off-by: Junaid Rahim <[email protected]> * feat: added supported features matrix Signed-off-by: Junaid Rahim <[email protected]> * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * feat: agents * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <[email protected]> * ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * update deps * add mkdocs temporarily * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * ci: correct commit command Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * feat: interface for agents * fix: req, langgraph agent, loggers * fix: rename agent files * fix: lock dependencies * fix: pre-commit formatting * fix: langgraph agent * fix: lock dependencies * docs: added example on how to define and use langgraph agent * feat: created the langgraphagent related workflow, activities, 2. some changes for temporal.py to pass in workflow args properly * chore: examples updated * fix: remove poetry.lock * feat: tests are added for the agentic changes made to sdk * fix: gh actions to install the deps * fix: circular import problem * fix: deps install if deps are added in pyproject.toml * fix: deps duplication * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#238) * test2 * test3" " * chore: poetry lock * chore: lock updates * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- Co-authored-by: Nishchith Shetty <[email protected]> * chore(release): bump version (#244) * feat: fastapi app added for the agent * fix: lock deps * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * feat: moved everything realted to agent workflow into the code sdk * chore: added examples, and readme to init the fastapiagentapp * fix: poetry lock * fix: failing tests, types * fix: agent type issues, validation, improve error message, llm import * fix: prefix / added in the fastapiagentapp * fix: langgraph related imports * chore: update readme for the agents * fix: error msgs, dependencies, readme * fix: api endpoint, readme example * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) * chore(release): bump version (#260) * fix: lock deps * chore(release): bump version (#264) * feat(docgen): APP-5643 add sql parsing (#237) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#267) * fix: Make e2e test base branch dynamic (#266) * chore(release): bump version (#268) * docs: added decorator doc strings and examples (#265) * chore(release): bump version (#269) * docs: APP-5752 setup mkdocs with pydoctor (#263) * chore: cleanup * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#270) * fix(docs): APP-5272 correct docstrings for decorators (#271) * fix(docs): correct docstrings for decorators * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#272) * fix: heartbeat, timeout configurable * chore(release): bump version (#275) * fix: usage of workflow_config and workflow args (#276) * chore(release): bump version (#277) --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Hrushikesh Dokala <[email protected]> Co-authored-by: Nishchith Shetty <[email protected]> Co-authored-by: Atlan CI <[email protected]> Co-authored-by: Amit Prabhu <[email protected]>
inishchith
added a commit
that referenced
this pull request
Mar 6, 2025
* refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Moved scale_data_generator to tests * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: update guide Signed-off-by: Junaid Rahim <[email protected]> * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class Signed-off-by: Junaid Rahim <[email protected]> * feat: added local server Signed-off-by: Junaid Rahim <[email protected]> * feat: added supported features matrix Signed-off-by: Junaid Rahim <[email protected]> * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * feat: agents * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <[email protected]> * ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * update deps * add mkdocs temporarily * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * ci: correct commit command Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * feat: interface for agents * fix: req, langgraph agent, loggers * fix: rename agent files * fix: lock dependencies * fix: pre-commit formatting * fix: langgraph agent * fix: lock dependencies * docs: added example on how to define and use langgraph agent * feat: created the langgraphagent related workflow, activities, 2. some changes for temporal.py to pass in workflow args properly * chore: examples updated * fix: remove poetry.lock * feat: tests are added for the agentic changes made to sdk * fix: gh actions to install the deps * fix: circular import problem * fix: deps install if deps are added in pyproject.toml * fix: deps duplication * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#238) * test2 * test3" " * chore: poetry lock * chore: lock updates * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- Co-authored-by: Nishchith Shetty <[email protected]> * chore(release): bump version (#244) * feat: fastapi app added for the agent * fix: lock deps * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * feat: moved everything realted to agent workflow into the code sdk * chore: added examples, and readme to init the fastapiagentapp * fix: poetry lock * fix: failing tests, types * fix: agent type issues, validation, improve error message, llm import * fix: prefix / added in the fastapiagentapp * fix: langgraph related imports * chore: update readme for the agents * fix: error msgs, dependencies, readme * fix: api endpoint, readme example * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) * chore(release): bump version (#260) * fix: lock deps * chore(release): bump version (#264) * feat(docgen): APP-5643 add sql parsing (#237) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#267) * fix: Make e2e test base branch dynamic (#266) * chore(release): bump version (#268) * docs: added decorator doc strings and examples (#265) * chore(release): bump version (#269) * docs: APP-5752 setup mkdocs with pydoctor (#263) * chore: cleanup * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#270) * fix(docs): APP-5272 correct docstrings for decorators (#271) * fix(docs): correct docstrings for decorators * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#272) * fix: heartbeat, timeout configurable * chore(release): bump version (#275) * fix: usage of workflow_config and workflow args (#276) * chore(release): bump version (#277) * fix: langchain-openai dependency (#282) --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Onkar Ravgan <[email protected]> Co-authored-by: Hrushikesh Dokala <[email protected]> Co-authored-by: Atlan CI <[email protected]> Co-authored-by: Amit Prabhu <[email protected]>
inishchith
added a commit
that referenced
this pull request
Mar 12, 2025
* refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits * Moved scale_data_generator to tests * chore: cleanup * chore: update guide * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class * feat: added local server * feat: added supported features matrix * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * feat: agents * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) * ci: update secret for releases (#216) (#217) * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional * chore: cleanup * chore: cleanup * update deps * add mkdocs temporarily * chore: cleanup * chore: cleanup * chore: cleanup * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user * chore: cleanup * chore: cleanup --------- * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup * chore: cleanup * ci: correct commit command --------- * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * feat: interface for agents * fix: req, langgraph agent, loggers * fix: rename agent files * fix: lock dependencies * fix: pre-commit formatting * fix: langgraph agent * fix: lock dependencies * docs: added example on how to define and use langgraph agent * feat: created the langgraphagent related workflow, activities, 2. some changes for temporal.py to pass in workflow args properly * chore: examples updated * fix: remove poetry.lock * feat: tests are added for the agentic changes made to sdk * fix: gh actions to install the deps * fix: circular import problem * fix: deps install if deps are added in pyproject.toml * fix: deps duplication * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases * chore: cleanup --------- * chore(release): bump version (#238) * test2 * test3" " * chore: poetry lock * chore: lock updates * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- * chore(release): bump version (#244) * feat: fastapi app added for the agent * fix: lock deps * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * feat: moved everything realted to agent workflow into the code sdk * chore: added examples, and readme to init the fastapiagentapp * fix: poetry lock * fix: failing tests, types * fix: agent type issues, validation, improve error message, llm import * fix: prefix / added in the fastapiagentapp * fix: langgraph related imports * chore: update readme for the agents * fix: error msgs, dependencies, readme * fix: api endpoint, readme example * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) * chore(release): bump version (#260) * fix: lock deps * chore(release): bump version (#264) * feat(docgen): APP-5643 add sql parsing (#237) * chore(release): bump version (#267) * fix: Make e2e test base branch dynamic (#266) * chore(release): bump version (#268) * docs: added decorator doc strings and examples (#265) * chore(release): bump version (#269) * docs: APP-5752 setup mkdocs with pydoctor (#263) * chore: cleanup * chore: cleanup * chore: cleanup * chore: cleanup --------- * chore(release): bump version (#270) * fix(docs): APP-5272 correct docstrings for decorators (#271) * fix(docs): correct docstrings for decorators * chore: cleanup * chore: cleanup * chore: cleanup --------- * chore(release): bump version (#272) * fix: heartbeat, timeout configurable * chore(release): bump version (#275) * fix: usage of workflow_config and workflow args (#276) * chore(release): bump version (#277) * fix: langchain-openai dependency (#282) --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Onkar Ravgan <[email protected]> Co-authored-by: Hrushikesh Dokala <[email protected]> Co-authored-by: Atlan CI <[email protected]> Co-authored-by: Amit Prabhu <[email protected]>
inishchith
added a commit
that referenced
this pull request
Mar 12, 2025
* refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits * Moved scale_data_generator to tests * chore: cleanup * chore: update guide * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class * feat: added local server * feat: added supported features matrix * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * feat: agents * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) * ci: update secret for releases (#216) (#217) * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional * chore: cleanup * chore: cleanup * update deps * add mkdocs temporarily * chore: cleanup * chore: cleanup * chore: cleanup * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user * chore: cleanup * chore: cleanup --------- * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup * chore: cleanup * ci: correct commit command --------- * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * feat: interface for agents * fix: req, langgraph agent, loggers * fix: rename agent files * fix: lock dependencies * fix: pre-commit formatting * fix: langgraph agent * fix: lock dependencies * docs: added example on how to define and use langgraph agent * feat: created the langgraphagent related workflow, activities, 2. some changes for temporal.py to pass in workflow args properly * chore: examples updated * fix: remove poetry.lock * feat: tests are added for the agentic changes made to sdk * fix: gh actions to install the deps * fix: circular import problem * fix: deps install if deps are added in pyproject.toml * fix: deps duplication * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases * chore: cleanup --------- * chore(release): bump version (#238) * test2 * test3" " * chore: poetry lock * chore: lock updates * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- * chore(release): bump version (#244) * feat: fastapi app added for the agent * fix: lock deps * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * feat: moved everything realted to agent workflow into the code sdk * chore: added examples, and readme to init the fastapiagentapp * fix: poetry lock * fix: failing tests, types * fix: agent type issues, validation, improve error message, llm import * fix: prefix / added in the fastapiagentapp * fix: langgraph related imports * chore: update readme for the agents * fix: error msgs, dependencies, readme * fix: api endpoint, readme example * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) * chore(release): bump version (#260) * fix: lock deps * chore(release): bump version (#264) * feat(docgen): APP-5643 add sql parsing (#237) * chore(release): bump version (#267) * fix: Make e2e test base branch dynamic (#266) * chore(release): bump version (#268) * docs: added decorator doc strings and examples (#265) * chore(release): bump version (#269) * docs: APP-5752 setup mkdocs with pydoctor (#263) * chore: cleanup * chore: cleanup * chore: cleanup * chore: cleanup --------- * chore(release): bump version (#270) * fix(docs): APP-5272 correct docstrings for decorators (#271) * fix(docs): correct docstrings for decorators * chore: cleanup * chore: cleanup * chore: cleanup --------- * chore(release): bump version (#272) * fix: heartbeat, timeout configurable * chore(release): bump version (#275) * fix: usage of workflow_config and workflow args (#276) * chore(release): bump version (#277) * fix: langchain-openai dependency (#282) --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Onkar Ravgan <[email protected]> Co-authored-by: Hrushikesh Dokala <[email protected]> Co-authored-by: Atlan CI <[email protected]> Co-authored-by: Amit Prabhu <[email protected]>
inishchith
added a commit
that referenced
this pull request
Mar 12, 2025
* develop -> main (#283) * refactor: update logger adapter test to use 'request_id' instead of 'extra' key * APP - 5069 : Added pyright pre-commit checks to application-sdk * Used pyright version 1.1.393 which is the latest one. * cleanup of stale dependencies in application-sdk - Removed alembic, fastapi-sqlalchemy, aiofiles, pygwalker from the pyproject.toml * Added a new-line at EOF for .pre-commit-config.yaml * APP-5418 : trivy github action should add comment to the PR and block the PR merge (#187) * APP-5418 : trivy github action should add comment to the PR and block the PR merge - Modified the flow so that trivy github action adds comments to the PR, block merging of that PR and uploads the scan details to Github Security tab too. * Updated aquasecurity/[email protected] to aquasecurity/[email protected] * APP-4989 : Rename `dapr_router` (#188) - dapr router is renamed to pubsub_router since it only handled subscription to events via dapr. * Removed otel-deps, numpy and pyyaml * update poetry.lock file * feat: enforce conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Moved scale_data_generator to tests * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: update guide Signed-off-by: Junaid Rahim <[email protected]> * Made pyarrow an optional dependency, removed pydata-sphinx-theme, pylint and myst-parser * Added back myst-parser as it was being used by "Build Docs and Push to Kryptonite / generate-docs" * Added back pydata-sphinx-theme * added back pylint * Updated pyproject.toml wrt PR comments. * made numpy an optional test dependency. * refactor: remove redundant logging from SQL query fetch method * refactor: update logger format string for improved clarity * APP-5362 : feat: integrate uvloop for improved asyncio performance (#179) * feat: integrate uvloop for improved asyncio performance * feat: add uvloop integration for improved asyncio performance across the application * refactor: remove uvloop integration from various files * test: add unit tests for event loop implementations and performance comparison * refactor: clean up whitespace and formatting in event loop tests * test: simplify event loop tests by removing unnecessary assertions and imports * refactor: add whitespace for improved readability in decorators module * refactor: remove obsolete event loop tests and related imports * APP-5369 feat: Add ENVs support for Dapr Components (#186) * feat: Add ENVs support for Dapr Components * fix: Pre-Commit Check * APP-5081 : fixed healthchecks (#182) * update: Adding configuration for max concurrent activities * fix: precommit * fix: unit tests * fix: Added health timeouts and health checks for transform batch * fix: Code refactoring * refactor: enhance logger test messages with formatted workflow and activity context * refactor: simplify expected message in logger tests by removing context details * refactor: update expected logger messages to include detailed workflow and activity context * refactor: replace logging with loguru in logger adaptors and directory parser * refactor: remove outdated logger tests and fixtures for AtlanLoggerAdapter * test: add unit tests for AtlanLoggerAdapter and logger functions * Updating pyproject.toml * refactor: update expected logger messages to use dynamic workflow and activity context variables * refactor: update logger adapter test to use run_id instead of workflow_run_id * refactor: update test command to use quiet mode and short traceback format * refactor: improve error logging in AtlasTransformer for better context * refactor: comment out cached venv loading in GitHub Actions workflow * refactor: comment out condition for cached dependencies installation in GitHub Actions workflow * refactor: update logging format in FastAPIApplication for improved readability * refactor: update logging format in SQLClient for improved string interpolation * refactor: improve error logging format in SQLClient and AtlasTransformer for better clarity * refactor: update logging format in SQLClient for improved string interpolation * refactor: enhance logging context handling in AtlanLoggerAdapter for improved robustness * refactor: streamline workflow and activity context handling in AtlanLoggerAdapter for improved clarity * APP-5495 stale prs github action does not seem to be working - Made the workflow callable * Modified the cron to run daily instead of weekly * refactor: simplify expected message formatting in logger adapter tests for improved readability * refactor: add print statements in logger adapter tests for debugging purposes * feat(docs): add driver class Signed-off-by: Junaid Rahim <[email protected]> * feat: added local server Signed-off-by: Junaid Rahim <[email protected]> * feat: added supported features matrix Signed-off-by: Junaid Rahim <[email protected]> * refactor: clean up error logging and improve message formatting in SQLClient and AtlanLoggerAdapter * refactor: enable loading of cached virtual environment in GitHub Actions workflow * refactor: update GitHub Actions workflow to improve test coverage reporting * No code changes made. * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * debug * debug * fix: adding skip in the right places * fix: Add Cloud specific metadata for Output Bindings * updated pyproject.toml * chore: update poetry.lock to remove unnecessary groups and markers * debug * debug * debug * debug * debug * made boto3-stubs as optional and as a module * fix: required * boto3-stubs and boto3 are part of the same module * Updated poetry.lock * chore: Update metadata payload * chore: Update metadata payload * feat: Allow apps to override heartbeat timeout * chore: Update metadata payload * Refactored pyproject.toml - Removed aiofiles - Made pyarrow an optional dependency - added pandas to dev dependency - Removed starlette-context - removed pyyaml - renamed boto3 module to iam_auth - removed pandas and pyarrorw from test dependency - Removed pygwalker and numpy * Updated pyproject.toml by adding comments * fixed pre-commit checks * Updated pyproject.toml - Moved boto3-stubs to dev-dependency * Updated comments on pyproject.toml * fix: Pre-Commit Check * feat: agents * refactor: APP-4889 Refactor `ChunkedObjectStoreWriterInterface` to the Inputs / Outputs interface * APP-5484 e2e test framework * fix poetry * handle exceptions * fix: APP-5585 Bug: Incorrect credentials inaccurate error message - Preflight check hander - Added error handling for sql_client.load() method - modified the metadata extraction activity to create and store handler before awaiting for the sql_client * update custom test * fix: APP-5438: Description/remarks causing diff (KIWI) (#203) * refactor: remove unnecessary JSON serialization in process_text function * refactor: improve test output format in CI workflow * fix: remove unnecessary JSON parsing in schema and table tests * refactor: update test descriptions and improve coverage command * fix: correct command syntax in CI workflow for running tests with coverage * refactor: enhance text processing by removing HTML tags and normalizing whitespace * refactor: streamline text processing by combining HTML tag removal and whitespace normalization * converted exception handling into seperate function and used that in to handle exceptions. * improved code readability * APP-5037 Add unit tests for tables and views (#209) - Added unit tests to validate this scenario : ”In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added.” * Changes to ParquetInputs and ParquetOutputs - Added decorator based tests for both the input and the output classes - Added a parquet Input class - Used native daft and pandas methods to implement the Input and the Output classes. * renamed decorator from transform_pandas to transform * Made the tests functional for testing parquet with pandas and daft both * Instead of using string matches, used SQLAlchemy specific exceptions * Input will read from objectStore and Output will push to objectstore * feat(docs): APP-5321 mount doc server with app * feat(docs): mount doc server with app * chore: cleanup * chore: APP-5393 create different action for conventional commits Signed-off-by: Junaid Rahim <[email protected]> * Pathed read_file and upload_file methods. * Removed sqlalchemy specific error handling alltogether * Made engine creation and connection under single try / Except * APP-5482 [Unit Tests] Improve coverage and maintain coverage report (#208) * Improvement : APP-5482 [Unit Tests] Improve coverage and maintain coverage report Added command for the coverage to omit tests folder Added command for the unit test step to fail if the coverage falls under 60 * APP-5037 Add unit tests for tables and views - Added a test to test_workflow.py which would test the following scenario : "In the current application-sdk, even if temp_table_regex is None, we still add a regex condition in our SQL statements - NOT TABLE_NAME RLIKE '^$'. With this change, if temp_table_regex is empty, then this condition would not be added." * Revert "APP-5037 Add unit tests for tables and views" This reverts commit 4392fda. * feat: APP-5393 add semantic releases (#211) * feat: add semantic releases * ci: update secret for releases (#216) Signed-off-by: Junaid Rahim <[email protected]> * ci: update secret for releases (#216) (#217) Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5393 correct duplicate env (#218) * add gh action to trigger workflow from sdk * fix poetry conflicts * check api resp * App-5256 Transformation time improvements (#193) * single assignment for columns * single assignment for all types * fix: transformed data didnt have attributes * fix(docgen): make mkdocs dep optional Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * update deps * add mkdocs temporarily * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * re-added scale data generator module in pyproject.toml * ci: APP-5393 correct release commit user (#225) * ci: correct release commit user Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * fix poetry file * rmv mkdocs * fix: custom attributes * fix: tests * ci: APP-5393 correct commit command (#226) * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * ci: correct commit command Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * ci: APP-5661 Added Resource Attributes in the otel logger (#219) * added resoruce attributes via the code * refactor: improve formatting and readability in logger_adaptors.py * refactor: enhance resource attribute parsing in logger_adaptors.py * refactor: improve attribute parsing logic in logger_adaptors.py * refactor: add error handling to resource attribute parsing in logger_adaptors.py * fix: correct typo in method name for resource attribute parsing in logger_adaptors.py * fix: ruff errors * feat: interface for agents * fix: req, langgraph agent, loggers * fix: rename agent files * fix: lock dependencies * fix: pre-commit formatting * fix: langgraph agent * fix: lock dependencies * docs: added example on how to define and use langgraph agent * feat: created the langgraphagent related workflow, activities, 2. some changes for temporal.py to pass in workflow args properly * chore: examples updated * fix: remove poetry.lock * feat: tests are added for the agentic changes made to sdk * fix: gh actions to install the deps * fix: circular import problem * fix: deps install if deps are added in pyproject.toml * fix: deps duplication * chore(release): bump version (#231) * ci: APP-5393 correct commit command (#229) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#232) * chore(release): bump version (#233) * chore(release): bump version (#234) * use dispatch gh * fix poetry 3 * ci: APP-5393 handle edge case for cascading releases (#236) * ci: handle edge case for cascading releases Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#238) * test2 * test3" " * chore: poetry lock * chore: lock updates * check commit hash * checkout repo * test6 * echo commit hashes * add event after for commit hash * rmv whitespace * update the branch to develop * chore(release): bump version (#240) * fix: Trivy Action to ignore the unfixed vulnerabilities (#239) * chore(release): bump version (#242) * fix: APP-5598 Health check timeout 1m assets (#243) * Removing max transform concurrency * fix: pin dapr CLI to 1.14.1 --------- Co-authored-by: Nishchith Shetty <[email protected]> * chore(release): bump version (#244) * feat: fastapi app added for the agent * fix: lock deps * Configurable timeouts (#245) * configurable timeouts * default_schedule_to_start_timeout - increase to 6 hours * chore(release): bump version (#246) * APP-5482: [Unit Tests] Improve coverage and maintain coverage report (#224) - add coverage reports to s3 bucket. * chore(release): bump version (#247) * feat: moved everything realted to agent workflow into the code sdk * chore: added examples, and readme to init the fastapiagentapp * fix: poetry lock * fix: failing tests, types * fix: agent type issues, validation, improve error message, llm import * fix: prefix / added in the fastapiagentapp * fix: langgraph related imports * chore: update readme for the agents * fix: error msgs, dependencies, readme * fix: api endpoint, readme example * chore: APP-5485 Optimise and add workflow timeout to e2e test framework github action (#248) * APP-5485 add workflow timeout to e2e tests * Modified echo * chore: fetch correct commit sha * chore(release): bump version (#249) * bump atlan-python to v5.0.1 (#252) * chore(release): bump version (#253) * fix: APP-5439 add exclude_unset to true (#254) * bump atlan-python to v5.0.1 * update: add exclude_unset to true * chore(release): bump version (#255) * fix: Clean up logger messages and remove color formatting (#256) * removed addition of colors from message * fix: remove HTML tags from log messages and clean up logger context * chore(release): bump version (#257) * chore(release): bump version (#260) * fix: lock deps * chore(release): bump version (#264) * feat(docgen): APP-5643 add sql parsing (#237) Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#267) * fix: Make e2e test base branch dynamic (#266) * chore(release): bump version (#268) * docs: added decorator doc strings and examples (#265) * chore(release): bump version (#269) * docs: APP-5752 setup mkdocs with pydoctor (#263) * chore: cleanup * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#270) * fix(docs): APP-5272 correct docstrings for decorators (#271) * fix(docs): correct docstrings for decorators * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> * chore: cleanup Signed-off-by: Junaid Rahim <[email protected]> --------- Signed-off-by: Junaid Rahim <[email protected]> * chore(release): bump version (#272) * fix: heartbeat, timeout configurable * chore(release): bump version (#275) * fix: usage of workflow_config and workflow args (#276) * chore(release): bump version (#277) * fix: langchain-openai dependency (#282) --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: SanilK2108 <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Onkar Ravgan <[email protected]> Co-authored-by: Hrushikesh Dokala <[email protected]> Co-authored-by: Atlan CI <[email protected]> Co-authored-by: Amit Prabhu <[email protected]> * fix: poetry lock --------- Signed-off-by: Junaid Rahim <[email protected]> Co-authored-by: Nishchith Shetty <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Abhishek Agrawal <[email protected]> Co-authored-by: Junaid Rahim <[email protected]> Co-authored-by: Mrunmayi Tripathi <[email protected]> Co-authored-by: Mustafa <[email protected]> Co-authored-by: Ujala Singh <[email protected]> Co-authored-by: ujala-singh <[email protected]> Co-authored-by: Onkar Ravgan <[email protected]> Co-authored-by: Hrushikesh Dokala <[email protected]> Co-authored-by: Atlan CI <[email protected]> Co-authored-by: Amit Prabhu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes changes to improve the performance and efficiency of the application by integrating
uvloopand updating dependencies.Performance improvements:
application_sdk/worker.py: Addeduvloopimport and set the event loop policy touvloop.EventLoopPolicy()to enhance the performance of the asynchronous event loop.Dependency updates:
pyproject.toml: Addeduvloopversion^0.21.0to the dependencies to ensure compatibility and leverage the performance benefits ofuvloop.JIRA Ticket
Description
Type of Change
How Has This Been Tested?
1.
2.
3.
Checklist
mypyandpylintwith no errorsSDK Specific Considerations
requirements.txtorsetup.pyif new dependencies were addedAdditional Context
Reviewer Checklist
PR Naming Convention
Recommended Title Format:
[PROJ-XXX] Brief description of changesExample:
[PROJ-1234] Add authentication middleware to SDKSummary by CodeRabbit
New Features
uvlooplibrary, boosting overall performance.Tests
uvloopimplementations.