Live and recorded tests skill

Author: timovv

.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: azsdk-common-live-and-recorded-tests
+license: MIT
+metadata:
+  version: "0.1.0"
+  distribution: shared
+description: "Deploy test resources and run Azure SDK tests in live, record, or playback mode. WHEN: \"run live tests\", \"run recorded tests\", \"deploy test resources\", \"record tests\", \"run tests in record mode\", \"clean up test resources\", \"run tests against live resources\". DO NOT USE FOR: writing new tests, authoring Bicep templates, playback-only test runs without resource deployment. INVOKES: azure-sdk-mcp:azsdk_package_run_tests."
+compatibility:
+  requires: "azure-sdk-mcp server, Azure PowerShell (Az module), local azure-sdk-for-{language} clone"
+---
+
+# Live and Recorded Tests
+
+## MCP Tools
+
+| Tool | Purpose |
+|------|---------|
+| `azure-sdk-mcp:azsdk_package_run_tests` | Run tests in playback, record, or live mode |
+
+## Prerequisites
+
+- azure-sdk-mcp server must be running
+- Azure PowerShell module (`Az`) installed
+- For live/record modes: an Azure subscription with permissions to create resources
+  - Use the Azure SDK Test Resources - TME (id: 4d042dc6-fe17-4698-a23f-ec6a8d1e98f4) subscription if not already authenticated and no other subscription is specified.
+
+## Steps
+
+1. **Identify service directory** — Determine the service directory for the package under test (e.g., `keyvault`, `storage`). This is required by the test resource deployment script.
+2. **Check for existing deployment** — Look for a `.env` file at the service directory level (next to `test-resources.bicep`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7.
+3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount`. Confirm the correct subscription is selected.
+4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs.
+5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs.
+6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output.
+7. **Run tests** — Run `azure-sdk-mcp:azsdk_package_run_tests` with the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo.
+8. **Clean up** — Ask the user whether to clean up test resources. If yes, run `eng/common/TestResources/Remove-TestResources.ps1`. If no, inform the user that resources remain deployed for subsequent test runs. See [cleanup details](references/test-resource-cleanup.md).
+
+## Examples
+
+- "Deploy test resources and run live tests for this package"
+- "Run the tests in record mode using the live test deployment specified in .env"
+- "Clean up my test resources for keyvault"
+- "Run tests in record mode against my existing deployment"
+- "Set up live test resources for storage and run all tests"
+
+## Troubleshooting
+
+- **No Azure context:** Run `Connect-AzAccount` and select the target subscription with `Set-AzContext -SubscriptionId <id>`.
+- **Deployment fails with auth error:** Verify that the signed-in account has Contributor or Owner role on the target subscription. Check `Get-AzContext` output.
+- **Deployment fails with resource conflict:** A previous deployment may still exist. Try running `Remove-TestResources.ps1` first, or use a different `BaseName`.
+- **Tests fail in record mode:** Check that all environment variables from the deployment are being passed correctly. Verify the `.env` file path is correct.
+- **Assets push fails after recording:** Ensure the assets repo is configured and accessible. Check git authentication.

.github/skills/azsdk-common-live-and-recorded-tests/eval.yaml

@@ -0,0 +1,29 @@
+name: azsdk-common-live-and-recorded-tests-eval
+description: Evaluation suite for live and recorded tests skill.
+skill: azsdk-common-live-and-recorded-tests
+version: '1.0'
+config:
+  model: gpt-4o
+  executor: mock
+  trials_per_task: 1
+  timeout_seconds: 600
+  parallel: false
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did the skill complete trigger and anti-trigger checks?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: text
+    name: no_fatal_errors
+    config:
+      regex_not_match:
+        - (?i)error occurred
+        - (?i)failed
+        - (?i)re-run the tool
+tasks:
+  - "tasks/*.yaml"

.github/skills/azsdk-common-live-and-recorded-tests/references/test-modes.md

@@ -0,0 +1,31 @@
+# Test Modes
+
+## Overview
+
+| Mode | Description | Requires Azure Resources | Records Traffic |
+|------|-------------|--------------------------|-----------------|
+| **Playback** | Runs against previously recorded HTTP interactions | No | No |
+| **Record** | Runs against real Azure resources and records HTTP interactions | Yes | Yes |
+| **Live** | Runs against real Azure resources without recording | Yes | No |
+
+## Default Behavior
+
+When no test mode is specified, **Playback** is the default. Playback mode does not require test resource deployment or environment variables.
+
+## Record Mode
+
+- Requires deployed test resources and environment variables passed via a `.env` file.
+- When all tests pass in record mode, the tool automatically pushes recorded test assets to the assets repo.
+- This is the standard workflow for creating or updating test recordings.
+
+## Live Mode
+
+- Requires deployed test resources and environment variables passed via a `.env` file.
+- Tests run against real Azure resources without recording HTTP traffic.
+- Useful for validating that tests work against real services without updating recordings.
+
+## Playback Mode
+
+- Does not require deployed Azure resources.
+- Tests replay previously recorded HTTP interactions via the test proxy.
+- This is the default mode and the fastest way to run tests locally.

.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-cleanup.md

@@ -0,0 +1,37 @@
+# Test Resource Cleanup
+
+## Script Location
+
+`eng/common/TestResources/Remove-TestResources.ps1`
+
+## Common Parameters
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `ServiceDirectory` | Yes* | Service directory name (e.g., `keyvault`). Required if `ResourceGroupName` not provided. |
+| `-ResourceGroupName` | No | Explicit resource group name to delete |
+| `-Force` | No | Skip confirmation prompt |
+
+*Either `ServiceDirectory` or `-ResourceGroupName` must be provided. If the test resources were deployed to the TME tenant, you must use the resource group name, which should include the `SSS3PT_` prefix as output by the test resource creation script.
+
+## Usage Examples
+
+### Basic cleanup
+
+```powershell
+eng/common/TestResources/Remove-TestResources.ps1 keyvault -Force
+```
+
+### Cleanup by resource group name
+
+```powershell
+eng/common/TestResources/Remove-TestResources.ps1 `
+    -ResourceGroupName 'rg-mytest' `
+    -Force
+```
+
+## Notes
+
+- The script also purges purgeable resources (e.g., Key Vault soft-deleted vaults).
+- Any `.env` files created by `New-TestResources.ps1` are also removed.
+- Always ask the user before cleaning up — they may want to reuse resources for additional test runs.

.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md

@@ -0,0 +1,48 @@
+# Test Resource Deployment
+
+## Script Location
+
+`eng/common/TestResources/New-TestResources.ps1`
+
+## Common Parameters
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `ServiceDirectory` | Yes | Service directory name under `sdk/` (e.g., `keyvault`, `storage`) |
+| `-OutFile` | Yes | Required for a .env file to be output. |
+| `-BaseName` | No | Base name for resource group (default: auto-generated) |
+| `-Location` | No | Azure region for deployment (default: script default) |
+| `-SubscriptionId` | No | Target subscription (uses current Az context if omitted) |
+| `-ResourceGroupName` | No | Custom resource group name |
+
+## Usage Examples
+
+### Basic deployment (interactive, uses current Az context)
+
+```powershell
+Connect-AzAccount -Subscription 'SUBSCRIPTION_ID'
+eng/common/TestResources/New-TestResources.ps1 keyvault -OutFile
+```
+
+### Deployment with custom parameters
+
+```powershell
+Connect-AzAccount -Subscription 'SUBSCRIPTION_ID'
+eng/common/TestResources/New-TestResources.ps1 `
+    -BaseName 'azsdk' `
+    -ServiceDirectory 'keyvault' `
+    -SubscriptionId 'SUBSCRIPTION_ID' `
+    -ResourceGroupName 'rg-mytest' `
+    -Location 'eastus'
+    -OutFile
+```
+
+## Output
+
+The script outputs environment variables needed for live and record test runs. If `$SupportsTestResourcesDotenv=$true` in the language repo's `LanguageSettings.ps1`, a `.env` file is written next to the `test-resources.bicep` file.
+
+Environment variables typically include credentials and resource endpoints needed by test frameworks.
+
+## Additional Parameters
+
+Some services require extra parameters (e.g., `-AdditionalParameters @{enableHsm=$true}` for Key Vault HSM tests). Check the service's `test-resources.bicep` or `test-resources.json` for required parameters.

.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger-write-tests.yaml

@@ -0,0 +1,15 @@
+id: live-tests-negative-002
+name: Anti-Trigger - Write New Tests
+description: |
+  Test that the skill does NOT activate when the user wants to
+  author new test cases rather than run existing tests.
+tags:
+  - negative-test
+  - anti-trigger
+inputs:
+  prompt: "Help me write unit tests for my new Azure SDK client"
+expected:
+  output_not_contains:
+    - "New-TestResources"
+    - "azsdk_package_run_tests"
+    - "Connect-AzAccount"

.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger.yaml

@@ -0,0 +1,15 @@
+id: live-tests-negative-001
+name: Anti-Trigger - Playback Only Test Run
+description: |
+  Test that the skill does NOT activate for playback-only test runs
+  that don't need resource deployment.
+tags:
+  - negative-test
+  - anti-trigger
+inputs:
+  prompt: "Run the tests for this package in playback mode"
+expected:
+  output_not_contains:
+    - "New-TestResources"
+    - "Connect-AzAccount"
+    - "Get-AzContext"

.github/skills/azsdk-common-live-and-recorded-tests/tasks/cleanup-resources.yaml

@@ -0,0 +1,18 @@
+id: cleanup-resources-001
+name: Clean Up Test Resources
+description: |
+  Test that the skill offers and performs cleanup of test resources
+  after a test run.
+tags:
+  - basic
+  - happy-path
+  - cleanup
+inputs:
+  prompt: "Clean up the test resources I deployed for keyvault"
+expected:
+  output_contains:
+    - "Remove-TestResources"
+  outcomes: []
+  behavior:
+    max_tool_calls: 5
+    max_response_time_ms: 60000

.github/skills/azsdk-common-live-and-recorded-tests/tasks/deploy-and-run-live.yaml

@@ -0,0 +1,22 @@
+id: live-tests-basic-001
+name: Deploy Test Resources and Run Live Tests
+description: |
+  Test that the skill guides through deploying test resources and
+  running tests in live mode for a package.
+tags:
+  - basic
+  - happy-path
+  - live-mode
+inputs:
+  prompt: "Deploy test resources and run live tests for the keyvault package at ~/repos/azure-sdk-for-python/sdk/keyvault/azure-keyvault-keys"
+expected:
+  output_contains:
+    - "Get-AzContext"
+    - "New-TestResources"
+    - "live"
+  outcomes:
+    - type: tool_called
+      tool_name: azsdk_package_run_tests
+  behavior:
+    max_tool_calls: 10
+    max_response_time_ms: 60000

.github/skills/azsdk-common-live-and-recorded-tests/tasks/record-with-env.yaml

@@ -0,0 +1,22 @@
+id: record-tests-with-env-001
+name: Run Tests in Record Mode with Existing Environment
+description: |
+  Test that the skill runs tests in record mode when a .env file
+  is already provided, skipping resource deployment.
+tags:
+  - basic
+  - happy-path
+  - record-mode
+inputs:
+  prompt: "Run the tests in record mode using the live test deployment specified in sdk/keyvault/azure-keyvault-keys/.env"
+expected:
+  output_contains:
+    - "record"
+  output_not_contains:
+    - "New-TestResources"
+  outcomes:
+    - type: tool_called
+      tool_name: azsdk_package_run_tests
+  behavior:
+    max_tool_calls: 5
+    max_response_time_ms: 60000