Skip to content

Add example validation process and descriptive values rule#42567

Open
ramoka178 wants to merge 2 commits intomainfrom
ramoka/examples
Open

Add example validation process and descriptive values rule#42567
ramoka178 wants to merge 2 commits intomainfrom
ramoka/examples

Conversation

@ramoka178
Copy link
Copy Markdown
Contributor

Adds two new sections to the example file review instructions:

  • EX-PROCESS (22.0): Step-by-step procedure for validating examples against their parent operation -- parameter checks, schema validation, URL/body cross-referencing, and coverage expectations.

  • EX-DESCRIPTIVE-VALUES (22.10): Forbids filler values like "aaaaaaa" and "string" in examples. Requires realistic, human-readable names since examples are published in public Azure docs and SDK samples.

Also updates the review checklist summary to include descriptive value checks.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 21, 2026

Next Steps to Merge

Next steps that must be taken to merge this PR:
  • ❌ The required check named Protected Files has failed. Refer to the check in the PR's 'Checks' tab for details on how to fix it and consult the aka.ms/ci-fix guide


Comment generated by summarize-checks workflow run.

@ravimeda
Copy link
Copy Markdown
Member

Thanks for putting this together, Ramakant! The motivation behind this PR is spot on. The VMSS Redeploy doc page with "aaaaaaaaaaaaaaaaaaaaaaaaaaaaa" as the VMSS name is a great example of why we need these checks. Catching low-quality example values before they land in public docs is genuinely valuable.

I have some suggestions on how to structure the changes so they fit the repo's skill architecture and maximize reuse. The content itself is strong, especially EX-DESCRIPTIVE-VALUES.


Suggestion: Extract shared rules into a reference file

The repo's azure-api-review skill uses shared reference files (under .github/skills/azure-api-review/references/) as the single source of truth for cross-cutting rules. Each reference file is then cross-referenced by the format-specific instruction files (openapi-review.instructions.md, typespec-review.instructions.md). This pattern enables reuse across reviewers, authors, and formats without duplication.

For example, SEC-SECRET-DETECT is defined once in references/secret-detection.md and cross-referenced from both OpenAPI and TypeSpec instruction files.

I'd recommend following the same pattern here:

1. Create .github/skills/azure-api-review/references/example-quality.md

This becomes the single source of truth. Move three rules into it:

Rule ID Source in this PR What it covers
EX-ORPHAN EX-PROCESS step 1 Every example must be referenced by an operation; flag orphans
EX-COVERAGE EX-PROCESS step 6 PUT/PATCH need min + max examples; polymorphic variants need separate examples; LROs should show both 202 and 200
EX-DESCRIPTIVE-VALUES Section 22.10 (keep as-is, it's well-written) Forbids filler values, requires realistic human-readable names

Structure it like the existing reference files (upstream alignment comment, rule IDs, severity, fix guidance, format-specific notes for OpenAPI and TypeSpec).

One small tweak for EX-DESCRIPTIVE-VALUES: The "20+ characters composed of a single repeated character" threshold is oddly specific. "aaa" as a resource name is equally bad. Consider simplifying to "resource names consisting entirely of a single repeated character."

One addition: Auto-generated examples (e.g., from oav generate-examples or tsp compile) are a common source of filler values. A note like "Auto-generated examples SHOULD be hand-edited to replace filler values before submitting" would help authors catch these at the source.

2. Restructure the EX-PROCESS section (22.0)

The proposed EX-PROCESS section has useful new content, but several steps duplicate existing rules:

Step Recommendation Reason
Step 1 (trace to operation) Move to reference file as EX-ORPHAN Genuinely new content
Step 2 (validate params) Remove Already covered by 22.3 EX-PARAM-CONSISTENCY
Step 3 (validate request body) Remove Already covered by 22.7 EX-PAYLOAD
Step 4 (validate response body) Remove Already covered by 22.7 EX-PAYLOAD + 22.2 EX-RESOURCE-ID
Step 5 (cross-reference URLs) Remove Already covered by 22.3 + 22.2
Step 6 (check coverage) Move to reference file as EX-COVERAGE Genuinely new content
Step 7 (apply EX-* rules) Remove Redundant (the reviewer applies rules below anyway)

Replace the entire 22.0 section with three concise subsections (22.10, 22.11, 22.12) that cross-reference the shared file:

### 22.10 Orphan Detection (EX-ORPHAN)

> See [`.github/skills/azure-api-review/references/example-quality.md`](../skills/azure-api-review/references/example-quality.md) for the full rule.

- Every example file in the PR **MUST** be referenced by exactly one `x-ms-examples` entry.
  Flag any orphaned example file not referenced by any operation.

### 22.11 Example Coverage (EX-COVERAGE)

> See [`.github/skills/azure-api-review/references/example-quality.md`](../skills/azure-api-review/references/example-quality.md) for the full rule.

- PUT and PATCH operations **SHOULD** have at least two examples (minimum + maximum property set).
- Operations with polymorphic discriminators **SHOULD** have separate examples per variant.
- LRO operations **SHOULD** show both the initial response (e.g., `202`) and the final response (e.g., `200`).

### 22.12 Descriptive Example Values (EX-DESCRIPTIVE-VALUES)

> See [`.github/skills/azure-api-review/references/example-quality.md`](../skills/azure-api-review/references/example-quality.md) for the full rule.

- Example values **MUST** be realistic and descriptive, not filler like `"aaaaaaa"` or `"string"`.
- Resource names **MUST** use recognizable, human-readable names (e.g., `"myVmScaleSet"`).
- Auto-generated examples **SHOULD** be hand-edited to replace filler values before merging.

3. Update typespec-review.instructions.md

TypeSpec specs produce the same example files with the same quality problems, but the TypeSpec review instructions currently only say "Example files present for all operations" with no quality checks. Add a cross-reference:

  • In the anti-patterns list (around line 201): add a bullet for low-quality example values pointing to the reference file.
  • In the checklist: update to Example files present for all operations, with realistic descriptive values (EX-DESCRIPTIVE-VALUES).

4. Update SKILL.md

Add the new reference file to the table in .github/skills/azure-api-review/SKILL.md:

| [example-quality.md](references/example-quality.md) | Example file quality: orphan detection, coverage, descriptive values | EX-ORPHAN, EX-COVERAGE, EX-DESCRIPTIVE-VALUES |

5. Checklist summary update (keep yours)

The checklist line update in the PR is good. Expand it slightly to cover the new rules:

- Example files validated: titles match operations, resource IDs are valid and consistent,
  no null nextLink, LRO headers correct, timestamps in RFC3339, no malformed values,
  values are realistic and descriptive -- not filler like aaaa or string,
  no orphaned example files, adequate coverage for PUT/PATCH/LRO (EX-*)

Resulting reuse flow

.github/skills/azure-api-review/references/example-quality.md  (single source of truth)
    |
    +-- openapi-review.instructions.md     (sections 22.10-22.12, cross-ref)
    +-- typespec-review.instructions.md    (anti-pattern + checklist, cross-ref)
    +-- ARM API Reviewer agent             (loads skill, gets rules)
    +-- Authors preparing PRs              (loads skill, self-checks)

This matches exactly how SEC-SECRET-DETECT, naming-conventions, and other cross-cutting rules work today.


CI fixes

  • SpellCheck: Run npx cspell locally on the changed files to find unrecognized words. Add legitimate technical terms to cspell.yaml.
  • Protected Files: openapi-review.instructions.md is owned by @praveenkuttappan and @maririos per CODEOWNERS. Their approval is required.

Happy to help if you have questions on any of this. The rule content is solid; this is just about packaging it in the right place so it benefits the broadest set of consumers.

@ravimeda
Copy link
Copy Markdown
Member

@praveenkuttappan please could you review too

Copy link
Copy Markdown
Member

@haolingdong-msft haolingdong-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good from typespec authoring side. just as discussed, it is better to have benchmark tests: 1. to avoid regression 2. it is hard to review the prompts (and its impact on the result), but if we have benchmark test report, it would be much easier to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants