Add example validation process and descriptive values rule#42567
Add example validation process and descriptive values rule#42567
Conversation
Next Steps to MergeNext steps that must be taken to merge this PR:
Comment generated by summarize-checks workflow run. |
|
Thanks for putting this together, Ramakant! The motivation behind this PR is spot on. The VMSS Redeploy doc page with I have some suggestions on how to structure the changes so they fit the repo's skill architecture and maximize reuse. The content itself is strong, especially Suggestion: Extract shared rules into a reference fileThe repo's For example, I'd recommend following the same pattern here: 1. Create
|
| Rule ID | Source in this PR | What it covers |
|---|---|---|
EX-ORPHAN |
EX-PROCESS step 1 | Every example must be referenced by an operation; flag orphans |
EX-COVERAGE |
EX-PROCESS step 6 | PUT/PATCH need min + max examples; polymorphic variants need separate examples; LROs should show both 202 and 200 |
EX-DESCRIPTIVE-VALUES |
Section 22.10 (keep as-is, it's well-written) | Forbids filler values, requires realistic human-readable names |
Structure it like the existing reference files (upstream alignment comment, rule IDs, severity, fix guidance, format-specific notes for OpenAPI and TypeSpec).
One small tweak for EX-DESCRIPTIVE-VALUES: The "20+ characters composed of a single repeated character" threshold is oddly specific. "aaa" as a resource name is equally bad. Consider simplifying to "resource names consisting entirely of a single repeated character."
One addition: Auto-generated examples (e.g., from oav generate-examples or tsp compile) are a common source of filler values. A note like "Auto-generated examples SHOULD be hand-edited to replace filler values before submitting" would help authors catch these at the source.
2. Restructure the EX-PROCESS section (22.0)
The proposed EX-PROCESS section has useful new content, but several steps duplicate existing rules:
| Step | Recommendation | Reason |
|---|---|---|
| Step 1 (trace to operation) | Move to reference file as EX-ORPHAN | Genuinely new content |
| Step 2 (validate params) | Remove | Already covered by 22.3 EX-PARAM-CONSISTENCY |
| Step 3 (validate request body) | Remove | Already covered by 22.7 EX-PAYLOAD |
| Step 4 (validate response body) | Remove | Already covered by 22.7 EX-PAYLOAD + 22.2 EX-RESOURCE-ID |
| Step 5 (cross-reference URLs) | Remove | Already covered by 22.3 + 22.2 |
| Step 6 (check coverage) | Move to reference file as EX-COVERAGE | Genuinely new content |
| Step 7 (apply EX-* rules) | Remove | Redundant (the reviewer applies rules below anyway) |
Replace the entire 22.0 section with three concise subsections (22.10, 22.11, 22.12) that cross-reference the shared file:
### 22.10 Orphan Detection (EX-ORPHAN)
> See [`.github/skills/azure-api-review/references/example-quality.md`](../skills/azure-api-review/references/example-quality.md) for the full rule.
- Every example file in the PR **MUST** be referenced by exactly one `x-ms-examples` entry.
Flag any orphaned example file not referenced by any operation.
### 22.11 Example Coverage (EX-COVERAGE)
> See [`.github/skills/azure-api-review/references/example-quality.md`](../skills/azure-api-review/references/example-quality.md) for the full rule.
- PUT and PATCH operations **SHOULD** have at least two examples (minimum + maximum property set).
- Operations with polymorphic discriminators **SHOULD** have separate examples per variant.
- LRO operations **SHOULD** show both the initial response (e.g., `202`) and the final response (e.g., `200`).
### 22.12 Descriptive Example Values (EX-DESCRIPTIVE-VALUES)
> See [`.github/skills/azure-api-review/references/example-quality.md`](../skills/azure-api-review/references/example-quality.md) for the full rule.
- Example values **MUST** be realistic and descriptive, not filler like `"aaaaaaa"` or `"string"`.
- Resource names **MUST** use recognizable, human-readable names (e.g., `"myVmScaleSet"`).
- Auto-generated examples **SHOULD** be hand-edited to replace filler values before merging.3. Update typespec-review.instructions.md
TypeSpec specs produce the same example files with the same quality problems, but the TypeSpec review instructions currently only say "Example files present for all operations" with no quality checks. Add a cross-reference:
- In the anti-patterns list (around line 201): add a bullet for low-quality example values pointing to the reference file.
- In the checklist: update to
Example files present for all operations, with realistic descriptive values (EX-DESCRIPTIVE-VALUES).
4. Update SKILL.md
Add the new reference file to the table in .github/skills/azure-api-review/SKILL.md:
| [example-quality.md](references/example-quality.md) | Example file quality: orphan detection, coverage, descriptive values | EX-ORPHAN, EX-COVERAGE, EX-DESCRIPTIVE-VALUES |
5. Checklist summary update (keep yours)
The checklist line update in the PR is good. Expand it slightly to cover the new rules:
- Example files validated: titles match operations, resource IDs are valid and consistent,
no null nextLink, LRO headers correct, timestamps in RFC3339, no malformed values,
values are realistic and descriptive -- not filler like aaaa or string,
no orphaned example files, adequate coverage for PUT/PATCH/LRO (EX-*)
Resulting reuse flow
.github/skills/azure-api-review/references/example-quality.md (single source of truth)
|
+-- openapi-review.instructions.md (sections 22.10-22.12, cross-ref)
+-- typespec-review.instructions.md (anti-pattern + checklist, cross-ref)
+-- ARM API Reviewer agent (loads skill, gets rules)
+-- Authors preparing PRs (loads skill, self-checks)
This matches exactly how SEC-SECRET-DETECT, naming-conventions, and other cross-cutting rules work today.
CI fixes
- SpellCheck: Run
npx cspelllocally on the changed files to find unrecognized words. Add legitimate technical terms tocspell.yaml. - Protected Files:
openapi-review.instructions.mdis owned by@praveenkuttappanand@maririosper CODEOWNERS. Their approval is required.
Happy to help if you have questions on any of this. The rule content is solid; this is just about packaging it in the right place so it benefits the broadest set of consumers.
|
@praveenkuttappan please could you review too |
There was a problem hiding this comment.
The changes look good from typespec authoring side. just as discussed, it is better to have benchmark tests: 1. to avoid regression 2. it is hard to review the prompts (and its impact on the result), but if we have benchmark test report, it would be much easier to review.
Adds two new sections to the example file review instructions:
EX-PROCESS (22.0): Step-by-step procedure for validating examples against their parent operation -- parameter checks, schema validation, URL/body cross-referencing, and coverage expectations.
EX-DESCRIPTIVE-VALUES (22.10): Forbids filler values like "aaaaaaa" and "string" in examples. Requires realistic, human-readable names since examples are published in public Azure docs and SDK samples.
Also updates the review checklist summary to include descriptive value checks.