Add agent-readiness improvements (Link headers, markdown negotiation, Content-Signal, agent-skills, WebMCP) - #807
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds “agent-readiness” features to the StaticHost and frontend to align aspire.dev with common agent discovery/interaction checks (Link headers, markdown negotiation, Content-Signal, agent-skills artifacts + digesting, and WebMCP tool registration), plus a dedicated StaticHost test project and Playwright coverage.
Changes:
- Introduces
UseAgentReadiness()middleware composition (markdown content negotiation + RFC 8288 Link headers) and wires it into the StaticHost pipeline beforeUseDefaultFiles/UseRouting. - Adds
.well-known/agent-skillsdiscovery artifacts (plus digest computation + LF enforcement) and arobots.txtContent-Signal directive. - Adds frontend WebMCP registration (
search-aspire-docs) backed by a pluggable search provider layer, with Playwright e2e tests.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/StaticHost.Tests/WellKnownArtifactTests.cs | Verifies robots.txt Content-Signal and agent-skills discovery/digests + LF-only enforcement. |
| tests/StaticHost.Tests/StaticHost.Tests.csproj | New StaticHost-focused test project wiring. |
| tests/StaticHost.Tests/SamplePages.cs | Shared HTML/MD fixtures for middleware tests. |
| tests/StaticHost.Tests/MarkdownPathMapperTests.cs | Unit coverage for request-path → markdown-companion mapping rules. |
| tests/StaticHost.Tests/MarkdownNegotiationTests.cs | Host-level tests for Accept negotiation (GET/HEAD/406/Vary/Cache-Control). |
| tests/StaticHost.Tests/LinkHeaderTests.cs | Host-level tests for Link header attach/skip rules. |
| tests/StaticHost.Tests/GlobalUsings.cs | Global usings for the new test project. |
| tests/StaticHost.Tests/AgentReadinessTestServer.cs | In-proc TestServer harness mirroring production middleware ordering. |
| tests/StaticHost.Tests/AcceptHeaderParserTests.cs | Direct q-value parsing/predicate tests. |
| src/statichost/StaticHost/StaticHost.csproj | Makes frontend ESProj reference non-transitive via PrivateAssets=all. |
| src/statichost/StaticHost/Properties/AssemblyInfo.cs | InternalsVisibleTo for StaticHost.Tests. |
| src/statichost/StaticHost/Program.cs | Wires app.UseAgentReadiness() before default files/routing. |
| src/statichost/StaticHost/GlobalUsings.cs | Adds global using for AgentReadiness namespace. |
| src/statichost/StaticHost/AgentReadiness/MarkdownPathMapper.cs | Central path mapping + infrastructure skip rules. |
| src/statichost/StaticHost/AgentReadiness/MarkdownNegotiationMiddleware.cs | Serves .md companions based on Accept preferences + caching semantics. |
| src/statichost/StaticHost/AgentReadiness/LinkHeaderMiddleware.cs | Emits discovery Link headers on successful HTML responses. |
| src/statichost/StaticHost/AgentReadiness/AgentReadinessExtensions.cs | Composition root extension and enforced middleware ordering. |
| src/statichost/StaticHost/AgentReadiness/AcceptHeaderParser.cs | Minimal Accept parser focused on html/markdown preference logic. |
| src/frontend/tsconfig.json | Adds @scripts/* path mapping. |
| src/frontend/tests/e2e/webmcp.spec.ts | Playwright coverage for WebMCP tool registration + non-fatal absence. |
| src/frontend/src/scripts/webmcp.ts | Registers search-aspire-docs tool when navigator.modelContext is present. |
| src/frontend/src/scripts/search/typesense-provider.ts | Stub Typesense provider for future migration. |
| src/frontend/src/scripts/search/SearchProvider.ts | Shared provider interface + response/result shapes. |
| src/frontend/src/scripts/search/pagefind-provider.ts | Pagefind-backed provider (dynamic import) with graceful unavailability. |
| src/frontend/src/scripts/search/index.ts | Provider selection via PUBLIC_SEARCH_PROVIDER. |
| src/frontend/src/components/starlight/Head.astro | Imports WebMCP registration script site-wide. |
| src/frontend/scripts/compute-skill-digests.mjs | Recomputes/validates agent-skills SHA-256 digests from raw bytes. |
| src/frontend/public/robots.txt | Adds Content-Signal directive under User-agent: *. |
| src/frontend/public/.well-known/agent-skills/index.json | New agent-skills discovery index (v0.2.0). |
| src/frontend/public/.well-known/agent-skills/getting-started-with-aspire/SKILL.md | New skill document for “getting started” guidance. |
| src/frontend/package.json | Adds digest compute/verify scripts; runs them in dev/build/lint. |
| Aspire.Dev.slnx | Adds StaticHost.Tests to solution. |
| .gitattributes | Forces LF in agent-skills artifacts to keep digests byte-stable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
David Pine (IEvangelist)
added a commit
that referenced
this pull request
May 4, 2026
* MarkdownNegotiationMiddleware: drop the stale reference to the `HasMarkdownCompanion` API (which never existed in this PR's final form); point readers at `MarkdownPathMapper.TryGetMarkdownCompanion` instead so the comment matches the live code. * compute-skill-digests.mjs: harden `resolvePublicPath` against `..` traversal. Switches from `path.join` to `path.resolve` and asserts the result stays under `publicRoot` so a malicious or malformed `url` in index.json (e.g. `/.well-known/agent-skills/../../../../etc/passwd`) cannot read bytes outside the published public/ tree in dev/CI. * WellKnownArtifactTests: rename `Agent_skills_files_are_LF_only` to `AgentSkills_files_are_LF_only` to match the surrounding `AgentSkills_*` PascalCase prefix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
David Pine (IEvangelist)
added a commit
that referenced
this pull request
May 11, 2026
Two follow-ups from @eerhardt's review on PR #807: * Move InternalsVisibleTo into StaticHost.csproj using Include="$(AssemblyName).Tests" to match the convention established in PR #758 and the existing src/tools/*.csproj files. Delete the now-unnecessary Properties/AssemblyInfo.cs. * Mirror MarkdownNegotiationMiddleware's positive ShouldHandle pattern in LinkHeaderMiddleware. The two were inverted: one returned "should I handle?" while the other returned "should I skip?". Both now use the same shape — guard with !ShouldHandle, then check MarkdownPathMapper.IsInfrastructurePath separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… robots Content-Signal, agent-skills, WebMCP) Bring aspire.dev up to spec for the checks at https://isitagentready.com: * RFC 8288 Link headers on HTML responses - LinkHeaderMiddleware advertises </llms.txt>; rel="llms", </.well-known/agent-skills/index.json>; rel="agent-skills", </sitemap-index.xml>; rel="sitemap", and a per-page rel="alternate" type="text/markdown" link when a .md companion exists. - Header attached via Response.OnStarting on 2xx text/html responses only; redirects, JSON, static assets, and well-known JSON are skipped. * Cloudflare-style "Markdown for Agents" content negotiation - MarkdownNegotiationMiddleware handles Accept: text/markdown by streaming the .md companion (emitted by starlight-page-actions) directly via IFileProvider.SendFileAsync. No path rewrite, so no interaction with UseRouting / MapStaticAssets endpoint selection. - Cache-Control: private, max-age=0, must-revalidate ensures Front Door does NOT cache, avoiding Vary: Accept cache-key explosion. - 406 when markdown preferred but no companion AND no HTML acceptable. - HEAD parity, Vary: Accept on negotiated responses, infrastructure paths (.well-known, _astro, healthz, install., pagefind) bypass negotiation. * Both new middlewares run BEFORE UseDefaultFiles + UseRouting (UseDefaultFiles rewrites /foo/ -> /foo/index.html, breaking companion mapping; MapStaticAssets registers endpoints during UseRouting, so post-routing path rewrites do not re-trigger endpoint selection). * robots.txt declares Content-Signal: ai-train=yes, search=yes, ai-input=yes inside the User-agent: * group (per draft-romm-aipref-contentsignals). * /.well-known/agent-skills/index.json (Agent Skills Discovery RFC v0.2.0) with a getting-started-with-aspire SKILL.md and a digest field of the form sha256:<lowerhex>. compute-skill-digests.mjs recomputes / verifies on every build; pnpm lint runs verify-skill-digests in --check mode. .gitattributes pins LF for the agent-skills artifacts so digests are byte-stable across Windows / Linux checkouts. * WebMCP integration on the Astro side - src/scripts/webmcp.ts feature-detects navigator.modelContext.registerTool and registers a single search-aspire-docs tool with a JSON Schema input. - Backed by src/scripts/search/* (SearchProvider abstraction with Pagefind today and a Typesense stub for the upcoming migration). The WebMCP tool surface is engine-agnostic so the Pagefind -> Typesense swap is a one-line change in src/scripts/search/index.ts. - Hooked into Head.astro via a single import line. * New host-level tests in tests/StaticHost.Tests/ - In-process TestServer with a temp wwwroot fixture (no frontend build required; PrivateAssets="all" on the frontend.esproj reference prevents the dist/ directory from leaking into test compilations). - 51 tests covering markdown negotiation (incl. HEAD, 406, fallback, Vary behavior, infrastructure-path skip), Link header content + skip rules, AcceptHeaderParser q-value handling, and the well-known artifacts. * New Playwright spec tests/e2e/webmcp.spec.ts asserts that the homepage registers exactly one WebMCP tool (search-aspire-docs) when the runtime exposes navigator.modelContext, and that the absence of the API is non-fatal. Out of scope (intentionally not advertised, would mislead agents): * /.well-known/openid-configuration / oauth-authorization-server (no protected APIs). * /.well-known/oauth-protected-resource (no protected resource). * /.well-known/mcp/server-card.json (aspire.dev does not host an MCP server). * /.well-known/api-catalog (RFC 9727 requires real API endpoints; aspire.dev exposes documentation, an LLM corpus, a sitemap, and an RSS feed - none of which are APIs in the RFC's sense). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ill conventions The first cut was inaccurate — it told agents to `mkdir my-aspire-app && cd my-aspire-app && aspire new` and then non-interactively pick a template. In reality `aspire new` is fully interactive and creates its own project folder, so the mkdir+cd pattern is wrong and the fabricated template flags would mislead agents. Rewrite the skill to align with the conventions used by the official skill at github.com/microsoft/aspire/tree/main/.agents/skills/aspire while keeping this one short and focused on getting started: * Frontmatter description is now a long when-to-use / when-not-to-use sentence in the same shape as the official skill. * Body: install, `aspire new` (interactive, no fabricated flags), `aspire start` (called out as the agent-friendly path vs. `aspire run` which blocks the terminal), and a concise list of authoritative references. * Explicit pointer to the official `aspire` skill for the operate-an-existing- app workflow, so an agent that has both available picks the right one. * Updated index.json description to match the new framing. * compute-skill-digests.mjs refreshed the sha256 to reflect the new bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Aspire is a polyglot stack — the AppHost can be authored in C# or TypeScript today, with additional languages (Java, Go, Python, Rust, …) on the roadmap. Calling it "the .NET cloud-native stack" was both inaccurate and misleading to agents who would then assume C#-only tooling and dismiss the TypeScript AppHost path. Re-runs compute-skill-digests.mjs to update the index.json digest to match the corrected SKILL.md bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Concerns flagged by the user: 1. LinkHeaderMiddleware.ShouldSkip duplicated the infrastructure path list maintained on MarkdownPathMapper.IsInfrastructurePath. Fixed by routing the path-skip check through the helper so both middlewares stay in lock-step. 2. Not every page on aspire.dev produces a `.md` companion (DocFX-rendered /reference/api/**, the search route, Lunaria stats, redirects, the 404 page). The previous mapper accepted any `.md` that happened to exist on disk, so a stray markdown file with no real HTML page would have been advertised via the `Link: rel="alternate"; type="text/markdown"` header and served by the negotiation middleware on `Accept: text/markdown`. Fixed by requiring BOTH the `.md` AND the corresponding HTML page to exist before declaring a companion. Adds new xUnit cases pinning the stray-md scenario for both middlewares. Additional cleanup along the way: * AcceptHeaderParser.PrefersMarkdown: removed a dead `htmlQ` assignment and hoisted `HighestExplicitQuality` to a private static method so markdown and html lookups go through the same helper. * Extracted shared sample HTML/Markdown bodies and seed helpers used by LinkHeaderTests + MarkdownNegotiationTests into SamplePages so the on-disk Starlight layout is described in one place. Tests: 57 passing (was 51), 0 warnings, 0 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* MarkdownNegotiationMiddleware: drop the stale reference to the `HasMarkdownCompanion` API (which never existed in this PR's final form); point readers at `MarkdownPathMapper.TryGetMarkdownCompanion` instead so the comment matches the live code. * compute-skill-digests.mjs: harden `resolvePublicPath` against `..` traversal. Switches from `path.join` to `path.resolve` and asserts the result stays under `publicRoot` so a malicious or malformed `url` in index.json (e.g. `/.well-known/agent-skills/../../../../etc/passwd`) cannot read bytes outside the published public/ tree in dev/CI. * WellKnownArtifactTests: rename `Agent_skills_files_are_LF_only` to `AgentSkills_files_are_LF_only` to match the surrounding `AgentSkills_*` PascalCase prefix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two follow-ups from @eerhardt's review on PR #807: * Move InternalsVisibleTo into StaticHost.csproj using Include="$(AssemblyName).Tests" to match the convention established in PR #758 and the existing src/tools/*.csproj files. Delete the now-unnecessary Properties/AssemblyInfo.cs. * Mirror MarkdownNegotiationMiddleware's positive ShouldHandle pattern in LinkHeaderMiddleware. The two were inverted: one returned "should I handle?" while the other returned "should I skip?". Both now use the same shape — guard with !ShouldHandle, then check MarkdownPathMapper.IsInfrastructurePath separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
David Pine (IEvangelist)
force-pushed
the
dapine/agent-readiness
branch
from
May 19, 2026 16:02
fabfbe8 to
4575785
Compare
The original PR shipped a SearchProvider interface plus a Typesense stub in src/frontend/src/scripts/search/ so the Pagefind -> Typesense swap would be a one-line change. We're no longer doing that migration, so the abstraction layer has no remaining justification. * Delete src/frontend/src/scripts/search/typesense-provider.ts * Delete src/frontend/src/scripts/search/SearchProvider.ts * Delete src/frontend/src/scripts/search/index.ts * Delete src/frontend/src/scripts/search/pagefind-provider.ts * Inline the Pagefind logic and types into a single src/frontend/src/scripts/search.ts that exports searchAspireDocs(query, limit). * webmcp.ts now imports searchAspireDocs directly; drop the stale "Pagefind today, Typesense later" comment and the provider-selector indirection. No behavior change for the WebMCP search-aspire-docs tool. Pagefind is the only backend now, and the unavailable-fallback contract is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
David Pine (IEvangelist)
enabled auto-merge (squash)
May 19, 2026 17:52
Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
Eric Erhardt (eerhardt)
left a comment
Eric Erhardt (eerhardt)
left a comment
Member
There was a problem hiding this comment.
Just some performance feedback. Once that is done, I think this can be merged,
Addresses PR #807 review feedback on AcceptHeaderParser: - Replaces string.Split / List<MediaTypeWithQ> allocations with span-based parsing using MemoryExtensions.Split, so the middleware hot path no longer allocates per request. - Memoizes negotiation outcomes in a bounded ConcurrentDictionary (max 256 entries, max 256-char keys). Real-world Accept headers cluster around a small set of distinct values so cache hits dominate. - Introduces AcceptHeaderParser.Negotiate(string?) returning a NegotiationResult struct so MarkdownNegotiationMiddleware can ask both questions (PrefersMarkdown / AcceptsHtml) with one cache lookup instead of two. Adds tests for the new Negotiate API: defaults, quoted profile params, case-insensitive q-token, comma-only headers, mixed-validity entries, and a cache-hit consistency check (via internal ClearCacheForTests). All 63 StaticHost.Tests pass (57 prior + 6 new). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses PR #807 review feedback (eerhardt): swap the hand-rolled ConcurrentDictionary + soft-cap with Microsoft.Extensions.Caching.Memory.MemoryCache configured with SizeLimit = 256. Each entry registers Size = 1 so the limit caps distinct Accept headers; MemoryCache compacts/evicts older entries on overflow rather than refusing new inserts. MaxCacheKeyLength = 256 is retained as a pre-filter so adversarial multi-KB Accept headers are never memoized at all. ClearCacheForTests now calls MemoryCache.Clear() (available since .NET 9). MemoryCache is part of the Microsoft.AspNetCore.App shared framework, so no new package reference is required. All 63 StaticHost.Tests still pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eric Erhardt (eerhardt)
approved these changes
May 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation of the checks at https://isitagentready.com. See commit message for full details. Verified: all 51 new + 16 existing C# tests pass, frontend lint clean, full solution build clean. Tests cover middleware ordering, markdown negotiation (incl. HEAD/406/Vary), Link header skip rules, AcceptHeaderParser q-values, agent-skills schema + digest verification, robots Content-Signal, and WebMCP tool registration via Playwright init-script stub. Out of scope (intentionally): OAuth/OIDC discovery, OAuth Protected Resource, MCP Server Card, and api-catalog (RFC 9727 requires real API endpoints; aspire.dev has none).