AEO: structured data, agent surfaces, crawlable content, robots hardening

[ragojose]

Ports the basement.studio AEO system (website-2k25) to Shader Lab.

Fixes

• Root-layout canonical landmine: alternates.canonical: "/tools/shader-lab" was inherited by every page that didn't override it (not-found, any future page). Removed — each page owns its canonical.
• OG siteName / applicationName was the npm package name (@basementstudio/shader-lab); now "Shader Lab", with a richer default description.
• Previews were fully indexable: non-production deployments (VERCEL_ENV !== "production") now serve robots.txt disallow-all, a noindex meta on every page, and an empty sitemap.

Structured data

One JSON-LD @graph per page via PageJsonLd, always inlining the Organization node anchored at https://basement.studio/#organization so both properties resolve to the same entity. Canonical facts live in src/lib/structured-data/product-facts.ts; effect names derive from LAYER_CATALOG so they can't drift.

• Editor: WebSite + WebApplication (free offer, WebGPU requirement, feature list)
• Scenes: CreativeWork with like/remix InteractionCounters, isBasedOn for remixes, effect keywords, author Person
• Profiles: ProfilePage (only when publishedCount > 0, matching the noindex rule)
• Community: CollectionPage + ItemList; breadcrumbs everywhere; FAQPage on the about page

Scene titles/descriptions are user-generated, so the JSON output escapes < to prevent </script> breakout.

Crawlable content

• New /tools/shader-lab/about: what Shader Lab is, how layers work, the full effect catalog with preview images, both npm packages, and a 6-question FAQ. The editor page had zero crawlable text (full-viewport WebGPU app, no h1) — this page is the fix. Editor shell also gets an sr-only h1.
• Linked from the privacy page (cross-linked back), sitemap, llms.txt, and the markdown mirrors.

Agent surfaces

• /llms.txt + /agents.md as route handlers (absolute URLs derive from the runtime base URL; community links gated on deployment config)
• Markdown mirrors: /index.md (product overview), /sitemap.md (content index), and per-scene .md twins — /community/<slug>.md rewrites through middleware, Accept: text/markdown negotiation on the HTML path, and Link rel="alternate" advertised on scene HTML responses
• robots.ts: named allow groups for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended; disallow /api/, /auth/, /monitoring
• manifest.ts (PWA icons pending brand assets — favicon placeholder for now) and /.well-known/mcp.json discovery card pointing at @basementstudio/shader-lab-mcp

Everything degrades gracefully when community is disabled (no community URLs, no throws).

Verified

• next build passes; lint clean (2 pre-existing warnings); tsc --noEmit clean
• Curl-checked on a prod server: robots groups, llms/agents/index/sitemap .md content types + canonical Link headers, mcp.json, manifest, canonicals on every page type, JSON-LD graphs, .md rewrite + Accept negotiation + alternate header
• VERCEL_ENV=preview build: disallow-all robots, noindex meta, empty sitemap

Review notes

• Copy is drafted for review: APP_DESCRIPTION, editor/about descriptions, about-page prose, FAQ answers.
• Needs 512px + 180px PNG icons to finish the manifest.
• All URLs are APP_BASE_URL-relative; set NEXT_PUBLIC_BASE_URL in Vercel prod once the domain is final.
• The about page isn't linked from the editor UI yet — open to suggestions on placement.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
shader-lab	Ready	Preview	Aug 31, 2026 2:20pm

[greptile-apps]

Greptile Summary

This PR adds structured data, crawlable product content, Markdown and agent discovery surfaces, deployment-aware crawler controls, and page-specific metadata.

• Adds JSON-LD graphs for the editor, community, scenes, profiles, breadcrumbs, and FAQ content.
• Adds about, llms.txt, agents.md, Markdown mirrors, manifest, MCP discovery, sitemap, and robots surfaces.
• Extends middleware with scene Markdown rewrites and content negotiation.

Confidence Score: 3/5

The PR should not merge until user-authored Markdown is escaped and non-Vercel development or staging deployments retain the promised indexing protections.

Published metadata can alter the new Markdown documents, while an unset VERCEL_ENV enables indexing across all three crawler-control surfaces; Accept negotiation also has a non-blocking standards edge case.

Files Needing Attention: src/app/api/md/scenes/[slug]/markdown.ts, src/app/sitemap.md/route.ts, src/lib/app.ts, src/middleware.ts

Prompt To Fix All With AI

### Issue 1
src/app/api/md/scenes/[slug]/markdown.ts:32-36
**Unescaped metadata alters Markdown**

When a published title, description, author name, or remix-source field contains Markdown syntax or newlines, this builder inserts it directly into headings, links, and body content, allowing malformed links or arbitrary agent-facing content in the scene mirror and `sitemap.md`.

### Issue 2
src/lib/app.ts:32-35
**Unset environment enables indexing**

If a development, staging, or self-hosted deployment runs without `VERCEL_ENV`, this function classifies it as production, causing the layout to omit `noindex`, robots.txt to permit crawling, and the sitemap to expose pages despite the stated non-production invariant.

### Issue 3
src/middleware.ts:71-73
**Accept qualities are ignored**

The substring check selects Markdown even for headers such as `text/html;q=1, text/markdown;q=0`, so clients that reject Markdown or prefer HTML receive the Markdown mirror instead of the canonical scene page.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "AEO: structured data, agent surfaces, cr..." | Re-trigger Greptile}