Docs SEO & Crawl Policy

1. Context (verified in repo)

• Legal (gograb-space): tools/scripts/build/prerender-legal-pages.js runs after production build; slugs and SEO metadata come from libs/libs-structural/legal-lib/.../legal.constants.ts; outputs static index.html per route for bots.
• Docs: tools/scripts/build/prerender-docs-pages.js runs as part of docs:build:prod (package.json); injects markdown from docs/english/{source}/index.md into the Angular shell for a fixed list of paths (DOC_PAGES in that script).
• Runtime: DocumentationSeoService + @libs-foundational/brand-seo update meta/JSON-LD for clients that execute JavaScript; this does not replace bot-readable HTML for URLs that matter for indexing.
• Gap: robots.txt references https://docs.gograb.space/sitemap.xml — confirm that file is produced at deploy; if missing, crawlers get 404 on sitemap.

2. Goals

1. Selective indexing: Not every documentation project needs to be static/SEO-oriented; internal or session-heavy trees should default to noindex and no prerender cost.
2. Single source of truth: Prerender targets, robots/noindex, and optional sitemap entries must derive from the same project metadata the app uses (e.g. folders config / manifest), not a second hardcoded list.
3. Operational clarity: Engineers know which routes get static HTML, which are SPA-only, and what is indexed.

3. Policy model (proposed)

Per root documentation project (or per language root):

• public-index: Allow indexing; include in generated sitemap.xml (when implemented); prerender root if prerenderRoot is true.
• noindex: Runtime meta noindex; exclude from sitemap; no prerender unless product explicitly needs a share preview (then use noindex + OG tags only — decide per product).
• internal: Same as noindex for crawlers; may additionally be excluded from public builds (future).

Assumption: crawlPolicy lives in the same JSON/YAML that backs project selection (e.g. generated docs-folders-config / manifest). Verify the exact schema file name and build step before implementing.

4. Build pipeline changes (plan)

1. Generate prerender config during configure-docs-path / generate-folders-config (or a small pre-build script): emit tools/scripts/build/docs-prerender-targets.json (or TS module) listing { path, sourceDir, crawlPolicy, prerenderRoot }.
2. Refactor prerender-docs-pages.js to read that artifact instead of hardcoded DOC_PAGES; skip entries where prerenderRoot is false or crawlPolicy is not public-index.
3. Sitemap: Add tools/scripts/build/generate-docs-sitemap.js invoked in docs:build:prod, output dist/.../browser/sitemap.xml with only public-index URLs that have a stable public path. Align URL shape with production routing (hash vs path — must be validated against Cloudflare _redirects and Angular withHashLocation()).
4. Robots: Keep Allow/Disallow; ensure sitemap URL matches deployed asset.

5. Runtime changes (plan)

• For projects with noindex / internal, set robots meta (and optionally X-Robots-Tag via hosting) on doc routes — implementation detail in DocumentationSeoService or route metadata.
• Hash-based URLs complicate canonical and crawler behavior; document the public URL contract for “SEO-important” pages (prefer path-based public URLs for those routes only, if feasible).

6. Risks & edge cases

• Duplicate content (EN/ES): require explicit canonical / hreflang before scaling indexed locales.
• Preview deploys: Non-prod hosts should ship noindex and a different robots.txt or block crawlers.
• Drift: Any hand-maintained list in prerender-docs-pages.js will rot; automation from folder config is mandatory at scale.

7. Acceptance criteria

• No duplicate manual list: prerender inputs come from generated config.
• sitemap.xml exists at the URL advertised in robots.txt or robots.txt is updated.
• Internal/session projects are not in sitemap and carry noindex when loaded.
• CI fails or warns if a public-index project lacks a prerender source file.

8. Out of scope (for this spec)

• Full SSR of the entire docs app.
• Server-side analytics aggregation (admin dashboards currently use session-local data unless environment.enableAnalytics persistence is enabled).