# robots.txt for globalgov.io
#
# A named User-agent group REPLACES the star group, it does not merge with it.
# Every named group below therefore RESTATES Disallow: /api/. Deleting those
# repeated lines as redundant duplication is the exact defect recorded as M-62
# and fixed 2026-09-06: Googlebot, GPTBot, anthropic-ai and PerplexityBot each
# inherited nothing from the star group and were free to spend crawl budget on
# the JSON API, while Bingbot, which does the overwhelming majority of verified
# crawling on this site, was unnamed, fell under the star group, and was the
# only crawler actually restricted. The rules read as duplication and are not.
#
# Adding a new named group means copying every rule it needs to keep.
#
# /login and /public/ are deliberately NOT disallowed here (LANE-GOOGLE-CRAWL-
# RECLAIM, 2026-09-16). Google holds ~545 /login?redirect=* URLs and ~33
# /public/solicitations?country&subcategory URLs, all serving the SPA shell at
# HTTP 200 with no canonical, and it reports them as duplicates. They now
# answer X-Robots-Tag: noindex, follow (nginx, sites-enabled/globalgov).
# A Disallow here would block the fetch, Googlebot would never READ that
# noindex, and the set would freeze in the index instead of draining out.
# Add the Disallow ONLY after GSC shows the duplicate buckets empty -- then it
# stops the residual crawl. Doing it in the other order is the defect.

User-agent: *
Allow: /
Disallow: /api/

User-agent: Googlebot
Allow: /
Disallow: /api/

User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: anthropic-ai
Allow: /
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /api/

Sitemap: https://globalgov.io/sitemap.xml