# Robots.txt for SPT Pension Website User-agent: * Allow: / # Disallow admin and test pages Disallow: /admin/ Disallow: /test-api/ Disallow: /test-knowledge/ Disallow: /api/ # Disallow form processing endpoints Disallow: /forms/*/submit # Note: crawling of portal.standardpensions.com is governed by that # subdomain's own robots.txt — a Disallow directive here cannot control it, # and Disallow takes a path rather than an absolute URL. # Allow important pages Allow: /about/ Allow: /schemes/ Allow: /services/ Allow: /forms/ Allow: /media/ Allow: /contact Allow: /testimonials Allow: /pension-calculator Allow: /chat # Sitemap location Sitemap: https://www.standardpensions.com/sitemap.xml