# robots.txt for http://gssg-www.stanford.edu/ # disallow everything User-agent: * # Disallow: * Disallow: /DEFUNCT/ Disallow: /RCS/ Disallow: /admin/ Disallow: /backup/ Disallow: /cgi-bin/ Disallow: /gssg/ Disallow: /jitter/ Disallow: /libexec/ Disallow: /logs/ Disallow: /netdev/ Disallow: /presentations/ Disallow: /stanford/ Disallow: /tools/ Disallow: /work/ Disallow: /~*/ User-Agent: W3C-checklink Disallow: