Robots.txt

Robots.txt

• Add to root directory when there are things you don’t want spiders to index
– Example robots.txt file:
• Disallow: /special_files/
• Disallow: /church/leaders.html

• Disallow: /*.pdf
– Translation: For all spiders, don’t index files in “special_files” folder, don’t index “leaders.html” in “church” folder, don’t index any .pdf files

• User-agent: googlebot

• Disallow: /
– Translation: For specific google spider, don’t index anything on this site

– Example on individual site pages
•
– Translation: don’t index this page, don’t follow the links on this page to index other pages

– Can use combinations: “noindex, follow”, “index, nofollow”