How do I stop a page from being spidered?

Wednesday, December 26th, 2007 at 12:02 pm

How to stop spiders [ 1:55 ] Play Now | Play in Popup | Download

Todays question was submitted through the survey form. If you would like your question addresseed here you may submit the question there or leave a comment.

Google says that I can stop spiders from indexing a particular page by putting this in the header: How do I put this into a header of a specific page?

Meta commands for robots and robots.txt files are what you are referring to. The problem with WordPress is that there is no easy way to put a specific meta tag on a single page for this. You could do it for the whole blog but doing it for a specific page is not doable at least to my knowledge.

Because the blog pulls different pieces to create the page you see any changes in the theme would be universal not just for a single page.

That being said there are plugins that will allow you to change some of the meta tags like the title and description but I haven’t seen on that allows for the meta tags for Robots.

You could create a robots.txt file and add that page to it as not being crawlable.

If you want to create a robots.txt file you can find a tutorial at Robtstxt.org.

Hope this helps and if you do find a plugin that will allow for this would love to hear about it.

As always your comments and questions are welcomed.

Mike Paetzold