Copyright © Roddy McKay - 2009 - All Rights Reserved

The search engine spiders need to navigate your site - give them a map.

A site map is essential. Without it, the spiders cannot crawl your website.

Find out more about sitemaps HERE.

There’s an good, free sitemap generator at

I use Rage Software’s Google Sitemap Automator. Its ability to track down broken links makes it money well spent. Click the icon to find out all about it.

This excellent application creates your sitemap, uploads it, verifies it and enters it into the four major search engines - all in a few minutes.

There are other, cheaper applications that do this but I can’t recommend them as I have had reports of problems with them. I’m not going to name names. Try them out for yourself at your own risk.

If you do decide to create your own sitemap, upload it to your server main folder and test it from -

Google Sitemap


The latest version of Google Sitemap Automator uploads your sitemap URL to Bing instead of MSN.

Click the Bing icon to check it out and, for web master’s tools, go to.......

To find out how to submit your sitemap to Yahoo go HERE.

Check out the Apptorial Video about the latest version of Rage Sitemap Automator.

This describes the new method of verifying your site through Google Webmaster tools.

How to BLOCK the Spiders - The robots.txt file.

Now that you have learned how to encourage the search engine spiders, you might be surprised to learn about how to stop them!

Why would you want to do this? Well, you may have published some test pages or ones that are full of secrets that are not included in the navigation menu and you don’t want to let the spiders tell people where they are.

To create your robots.txt file go HERE and to test it HERE. Then upload it to the root folder on your server.

Note that you can tell the spiders to leave your whole site alone if you enter “/” into one of the fields in the online robots.txt file creator.

MobileMe users note that you have to enter your username into the file path: username/test/

Next Page

Get step by step instructions for publishing, hosting and SEO...