Creating Sitemaps for Google and Other Major Search Engines

Let's look at how some of the major search engines find your website. All search engines use software programs or "robots" (also known as "spiders" or "crawlers') that browse the Web in an automated fashion to identify and index web pages. Once a search engine robot finds a web page, it sends information about that web page back to the search engine's database. The information is included in the database the next time the search engine updates its index. (If you've just created a new website, that's why it takes a bit of time for any of your pages to be displayed by a search engine.)

Many search engine robots, including Googlebot, Yahoo's robot ("Slurp"), and MSNBot, find certain types of content more readily than others. For example, search engine robots can't "see" images or frames, so text content is very important for search engine optimization. To feed the gods of page ranking, making a page particularly tasty for a search engine robot is a good idea. And that's where the XML sitemap comes in.

What is an XML Sitemap?

You may be used to seeing HTML sitemaps, which are often included in websites to help users find their way around the site. These typically have the same look and feel as other web pages, but present a summary of all the pages on the site and the connections between them.

An XML sitemap, which is what you'll need to feed the robots, is not a pretty thing to look at. It's essentially a list of all the URLs that correspond to your website's web pages in XML code.

An XML sitemap also includes information ("metadata") about each URL such as: when the page associated with the URL was last updated; how often the page is updated ("change frequency"), and how important that page is relative to other pages (using a 0.0 to 1.0 scale, with 1.0 being most important).

Having an XML sitemap doesn't guarantee a search engine will find your site, but it definitely increases your odds.

The code for generating an XML site is not complicated, but if you have a large number of pages, it's a pain. So why not avail yourself of the free programs being offered by some kind webmasters? The one provided by XML-Sitemaps.com will map a site of up to 500 pages. Another free site is Sitemaps^pal and there are many others.

Creating a Sitemap

Step 1: Generate an xml file. If you are using one of the free programs described above, you simply enter your URL into a field provided and the sitemap generator determines how many links are present in your site and will generate the appropriate XML code. You may want to fool with the change frequency and page rankings identified, since the free programs will set some defaults. Alternatively, you can generate all your own code using the instructions provided at sitemaps.org .

Step 2: Save the file and name it - for example, "sitemap.xml."

If you coded your own sitemap or you just want to see if the file you have meets the right standards, you may want to check it using Smart IT Consulting's free sitemap validator, which will let you know if there are any errors.

Step 3: Once you're satisfied, upload the file to your website directory.

Now Submit Your Sitemap to Search Engines

Google requires a Google account to sign in, but it's free. You will need to prove ownership of your site. They'll provide you with a unique metatag to include in each of your web pages or a separate file that you can just save and upload - it's up to you which method you use. Once Google determines that you actually own your site, you simply provide them with the URL to your sitemap file, e.g., http://www.mycompany.com/sitemap.xml.

Having a Google account is handy in any case because Google provides many useful webmaster tools. These allow you to monitor website traffic, identify links to your site, and obtain information about PageRank, etc. An extremely nice feature is the information Google provides about the top search queries that lead people to your site.

Yahoo! also requires you to prove you own your site and gives you a file to upload into your directory. Like Google, once you're verified, you can submit your XML sitemap.

MSN does not require you to prove you own your site, but they also don't make it easy for you to directly submit your sitemap. You can submit your home page URL (e.g., http:www.mysite.com) to a web page MSN provides and the MSNBot robot will eventually find your site. There's also a trickier way to give MSN your site info and that's by using this URL: http://api.moreover.com/ping?u=http://www.mycompany.com/sitemap.xml, inserting your company URL where I've indicated "www.mycompany.com."

Other search engines have their own idiosyncratic rules. For example, Ask.com also doesn't require authentication, but you do have to know the secret code to get their robot to wake up and see your sitemap. (Well, not so secret, it's describe in the text of their website, but it's rather hard to find.) Copy the following into the address bar of your browser: http://submissions.ask.com/ping?sitemap=http://www.mysite.com/sitemap.xml. Hit enter and you'll be put in the queue. You may have to do this more than once over the next few months.

Site Maintenance

If you add web pages or change the frequency at which you update your site, don't forget to update your sitemap.xml file. You don't have to resubmit to the individual search engines (they're searching for the file name and you won't change that), but you should make sure you upload a current sitemap.xml file to your website.

How do Sitemaps Affect Page Ranking?

You won't improve your Page Rank by indexing your website with a sitemap, but your site map will at least ensure that your web pages are crawled and indexed by search engines.

Now write that great content so you get a good Page Ranking!