Auto Discovery of Sitemaps

I was surprised to see Yahoo come out with the news that sitemap.xml files can now be referenced within your robots.txt file. It’s a great idea and I’m glad to see Yahoo announce it. I can only imagine that this was because it was their idea?

sitemap.xml files let you tell Google, Yahoo, and Microsoft (yes they do have a search engine ) exactly which pages exist on your site and how often they are updated. This is a more efficient and effective method than just letting the search engines spider based off of links.

the robots.txt file tells search engines which pages not to spider. The search engines always look for this file and should respect any listings you have. Combining the two, at least referencing the sitemap.xml in the robots.txt, is like peanut butter and jelly.