One of the first checklists on any SEO audit is, “Does the website have a sitemap?” According to Google, nowadays, your website can be crawled and indexed easily without one. At SEO Gone Wild, we have seen websites get crawled okay without even setting up Google Search Console. According to a recent article in Search Engine Journal, there are certain situations where your website does not need a sitemap. However, we find that adhering to having a sitemap is a good practice. Here is what we recommend:
Use an Automatically Generated XML Sitemap
Most ecommerce platforms and website CMS’s will automatically generate a sitemap for you. WordPress works best with Yoast SEO. Shopify, BigCommerce, and other platforms will generate a sitemap for you. Whenever you add new content to these types of sites, the new page or post will automatically be generated for you.
Software like Screaming Frog can generate a manual sitemap, but you will have to update this every time you generate new content for your website. You should use a standard XML sitemap URL parameter, too. The standard XML sitemap parameter is /sitemap.xml. If you don’t specify a standard sitemap parameter, it will be harder for search engines to find your sitemap and crawl your website.
Avoid Dynamic Sitemaps
Many platforms will create a dynamic sitemap, one that is rendered browser side. While your platform may use a dynamic sitemap, you should still be okay if you test your sitemap in Search Console. But, search engines may not crawl these as readily as a static sitemap.
Specify Your Sitemap In Your Robots.txt File
The robots.txt file is the first thing that search engines review when they crawl your website. If you forget to submit your sitemap to Search Console and the bots find your sitemap in your robots.txt file, this will still help search engines crawl your sitemap. But, you should always submit your sitemap and specify it in your robots.txt file.
When Must You Absolutely Have a Sitemap?
There are three criteria when you must absolutely have a sitemap. They are:
- When your site is large
- When you have isolated pages that aren’t internally linked
- When your site changes frequently (with new content)
Does Having a Sitemap Guarantee That I Will Have Every Page Indexed and Ranking?
No. Google may choose not to include certain pages in its index. It also will not help with rankings. Essentially, a sitemap is a way of telling Google where everything is on your website. It’s kind of like an index for search engines to find all the useful content that need to be included for search results.
How To Optimize Your Google XML Sitemap’s Crawl Budget
If you create a manual sitemap, make sure that you audit it monthly so that you can remove any 301s, 302s, or 404 pages that are no longer on your website. You should also consider breaking your sitemap up into smaller sitemaps. Many large ecommerce websites will automatically generate sitemaps for your pages, categories, products, and blogs.
Does your website also use tags and categories for its blog? Consider NOT including these in your XML sitemap and block them via robots.txt to optimize your site’s crawl budget even more.
How To Make Search Engines Find Your Google XML Sitemap
If you have Search Console set up, make sure that you go to the “sitemaps” section. Simply copy and paste your URL for your sitemap into the sitemap field and click submit. Google will take care of the rest.
How Often Does Google Crawl Your XML Sitemap?
No one really knows how often Google crawls sitemaps. The best answer that we could find was that Google crawls a site up to six times per month. If you have a piece of content that you would like to add to the index, you can use the “Fetch URL” in the search console. The message under this tool says that Google will index the page in 1 to 2 weeks. However, in experiments we have done, we’ve seen the “Fetch URL” tool add pages to Google’s index within seconds.
You should use the “Fetch URL” tool for a few pages. But if you have a large website and you add several products or pages, we recommend letting Google crawl your website naturally. Also, keep in mind that using the “Fetch URL” tool repeatedly for the same page will not help increase the likelihood of the page getting indexed.
How Often Should You Resubmit a Google XML Sitemap?
You should theoretically never have to resubmit your sitemap. If you automatically generate it, like with WP’s Yoast SEO plugin, Google will pick up updates to your site automatically. If you manually update your sitemap and you have a large website, we recommend resubmitting your sitemap.
What’s the Difference Between a Google XML Sitemap and an HTML Sitemap?
An XML sitemap is a specific web document for search engines. HTML sitemaps are great for humans. They can be a page that lists all the different URLs on your website. We recommend placing an HTML sitemap as a footer link on your website. Make sure that you don’t just drop a list of all your pages. We recommend organizing your HTML sitemap exactly how your website would look as a Table of Contents in a book. You might also want to include a brief description after each link on what the particular page is about.