A site with several hundred pages, half of which are never crawled by Google, we’ve all seen that. The problem rarely comes from the content or internal linking: it’s often the XML sitemap that sends conflicting signals to the crawling bots. Optimizing the structure of your sitemap is about cleaning up this communication channel so that each submitted URL truly deserves to be crawled and indexed.
The sitemap ping is dead: adapt your submission in 2025
Many SEO guides still recommend sending an HTTP ping to Google after each sitemap update. This method is outdated. Google deprecated the sitemap ping endpoint in June 2023, meaning that requests to /ping?sitemap=… have no effect anymore.
In practice, we end up with two official submission channels. The first is direct submission via Google Search Console, in the Sitemaps report. The second is declaration in the robots.txt file with the Sitemap: directive followed by the full URL of the XML file. These two methods can coexist, and this is indeed the recommended combination.
The classic trap: technical teams automating a ping with every deployment, thinking they are speeding up indexing. This script has been running in vain for over two years. If you use an automated publishing system, ensure that the notification relies on the Search Console API or on IndexNow (supported by Bing), not on an outdated ping. On the Amplement site, the sitemap structure clearly illustrates how to organize URLs to facilitate discovery by search engines.
XML Sitemap and crawl budget: what Google really accepts
A misunderstanding persists in many SEO strategies: the sitemap is seen as an indexing instruction. The recently updated Google Search Central documentation clarifies this point. The sitemap is a discovery file, not a guarantee of crawl or indexing. Google treats it as a hint among others.
This distinction changes how to structure the file. A sitemap that lists all the site’s URLs without prior sorting only dilutes the signals. Crawling bots have limited time per domain. If half of the URLs point to noindex pages, 301 redirects, or low-content pages, the crawl budget is wasted on unnecessary resources.
What to exclude from the sitemap
- Pages with a noindex tag or blocked by robots.txt, which send a conflicting signal when they appear in the sitemap
- URLs with sorting, filtering, or pagination parameters that generate duplicate content (typically catalog pages with ?sort=price&page=3)
- Old URLs redirected in 301 to other pages, as they consume a crawl to result in a redirect
- Utility pages with no search value (legal notices, login pages, terms and conditions)
The desired result: a sitemap where each URL returns a 200 code, contains unique content, and has an index tag. Nothing else.
Splitting the sitemap into an index for large sites
When working on an e-commerce site or a media site with thousands of pages, a single sitemap file quickly becomes unmanageable. The standard allows for an index file (sitemap index) that points to several sub-sitemaps. This thematic splitting improves readability for bots and the SEO team.
Generally, we split by content type: one sub-sitemap for product sheets, one for blog articles, one for category pages, one for images or videos if the site publishes many. This segmentation allows for quick identification of which group of pages is causing a crawl issue in Search Console.
Lastmod: the only tag that really matters
The priority and changefreq tags are largely ignored by Google. Feedback varies on this point among engines, but on Google’s side, the documentation is clear: these parameters are no longer taken into account in crawl prioritization.
The lastmod tag, on the other hand, has a concrete role. It indicates the actual last modification date of a page. Update lastmod only when the content changes significantly (not with every technical rebuild) sends a freshness signal that Googlebot can use. Forcing a daily lastmod date on all pages, a common practice on some CMS, is like crying wolf constantly: the bot eventually ignores the signal.
Specialized image and video sitemaps: when to activate them
Sitemaps are not limited to HTML pages. Google supports specific extensions for images and videos, with dedicated tags (image:loc, video:content_loc). The interest entirely depends on the site’s profile.
For a site where visual traffic represents a significant share of visits (photographer portfolio, real estate listing site, recipe platform), a dedicated image sitemap allows for indexing visuals that the crawler would not discover by only browsing the HTML. For a typical text blog, adding an image sitemap brings almost nothing.
The same logic applies to videos: if you host your own video files (not YouTube embeds), a video sitemap specifying duration, thumbnail, and description helps Google display rich results. A specialized sitemap is only justified if the content in question has its own traffic potential in search.
Verification and maintenance of the XML sitemap
An optimized sitemap at a given moment degrades over time. Pages are deleted, others redirected, new ones appear without being added to the file. Regular checking via Google Search Console allows for spotting coverage errors related to the sitemap: URLs returning 404 errors, pages indexed that shouldn’t be, or conversely valid URLs missing from the file.
The “Sitemaps” report in Search Console displays the number of discovered URLs and the number actually indexed. A significant gap between these two figures signals a quality issue in the file. This is the first indicator to monitor, even before positions or organic traffic.
Structuring your sitemap correctly does not produce miracles on positions. It is a technical hygiene task that ensures bots crawl the right pages, at the right time, without wasting resources on unnecessary URLs. The gain is mainly measured in the speed of indexing new content and in reducing coverage errors.
