Table of Contents
Crawl budget definition: What is crawl budget?
Crawl budget it is the number of resources (time/kb) that Google allocates to your web page to crawl on a daily basis. In other words, is the number of pages Google crawls and indexes on a website within a given timeframe.
Crawl budget for SEO: Why is crawl budget important for SEO?
In short: if Google doesn’t index a page, it’s not going to rank for anything. So if your number of pages exceed your site’s crawl budget, you’re going to have pages on your site that aren’t indexed.
As you can imagine this is a very important aspect, especially in large pages (> 1,000 pages). For small pages, it is not advisable to become obsessed with it because you will not notice much difference.
Crawl budget calculation: How does Google calculate/assign crawl budget?
This crawl budget that Google assigns to your website depends mainly on:
- Page Authority
- Frequency of publication or content update (you can have the same number of URLs, but keep them up to date, which increases the crawl frequency)
- Web load time
In other words, the more links your website receives, the more content you publish and the less time it takes to load your website, the greater the crawl budget will be assigned by Google to its robots.
Crawl budget optimization: How can we optimize the crawl budget?
Let’s see the options (beyond the obvious ones) that we have to optimize our website’s crawl budget:
- Remove links to pages with errors or redirects
- Optimize web load time
- Increase the frequency of publication/update
- Increase the number of backlinks to your website
This is the beginning of the ‘beautiful’ part… It is VERY important to prevent Google from crawling pages that we are not interested in positioning (duplicate content, low value for SEO, etc.). For this I recommend:
- Add a no-index value to that page
- Exclude the page from the sitemap (if generated automatically)
- Block that page from your robots.txt:
- Obfuscate links
Do you need help optimizing your crawl budget?
Get in touch with us and we will give you a customized solution
How to Obfuscate links to optimize our crawl budget?
<a class=”pointer” onclick=”location.href=’/legal-notice/'” title=”Legal Notice”> Legal Notice</a>
This plugin allows you to obfuscate links in 3 parts of our WordPress:
- Elements of a menu.
- Links within the content of post and pages.
- Links within the content of HTML widgets.