Google: Crawl Rate & Crawl Demand = Crawl Budget

Jan 17, 2017 - 7:52 am 2 by

Google Crawl Budget

Several weeks ago, Google told us crawl budget is not what we think it is and promised to write a blog post explaining what Google defines as crawl budget. Well, Gary Illyes posted on the Google blog What Crawl Budget Means for Googlebot.

In short, "crawl rate and crawl demand together," Google said is how they "define crawl budget as the number of URLs Googlebot can and wants to crawl."

Crawl rate limit is how much GoogleBot allows itself to crawl your pages in order not to overload your server and do harm. It is the "number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches." Google won't crawl too much to take down your server and it will respect the limit you set in the Google Search Console.

Crawl demand is the demand for Google to crawl your pages, new and old pages. Demand is determined by popularity of your site and URLs and the staleness of the Google systems attempt to prevent URLs from becoming stale in the index. Even if you have more crawl rate left, if demand doesn't require more crawl, GoogleBot won't crawl.

Google said, these two make up crawl budget. "Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl," Google wrote.

What impacts crawl budget?

  • Faceted navigation and session identifiers
  • On-site duplicate content
  • Soft error pages
  • Hacked pages
  • Infinite spaces and proxies
  • Low quality and spam content

Crawl rate is not a ranking factor, said Google. Google wrote, "an increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it's not a ranking signal."

For more details and a lot more FAQs in the blog post.

Forum discussion at Twitter and Google+.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google SGE In Wild, Stop Doing SEO For Google, Maps & Shopping Features & Google Ads Safety Report - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Video Recaps

Search News Buzz Video Recap: Google SGE In Wild, Stop Doing SEO For Google, Maps & Shopping Features & Google Ads Safety Report

Mar 29, 2024 - 8:01 am
Google Ads

Google Ads Posted Zero New Feature Announcements In March 2024

Mar 29, 2024 - 7:51 am
Google Search Engine Optimization

Correction: Web Stories Still Show In Google Images & Google Fixes Search Console Reporting Bug

Mar 29, 2024 - 7:41 am
Google Maps

Google Business Profiles New Add Amenity To Your Website

Mar 29, 2024 - 7:31 am
Web Analytics

New Microsoft Copilot Referrer Data In Google Analytics

Mar 29, 2024 - 7:21 am
Google

Google Search Recommendation For "Google Can Help You Stand Out"

Mar 29, 2024 - 7:11 am
Previous Story: Tell Google What Technical SEO Topic They Should Write About