Google: Disallowed URLs Through Robots.txt Does Not Affect Crawl Budget

Jun 21, 2019 - 7:40 am 0 by

Google Disallow

Gary Illyes from Google added to the big crawl budget article he published on the webmaster blog in January 2017 a new question and answer. It basically says that if you disallow a URL in your robots.txt file, then those URLs do not affect your crawl budget.

He added this question and answer:

Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?
A: No, disallowed URLs do not affect the crawl budget.

He posted last night that he added the Q&A for a former Googler, Pierre Far:

More details:

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 27, 2024

Mar 27, 2024 - 4:00 pm
Google Ads

Google Local Service Ads Sends Email Asking You To Upload Photos

Mar 27, 2024 - 7:51 am
Google Maps

European Hotels Can Remove Pricing Information From Google Hotel Search

Mar 27, 2024 - 7:41 am
Google Search Engine Optimization

SEOs, Please Don't Remove Contact Us & About Us Pages

Mar 27, 2024 - 7:31 am
Misc Google

Google Search Developer Docs Gain AI Generated Help Features

Mar 27, 2024 - 7:21 am
Google Maps

Google Local Reviews Reactions Notice

Mar 27, 2024 - 7:11 am
Previous Story: Summer Season Logos From Google, Bing, Yahoo & More