Google revised its crawler guidelines to provide clearer details on the Mediapartners-Google crawler, commonly linked with AdSense. Web administrators, content creators, and SEO professionals should pay attention to these updates, especially if they utilize the Mediapartners-Google user agent in robots.txt or similar settings.
What was different
Google modified the Mediapartners-Google documentation to make it less focused on AdSense and more applicable to various Google Ads products.
The initial language in the documentation provided guidance.
Preferences for crawling directed at the Mediapartners-Google user agent impact Google AdSense, as the AdSense crawler visits approved websites to display pertinent advertisements.
The revised wording now reads:
The Mediapartners-Google crawler goes to sites that have agreed to it, to show them ads that are appropriate. Preferences set for the Mediapartners-Google user agent impact different Google products related to ads, such as Google AdSense and Google Ad Manager.
The new documentation provides more information on the Google services related to Mediapartners-Google but does not disclose all the services included.
It does not specify whether it impacts Google Marketing Platform, Local Services Ads, and Google AdMob.
Check out the revised roster of Google’s Special-Case Crawlers.
Image provided by Natalya Bardushka on Shutterstock.