Google Unveils New Crawlers for Image and Video Content
Google recently announced the introduction of two new crawlers specifically designed to scrape image and video content for “research and development” purposes. While the official documentation does not explicitly state this, it is widely assumed that blocking these new crawlers will not have any impact on website ranking.
Introduction of GoogleOther Crawlers
The two new crawlers are actually variants of Google’s existing GoogleOther crawler, which was initially launched back in April 2023. The original GoogleOther crawler was primarily intended for use by Google product teams for conducting research and development through one-off crawls.
The new variants, GoogleOther-Image and GoogleOther-Video, are specifically tailored for crawling binary data, such as image and video content. Unlike text data which can be viewed in text files, binary files like images, audio, and videos require specialized crawlers to extract relevant information.
GoogleOther-Image Crawler
The GoogleOther-Image crawler includes the following user agent tokens:
- GoogleOther-Image
- GoogleOther
Its full user agent string is: GoogleOther-Image/1.0
GoogleOther-Video Crawler
Similarly, the GoogleOther-Video crawler utilizes the following user agent tokens:
- GoogleOther-Video
- GoogleOther
Its full user agent string is: GoogleOther-Video/1.0
Updated User Agent Strings
Google has also updated the user agent strings for the regular GoogleOther crawler. The data sent to servers to identify these crawlers remains consistent, with the technology used being Chrome. Publishers can continue using the user agent token “GoogleOther” for blocking purposes.
The updated GoogleOther user agent strings include:
- Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
GoogleOther Family of Bots
It is important for publishers to be aware of these new crawlers, as they may appear in server logs. Understanding these bots as legitimate Google crawlers can help publishers make informed decisions about opting out of having their images and videos scraped for research and development purposes.
For more information, refer to the updated Google crawler documentation on GoogleOther-Image and GoogleOther-Video.
Image/Photo credit: source url
