Gold Penguin Logo with Text

OpenAI Is Now Crawling The Web – Say Hi To GPTBot

OpenAI soft-launched their new web crawler GPTBot to enhance AI accuracy, capabilities, and safety for users.
Photo by Mariia Shalabaieva on Unsplash
Photo by Mariia Shalabaieva on Unsplash
August 7, 2023 11:40 pm

OpenAI recently soft launched its new web crawler ‘GPTBot’ to enhance and improve AI accuracy, capabilities, and safety for users and admins.

GPTBot is designed to gather data to improve the performance of the company’s AI models (such as GPT-4) by enabling the crawler to access the websites’ data that will contribute to its own data pool.

“Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety,” OpenAI said in a release.

(Web crawlers are computer programs that collect data to index the content of websites across the internet to improve search engine results).

The crawler is also capable of restricting and filtering out sources that violate data privacy and policies implemented by OpenAI. This includes, text that violates their policies, paywall-restricted sources, and sources that gather personally identifiable information.

Consequently, the company added a feature to safeguard users by providing privacy controls to web administrators to enable or disable the crawler’s access to their sites.

More so, OpenAI gave out instructions to customize, enable or disable GPTBot’s access by modifying the robots.txt file. 

In addition, they also released the IP egress ranges, to provide transparency as to where the traffic source comes from on the web admins’ sites.

It is definitely nice to have improved security options that enable web owners to control what programs could access their data. Having the privacy controls to enable or disable the crawler only proves that AI systems exert effort to provide limitations. 

Article Sources:
Stay Updated On The Latest AI Tools
Get notified with the biggest releases and tips in Artificial Intelligence delivered straight to your email. We don't sell your data.
Written by Andy Hoo
Andy is an investigative tech journalist at Gold Penguin. Besides being a journalist with the heart and mind for truth and credibility, he is also a passionate content creator who loves making informative and recreational videos. He writes all types of news in the technology & AI industry.
Notify of

Inline Feedbacks
View all comments