Website owners can use a robots.txt file to tell participating AI crawlers not to use their content to train models. OpenAI, Google, Microsoft and Anthropic are backing a shared opt-out signal, though the rule depends on crawlers choosing to respect it.
The file sits in a website’s root directory and provides instructions for bots. To block all AI crawlers that honor the instructions, site owners can add a section headed “# AI crawlers” and include “Disallow: /”. They can also target a specific crawler by naming it with a “User-agent” line, or restrict access to particular paths. The updated file must then be placed in the website’s source files.
Robots.txt is not a technical barrier: bots that ignore its instructions can still access a site. OpenAI has encouraged other AI providers to honor the signal, but the approach is most useful against compliant crawlers.
Other options include requiring visitors to sign in, limiting the number of requests from an IP address, and using captchas. Each has drawbacks. Login requirements can discourage people from visiting, while IP limits may be less effective against scrapers that rotate addresses. Captchas can frustrate users and may not stop more capable AI systems.
These measures can help reduce unwanted scraping, but they may also affect the experience of legitimate visitors. Website owners must weigh that trade-off when choosing how to restrict access.
Sources: TechRadar.