source: techcrunch ai: cloudflare’s new policy pushes ai companies to pay for publishers’ content
level: business
starting september 15, 2026, cloudflare will change its default settings to block mixed-use crawlers from any pages that host ads. these crawlers blend traditional search indexing with ai training and agentic services. the new defaults apply to new customers, new sites from existing customers, and all existing free customers. site owners can adjust the settings if they choose.
cloudflare says most website owners want their content discoverable via search and ai services but also want to protect their intellectual property. the company notes that the largest search engine has access to about twice as much information as other ai companies because it makes it hard for customers to stay discoverable without being used for ai. google has said it provides a separate bot, google extended, to opt out of ai training without affecting search.
cloudflare is also expanding its pay-per-crawl marketplace into pay-per-use, letting publishers charge ai companies when their content creates value, not just when it is fetched. initial partners include ceramic.ai and you.com. publishers get paid when their content appears in ceramic's ai search results or when you.com accesses premium content. cloudflare says over half of crawl traffic from ai crawlers re-fetches unchanged pages, wasting resources.
why it matters: this shift could limit free access to web data for ai training and force clearer separation between search and ai bots, affecting how ai models gather information.
source: techcrunch ai: cloudflare’s new policy pushes ai companies to pay for publishers’ content