AI companies are cutting checks to marquee news brands while the long tail of independent publishers watches from a distance. The most recent are Meta’s new set of content deals to pipe real-time news into Meta AI, from a range of outlets that include CNN, Fox News and USA Today (and The Daily Caller, People and Le Monde). It’s a momentous change in how much data gets licensed for AI — and yet, it is not a rising tide for all.
Why Major AI Licensing Deals Favor Big News Brands
Major publishers are making money because they have leverage. An aggregator tracked by Columbia Journalism School’s Tow Center for Digital Journalism lists more than 120 licensing deals that AI operators have signed with news publishers since mid-2023, among them in the past year or so: some high-profile pacts like OpenAI with The Financial Times and Perplexity with The Washington Post and Los Angeles Times for its Comet browser. Recent 2024 content-licensing deals with Reddit by both Google and OpenAI highlight the trend: AI companies need high-quality, trusted sources to improve outputs even while they seek to reduce liability.
These are not just data-access deals; they are a wager by AI platforms that premium, vetted content can make their products stand out. The money and the legal jargon go where publishers can talk tough — which these days means ones with readership scale, well-known brands and litigation budgets.
Smaller Outlets Shoulder More Risk in the AI Shift
For local newsrooms, niche blogs and solo creators, the calculus is different. They encounter the same risks of extraction without equivalent bargaining power — and at times, are collateral damage as AI responses replace clicks. In a study this summer by the Pew Research Center, Google’s AI Overview reduced clickthrough among survey respondents from 15 percent to 8 percent on average. Google says that it’s not seeing an overall traffic decline, and asserts that AI Overviews actually send “higher-quality” visits, yet has provided no data to back up those assertions.
There is also a cost of being crawled. This year Wikipedia warned about an influx of bots, particularly AI ones scraping Wikimedia Commons images that were putting a strain on infrastructure and budgets. And one food blogger told Bloomberg how AI recipe summaries can be inaccurate and suck traffic away from creators, with such a loss of revenue potentially career-ending.
Law and Liability Are Still in Development
Courtrooms are offering mixed signals. In February, a federal judge determined that executives from a now-shuttered AI start-up violated Thomson Reuters’s copyrights as they built an AI rival to its Westlaw service. In June, another judge ruled that buying books and scanning them for training purposes could be fair use, but downloading pirated versions would not. The schism highlights why publishers big enough to flex their legal muscle are rushing to obtain explicit licenses, even as smaller players languish in wait-and-see limbo.
Trade groups have lobbied for more wholesale solutions. The News/Media Alliance has pushed for structured licensing as well as transparency around training data, saying news is a public good and expensive to produce. Without clearer rules or enforceable standards, the market is now rewarding those that are big and strong enough to fight — or worth paying.
Cloudflare’s Experiments With Pay Per Crawl
There may be a fledgling tech path for everyone else. Cloudflare, which already has a service that allows sites to block AI bots, recently added an optional “pay per crawl” at the edge of its network that lets approved AI crawlers access content for a price. CEO Matthew Prince once cast it to us at Web Summit as a survival tactic: If AI tools don’t send traffic, the ad-supported web fails. He also called out Google over the links between its traditional indexing and AI crawling, which prevented the selective blocking of robots.txt.
Google says publishers can opt out of having their content used to train Gemini, but it does not cover AI Overviews. A “nosnippet” command blocks text previews but it also limits exposure in classic search. Cloudflare’s pay-per-crawl is still in private beta, and its levels of uptake among small newsrooms are unclear. Yet it does suggest a model of metered access that could unbundle AI training from indiscriminate scraping.
Licensing Standards and Enforcement for AI Crawling
Standards could bring the long tail to market. The nascent Really Simple Licensing framework — backed by firms like Reddit, Yahoo, Ziff Davis — enables publishers to declare machine-usage terms in a machine-readable fashion. Paired with bot-blocking or access gateways, it could become a sort of enforcement checkpoint: pay to crawl, or get denied at the edge.
For smaller sites, the short-term playbook is practical: make public AI-use policies, implement bot management to weed out unauthorized crawlers and monitor log data for scraping behavior that looks different. Forming alliances or consortia may enhance negotiating power and generate common price references. “Even if it’s small, it makes a really big difference to the outcome” when AI companies need varied, high-quality data.
The Next Phase of AI Content Economics for Publishers
Increasingly, AI platforms will battle for customers on data provenance. That buys original reporting and exclusive archives, which in turn lowers legal risk — and raises trust signals, particularly important as regulators, courts and consumers shine a light on the black box of AI outputs. The risk is a pay-to-play internet where only the largest publishers get paid and everyone else contributes free training data.
The solution won’t involve something big; it will be plumbing. Metered crawling, machine-readable licenses and auditable logs can turn scraping into a priced transaction. Should that infrastructure scale, AI’s hunger for content might eventually come to represent a steadier stream of revenue for thousands of smaller publishers — rather than a windfall for the few who already had the means to be heard.
