Meta removes contested AI‑scraping clause from terms, sparking debate among publishers

Meta has removed a clause from its commercial terms that critics said permitted the company to scrape news sites for AI training, a change that could alter how publishers enforce robots.txt.

Key points

• Meta deleted language that could allow scraping of news sites for AI training.
• The clause was interpreted as overriding robots.txt preferences.
• Meta’s spokesperson said the change does not affect crawler use.
• The update was made on 2026‑10‑17 after public backlash.
• The removal reflects ongoing regulatory scrutiny of AI data sourcing.

What happened

On 17 Oct 2026, Meta updated its Commercial Terms page, eliminating a phrase that read: “Meta may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain …” The clause had been flagged by experts such as Neiman Lab and Cornell professor James Grimmelmann, who argued it granted Meta the right to scrape content even when robots.txt blocked it.

In response, Meta’s vice president of communications, Andy Stone, and spokesperson Daniel Roberts issued statements. They denied any change to crawler behavior, citing, “There is no change to how Meta uses web crawlers; any suggestion to the contrary is false.” Roberts added that the consolidated terms applied only to ads and business products and that the updates do not grant new rights superseding site preferences.

The removal follows a broader backlash over AI training data practices and similar controversies that have prompted calls for clearer terms and stronger enforcement of content‑owner signals.

The backstory

Since the rise of large‑language models, tech firms have sought vast amounts of public web content to train AI. Companies often rely on web crawlers and ignore robots.txt directives, leading to legal and ethical concerns. Meta’s earlier terms had been criticized for giving it broad rights to use scraped content.

The dispute intensified after a 2025 report highlighted Meta’s use of news articles in its AI products. Regulatory scrutiny, coupled with pressure from publishers, pushed Meta to revise its terms in 2026, culminating in the latest update.

Why the clause mattered

The clause was a legal lever that could have allowed Meta to claim ownership over scraped news text, undermining publishers’ control and revenue models. By removing it, Meta reduces potential liability but also signals a shift toward more transparent data‑sourcing practices. The change may encourage other firms to tighten their own terms, but it also leaves open the question of whether Meta still aggregates data without explicit permission.

Implications for AI data sourcing

The removal does not alter Meta’s crawler operations, meaning it can still gather data; however, it must now respect robots.txt signals more visibly. For AI developers, this shift could mean tighter compliance requirements and a need to verify source permissions before training models. Publishers may see a temporary reprieve, but the broader industry must confront how to balance AI innovation with content‑owner rights.

What builders and businesses should do

• Review your own terms of service for clauses that could be interpreted as allowing data scraping.
• Monitor changes to major platform policies that affect data access.
• Implement robust robots.txt handling in your crawlers to respect site preferences.
• Conduct legal risk assessments around using publicly available content for AI training.
• Engage with industry groups to advocate for clearer data‑usage standards.

The bottom line

Meta’s term revision removes a potentially sweeping data‑scraping right, but the company’s crawler practices remain unchanged. Watch for further policy updates and court rulings that could redefine AI data sourcing.