Meta Allows Crawling of Advertiser Websites: Terms Effective October 30 and Scraping Insights

Starting from October 30, 2026, Meta's updated Commercial Terms will come into effect for everyone using Facebook, Instagram, Messenger, and WhatsApp for commercial purposes: launching ads, managing a brand page, selling through a catalog. A paragraph that lawyers and media noticed only in mid-October has been included in the new text: businesses allow Meta to automatically bypass their sites — to index, cache, collect, and analyze content. This clause cannot be opted out of separately; the only way to avoid it is to stop using Meta's business products.

This story is interesting not only to publishers. This is the same Meta that has been in legal battles with scrapers, including Bright Data, for years and lost a key dispute specifically regarding the question of "who is bound by the terms of use." Now the company is using the same contractual mechanism to its advantage. Let's analyze what is written in the terms, what Meta has managed to amend after the scandal, and what practical conclusions can be drawn for those who collect data themselves or protect their sites from bots.

What is specifically stated in the new Meta terms

Meta announced the update in early October in the Business Help Center and in Meta Business Suite. It affects several documents: Commercial Terms, product catalog terms, self-service advertising terms, and terms for generating ad creatives using AI. The previous version of the Commercial Terms was in effect since March 6, 2024, and, according to PPC Land's analysis, there was no language regarding crawling in it.

In the new version, Section 1 has been renamed from "Licenses" to "Licenses and Rights You Grant to Meta." Key phrases in the original version include:

• The business allows Meta "or any person acting on its behalf" to use automated software to access, index, cache, analyze, collect, and crawl domains, pages, and URLs, as well as to periodically retrieve updates;
• Websites "related" to the business's products or services or accessible through the use of Meta's products fall under this clause. The text does not define what "related" means;
• The content obtained by Meta may be used to "provide, develop, improve, research, train, and maintain the integrity" of its products;
• Meta has the right to create "test credentials" to access these resources. The practical implications of this are not explained in the terms.

Consent is framed in a familiar way for platforms: if you continue to use business products after being notified of the changes, you agree to them. A post on the company page on October 31 is already considered acceptance. There is no partial opt-out from the crawling clause.

Reactions from lawyers and Meta's response

The story was raised on October 15 by Nieman Lab, which surveyed three legal experts. Their assessments were harsh:

• Matt Topic (Loevy & Loevy) stated that it is unclear whether a court would recognize such a clause, but publishers "should assume that it will be enforceable";
• James Grimmelmann (Cornell Law School) described the clause as a clear permission for Meta to train AI on publishers' content. He also pointed out the "test credentials": in his opinion, this could allow for the creation of fake users and bypassing registration walls;
• Benjamin Sobel (University of Wisconsin) highlighted a potential issue with "binding": to use the necessary products, a business is forced to agree to crawling as well.

Daniel Coffee, head of the News/Media Alliance, called the terms "outrageous and likely illegal." Jason Kint from Digital Content Next questioned why Meta would require broad rights and explicitly state AI training in a binding contract if, according to them, nothing is changing.

Meta's position: On October 16, communications director Andy Stone called the publications "false" and stated that there is no change in how Meta uses web crawlers. According to the company, the language about crawling was previously scattered across various documents, including terms for generative AI in advertising, and is now simply consolidated in one place. Content owners, according to Meta, can still manage access through robots.txt.

On October 17, the text was amended. According to Nieman Lab, the word "train" was removed from the list of purposes, and usage was limited to advertising and business products instead of all Meta products. Some publications, citing The Verge, report that the controversial clause has been completely removed. However, experts consulted by Nieman Lab believe that the broad rights for crawling still remain in the terms, and business accounts do not have the option to opt out before October 30. The final wording should be checked on the Commercial Terms page before they come into effect.

Double standard: how Meta lost to Bright Data and what it learned

To understand why lawyers are so concerned, one must recall the case Meta Platforms v. Bright Data. On January 23, 2024, Judge Edward Chen (Northern District of California) ruled in favor of Bright Data in a breach of contract lawsuit. The court's logic: Meta's terms prohibit scraping but only bind users who have logged into an account. Collecting public data without logging in does not fall under these terms. The court also found no evidence that Bright Data collected data only available after login and rejected the idea that a former user is bound by the terms indefinitely.

The takeaway from that case, which the industry has relied on since then: a contract binds the party that accepted it. The new Commercial Terms operate on the same principle, only in reverse. Meta cannot force a third-party site to allow its crawler, but it can obtain consent from the owner through a contract that they have already accepted to run ads. And robots.txt is a weak consolation here: the terms do not mention it, and lawyers fear that Meta may refer to the contract as a stronger permission than the prohibition in the technical file.

Which crawlers does Meta use on websites

In the documentation for webmasters, Meta currently lists five agents. Depending on which one comes, the robots.txt prohibition may or may not apply:

• meta-externalagent — collects pages for training basic AI models and for indexing in products. One token serves both tasks, so indexing cannot be allowed while separately prohibiting training;
• meta-webindexer — indexes the web for search and citations in Meta AI;
• meta-externalads — a crawler for advertising and business products, which is exactly the area that Meta has limited in the new clause;
• meta-externalfetcher — fetches individual links upon user request, including for AI agent tasks. According to the documentation, it may ignore robots.txt because it is a human request;
• facebookexternalhit — builds link previews when shared. It may also bypass robots.txt for security and integrity checks.

What should a website owner with a Meta business account do

If you are selling through Instagram or running ads on Facebook, you cannot change the contractual part. The technical part is in your hands.

1. Decide what you are willing to give up. Link previews (facebookexternalhit) are necessary; otherwise, posts from your site will lack images. The advertising crawler (meta-externalads) is needed if you use a product catalog and dynamic ads. Training models (meta-externalagent) does not provide any benefit for a store.
2. Specify rules in robots.txt for individual agents, for example: User-agent: meta-externalagent and Disallow: /. Meta warns that crawlers cache the file, and changes take effect within 24 hours.
3. Check the logs. Filter requests by the strings meta-externalagent, meta-externalads, and meta-webindexer, and count which sections they bypass and how many. This is the baseline to compare behavior after October 30.
4. Restrict valuable content at the server level, not just through robots.txt: prices, knowledge bases, paid materials — behind authentication or with User-Agent restrictions on CDN. robots.txt remains a voluntary agreement, not a lock.
5. Save the current version of the terms (screenshot or PDF with date). The wording has already been amended within a couple of weeks after the announcement, and in case of a dispute, the version in effect on a specific date will be useful.

What this means for those who scrape data themselves

For teams that collect data from platforms, Meta's story is a vivid illustration of where the legal boundary actually lies.

• Login is a signature on the contract. The ruling in Bright Data protects the collection of public data without logging in. Once a scraper operates from an account, the platform's terms apply to you, and the prohibition on automated collection becomes a contractual obligation. We elaborated on the practical implications of this in our analysis of the risks of violating Terms of Service when working through proxies.
• Separate infrastructure. Business accounts through which you run ads and data collectors should not overlap in accounts or IPs. For public collection without login, data center proxies are usually sufficient, while for platforms with strict anti-bot filters, residential IPs with rotation are necessary.
• robots.txt is a signal, not a law, in both directions. Meta itself acknowledges that two of its agents can bypass it. However, for your project, adhering to robots.txt remains an argument of good faith if a dispute arises. The practice is described in the material on ethical scraping and robots.txt.
• Keep an eye on changes to the terms. Platforms amend them through "continued use means consent." It is useful to compare the current terms of the platforms you work with under your account with saved copies quarterly.

Conclusion

The new Commercial Terms from Meta illustrate the direction of the battle for web data in 2026: from technical barriers to contractual ones. The company that lost a dispute over who is bound by its terms is now using those same terms to obtain consent to bypass the sites of its advertisers. After the amendment on October 17, the wording became softer, but the option to opt out is still unavailable, and the terms come into effect on October 30. Website owners should configure robots.txt according to agents and save a copy of the terms, while teams that scrape data should double-check where their collection is done under login and what risks this entails.