AI Firms Are Getting Around Web Protocol That Lets Publishers Block Scraping (mediapost.com)

The debate over content scraping took a new turn on Friday when TollBit, a content licensing startup, alleged that artificial intelligence companies are bypassing a web standard used by publishers to block scraping, Reuters reports.

The web standard is Robots Exclusion Protocol, or robots.txt, which was created in the 1990s to prevent websites from being overwhelmed with web crawlers, according to Reuters.

A Wired probe alleges that AI search startup Perplexity has likely bypassed efforts to block its web crawler via robots.txt, Reuters continues. And TollBit says its analytics show that numerous AI agents are bypassing the robots.txt protocol.
more at: https://www.mediapost.com/publications/article/397097/ai-firms-are-getting-around-web-protocol-that-lets.html

Related Posts

Macy’s, Inc. Fourth Quarter and Full-Year 2021 Results Exceed Expectations

*Comparable sales up 28.3% on an owned basis and up 27.8% on an owned-plus-licensed basis versus Q4 2020; up 6.6% and up 6.1%, respectively, versus Q4 2019 *Digital sales up 12% over Q4 2020, up 36% over Q4 2019 *7.2 million new customers shopped the Macy's brand, an 11% increase over Q4 2019 *Generated $2.7 billion in Operating Cash Flow and $2.3 billion in Free Cash Flow in FY 2021 *New $2 billion share repurchase program authorized after completing current $500 million program and raised quarterly dividend by 5%

AAP November 2021 StatShot Report: Industry Up 8.3% for Month and 13.1% Year-to-Date

The Association of American Publishers (AAP) released its StatShot report for November 2021 reflecting reported revenue for all tracked categories, including Trade (Consumer Books), K-12 Instructional Materials, Higher Education Course Materials, and Professional Publishing. Total revenues across all categories for November 2021 were up 8.3% as compared to November 2020, coming in at $1.3 billion. Year to date revenues were up 13.1%, at $14.3 billion for the first eleven months of the year.

REI Co-op enters the running shoe business with the Swiftland MT, its first multi-terrain trail shoe designed using recycled and bio-based materials

Based on years of gear designing, experience, customer feedback, and industry-leading sustainability insights, REI Co-op introduces its first foray into running shoes since the 1970s, with the introduction of the REI Co-op Swiftland Multi-terrain (Swiftland MT). This versatile trail runner maximizes recycled and bio-based materials without compromising technical expectations, while giving runners the freedom to take their adventures over multiple surfaces beyond the pavement. The Swiftland MT is a foundational running shoe that joins a wide variety of running apparel, equipment and gear for REI customers to choose from, and continues the co-op's commitment to broaden its running assortment and more deeply invest in this activity category.
Back To Top
×Close search
Search