Reddit's New Block on the Internet Archive will Hinder AI Data Scraping
News/Tech

Reddit's New Block on the Internet Archive will Hinder AI Data Scraping

Reddit is set to prevent the Wayback Machine from indexing most of its content, affecting how AI companies access historical data.

The usefulness of the internet is about to decline as Reddit moves to block the Wayback Machine from indexing its content. According to a report from The Verge, Reddit plans to restrict access to its archives, allowing only the homepage to be indexed. Individual posts and subreddits will become unreachable.

The Wayback Machine, part of the Internet Archive, historically snapshots web pages to preserve information, even for sites that are no longer active. For instance, it’s possible to view the historical layout of closed communities or track website revisions. However, with Reddit’s new restrictions, the machine will lose access to a vast database of community knowledge.

Reddit stated that this decision responds to instances where AI companies have been found violating platform policies by scraping content unlawfully. Tim Rathschmidt, a spokesperson for Reddit, explained that this measure aims to curb this type of misuse.

This change, however, seems to prioritize financial gains over principles. In 2024, Reddit struck a partnership with Google to make its content available for AI training, followed by a similar deal with OpenAI. Hence, limiting the Wayback Machine’s access does not seem purely about protecting data but rather about monetizing it.

Mark Graham of the Wayback Machine expressed hopes for continued discussions with Reddit on this matter, indicating that there might be possibilities for a resolution that allows broader access in the future.

As a non-profit organization, the Internet Archive provides a crucial service to preserve the web’s history—an endeavor that could be significantly hampered by these new limitations.

Next article

Ubisoft's Upcoming Far Cry TV Series Announcement Gets Pulled

Newsletter

Get the most talked about stories directly in your inbox

Every week we share the most relevant news in tech, culture, and entertainment. Join our community.

Your privacy is important to us. We promise not to send you spam!