What does a Reddit search engine block even mean? Does Reddit have a search engine of its own? No, Reddit has a search option in the app but that’s not what we’re talking about. The company has recently decided to fire up its own efforts to stop popular search engines from crawling through the data on its website, especially for training generative AI data bots. To do this, Reddit has enforced a search engine ban by updating its robots.txt file, which prevents these search engines from showing data from the website among the search results. Want to see the Reddit reviews on the latest GTA 6 news?
A quick search on Bing or DuckDuckGo will tell you that you can still access Reddit if you specifically search for the website, but the recent posts or popular threads don’t come out on top as easily.
Image: Snoo, the Reddit mascot
Reddit Moves to Block Search Engines Other than Google
Reddit has updated its robot.txt file to block search engine scrapers or crawlers from accessing the contents of their site and collecting the information available on the platform. If you thought this was a move in support of privacy and rejection of the selling of data, it’s not. This update prevents search engines from crawling the website for information that it might present in response to a search query. It also stops search engines from being able to use the data available on the website to train its AI models without paying the company for the information first. That privilege is exclusively held by Google right now.
What Is the Reddit-Google Deal About?
Earlier this year, around February, Reddit and Google signed a deal that would allow the tech company to use data from Reddit to train its AI. Reports suggest Google is paying the social media platform $60 million USD per year for that data, and in exchange for it, it also appears to be boosting the position of Reddit’s results on its webpage. A significant number of searches now yield Reddit as a top result, often outranking the original articles shared to Reddit for discussion.
Reddit outranking the @ThisHouseFresh article for “How Google is killing independent sites like ours” is probably the most “so sad it’s funny” example of how bad they are that I’ve seen yet. pic.twitter.com/L1ca9JEUct
— Chris M. Walker (@cmwalker) February 20, 2024
Timeline:
– Google boosts Reddit in search by over 400%
– Google announces a 60M deal with Reddit right after
– Reddit announces it seeks to go public 48-72 hours after
– Reddit blocks all scraping for competition search engines shortly after👏Anti 👏Competitive👏Behavior https://t.co/Hzrw0zkjDo
— Ori Zilbershtein (@OriZilbershtein) July 24, 2024
Reddit blocks all search engines except Google, breaking Bing.
As much as Google’s anti-competitive behavior is angering, I’m playing the world’s smallest violin for Bing, whose content theft to power AI is threatening the open web too. https://t.co/7xy9jpuLBP#Bing #Google
— Jez (@JezCorden) July 25, 2024
Google’s AI Summary feature had faced some criticism earlier for providing misleading results when it was released. One of their solutions to the problem was to limit “the inclusion of satire and humor content.” A lot of such information comes from websites like Reddit and other discussion forums where user response drives the content, as well as satirical news sources like The Onion. Now we have to wonder how Google is using the data it acquires from Reddit to train its AI and where exactly this information plays a role in the capabilities of its AI services.
Reddit’s exclusive shift to Google may cause more users who traditionally turn to Reddit to switch browsers following the move to block other search engines. It may also cause a drop in stray users arriving on relevant threads now that they can’t comfortably use their preferred browsers to get there. The company does not appear to be too worried about this possibility.
Reddit Says the Search Engine Ban is Not a Result of Their Deal with Google
Reddit’s approach to search engine scrapers from sources apart from Google might make it sound like an exclusive deal, but the company isn’t planning on refusing offers other companies make to pay and use its data. In an update on the r/redditdev subreddit, user traceroo provided an explanation of the changes in the company’s Public Content Policy. “We’ve seen an uptick in obviously commercial entities who scrape Reddit and argue that they are not bound by our terms or policies. Worse, they hide behind robots.txt and say that they can use Reddit content for any use case they want,” the user explained in the post.
In the next few weeks, we’ll be updating our robots.txt instructions to be as clear as possible: if you are using an automated agent to access Reddit, you need to abide by our terms and policies, and you need to talk to us. We believe in the open internet, but we do not believe in the misuse of public content.
It’s a brazen move from Reddit as the search engine block might reduce the number of people who stumble into a particular thread while looking for information, whether that is the latest Android games on the market or what exactly is happening with the latest CrowdStike bug.
Reddit is full of polarized opinions and edgy memes, but there are also useful threads where people provide knowledgeable insight into topics that other websites may not have covered as candidly. We wouldn’t go so far as to call it a good source of information but it’s definitely an extensive source of information, powered by a lot of users who are genuinely willing to share links and help out.
Image: While Bing and Google were always designed to put different search results on top, the direct access to Reddit posts make Google easier to use in this particular case.
Is the Reddit Restriction on Search Engines Working?
According to 404 Media, if you use alternate search engines and search for “site:reddit.com,” you will no longer be able to see the most recent posts that were made on Reddit. DuckDuckGo still shows you Reddit links, but the engine cannot access any further details about the linked content.
From our own search on Bing, older content from a month or two ago can still show up in the results easily, but Reddit’s restriction on search engine scrapers will prevent it from crawling and presenting newer content. You can also use the independent platform Kagi to access information considering it buys sections of its search index from Google, but other engines may fall behind.
Microsoft spokesperson Caitlin Roulston told The Verge, “Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models.” Bing has reportedly stopped crawling Reddit since July 1, 2024, which is when the robots.txt file was updated.
Will Reddit’s decision to block other search engines finally prompt them to enter into an agreement just like the Reddit-Google deal? We’ll certainly be on the lookout for more updates on this story.