+31 20 820 3777
Support 
Documentation 
Login 
Behind almost every website stands someone who cares about whether people visit. For some, a site is a business—a source of orders and clients. For others, it’s a hobby, a personal blog, or a community of interest. The difference is significant, yet owners in both cases share one common goal: they want visitors, and ideally, those who genuinely need the content.
It seems logical that the number of such visits depends primarily on content quality. The more useful and interesting the site, the more likely people are to visit. In reality, the connection is far less direct. The volume and quality of traffic from search engines depend on much more than just published content. Often, visitor metrics are influenced by factors unrelated to the content itself.
Sometimes, a website’s fate in search results hinges on a tiny technical file that the owner might not even know exists. It’s called robots.txt. Located in the root directory of nearly every site, it’s just a few lines long, yet it’s the first thing any search engine bot encounters. What’s written in those lines determines which pages appear in search results and which remain invisible. A single mistake here can silently remove an entire site from search results, and no monitoring system will raise an alarm because, technically, everything is working fine.
This issue most often surfaces during site migrations. A team completes a major migration of an e-commerce store to a new server. Weeks of work are behind them; everything is up and responding, so they can breathe easy. A couple of weeks later, traffic from search drops noticeably, then drops further. Emergency calls begin, and the hunt for the culprit starts. They review the code, redirects, load speeds, and hosting. The cause? Two lines. Along with the rest of the files, a test robots.txt was migrated to the production server—the very one that blocked indexing for everything. Search engines faithfully read the instructions and left.
Interestingly, despite the serious consequences, many treat the file with humor. Yelp once included a reference to Asimov’s Three Laws of Robotics in its robots.txt, TripAdvisor used it to recruit developers, and in 2014, Google created a separate file called killer-robots.txt that forbade T-800 and T-1000 Terminators from touching the company’s founders. The joke coincided with the 20th anniversary of robots.txt, and the file was later removed. robots.txt has long become a playground for small easter eggs.
Despite this, robots.txt remains in a blind spot. Almost everyone has heard of it, few understand it, and even fewer look inside. Below, we’ll break down what this file does, what people mistakenly expect from it, and where it most often fails.
robots.txt and Changes in Search
Not long ago, robots.txt was of interest only to SEO specialists. Today, the circle of people it affects is significantly wider, for several reasons.
The first is related to how search itself is changing. In a recent interview with The Verge, Google’s CEO discussed how search engines are increasingly answering queries directly on the results page, without sending users to external sites. Major publishers mentioned in the same conversation are already restructuring their business models based on the premise that search traffic is trending toward zero. Whether this will fully materialize is hard to judge.
The second reason emerged recently. Language models train on texts from the open internet, and site owners are increasingly wondering if they’re willing to give their content for training. robots.txt became the first tool used to try to restrict such access. It turns out that a file designed for managing indexing has unexpectedly become a lever for regulating artificial intelligence (AI) access to site content. So, it makes sense to understand it more deeply than just “block the admin panel and forget about it.”
What robots.txt Actually Is
In short, it’s a text file in the site root located at /robots.txt that hints to search engine bots which sections to skip and which best not to touch. Formally, it’s described in the Robots Exclusion Protocol, which was formalized as RFC 9309 in September 2022. Before that, the standard relied on good faith for nearly thirty years. No one officially approved it; major search engines simply agreed to follow it and largely did.
The key word here is “hints.” robots.txt does not prohibit access and offers no protection. It’s not a password, firewall, or access control. A bot that wants to will easily read a blocked section. Well-behaved bots from major search engines follow the rules because it’s beneficial for them, but scrapers collecting emails for spam or hunting for vulnerabilities view robots.txt merely as a list of interesting places to visit first.
In practice, this means you can block paths like /admin or /wp-login in robots.txt, but you shouldn’t count on protection. It won’t stop anyone seriously; quite the opposite. A public file literally points out where to find the site’s admin entry point.
Why robots.txt Doesn’t Remove Pages from Search
The most common confusion revolves around the Disallow directive. It doesn’t mean “remove the page from search”; it means “do not crawl this page.” The difference is subtle, but because of it, a blocked page can remain in search results, while a needed page might drop out.
If you block a page in robots.txt, the bot indeed won’t crawl it. However, if other sites link to that page, the search engine may still show it in results. It will appear without a proper description, sometimes with a note like “description unavailable due to robots.txt,” but the page will still appear in the index. The effect is the opposite of the intention. You wanted to hide the page, but instead, it ended up in the index, looking unappealing.
This is where the second trap lies. To truly remove a page from the index, you use the noindex meta tag or the X-Robots-Tag response header, which Google recommends, explicitly noting that robots.txt itself is designed for managing site load, not hiding pages. There’s a nuance with noindex. The bot will only see it if it can access the page and read it. If you block that same page in robots.txt, the bot won’t go there, meaning it won’t see noindex. The two tools start interfering with each other. Google’s documentation separately warns that crawling and indexing rules can cancel each other out when mixed.
Hence the standard logic. When a page needs to stay out of the index, it’s left open for crawling and marked with noindex. robots.txt is reserved for other purposes, such as preventing bots from crawling infinite filters and wasting crawl budget.
robots.txt Syntax Structure
A minimal working file looks roughly like this:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlHere, the asterisk in the User-agent line means “rules for all bots,” an empty Disallow means “we block nothing,” and Sitemap points to the sitemap. Essentially, “everything is open, here’s the map.”
The file that crashed the store in the introduction looks like this:
User-agent: *
Disallow: /A single slash after Disallow blocks crawling for everything. This is the file usually found in test environments, and it’s the most dangerous during migrations.
Rules can also be set for specific bots. For example, open the site to everyone but block one specific scraper:
User-agent: *
Disallow:User-agent: SomeBot
Disallow: /There are two useful special characters in the rules. The asterisk * replaces any sequence of characters, and the dollar sign $ denotes the end of the address. Suppose you need to block all links with sorting parameters, as well as all PDF files:
User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$Here, * substitutes any part of the address, and $ requires the address to end specifically with .pdf.
Allow works as an exception to Disallow. It’s used to block an entire section while leaving one folder inside open:
User-agent: *
Disallow: /private/
Allow: /private/public/The entire /private/ section is blocked, except for the nested /private/public/. In case of conflicting rules, the longer and more specific path match wins, and if lengths are equal, the less restrictive rule prevails according to Google’s guidelines. Yandex’s logic is similar, although details should always be checked against current documentation, as they change.
The trickiest part here is path length. Suppose someone blocked all PDFs and also opened a downloads folder:
User-agent: *
Disallow: /*.pdf
Allow: /downloads/By design, PDFs are blocked. In reality, for /downloads/report.pdf, Allow: /downloads/ triggers because its path is longer than /*.pdf, leaving those PDFs open despite the ban.
Where robots.txt Breaks Most Often
The most painful error was described above: an accidental full block via Disallow from a test environment. There are other, less obvious ones.
Blocked Styles and Scripts
It was once considered reasonable to hide CSS and JavaScript folders from bots. Now it’s more harmful because Google processes pages almost like a browser, rendering them. If you block styles and scripts, it will see broken layout and may decide the page is not mobile-friendly. Google explicitly writes about this in its recommendations.
Confusion Between Disallow and noindex
This has already been mentioned, but it’s so common that I’ll repeat it. Blocking in robots.txt does not remove pages from the index.
Case Sensitivity
It matters in paths, so /Catalog and /catalog are different addresses for the bot. In directive names, case doesn’t play a role; Disallow and disallow are understood identically. Bot names are usually matched case-insensitively; googlebot and Googlebot are the same for Google, but it’s better to write them in the official format. The filename itself must be lowercase, robots.txt, with no Robots.txt.
Wrong Address
The file works only in the domain root and only for its own address. A robots.txt from the main domain doesn’t apply to a subdomain, and the HTTP version doesn’t apply to HTTPS. Each domain and subdomain needs its own file.
Size
Google has a limit; it reads approximately the first 500 kilobytes of the file and ignores the rest. On a typical site, this isn’t a problem, but automatically generated giant robots.txt files sometimes hit this limit.
What to Block via robots.txt vs. Other Tools
Since robots.txt isn’t about indexing, it’s useful to keep in mind which tool does what.
Robots.txt is suitable for keeping bots out of heavy or meaningless-for-search sections, such as infinite filters, carts, or internal search results. This saves so-called crawl budget.
The noindex meta tag in the page code removes a specific page from the index, provided the page is open for crawling.
The X-Robots-Tag header does the same as noindex but at the server response level. It’s convenient for non-HTML files, such as PDF documents or images.
canonical is not a block at all. It hints which of similar pages to consider primary, helping with duplicates, not hiding.
A Ready-Made robots.txt for a Standard Site
For a typical site, a working template looks roughly like what’s shown below. Don’t copy it blindly; it’s more of a starting point to build from.
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xmlHere, the cart, checkout, internal search, and pages with sorting and filters are blocked. The bot crawls everything else as usual, so a separate Allow isn’t needed, and the sitemap is simply placed at the end. Still, this isn’t dogma. Those same filters aren’t always hidden; sometimes a filter page performs well in search, answering queries like “laptop with rtx 4060” more precisely than the general catalog. In reality, every site ends up with its own set of blocks.
robots.txt and Artificial Intelligence Scrapers
Besides classic search engine bots, sites are now visited by scrapers collecting texts to train language models. GPT-Bot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from the AI search engine of the same name, CCBot from the Common Crawl project, which many rely on. They are managed via the same robots.txt. Moreover, OpenAI and Anthropic have multiple bots. Besides training bots, they have bots for search and answering user queries. In robots.txt, they are blocked separately, so you can block the training bot while still appearing in ChatGPT and Claude answers.
If the idea of giving your content for training doesn’t appeal to you, you can block these scrapers by name.
User-agent: GPTBot
Disallow: /User-agent: CCBot
Disallow: /User-agent: PerplexityBot
Disallow: /Google-Extended stands slightly apart. With this token, the site owner decides whether their content will be used to train Google models. This block doesn’t affect regular search crawling, so you can block Google-Extended without dropping out of search results. The token appeared in 2023, when site owners began requesting this capability.
On the other hand, while some webmasters barricade themselves from neural networks using robots.txt, others try to adapt to them. In September 2024, Jeremy Howard (co-founder of fast.ai) proposed a new markup format—the llms.txt file. If robots.txt indicates where not to go, llms.txt is a text greeting for AI agents.
This file is written in simple Markdown and placed in the site root (e.g., ://site.com). Inside, a brief and concise summary of the project is published, along with a list of links to the most important and high-quality pages—documentation, FAQ, or key articles.
Often, llms-full.txt is created alongside it, dumping all the site’s textual content in a “flat” format—completely stripped of heavy HTML code, ads, design, and navigation menus.
Why is this needed? Modern AI assistants like Claude or ChatGPT find it difficult and expensive to “wade through” site layout in search of substance. A compact cheat sheet saves them context window space. The format quickly gained popularity: AI developers themselves (Anthropic, OpenAI, Google) use it for their documentation, smart code editors like Cursor, and popular WordPress SEO plugins (Yoast, RankMath) can already generate such files automatically.
However, both robots.txt and llms.txt work only on conscientious scrapers that agree to follow these rules. Major players say they comply, and it seems they do, but you can’t verify every one... Essentially, these files aren’t locks here; they’re polite signs. Those who need real control move to the server level and cut bots by name or address. Moreover, any rules only affect the collection of new data: they won’t remove already collected data from models. robots.txt affects only the collection of new data, not the training itself. It won’t remove already collected data, nor does it guarantee that content won’t end up in the training dataset.
How to Check Your robots.txt
Checking the file by eye is nearly useless; an obvious error will still slip through. So, it makes sense to use tools.
Google Search Console has a robots.txt report. It shows which version of the file Google sees and when it last downloaded it. It’s also convenient for catching situations where an old, previously saved version is displayed.
Yandex Webmaster has a robots.txt analyzer. You can enter an address and see if it’s open for crawling or blocked, and by which specific rule. This helps when there are many rules and it’s unclear which one triggers.
Separately, it’s useful to simply open your /robots.txt in a browser after any migration or deployment and read it with your eyes. It sounds banal, but this step is most often skipped. You need to get into the habit of checking the file immediately after “deploying” the site.
Therefore, it’s better not to hang a robots.txt with Disallow: / on staging at all. It’s safer to block the test environment with HTTP authentication; no bot or random person will get inside without a login, and there’s nothing to replace during deployment. It’s more convenient to generate the file on the fly based on where the application is running, so the production version physically can’t pick up the test version. And a simple check for Disallow: / directly in the deployment catches exactly this case before the file reaches the production site.
What to Remember About robots.txt
Search engines change their behavior, and what’s true today might look different in a year. This is especially true for handling conflicting rules and supporting specific directives, so specific details should be checked against current Google and Yandex documentation, not retellings, including this one.
Things with AI scrapers are still fluid. New names appear regularly, each follows rules differently, and there’s still no single standard here. It’s not excluded that a separate mechanism will replace robots.txt here over time.
We consciously left out some nuances, such as caching details of the file itself and the behavior of very exotic scrapers. These are topics for separate breakdowns.
robots.txt manages crawling, not indexing, and those who confuse them will sooner or later pay with traffic. It doesn’t work as a lock; there are other means for protection. After every migration, the file should be opened and reread; it’s the cheapest insurance available. Over time, it gained another role, as it’s now used to negotiate with AI scrapers, and how to handle them is up to each individual.
Next, it makes sense to look into crawl budget and how search engines distribute attention between pages, because that’s where robots.txt transforms from a formality into a tool.