How to Fix “Blocked by robots.txt” (Full Step-by-Step Guide)

Your pages might be perfectly written, but if they’re blocked by robots.txt, Google can’t even see them. That means no crawling, no indexing, and no traffic.

This issue is more common than most people think, and it can quietly stop your site from ranking.

One small line in your robots.txt file can block important pages without you realizing it.

The good news? It’s usually easy to fix once you know what to look for.

This guide is built for beginners and site owners who want clear, simple steps to find the problem and fix it fast, without needing technical skills.

Need help with other issues in Google Search Console? Check out this complete GSC troubleshooting guide.

What Does “Blocked by robots.txt” Mean?

“Blocked by robots.txt” means search engines are being told not to access a specific page on your site, so they can’t crawl its content.

In simple terms, your robots.txt file acts like a set of instructions for search engine bots, telling them which pages or folders they are allowed to visit and which ones to avoid.

When a page is “blocked,” it usually contains a rule like Disallow: /page-name/, which stops crawlers from loading that page at all.

Because crawling is the first step before indexing, this creates a problem because if Google can’t crawl a page, it can’t properly understand or rank it.

However, many people confuse this with “noindex,” and they are not the same thing: robots.txt controls crawling (access), while a noindex tag controls indexing (whether the page appears in search results).

A page with noindex can still be crawled and read, but it won’t show up on Google, whereas a page blocked by robots.txt may not be crawled at all and, in some cases, can still appear in search results without full details if Google finds links pointing to it.

This is why using robots.txt incorrectly can silently harm your SEO.

You might block pages you actually want ranking, or fail to properly remove pages from search, because the crawler never sees the noindex instruction.

Why This Issue Happens

Disallow Rules in robots.txt

The most direct cause is a rule inside your robots.txt file that tells search engines not to crawl certain pages or sections.

This is done using the Disallow directive, which can block a single page, a folder, or even your entire site if written as Disallow: /.

These rules are often added intentionally to hide low-value or private areas, but problems start when they are too broad or incorrectly written.

A small mistake, like blocking /blog/ instead of a specific page can stop dozens or hundreds of important URLs from being crawled.

Because search engines follow these instructions strictly, even one incorrect line can prevent your content from ever being seen.

CMS or Plugin Settings (WordPress, Shopify, etc.)

Many website platforms and SEO plugins can automatically control crawling settings, sometimes without you realizing it.

For example, WordPress has a “Discourage search engines from indexing this site” option, which can add blocking rules behind the scenes, while Shopify manages parts of robots.txt automatically and may restrict certain pages by default.

SEO plugins can also modify robots.txt or add rules based on templates.

These tools are helpful, but if a setting is enabled by mistake or left on after a site launch, it can block important pages without any obvious warning.

Developer Restrictions (Staging/Dev Environments)

Developers often block search engines while a site is being built or tested to prevent unfinished content from appearing in search results.

This is common on staging or development versions of a site, where robots.txt is used to stop all crawling.

The issue happens when these restrictions are accidentally carried over to the live site during launch.

A site can go public while still telling search engines to stay out, which leads to pages not being crawled or indexed at all.

This is one of the most common causes of sudden traffic drops after a redesign or migration.

Accidental Blocking of Important Pages

Not all blocking is intentional. Sometimes pages are blocked simply due to oversight, like an outdated rule, a copied robots.txt file, or a quick fix that was never reviewed.

Important pages like blog posts, product pages, or landing pages can end up inside blocked folders or match a broader rule without you noticing.

Because robots.txt works at the crawl level, these mistakes often go unnoticed until you check Google Search Console and see that key pages are excluded.

The impact builds over time, as search engines stop revisiting those pages, leading to lost visibility and rankings.

How to Check If a Page Is Blocked

Using Google Search Console (Pages Report)

The fastest way to spot this issue is inside Google Search Console. Go to the Pages (formerly Coverage) report and look for the status “Blocked by robots.txt.”

This report shows exactly which URLs are affected and groups them clearly, so you can see patterns, like entire folders being blocked.

Click on any listed URL to view more details, including when Google last tried to crawl it.

This gives you a clear starting point and confirms whether the issue is active or already resolved.

Robots.txt Tester Tool

The robots.txt Tester inside Google Search Console lets you check if a specific URL is being blocked by your current rules.

You simply enter the page URL, and the tool will tell you if it’s allowed or disallowed based on your robots.txt file.

It also highlights the exact line causing the block, which removes guesswork.

This is especially useful when your file has multiple rules, and you’re not sure which one is affecting the page.

Manual Check (yourdomain.com/robots.txt)

You can quickly review your robots.txt file by visiting yourdomain.com/robots.txt in your browser. This file is public, so anyone, including search engines, can see it.

Look for Disallow rules and check if any match the page or folder you’re investigating.

Pay attention to broad rules like Disallow: / or folder-level blocks such as /blog/ or /products/.

Even a small pattern can affect many URLs, so reading the file carefully helps you understand the full scope of the problem.

URL Inspection Tool

The URL Inspection tool in Google Search Console gives a deeper, page-level view.

Enter the exact URL, and it will show whether the page is blocked from crawling, along with other indexing details.

If the page is blocked, you’ll see a clear message explaining that robots.txt is preventing access.

This tool also shows the last crawl attempt and lets you test the live URL, so you can confirm whether your fix worked before requesting reindexing.

Common Robots.txt Mistakes

Blocking Entire Site (Disallow: /)

One of the most damaging mistakes is adding Disallow: / under a user-agent, which tells search engines not to crawl any part of your site.

This rule is often used during development to keep unfinished sites out of search results, but if it remains after launch, your entire website becomes invisible to search engines.

Googlebot will respect this directive and stop crawling completely, which means no pages get indexed or updated.

This issue is easy to miss but has a major impact, especially after site migrations or redesigns.

Blocking Important Folders (e.g., /blog/, /products/)

Another common issue is blocking entire directories that contain valuable content.

For example, a rule like Disallow: /blog/ or Disallow: /products/ prevents search engines from accessing all pages within those sections.

This often happens when trying to block a specific page but using a broader path instead.

Since robots.txt works at the folder level, one rule can affect hundreds of URLs at once, leading to lost rankings and traffic across key parts of your site.

Incorrect Wildcard Usage

Wildcards like * and $ are used to match patterns in URLs, but they can easily be misused.

For example, a rule such as Disallow: /*? might block all URLs with query parameters, even if some of those pages are important.

Small syntax mistakes can expand the scope of a rule far beyond what was intended.

Because search engines interpret these patterns literally, even a minor error can lead to large sections of your site being blocked without a clear warning.

Blocking CSS/JS Files

Some sites accidentally block access to CSS and JavaScript files, often by disallowing folders like /wp-content/ or /assets/.

This prevents search engines from properly rendering your pages, which affects how Google understands layout, usability, and mobile friendliness.

When Googlebot cannot load these resources, it may not see your page the way users do, which can hurt rankings.

Best practice is to allow essential resources so search engines can fully process your content.

Conflicting Rules

Robots.txt can contain multiple rules for different user agents, and conflicts can occur when they overlap.

For example, one rule might allow a page while another blocks it for the same or a different bot.

Search engines follow specific precedence rules, but these conflicts can still create confusion and unintended blocking.

If your file is cluttered or poorly structured, it becomes harder to predict how bots will behave. Keeping rules simple and clearly organized reduces the risk of these hidden issues.

How to Fix “Blocked by robots.txt” (Step-by-Step)

Step 1: Locate Your robots.txt File

Your robots.txt file is always found at the root of your domain, which means you can access it by visiting yourdomain.com/robots.txt in your browser.

This file is public and shows exactly what search engines are allowed or blocked from crawling.

To edit it, you’ll need access to your site’s backend.

This could be through your hosting provider, file manager, or a CMS like WordPress, where plugins or settings may control the file.

Once you open it, you’re looking at the exact instructions search engines are following, so any change you make here directly affects crawling behavior.

Step 2: Identify the Blocking Rule

Next, scan the file for Disallow directives that might be blocking your page.

These rules tell search engines what not to crawl, so your goal is to match the blocked URL with a specific rule.

For example, if your page is /blog/post-name/, check if there’s a rule like Disallow: /blog/ or a pattern that covers it.

This step is about clarity because once you find the exact line causing the issue, the problem becomes much easier to fix.

Step 3: Update or Remove the Rule

After identifying the rule, decide whether it should stay or be changed.

If it’s blocking an important page, you should either remove the rule or make it more specific so it no longer affects that URL.

For instance, instead of blocking an entire folder, you can target a single page.

At the same time, keep rules that protect sensitive or low-value areas like admin pages or checkout flows.

Step 4: Test Your Changes

Before assuming the fix worked, test it. Use the robots.txt Tester in Google Search Console to check if the updated rules now allow your page to be crawled.

Enter the URL and confirm that it’s no longer blocked. You can also use the URL Inspection tool to validate live accessibility.

This step ensures you’re not guessing because you’re confirming that search engines can now reach your content.

Step 5: Request Reindexing

Once the page is accessible, the final step is to ask Google to crawl it again. In Google Search Console, use the URL Inspection tool, enter your page, and click “Request Indexing.”

This prompts Google to revisit the page and update its status. While indexing is not instant, this step speeds up the process and ensures your fix is recognized.

After that, monitor the Pages report to confirm the issue is resolved and your page is back in the crawl cycle.

robots.txt Best Practices

Only Block Low-Value Pages (Admin, Cart, etc.)

Use robots.txt to guide search engines away from pages that don’t provide value in search results.

This includes areas like admin panels, login pages, cart/checkout flows, and internal system folders.

These pages don’t help users when found on Google and can waste crawl budget.

By blocking them, you help search engines focus on your important content instead of irrelevant sections.

Never Block Pages You Want Indexed

If you want a page to rank on Google, it must be crawlable. Blocking it in robots.txt prevents search engines from accessing its content, which stops proper indexing and ranking.

This is a common mistake, especially when trying to control visibility.

If your goal is to keep a page out of search results but still allow crawling, use a noindex tag instead and not robots.txt.

Keeping this distinction clear protects your key pages from being accidentally excluded.

Keep the File Clean and Simple

A robots.txt file should be easy to read and maintain. Avoid adding too many complex rules, overlapping directives, or unnecessary patterns.

Search engines follow clear and direct instructions more reliably, and a simple structure reduces the risk of mistakes.

A short, well-organized file is easier to audit and less likely to cause unexpected blocking.

Use Comments for Clarity

You can add comments in robots.txt by starting a line with #. These notes are ignored by search engines but help you and your team understand why certain rules exist.

For example, labeling a rule as “# Block checkout pages” makes future edits safer and faster.

Clear comments reduce confusion, especially when multiple people manage the site.

Regular Audits

Robots.txt should not be a “set and forget” file. Changes to your site, like new sections, redesigns, or migrations, can make old rules outdated or harmful.

Regularly reviewing the file ensures it still aligns with your SEO goals.

Tools like Google Search Console can help you spot blocked pages and confirm everything is working as expected.

Frequent checks keep you in control and prevent small issues from turning into bigger problems.

robots.txt vs Meta Robots (Important Difference)

robots.txt and meta robots tags serve different roles, and understanding this difference gives you full control over how your pages appear in search.

robots.txt controls crawling, which means it tells search engines whether they are allowed to access a page at all; if a page is blocked here, search engines cannot read its content.

Meta robots, on the other hand, control indexing, which determines whether a page should appear in search results after it has been crawled.

This means a page with a noindex tag can still be crawled and understood, but it won’t show up on Google, while a page blocked by robots.txt may still appear in search results without details if other sites link to it, because Google knows the URL exists but cannot access the content.

The key rule is simple: use robots.txt when you want to prevent crawling of low-value or sensitive areas, and use meta robots (like noindex) when you want search engines to access the page but not include it in search results.

A common mistake is blocking a page in robots.txt and expecting it to disappear from Google, but this often fails because the crawler never sees the noindex instruction.

To remove or control visibility properly, the page must be crawlable first.

When You SHOULD Block Pages

Admin/Login Areas

Admin and login pages should always be blocked because they are not meant for public access and provide no value in search results.

These areas often contain sensitive functionality, and allowing search engines to crawl them can waste crawl budget and expose unnecessary URLs.

A simple rule like Disallow: /wp-admin/ helps keep these sections out of the crawl path while allowing search engines to focus on your actual content.

Duplicate Content Sections

If your site generates multiple URLs with the same or very similar content, blocking certain versions can help reduce unnecessary crawling.

This often happens with filtered pages, sorting options, or parameter-based URLs.

While canonical tags are usually the better solution for managing duplicates, robots.txt can still be useful for preventing search engines from spending time on low-value variations that don’t need to be crawled at all.

Internal Search Results

Internal search pages (like /search?q=keyword) should not be indexed or heavily crawled because they create endless combinations of URLs with little unique value.

Search engines like Google have explicitly recommended blocking these pages, as they can be seen as low-quality or even spam-like in large volumes.

Thank-You/Confirmation Pages

Pages shown after a form submission, purchase, or sign-up, such as “thank you” or confirmation pages, should also be blocked.

These pages are not useful in search results and can create confusion if users land on them directly without completing the intended action.

Blocking them ensures they stay part of the user journey rather than becoming standalone entry points from search engines.

When You SHOULD NOT Block Pages

Blog Posts

Blog posts are often your main source of organic traffic, so they must be fully accessible to search engines.

If a blog page is blocked in robots.txt, Google cannot crawl or understand its content, which means it won’t rank properly or may not be indexed at all.

Even if the post is high quality, it becomes invisible in search.

Always make sure your blog directory (e.g., /blog/) is allowed unless there is a very specific reason to block a single page.

Product Pages

Product pages are critical for eCommerce visibility and conversions.

Blocking them prevents search engines from seeing product details, pricing, and relevance signals, which directly impacts rankings and sales.

Platforms can sometimes create complex URL structures with filters or parameters, but the core product URLs should never be disallowed.

If these pages are blocked, you lose the opportunity to appear in search results for buying-intent keywords.

Landing Pages

Landing pages are designed to attract targeted traffic, often from search engines or campaigns. If they are blocked, they cannot perform their primary role.

Whether it’s a service page, campaign page, or lead-generation page, search engines need full access to evaluate and rank it.

Blocking landing pages cuts off a key entry point to your site and limits your ability to capture new visitors.

Important SEO Content

Any page that is meant to rank, like guides, category pages, cornerstone content, or high-value resources, must remain crawlable.

Blocking these pages stops search engines from reading internal links, understanding content structure, and passing ranking signals.

According to guidance from Google, pages need to be accessible for proper indexing and ranking.

As a rule, if a page is important for traffic, visibility, or user discovery, it should never be blocked in robots.txt.

Final Thoughts

“Blocked by robots.txt” can stop your pages from being seen, but the fix is usually simple once you know where to look.

A quick check of your robots.txt file and a small update can restore crawling and get your content back on track.

You’re in control of how search engines access your site. Review your settings regularly, catch issues early, and keep your important pages open for indexing.

Want to be a pro at fixing issues in GSC? See this detailed guide on Google indexing issues.

FAQs