A robots meta tag tells search engines how a specific page should be indexed, and how its content may appear in search results. Depending on the directive, it can prevent the page from being indexed, stop crawlers from following links, or limit how the page’s content appears in search results.
If Track My Visibility has flagged a Robots Meta Tag opportunity for one of your pages, it does not necessarily mean something is wrong. The goal is to identify which directive is present and check whether it matches the page’s purpose.
In this guide, you’ll learn how to identify the robots directive, decide whether it needs fixing, correct it when necessary, and verify that the change has been applied properly.
TL;DR
- Check the robots tag before changing it. A flagged tag may be intentional.
- Use noindex only when you do not want the page to appear in search results.
- You usually do not need index, follow because search engines use it by default.
- robots.txt controls crawling, while robots meta tags control indexing and how pages appear in search results.
- Ensure Google can access a noindex page so it can read and apply the directive.
- Protect private content with login or access controls, not noindex.
- Use canonical tags for duplicate pages and noindex when you want pages excluded from search.
- After making changes, check the page again in Google Search Console and give Google time to process the update.
1. What Is a Robots Meta Tag?
A robots meta tag is an HTML element that gives search engines instructions about how to index and display a page in search results. It can control whether the page appears in search and how its content is presented.
A simple example looks like this:
<meta name=”robots” content=”noindex”>
There are two important parts:
- name=”robots” tells supported search crawlers that the instruction applies to them.
- content=”noindex” contains the directive the crawler should follow.
The robots meta tag is normally placed inside the <head> section of the page’s HTML.
One important thing to know is that you do not need to add:
<meta name=”robots” content=”index, follow”>
to every page you want indexed. For Google, index and follow are the default behaviors, so you usually do not need to add them explicitly when no restrictive robots directives are present.
So, if a page has no robots meta tag, that is usually normal and does not by itself indicate a problem.
2. Why Are You Seeing This Robots Meta Tag Opportunity?
Track My Visibility may flag a Robots Meta Tag opportunity when it detects a restrictive, conflicting robots directive that should be reviewed.
For example, the page may:
- contain noindex
- use nofollow
- include nosnippet
- have conflicting robots directives
- use a restriction that may not match the page’s purpose
Seeing this opportunity does not mean the page is set up incorrectly. Some pages are intentionally kept out of search using noindex, such as checkout, login, or thank-you pages.
For example, a checkout page with noindex may be perfectly normal. But if an important product, service, category, or blog page has noindex by mistake, it could prevent that page from appearing in Google Search.
The goal is to check which directive is present and confirm that it matches what you want that page to do.
3. Check What Robots Directive Your Page Has
Before changing anything, first check which robots directive your page currently has. To check robots directives on your page:
- Open the affected page.
- Right-click and select View Page Source.
- Press Ctrl+F on Windows or Cmd+F on Mac.
- Search for name=”robots” or <meta name=”robots”.
- Look for a tag similar to:
<meta name=”robots” content=”noindex”>
You can also check the page using your browser’s Inspect or DevTools options.
Here’s what some common results mean:
| What you find | What it means |
| noindex | Tells Google not to show the page in search results |
| nofollow | Tells Google not to follow links on the page |
| noindex, nofollow | Applies both restrictions |
| nosnippet | Prevents normal text snippets and video previews from being shown |
| No robots meta tag | No special robots restrictions are being set. |
If no robots meta tag is present, you usually do not need to add one just to allow indexing. However, check whether an X-Robots-Tag, robots.txt rule, or another setting is affecting the page.
4. How to Know If a Robots Meta Tag Is a Problem
A robots meta tag is a problem when its directive conflicts with what you want the page to do in search.
The easiest way to decide is to ask:
If You Want the Page to Appear in Search
Check for restrictive directives such as noindex, nofollow, or nosnippet, and confirm whether each restriction matches what you want the page to do in search.
Pages that usually need to be indexable include:
- service pages
- product pages
- useful category pages
- important landing pages
- blog posts
- informational resources
If an important page has noindex, Google is instructed not to show it in search results. In that case, the robots meta tag likely needs to be fixed.
If You Do Not Want the Page to Appear in Search
A restrictive directive may be intentional.
This may apply to pages such as:
- login pages
- cart pages
- checkout pages
- thank-you pages
- certain internal search pages
- low-value utility pages
For these pages, a noindex directive may be appropriate and does not necessarily need to be changed. However, if the content needs to remain private, use authentication or access controls rather than relying on noindex.
Also check whether directives such as nofollow or nosnippet are intentional, since they can affect link following or how the page appears in search results.
5. How to Fix the Robots Meta Tag Issue
Once you know which directive is present and have confirmed that it does not match the page’s purpose, update or remove the restrictive directive.
| Directive found | What to do if the directive is unintended |
| <meta name=”robots” content=”noindex”> | Remove noindex, then inspect the URL in Google Search Console and use Request Indexing if appropriate. |
| <meta name=”robots” content=”nofollow”> | Remove nofollow, then recheck the page source to confirm the change. |
| <meta name=”robots” content=”noindex, nofollow”> | Remove both directives, then use URL Inspection in Google Search Console and request recrawling if needed. |
| <meta name=”robots” content=”nosnippet”> | Remove nosnippet, then allow Google to recrawl the page and update its search appearance. |
Remove a directive only when you are sure it does not match what you want the page to be displayed in search.
6. How to Add or Change the Robots Meta Tag on Different Platforms
The steps depend on your CMS or framework. Some platforms provide page-level SEO settings, while others require changes to the theme, template, or code.
1. Plain HTML
Open the page’s HTML and add the robots meta tag inside the <head>:
<head>
<meta name="robots" content="noindex">
</head>To change the directive, edit the content value. To remove the restriction, remove the unwanted directive or the entire tag if no other directives are needed.
2. WordPress
If you use Yoast SEO, open the page or post → Yoast SEO → Advanced → Allow search engines to show this content in search results. Select No to add noindex or Yes to allow indexing.
If you use Rank Math, open the page or post → Rank Math SEO → Advanced → Robots Meta, then select Index, No Index, or another required directive and update the page.
3. Shopify
Go to Online Store → Themes → Edit code → Layout → theme.liquid.
Add the robots tag inside <head>. For example:
{% if handle contains ‘page-handle’ %}
<meta name=”robots” content=”noindex”>
{% endif %}Replace page-handle with the page or product handle. To make the page indexable again, remove the condition or noindex tag and save the theme.
4. Wix
Go to Pages & Menu → select the page → SEO Basics.
Use Let search engines index this page:
- Turn it off to add a noindex instruction.
- Turn it on to allow the page to be indexed.
Publish the site after making the change.
5. Webflow
Open the page → Page settings → SEO settings → Sitemap indexing.
- Turn Sitemap indexing off to add a noindex robots meta tag.
- Turn it on to allow indexing again.
Publish the site after changing the setting.
6. Squarespace
Open Pages, click the page’s gear icon → SEO, then use Hide page from search results.
- Turn it on to add noindex.
- Turn it off to allow the page to be indexed.
You can also manually add <meta name=”robots” content=”noindex”> through the page header code injection when needed.
7. Next.js
For the App Router, add or change the robots property in the page or layout metadata:
export const metadata = {
robots: {
index: false,
follow: true,
},
}Change index to true or remove the restrictive setting when you want the page to be indexable. Next.js generates the corresponding robots meta tag automatically.
8. Magento / Adobe Commerce
For store-level robots settings, go to Content → Design → Configuration → Edit → Search Engine Robots.
Change Default Robots to the required option, such as:
- INDEX, FOLLOW
- NOINDEX, FOLLOW
- INDEX, NOFOLLOW
- NOINDEX, NOFOLLOW
Then click Save Configuration. For robots directives that apply only to individual pages, check the page template, extension, or custom implementation generating the metadata.
After making the change, check the published page source to confirm the expected robots meta tag is present, then recheck the page in Track My Visibility to confirm the opportunity is resolved.
9. Important: Check Robots.txt Too
If a page uses noindex, make sure robots.txt is not blocking Google from accessing that page.
The two controls do different jobs:
- Robots meta tag: Controls indexing and how a page appears in search.
- robots.txt: Controls whether search crawlers can access a URL.
For example, this robots.txt rule blocks Google from crawling the page:
User-agent: *
Disallow: /example-page/
If the same page also contains:
<meta name=”robots” content=”noindex”>If robots.txt blocks the page, Google cannot crawl the page to see and process the noindex directive.
So, if you are using noindex to remove a page from Google Search, make sure the URL is still crawlable.
Also, do not rely on robots.txt alone to keep an HTML page out of search results. A blocked URL can still sometimes appear in Google if it is discovered through links or other sources.
7. How to Verify the Robots Meta Tag Fix
After updating the robots meta tag, verify that the change is actually live and that Google can process it.
Step 1: Check the Page Source Again
Open the page, select View Page Source, and search for robots.
Confirm that the unwanted directive has been removed or updated correctly.
Step 2: Check the URL in Google Search Console
Open URL Inspection and enter the affected URL.
Check whether:
- Google can access the page
- Indexing is allowed
- The updated directive is being recognized
Step 3: Request Indexing
If the page should be indexed and you have fixed the issue, use Request Indexing in Google Search Console where appropriate.
Step 4: Give Google Time to Process the Change
Removing noindex or changing another robots directive does not update Google Search immediately.
Google needs to recrawl and process the page before the change can be reflected in search results.
8. Common Robots Meta Tag Mistakes to Avoid
Even a small robots configuration mistake can affect whether an important page appears in search. Check for these four common issues.
1. Accidentally Leaving noindex on an Important Page
This often happens after:
- Staging
- Redesigns
- Migrations
- SEO plugin changes
If noindex is left on an important page, Google may keep that page out of search results.
2. Blocking a noindex Page in robots.txt
If Google cannot crawl the page, it may not be able to see and process the noindex directive.
If you are using noindex to remove a page from search, make sure robots.txt is not blocking Google from accessing it.
3. Using noindex When Canonicalization Is the Better Fit
noindex and rel=”canonical” serve different purposes:
- noindex tells Google not to show the page in search.
- rel=”canonical” tells Google which URL you prefer among duplicate or very similar pages.
If you want duplicate URLs to consolidate into one preferred version, a canonical tag may be more appropriate than noindex.
4. Having Conflicting Robots Directives
Robots restrictions can come from different places, including:
- Page HTML
- CMS settings
- SEO plugins
- Templates
- HTTP headers
For example, one setting may allow indexing while another adds noindex.
When robots meta or X-Robots-Tag directives conflict, Google applies the more restrictive rule. Check the page HTML, CMS settings, plugins, templates, and HTTP headers for conflicting directives.
Use Robots Meta Tags With the Right Intent
A robots meta tag is not automatically a problem. The important part is making sure the directive matches what you want the page to do in search.
Check the directive, fix it if needed, make sure robots.txt is not blocking Google from accessing the page, and verify the update in Google Search Console.
Once the page is configured the way you intended, you’ve addressed the Robots Meta Tag opportunity in Track My Visibility.
FAQ
No. You usually do not need to add index, follow because indexing and link following are the default behavior when no restrictive robots directives are present.
No need to remove noindex from every page. Keep it on pages that should stay out of search, such as thank-you pages, login pages, cart pages, internal search results, or thin archives.
robots.txt tells search engines which pages or files they can crawl. A robots meta tag tells them whether they can index a page or how it should appear in search results. Do not block a noindex page in robots.txt.
The timing can vary. Google needs to recrawl and process the page before the updated directive is reflected in search. You can use Request Indexing in Google Search Console, but this does not guarantee an immediate update.
