Checking Indexation

Why Indexation Is the First Audit

Technical SEO audit always starts with indexation. If Google cannot find and index your pages, every other optimization effort is wasted. The goal is simple: important pages should be indexed; unimportant pages should not waste crawl budget.

Both extremes are problematic. Too few pages indexed means important content is invisible. Too many pages indexed means Google spends crawl budget on filters, pagination, and duplicates instead of your real content.

Step 1: Check Total Indexed Pages in GSC

Open GSC → Indexing → Pages. The first screen shows two numbers: "Indexed" and "Not indexed." Compare the "Indexed" count against your site's actual page count (you can check this via a sitemap or Screaming Frog crawl).

• Indexed count much lower than actual pages → important pages may be blocked or failing quality checks
• Indexed count much higher than actual pages → duplicate or parameter-based pages are being indexed, wasting crawl budget
• Indexed count dropped sharply on a specific date → something changed on that date; check deployments and content changes

Step 2: Audit robots.txt

Open yourdomain.com/robots.txt and check every Disallow directive. Common critical mistakes:

• Blocking / — the entire site is blocked (this happens after migration from staging)
• Blocking *.js or *.css — prevents Googlebot from rendering JavaScript-dependent pages
• Blocking important sections like /blog/ or /products/
• Missing clean-param for filter/sort URL parameters

Step 3: Check Noindex Tags

The <meta name="robots" content="noindex"> tag completely removes a page from Google's index. It's useful for thank-you pages, admin areas, and internal search results — but frequently gets applied incorrectly to important pages after CMS updates or template changes.

Use Screaming Frog to crawl all pages and filter for noindex in the response headers or meta tags. Export the list and review it manually. Pages that should be indexable but have noindex set are an immediate technical fix priority.

Step 4: Canonical Tag Audit

Canonical tags tell Google which version of a page is the authoritative one. Three common canonical errors:

1. Pagination canonicalized to page 1 — e.g., /blog/page/2/ has canonical pointing to /blog/. This hides deep content from Google.
2. JavaScript-rendered canonicals — Googlebot reads the raw HTML, not the JS-rendered DOM. If your canonical tag is set via JavaScript, Googlebot won't see it.
3. Duplicate pages without any canonical — URL parameters like ?sort=price and ?sort=name create duplicate pages. Without canonical or robots.txt clean-param, both get indexed separately.

Step 5: Sitemap.xml Health Check

Open yourdomain.com/sitemap.xml and verify:

• The file returns HTTP 200 (not 404 or redirect)
• It contains only URLs that should be indexed — no noindex pages, no canonicalized-away URLs, no parameter variants
• The lastmod values are accurate (not all set to the same date, not future dates)
• The sitemap is submitted in GSC under Indexing → Sitemaps
• If the site has more than 50,000 URLs, use a sitemap index file

Step 6: Crawl Budget Awareness

Crawl budget is the number of pages Googlebot visits per day on your site. Wasted crawl budget means important pages get crawled infrequently. The biggest crawl budget wasters:

• URL parameters (filters, sorts, sessions, tracking IDs)
• Infinite scroll or pagination without proper handling
• Duplicate pages accessible via multiple URLs
• Server errors (5xx) — each failure teaches Googlebot to crawl more slowly