Robots.txt
- [ ] P1 Robots.txt returns 200 and does not disallow important sections, CSS or JavaScript files needed for rendering
- [ ] P1 No leftover
Disallow: /from a staging environment - [ ] P2 Parameter patterns, internal search results and infinite calendar or filter spaces are disallowed where crawl waste is a problem
- [ ] P2 Sitemap location is listed in robots.txt
- [ ] P3 Rules are tested against real URLs in the Search Console robots.txt report
Remember that robots.txt controls crawling, not indexing. A disallowed URL can still appear in results if other pages link to it. Use noindex when the goal is keeping a page out of the index, and leave that page crawlable so the tag can be seen.
XML sitemaps
- [ ] P1 Sitemaps are submitted in Search Console and return without errors
- [ ] P2 They list only canonical, indexable URLs that return 200
- [ ] P2 No redirected, noindexed or 404 URLs in sitemaps
- [ ] P2 Large sites split sitemaps by type (products, categories, articles) to make indexing gaps visible per section
- [ ] P3 The lastmod value reflects real content changes, not the date of every regeneration
Status codes and redirect chains
- [ ] P1 Important pages return 200, removed pages return 404 or 410, moved pages return 301 or 308
- [ ] P1 No 5xx errors on key templates, and server errors in the crawl stats report are investigated
- [ ] P2 Redirect chains are collapsed to a single hop, and redirect loops are removed
- [ ] P2 Internal links point to final destination URLs, not to redirects
- [ ] P2 Soft 404s (thin "not found" pages returning 200) are fixed
- [ ] P3 Temporary 302s that have been in place for months are reviewed and switched to 301 if the move is permanent
Canonicals, noindex and duplicates
- [ ] P1 No important page carries a noindex tag or an X-Robots-Tag noindex header by mistake
- [ ] P1 Canonical tags point to the correct, indexable, 200 status URL
- [ ] P2 Each indexable page has a self referencing canonical
- [ ] P2 HTTP and HTTPS, www and non-www, trailing slash and case variants resolve to one version with a 301
- [ ] P2 Duplicate content from parameters, print pages, session IDs and tag archives is consolidated
- [ ] P3 Canonical in the HTML and canonical in HTTP headers do not disagree
The Search Console page indexing report
The page indexing report is the first place to look for indexation problems. Read the reasons, not just the totals.
| Status in the report | What it usually means | Typical action |
|---|---|---|
| Crawled, currently not indexed | Google saw the page and chose not to index it, often quality or duplication | Improve content, consolidate near duplicates, strengthen internal links |
| Discovered, currently not indexed | Google knows the URL but has not crawled it yet | Check crawl capacity, server speed, internal links and crawl waste |
| Duplicate, Google chose different canonical than user | Your canonical hint was overridden | Check that the canonical target really matches the content |
| Excluded by noindex tag | Page carries noindex | Confirm it is intentional |
| Blocked by robots.txt | Crawl is blocked | Confirm it is intentional |
| Soft 404 | Page looks empty or like an error | Add content, return a real 404, or redirect to a relevant page |
| Page with redirect | URL redirects elsewhere | Remove it from sitemaps and update internal links |
Use the URL Inspection tool on individual URLs to compare the canonical you declared with the one Google selected, and to view the rendered HTML Google sees.
How Google processes JavaScript
Google crawls the HTML, then renders pages with an evergreen version of Chromium to execute JavaScript. Rendering can happen later than the crawl, and content or links that only appear after user interaction (clicks, scroll events, hover) may never be seen. Other search engines and many AI crawlers render JavaScript less reliably or not at all.
SSR vs CSR
| Approach | How content reaches the crawler | SEO risk | Good fit |
|---|---|---|---|
| Server side rendering (SSR) | Full HTML in the initial response | Low | Content sites, ecommerce, marketing pages |
| Static generation | Prebuilt HTML files | Low | Docs, blogs, landing pages |
| Hydration of SSR pages | HTML first, then JavaScript makes it interactive | Low if HTML is complete | Most modern frameworks |
| Client side rendering (CSR) only | Empty shell, content built in the browser | High | Logged-in apps that do not need to rank |
| Dynamic rendering | Different output for bots and users | Medium, now described by Google as a workaround | Legacy setups being migrated |
JavaScript checklist
- [ ] P1 Main content, titles, canonicals and meta robots tags are present in the rendered HTML (check with URL Inspection)
- [ ] P1 Internal links are real
<a href>elements, not click handlers on divs or buttons - [ ] P2 Critical content does not depend on user interaction to load
- [ ] P2 JavaScript and API endpoints needed for rendering are not blocked in robots.txt
- [ ] P2 Routes return correct status codes, so a missing product in a single page app does not return 200 with an error message
- [ ] P3 Important pages are rendered on the server or prebuilt, rather than built only in the browser
The three metrics in 2026
Core Web Vitals measure loading, responsiveness and visual stability from real Chrome users. Interaction to Next Paint (INP) replaced First Input Delay (FID) as the responsiveness metric in March 2024. FID only measured the delay before the first interaction was processed. INP looks at the latency of interactions across the whole visit, which makes heavy JavaScript much more visible.
| Metric | Measures | Good | Needs improvement | Poor |
|---|---|---|---|---|
| Largest Contentful Paint (LCP) | Loading of the main content | 2.5 s or less | 2.5 to 4 s | Over 4 s |
| Interaction to Next Paint (INP) | Responsiveness to interactions | 200 ms or less | 200 to 500 ms | Over 500 ms |
| Cumulative Layout Shift (CLS) | Visual stability | 0.1 or less | 0.1 to 0.25 | Over 0.25 |
Thresholds are assessed at the 75th percentile of page loads, separately for mobile and desktop. Core Web Vitals are part of Google's page experience signals. They rarely outrank relevance, but they influence close contests and strongly affect conversion.
Performance checklist
- [ ] P2 LCP element (usually the hero image or heading) is not lazy loaded, and is preloaded or prioritized where appropriate
- [ ] P2 Images are sized correctly, served in modern formats and compressed
- [ ] P2 Long JavaScript tasks are broken up, third party scripts are audited, and unused code is removed to improve INP
- [ ] P2 Images, embeds and ads reserve space with width and height or aspect ratio to prevent layout shift
- [ ] P3 Fonts load without causing large shifts, using font-display and matched fallback metrics
- [ ] P3 Server response time (TTFB) is low, with caching and a CDN where traffic is international
Mobile-first indexing
Google completed the move to mobile-first indexing, so the mobile version of a page is what gets crawled and indexed. Check that:
- [ ] P1 Mobile pages contain the same main content, headings and structured data as desktop
- [ ] P1 Internal links in mobile navigation reach the same important pages as desktop navigation
- [ ] P2 Content hidden in tabs or accordions on mobile is still in the HTML
- [ ] P2 Intrusive interstitials do not cover the main content on mobile
Structured data
- [ ] P2 Markup uses JSON-LD and matches what is visible on the page
- [ ] P2 Key types are present where relevant: Organization, BreadcrumbList, Article or BlogPosting, Product with offers, LocalBusiness, FAQPage where the content is genuinely a FAQ
- [ ] P2 No errors in the Search Console enhancement reports and the Rich Results Test
- [ ] P3 Markup is generated from templates, so it stays in sync with content changes
Structured data helps search engines and AI systems interpret entities on a page. It does not guarantee rich results, and markup describing content that is not on the page can lead to a manual action.
Internal linking and orphan pages
- [ ] P1 Every page you want indexed is linked from at least one other indexable page
- [ ] P2 Important pages are reachable within three or four clicks from the homepage
- [ ] P2 Orphan pages are found by comparing URLs from sitemaps, analytics and server logs against URLs found by a crawler
- [ ] P2 Anchor text describes the target page instead of "click here"
- [ ] P3 Breadcrumbs and related content blocks connect sibling pages
Hreflang
- [ ] P1 Each hreflang annotation is reciprocal: if page A references page B, page B references page A
- [ ] P2 Language and region codes are valid (en-us, en-gb, de-de, not en-uk)
- [ ] P2 Each page includes a self referencing hreflang tag
- [ ] P2 Hreflang targets are canonical, indexable 200 URLs
- [ ] P3 An x-default is set for language selectors or global fallback pages
Hreflang is easy to break at scale, so annotations are best generated from one source of truth. Our guide to international SEO and hreflang goes deeper.
HTTPS and security
- [ ] P1 Every page is served over HTTPS with a valid certificate
- [ ] P1 HTTP URLs 301 redirect to HTTPS in one hop
- [ ] P2 No mixed content (HTTP images, scripts or styles on HTTPS pages)
- [ ] P3 HSTS is enabled once HTTPS is stable
- [ ] P3 The Search Console security issues report is checked regularly
AI crawler access
AI search features and assistants rely on crawlers to read and cite your content. Decide deliberately which ones you allow.
- [ ] P2 Robots.txt rules for AI user agents (for example GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) reflect a conscious decision, not an accidental block from a copied template
- [ ] P2 You understand that Google-Extended is a control token for training use of content by Google's AI models, and does not affect Googlebot crawling or Google Search rankings
- [ ] P2 CDN or firewall bot protection does not block legitimate search and AI crawlers you want to allow
- [ ] P3 Important content is available in server rendered HTML, since many AI crawlers do not execute JavaScript
If visibility in AI answers matters to you, our guide on answer engine optimization covers the content side.
What logs show that crawlers cannot
A crawler shows what could be crawled. Server logs show what search engines actually crawl. Log analysis answers questions no other tool can:
- Which sections Googlebot visits most and least often
- How much crawl activity goes to parameter URLs, redirects and errors
- How quickly new pages are discovered after publishing
- Whether important pages are crawled at all
- How often AI crawlers visit and what they request
Steps for a first log review
- Export at least 30 days of access logs, including user agent, URL, status code and timestamp.
- Verify Googlebot by reverse DNS lookup, because many bots fake its user agent.
- Group requests by site section and by status code.
- Compare crawled URLs with your list of indexable URLs to find important pages that are rarely crawled.
- Identify crawl waste: parameters, redirects, 404s and duplicate paths with high hit counts.
- Fix the largest waste sources first, then repeat the analysis a month later.
| Check | Primary data source | Supporting source |
|---|---|---|
| Indexation status | Search Console page indexing report | URL Inspection tool |
| Crawl activity | Server logs | Search Console crawl stats report |
| Status codes and redirects | A crawler | Server logs |
| Canonicals, noindex, duplicates | A crawler | URL Inspection tool |
| JavaScript rendering | URL Inspection rendered HTML | A crawler with rendering enabled |
| Core Web Vitals, field data | Chrome UX Report | Search Console Core Web Vitals report |
| Core Web Vitals, lab diagnosis | PageSpeed Insights | Browser developer tools |
| Structured data | Rich Results Test | Search Console enhancement reports |
| Hreflang | A crawler | Search Console |
| Orphan pages | A crawler compared with sitemaps, analytics and logs | GA4 landing page report |
| AI crawler activity | Server logs | Robots.txt review |
PageSpeed Insights shows both field data from the Chrome UX Report (when the page has enough traffic) and lab data from a simulated load. Field data reflects real users and is what Core Web Vitals assessment uses. Lab data is better for diagnosing causes.
Migration checklist
Domain changes, platform changes, URL restructures and HTTPS moves are where technical SEO mistakes cost the most traffic.
- Crawl and benchmark the old site. Save every URL, its status, title, canonical, rankings, traffic and backlinks.
- Build a full redirect map. Map each old URL to the most relevant new URL with a single 301. Avoid mass redirects to the homepage.
- Test on staging. Check templates, canonicals, hreflang, structured data and internal links, with staging blocked from indexing by password protection.
- Launch and verify within hours. Remove any staging noindex or disallow, crawl the redirect map, submit new sitemaps and use the change of address tool in Search Console for domain moves.
- Monitor for several weeks. Watch the page indexing report, crawl stats, 404s, rankings and conversions. Keep redirects in place for at least a year, ideally permanently.
Fluctuation for a few weeks is normal. Persistent drops usually trace back to missing redirects, changed content or a blocked section.
Our technical seo services cover this checklist end to end: diagnosis, prioritized fixes written for your developers, and verification after release. For a one-off deep review, see our SEO audit services. Plans are a monthly retainer bought online with no sales calls and no setup fee, month to month, and you cancel in the panel. Small is 1 500 USD per month, Growth 3 500, Scale 7 500 and Enterprise 12 000, which adds migration support and release reviews. Annual prepay saves 20 percent.
To find out which of these items affect your site, start with a free SEO audit. It crawls up to 15 pages and shows technical issues and keyword gaps.