01 / Context
Why the 2026 audit is not the 2022 audit
Most technical SEO checklists still in circulation were written for a simpler crawler. Since late 2025 Google has changed the ground under them, and an audit that ignores those changes will miss real problems.
- New documentation home. On 20 November 2025 Google moved its crawling documentation from Search Central to a dedicated crawling site, and it has been editing it steadily since. Its crawling changelog is now required reading for any auditor.
- Hard fetch limits. In a Search Central blog post on 31 March 2026, Gary Illyes spelled out that Googlebot fetches up to 2 MB per URL for HTML and other non-PDF files, 64 MB for PDFs, with a 15 MB default for crawlers that do not state their own limit. Bytes past the cutoff are not rendered or indexed.
- More user agents. Google documented
Google-Agent for AI agents acting on a user's behalf in March 2026, added experimental Web Bot Auth request signing in May, and on 17 November 2026 published guidance on allowing Google-Agent through.
None of this replaces the fundamentals. It does change the order of work. We audit in the sequence Google processes a site: can it be fetched, can it be rendered, which URL is canonical, and is it worth indexing. Each layer depends on the one before it, so fixing titles on pages Googlebot cannot reach is wasted effort.
The checklist below is the one we use on engineering-led audits. It assumes access to Google Search Console, raw server or CDN logs, and a crawler that can render JavaScript. If logs are out of reach, say so in the report; an audit without them is an estimate, not a diagnosis.
02 / Crawl and index
Start with the Page indexing report, not a crawler
A third-party crawl shows what could go wrong. Search Console's Page indexing report shows what Google actually decided. Start there, export every reason, and sample URLs from each bucket in URL Inspection before you form an opinion.
Reasons worth separating early
- Discovered – currently not indexed: Google knows the URL but has not crawled it. On large sites this usually points to crawl demand or server capacity, not page quality.
- Crawled – currently not indexed: fetched, then left out. Typical causes are thin or near-duplicate templates, faceted URLs and weak internal signals.
- Duplicate without user-selected canonical and Duplicate, Google chose different canonical than user: your canonical signals disagree with each other or with what Google sees.
- Soft 404: a 200 response that looks empty or like an error. Check out-of-stock, empty search and filter pages.
- Excluded by 'noindex' tag, Blocked by robots.txt, Page with redirect, Not found (404): often correct, but confirm each one is intentional.
The report is a sample with a delay, so do not treat a single day's change as a trend. What matters is the ratio of valuable URLs in each bucket. A few thousand excluded parameter URLs is healthy; a few hundred excluded product pages is a revenue problem.
Map every bucket back to a template. Indexing problems are almost always template problems, and the fix belongs in code, not in a spreadsheet of individual URLs.
03 / Canonicals and hreflang
Make canonical and language signals agree
Google treats rel=canonical as a strong hint, not an order. It weighs the tag against redirects, internal links, sitemap entries and content similarity. When those signals disagree, Google picks its own canonical, and you see it in the report above.
Canonical checks
- Every indexable page carries one absolute, self-referencing canonical in the server-rendered HTML.
- Exactly one canonical exists after rendering. Google's JavaScript documentation supports injecting it with JavaScript but does not recommend it, and warns against conflicting tags.
- Internal links, sitemaps and hreflang point at the canonical URL, not at a redirect or parameter variant.
- Paginated series canonicalise to themselves, not to page one.
Hreflang checks
- Each language version lists all alternates, including itself, and the links are reciprocal. One-way annotations are ignored.
- Codes use ISO 639-1 language codes with optional ISO 3166-1 region codes. Ukrainian is
uk; ua is a country code and a common mistake. - An
x-default covers users who match no version, typically a language selector or the main market. - Hreflang targets return 200 and are canonical. Pointing hreflang at a redirected or noindexed URL breaks the cluster.
Multilingual sites fail here more than anywhere else, usually because translation and engineering teams own different halves. We cover the workflow side in localisation without friction.
04 / Sitemaps and redirects
Sitemaps, redirects and the host you actually serve
XML sitemaps are a discovery aid, not an indexing guarantee. Google said in 2023, when it retired the sitemap ping endpoint, that it does not use changefreq or priority, and that it uses lastmod only when the value is consistently accurate. A build that stamps today's date on every URL teaches Google to ignore the field.
Sitemap checks
- Only canonical, indexable URLs that return 200. No redirects, no noindex, no parameter duplicates.
lastmod changes only when the main content changes.- Sitemaps are split by template (products, categories, articles) so the Page indexing report can be filtered per sitemap.
- The sitemap index is declared in robots.txt and submitted in Search Console.
Redirect and host checks
- One preferred host.
http://, http://www. and the non-preferred https:// variant each reach the final URL in a single 301 or 308 hop. - Trailing-slash and letter-case policy is consistent and enforced at the edge, not partly in the app and partly in the CDN.
- No redirect chains over two hops, and no loops. Re-crawl the old URL list after every migration.
- Removed pages return 404 or 410. Redirecting everything to the homepage tends to be reported as soft 404.
Host and redirect logic often lives in three places: the app, the web server and the CDN. Audit all three. When we rebuild platforms in our web development work, this rule set is the first thing we move into one place.
05 / Rendering
JavaScript rendering and the 2 MB ceiling
Google renders JavaScript, but rendering is a separate stage with its own rules. In December 2025 Google revised its JavaScript SEO basics page on three points that belong on every audit.
- noindex in the original HTML wins. Google may skip rendering when it sees
noindex, so JavaScript that removes the tag later may never run. If a page should be indexed, do not ship noindex in the initial HTML. - Non-200 pages may not be rendered. Pages returning 200 go to rendering; pages with other status codes might not. Error pages that rely on JavaScript to show content or signals are fragile.
- Canonicals via JavaScript are supported but not recommended. Keep them in the server response.
The byte budget
The 2 MB fetch limit sounds generous until you meet a framework that inlines its full state as JSON, or a page builder that embeds SVG icons hundreds of times. Measure the uncompressed HTML size of your heaviest templates. If critical content, links or structured data sit near the end of a large document, move them up or slim the document. Google says the limit may change, but content past the cutoff currently does not exist for Googlebot.
Rendering checks
- Compare raw HTML with rendered HTML in URL Inspection for each key template.
- Primary content, internal links and product data are present without user interaction.
- Links are real
<a href> elements, not click handlers. - JavaScript and CSS needed for rendering are not blocked in robots.txt.
Heavy rendering also hurts users. If templates are slow as well as large, read our notes on Core Web Vitals for e-commerce.
06 / Logs and bots
Log files and bot verification
Logs are the only place you see what crawlers did, rather than what a report summarises. Pull at least 30 days of raw access logs from the origin and the CDN; the edge often absorbs requests the origin never sees.
What to extract
- Googlebot hits by template and status code. Large shares of 3xx, 4xx or 5xx point to wasted crawling.
- Crawl frequency on money pages against parameter, filter and internal search URLs.
- Response time for crawler requests. A slow origin reduces how much Google is willing to crawl.
- Valuable URLs that receive no crawler hits at all: these are usually orphaned or buried.
Verify before you trust a user agent
Anyone can send a Googlebot user agent string. Verify with a reverse DNS lookup that resolves to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup that returns the same IP, or match the IP against Google's published JSON range files. Those files moved to developers.google.com/crawling/ipranges/ in 2026; the old path still works for now, so update any firewall automation that fetches it.
Treat AI and agent traffic separately. Google-Agent is user-triggered, so it does not follow the same rules as Googlebot and should be judged on its own merits. Web Bot Auth, which lets crawlers sign requests cryptographically, is still experimental at Google.
Unverified bots that claim to be Googlebot are a security finding as much as an SEO one. If your logs show scraping or credential stuffing behind fake crawler strings, that belongs in a security audit, with WAF rules that let verified crawlers through.
07 / Edge and on-page
CDN quirks, internal links and title length
CDNs fix performance and quietly create SEO noise. The best-known case is Cloudflare's Email Address Obfuscation. When it is on, Cloudflare rewrites email addresses in your HTML into links under /cdn-cgi/l/email-protection, an endpoint Cloudflare adds to every proxied domain (see Cloudflare's /cdn-cgi/ documentation). Crawlers that follow those links can log 404 or soft 404 URLs that you never published. If you do not need the feature, disable it under Scrape Shield. If you keep it, make sure those URLs stay out of sitemaps and internal link reports.
Check the edge for other surprises too: bot challenges or rate limits that hit verified crawlers, cached redirects that outlive a migration, and HTML rewriting features that change markup after your QA has signed it off.
Internal linking
- Every indexable URL is reachable through crawlable links, ideally within three or four clicks of the homepage.
- No orphan pages: compare the crawl with sitemaps and analytics landing pages.
- Anchor text describes the target. Navigation, breadcrumbs and related-content modules do most of the work.
Titles and descriptions
Google's documentation sets no character limit for titles or meta descriptions; it shortens them to fit the device and may rewrite them. In practice we keep titles near 50–60 characters with the key phrase first, and descriptions around 120–160 characters, so the important part survives truncation. Check for duplicates by template, not one page at a time. If the copy itself needs work, that is a content strategy task, not a technical one.
Page indexing reasons: triage table
Use this to turn Search Console buckets into engineering tickets.
| GSC reason | What it usually means | First check | Typical fix |
|---|
| Discovered – currently not indexed | Google has not crawled the URL yet | Server response times, crawl hits in logs, internal links to the URL | Faster origin, stronger internal linking, prune low-value URLs |
| Crawled – currently not indexed | Fetched but judged not worth indexing | Template uniqueness, near-duplicates, rendered content | Consolidate or enrich thin templates; noindex true duplicates |
| Google chose different canonical than user | Your canonical conflicts with other signals | Redirects, internal links, sitemap and hreflang targets | Align every signal with one URL |
| Soft 404 | 200 response that looks empty or like an error | Out-of-stock, empty filters, mass homepage redirects | Return 404/410, or add real content |
| Blocked by robots.txt | Crawling disallowed | Whether the rule is intended; blocked JS/CSS | Fix the rule, or use noindex if the aim is deindexing |
Reason names as shown in Google Search Console's Page indexing report. Always sample URLs in URL Inspection before acting.
The one-page technical SEO checklist
Run these in order; each step depends on the one before it.
- [+] Export every Page indexing reason and map each bucket to a template.
- [+] Confirm robots.txt allows the CSS and JavaScript needed for rendering.
- [+] Check one self-referencing canonical per indexable page, in server HTML.
- [+] Validate hreflang: reciprocal links, valid codes (uk, not ua), x-default, 200 targets.
- [+] Strip redirects, noindex and parameter URLs from sitemaps; keep lastmod honest.
- [+] Enforce one host and one-hop redirects for http, www and trailing-slash variants.
- [+] Compare raw and rendered HTML; keep noindex and canonical out of JavaScript.
- [+] Measure HTML size on heavy templates against Googlebot's 2 MB fetch limit.
- [+] Analyse 30 days of origin and CDN logs by template and status code.
- [+] Verify Googlebot by reverse DNS or Google's IP range files at the new /crawling/ipranges/ path.
- [+] Audit CDN features: email obfuscation, bot challenges, cached redirects, HTML rewriting.
- [+] Find orphan pages and check titles and descriptions for duplicates by template.