HTTP status codes help Googlebot understand whether a URL delivered content, moved, became unavailable, or could not be served. For SEO, the priority is making the response match the page’s actual condition and investigating when important URLs return an unexpected result.
A deleted article returning 404 can be correct. A working product page returning 403 to Googlebot is a problem. A maintenance page returning 200 may conceal an outage from a basic availability monitor.
This guide explains how to choose the right response and diagnose failures, with particular attention to server errors.
Documentation checked: October 22, 2026. Diagnostic examples are hypothetical, not results from a Scale Xpert site audit.
Start With the Page’s Intended State
Before changing a status code, decide what the URL should do:
- Serve a public page.
- Move visitors to a relevant replacement.
- Remain temporarily unavailable.
- Confirm that content no longer exists.
- Require authentication because the resource is private.
That decision gives a developer a testable requirement. “Fix our crawl errors” is less useful than “this public service page must return its full content successfully to visitors and verified Googlebot.”
For the wider relationship between accessibility and indexing, see our technical SEO guide.
A Practical Status-Code Decision Table
| Response | Typical situation | First SEO check |
|---|---|---|
| 200 | The requested page is available | Does the response contain the intended content? |
| 204 | A successful request has no content | Is that intentional for this resource? |
| 301 or 308 | A permanent move | Is the destination relevant and working? |
| 302 or 307 | A temporary move | Does the temporary behavior still match the requirement? |
| 304 | A conditional request confirms unchanged content | Are cache validators accurate? |
| 401 or 403 | Authentication is required or access is denied | Is a public page being blocked accidentally? |
| 404 or 410 | Content is missing or removed | Should the URL remain gone or have a replacement? |
| 429 | Requests are being rate-limited | Which rule or capacity problem triggered it? |
| 500, 502, or 503 | The server could not serve the request normally | Which infrastructure layer failed, and how widely? |
This is a triage table. A response must be interpreted alongside the requested resource, body, headers, and request conditions.
Why a 200 Response Does Not Guarantee Indexing
Google’s technical requirements include crawler access, a successful response, and indexable content. Meeting those requirements makes a page eligible; it does not guarantee inclusion.
A URL returning 200 may still have an accidental indexing restriction, an unsuitable canonical, or content Google chooses not to index.
It may also deliver the wrong page entirely.
For example, a deployment could return a “temporarily unavailable” template with a 200 response across hundreds of URLs. A monitoring system checking only the status might label every request successful.
Inspect the body as well. Record an expected page title, main heading, or content marker for important templates. These checks help distinguish a working article from a successful response containing an error screen.
If access works but indexing does not, continue with our discovered versus crawled, currently not indexed guide.
Redirect Codes Should Describe a Real Move
Google’s redirect documentation distinguishes permanent redirects, such as 301 and 308, from temporary redirects, such as 302 and 307. The choice affects the signals sent about which URL should appear in Search.
Use a permanent redirect when content has permanently moved to a relevant destination. A temporary change should use the appropriate temporary behavior.
Then verify the full journey.
A migration can have a valid first redirect followed by another redirect, an outdated destination, or an inaccessible final page. Looking only at the first response misses the actual outcome.
For a hypothetical article migration, the desired result might be:
Old article URL → permanent redirect → equivalent new article with a successful response.
Update internal links to point directly to the intended destination. Document why each redirected URL has a meaningful replacement; avoid treating the homepage as a universal destination for removed content.
A Correct 404 Is Different From a Broken Public Page
Google’s Page indexing guidance recommends paying attention to missing URLs you link to internally or include in sitemaps. A moved page should lead to its new location.
Classify missing URLs before creating redirects:
- Accidentally missing: restore the intended page or repair the route.
- Moved with an equivalent replacement: redirect to that destination.
- Intentionally removed without a replacement: retain an appropriate missing-content response and clean up discovery signals.
- Malformed or irrelevant URL: investigate whether an internal system is generating it.
A soft 404 is a different situation: the response appears successful, but the page behaves like missing or unusable content.
The cleanup target should be accuracy. Replacing every 404 with an unrelated redirect can make a report look cleaner while giving visitors a worse experience.
For large groups of low-value URLs, our index bloat guide helps frame the decision about which pages should exist.
Why 5xx Errors and 429 Deserve Immediate Investigation
Google’s HTTP status documentation says 5xx responses and 429 cause crawlers to slow down temporarily. Content returned with 5xx is ignored. Previously indexed URLs can remain initially, but persistent server failures can eventually lead to removal.
When successful responses resume, crawling increases gradually.
There is no universal removal deadline in that guidance. Assess the breadth and persistence of failures instead of promising that a particular outage length is harmless.
For diagnosis, separate the likely source:
| Observed pattern | Investigation starting point |
|---|---|
| Failures start after a deployment | Application errors, configuration changes, dependencies |
| Gateway errors appear while the origin struggles | Proxy or load-balancer logs and upstream connectivity |
| Service-unavailable responses occur during maintenance | Maintenance configuration and restoration plan |
| Rate limiting appears primarily for automated requests | WAF rules, request limits, and crawler verification |
| Failures affect one URL pattern | Template, route, database query, or shared dependency |
These are hypotheses to investigate, not automatic diagnoses.
Google’s guidance on pausing a website describes using 503 when urgently disabling a site for a very short period. It recommends keeping functionality available where possible and using a Retry-After header when appropriate.
A 503 accurately communicates temporary unavailability. It does not preserve visibility indefinitely.
Plan the restoration and verify it. Check that the CDN stops serving a cached maintenance response after the application is back.
If only checkout is unavailable, consider keeping product information accessible while disabling the affected function. Taking every informational page offline may be unnecessary.
Do Not Use Access Errors as Informal Crawl Controls
A public landing page returning 401 or 403 tells you access has failed. Investigate authentication, WAF rules, geographic filtering, and bot protection before blaming the content.
Keep private resources private. The requirement is to correct accidental restrictions on public content, not remove authentication from protected systems.
Google’s status documentation specifically warns against using 401 and 403 to limit crawling. It treats 429 as an overload signal instead.
Verify crawler identity before creating exceptions. Google’s request-verification documentation describes checking published IP ranges or using DNS verification. A user-agent string alone is insufficient evidence.
Logs may contain different request methods and resource types. Our article on Googlebot rendering requests in server logs provides context; request method and response status answer different questions.
Check robots.txt Separately
Do not assume a status code has the same operational consequence on robots.txt as on an ordinary article.
Google’s robots.txt specification describes special failure handling. When robots.txt encounters server errors, Google initially pauses crawling while retrying. Later behavior depends on cached rules and site availability.
For robots.txt, most 4xx responses other than 429 are treated as if a valid file does not exist, so Google assumes no crawl restrictions.
Include robots.txt in incident checks even when the homepage works.
For a maintenance rollout, ask the developer to verify the intended robots.txt response separately. A site-wide rule that intercepts every path can accidentally change this file too.
How to Investigate the Response Google Actually Received
1. Capture a Small, Useful URL Sample
Choose an affected URL, a working URL from the same template, and a URL from another section. Add robots.txt and any failing resource required to display the page.
Record the expected outcome for each before testing.
This creates a comparison that helps distinguish a single-page defect from a template or infrastructure failure.
2. Use GET to Inspect Headers and Content
These commands are examples for your own site:
curl -sS -D headers.txt -o body.html "https://example.com/page/"For a redirect journey:
curl -sS -L --max-redirs 10 -D redirect-headers.txt -o final-body.html "https://example.com/old-page/"The second command stores headers across followed responses and saves the final body. Its redirect limit is a diagnostic choice, not a promise that every Google fetcher behaves identically.
Read the files. Check the response, Location headers, content type, cache headers, and whether the body contains the intended page.
A request from your computer cannot establish what Google received earlier. Changing the user agent also does not reproduce verified Googlebot identity or its network conditions.
3. Connect Search Console With Logs
Google’s Crawl Stats report shows crawling history, response information, and availability issues. Use it to identify patterns, then examine specific URLs and timestamps.
Collect evidence from both edge and origin systems where available:
- Timestamp and timezone.
- Requested host and path.
- Response status and request method.
- Client address and verification evidence.
- Cache result or security rule.
- Upstream status and response time.
An origin-only log may miss a request rejected by the CDN before it reaches the application.
The same distinction between observed access and later outcomes matters when auditing AI crawler requests to sitemaps and RSS, although other crawlers have their own behavior.
4. Identify Scope Before Prioritizing
Estimate affected important URLs, affected templates, failure frequency, and duration. Separate retired URLs from public pages that should work.
Prioritize widespread availability failures and broken business-critical pages. A list containing thousands of intentional missing URLs is a different task from intermittent failures across your active catalogue.
Confirm Recovery at Three Levels
Use an evidence record rather than a single green checkmark.
| Level | Confirmation |
|---|---|
| Serving | Repeated checks deliver the intended status, headers, and content |
| Crawling | Subsequent verified crawler requests receive the corrected response |
| Search processing | Later inspection and indexing evidence reflect the updated state |
Keep the deployment time, correction, example URLs, and follow-up observations together.
A corrected live response establishes that serving improved. It does not prove Google has already revisited every affected URL or restored previous visibility.
If you request reprocessing through Search Console, verify the fix first. Submission is a follow-up action, not a substitute for restoring access.
Frequently Asked Questions
Does a 200 status mean Google will index the page?
No. A successful response is part of technical eligibility. Content and indexing controls still matter.
Is 503 safe for SEO during maintenance?
It can accurately describe brief unavailability. It is not an unlimited protection against reduced crawling or indexing consequences.
Does 429 behave like a normal missing-page error?
No. Google treats it as an overload signal. Investigate the rate limit and the requests triggering it.
Should I redirect every 404 to the homepage?
Choose a destination only when it meaningfully replaces the missing content. Otherwise, an accurate missing-page response can be appropriate.
Is 304 an error I need to remove?
No. In conditional crawling, it indicates unchanged content. Investigate incorrect cache validation rather than treating every 304 as a failed fetch.
What should I check first when Google receives 403 but I see the page?
Compare authenticated and public access, edge security logs, and verified crawler requests. Your browser session may follow a different path.
Make Your Responses Match Your Content
Start with the intended state of the URL, then verify the response delivered at the edge, the content behind it, and subsequent crawler evidence. This makes the repair specific enough to test and reduces the chance of fixing a dashboard while leaving the underlying access problem unresolved.
If you want to connect technical cleanup with a broader visibility plan, you can discuss your priorities with Paul Gautheron on LinkedIn. Scale Xpert focuses on SEO and Reddit visibility, bringing technical accessibility, useful content, and discovery into the same growth conversation
