In one paragraph
Crawled pages returning 4xx or 5xx means observed targets produced statuses this check treats as errors. Determine whether each response is an intentional removal, a wrong reference, an access issue, or a service failure before choosing a repair. One request does not prove an outage affects every visitor. Excluded or unmeasured cases can still represent real problems and require separate investigation.
Why it matters
A 404 or 410 can be the correct result for content that no longer exists. Continuing to advertise it through navigation or a sitemap is a separate maintenance issue. Restore or redirect only when a meaningful replacement or intended resource actually exists.
A 5xx indicates failure for the observed request, not necessarily for every user at every time. Intermittent load, a particular edge, routing, bot policy, or a backend dependency can change the result. Verify the exact public URL and evidence instead of assuming a permanent universal outage.
No network response can also indicate real availability trouble, even when the scanner leaves it ungraded. Measurement limits should not be translated into a claim that the site is healthy or that the failure necessarily belongs to the scanner.
How WebAuditOS evaluates this finding
The supplied rule grades final observed status after its redirect handling. Eligible 4xx failures are high and 5xx failures critical, while 401, 403, 429, known bot challenges, and unavailable responses are excluded as documented. It also recognises a specific WordPress database-error body served with 200.
Some transient cases receive one retry. Start-URL error handling can stop a scan before a normal finding is generated. Those behaviours and crawl budgets limit coverage. A 2xx or accepted 3xx pass is not proof of useful content, successful rendering, or the absence of soft errors.
How to fix it
- Inspect the exact URL, final status, redirect chain, timing, and any retry evidence. Compare the same path and query rather than substituting a working menu destination.
- Reproduce safely from the public delivery path. For example:
curl -sS -L -D - -o /dev/null https://example.com/old-routeUse an appropriate actual URL. This GET inspection shows headers, not every browser interaction or authenticated workflow.
- For missing content, decide whether to restore it, redirect a genuinely moved equivalent, or retain the correct 404/410 and remove stale references. Avoid irrelevant homepage redirects.
- For server failures, inspect application, proxy, edge, and dependency diagnostics. Keep useful private error logging without exposing stack traces or masking every failure as a successful page.
- Investigate excluded access or rate-limit responses when they affect intended public use. A legitimate authentication boundary and an accidentally blocked sign-up are different situations despite sharing a status code.
- Find the discovery source: internal links, sitemap entries, scan input, or another supported crawl source. Do not assume the product always provides every referring page; verify its actual evidence and inspect your own templates or content.
- Test after deployment across relevant hostnames, edge paths, signed-out states, and direct routes. Record intermittent behaviour rather than dismissing it because one local request succeeded.
- Rescan and monitor real application outcomes. Correct statuses, useful error recovery, and source-link maintenance are separate tasks that should remain consistent.
What the fix can break
A bulk redirect can hide a genuine backend error or misroute valid pages. Make corrections according to intent and test live routes too.
Security controls can create different results for probes and visitors. Do not disable necessary protection globally just to make a scan succeed; diagnose the specific policy and authorised access.
Do not describe a successful retry as proof there is no reliability issue. Frequency and actual user impact require broader observation.
Questions
Are all 404 responses defects?
No. They can be correct for removed or unknown content. Review whether the route is still advertised or expected to work.
Does a 5xx prove everyone is down?
No. It records the failed observation. Investigate scope, repetition, and affected users.
Does ungraded mean healthy?
No. It can mean unavailable evidence or an excluded status requiring another test.
Which related checks help?
Review broken links, soft 404s, and the indexing checklist.