Every URL on your site answers with a status code: 200 when all is well, 301 or 302 when it redirects, 404 when it does not exist, 500 when the server fails. Checking them one by one does not scale, and the typical problems (multi-hop redirect chains, loops, intermittent errors) only show up when you crawl the whole site.

AD2SPIDER requests each URL without following redirects automatically, so it sees every hop: how many redirects there are before the final destination, whether any of them loops, whether an HTTP page is still live on an HTTPS site, or whether the server is too slow to answer at all.

Redirects and status codes errors

They stop the page from working or being indexed properly. Fix them first.

No response from server

Why it matters
The URL returned no response (DNS, connection, TLS or timeout). Neither users nor search engines can reach it.
How to fix it
Check that the domain resolves, the server accepts connections and the certificate is valid. If it was transient, recrawl.

Priority: highTechnical standardDocumentation ↗

Client error (4xx)

Why it matters
The internal URL returns a 4xx error: it does not exist or is not accessible. It wastes crawling and links.
How to fix it
Restore the content, 301-redirect to the equivalent URL, or remove the links pointing here.

Priority: highTechnical standardDocumentation ↗

Server error (5xx)

Why it matters
The server failed to respond. Persistent 5xx errors make Google slow down crawling and may drop URLs from the index.
How to fix it
Check the server logs for this URL and fix the failure; if it was transient, recrawl to confirm.

Priority: criticalTechnical standardDocumentation ↗

Redirect loop

Why it matters
The redirects form a cycle: neither users nor search engines ever reach a page.
How to fix it
Break the cycle so one URL returns 200 or redirects outside the loop.

Priority: highTechnical standardDocumentation ↗

Redirects and status codes warnings

Real problems worth fixing.

Destination blocked for security

Why it matters
The URL resolves to a private, local or cloud-metadata network and AD2SPIDER does not request it (SSRF protection).
How to fix it
If it is a legitimate intranet site, enable intranet mode for the project. If it is a public link, check where its DNS points.

Priority: mediumBest practiceDocumentation ↗

Blocked by robots.txt

Why it matters
robots.txt prevents crawling this internal URL. robots.txt controls crawling, not indexing: Google may index it without content if it gets links.
How to fix it
If it should rank, remove or adjust the rule shown. If it must not be indexed, allow crawling and use noindex instead.

Priority: mediumDocumented by GoogleDocumentation ↗

Blocked for this crawler, but not for Googlebot

Why it matters
robots.txt blocks this URL for the crawl's user agent, but the Googlebot group allows it: Google does crawl it and the audit cannot see its content.
How to fix it
To audit it as Google does, crawl with the Googlebot user agent or with «ignore robots.txt but report». If Google should not see it either, block it for Googlebot too.

Priority: lowDocumented by GoogleDocumentation ↗

HTTP URL on an HTTPS site

Why it matters
The site is served over HTTPS but this internal URL uses HTTP. It splits signals and exposes users to unencrypted connections.
How to fix it
301-redirect HTTP → HTTPS and update links to use HTTPS.

Priority: mediumDocumented by GoogleDocumentation ↗

Response too large

Why it matters
The body exceeded the configured limit and was not analyzed. Huge documents are slow and search engines may truncate them.
How to fix it
Reduce the document size (inline HTML, embedded data) or raise the crawl limit if intentional.

Priority: mediumBest practiceDocumentation ↗

Redirect via Refresh header

Why it matters
The response uses the HTTP Refresh header to redirect. Google understands it, but a 301 is clearer and faster.
How to fix it
Replace it with a 301 (permanent) or 302 (temporary) redirect.

Priority: lowDocumented by GoogleDocumentation ↗

Content-Type does not match the content

Why it matters
The Content-Type header declares a different type from the actual content (for example, HTML served as text/plain): browsers and search engines may mishandle it.
How to fix it
Configure the server to send the correct Content-Type.

Priority: lowTechnical standardDocumentation ↗

Redirect chain

Why it matters
The URL goes through two or more redirects before the destination: more latency and Googlebot may give up.
How to fix it
Redirect straight to the final destination and update links.

Priority: mediumDocumented by GoogleDocumentation ↗

Redirects and status codes information

Data to understand the site; not errors in themselves.

Redirect

Why it matters
The internal URL redirects elsewhere. Normal after migrations, but internal links should point to the final destination.
How to fix it
Check the destination is correct and update internal links to point straight to it.

Priority: lowTechnical standardDocumentation ↗