Every URL on your site answers with a status code: 200 when all is well, 301 or 302 when it redirects, 404 when it does not exist, 500 when the server fails. Checking them one by one does not scale, and the typical problems (multi-hop redirect chains, loops, intermittent errors) only show up when you crawl the whole site.
AD2SPIDER requests each URL without following redirects automatically, so it sees every hop: how many redirects there are before the final destination, whether any of them loops, whether an HTTP page is still live on an HTTPS site, or whether the server is too slow to answer at all.
Redirects and status codes errors
They stop the page from working or being indexed properly. Fix them first.
No response from server
- Why it matters
- The URL returned no response (DNS, connection, TLS or timeout). Neither users nor search engines can reach it.
- How to fix it
- Check that the domain resolves, the server accepts connections and the certificate is valid. If it was transient, recrawl.
Client error (4xx)
- Why it matters
- The internal URL returns a 4xx error: it does not exist or is not accessible. It wastes crawling and links.
- How to fix it
- Restore the content, 301-redirect to the equivalent URL, or remove the links pointing here.
Server error (5xx)
- Why it matters
- The server failed to respond. Persistent 5xx errors make Google slow down crawling and may drop URLs from the index.
- How to fix it
- Check the server logs for this URL and fix the failure; if it was transient, recrawl to confirm.
Redirect loop
- Why it matters
- The redirects form a cycle: neither users nor search engines ever reach a page.
- How to fix it
- Break the cycle so one URL returns 200 or redirects outside the loop.
Redirects and status codes warnings
Real problems worth fixing.
Destination blocked for security
- Why it matters
- The URL resolves to a private, local or cloud-metadata network and AD2SPIDER does not request it (SSRF protection).
- How to fix it
- If it is a legitimate intranet site, enable intranet mode for the project. If it is a public link, check where its DNS points.
Blocked by robots.txt
- Why it matters
- robots.txt prevents crawling this internal URL. robots.txt controls crawling, not indexing: Google may index it without content if it gets links.
- How to fix it
- If it should rank, remove or adjust the rule shown. If it must not be indexed, allow crawling and use noindex instead.
Blocked for this crawler, but not for Googlebot
- Why it matters
- robots.txt blocks this URL for the crawl's user agent, but the Googlebot group allows it: Google does crawl it and the audit cannot see its content.
- How to fix it
- To audit it as Google does, crawl with the Googlebot user agent or with «ignore robots.txt but report». If Google should not see it either, block it for Googlebot too.
HTTP URL on an HTTPS site
- Why it matters
- The site is served over HTTPS but this internal URL uses HTTP. It splits signals and exposes users to unencrypted connections.
- How to fix it
- 301-redirect HTTP → HTTPS and update links to use HTTPS.
Response too large
- Why it matters
- The body exceeded the configured limit and was not analyzed. Huge documents are slow and search engines may truncate them.
- How to fix it
- Reduce the document size (inline HTML, embedded data) or raise the crawl limit if intentional.
Redirect via Refresh header
- Why it matters
- The response uses the HTTP Refresh header to redirect. Google understands it, but a 301 is clearer and faster.
- How to fix it
- Replace it with a 301 (permanent) or 302 (temporary) redirect.
Content-Type does not match the content
- Why it matters
- The Content-Type header declares a different type from the actual content (for example, HTML served as text/plain): browsers and search engines may mishandle it.
- How to fix it
- Configure the server to send the correct Content-Type.
Redirect chain
- Why it matters
- The URL goes through two or more redirects before the destination: more latency and Googlebot may give up.
- How to fix it
- Redirect straight to the final destination and update links.
Redirects and status codes information
Data to understand the site; not errors in themselves.
Redirect
- Why it matters
- The internal URL redirects elsewhere. Normal after migrations, but internal links should point to the final destination.
- How to fix it
- Check the destination is correct and update internal links to point straight to it.