User guide

FAQ and troubleshooting

What to do when a crawl is slow, the server blocks you, pages are missing or an integration fails, and where your data is stored.

The crawl is slow

  • Speed: by default, 2 requests are made at a time to the same server with a 250 ms pause, a cautious pace. On a powerful server that you own, you can raise Concurrent per host and lower Min delay in Speed.
  • JavaScript rendering: it opens each page in a browser and is several times slower. Use it only if the website needs it.
  • Final phase: if the status is Auditing…, semantic analysis, AI or PageSpeed are running (each PageSpeed measurement takes 10 to 60 seconds). The header shows the progress of each phase.
  • Slow server responses: look at the explorer’s Response times panel. If the server takes seconds to respond, the problem is on its side, not the crawl’s.

The server blocks me or returns 429/503

  • Lower Concurrent per host to 1 and raise Min delay (for example, to 1,000 ms).
  • Keep Respect server Retry-After turned on.
  • If you’re using the Googlebot user agent, switch back to the AD2SPIDER one: many firewalls block anyone who impersonates Google.
  • On your own websites, ask for an exception to be added for the AD2SPIDER user agent.

The crawl never ends or finds too many URLs

This is usually caused by URLs that are generated endlessly: calendars, combinable filters, session parameters or sort orders.

  • Look at the explorer’s URL tab (parameters, repetitive paths) and at Site structure in the side panel to find the area that keeps growing.
  • Exclude those URLs with a regular expression in Include / Exclude, remove the parameters that don’t change the content in URL rewriting or limit the number of parameters in Limits.
  • Always set a reasonable Max URLs.

Pages I know exist are missing

  • robots.txt: check the Blocked by robots filter in Response codes. Each URL’s details show which rule blocks it.
  • nofollow: nofollow links aren’t followed by default.
  • JavaScript: if the menu or the links are generated in the browser, turn on rendering.
  • Subdomains or folders: check the Scope.
  • Unlinked pages: if they’re only in the sitemap, make sure Discover sitemaps from robots.txt is turned on or add the sitemap in Crawl. They’ll show up as orphan pages in Links.
  • Limits: the reason the crawl ended, shown in Crawls, tells you whether it stopped because of the URL limit or the time limit.

Every page shows up as a duplicate or as thin content

The content area probably includes blocks that repeat on every page (menu, footer, sidebar, cookie notice). Exclude their tags, classes or IDs in Content and crawl again.

Spelling flags correct words

  • Proper names, brands or technical terms: add them to Ignored words.
  • Language variant: if the website is in British English or Brazilian Portuguese, choose that variant instead of Automatic.

PageSpeed or AI returns no results

  • Look at the Crawl warnings block in the Summary panel: it says what wasn’t measured and why (see warnings).
  • Check the key with Test in its section of the configuration.
  • PageSpeed: the key must have the “PageSpeed Insights” and “Chrome UX Report” APIs enabled. If the daily quota runs out, the phase stops and the crawl still completes.
  • AI: the message in the column or in the details explains the reason: invalid key, no balance, model doesn’t exist or rate limit. Check the model name with ↻. With Ollama, make sure it’s running and the model is downloaded.

The configuration says it isn’t valid

The footer of the configuration window shows the first error; hover over it to see all of them. The most common ones: a badly written regular expression, an integration turned on without its key or without its module installed, or JavaScript rendering without its browser (WebView2 on Windows; on a Mac, not downloaded yet).

JavaScript rendering doesn’t work

On Windows it needs WebView2, the Windows browser component. It comes with Windows 11; on Windows 10, install it from Microsoft’s website (“Microsoft Edge WebView2 Runtime”).

On a Mac it needs AD2SPIDER’s own browser (Chrome for Testing): download it in Settings → Rendering → JavaScript browser. The same place tells you when a new version is out; it’s worth updating, because the browser opens third-party websites.

The Rendering section shows the installed version or the error.

Where my data is stored

What Where
Projects and crawls Documents\AD2SPIDER (on a Mac, Documents/AD2SPIDER), one folder per project
License, scheduled crawls and logs Windows: %APPDATA%\com.ad2place.ad2spider · Mac: ~/Library/Application Support/com.ad2place.ad2spider
Modules (spelling, semantic and, on a Mac, the browser) Windows: %LOCALAPPDATA%\com.ad2place.ad2spider\modules · Mac: ~/Library/Application Support/com.ad2place.ad2spider/modules
API keys, passwords and sessions Windows Credential Manager or macOS Keychain, encrypted with your user

To back up your work, just copy the AD2SPIDER folder in Documents. Passwords and keys don’t travel with the projects: on another computer you’ll have to enter them again.

I’ve run out of disk space

The crawl pauses on its own before the disk fills up. Free up space and click Resume. To reclaim space in the project, delete old crawls you don’t need in Crawls: each one shows how much space it takes up.

I need help

If something doesn’t work as you expect, write to us at soporte@ad2spider.com and tell us what you were doing and what happened. Attaching that day’s log helps a lot; it’s in %APPDATA%\com.ad2place.ad2spider\logs (Windows) or in ~/Library/Application Support/com.ad2place.ad2spider/logs (Mac; in the Finder, Go → Go to Folder).