User guide

Crawl configuration

Every option in the Configure window, section by section: what each one does, its default value and when it makes sense to change it.

The configuration decides what gets crawled, how fast, with which limits and which analyses run. Open it with the Configure button in the project header. It applies to the next crawl: finished crawls keep the configuration they were launched with, so you can always tell which options produced each result.

Configuration window with the section tree on the left

The window has three areas:

  • The section tree (left), grouped into Spider, Crawl control and Integrations. The search box at the top filters sections by name.
  • The open section (right). Every option has a “?”: hover over it to see what the option is for, and click More in the guide → to come to this chapter.
  • The footer, with live validation and profiles.

The default values work for the vast majority of websites. If this is your first crawl, just enter the URL in Start and click OK.

The “?” next to each option explains what it does and links to this chapter

Validation and profiles

As you change options, the footer shows ✓ Valid configuration or the first error found (for example, a badly written regular expression or an integration without its key). While there are errors, OK is disabled; hover over the message to see all of them.

Button What it does
Import profile… Loads a configuration saved in a .json file. The project’s start URLs are kept.
Export profile… Saves the current configuration (without the start URLs) to a .json file, so you can reuse it in other projects or share it with your team.
Set as default This configuration becomes the starting point for new projects.
Restore factory defaults New projects go back to starting with the factory defaults.

Profiles never include passwords, cookies or API keys: that data is stored separately and encrypted.

The robots.txt, URL rewriting, Scope and Include / Exclude sections also show the Test a URL box. Type any address and you’ll see, with the configuration currently on screen, which URL would be requested (and whether it gets rewritten), whether it would count as internal, external or excluded, whether your custom robots.txt would block it and which key it would be grouped under when looking for duplicates. It’s the fastest way to check a regular expression before you start the crawl.

Start

Option What it does
Spider (follows links) Starts from the start URLs and follows links until it has gone through the site. This is the usual mode.
URL list Checks only the URLs you provide, without following links. Useful for checking a migration, a list of redirects or the URLs in a sitemap.
Start URLs One per line. The home page is usually enough. Add more if some areas aren’t linked from the home page (for example, another language version).

In list mode you can paste the URLs, use Load file… (txt, csv, xlsx or a sitemap) or Download sitemap… from its address.

Crawl

Which kinds of links are followed in addition to regular <a> links. Anything not followed is still recorded (it shows up in each page’s links), but it isn’t requested.

Option What it does Default
Images Requests linked images to check that they load, their size and their alt text. Yes
CSS / JavaScript Requests stylesheets and scripts to check that they respond correctly. Yes
Audio and video Requests the files in <video> and <audio>. They can be very large. No
srcset variants Requests every size of responsive images. Multiplies the number of requests. No
Other <link> tags favicon, preload, manifest… No
Canonicals Requests the canonical URL even if it isn’t linked, to check that it exists and returns 200. Yes
Hreflang Requests the language versions to check that they exist and link back. They don’t count toward depth. Yes
Pagination (rel next/prev) Follows the old pagination in the head. No
Meta refresh Follows redirects made with <meta http-equiv="refresh">. Yes
Iframes Follows iframe content. Yes
Check external links Requests each external link once to detect broken ones, without crawling those sites. No
Follow internal / external nofollow links Also follows nofollow, ugc or sponsored links and links on pages with a nofollow meta robots tag. Google doesn’t follow them. No
Discover sitemaps from robots.txt Reads the declared sitemaps to find orphan pages and sitemap URLs that shouldn’t be there. Yes
Additional sitemaps Sitemaps that aren’t listed in robots.txt. One per line. —

Limits

Caps that keep a crawl from getting out of hand. An empty field means no limit.

Option What it does Default
Max URLs Maximum number of URLs in the crawl. 100,000
Max depth Maximum clicks from the start URL. No limit
URLs per depth level Maximum internal URLs at each level. Useful for taking a sample of a huge site. No limit
Max folder depth Folders in the path: /a/b/ is 2. No limit
Max query parameters Keeps the crawl from getting lost in filter combinations (?color=…&size=…). No limit
URLs per host Useful when crawling several subdomains. No limit
Max redirect hops Hops followed before the chain is considered broken. 10
Max URL length Longer URLs aren’t crawled: they’re usually parameter traps. 2,048
Max duration (s) The crawl stops when this time is reached. No limit
Limits by path Maximum URLs for paths that match a regular expression. Example: /blog/ with 500 crawls only 500 articles. —

In addition, the crawl pauses on its own if the project’s disk is almost out of space: free up space and click Resume.

Rendering

Option What it does Default
Rendering Text only analyzes the HTML the server sends; it’s fast and enough for most websites. JavaScript opens each internal page in a browser (on Windows, WebView2; on a Mac, the Chrome the app downloads the first time) and analyzes the final result, the way Google does. Use it on sites built with React, Vue, Angular… whose content or links are generated in the browser. Text only
Device Mobile (like Googlebot smartphone) or desktop. Mobile
Maximum wait after load (ms) How long to wait for the page to finish painting content; if the network goes idle earlier, it stops waiting. 5,000
Pages at once Each one is a browser (100–150 MB of memory): raise it only if your computer has memory to spare. 3
Screenshots Saves a screenshot of each page (30–80 KB), visible in the URL details. Yes
Accessibility audit (axe-core) Checks the WCAG criteria on each rendered page. No

With JavaScript turned on, the JavaScript tab is also enabled (console errors, blocked resources, links that only exist after rendering) and, if you choose it, Accessibility.

Advanced

Option What it does Default
Connect / First byte / Total Network timeouts in milliseconds. 10,000 / 30,000 / 60,000
Retries On temporary errors (5xx, network drops). 2
Accept cookies during the crawl Like a browser. Some websites need this to show content. No
Respect HSTS policy If a site has already sent HSTS, its http:// links are treated as a hop to https:// without requesting them. When it’s off, you see the real redirect that Google sees. No
Self-referencing meta refresh = non-indexable A page that refreshes to itself counts as non-indexable. Yes
In list mode, follow redirects Follows the redirects of the URLs in the list to their destination. Yes
Store downloaded HTML Don’t store takes up the least space; Sample stores 50 documents; All HTML pages or Everything let you look at the code afterward. Don’t store

Preferences

The rule thresholds. These aren’t Google rules but editorial criteria you can adapt to each project.

Option Default
Page title: min/max characters and pixels 30–60 characters · 200–561 px
Meta description: min/max characters and pixels 70–155 characters · 400–985 px
URL: max characters 115
H1 / H2: max characters 70 / 70
Alt: max characters 100
Image: max bytes 102,400 (100 KB)
Low content (words) 200
Near duplicates: minimum similarity (%) 90
Max click depth 3
Max internal / external outlinks 1,000 / 10
Readability “difficult” / “very difficult” below 50 / 30 (0–100 scale: Flesch in English, Fernández Huerta in Spanish)
Min words for readability 20
Non-descriptive anchors “click here”, “learn more”, “here”… one per line

Under Reporting:

  • Ignore non-indexable pages for content issues (on): title, meta, heading or word count issues are only flagged on indexable pages, because there’s no point optimizing a page that has noindex or is canonicalized to another one.
  • Report pages without canonical, Report missing HSTS and Report missing security headers (on): turn them off if you don’t care about those warnings in a project.

Structured data

Option What it does Default
JSON-LD, Microdata, RDFa Formats that are read. All three
Validate against schema.org Checks that the types and properties exist in the vocabulary. Yes
Validate Google rich results Missing required properties = error; missing recommended ones = warning. Yes
Case-sensitive types and properties Not recommended: Google is case-insensitive. No

Content

  • Content area: tags (by default nav and footer), classes and IDs that do not count as content. That way the menu, the footer, cookie banners or sidebars don’t inflate the word count or make every page look like a duplicate. Word counts, readability, duplicates, spelling, semantic analysis and the text sent to AI all come from this area.
  • Soft 404: phrases that give away an error page that returns 200 (“page not found”, “error 404”…). Add the ones your site uses if they’re different.
  • PDF: reads internal PDFs up to 30 MB (title, author, pages, words and links, which are also crawled). Their data appears in the PDF tab.

Spelling & grammar

The check uses LanguageTool, which you download once from this section (Download and install, about 290 MB) and which works offline: the text of your pages never leaves your computer.

Option What it does
Check the spelling of the content Checks the content area of each internal HTML page. It makes the crawl a little slower.
Check grammar too Besides spelling, agreement, punctuation…
Language Automatic (lang attribute or detection) uses each page’s language (the lang attribute or detection). Pick a fixed one for a specific variant, for example British English.
Ignored words Brands, proper names or technical terms that aren’t mistakes. One per line.

The results are in the explorer’s Spelling tab and in each URL’s details.

robots.txt

Option What it does Default
Respect robots.txt Doesn’t request URLs that robots.txt disallows, just like Google. Yes
Report URLs robots.txt would have blocked If you don’t respect it, it still reads it and flags what it would have blocked. —
Custom robots.txt A test robots.txt that replaces that host’s real one during the crawl. Ideal for testing changes before publishing them. —

URL rewriting

Changes applied to discovered URLs before they’re requested (never to the start URLs), in this order:

  1. Remove parameters: utm_source, sessionid… With * at the end, every parameter that starts that way (utm_*).
  2. Regex replacements, in order. Example: ^http:// to https://. Supports $1 groups.
  3. Lowercase URLs.

Ignore parameters when detecting duplicates is different: the URL is crawled as is, and only the way duplicates are grouped changes.

Scope

What counts as internal (crawled and audited) and what counts as external (only checked).

Option What it does Default
Include subdomains blog.example.com, shop.example.com… count as internal. No
If the start URL is the root domain, include its subdomains Starting at https://example.com/, its subdomains are internal. Yes
Crawl only inside the start folder Starting at /blog/, anything outside it is checked but not followed. No
Intranet mode Allows local network addresses and localhost. Off for security reasons. No
CDNs and extra hosts Other hosts that count as internal. Supports *.domain and a folder: cdn.example.com/static/. —

Include / Exclude

Regular expressions matched against the full URL, one per line.

  • Include: if you add any, only internal URLs that match one of them are crawled. Example: ^https://example\.com/blog/.
  • Exclude: matching URLs aren’t crawled. Example: \?orderby= to skip a store’s sort orders.

Start URLs are always crawled. Use Test a URL to check the patterns.

Speed

Option What it does Default
Concurrent requests In total. 8
Concurrent per host To the same server. This is what puts the most load on the site. 2
Min delay per host (ms) Between two requests to the same server. 250
Random jitter (ms) Added to the delay so the pace isn’t fixed. 100
Respect server Retry-After If the server asks you to wait (429, 503), the crawler waits. Yes

If the server starts returning 429 or 503 errors, lower the concurrent requests per host or raise the delay.

User-Agent

How the crawler identifies itself. AD2SPIDER identifies itself by name with an information page. Googlebot (smartphone or desktop) lets you see what the site serves to Google, but many firewalls block anyone who impersonates it: use it only on your own sites or with permission. Custom lets you write the string and the token used to look up the rules in robots.txt.

HTTP headers

Headers sent with every request to the crawled site, for example Accept-Language: es-ES to see the Spanish version. They’re stored unencrypted, so don’t put keys or tokens here: use the secret headers in Authentication for that. For security reasons, Authorization, Cookie and Host aren’t allowed.

Authentication

For crawling private areas. It’s available once the project exists, because the credentials are stored encrypted inside it.

  • Login (form): click Open browser to sign in, sign in with your account as you normally would and click Save session. The crawl will use those cookies until the session expires on the site.
  • Basic authentication (HTTP): for servers that ask for a username and password in a browser window (typical of staging environments). Enter the host, username and password and click Save.
  • Secret headers: for sites that ask for a token in a header instead of a username and password, for example a staging site behind a firewall or CDN, or Authorization: Bearer …. Enter the host, the header name and its value and click Save. It’s only sent to that host over HTTPS (or to the exact address of the start URL), also when rendering JavaScript; never to other domains.
  • Do not follow logout or delete links: while signed in, URLs that look like actions (sign out, delete, unsubscribe…) aren’t requested.

Passwords, secret header values and cookies are encrypted with a key stored in the system key store (Credential Manager on Windows, Keychain on a Mac): only your user can read them, and they’re never included in exported profiles.

Custom

  • Custom search (up to 100): looks for a text or a regular expression in the HTML, in the visible text or inside a CSS or XPath selector, and creates a filter with the pages that contain it (or don’t). Example: pages without the Analytics code.
  • Custom extraction (up to 100): pulls a piece of data from each page with XPath, a CSS selector (with an optional attribute) or regex: prices, authors, dates, SKUs… You can get the element, its inner HTML, its text or the value of an XPath function such as count(//h2).
  • Custom JavaScript (up to 20): your code runs on each rendered page, to extract data (whatever it returns) or to act before the page is analyzed (scroll to the bottom, click a button). It requires JavaScript rendering. Add from the library… brings in ready-made examples: scroll to the bottom, dataLayer events, Google Analytics and Tag Manager IDs, main content words or lazy-loaded images that haven’t loaded.

Each search, extraction or script becomes a column and a filter in the Custom tab.

Semantic analysis

Compares the meaning of your pages with an AI model that runs on your computer: no key needed, no cost, and the content never leaves your machine. The model (multilingual, about 490 MB) is installed once from this section with Download and install.

Option What it does Default
Analyze content with embeddings when the crawl ends Calculates the meaning of each page and compares it with the rest of the pages in its language. About 6 pages per second. No
Similar from (%) Similarity above which two pages are considered similar: possible cannibalization. 60
Suggest link from (%) Similarity above which a link is suggested between two pages that don’t link to each other yet. 45
Suggestions per page Maximum suggested links. 5

Anything repeated on every page (menu, footer, notices) is discarded before comparing, and each language is compared separately. The results are explained in Integrations.

AI (prompts per URL)

Sends a request (prompt) to ChatGPT (OpenAI), Claude (Anthropic), Gemini, Perplexity or Ollama for each page and saves the answer as a column. It uses your own key for each provider: the provider bills the cost of each query directly.

API keys and prompts with templates

  • API keys: paste the key for the provider you’re going to use, click Save and then Test (it costs nothing, except for a minimal query on Perplexity). It’s stored encrypted on your computer and works for every project. Ollama doesn’t need a key: install it, download a model and click Check.
  • Send the prompts to the pages when the crawl ends: turns on this phase.
  • Pages per prompt (max.) (100) and Text characters per page (max.) (6,000, about 1,500 tokens): these control the cost.
  • Prompts (up to 20): name, provider, model (↻ loads the ones in your account), what is sent (Content or Metadata only), maximum answer length and the request. Add from template… brings in six ready-made ones: suggested title, suggested meta description, search intent, main keyword, summary, and title, meta and H1 review.

More details, costs and privacy in Integrations.

PageSpeed Insights

Measures page performance with Lighthouse on Google’s servers and adds real Chrome user data (Core Web Vitals).

Option What it does Default
Google API key Free, with the “PageSpeed Insights” and “Chrome UX Report” APIs enabled (25,000 queries per day). It’s stored encrypted on your computer. —
Measure with PageSpeed Insights when the crawl ends Each page takes 10 to 60 seconds. No
Mobile / Desktop Each one is a separate measurement. Mobile
Maximum pages From shallowest to deepest. 100
Real-user data (Chrome UX Report) LCP, INP and CLS from real users over the last 28 days, for the URL or, if there isn’t enough data, for the domain. Yes

How to create the key and how to read the results, in Integrations.