User guide

Compare crawls

What changed between two crawls: new and resolved issues, new and missing URLs and field-by-field changes, including from staging to production.

Compare sets two completed crawls of the same project side by side. It’s how you check that a change has worked, make sure nothing breaks between releases or validate a migration.

Compare

Choose the crawls

Pick the Previous crawl and the New crawl (by default, the last two completed ones) and click Compare. You need at least two completed crawls.

If the two crawls were launched with different configurations, a box warns you about the differences: a URL may “appear” simply because subdomains are now being crawled, not because it’s new.

URL mapping

To compare sites with different addresses, for example a staging environment with production or the website before and after a migration, add mapping rules: a regular expression and its replacement, with $1, $2… groups.

Pattern Replacement
^https://staging.example.com/(.*) https://www.example.com/$1
^https://www.example.com/products/(.*) https://www.example.com/shop/$1

The URLs from the previous crawl are transformed with these rules before their match is looked up in the new one.

Results

  • Summary: new URLs, missing URLs, changed URLs, new issues (in red) and resolved issues (in green).
  • Issues by rule: for each check, how many URLs fail it for the first time, how many no longer fail it and how many are unchanged.
  • Change detection: URLs that exist in both crawls and have changed. Filter by field: status code, indexability, title, meta description, H1, canonical, content (with the similarity percentage), word count, crawl depth, inlinks or internal outlinks. Each row shows the before and after values.
  • Missing URLs: with the reason (removed, out of scope or unconfirmed).
  • New URLs.

The lists show the first 200 rows. Export CSV saves the full comparison.

Scheduled crawls can generate this comparison as a CSV automatically after each run.