Screaming Frog SEO Spider: how to crawl a site and interpret the reports

What Screaming Frog SEO Spider is, and what’s available without a license

SEO Spider (informally called «the Frog,» or just «Screaming Frog») helps check a site’s technical foundation — one of the parts that make up SEO: it shows what a robot that follows links finds on a site. According to the developer’s description, the program can find broken links, errors and redirects, analyze title tags and metadata, check robots directives, and find duplicate pages. It also builds an XML Sitemap, renders JavaScript, extracts data by custom rules, compares crawls, and pulls in data from Google Analytics, Search Console and PageSpeed Insights.

It isn’t an online service but an application you install on your computer. It’s downloaded from the program’s official page.

Free version vs. license: what’s the difference

Without a license, the program crawls no more than 500 URLs per crawl. A license removes this limit and unlocks settings and additional features:

Feature	Free version	Licensed
Crawl limit	up to 500 URLs per crawl	the 500 URL limit is removed
Crawl configuration	locked, crawls with default settings	available
Saving and reopening crawls	no	yes
JavaScript rendering	no	yes
Custom Extraction (scraping data from pages) and source code search	no	yes
Google Analytics, Search Console, PageSpeed Insights integrations	no	yes
Comparing crawls	no	yes, in database storage mode
Scheduled crawls	no	yes

A limit of 500 URLs is enough to get familiar with the program and check a small site. It won’t be enough for a full crawl of a large site, and without a license you can’t save the result or compare it with the next crawl.

How to crawl a site: preparation and launch

A crawl (crawling, or a scan) is a sequential loading of pages: the program opens the starting URL, finds the links on it, and follows them further. Before starting, you should choose a mode and check the settings, or the crawl may end up incomplete.

Crawl modes: the whole site or a list of addresses

• Spider — the main mode. The program crawls the subdomain you specify and follows links.
• List — checking a given list of addresses. This mode is turned on via Mode → List. You can paste the list, load it from a file, or pull it from an XML Sitemap. By default, this mode crawls only the addresses on the list. It’s useful, for example, for checking redirects after a site migration.

There are other modes too, including Compare, for comparing two crawls. More on it in the section on working with results.

What to check in the settings before you start

In the free version, the settings are locked, and the program crawls the site with default parameters. With a license, the settings are in the Configuration menu. Below are the parameters that affect how complete the crawl is, and what mistake each one guards against.

• robots.txt. The program follows robots.txt. It first looks for rules for its own user-agent, then for Googlebot, then for all robots. Blocked addresses aren’t crawled; you can see them on the Response Codes tab under the Blocked by Robots.txt filter. You can only ignore robots.txt in the licensed version. For more on which robots.txt directives restrict a crawl, see a separate article.
• JavaScript rendering. Text Only mode is on by default: the program reads only the raw HTML and doesn’t run client-side JavaScript. If a menu, links or text are generated by a script, the program won’t see them without JavaScript mode. JavaScript mode renders the page in a built-in Chrome browser and is available with a license. Google processes such pages in three stages: crawling, rendering, indexing.
• Include and Exclude. Regular-expression rules restrict the crawl. Include keeps only matching addresses, while Exclude removes addresses from the crawl entirely rather than just hiding them in the interface. This keeps the crawl from wandering into filter and parameter URLs.
• User-agent. This setting changes the user-agent of the requests and which robots.txt rules the program follows. By default it’s «Screaming Frog SEO Spider.»
• Speed. Speed is set by the number of threads and the maximum number of URLs per second. The developer warns: more threads means more requests to the server, which can slow down the site’s responses.
• Subdomains. By default, the program crawls only the starting subdomain and treats the rest as external sites. If the site has several subdomains, turn on crawling them separately.
• XML Sitemap. In Spider mode, sitemaps aren’t crawled by default. There’s a Crawl Linked XML Sitemaps setting for that.
• nofollow. By default, the program doesn’t follow links with the nofollow, sponsored and ugc attributes.

Crawling puts load on the server. For a large site, it’s better to crawl at a moderate speed and coordinate the crawl in advance with whoever is responsible for the server.

Starting the crawl and a first check of the result

Enter the address in the Enter URL to spider field and click Start. Data updates as the crawl runs, and the bottom of the window shows the speed and the number of URLs processed and remaining.

When the crawl finishes, check how complete it is. Does the number of HTML pages found match what you expected? Are there mass blocks or redirects right at the start, for example from http to https, or from www to a non-www address? If there are suspiciously few pages, go to the «What a crawler won’t show you» section and check robots.txt, JavaScript-driven links and nofollow.

Which reports to check for SEO

Data is organized into tabs, and each tab has filters. On the Issues tab on the right, the program collects issues, warnings and opportunities, and assigns each item a type and a priority rating. According to the developer, a warning isn’t necessarily an error — it needs to be checked.

Important: the error highlighting, priorities and length thresholds are the tool’s own settings and classification, not Google’s or Yandex’s rules. The length thresholds for title and description can be changed in the program’s settings. Before fixing anything, check against the search engines’ official documentation.

Response codes: 4xx, 5xx and broken links

A response code (HTTP status) is the number a server returns in response to a page request. The Response Codes tab shows the response codes of internal and external URLs. Broken links are links to addresses with a 4xx error. The Client Error (4XX) filter shows them.

To find the pages that link to such an address, select the URL and open the Inlinks tab at the bottom: the sources will be in the From column. You can export all «source — broken address» pairs via Bulk Export → Response Codes → Client Error (4XX) Inlinks.

External links are collected on the External tab: addresses on other subdomains end up there. This is how you check outbound links.

What this means for search:

• 4xx. Google doesn’t use the content of a URL with a 4xx code and doesn’t index it, and it removes already-indexed URLs like that from the index. Yandex’s documentation states that code 404 means «the document doesn’t exist,» and 410 means the resource has been permanently removed.
• 5xx. With 5xx and 429 errors, Google’s robots temporarily slow down crawling. Yandex’s documentation defines 500 as an unexpected server error, and 503 as a temporary overload or maintenance.

Redirects and redirect chains

A redirect is a server response that sends a robot to a different address. The Redirection (3XX) filter shows server-side redirects, such as 301 and 302. The Redirect Chain filter shows internal addresses that lead to a URL with yet another redirect. Full chains can be exported via Reports → Redirects → Redirect Chains.

How search engines interpret this:

• Google. A permanent redirect is a signal that the final address should become canonical. A temporary redirect isn’t treated as that kind of signal. By default, Google’s robots follow up to 10 hops in a chain.
• Yandex. The robot will index the page the redirect points to, if it’s accessible.

It’s better to point internal links straight to the final address: that way the robot doesn’t need to follow intermediate hops.

Canonical: self-referencing, pointing elsewhere, or missing

Canonical specifies a page’s primary address when its content is available at several URLs. The Canonicals tab has filters including Self Referencing (the canonical points to the page itself), Canonicalised (points elsewhere), Missing (not specified), and Non-Indexable Canonical.

A canonical is a hint, not a command. Google describes the ways to specify a canonical URL as recommended rather than mandatory, and will pick a primary address itself if none is given. Yandex treats the canonical address as a recommendation and may ignore it. For more on how the canonical link works, see a separate article.

Meta robots, X-Robots-Tag, and addresses blocked in robots.txt

The robots meta tag and the X-Robots-Tag HTTP header set indexing rules for a page. The Directives tab shows these directives, including the Noindex and Nofollow filters, as well as directives from the X-Robots-Tag header. Addresses blocked in robots.txt are visible in the Blocked by Robots.txt filter.

noindex prevents a page from appearing in Google’s search results. According to Yandex’s documentation, a page with noindex won’t take part in search results. If a page is blocked in robots.txt, the search engines won’t see a directive in its meta tag or header. For more on how noindex and nofollow work, see a separate article.

Title, description and H1: empty, duplicated, too long

The Page Titles, Meta Description and H1 tabs have Missing (the element is absent) and Duplicate (repeats across several pages) filters, as well as length filters. The length thresholds are the program’s own settings and can be changed.

Google has no title-length limit: it generates the title link in search results automatically, and can draw the text not just from the title but also, for example, from the page’s headings or link anchor text. The description in results is also generated automatically. Google uses the meta description if it describes the page more accurately than the page’s content does. So a length filter is a reason to check the page, not an error in itself.

Duplicate pages

On the Content tab, the Exact Duplicates filter shows pages that are completely identical, while Near Duplicates shows pages that are similar above a set similarity threshold. To populate Near Duplicates, you need to turn on near-duplicate detection in the settings (Configuration → Content → Duplicates) and run Crawl Analysis after the crawl. As a reminder, these settings are only available with a license.

Duplicates arise, for example, from GET parameters, a trailing slash, www vs. non-www versions, and http vs. https. The primary address is specified through a canonical, a 301 redirect, or, for Yandex, the Clean-param directive.

Crawl depth and internal links

Crawl depth is the number of hops from the starting page to the current one. The program shows it in the Crawl Depth column on the Internal tab. The same tab has Inlinks and Unique Inlinks columns — the number of internal links to a URL. The pages that link to a selected URL are visible in the Inlinks tab at the bottom.

Link Score is the program’s own metric from 0 to 100. It estimates a page’s relative weight based on internal links and is populated after Crawl Analysis. It’s the tool’s own calculation, not a search engine signal.

According to Yandex’s documentation, how long it takes to index an internal page depends, among other things, on its crawl depth. The documentation doesn’t name a specific number of levels. For more on how to design depth levels and what to do about pages with no inbound links, and about anchor text, see separate articles.

Summary table: what to check and how to interpret it

What to check	Where in the program	How to interpret it	What to do next
Broken links (4xx)	Response Codes → Client Error (4XX); sources — the Inlinks tab at the bottom	Google doesn’t index URLs with 4xx and removes them from the index	Fix or remove internal links to such addresses (recommendation)
Server errors (5xx)	Response Codes → Server Error (5XX)	Google temporarily slows crawling; for Yandex, 503 means temporary overload or maintenance	Pass it on to developers, recheck later
Redirects and chains	Response Codes → Redirection (3XX), Redirect Chain; Reports → Redirects → Redirect Chains	A permanent redirect signals canonicalization of the final address; a temporary one doesn’t	Point internal links to the final address (recommendation)
Canonical	Canonicals → Canonicalised, Missing, Non-Indexable Canonical	A hint, not a command	Check that the canonical points to the right address (recommendation)
noindex, meta robots	Directives → Noindex, Nofollow; X-Robots-Tag	A page with noindex doesn’t show up in search	Check that noindex is only on pages that need it
Blocked in robots.txt	Response Codes → Blocked by Robots.txt	Search engines won’t see directives on such a page	Check against the robots.txt rules
Title, description, H1	Page Titles, Meta Description, H1 → Missing, Duplicate, length filters	Google generates the title and description automatically; there’s no title-length limit	Fix empty ones and template-driven duplicates (recommendation)
Duplicates	Content → Exact Duplicates, Near Duplicates	Primary address via canonical, a 301, or Clean-param	Pick the primary address
Depth	Internal → Crawl Depth, Inlinks	Yandex: indexing time depends, among other things, on depth	Check how important pages are linked

How to export a list of all pages on a site

Exporting addresses after a crawl

The Internal tab holds all the data found on the crawled site. To keep only pages, select the HTML filter on it. The Export button in the top-left corner exports the current tab and filter’s data to a table. The export has Status Code and Indexability columns, which make it easy to keep only the pages with a 200 code that the program considers indexable.

For bulk exports, there’s the Bulk Export menu. The program can also create an XML Sitemap via the Sitemaps menu. For more on how to structure and submit a sitemap, see a separate article.

Pages with no links to them

In Spider mode, the program finds only the pages that links lead to. An orphan page is a page that can’t be found via internal links from the starting page. To find such pages, the program needs additional sources of addresses: an XML Sitemap, and data from Google Analytics and Search Console.

The sequence is:

1. Turn on Crawl Linked XML Sitemaps in Configuration → Spider → Crawl.
2. Connect Google Analytics and Search Console in Configuration → API Access.
3. After the crawl, run Crawl Analysis → Start.
4. Check the Orphan URLs filter on the Sitemaps, Analytics and Search Console tabs, or export the Reports → Orphan Pages report.

This check requires a license: without one, a full site crawl and integration setup aren’t available. Another way is to load the addresses from the Sitemap in List mode and compare them with the addresses found while crawling links.

A URL list from a crawler is the addresses that were reachable via links at the time of the crawl. It isn’t the pages in Google’s or Yandex’s index. To find out which pages made it into the index and how to check that, you need data from the search engines themselves.

What a crawler won’t show you

• The index and the search engine’s own decisions. Whether a page made it into the index and which canonical version the search engine chose is visible in Search Console’s URL Inspection tool and in Webmaster’s «Page check.» A crawler doesn’t contain this data.
• Content and links generated by JavaScript, if rendering is turned off. In Text Only mode, the program reads only the raw HTML, and rendering isn’t available in the free version. Googlebot, meanwhile, does render pages, so results can diverge. That said, Google generally only processes a link if it’s an <a> element with an href attribute.
• Addresses with no inbound links, if you haven’t added other URL sources (see the previous section).
• The site’s state at a different point in time. A crawler only records server responses during its own run. A search engine’s robot may have gotten a different response, for example under different server load or with a different user-agent.
• Priorities for your site. The program shows a list of deviations and its own rating of their importance, but it doesn’t know which sections matter for the business. That decision is up to the specialist.

What to do with crawl results

1. Group errors by template and section. One error in a template repeats across many pages. Fixing the template is usually faster than fixing each URL.
2. Check your interpretation against the search engines’ rules, not against the program’s highlighting. Use the summary table above as your reference.
3. Re-crawl the site after fixes. Mode → Compare compares two crawls and shows what changed across tabs and filters, for example in title, description and depth. Comparing requires a license and database storage mode. Without a license, compare exports manually.

A crawl is only one part of checking a site. For more on what a technical site audit consists of, see a separate article.

Post-crawl checklist:

• no internal links to addresses with 4xx or 5xx codes;
• internal links don’t go through redirect chains;
• standalone pages’ canonical points to themselves;
• noindex is only on pages that genuinely shouldn’t be in search;
• title and description aren’t empty and don’t repeat by template;
• important pages are reachable by links and aren’t buried deep in the structure;
• the re-crawl’s result has been compared with the original.

If the errors are template-driven, recur after every release, or there are too many for manual fixes, you need systematic work on your site’s technical errors.

Frequently asked questions

What is a crawler, and how is it different from a search engine robot?

A search engine robot is the search engine’s own program. Yandex describes its robot as a system that crawls site pages and loads them into its own database; Google calls robots automated programs that download content from sites. An SEO crawler crawls a site in a similar way, but a site owner or specialist runs it to see their own site’s technical data. This data never enters the search engine’s index.

Can Screaming Frog be used online?

No. It’s an application for Windows, macOS and Linux that you install on your computer.

Is the free version enough?

It depends on the site’s size and the task. The free version crawls up to 500 URLs per crawl with default settings. Saving crawls, JavaScript rendering and integrations aren’t available in it. See the table in the «Free version vs. license» section for details.

Why did the program find fewer pages than the site actually has?

Check, in order:

• whether the crawl hit the 500 URL limit;
• whether sections are blocked in robots.txt;
• whether JavaScript is generating links while rendering is off;
• whether links carry nofollow, which the program doesn’t follow by default;
• whether there are pages with no inbound links;
• whether the crawl is restricted by Include / Exclude rules.

For more, see the sections on settings and on what a crawler won’t show you.

How often should a site be crawled?

There’s no single rule. Crawl the site after releases, template changes, migrations and bulk changes, and also on a regular basis — at a frequency that matches how often the site changes. Scheduled crawls (File → Scheduling) are available with a license.