Screaming Frog SEO Spider is a crawling tool that traverses a website link by link, much like a search engine bot, and identifies technical issues that hinder its organic search performance. It is free for up to 500 URLs, costs €245 per year for the paid version, and is used in four steps: installation, configuration, crawling, and tab analysis.
The tool has evolved significantly in recent versions. Four major releases, from 22 to 25, have introduced semantic analysis, an MCP server that connects the crawler to AI assistants, automatic crawl comparison, and custom dashboards.
This guide details how to get started with SEO Spider, its pricing, the method for conducting a technical SEO audit, and the latest features you should know. It is designed for both beginners and teams at an SEO agency agency that crawl sites every day.
What is Screaming Frog SEO Spider?
Screaming Frog SEO Spider is a desktop crawler: you download and install it on your computer, and it explores a site's URLs by following its internal links. For every page, it records the HTTP response code, tags, robot directives, content, and links.
The software is primarily used for three purposes:
- conducting atechnical site audit : errors, redirects, non-indexable pages, and duplicate content;
- preparing and testing a site redesign or migration, including a redirect plan;
- monitoring a site over time, thanks to scheduled crawls and their comparison.
It does not provide keyword or backlink data without third-party integration: it complements the suites in our selection of the best SEO tools, rather than replacing them.
Is Screaming Frog free? Pricing for the free version and the license
Screaming Frog is free for up to 500 URLs per crawl. Beyond that, the paid version costs €245 per license per year, according to the publisher's pricing page. The license is valid for one year and then renews.
The price decreases with volume: €235 per license for 5 to 9 licenses, €219 for 10 to 19, and €209 for 20 or more. The free version is sufficient for exploring the tool or checking a small showcase site. As soon as a site exceeds 500 URLs, or if you need to configure the crawl and save your results, the license becomes necessary.
How do you install and configure Screaming Frog?
Installation takes a few minutes: you download the SEO Spider from the publisher's website, run the installer, and then adjust the memory and crawl configuration before your first exploration.
Downloading and installation
The SEO Spider is available for Windows, macOS, and Linux (Ubuntu and Fedora). The developer requires at least a 64-bit system and 4 GB of available RAM.
Without a license, the software starts in free mode. After purchasing, you enter your username and license key into the software to unlock the paid version.
Memory and storage mode
The storage mode can be set under "File > Settings > Storage Mode". Database storage, which is enabled by default, writes the crawl to the disk: it saves automatically and is suitable for large sites. Memory storage keeps everything in RAM and is reserved for small crawls.
Allocated memory can be set under "File > Settings > Memory Allocation". The developer recommends allocating 4 GB for crawls of up to 2 million URLs, and a machine equipped with an SSD and 16 GB of RAM for anything over one million URLs.
Crawl configuration
The "Configuration > Spider" menu defines what the crawler explores and records. The settings to check before each crawl are as follows:
- resources to explore: images, CSS, JavaScript, external links;
- adherence to the robots.txt file and noindex, canonical, and nofollow directives;
- JavaScript rendering ("Configuration > Spider > Rendering"), which is essential for sites where content is injected via script;
- limits: depth, number of URLs, and, since version 23, a URL cap per subdomain;
- Include and Exclude rules, which restrict the crawl to a specific directory or exclude URLs with parameters.
Once set, the configuration can be saved as a profile to be reused for your future projects.
{{audit-primary="/cta"}}
How to perform a technical SEO audit with Screaming Frog?
To audit a site, enter its URL in the top bar, click "Start," and then read the results tab by tab. Each tab isolates a category of checks, and its filters lead directly to the URLs with errors.
- Launch the crawl in Spider mode on the homepage.
- Open the "Issues" tab, which categorizes detected problems by type and priority.
- Review the "Response Codes," "Page Titles," "Meta Description," "H1," "Content," "Images," and "Links" tabs.
- Export the relevant URLs, fix the issues, and then re-crawl to verify.
This reading is part of a broader approach, detailed in our guide on conducting a successful SEO audit.
Response codes, broken links, and redirects
The "Response Codes" tab categorizes each URL by its HTTP code: 200, 3xx, 4xx, or 5xx. The client error filter lists broken links, and the "Inlinks" tab at the bottom shows the pages that point to each 404 error.
Redirects are managed in the same place. The "Reports > Redirects" report isolates chains and loops, and version 23 added "Redirects to Error," which identifies redirects that lead to an error. Every 301 redirect must point directly to a page with a 200 status code.
Title tags, meta descriptions, and H1s
The "Page Titles," "Meta Description," and "H1" tabs flag missing, duplicate, too long, or too short tags. Length is measured in both characters and pixels, allowing you to anticipate truncation in search results.
Duplicate content and image ALT attributes
The "Content" tab detects strictly identical pages and near-duplicate content, based on a similarity threshold that can be adjusted in "Configuration > Content > Duplicates." Since version 22, it also flags semantically similar pages, even when the text is paraphrased.
The "Images" tab lists images that are too large and those missing ALT attributes. Version 25 adds a check for missing title attributes.
Internal links, depth, and orphan pages
The "Links" tab and "Crawl Depth" column show how internal links distribute authority: pages that are too deep, pages with few links, and empty anchors. Version 24 adds a filter for non-crawlable links, such as those contained within a span tag or an onclick attribute.
To find an orphan page, a crawl alone is not enough, as no links lead to it. You must connect your XML sitemap, Google Analytics, or Search Console, then run the crawl analysis.
Export and reporting
Each tab can be exported to CSV, Excel, or Google Sheets, and the "Bulk Export" menu groups mass exports together. Since version 22, "Bulk Export > Multi Export" saves a selection of exports and reports, delivered in a single click as a multi-tab workbook.
What are the latest updates in Screaming Frog?
The last four major versions of Screaming Frog have focused on three areas: artificial intelligence, crawl automation, and tracking changes between crawls.
Version 25: dashboards, configurable issues, and the Social tab
Version 25 makes the SEO Spider easier to adapt to any project. Its main new features include:
- a "Dashboard" tab for building custom dashboards with charts, counters, and tables, which can then be shared or exported;
- configurable issues: each check in the "Issues" tab can be disabled, or have its type and priority changed;
- a "Social" tab that validates Open Graph and X Card tags;
- automatic crawl throttling when the server returns 429 or 503 status codes;
- link exclusion based on their location on the page, such as faceted navigation;
- configuration files in JSON format, which are readable and editable in a text editor.
Version 24: the MCP server and automatic comparison
Version 24 allows you to control the SEO Spider via an AI assistant. Its MCP server exposes crawl data to Claude, LM Studio, and other assistants: you can request to run a crawl, export a report, or cross-reference two datasets using natural language.
The MCP server is enabled under "File > Settings > MCP Server." It requires a paid license and database storage mode, and the developer notes that not all software functions are supported yet.
The same version automatically compares the last two scheduled crawls of a project. The end-of-crawl email then summarizes what has changed (pages that appeared or disappeared, an increase in non-indexable pages, deleted titles) and can attach the relevant exports for each recipient.
Versions 22 and 23: semantic analysis via embeddings
Version 22 introduced semantic similarity analysis. The SEO Spider converts the content of each page into embeddings using a connected AI provider (OpenAI, Gemini, Anthropic, or Ollama), then compares the pages based on their meaning rather than their words.
The tool identifies similar pages at risk of cannibalization, isolates off-topic content, and displays content clusters in a diagram ("Visualisations > Content Cluster Diagram").
Version 23 completed the set. It allows for limiting semantic comparison to specific URL patterns, which is useful for comparing a staging site to the live site to build a redirection plan.
{{rdv-primary="/cta"}}
Which advanced Screaming Frog features should you use?
Four advanced features save the most time during an audit: list mode, custom extraction, Google integrations, and checking what AI search engine bots are reading.
List mode and migration recipes
List mode crawls a provided list of URLs instead of following a site's links. It is used to validate a redirection plan: you import the old URLs, enable redirect tracking, and then verify that each one leads to a 200 status page.
Custom extraction and regular expressions
The "Configuration > Custom" menu adds custom columns to the crawl report. Custom search identifies pages that contain—or do not contain—text or a pattern defined by a regular expression. Custom extraction retrieves any element from the HTML code, such as a price, a breadcrumb trail, or a structured data block.
Google Analytics, Search Console, and PageSpeed Insights
The "Configuration > API Access" menu connects the crawler to Google Analytics, Search Console, and PageSpeed Insights. Each crawled URL then receives its session, click, impression, and performance data.
This cross-referencing reveals crawled pages that generate no traffic. To find out what Googlebot is actually crawling, it is complemented by a log analysis.
Checking what AI search engine bots are reading
A crawl without JavaScript rendering shows the content available to bots that do not execute scripts, which is the case for some AI assistant bots. If the main text of a page disappears in this crawl, it is also invisible to them.
The "Configuration > User-Agent" menu changes the crawler's identity to test the server's response to a specific bot. This check is paired with a review of the robots.txt file, which may be blocking these bots unintentionally.
How datashake leverages crawling in its SEO and GEO audits
At datashake, crawling is the starting point of a technical audit, not the end goal. A tool like Screaming Frog collects thousands of data points, but not all of them impact visibility: the real work lies in sorting, prioritizing by impact, and then tracking the fixes.
The datashake audit covers over 100 checkpoints: technical SEO, semantics, authority, and visibility in AI search engines. The team SEO and GEO team at datashake pairs this with a proprietary monitoring tool that tracks a brand's presence in responses from ChatGPT, Perplexity, and AI Overviews.
FAQ
What is Screaming Frog?
Screaming Frog SEO Spider is SEO crawling software developed by the British agency Screaming Frog. It explores a website's pages just like a search engine and flags technical errors: broken links, redirects, missing tags, duplicate content, and non-indexable pages.
How much does Screaming Frog cost?
A Screaming Frog license costs €245 per year, per user. The price drops to €235 for 5 or more licenses, and to €209 for 20 or more. The free version remains available indefinitely for up to 500 URLs per crawl.
How do you use Screaming Frog as a beginner?
Install the software, enter your site's URL, and click "Start." Then, open the "Issues" tab, which categorizes problems by priority. Start by addressing 4xx errors, redirect chains, and missing or duplicate title tags.
What is the Screaming Frog MCP server?
The MCP server, introduced in version 24, connects the SEO Spider to an AI assistant like Claude or LM Studio. The assistant can then launch a crawl, read the results, and generate exports based on a simple prompt. This feature is exclusive to the paid version.


