You've heard of Screaming Frog, but you're not quite sure how to use it? Screaming Frog is a powerful crawling tool designed to navigate and analyze web pages by following the links connecting them.
A must-have for any SEO agency running technical audits, this tool is essential for optimizing your SEO (Search Engine Optimization). It lets you detect technical issues, identify areas for improvement, and apply best practices for effective search rankings.
In this article, we'll walk you step by step through how to use Screaming Frog SEO Spider, the most advanced and fully featured version of this crawler. We'll cover installation, configuration, how to deploy it on your site or a competitor's, and how to make the most of the data it collects. You'll also discover the advanced features of Screaming Frog that will help you run a thorough and effective SEO audit of your website.
Installing and configuring Screaming Frog SEO Spider
Getting started with Screaming Frog SEO Spider is straightforward: download and install the program on your computer. The software is available for Windows, macOS, and Linux.
Installation is done through an executable file. Launch it and follow the on-screen instructions. A key step in the process is accepting the ttf-mscorefonts-install license, which is essential for the crawler to function properly.
Download and installation
After installation, you'll need to activate the software. If you don't have a paid license, the free version is available and lets you analyze up to 500 URLs per site. Simply click "Start" in the main window. If you have a paid license, you can activate it via "Help" > "License", which unlocks the ability to crawl an unlimited number of URLs.
License activation and initial settings
Before running your first crawl, a few initial settings are needed. Go to "Configuration" > "Spider" to adjust parameters to your needs, including the types of files to analyze (such as images, CSS, JavaScript), how internal and external links are handled, whether to respect the site's robots.txt file, and integration with the Google Analytics and Search Console APIs, among other options.
You can also create custom configuration profiles tailored to your specific projects.
Memory and storage settings
To adjust the crawler's performance based on the size of the sites you're analyzing, go to "Configuration" > "System". Here you can choose between RAM storage for greater speed or database storage for analyzing large sites. This menu also lets you customize cache size, thread count, and request delay to optimize how you use the software.
{{audit-primary="/cta"}}
Using Screaming Frog to optimize your SEO
Once Screaming Frog is installed and configured, you'll see how it becomes an indispensable tool for SEO optimization. It helps you perform a quality technical audit, identify and fix broken links, optimize title tags and meta descriptions, analyze image ALT attributes, and spot duplicate content.
Here are the steps to follow.
Running a comprehensive SEO audit of your site
To carry out a full technical SEO audit, simply enter your site's URL in Screaming Frog and click "Start". The crawler scans all pages and provides detailed information accessible through the "Internal", "External", "Response Codes", "Page Titles", "Meta Description", "H1", "H2" tabs, and more. This lets you check whether your site follows SEO best practices such as load speed, internal structure, and HTML markup. Data can be exported in CSV or XLS format for deeper analysis.
Identifying and fixing broken links
Broken links, which redirect to non-existent or moved pages, degrade both the user experience and search rankings. The "Response Codes" tab in Screaming Frog lets you spot these links using HTTP response codes such as 200 (OK), 301 (permanent redirect), 404 (page not found), and others. You can then filter and fix these links, improving the overall accessibility of your site.
Optimizing title tags and meta descriptions
Title tags and meta descriptions are critical because they define the content and intent of your pages in search results. Screaming Frog lets you review them under the dedicated "Page Titles" and "Meta Description" tabs, checking their length and content. It's an opportunity to optimize them in line with SEO best practices and improve your visibility.
Analyzing image ALT attributes and identifying duplicate content
ALT attributes describe the images on your site, improving accessibility and helping search engines and visually impaired users understand visual content. The "Images" tab lets you check whether your images are SEO-optimized in terms of size, filename, and alt text. Additionally, duplicate content hurts your rankings and credibility. The "Content" tab helps you identify similar or identical content, which you can then update or remove using methods such as rewriting or canonicalization.
Leveraging Screaming Frog's advanced features
Screaming Frog is a comprehensive tool, essential for SEO optimization. Let's look at some of its advanced features. These will help you customize your analysis, refine your searches, integrate external data, and generate detailed reports.
Setting custom crawl rules with the Include/Exclude tool
With the Include/Exclude tool, you can create crawl rules tailored to your specific needs. Choose which pages or file types to include or exclude from the analysis. For example, you could exclude pages containing a specific keyword, or include only pages with a particular extension. To do this, go to "Configuration" then "Include/Exclude" and enter the criteria you want.
{{expert-primary="/cta"}}
Using regular expressions for more precise searches
Regular expressions let you define complex search patterns. They significantly sharpen searches across your site. For example, you could look for pages with titles longer than 60 characters or meta descriptions that include a specific keyword. Enable the "Regex" option in Screaming Frog and enter your expression to get started.
Integrating Screaming Frog with Google Analytics and Google Search Console
By connecting to the Google Analytics and Google Search Console APIs, Screaming Frog enriches your SEO audits with real performance data. Access valuable information such as visits, bounce rate, and keyword rankings. To integrate these tools, click "Configuration" then "API Access" and select the tool you want after signing in to your account.
Exporting and analyzing data for a detailed SEO report
Once the analysis is done, export and analyze the data to build a detailed SEO report. Screaming Frog supports export in multiple formats (CSV, XLS, PDF, etc.). Select the data relevant to your goals, such as titles, meta descriptions, images, or links. Use predefined report templates or create custom ones to effectively present the results of your audit.
{{rdv-primary="/cta"}}
Conclusion
Screaming Frog stands out as an essential tool for boosting your organic search rankings (SEO). Its ability to crawl and analyze your website, or your competitors', helps you identify and fix technical issues, optimize HTML elements, integrate performance indicators, and produce accurate reports. The tool offers unique flexibility, letting you customize your crawl, use regular expressions, configure APIs, and export data according to your requirements.
If you haven't explored Screaming Frog yet, we strongly encourage you to download it and give it a try. The difference it makes to your SEO strategy will be clear.
Want guidance on your digital marketing strategy? Don't hesitate to reach out!
FAQ
What is Screaming Frog?
Screaming Frog is an SEO crawling tool used to analyze and audit websites. It helps identify technical errors and content issues, and optimize site structure to improve search engine rankings.
How do you use Screaming Frog for SEO?
Download and install Screaming Frog, then enter your site's URL and launch the crawl. The resulting report highlights issues such as missing meta tags, broken links, and 404 errors. Fixing these issues improves your site's indexation and visibility.
Is Screaming Frog suitable for beginners?
Yes, Screaming Frog is beginner-friendly, with guides and tutorials available online. The free version offers basic features to get familiar with the tool before upgrading to the more fully featured paid version.
.jpg)
.jpg)