Automating Meta Description Generation Using Screaming Frog and OpenAI: A Practical Workflow
Writing meta descriptions for numerous web pages can be an arduous task, often consuming hours of manual effort. Recently, a colleague shared that they spent over 11 hours crafting meta descriptions for 139 pages—an experience many SEO professionals can relate to. In response, I developed an automated workflow leveraging Screaming Frog and OpenAI’s API, dramatically reducing the time and effort involved.
Below, I outline a step-by-step approach to set up this automation, providing a framework that balances efficiency with quality.
Essential Tools and Requirements
- Screaming Frog SEO Spider (with a valid license)
- OpenAI API Key (requires a free or paid account on the OpenAI platform)
- Basic JavaScript familiarity (minimal coding skills needed)
Step 1: Enable JavaScript Rendering in Screaming Frog
To ensure that dynamic and JavaScript-rendered content is accessible:
- Navigate to Configuration > Spider > Rendering
- Select “JavaScript” from the dropdown menu
- Confirm with OK
This setting allows Screaming Frog to process content more similarly to browsers, ensuring accurate data extraction.
Step 2: Manage API Rate Limiting
To prevent exceeding OpenAI’s API limits:
- Go to Configuration > Speed
- Enable “Limit URL/s”
- Set Maximum URLs per second to 0.5 or lower
- Additionally, in Configuration > Spider > Crawl, uncheck resource types such as CSS files—meta descriptions don’t require these assets
Proper rate limiting ensures smooth processing without API throttling.
Step 3: Incorporate Custom JavaScript for API Calls
Screaming Frog allows custom scripting to automate API interactions:
- Navigate to Configuration > Custom > Custom JavaScript
- Click “+Add from Library”
- Select the template named “(ChatGPT) Template” (or create your own)
This script will handle sending prompts to OpenAI and receiving responses programmatically.
Step 4: Obtain Your OpenAI API Key
Secure your API key:
- Create an account on the OpenAI platform (distinct from ChatGPT login)
- Access Settings > API Keys
- Generate a new secret key
- Copy the key and input it into Screaming Frog during setup—note that API keys are only accessible at creation time, so store it securely
Step 5: Craft Your OpenAI Prompt for Meta Descriptions
The core of this automation is a meticulously designed prompt. Here is an example tailored for generating SEO-friendly meta descriptions:
javascript
const question = "Identify the primary keyword from the following web page content and generate an SEO optimized meta description using that keyword in the style of a SEO expert. IMPORTANT: Do NOT start the description with the words explore, embark, experience, join, dive into, or discover.\n"+
"# Steps\n"+
"1. Analyze Content: Examine the provided content to determine the primary theme or subject matter.\n"+
"2. Identify Keyword: Select the primary keyword that best represents the main subject of the web page.\n"+
"3. Craft Meta Description: Write a meta description between 120-155 characters that incorporates the keyword, is engaging, and succinctly summarizes the page content using the language in the text.\n"+
"# Output Format\n"+
"**Primary keyword:** The keyword that best represents the main subject of the web page\n"+
"**Meta Description:** Meta description between 120-155 characters\n"+
"# Examples\n"+
"**Example 1:**\n"+
"- Primary Keyword: Keyword Research Tool\n"+
"- Meta Description: The most complete keyword research tool on the market. Generate thousands of keyword ideas, group them into topical clusters, and nail search intent.\n"+
"**Example 2:**\n"+
"- Primary Keyword: Authority checker\n"+
"- Meta Description: Free tool to check the authority of any website based on the quality and quantity of its external backlinks.\n"+
"**Example 3:**\n"+
"- Primary Keyword: Screaming Frog alternatives\n"+
"- Meta Description: Looking for a Screaming Frog alternative? MetaMonster could be the tool for you.\n"+
"**Example 4:**\n"+
"- Primary Keyword: SEO spider software\n"+
"- Meta Description: Clean up your metadata in minutes instead of months with MetaMonster, the SEO spider that fixes issues for you.\n"+
"**Example 5:**\n"+
"- Primary Keyword: Page title generator\n"+
"- Meta Description: Generate page titles for your website with AI. MetaMonster is the SEO spider software that fixes issues for you.\n"+
"**Example 6:**\n"+
"- Primary Keyword: WordPress SEO plugin\n"+
"- Meta Description: Easily publish optimizations to your site with the MetaMonster WordPress SEO Plugin.\n"
This prompt guides the AI to extract keywords and generate descriptions within specified constraints. You can further customize it to align with your brand voice or SEO strategy.
Bonus Tips for Enhanced Results
- Set the temperature parameter to 1.0 for more diverse outputs
- After receiving responses, use custom scripts to extract just the meta description text—e.g., employing regex patterns to isolate the relevant content
Sample extraction function:
javascript
function extractMetaDescription(content) {
const metaDescriptionRegex = /\*\*Meta Description:\*\*\s*(.+)/;
const match = content.match(metaDescriptionRegex);
return match ? match[1].trim() : null;
}
Limitations and Considerations
- Context Limitations: Each page is processed independently; the AI has limited awareness of other pages’ content.
- Complex Prompt Engineering: Advanced chaining or context-aware prompts are constrained by Screaming Frog’s scripting environment.
- Data Security: Ensure API keys are kept secure; avoid sharing scripts with sensitive info in public repositories.
Real-World Results
Applying this workflow to a client’s website previously involved manual effort spanning approximately 11 hours. Implementing the automated system took about 30 minutes to set up, with subsequent pages generated swiftly—saving considerable time and avoiding the tedium of manual editing.
While the generated descriptions might require some fine-tuning, they serve as high-quality starting points, enabling rapid deployment and optimization.
Final Thoughts
This approach exemplifies how combining SEO tools with AI APIs can streamline repetitive tasks. Although the process involves some initial configuration, it provides a flexible, scalable solution to generate meta descriptions efficiently.
Note: For those seeking an even more seamless experience, I am developing a tool called MetaMonster that automates this process without coding and delivers superior output quality. However, the Screaming Frog + OpenAI setup remains a cost-effective and hands-on alternative worth exploring.
Disclaimer: This guide reflects my personal setup and experiences. Feel free to adapt and enhance it based on your specific needs and technical comfort level. Happy optimizing!
