You optimized your content in Lesson 4. Your articles have proper headings, keywords, and structure. But here's the problem: even perfectly optimized content won't rank if Google can't access it. In this lesson, you'll learn which technical issues hurt rankings — and which ones you can safely ignore.

Technical SEO is the backend work that makes your site accessible to search engines. Unlike on-page SEO (your content and keywords), technical SEO focuses on how your site functions:

When any of these break down, your content doesn't rank — no matter how good it is. And since not all technical issues are equal, it helps to know which ones to fix first.

Issues fall into three priority levels:

For example, your home decor site might have 15 technical issues. But if three are high priority (broken pages, slow speed, not mobile-friendly) and 12 are low (minor schema tweaks), fix the three high ones first. You'll see bigger ranking improvements faster. This is called impact-based prioritization — fixing what moves the needle.

Remember: you can't fix what you can't see. Technical issues hide in your site’s code and server responses. Tools like Frase and SEOWriting.ai are great for content, but they don't crawl your site or spot broken links. You need a tool that mimics Google's crawling process to find these hidden issues — enter Screaming Frog.

Screaming Frog is a website crawler that scans your site the way Google does. It visits every page, follows every link, and reports technical issues: broken links, slow pages, missing meta descriptions, duplicate content, and more.

If your homepage links to a product page but it returns a 404 error, Screaming Frog catches it. If 10 pages have identical meta descriptions, it flags them as identical. This is how you find critical issues before they hurt rankings.

Download Screaming Frog from their website — the free version crawls up to 500 URLs, which works for most small to medium sites. Install it on your computer (Windows, Mac, or Linux), open the app, and enter your website URL to start.

Let's pick a well-known company for today’s analysis — IKEA. For your real audit, use your own website. Enter the website URL in the search bar and click "Start." Screaming Frog will now crawl your entire site, following every internal link. This may take up to 10 minutes, depending on your site's size. You'll see a progress bar showing the number of pages crawled.

Once the crawl completes, you're looking at your complete site data in Screaming Frog. The top navigation tabs (Internal, External, Response Codes, Page Titles, Meta Description, H1/H2) show specific elements you can examine. The right sidebar Summary displays total URLs crawled and critical stats like blocked pages or indexing issues.

Click on "Issues" in the right panel. It highlights critical problems organized by Issue Name, Issue Type (Warning, Issue, Opportunity), and Issue Priority (High, Medium, Low). Start with the issues that show high priority — these are your critical blockers.

Screaming Frog also lets you see what caused the issue! Click on a specific one, and in the "Issue Details" section, you'll see a description of the problem and "How To Fix" instructions.

For example, we found a "Response Codes: Internal Blocked by Robots" issue with High priority. From Issue Details, you know that important pages are blocked from Google's crawlers — a critical problem that stops those pages from ranking entirely. Now, let's understand why this happens and how to fix it by sharing issue details with ChatGPT.

ChatGPT explained the "why," and Screaming Frog shows you the "what": which pages are affected and need fixing. Now, let's create a prevention plan to ensure the problem will not occur again.

Prevention plan is ready, let's start fixing! In Screaming Frog, view URLs that link to blocked pages using the "Inlinks" tab. Then export them in bulk via Bulk Export > Response Codes > Internal > Blocked by Robots.txt Inlinks. This gives you a spreadsheet showing every internal link pointing to blocked pages — so you know what to fix and how.

For each blocked URL, check whether it should actually be blocked. A page is blocked by mistake if it's important content — like a product page or blog post — that you want Google to find. A page should stay blocked if it's private, like an admin panel or checkout page. If it's blocked by mistake, update your robots.txt to allow access. If it's meant to stay private, remove any internal links pointing to it so the warning stops appearing.

Understanding the root cause means you won't just fix the problem today — you'll stop it from breaking again tomorrow. For a final step, let's verify your fix worked with a follow-up crawl.

After fixing the robots.txt blocking issue, run Screaming Frog again. Enter your URL, then click "Start" to run a fresh crawl.

Compare the new crawl to your first one — the "Internal Blocked by Robots" issue should be gone from the High Priority list. If new issues appear, apply the same priority system and use your prevention checklist to understand why they showed up.

You've come a long way from Lesson 1! You started by understanding SEO basics, found valuable keywords, planned strategic content, optimized it, and now removed the technical barriers that could block your rankings. See you in the next lesson!