Select Page

Understanding crawl errors with free webmaster tools to improve web traffic

Google Search ConsoleIf you’ve taken any SEO classes, you know that the Google Search engine is the most important search tool for users to find your site.  After all, Google searches control 75% of worldwide search requests according to Netmarketshare.com.  I tell my students that “Google controls the search game, so you need to learn how to play it.”

To learn how you are doing with your ranking in search results, one of the most under-utilized and useful tools that Google provides you is called Google Search Console, a part of your Webmaster tools.  Through this site, Google provides useful data for you to view basic search information, and detect search-related issues on your website.

In this post, let’s take a look into the mind of the Google search engine and try and use Google Search Console to examine errors.

What is a crawl error?

Simple stated according to Google.com, a crawl error “can prevent your page from appearing in search results.”  Google uses its own “bots” (called Googlebots) to crawl your website and to determine what the content and relevancy of your site.  As the Googlebots do their job, they rely on certain information within the article itself, as well as several pieces of code (header tags, titles, meta tags) to rank and index your site.

While indexing, the Googlebot is deciding which pages are going to be appropriate for a user when they come to search for some information relevant to what you offer.  If you have optimized your site using solid SEO practices, then your search engine result should be organically high.

However, if you have any errors on your site, Google flags that page and/or your site with caution. The more fatal the error, the worse your ranking may be effected.

Google Crawl ErrorsTo combat errors, there is a free report tool within Google Search Console called “crawl error report.”  This report “provides details about the site URLs that Google could not successfully crawl or that returned an HTTP error code” according to Google help.

The crawl error report will tell you the URL that is causing the issue, as well as an error code that may be useful.

To check your site for errors, login to your Google Search Console, go to your primary domain name, click on “Crawl”, and click on “Crawl Errors.”

The errors you will find on this report will be broken down into three categories: server errors, soft 404s, and not found.  There are also different sections for desktop versus mobile crawl errors.  Here are some useful definitions (from Google help) of what these errors mean:

Server errors: Shows any instances when Googlebot’s request timed out or the site is blocking Google.

Soft 404s: The target URL doesn’t exist, but your server is not returning a 404 (file not found) error.

Not found: URL points to a non-existent page.  This is commonly referred to as a 404 error and displays a specific 404 error page to users.

Smartphone errors: Errors that occurred only when your site was crawled by Googlebot (errors didn’t appear for desktop).

I’ve got crawl errors! What do I do now?

First, don’t panic.  With some basic steps you can fix the errors.  The basic idea is that you need to 1) correct the error on your server and 2) tell Googlebots to go back and re-index the pages in question.

However, you should be aware that the timeline to see the results of your work here can vary greatly.    You are relying on Googlebots to determine the changes are sufficient and the errors are gone.  Despite my pleas with Google support, there is no timeline they can provide to help speed this up.

Once you have fixed the error on the server, to fix a crawl error with Google you need to use the “fetch” command and see into the brain of the Googlebot.  This command will tell you if Google is seeing the page correctly or if there are remaining issues.

Crawl errors - fetch as googleOn the crawl error report, click on a URL you have fixed.  It will tell you the original error, and provide a “fetch as google” option.  Click this and you will jump to the fetch as google page.

Here, you can verify the URL in question, and then choose “Fetch” to see how Google will view this page.  If you get a green check mark, you’re good to continue.  At this point, you can click Fetch and Render, which should provide an option for “Submit to Index.”

There are 2 options on the next screen: 1) to index only this page or 2) to index this page and its direct links.  There are limits to both so choose wisely.

You have now told Google to re-index this page and the error is fixed.  After approximately 48 hours, you should check back every day or so, to make sure the error has cleared.

Why is the crawl error report important?

Why spend the time building a website if new, potential customers can’t find you through search?

Think about this every time you re-visit the crawl error report.  What will only take you a few seconds to check could prevent a huge headache with the Google search engine.  This will only help your search engine rankings and help customers find you easier.

If you would like to learn how to control your website’s SEO on your own, take a look at Digital Workshop Center classes and digital marketing certificate programs, which are available in Fort Collins, Denver, and online.

We can also customize a SEO training plan for you and your business. Fill out the following form and we will get in touch with you as soon as possible.