Search Engine Marketing

Understanding Crawl Errors in Google Search Console

JannelleChemko
ByJannelleChemko

Some people may think I’m crazy, but fixing crawl errors in Google Search Console is one of my favorite tasks. There’s something so satisfying about going through those errors and fixing them, one by one.

Website crawl errors can prevent your pages from appearing in search results. The Crawl Errors report in the Search Console provides details about these errors, by showing a list of all site URLs that Google is unable to crawl, or that are returning HTTP error codes.

The Crawl Errors report has two sections: Site Errors and URL Errors. Here’s an overview of each of them, and how to fix the errors that you might come across.

Site_and_URL_Errors.png

Site Errors

This section of the report shows data from the last 90 days. In most cases, you’ll find a green check mark “status” next to each error type, meaning that your site is functioning well. If your status is not a green check, you can click on the box to expand it and view the historical crawling data from the last three months. This would only happen if your website is down, overloaded, or configured improperly. If this is the case, Google will send you a message to give you the heads up.

Site Errors are split into three different types:

DNS Errors

A DNS error means that a bot crawling your site cannot communicate with the DNS server — either because the server is down, or perhaps there is an issue with the routing to your domain. Whatever the case, a DNS error might indicate speed issues, which won’t affect the bot’s ability to access your site, but will affect your users and their ability to use your site.

Server Errors

A server error means that the bot cannot access your URL for some reason. Perhaps the request timed out or maybe the site was busy; either way, as a result, the bot had to quit the original request and cannot access your site.

Robot Failure

A Robot Failure is an error in retrieving your site’s robots.txt file. Approximately once a day, before a bot crawls your site, it retrieves your robots.txt file in order to see which pages it should not crawl. If your robots.txt file is unavailable, Google will stop the crawl so that it doesn’t accidentally crawl URLs that should not be crawled.

URL Errors

URL Errors are ranked by Google and sorted to show the most important and urgent issues at the top of the report, to a maximum of 1,000. They are also split between Desktop, Smartphone and Feature Phone errors and you can mark them fixed as you go through each one (which is what I find satisfying!). However, if you do mark a URL as “fixed” and it actually remains unfixed, the URL will reappear the next time Google crawls your site until you actually sort out the issue.

URL_Errors.png

Common URL errors include:

Server Error: just like under the DNS errors, a server error for your URLs means the same: that a Googlebot cannot access your URL, the request was timed out, or your site was busy, and the bot was forced to abandon the request.

404: this error is one that most people see on an almost-daily basis. A 404 error occurs when a user tries to access a webpage that doesn’t (or no longer) exists. The causes of this error could be from broken links, incorrect URLs, or a redirected page that is no longer valid.

Access Denied: these errors occur when a Googlebot cannot access your site. Perhaps a URL requires a login in order to view it, or your server requires proxy authentication, or your hosting provider is blocking Google from accessing your site.

Not Followed: these errors list URLs that Google cannot completely follow. Perhaps your redirects are not redirecting to the right page, or because some features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash can make it difficult for search engines to crawl your site.

URLs Blocked for Smartphone: this “Blocked” error appears on the Smartphone tab. If you get the “Blocked” error for a URL on your site, that means that the URL is blocked for Google’s smartphone Googlebot in your site’s robots.txt file.

Flash Content: most mobile devices cannot process Flash content, so those Smartphone URLs will return this error. You may need to upgrade to a more responsive web design to avoid this error.

Summary

Some of these errors are definitely more technical than others; it’s our recommendation that you make use of Google’s support documents to help you learn which are important to act on, and which might be safe to ignore (at least for now).

In the meantime, you should also check out some of our blog posts on the topic of Google Search Console (formerly Google Webmaster Tools) and website errors:

Who knows…maybe you’ll enjoy fixing these errors as much as I do 🙂 And while you’re making these updates to your site, you might want to make other improvements as well. For a list of 25 website “must-haves” download a copy of our free eBook:

About the Author

JannelleChemko

JannelleChemko

Numbers Ninja & Digital Dynamo
Jannelle Chemko has been working in Operations and Accounting since 2007. After earning a Bachelor’s Degree in English, she is now in the midst of her CGA designation.

As strange as it sounds, Jannelle is a numbers and a letters guru: in addition to extensive full-cycle accounting experience in the technology and retail industries, Jannelle is also passionate about writing. In between crunching numbers and building excel reports, she researches, creates content, and keeps up to date with digital trends.

When she’s not working to meet school and month-end deadlines, you can find Jannelle outside walking her dog, and enjoying the beautiful Vancouver air.
Follow Me On: Facebook

You may also like...

By continuing to browse or by clicking “Accept” you agree to the storing of first- and third-party cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts.
Cookie policy | Privacy Policy

Privacy Preference Center

Close

Your Privacy

Umami Marketing Inc. appreciates your interest in its products and your visit to this website and respects the privacy and the integrity of any information that you provide us as a user of this Site. The protection of your privacy in the processing of your personal data is an important concern to which we pay special attention during our business processes.

Privacy Policy

Required
Personal data collected during visits to our websites are processed by us according to the legal provisions valid for the countries in which the websites are maintained. Our data protection policy is also based on the data protection policy applicable to Umami Marketing Inc. Read more

Cookie Policy

Required
Umami Marketing uses cookies and similar technologies, such as HTML5 web storage and local shared objects (all referred to as ‘cookies’ below), to record the preferences of users and optimize the design of its websites. They make navigation easier and increase the user-friendliness of a website. Read more

Essential cookies

These cookies are essential for websites and their features to work properly. Without these cookies, services such as the vehicle configurator may be disabled.

Cookies used

  • WordPress Required

Performance Cookies

These cookies collect information about how you use websites. Performance cookies help us, for example, to identify especially popular areas of our website. In this way, we can adapt the content of our websites more specifically to your needs and thereby improve what we offer you. These cookies do not collect personal data. Further details on how the information is collected and analyzed can be found in the section ‘Analysis of usage data’.

Cookies used

Third-party cookies

These cookies are installed by third parties, e.g. social networks. Their main purpose is to integrate social media content on our site, such as social plugins.

Third-party cookies