Screaming Frog is a tool that you can use that will crawl your entire website for you. This crawl will show you everything you need to know about your website. It will show you all the ALT tags, title tags, broken links, and much more for your website. Today, we are going to focus on how to use Screaming Frog to find 400 code errors on your own site. Just for when you don’t know what a 404 error is, you can click here. If you don’t have Screaming Frog, you can follow this link.
Crawl The Website with Screaming Frog
With Screaming Frog open, we are going to start the download. Simply put in the website you want to crawl and hit the start key at the top of the page. If you are just using the free copy, you will have a limit on how many pages you are able to crawl. I don’t have that many pages on my website, so the free copy works fine for me. I have used the paid version as well when I worked in agencies and can truly say it is money well spent.
Bulk Export the 400 Codes
Once the crawl has been completed, we are going to go to the top of the page and look for bulk export. From there, we are going to go down the drop down menu and head over to response codes. From there, we are going to select crawl error (4xx) inlinks.
Excel File Check
Once we have clicked on that report, we are then presented an excel file with our data. I have a pretty small list of places I need to check out, but I can see the posts on my site that are linking to a dead page.
What I like about this report is that I can see exactly where on my site the error has occurred, but I can also see the anchor tag that was used that has the 404 error. I am going to go to the first website on the excel document to take a look to see this problem.
Website Check
I head right over to that URL on my site and I am going to look for that anchor tag that the issue is from. I spot it right at the top of the page. For my own check, I right clicked on my name and went to inspect element. From there, I can see the back-end code that is visible to the search engines when they see my site. No surprise here that I can see the link to the Google + page on my name.
I clicked on my name just to confirm that the dead page is still where the link is pointing to. I can confirm that is the case as I see this picture below from Google.
Summary
Fixing this 404 error can be fixed in a few ways. I can go back into my page and change the destination URL to go to my most up to date Google + page. I could change the destination to my more active social media account, Twitter. I can even get rid of that link all together since I have a link to my social media accounts on the right hand side on my template. Regardless, I have found the issues on my site and I have gone in and cleaned up the 400 error. I want to make my website as SEO friendly as possible, so I want search engine bots to be able to maneuver around my website and get to other web pages without any problems. It of course depends on your workload and how big of a site you are working on, but it would not hurt doing a quick 1 month crawl just to see what is going on with your website. If you notice a lot of crawl issues, you can then start to rank the list and see what you can fix going forward.