Crawl control is a section within Bing Webmaster Tools that allows you to tell Bing how aggressive they should crawl your site at certain hours. By limiting the crawl at your most busy hours during the day, you will keep your bandwidth and deliver a better experiencing for your audience since the site will load faster. See how!
The past two weeks or so, I have been focused on improving the speed performance on my own site. I wrote a quick blog about how to compress images on your site to improve the speed, but I have not stopped there. The next area I focused on is how Bingbot crawls through my site. In Bing Webmaster Tools, you have the ability to tell Bing when you want Bing to crawl through your site. This is based on a metric of ten units that are represented in blue columns. Ten represents when you want Bing to crawl your site most. One represents when you tell Bing to slow down their crawl rate. This ability allows you to get the best performance from your site by limiting Bing’s crawl through your site when you are getting the most traffic. Likewise, you can allow Bing to crawl through your site during the off hours to allow for SEO indexing while your audience is away.
Crawl Control Steps in Bing
- Head to Configure My Site in Bing Webmaster Tools
- Click on Crawl Control in Bing Webmaster Tools
- Choose a Custom Time Range
- Move the Columns up or down based on how heavy you want your crawl rate to be during a normal 24 hour day
- Click Save
If you are able to see the logs of your bandwidth, you will have a good story to show the before and after when it came to the crawl of the site.
How to See What Time You get Traffic?
I am going to use Google Analytics for this demonstration. I am also going to view all of my traffic for my website with all the channels. While I will have a lot of rows to deal with in GA, I will use excel to quickly sort through all of this data, but let’s first start off with how to get the hour report in Google Analytics.
Under the acquisition tab, I clicked on the channels button. Next, I went to secondary dimension and clicked on the word hour. Once clicked on, Google breaks down my visits in the military time display. As a note, I would recommend getting a few months of data if you have it available. For now, I just decided to use the last few months to get a baseline from my blog. Anyway, I now see a report like this in Google Analytics.
From here, I went to the export key at the top and downloaded everything into excel. Again, I want to save time, so we will be creating a pivot table to see the data at a high level.
Before the Pivot Table
Here is the data straight from Google. This is good, but this will take me a lot of time to go through. I am going to select the entire document and create a pivot table to better sort this information.
After Creating a Pivot Table
I took all of this data in excel and created a Pivot Table to view everything just at a high level. As a note, I purposely did not check off the default channel grouping because I want to see all the traffic .
Sort the Data with a Condition Format
From here, I like to take the data from the pivot table and sort it all with a condition statement. I am going to set a reverse color scheme to show me in red when I should not have Bingbot crawl my site. Here is that data below. Not surprising, but the heavy times are during the late morning / early afternoon times.
Conclusion
So again, I go back in Bing Webmaster Tools with this data and set Bing to crawl my site during the off hours. I also tell Bing to hold back crawling my site during the busy times. How much of a difference does this really make? I see this more as a best practice then a ranking statistic. If I can give my website any sort of edge when it comes to SEO, I am going to look into it. While I look for other ways to improve my site’s speed, I will share them on this blog!