How to Bulk Check SEO Stats on a List of Websites

Harvesting SEO information about a site can be a tedious process. Just look at how much work you have to do to get relevant information about one site. What happens if you’re trying to check more than one site, then? Maybe you need to find information on half a dozen competitors. Maybe your client has a network of a dozen sites they want audited. Maybe you’re harvesting information on the top 500 niche sites for an infographic. Let me tell you; harvesting data about each of them one at a time is annoying, tedious, and inefficient. There has to be a better way, right?

The answer, of course, is yes. There are many bulk SEO tools out there. The problem is, most of them only check one thing at a time. So your first step is to determine what metrics you’re trying to measure. Then you can find a tool to measure them in bulk. Here are some of the most common.

Alexa Ranking

This tool, from the appropriately named Bulk SEO Tools site, checks the Alexa global ranking, reach, and 3 month change amount of the URLs you plug into it. It can support up to 500 domains in one run. This is all information you can get by manually checking Alexa, but it’s faster to run the bulk search here. When it’s done, it can play an alert if you choose, since it can take a little time to run the search for hundreds of domains.

The data will be displayed on a table in the page. It contains the length in characters of the domain, the domain name, the global rank, reach, change, country of origin, and country rank, where applicable. There will also be an export to CSV button. One thing to note is that when you click submit the first time, it doesn’t run; it scans your list for duplicate entries and removes them. Then you can click it again to run the search.

Now, I personally don’t like Alexa rank. It’s not really a viable or valuable ranking for any sort of objective measurement. Still, if it’s something you want to monitor, go right ahead.

Basic Onsite SEO

This tool checks a bunch of different metrics and gives you a readout of them. These include page meta info, meta tags, alexa rank, social stats, image meta data, H# tags, link tags, sitemap, robots.txt, and favicon presence. It also has a built-in captcha system to help with harvesting data when Google blocks bot activity.

There are two drawbacks to this tool. The first is that it’s not free. A license will run you $19 and it only includes six months of support from the developer. It also only checks ten URLs at once, so if you have a list of 100+ sites you need to check, it can still be tedious to check them all, just 1/10^th as tedious as it would be doing it one by one.

If you have one particular site you want a more detailed audit for, you can use this tool instead. It analyzes a number of additional onsite SEO factors, including meta title, description, keywords, and H1/H2 tags, image meta data, robots meta data, the Google Earth KML tag, canonicalization, stylesheets, scripts, favicons, link meta data, and compression data.

The reason I like this tool is that it not only gives you information about how your site is handling each of these issues, but it gives you context for why they’re important. As an SEO I already know that information, but it’s great for showing to a client that might not be on the ball.

Server Response Codes

This free web-based tool checks the server response codes – 404, 301, etc – on a URL when you plug it in. This helps you identify if a page is successfully displaying, if there’s a redirect in place, or if there are other server codes throwing errors. For example, plugging in “seoblog.com” shows me that there’s a 301 redirect on that URL to https://www.seoblog.com, the appropriate URL for this site.

You can run up to 50 URLs at a time using the web tool, or possibly more if you want to run it via WordPress plugin on your own site, linked in their post. It also has a “download excel” button for the spreadsheet version of the data when you’re done.

A similar tool here will show you the same information, but with a trick; it allows you to specify the user agent you want to use for the search. This way you can pretend to be using a different web browser, or even a different device, so you can see if the site in question redirects to different URLs based on user agent.

Keyword Search Ranking

This tool essentially allows you to select a keyword and then run a site:yoururl.com search for that keyword, and then pulls the top ranking pages from your site for that keyword. It uses Google’s API so you don’t need to worry about breaking search captchas, and it’s a lot faster than doing it all manually.

This is a “bulk” tool, but it only does one URL at once. The trick is that it runs the search on that URL for as many keywords as you want to plug into the list.

Majestic FlowMetrics

Ever since the recent death of PageRank and the long slow decline before that, different metrics have been cropping up as attempted replacements. One of the best, with the largest site index and the most value to a basic SEO, are the Majestic metrics. These are Citation Flow and Trust Flow. You can normally check them from Majestic themselves, but it can be time consuming to do it manually.

This tool checks both metrics and pulls basic data about them, showing you the flowmetrics graph. For more detailed information, you will have to click the graph to go to the actual Majestic page for the sites, unfortunately. This isn’t a very robust tool for that reason.

Code Validation

Two things that can kill a search ranking, and yet be tricky to spot from the outside, are invalid bits of code. HTML and CSS both need to be valid to rank highly, but they can function even if the code is slightly malformed or if it’s adhering to old, out of date standards.

The standard for validation is the W3C, but using their validator is slow because it’s designed for you to test your own pages one at a time. There are two tools here, one for HTML and one for CSS, that you can use to validate both types of code.

• Bulk SEO Tools W3C HTML Validator
• Bulk SEO Tools W3C CSS Validator

One thing to note is that the HTML validator does HTML, HTML5, and XHTML equally. You don’t have to find a different set of tools for each.

MozMetrics

Up there when I mentioned there were other metrics popping up when PageRank died, I didn’t mention all of them. Majestic is one, and they’re a great link analysis, but there is a strong competitor. Moz, the SEO giants, have maintained their Domain Authority, Page Authority, and MozRank metrics for a very long time. Consequently, they are very potent metrics, and are the closest analogue to PageRank that we have.

This tool is interesting because it checks four metrics; Page Authority, Domain Authority, MozRank, and total number of links on a page. It uses the Moz API on its own to allow you to check up to 200 URLs at a time. If you happen to have an API key of your own from Moz, you can plug in your information on the page and check up to 1,000 URLs at a time.

Your data will be displayed with the domain name, Authority metrics, number of backlinks, number of links, and MozRank. You can, of course, download the data as a CSV.

An alternative tool is MozCheck. Again, this will have a limit of up to 200 URLs at a time, and a cap of 3,000 per month. The site also requires that you create a free user account, which may be more work than you want to put in. It’s up to you to decide which tool you prefer.

Domain Index Check

This is a very simple tool that simply checks a URL to see if it is indexed with Google and Bing or not. It doesn’t really give you any more information than that, but that’s fine. It’s not designed for SEO use so much as it is research for buying expired domains. You want to know if a domain has been deindexed before you buy it, and sometimes you’re considering bulk purchasing. This helps you make a smart decision.

A related research tool from the same people is the Domain Age checker. This one checks domains based on their registration information. You will get to see the original registration date, the current expiration date, and the name of the domain registrar. If you’re interested in buying the domain, you can see when it will drop publically – barring re-registration from the current owner – with the Domain Expiration checker as well.

A similar but unrelated checker is a check to see if the domain is listed in DMOZ, or the Open Directory. DMOZ listing is a simple SEO boost a lot of sites do, but is easy to forget. You can check if a site has been listed in DMOZ here. As usual, up to 500 at once.

Social Media Shares

Tracking social metrics can be time consuming if you don’t have access to the analytics API for the accounts involved. This tool helps you speed up the process by harvesting data and displaying it in a nice table, as always downloadable in CSV format. This one shows domain name, Facebook shares, likes and comments, Google Plus shares, Twitter and Pinterest shares, LinkedIn shares, and StumbleUpon upvotes.

One thing to note is that Twitter does not work. Twitter removed the share count from their buttons a while back, and this tool used the public API to pull that data. As a consequence, the tool always reports 0 in the Twitter column. If you use this tool and notice it has changed, let me know and I’ll report that it has been updated.

Hosting Stats

There are two related tools I’m adding to this section. The first is a WHOIS lookup tool that records the domain name WHOIS results. This will show you the owners information including name, address, registrar, phone, email, and more. Of course, if the information is protected or if the domain is owned by a registration company or squatter, that information will be listed in the table rather than the real owner’s information.

The second is a nameserver lookup tool. It’s fairly rare that you will want to look up the nameservers for hundreds of sites, but if you ever find yourself in that situation, you’ll recognize how valuable it is to have a tool to do it all for you.

Are there any tools I missed? I feel as though I’ve covered pretty much all of the major bulk searches you want to do, other than perhaps deep keyword analysis. I don’t think you can automate keyword analysis on multiple sites very easily, though; it often requires a sanity check to keep from duplicating or running unnecessary searches.

If you have a favorite tool I didn’t list, feel free to let me know. I’ll check it out and see if it passes muster. I’ll say this, though; I’m avoiding the content scraper tools that allow you to scrape and spin content or links. They’re too black hat for me, and they don’t actually provide you much benefit anyways. I won’t be adding those to this post.