Your websites Search Engine Results Pages (SERPs) are determined by various SEO factors, on page content (your body text, the title tag etc…), anchor text of links to your web sites pages, the search engines algorithms tweak of the day and of course the number of websites/pages competing for that search phrase: are there a lot of webmasters targeting the phrase.
In this section of the SEO Tutorial we’ll deal with estimating the number of competitors for a Search Engine Results Page (SERP).
Determining your Search Engine Results Pages SEO Competition
The first mistake a lot of amateur SEOs (and professional SEOs for that matter) make is looking at total number of pages found for a given SERP and from this limited information concluding the difficulty and number of competitors for that search phrase.
This method is inaccurate and should not be relied upon, for example the word HOME is found on a lot of pages (5,960,000,000 pages!!), not because they are targeting SERPs related to the word home, but because they use home as the anchor text of home page links and the word home is quite common: there’s almost 6 billion pages indexed in Google for home!
Check out the ‘SEO Home’ SERP, 318,000,000 pages found with a loose search (just type the keywords and search, no speech marks etc… so a loose search) for SEO Home, which would suggest a difficult SERP with quite a lot of competition. But this loose search alone tells us nothing about real competition for the SERP, what we need to know is of those millions of pages how many are targeting or at least mentioning the exact phrase SEO Home, those are your real SERPs competition.
Interestingly in June 2014 the top result for SEO Home isn’t a search engine optimization site, see screenshot below
The top SERP result is for a site about education: Sponsors for Educational Opportunity:
Sponsors for Educational Opportunity (SEO) provides superior educational and career programs to young people from underserved and underrepresented communities to maximize their opportunities for college and career success.
It’s interesting because no SEO expert would ever target a SERP like SEO Home, no one is going to build backlinks with anchor text “SEO Home”. Guess it’s a good thing it’s not an SEO site at number one.
Keyword Proximity Suggests True SEO SERP Competition
An exact phrase match search is where you enclose your search phrase within speech marks (“search phrase goes here”) when making a search in Google. An exact phrase search removes all loose matches (see below).
An exact match would be the bold text below-
1. ‘At SEO Home search engine services we deliver great results’
A loose search match would be text like this-
2. ‘SEO Gold the Home of amazing SEO Consultants’
Keyword proximity (how close the keywords are to each other) is important to Google SEO, as you can appreciate an exact phrase match (like text #1 above) to a search engine is considered much more important than finding the keywords separated (you can’t get a better keyword proximity than next to one another), or a loose match (like text #2 above).
Two similar pages with the above strings of text and no other mention of the keywords SEO and Home you would expect to find the page with text number 1 way above the page with text number 2 for the SEO Home SERP.
We basically assume web pages with perfect keyword proximity for the given search are targeting the SERP either by choice or by luck (not all webmasters are SEO’s).
When you perform the exact phrase match “SEO Home” it gives 119,000 pages. This tells us the vast majority of the 318,000,000 pages found for SEO Home (loose search) are not interested in the SEO Home SERP; only 119,000 pages (less than 1%) are your true competition since they use the exact text on their page.
That is not to say a loose matching page or even a page with some or all of the keywords not present at all can’t rank highly since anchor text alone from high PageRank (PR) links or lots of links with the right anchor text (like the Miserable Failure Googlebomb SERP many years ago) can beat exact phrase matches.
Interestingly the same Sponsors for Educational Opportunity site is ranked number one for the exact search and some of the top ranked SEO sites (they are ranked for loads of SEO phrases) are NOT in the top 10. Couldn’t find Search Engine Watch in the top 50 for this SERP, they were number 2 for the loose search because of a combination of a lot of links with anchor text SEO and the text home used within body content etc… that’s how it should be, shows they aren’t wasting link benefit on home page links with anchor text Home**.
** Hmm, actually they are potentially wasting anchor text benefit by linking to home with the anchor text home, but because it’s not the first link in code Google is ignoring the internal links with anchor text Home. Google reads the code from top to bottom looking for links, the first unique link found is the anchor text used for all the links to a particular page, so if you have 3 links to home on every page only the 1st ones anchor text is counted for the homepages SERPs. See my SEO Test.
An Example SERPs Competition Calculation
We’ll use two SERPs that at first glance appear to have similar levels of competition, at least from just looking at the total number of pages found in Google with a loose search (the inaccurate measure of competitor pages). Then using exact phrase match searches to show how one is relatively easy and the others is relatively difficult.
This section was written in 2006, since then I think the MSN Today service has closed down (or at least changed: the original MSN site for MSN Today signup no longer exists), so no longer a good SERP for SEO analyses, will have to find a new one when I get the time.
The SEO concept and analysis is still valid, only problem is you can’t do the same SERP checking I did many years ago because the SERPs have changed significantly. Real shame as it was a really good example.
It’s very interesting to see the significant increase in pages found for all my example SERPs, I’ve edited the earlier ones (in June 2014) and some went up a lot. For example loose search for SEO Home was 5,360,000 pages, now it’s 318,000,000 pages (from 5 million to 300 million!!!. Exact search for “SEO Home” was 13,900 pages, now it’s 119,000.
That’s an enormous increase in pages Google is indexing today relative to 2005/2006: no wonder SEO is so much harder today, SEO competition is fierce. Even though it looks like MSN Today was closed years ago (was part of Windows Live Messenger, MSN Today was a website that appeared in a small browser or “trident” window.) there’s still almost 2,000 Google searches for MSN Today a month in 2014. So though there’s a lot more competition for all SERP, there’s a lot more visitors as well and they search for weird stuff :-)
MSN Today : 9,850,000 pages
Psychology Today : 8,700,000 pages
Based on this information alone MSN Today should be the harder SERP by a small margin, more pages using those two words should mean more competition.
As you can appreciate an exact phrase match to a search engine is considered much more important than finding the words separated (loose match). So finding a text match like-
1. “At MSN Today you’ll find a search engine hardly anyone uses”
Will be considered more important for the SERP MSN Today than the loose matching text below-
2. “Earlier Today the search engine MSN crashed due to excessive use from the alt.internet.search-engines newsgroup community!”
Both have the two keywords, but number 1. will be considered more relevant (all other things being equal of course) than number 2. This doesn’t mean number 2 can’t show high for a search for MSN Today, but without supporting anchor text (links using the anchor text MSN Today or derivatives) it is highly unlikely.
Exact Phrase Match Searches
“MSN Today” : 10,600 pages
“Psychology Today” : 1,150,000 pages
The exact phrase “MSN Today” is found on only 10,600 pages out of almost 10,000,000 (just 0.1%).
The exact phrase “Psychology Today” is found on 1,150,000 pages out of almost 9,000,000 (~15%).
For the purpose of determining real SEO competition for a SERP you can all but ignore the loose search matches.
The real competition for MSN Today is just 10,600 pages (not that many) since the remaining 9.5 million won’t have enough content relevance to compete effectively (unless they have significant links using the right anchor text as well, which is unlikely). So MSN Today is most likely a relatively easy SERP, especially in comparison to a SERP like Psychology Today where there are 1.15 million pages with good content relevance to that SERP (this could be a difficult SERP to obtain).
Other Factors Impacting on SERP Difficulty
As you saw with the MSN Today example above from the millions of pages found with a loose search only 10,600 are using the exact phrase. This information is vital to determining the difficulty of a SERP, but does not always reflect true difficulty since if there are just 10 pages online targeting a SERP and they all do better than your page you don’t get the prized top 10 listing.
For a search phrase to provide traffic to your site you ideally want one of the top 5 listings and no less than top 10 (first page result). Other than very high traffic SERPs after the 10th result traffic practically stops and so you need to look at what actually has the top 10 listings for a given SERP.
Keeping with our MSN Today example the first two pages are-
1. www.msn.com PR9 with 815,000 backlinks in Google with many using the anchor text MSN.
2. join.msn.com PR7 with 130 backlinks in Google with many using the anchor text MSN.
Even though there are only 10,600 pages listed for this SERP it would be very difficult to beat these two pages, they have high PR and more importantly a lot of the links will use one of the keywords of this SERP (MSN). To beat these two pages you will need good on site optimization and quite a lot of links (probably enough to make your page PR6) using the anchor text MSN and Today (not necessarily together). You will notice you don’t need as many links as the two pages above since neither of them is optimized well for the phrase MSN Today. They are optimized for MSN though, so the lack of optimization for the Today part makes obtaining the SERP a little easier (but still difficult).
Looking beyond those two pages we have- www.slate.com PR7 with 8,310 backlinks which appears to do well thanks to the word today(s) within the content and links using the word MSN as anchor text. This is the type of page that can be relatively easily beat with good on page optimization and relatively small number of links using the exact phrase. So top 3 for this SERP is a real possibility without having to go overboard on the SEO and top 20 should be obtainable with little to no use of anchor text (mentioning the phrase MSN today several times, like this page).
MSN Today SERP Update: July 1st 2005 this page is 6th for the MSN Today SERP. Not bad for no links using that as anchor text or even basic on page optimization. I’ve added a couple of links to the page now with anchor text MSN Today, top 3 is a real possibility.
Quick SEO Tip – If you want to ONLY see the loose matching pages of a SERP use this search format in Google:
Search Phrase -“Search Phrase”
This search will only show pages that have both keywords Search and Phrase, but not in the format Search Phrase (so NOT perfect keyword proximity).
A few SEO caveats about SERPs and how some SEO’s manipulate potential clients. You will find on a lot of SEO company websites lists of SERPs the site or their clients sites rank well for in Google and other search engines. Read through these SERPs lists carefully you will almost always find they consist of really easy search phrases, SERPs that you would get with little to no optimization.
One SEO firms web site I visited recently listed phrases like-
SEO software reviews
And stated they have top 20 listings in Google for them all. Sounds good must be pulling in thousands of targeted search engine traffic, but look closer only the 1st two (in bold) are of any real benefit to a SEO firm, the rest will not result in much if any traffic. They are listed to ‘bulk up’ the SERPs (make it look good to a potential client). The reality is the majority of the above SERPs will result in a handful of visits a month, not exactly the SEO benefits the firm wants to portray!
So when you see a list like this go through it and see if this really is a successful site or just flim flam.
Here’s a few worthless top 10 SERPs this site had in Google (many years ago as I update, not going to add new ones as they’ll tend to come and go over time since not specifically targeting them)-
Some of these sound fine, but I got them from the August 2005 web logs, specifically looking for phrases that have resulted in less than 5 visits that month. So as we’ve listed SERPs above that comes to less than 50 search engine visitors per month (closer to 20 visitors a month). Not very impressive when you do the maths.
Would you hire an SEO company that used those SERPs as proof of how they’ll gain high traffic for your site?