Foundem’s Google Story
Stifling Innovation and Eroding Consumer Choice
Search engines are the undisputed gateway to the internet, and Google, with a 90% share of the search market in the UK and 72% in the US, wields unprecedented market power. Google’s own revenues last year exceeded $23 billion, but this pales next to the hundreds of billions of dollars of other companies’ revenues that Google controls indirectly through its search results and AdWord listings. Recently, it seems that Google has started quietly exercising this control in ways that can suppress competition, stifle innovation, and erode consumer choice.
Google has always used various penalty filters to remove certain sites from its search results or place them so far down the rankings that they will never be found.
Figure 1: Google search penalties
Typically, Google search penalties take a result’s "natural" ranking position and artificially lower it by some factor (depending on the severity of the penalty). Usually, the severity of a penalty is of little consequence, as any result placed below the first three pages is effectively invisible to users.
Whereas these penalties used to be reserved for spam, or sites caught attempting to cheat Google’s algorithms, they are now increasingly targeted at perfectly legitimate vertical search and directory services. It may not be coincidence that, collectively, these services present a nascent competitive threat to Google’s share of online advertising revenues.
Google grants certain sites immunity to these algorithmic penalties through ‘whitelisting’ (a.k.a. ‘manual lift’). But, these manual interventions are cloaked in secrecy and seem to be reserved mainly for high profile sites that would be conspicuous if absent from Google’s search results.
These vertical-search-targeted penalties, coupled with a manual whitelisting process that seems heavily biased towards established brands, are a significant barrier to new entrants and inevitably suppress innovation.
The experience of Foundem illustrates the innovation-suppressing effect of these penalties.
Founded in 2005, Foundem is a UK-based technology company specialising in Vertical Search. Vertical search is different from and complementary to keyword-based, horizontal search: for example, whereas a traditional horizontal search engine like Google can return a list of sites that sell flights, an appropriate vertical search engine like Foundem or Kayak can go deeper into those sites to return details of the actual flights.
Because vertical search requires an “understanding” of a vertical, and because each vertical is different, vertical search services tend to be highly specialised, focusing on just one or perhaps a small handful of verticals. Before Foundem, there was effectively a technology barrier that prevented horizontal search engines from going deeper and vertical search engines from going broader. Foundem’s unique WebSentient technology allows it to overcome this barrier to go both deep and broad.
Figure 2: The Search Technology Barrier
Foundem’s patented WebSentient™ technology is the world’s first and only universal vertical search platform, enabling Foundem to deliver class-leading vertical search services to virtually any vertical with just a straightforward configuration. Foundem may already be the world’s broadest vertical search service, covering product price comparison, travel, jobs, and property, and it can expand rapidly into myriad additional verticals, including many that lie beyond the reach of traditional services.
In December 2008 The Gadget Show, the UK’s leading technology television programme, named Foundem the UK’s best price comparison site [Video Clip].
Figure 3: The Gadget Show recommends Foundem (December 2008)
And, in October 2009, Which?, the UK’s leading consumer body, surveyed the UK’s twelve leading flight search sites and ranked Foundem third—five places above Kayak (a dedicated travel search service, and one of the leading lights of the US Vertical Search industry).
Figure 4: From Holiday Which? October 2009
Foundem also powers vertical search and price comparison services for many of the UK’s leading media companies, including Bauer, IPC Media, and Future.
Figure 5: Powered by Foundem Examples
That Foundem has achieved all of this while labouring under a three and a half year Google penalty, and with just a tiny fraction of the resources of any of its competitors, is testament to the power and innovation of its unique technology.
Foundem’s Google Search Penalty Part I
Despite Foundem’s strong credentials and track record of innovation, from June 2006 to December 2009 Foundem was effectively “disappeared from the Internet” by a Google penalty that systematically excluded all of its content from Google’s search results.
Throughout Foundem’s three and a half year penalty, Foundem continued to rank normally in Yahoo and Bing.
Figure 6: Example – Foundem ranked normally in Yahoo and Bing throughout its Google penalty
For the query “compare prices shoei xr-1000”, Foundem is one of only two or three truly relevant pages. During its penalty, Foundem ranked #1 in Yahoo, #7 in Bing, and #144 in Google. This broad pattern was repeated for all of Foundem’s price comparison results.
As an example, for the query “compare prices shoei xr-1000”, during Foundem’s penalty it ranked #1 in Yahoo, #7 in Bing, and #144 in Google. To put this in perspective, this is an example of a highly targeted query aimed specifically at a price comparison for a particular motorcycle helmet, and Foundem’s page was one of only one or two actually relevant to the query. Yet, the Google penalty ignored this relevance and artificially dropped Foundem’s result to page 14, well beyond the reach of any Google users. Now that the penalty has been lifted, Foundem ranks #5 in Google for this same query.
This is not an isolated example, this same pattern was repeated for all of Foundem’s tens of thousands of product price comparison pages.
Foundem’s AdWord Penalty and Whitelisting
Just weeks after Foundem’s Search penalty began, Foundem was also struck by an equally sudden and severe AdWord penalty (AdWords are the ‘sponsored links’ above and to the right of Google’s search results, paid for by advertisers on a per click basis).
Google assigns automated ‘Quality Scores’ to its advertisers’ landing pages (the page you land on when you click). The scores are meant to reflect the page’s relevance to the keywords being bid for. Foundem’s AdWord penalty meant that, overnight, Google artificially lowered all of Foundem’s Quality Scores from a typical rating of “excellent” (10/10) to a typical rating of “very poor” (1/10). This instantly raised all of Foundem’s costs per click (minimum bids) from around 5p to around £5. Foundem was now being asked to pay a staggering and prohibitive 10,000% more than its unpenalised competitors.
After a protracted series of appeals, Foundem’s case was eventually escalated to a senior AdWord evangelist who explained that Foundem’s Quality Scores had been automatically downgraded by a new algorithm designed to detect and penalise certain types of service, including price comparison and travel search. In other words, Google had systematically and artificially lowered Foundem’s Quality Scores, not because of any quality or relevance issues, but purely because Foundem had been identified as a vertical search engine.
The same AdWord evangelist then championed Foundem through Google’s “whitelisting” process and, six weeks later, Google granted Foundem immunity from its AdWord penalty by manually “whitelisting” the site names ‘www.foundem.co.uk’ and ‘www.foundem.com’. Within hours, all of Foundem’s Quality Scores and costs per click returned to normal.
Figure 7: Illustrating the Effect of Foundem’s AdWord Penalty and Eventual Manual "Whitelisting"
It is important to note that the sudden dramatic changes to Foundem’s Quality Scores were entirely unrelated to the underlying quality or relevance of Foundem’s pages: the pages did not change before the sudden drop from 10/10 to 1/10, and they did not change before the sudden increase from 1/10 back to 10/10. These quality score changes were purely the result of a penalty being applied and then lifted. It is also important to note that, once penalised in this way, a site’s only means of escape is by manual intervention (e.g. “whitelisting”) by Google.
One of the reasons it is so difficult for legitimate sites struck by these algorithmic penalties to obtain relief is Google’s reluctance to acknowledge publicly, or even to a large extent internally, any kind of manual override of its algorithms. In September 2007, Google confirmed Foundem’s AdWords whitelisting in writing. Yet, in September 2008, Google seemed to conceal the existence and crucial role of these penalties and their manual remedies: “Google told me that it never made judgments of what was ‘good’ and ‘bad’ because it was all in the hands of the algorithm” (Joe Nocera, Stuck in Google’s Doghouse, New York Times, September 12 2008).
Foundem’s Google Search Penalty Part II
Foundem’s Search and AdWord penalties were both triggered by the same underlying Google penalty algorithm. It was therefore clear that Foundem’s search penalty, like its AdWord penalty, had everything to do with Foundem’s role as a vertical search engine, and nothing to do with the quality of its service or pages. It was equally clear that Foundem’s only means of escaping its search penalty was for Google to manually intervene and grant Foundem immunity from it, just as it had done for Foundem’s AdWord penalty.
Yet, despite a long list of credentials that included being vetted by Google’s own AdWord Quality Team, it took a further two years and a hard-fought public campaign to finally open the communications channel that led to Foundem being manually “whitelisted” out of the Search equivalent of its AdWord penalty.
Figure 8: Illustrating Foundem’s Rankings across a broad sample of relevant product price comparison keywords for the period leading up to Summer 2009
Notice that Foundem only appears for the keyword “foundem” (ranked #1, on the far left hand side of the graph).
The first significant change to Foundem’s Google search penalty occurred three years after it began, just days after journalists from the Guardian and the BBC began to question Google about Foundem’s case. On 14 July 2009, Foundem’s home page suddenly emerged from the penalty and started ranking normally for certain phrases, such as “price comparison”:
Figure 9: Illustrating Foundem’s Rankings across a broad sample of relevant product price comparison keywords for the period between July 2009 and 1st December 2009, collected just before Foundem was whitelisted
Unfortunately, the vast majority of Foundem’s pages remained penalised, so this change was largely superficial, and the only material impact was to make Foundem’s penalty more complicated to describe.
In October 2009, a direct communications channel finally opened between Foundem and senior Google personnel, including a senior member of Google’s Search Quality Team. Google has asked that Foundem keep the details of this dialogue private. As a direct result of this dialog, Foundem’s search penalty was finally lifted on December 1, 2009, and Foundem’s organic traffic from Google increased by around 10,000% overnight.
Figure 10: Illustrating Foundem’s Rankings across a broad sample of relevant product price comparison keywords from 2nd December 2009, collected just after Foundem was whitelisted (Note: the dramatic and instantaneous improvement from just hours earlier on December 1).
Power without Responsibility
Google’s overwhelming dominance of search means that no one can doubt the immediate and substantial economic impact of a Google penalty. Any policy that, inadvertently or otherwise, whitelists established sites while leaving emerging sites penalized inevitably suppresses innovation.
Remarkably, Google exercises this immense power without oversight of any kind.
The Google Gap – Public Perception versus Reality
Users expect Google to try to return the most relevant results to their queries, while filtering out any obvious Spam. Users do not expect Google to deliberately filter out high quality sites, irrespective of relevance; yet, with the introduction of these vertical search penalties, this is precisely what Google is now doing.
Google is performing a precarious balancing act between the enduring public perception of it as comprehensive and impartial and the reality that it is increasingly neither. In the court of public opinion, Google continues to nurture the appearance of absolute automated objectivity, while, in the court of law, it vehemently asserts and defends its right to manually and subjectively promote, penalise, or omit whatever it chooses.
Every time Google ranks its search results, it is by definition expressing an opinion. There is a commonly held misconception that, in court, Google is merely defending the right of its algorithms to autonomously express this opinion. But this is not the case. When Google’s opinion lay entirely in the hands of an algorithm that acted in a uniform and methodical way and relied solely on information it found for itself, it was easy to argue that its opinion was fair and objective. With the introduction of vertical search penalties, however, Google has become increasingly reliant on manual intervention to ensure that high-profile sites are not visibly penalised. By introducing special treatment for particular site names manually fed to the algorithm (such as ‘whitelists’), objectivity is lost, and the opinion becomes undeniably subjective.
In court filings, Google confirms such manual intervention, stating categorically that PageRank—one of the factors used to rank Google’s search results—is manually manipulated: “Even if PageRank were entirely determined by an algorithm, which…it is not,…PageRank is no less Google’s expression of opinion about the ‘appeal, popularity and relevance’ of a website than a movie review using a scale of: ‘I hated it’ … or ‘I loved it’” [Attorneys for Google Inc., June 16 2006].
Remarkably, at the same time as Google’s attorneys were arguing in court that "PageRank constitutes Google’s subjective opinion concerning the relative importance of a website…", the information on Google’s website directly contradicted this: "PageRank performs an objective measurement of the importance of web pages", [Source]. This page was eventually altered in May 2007 in a way that accommodates Google’s manual manipulations: "PageRank reflects our view of the importance of web pages" [Source].
Outside of the court room it seems that Google is not quite so comfortable with its shift away from automated impartiality toward manual bias; subtle changes to the text on Google’s website reveal a company struggling to come to terms with this new, and less palatable, reality.
Until recently, Google’s online explanation of its search results read: "A site’s ranking in Google’s search results is automatically determined by computer algorithms…", but this was altered in May 2007 to read "A site’s ranking in Google’s search results relies heavily on computer algorithms …" [Source].
Similarly, where Google used to reassure us that "there is no human involvement or manipulation of results…", [Source], it now offers the considerably less reassuring "We have always taken a pragmatic approach to help improve search quality" [Source].
Even now, Google still regularly denies all manual intervention:
"At Google we do not manually change results.” (Udi Manber, Google VP in Charge of Search Quality, April 16, 2008) [Source].
"Our third philosophy: no manual intervention…The final ordering of the results is decided by our algorithms…, not manually by us. We believe that the subjective judgment of any individual is…subjective, and information distilled by our algorithms…is better than individual subjectivity." (Amit Singhal, Google Fellow in Charge of the Ranking Team, 7 September, 2008) [Source].
Search Neutrality – The Need for a Level Playing Field
There is a pressing need to re-establish the level playing field on which the web was founded, so that innovative sites and services can once again compete on utility and relevance, rather than on the whims of an all-powerful search engine.
Because Google’s new penalties target perfectly legitimate services, they rely on a lack of transparency and openness for their very existence. Unlike previous Google penalties, which were aimed at spam and sites caught attempting to cheat Google’s algorithms, these new penalties are difficult to defend publicly. Instead Google conceals, confuses, and obfuscates; keeping the debate focused firmly on “if” a site is penalised, rather than on “why”.
Google’s overwhelming dominance of European and US Search markets means it probably isn’t allowed to behave like this. Certainly, EU competition law is clear: dominant organisations are not allowed to act purely in their own self interest.
Google needs to introduce an effective, accessible, and transparent appeal process by which all inappropriately penalised sites can obtain whitelisting. There should also be an open and informed discussion about the rationale behind these sweeping, discriminatory, and highly questionable penalties.
If Google is allowed to continue on its present course unchecked, it will have far-reaching consequences for the web’s future development. In the words of Google’s CEO, Eric Schmidt, “Incumbents very seldom invent the future”.
[First Published: 18 August 2009]
[Significantly Updated: 14 October 2009]
[Significantly Updated: February 2010]