Google’s Chief Economist, Hal Varian, noted in a post on the AdWords blog today that conversion rates are pretty much the same across positions on AdWords. To quote the post:

… on average, there is very little variation in conversion rates by position for the same ad. For example, for pages where 11 ads are shown the conversion rate varies by less than 5% across positions. In other words, an ad that had a 1.0% conversion rate in the best position, would have about a 0.95% conversion rate in the worst position, on average. Ads above the search results have a conversion rate within ±2% of right-hand side positions.

In the great scheme of things, I suppose this is possible. If you are analyzing millions or billions of pieces of data, I imagine that the data just normalizes down to small percentage differences between different positions.

The problem here, however, is that the data set is too big to draw real conclusions. For example, if I told you that 95% of Americans make between \$0 and \$166,000, and that 60% make between \$19K and \$91K you might conclude from this data that wealth is evenly distributed in the US, right? But then if I told you that 10% of the US population controlled 71% of the wealth in America, and 1% controlled 38%, you might think of the data in a different light.

Here’s a different way to think about the Google conversion rate data that I think would probably have a much different outcome. Do a conversion rate analysis by “token length”, which is search engine language for the number of words in a search query. If someone types in “baseball” for example, what’s the conversion rate differential between position #1 and position #10, versus a query for “buy Louisville slugger size 28 wooden baseball bat.”

I suspect that the conversion rate for the first query is going to be very low for the first position, simply because you are going to have a lot of browsers who simply click on the first ad that they see. Any browser who eventually makes it down to position #10’s result may very well have turned into an actual shopper after clicking on all the other ads. Conversely, if you already know the exact product you want – and your search query indicates that intent – you are much more likely to convert on the first ad you see that actually offers the specific product you want.

Suffice to say, I’ve seen plenty of examples where conversion rate does vary dramatically by position. To make a general statement that position does not factor into the equation is something that, well, a smart economist who loves statistics but doesn’t really understand SEM might be apt to conclude.

Postscript: for those of you wondering about the picture at the top of the post, it’s a scientist explaining why the Space Shuttle Challenger exploded. For a great explanation of why statistics and the presentation of statistics failed to alert NASA that this disaster would occur, check out Edward Tufte’s book, Visual Explanations.