Why Aren’t My ACE Tests Rotating Evenly?!
Published: March 4, 2015
Author: Jesse Morris
A/B testing is the backbone of any long-term SEM strategy, and ACE testing is a valuable tool for determining whether changes drive awesome improvements or sad declines in performance. However, it can be frustrating when you see ad rotations like this:
Clearly, the impression volume for test #1 is much higher, leading to more clicks and conversions. But shouldn’t these tests be showing as a 50/50 split? That’s how the test is set up, right? But no. Thanks to an explanation from my friendly Google rep, I know there is a reason for what appears to be an uneven rotation of ads.
It comes down to the ever-important Quality Score. The important point to remember is that the ACE testing rule guarantees that ads will split the numbers of auctions 50/50. It cannot ensure that the ads will win the auctions 50/50.
Look at ad #1. CTR is 54% higher, which is a major factor in determining QS. What this means is that ad #1 has a higher QS and a higher ad rank and is winning a larger number of the auctions. This also leads to #1 showing in higher position, getting more clicks and (in this case) more conversions. This means that when you see an ad starting to pull ahead in impression volume, you may be looking at the future winner of the test even before you actually gain statistically significant results.
However, be careful! The highest CTR ad is not always your best bet. So do wait for significance. An ad that touts a “Free Gift” may have an amazing CTR, but the user may show up only for the gift, see the price of the product, and fail to convert. Google loves CTR, but the true driver of value for most SEMs is conversions.