Today’s post is by Terry Whalen, Managing Director at Sum Digital, a San Francisco digital marketing agency. Terry has managed millions in PPC spend for consumer and B2B advertisers since 2006.

Disclosure: Terry may be an admitted Optimizely fan-boy, but he has no affiliation and zero financial interest in touting the tool. He just thinks it’s the bee’s knees.

Optimizely is an A/B testing tool that works well and – all-importantly – is easy to use. I’ve not been able to think of any functionality that it is missing, and that’s saying a lot.

The Benefits of A/B Testing Tools

A/B testing tools let non-technical users modify website pages and then test the modified page against the original, or control page. To get a sense of the true benefit of this, you have to have experienced the bottlenecks that occur when A/B testing is done the more traditional way.

Typically, graphic design, web development and web admin resources are needed. And there are usually big delays in procuring these resources, even for the smallest changes. Those of you who have waited months for your company or client to paste a simple conversion tag know just what I’m talking about! Using a 3rd-party testing tool enables online marketers to circumvent these bottlenecks, and this makes all the difference for successful testing. Online marketing is about improving performance from impressions all the way through to revenue. It’s also about… momentum! If you can eliminate a bottleneck in the chain, you’re on your way to nirvana.

Functionality for any type of software often sounds fantastic – but when the software doesn’t work well or it is difficult to use, then functionality becomes a moot point. Think about Google’s Website Optimizer – remember what amazing news it was when it was released? The online marketing world would be changed forever. But then folks actually used it. For each day I interacted with the interface, I think I aged five days. I haven’t used it at all in the last few years.

Optimizely Works

Optimizely works well and it’s easy to use. The interface is intuitive, and the UI guys are iterating quickly to make it even better. I greatly dislike most software I come into contact with – but I am a huge fan of this tool.

I’ve been doing landing-page optimization for many years. The usual process has been to take a page and “redline” it with suggested changes, including content I think should be added and also deleted. I don’t know Photoshop, so instead I use Snagit, which is a great tool itself – but communicating suggested changes is a time-consuming process. It’s faster to use Optimizely’s what-you-see-is-what-you-get (WYSIWYG) editor to move things around and get the page the way I want it for testing.  And, the webdev folks don’t have to do any work until a variation has been proven to be a winner. At that point, the webdev folks do need to create the page and host it, which they are happy to do since this isn’t just a test – it’s now been proven to be a winner!

You save time by not having to communicate what you’re thinking, and you save time in implementation – no waiting for your client to tell you that the page has been delayed; no cloning ads and then waiting for ad approvals (or seeing disapprovals) in AdWords; and no worries about Google not rotating ads evenly. Lastly, statistical significance measurements are built into the product. And you can send a URL to view experiment results to anyone, even people who aren’t listed users for that project.

Optimizely Features & Functionality

Visitor Targeting – you can choose which visitors will be included in your test by referring URL, query parameter, geo location, browser, and even language preference. You can also specify what percentage of traffic to test on.

Goals – page views, clicks, or custom events are all options, and there is an ‘engagement’ default goal that seems useful.

Page Editor – the WYSIWYG editor is powerful and easy to use – it seems intuitive. You can work at the CSS level or the html level or even the JavaScript level if you want to.

Run Experiments on Multiple Pages – for example, you can test a change to a site-wide element; just modify the element and then specify sets of URLs on which the experiment should run.

Scheduling – you can schedule experiments to start and stop at certain days/times.

Duplicate – easily duplicate a page variation or an entire test.

Multivariate – enabled for higher-cost plans.

Speed – Akamai is used for content delivery; the code snippet takes about 100ms to load.

Using Optimizely

First sign up for a 30-day free trial, which is available for all pricing plans offered. If you have questions, call (800) 252-9480 – they actually answered the phone and were extremely knowledgeable when I called.

Once you’re up and running, the first step is to create a project, which is a default step as you get started. Projects correspond with website domains. So, if you are an advertiser with 1 domain, you’ll probably never need more than 1 project.

Even with cross-domain shopping cart or tracking scenarios, you’d use just one project. Every project has its own code snippet, a single line of JavaScript. Importantly, the snippet needs to be inserted into the *universal header* of the website associated with each project.

Curiously, you are only able to name your project after you create it. To name it, click on the ‘rename project’ option under project settings. Although the project code snippet seems to “update” after you give it its name, the snippet does not actually change. You need not worry whether you copy the snippet before or after naming your project.

Now, click the green ‘new experiment’ button to start creating your first experiment! Specify the ‘experiment URL,’ which is the page you want to modify and test. Importantly, if you modify page elements that are common to other pages besides just your experiment page, you can tell Optimizely to include those pages in your test. For instance, if you are altering a ‘Free Trial’ button that is included in the header of a bunch of pages on the website, you only need to alter the test page button, but you can choose to include in your test all the other pages that have this common button.

Once you’ve created your experiment, you are thrown onto the experiment page, which is also the WYSIWYG editor page. You can create multiple variations to test against the control page (the ‘experiment URL’).

Just hover over various page elements and click to specify whether you want to resize, move, or edit the element. If you mess up, there is an undo button. Another benefit to using the page editor is that it saves time in communicating proposed changes.

Once you are finished with your variation page(s), click ‘set up goals’ and add pageview, clicks, or custom event goals. Next you can click on ‘options’ found on the upper far right of the page, and then click on ‘targeting’ to choose which visitors should participate in your test.

A great tip to remember is that you can start modifying your test page immediately – you do not need to wait for the snippet to be inserted on the domain associated with your project and experiment. Once you actually start to run your experiment, Optimizely lets you know if the snippet is not in place.

Before you actually start your experiment, you’ll want to preview your variation pages to make sure they appear as you want them to. You can do this by clicking the ‘variation’ drop-down. Cross-browser testing is also located in this drop-down (the cross-browser test option comes with all plans but the lowest-priced one).

You are now ready to start running your experiment by clicking the ‘start experiment’ button in the upper right-hand corner of the page. As of this writing, you should expect to get a pop-up dialogue box telling you to insert the code snippet onto the domain associated with the project, even if you have already done that. Not to worry – just ignore this message (if it confirms the snippet is not in place, that’s a different matter). Curiously, in order to start seeing results from the experiment, you have to click back into ‘options’ and choose ‘view results’.

At some point, I’m sure ‘view results’ will be a large button on the each experiment (editor) page. You can also see results on the ‘my experiments’ page, but those don’t seem to be updated as often. If you don’t see results populating, wait 5 minutes.













Here is a sample results page.

Of course, there are other A/B testing tools on the market – Visual Website Optimizer, for one, is robust and works well. But if you try Optimizely, I think you will be impressed.

Terry Whalen, CPC Search


  1. JaimeTSikora June 28th, 2012

    Optimizely: A Secret Weapon for Online Marketers | PPC Associates Blog #ppcchat @ppcassociates

  2. malthauser June 28th, 2012

    #Optimizely #abtest

  3. jmthefourth June 28th, 2012

    Optimizely: A Secret Weapon for Online Marketers | PPC Associates Blog via @tdwhalenho

  4. GergelyTilly June 28th, 2012

    Jó alkalmazásnak tűnik, kipróbáljuk! #ABtest RT @PPCAssociates: @Optimizely: a secret weapon for online marketers:

  5. rodnitzky June 29th, 2012

    Optimizely: A Secret Weapon for Online Marketers – Today’s post is a guest entry from Terry Whalen, Managing Directo…

  6. t_orth June 29th, 2012

    Optimizely: A Secret Weapon for Online Marketers | PPC Associates Blog #sm #analytics

  7. 1MattHopkins June 29th, 2012

    Optimizely: A Secret Weapon for Online Marketers << love Optimizely

  8. DanButiu June 29th, 2012

    Optimizely: A Secret Weapon for Online Marketers | PPC Associates Blog via @wibiya

  9. therustybear June 29th, 2012

    “Optimizely: A Secret Weapon for Online Marketers”

  10. ThinkFirstWeb June 29th, 2012

    Optimizely: A Secret Weapon for Online Marketers

  11. Biquitous July 2nd, 2012

    Optimizely: A Secret Weapon for Online Marketers

Leave a Comment

Terry Whalen
Terry Whalen is Managing Director at Sum Digital, a San Francisco digital marketing agency. Terry has managed millions in PPC spend for consumer and B2B advertisers since 2006.