Since Google recently opened the gates for its Website Optimizer to a wider audience, and because we have a paid alternative called MuVar for quite a while, I figured it’s time to compare both programs and see which one brings greater returns in terms of profitability and actual results.
First though, what is multivariate testing anyway?
Before I answer that question, ever heard of split testing, or A/B testing? It is an approach where you show one piece of “something” to 50% of your visitors and another piece of that “something” to the other 50%. After a while, you’ll see which version brings the greater returns (in terms of profitability, conversion rate, etc.)
So split testing is indeed a powerful thing when optimizing sales letters, sign-up pages, and the like. You have a number, a conversion rate, and you want to increase that number. Sure, the first thought would be to increase traffic then you often automatically increase sales or sign-ups. But what if traffic stays the same and you still can optimize the process?
That’s where testing comes into place. And while split testing is a well-known option, multivariate testing is more often than not a much better one because you get results much faster. What it does is it compares multiple variations with each other and shows you which versions or combinations convert better.
So let’s say you want to compare 3 different headline versions with each other, then at the same time you also have 4 different testimonials, and 3 different pictures, different font sizes, and on and on. You get the idea. It actually doesn’t matter what, just that you compare whatever you have in mind, and various options with each other at the same time (not just two.)
Lots of ideas and possibilities, and multivariate testing helps you find out which combination sells better or converts more visitors into subscribers, etc. You have different variables, plug them into the testing system, and compare them with each other to find THE combination.
Now, let’s take a closer look at the two options at hand, MuVar and Google Website Optimizer (gonna call it GWO from now on.) I’m going to continue with bullet points to illustrate the main elements…
- GWO gives you the opportunity of setting up multivariate as well as split tests while MuVar focuses on multivariate tests only. Google says that split tests may be better when your site gets less than 1,000 page views per week. I’m not sure about that. You need a considerable amount of traffic anyway in order to get statistically significant results. When you have only 100 page views per week any testing result is not as meaningful as with 1,000 or 10,000.
- Both MuVar and GWO provide you with video tutorials to walk you through the set-up process. GWO further explains the differences between split and multivariate testing, which does MuVar on its sales page. Google may have an advantage here since it offers an extensive online help (in many different languages!) “behind” the two video tutorials (and its integrated powerful search feature which we’re using every day, aren’t we?)
- The video tutorial MuVar provides walks you through the whole installation and configuration process, so does the GWO, too. But I found the lady in the GWO tutorial is rushing through all the steps within 14 minutes, while the gentleman in the MuVar tutorial takes more than 30 minutes and makes sure each and every step is explained thoroughly. Besides, he shows you a real computer screen instead of a Powerpoint presentation. In other words, I find the GWO tutorial a bit superficial. Yes, I do see what I can do, but I miss the real live examples and “walk-through” shown in the MuVar tutorials. Further, the Google lady speaks into the microphone instead of aside from it, which is the reason why you hear those annoying pop sounds when she’s virtually spiting into the mic. (I’m allergic against this; it’s something many video tutorialists do wrong, often unknowingly.)
- GWO is placed on your Google Account’s home page and is accessible from within your Google Analytics account. I like it when many of my daily tools are available from one place and I don’t have to go to multiple locations in order to get what I want. With MuVar, you need to log in to your site’s MuVar account. When you installed MuVar on, say, 5 sites you need to log in at 5 different locations.
- GWO provides you with reports that show you which combination gives the best results. MuVar gives you these kind of reports as well, but it does a very important task… it optimizes your page by itself without me having to lift a finger! This is a huge advantage and a great time saver. One can say GWO tests while MuVar tests and improves. It’s like having a human resource do that for you every single day, for free. (OK, there is a sort of “batch processing” thing in MuVar that you have to perform occasionally, going to a particular URL once a week or so. I did it myself already; it’s pretty straight forward and can be done within a couple minutes.)
- MuVar can only be used on HTML or HTM pages, whereas (as far as I can see) GWO can be used on PHP pages as well, is that right? They only show HTML extensions in their examples, but they do not explicitly state that you cannot use it in PHP, ASPX, or something like that. I don’t see a need for PHP when having a sales letter or a sign-up page, but more technically inclined Webmasters may have some arguments towards PHP and ASPX.
- Setting up a multivariate test with GWO is a bit more challenging (and perhaps complicated?) than with MuVar. It may be because the lady in the tutorial rushed through the process and even after watching it the third time I haven’t completely understood it quite yet. They do show you where to put which JavaScript code. Though, with MuVar you simply upload a couple files, go to the MuVar administration area on that site, and then enter all the variables you want to use. It’s pretty straight forward. You can even add variables without logging in! And you can use existing HTML code, copy & paste it into MuVar, and then modify it or create multiple variations right there. Pretty simple, pretty neat.
Bottom line, GWO is available for free while MuVar cost $300. I see the quality in what Google does since I use a couple of their tools myself every day. So the word “free” may not apply here when talking about quality. How often did it happen you had to pay for a certain service, then Google bought that company and made that service available free of charge from that point on. (Now, you may wish Google goes ahead and purchases MuVar, but let’s not go that direction for now…)
The biggest advantage MuVar has is its automation and “built-in human resource.” It optimizes whatever you wish to optimize by itself, without me having to do anything (except the occasional “batch processing” and perhaps adding more variables.)
I don’t know about you, but when there is a tool available that saves me a bunch of time I become VERY interested and excited, and I’m more than happy to pay for that excitement. There are lots of things I have to do during the day. And when there is a tool that helps to empty my desk, I’m very grateful for that.
And although all your GWO tests are available from one location, it does not take so much time to go to a few Web sites in order to manage your MuVar installations.
With all that being said, my vote goes to MuVar. I’m going to use it from now on and am really looking forward to substantially improve my business and conversion rates.
—Marcus Hochstadt
