Summary

Fractional Factorial Multivariate Testing (also known commonly as the Taguchi Method) allows you to simultaneously test several key elements of your website or landing pages. In effect, you get to run several A-B Split Tests at once, with the total traffic requirements that are significantly less than an equivalent number of separate A-B Split Tests.

As an official Google Authorized Consultant we use the Google Website Optimizer for all Multivariate Testing. Unlike fractional factorial methods, the Google Website Optimizer uses full factorial data collection, allowing a more accurate subsequent analysis that can consider variable interactions.

Description

The Taguchi Method has the advantage of compressing the amount of data (website traffic) required, while still giving you the benefits of multiple simultaneous A-B Split Tests.

Let's look at a specific example. You are constructing a landing page and want to test the following three variables:

• Variable 1 - Headline: A or B
• Variable 2 - Picture: A or B
• Variable 3 - Call-to-action: A or B

We can encode each possible version of the site as a unique "recipe":

It might take too long to collect data on all eight recipes to get an answer using traditional a-b-c-d-e-f-g-h split testing. So we carefully constuct a Taguchi Method design and decide to collect data on only two out of the eight recipes: Recipe 0 "AAA", and Recipe 7 "BBB". Note that for each variable we will have a 50/50 mix of A's and B's. We are essentially collecting the data once, but then taking different views of it to isolate the effect of each variable. From the data required to run just one split test we are getting information about three variables. We can then use the actual performance of the chosen recipes to model and predict how the other six should perform.

But nothing comes without a price. Multivariate Testing assumes that all of the page elements that you are considering are independent of each other. In other words, the context in which something is seen has no impact on its effectiveness (i.e. that there are no variable interactions). Clearly this is not the case in online marketing, where we want to create synnergies among the various page elements. This does not mean that you can not get good results with the Taguchi Method. But you may not get the best possible results without considering variable interactions.

This problem can be alleviated by sampling a higher percentage of the possible recipes. In the extreme, you would sample them all (a so-called "full factorial design"). However, this approach explosively increases the amount of data required for the test, and negates the data compression advantages that the Taguchi Method enjoys over A-B Split Testing. The tests grow in size so quickly that it is usually not possible to test more than a handfull of individual page elements unless you have a very high traffic rate to your website.

Test Size & Data Rate

Typically 10-100 unique site versions or "recipes". Minimum of 50 conversion actions per day.

Advantages

• Allows you to test a few page elements at once.
• Requires less data than a comparable number of A-B Split Tests run back to back.

Disadvantages

• Small test sizes (although larger than a-b splits).
• Requires specialized design of experiments (DoE) knowledge to properly set up the test.
• Requires specialized statistics knowledge to understand the results.
• Does not take variable interactions into account.

Also Known As

Multivariate Analysis, Design of Experiments, DoE, Taguchi Methods

Other Conversion Tuning Methods

A-B Split Testing, Non-parametric Tuning, [Comparison of Methods]