Introduction to Multivariate Testing

Overview

Multivariate Testing (MVT) is the testing technique wherein multiple elements on a single page are changed to determine the optimal combination of those changes. The objective of MVT is to find out which combination of elements works best for your audience. In Wingify, you would typically run an MVT to test the visual elements, CTA buttons, form fields, and texts on your webpage.

MVT helps you to:

• Optimize smaller changes on your webpage
• Determine the impact of each element on your webpage
• Test multiple changes in a single test
• Reuse the data about the impact of a certain element’s design as it can be applied to future campaigns, even if the element's context has changed.

While A/B testing and Split URL testing allow you to compare two or more versions of a webpage by splitting traffic equally among them, MVT allows changes on more than one element on a page and helps you find the best converting combination. Also, the MVT test is more complex as it compares several elements to find out those elements which have the highest impact on your website. This helps you determine which combination of elements has the highest conversion rate.

The total number of variations in an MVT is calculated as:

[# of Variations on Element A] X [# of Variations on Element B] … = [Total # of Variations]

For example, you might want to test two versions, each heading and an image on a webpage. Using MVT, you create one variation for the heading and one for the image. To test all the versions now, create a combination of all the variations (2 X 2 = 4), as shown here:

• Version 1: Heading 1 + Image A
• Version 2: Heading 2 + Image A
• Version 3: Heading 1 + Image B
• Version 4: Heading 2 + Image B

1. To get the best out of MVT testing, it's recommended to use it only when your website receives high traffic. Based on this, you can calculate the estimated time that your MVT test will take to produce results. To do this, use the Wingify’s Test Duration Calculator.

NOTE: If your MVT test takes too long to declare results, we recommend breaking down your MVT test into various A / B tests. This helps to reduce the number of combinations required for testing.

2. A Multivariate test is best suited for situations wherein you hypothesise multiple ways to implement a variation. Still, you are unable to decide which combination should be tested.

When Should You Choose Multivariate over A/B?

If your objective is to test the smaller elements instead of completely revamping your web page to figure out the element that makes the most impact on the conversion rate, you must use Multivariate Testing. Also, it would help if you were sure that there is enough traffic and conversions on your website.

Things to Consider While Working with Multivariate Testing

• The only way to add multiple changes to a variation is by using custom JS/ CSS.
  
   
• The EDIT CODE and ADD ELEMENT options are not present in the Visual Editor for Multivariate test.
  
   
• A new section is to be created in a Multivariate test whenever you change any element. When you further modify that element, the section would remain the same. However, on selecting any other element, another section is created. While the test is running, Wingify will serve the combinations of those sections. If there are 2 sections, the number of combinations would be 2*2 = 4.
  
   
• You cannot add the custom changes once the test has been started or paused.
  
   
• You cannot delete a variation once a test has started but can disable it.
  
   
• Running a Multivariate test is useful only when the website receives high traffic.
   

• In a Multivariate test, the combi cookie value differs from the one generated in an A/B test. This is because Wingify automatically creates various combinations for the applied set of changes. The combi cookie value in a Multivariate test is a combination of the Control and Variations selected for different sections. If Control is selected, the section will have a value of 1, and if Variation is selected, the section will have a value of 2.

  For example, if the combi cookie’s value appears as 1%2C1%2C1%2C2, it is interpreted to 1,1,1,2 as “%2C” is the ASCII value of comma separator. This means, in your combination, Wingify has chosen CONTROL for the first three sections and VARIATION for the last section.

  INFO: Combi cookie is generated when a user is chosen for a particular version for a test. This cookie ensures that users see the same combinations they became a part of in the first visit.