Set Up Multivariate Rules in Feature Experimentation

Feature Availability: This feature is available with Wingify Feature Experimentation Pro and Enterprise plans.

This article covers the following:

• Overview
• Prerequisites
• Set Up a Multivariate Rule
• Troubleshooting
• FAQs

Overview

Multivariate rules are experimentation rules that enable you to run sophisticated multivariate tests directly on flag-controlled features. Multivariate Testing (MVT) is a technique in which multiple elements are changed to determine the optimal combination of those changes. It helps you identify not just which individual changes work best, but how they work together. The objective of MVT is to determine which combination of elements works best for your audience.

For example, an e-commerce brand wants to test two pricing models and a feature flag for a promotional banner.

With MVT, the brand can define two variables:

Variable 1 (Pricing): $10, $20

Variable 2 (Promo Banner Flag): False, True

Wingify automatically generates four combinations (2 variables x 2 values) as follows:

Wingify shows each combination to a different set of visitors to identify which pair performs best. The multivariate test helps you understand how changes to one element affect the performance of another. For example, the $20 price might lead to more conversions, but only when the Promo Banner Flag is enabled. On the other hand, the $10 price might work better when shown without the banner.

This article further explains how to use multivariate testing for feature rollout optimization and set up a multivariate test in Wingify Feature Experimentation.

Prerequisites

The following prerequisites must be in place before creating a rule:

• Ensure you have:
  • An active Wingify account.
  • The appropriate user permissions within Wingify to create and manage feature flags and rules. You need Design-level access or higher to create a feature flag and Publish-level access or higher to publish it. For more information, see Wingify Account Hierarchy.
• Install and initialize the Wingify Feature Experimentation SDK in your application.
• Create a Feature Flag in Wingify Feature Experimentation that represents the feature you want to release, test, or personalize. Make sure you define variables and assign more than one value to each. Using the variables and their different values, Wingify automatically creates Combinations, which are served to the visitors in the experiment.
• Define the metrics relevant to your experiment's hypothesis.

Set Up a Multivariate Rule

To use multivariate testing for feature rollout optimization, you must first set up a multivariate rule. Follow the steps outlined in this section:

1. Log in to your Wingify account.
2. From the main panel on the left, go to Feature Experimentation > Feature Flags. Create a new feature flag or select the one you want to work with.
3. Create variables for the components of your feature that you want to control directly from the Wingify dashboard without deploying new code. Go to Configuration > Variables > Add Variables. For each variable:
4. Assign a clear Name.
5. Select the appropriate Type: String, Boolean (true/false), Integer, Float, or JSON (for more complex, dynamic configurations).
6. Assign the different values you want to test.
7. Configure the metrics you want to track to measure the impact of the feature flag. Go to Metrics and add the primary and secondary metrics. For more information on how to do that, see Working with Metrics in Wingify.
8. To configure the rule, navigate to the Rules tab.
9. Choose the Environment in which you want these rules to work. You'll see options like Production (for live users), Staging, or Development.
10. On the Testing and Personalize tab, click Create new Multivariate rule.
11. Configure the following details for the rule:
    • Name: Enter a clear and descriptive name for your experiment.
    • (Optional) Hypothesis: Select an existing hypothesis from the dropdown to link it to your rule. This helps keep your experimentation efforts organized and tied to strategic goals.
    • Audience: Specify which users should be included in this experiment. By default, all visitors become part of the experiment. This option allows you to use Wingify's default segments or create a custom visitor target segment that meets your requirements. For more information, see Using Segmentation in Wingify.
    • Traffic Allocation: Specify the percentage of visitor traffic you want to include in the experiment. Move the slider from left to right to increase the traffic percentage, or manually enter the percentage in the text box.
    • Experimentation Mode: Choose the experimentation mode. The default selection is set to Full Factorial.
      1. Full Factorial: This method tests all possible combinations of variables. It provides the most accurate results but requires more samples and time.
      2. Partial Factorial: Tests a subset of variable combinations using optimal arrays. In this method, Wingify strategically selects only a subset of combinations to test. By analyzing how these selected combinations perform, Wingify can estimate the impact of all possible combinations (including the ones that are not tested), while using fewer visitors and resources. Note: To use Partial Factorial, ensure you have added at least five variables with at least two values for each variable when setting up the feature flag.
    • Combinations to serve: When you define variables and assign values while setting up the feature flag, Wingify automatically generates all possible combinations of those values. Each row under Combinations to serve represents a unique combination of variable values. Hover over a combination to see the actual values used in that combination.
    • Distribute Traffic: Choose how to split visitor traffic among the combinations. Assign the percentage of the targeted audience traffic to each combination. You can choose from:
      1. Equal Distribution: Wingify automatically splits traffic evenly among all combinations.
      2. Custom Distribution: Manually enter the traffic percentage for each combination. The total must sum to 100%.
      3. Auto-distribute (Multi-arm bandit): Wingify dynamically allocates more traffic to combinations that perform better over time, maximizing the performance of the selected goal during the experiment. This is often used when you want to optimize conversions during the test itself.
    • Force users to combinations: This option allows you to force specific users to see specific combinations instead of random combinations. Check the box, mention the user IDs, and select the combination you want them to see.
12. (Optional) Add a description: Enter a clear description for your experiment.
13. (Optional) Configure the Advanced Settings to set up your reports using advanced statistical adjustments.
    • Set the Testing Approach: When running a test, you might need to check the results frequently to make timely decisions. This can lead to confusing results and false winners. This section lets you choose from two Wingify tools to help you make accurate decisions, even if you check results early or have multiple variations in your test. Choose between:
      1. Fixed Horizon: The traditional approach where the experiment runs until a pre-calculated sample size or duration is met, after which statistical significance is determined.
      2. Sequential Testing: Allows you to analyze results and potentially reach a conclusion faster, as significance is evaluated continuously as data comes in. It's often suitable when speed is crucial, but requires careful interpretation.
    • Apply Bonferroni Correction: This is recommended when testing multiple combinations. When making multiple comparisons, it adjusts the significance threshold to reduce the chance of incorrectly declaring a winner due to random fluctuations (false positives).
    • Pause rule once duration is completed: Check this box if you want Wingify to automatically pause the experiment rule once a statistically significant winner is declared according to your settings.
    • Enter Salt Value: This ensures sticky bucketing, meaning a specific user is consistently assigned to the same combination across multiple rules and flags, providing a better user experience during the test. For more information, see Salting.
14. Create and enable the rule: Once you have configured all the settings, click Create. You are auto-navigated to the Testing and Personalize tab. Switch the toggle on to enable the rule. Ensure both the main Feature Flag and the Experimentation Rule are activated (toggle switched on) for the test to start collecting data.

Monitor the results of your campaign. Allow the experiment to run for a sufficient duration to collect meaningful data. To monitor the performance of your combinations and check for statistical significance, go to Feature Experimentation > Flags Multivariate > Your Campaign > Reports.

Troubleshooting

FAQs

• Can I add or remove variables after launching a Multivariate test?
  No. Modifying the variables or their values after the test starts will invalidate the results. You should pause the current test and create a new one with the updated variable configuration.

Need more help?

For further assistance or more information, contact Wingify Support.