Mathematically speaking, even a very small improvement over the baseline can be considered and given enough visitors, it will be declared statistically significant. However, from a business perspective, not all improvements are worth deploying, and we have found that experimenters often ignore such infinitesimal changes. The case is the same for any equivalent decline, as well.
Such a region where any minuscule improvement or decline is regarded as equivalent to the baseline’s conversion rate is called the Region of Practical Equivalence (ROPE). Businesses can configure the ROPE for their campaigns individually for all metrics, depending on their scenario.
Assume that your baseline conversion rate (aka the baseline average) is 40%. You have decided that any conversion rate between 38% and 42% will be regarded as equivalent to that of the baseline. For this, ROPE is calculated as follows:
Step 1: Obtain the difference between these individual extremes and the baseline average. Here, the difference is ± 2.
Step 2: Divide the difference by the baseline average.
Now, ROPE = ± 2 / (40%) = ± 5
You can set ± 5 as the ROPE for your campaign in this case.
Unlocking Insights: The Value of ROPE in Decision-Making
There are three main ways in which Wingify uses ROPE to deliver value to customers. Note that a higher value of ROPE will lead to an increased intensity of these benefits but also has some tradeoffs that are discussed in the next section.
- Declaring when to stop a variation: Using ROPE, Wingify can give an early decision to disable a variation if it detects that the variation does not have the potential to be better than the baseline.
- Minimize False Positives: Without the concept of ROPE, we might mistake random fluctuations in data for significant improvements. ROPE acts as a safeguard against false positives, helping us focus on changes that truly matter.
- Saving Visitors on Average: Including ROPE helps accelerate the decision to disable underperforming variations and campaigns, which can lead to overall resource savings. Since campaigns with significant winners are typically less common than those that don't produce a clear winner, using ROPE often results in conserving visitors on average.
Considerations for ROPE
Determining the width of your Region of Practical Equivalence (ROPE) is a nuanced decision that depends on various factors, including the nature of your experiment, the desired level of confidence, and the practical significance of the changes you seek.
- Default Values: A conservative ROPE value of ±1% is a good default to start with, as it will save you a good amount of the False Positive Rate. Wingify has configured this by default.
- Understand Your Business Context: Consider the unique aspects of your business and the target metric. Some businesses might have smaller margins of acceptable change, while others may tolerate larger values. Your understanding of what constitutes a meaningful change in your specific context will influence the value of your ROPE.
- Iterative Refinement: A/B testing is an iterative process. Initially, you might start with a broader ROPE and then refine it based on the results of your experiments. This adaptive approach ensures that your ROPE aligns more closely with the actual impact on your business.
In total, ROPE is a vital feature in the optimization toolkit for making informed decisions based on statistically significant and practically relevant data. After all, the essence lies not just in making changes but in implementing those that truly matter for a more optimized and seamless online experience.