In this article, you’ll learn the following:
- What is MDE?
- Why is MDE Important?
- Visualization of MDE
- MDE and Experiment Design
- Best Practices for MDE
What is MDE?
Minimum Detectable Effect (MDE) is a critical concept in any type of testing. It represents the smallest improvement (uplift) you want to be able to confidently detect as statistically significant in a test metric. In essence, MDE defines the sensitivity of your experiment.
Note: For guardrail metrics, you are trying to guard against maximum reduction in the test metric instead of improvement and to convey this we appropriately rename MDE as MDR (Minimum Detectable Reduction).
Why is MDE Important?
MDE directly impacts the sample size needed for your test. Smaller effects are harder to detect and require more data (samples) compared to larger effects. By defining the MDE upfront, you can ensure your test has the power to identify meaningful changes, even if they are subtle.
Visualization of MDE
Imagine comparing two objects. A significant difference in size is immediately noticeable, while a smaller difference might take longer to detect. Similarly, in testing, larger uplifts are easier to identify than smaller ones.
For instance, you want to test a new design for your website’s homepage to see if it will hike up your conversion rate.
Let's say your baseline conversion rate is 10%. This means that currently, 10% of visitors who come to your homepage convert. You set your MDE to ±5%. This means that you want to be able to detect changes in the conversion rate that are at least 5% larger or smaller than the baseline conversion rate.
In other words, if the new design results in a conversion rate of 9.5% or 10.5%, it won't be statistically significant according to your MDE. This is because the change is within the undetectable range of ±0.5% around the baseline conversion rate of 10%.
Here's a table to summarize the concept:
| Baseline Conversion Rate | MDE | Undetectable Range |
|---|---|---|
| 10% | ±5% | 9.5% - 10.5% |
The undetectable range is simply the range of values that could be due to random chance rather than a real effect of the new design. By setting a smaller MDE, you can reduce the undetectable range and make your test more sensitive to smaller changes in the conversion rate. However, this will also require a larger sample size (more visitors to your website) to achieve statistical significance.
MDE and Experiment Design
While the exact uplift from a test is unknown beforehand (otherwise, the test wouldn't be needed), MDE is crucial for proper experimental design. Statistical principles require considering the minimum effect size you're willing to overlook to determine the necessary sample size.
Best Practices for MDE
Here are some key considerations for setting an MDE:
- Conservative Estimate: Choose an MDE value from the lower end of your expected uplift range. This ensures that larger uplifts are detected while minimizing the chance of missing smaller but relevant improvements.
- Metric Sensitivity: A smaller MDE might be appropriate for stable metrics where large significant uplifts are hard to achieve. Conversely, a larger MDE is preferred for sensitive metrics to detect even minor positive effects.
For example, Imagine you're running an A/B test to improve your website's conversion rate. The size of the change you want to detect will influence how long you need to run the test.
- Stable Metrics (e.g., Revenue per Visitor): These metrics exhibit less fluctuation and require a smaller MDE to detect significant changes.
- Sensitive Metrics (e.g., Time Spent on Page): These metrics are more prone to change, making it easier to detect positive effects with a larger MDE.
- Sample Size and Traffic Volume: A larger sample size allows you to utilize lower MDE values, enabling the detection of smaller uplifts.
- MDE During Experiments: Avoid increasing MDE during a running test, as it can compromise results.
- MDE Adjustment if Needed: If the observed effect is smaller than the MDE, consider increasing the sample size and lowering the MDE to ensure it's below the observed effect. This ensures that you have enough sample size to detect significance for a smaller uplift.
By effectively utilizing MDE, you can design well-structured tests that deliver reliable results. MDE helps you strike the right balance between detecting meaningful changes and ensuring your tests are efficient and cost-effective.