|
This article covers the following: |
Overview
Note: This article covers SmartStats, Wingify’s Bayesian-powered statistics engine, which applies only to:
- Accounts on the non-Data360 framework, and
- Campaigns created before an account was migrated from non-Data360 to Data360.
For new campaigns (created after migration) or all campaigns in accounts created directly on the Data360 framework, refer to Campaign Reports in Data360, Wingify’s upgraded statistical engine that offers deeper insights and improved control on statistics.
Wingify calls a variation a Winner when its potential loss is below the threshold of caring and its Probability to be best is greater than or equal to 95%. The probability to be best is typically the probability of a variation outperforming all other variations to be the best. Wingify declares only one Winner.
Note: To declare a variation as a winner, Wingify requires a minimum of 25 conversions per variation, 1500 visitors for the test, and the test must be running for at least a week.
How Wingify Calculates the Threshold of Caring
The threshold of caring is the magnitude that decides whether to consider a variation to declare a Smart Decision/Winner or not.
The threshold of caring = Current conversion rate * multiplier * 100
where the multiplier is 0.001 for revenue goal and 0.0075 for non-revenue goal
In the current reporting system, the Conversion Rate of a variation is taken from the test report. Say it is 2.5%, the threshold of caring will be: 0.025 * 0.001 * 100 = 0.0025. Hence, for the variation to be a winner, its potential loss must be below 0.0025, and the Probability to be best is greater than or equal to 95%
Note:
- If the threshold of caring for any test is > 1%, we will set it to 1%, and if the threshold is <0.001%, we will set it to 0.001%.
- The calculations of the threshold of caring were changed in September 2019. The formula earlier was: The threshold of caring = Current conversion rate * Expected lift in conversion rate * multiplier * 100, where multiplier = 0.01 for High Certainty mode; 0.075 for Balanced mode; and 0.2 for Quick Learnings mode, and the current conversion rate and expected lift in conversion rate were accepted from the user.
When is a Winner Retracted in Wingify?
Once a variation has been declared as a winner, the decision will be reversed if the Probability to beat the baseline is below 90% or if the absolute potential loss is greater than 105% of the threshold of caring. Whenever the variation's probability to beat baseline or the potential loss breaches the threshold, the Smart Decision is retracted.
Need more help?
For further assistance or more information, contact Wingify Support.