This article covers the following:
- Overview
- Available Statistical Models in Wingify
- Available Testing Approaches in Wingify
- Understand the Peeking Problem
- Understand Multiple Variations and Bonferroni Correction
- Access SmartStats Configuration
- When to Apply Each SmartStats Configuration
- FAQs
Overview
SmartStats Configuration lets you choose how Wingify calculates and reports statistical significance for your campaign. You select a statistical model, a testing approach, and optional corrections for multiple variations. Configuring these correctly reduces false winners and helps you make reliable decisions faster.
Available Statistical Models in Wingify
The statistical model determines the mathematical framework Wingify uses to analyze variation performance. Select the model that matches your experimentation methodology before starting a campaign. The model can be changed for campaigns in Draft status. Changes to a running campaign may affect statistical accuracy, hence, Wingify doesn't allow switch the statistical approach in running and paused state.
| Model | How It Works |
|---|---|
| Bayesian Model (Default) | Evaluates when the improvement distribution crosses the ROPE boundary. Results are reported in terms of the probability to be better, worse, or equivalent to the baseline. |
| Frequentist Model | Tests whether observed differences could occur by chance under a null hypothesis. Results are reported in terms of significance levels. |
Note: The Bayesian model is the default for all new campaigns. The report columns displayed in your campaign's Statistics view differ depending on which model is selected. For a full column reference, see the Understand, Navigate, and Customize Your Report Table.
Available Testing Approaches in Wingify
The testing approach controls when and how Wingify evaluates your data for decisions. Three options are available.
| Approach | How It Works |
|---|---|
| Sequential | Continuously checks results and issues early winner or disables recommendations as soon as statistical thresholds are crossed. Adjusts significance levels to account for repeated looks at the data. |
| Fixed Horizon | Calculates the required sample size upfront, then evaluates results once after that visitor count is reached. More robust to weekly traffic fluctuations. |
| Dynamic | Retrospective testing with no fixed sample size bounds. Continuously checks for statistical significance until the campaign is paused. |
Understand the Peeking Problem
Checking A/B test results before the test is complete inflates the false positive rate, which is the chance of declaring a winner that is not genuinely better. Each additional look at the data increases the cumulative probability of a spurious result.
Sequential Testing: The Solution to Peeking
Sequential Testing adjusts statistical thresholds dynamically to account for how many times you have checked the results. You can review the report at any time without inflating the false positive rate. Wingify issues disable and winner recommendations only when the adjusted thresholds are crossed.
Fixed Horizon Testing: For Single-Look Evaluations
Fixed Horizon Testing is appropriate when you commit to reviewing results exactly once, after the pre-calculated visitor count is collected. All statistical calculations are performed at that point. Experiment Vital checks continue to run throughout the campaign duration even in Fixed Horizon mode.
Understand Multiple Variations and Bonferroni Correction
When a campaign includes more than one variation, running multiple simultaneous comparisons increases the overall chance that one variation appears as a winner by luck. Bonferroni Correction compensates for this by tightening the significance threshold for each individual comparison, keeping the experiment-wide false positive rate at the configured level.
Apply Bonferroni Correction whenever your campaign has two or more variations (excluding the baseline). For single-variation campaigns, it has no effect.
Note: Bonferroni Correction also raises the visitor requirement in proportion to the number of variations. The more comparisons being corrected for, the higher the threshold each one must clear.
Access SmartStats Configuration
SmartStats Configuration is accessible from two places.
From the Campaign Configuration
- Open the campaign and go to Configuration.
- Expand More configurations.
- Select SmartStats Configuration from the submenu.
From the Campaign Report
- Open the campaign and go to the Reports tab.
- In the report header toolbar, click the Statistical Configuration icon.
- The SmartStats Configuration pop-up displays.
For more information on setting up the configuration, see Advanced Campaign Configurations: SmartStats Configuration.
Note: Accessing SmartStats Configuration from the report opens the same page as from the campaign configuration. Changes saved here apply immediately to subsequent report calculations.
FAQs
What is the default statistical model for new campaigns?
The Bayesian Model is the default. It is selected automatically when you create a new campaign. You can switch to the Frequentist Model before starting the campaign.
Can I change the statistical model after a campaign starts?
No, you cannot change the statistical model after a campaign is started.
What is Dynamic testing and when should I use it?
Dynamic testing is a retrospective approach with no fixed sample size. It checks for statistical significance continuously until you pause the campaign. Use it for low-traffic tests where you need directional data quickly and can accept lower statistical rigour.
What does Bonferroni Correction do to my required visitor count?
It increases the visitor count required to reach statistical significance. This is expected behaviour. The correction tightens individual significance thresholds to compensate for multiple simultaneous comparisons, which requires more data per comparison.
Why does my report show different columns for Bayesian vs. Frequentist campaigns?
The two models produce different statistical outputs. Bayesian reports show Decision Probabilities. Frequentist reports show Significance Level. Both are accessed through the Statistics view in the report table. Frequentist campaigns in Dynamic mode also show an additional Observed Power column.
Is Sequential Testing compatible with Bonferroni Correction?
Yes. You can apply both simultaneously. Sequential Testing addresses the peeking problem; Bonferroni Correction addresses the multiple-comparisons problem. Applying both provides the strongest protection against false winners when you have multiple variations and check results frequently.
Need more help?
For more information or further assistance, contact Wingify Support.