Note: This article covers SmartStats, Wingify’s Bayesian-powered statistics engine, which applies only to:
- Accounts on the non-Data360 framework, and
- Campaigns created before an account was migrated from non-Data360 to Data360.
For new campaigns (created after migration) or all campaigns in accounts created directly on the Data360 framework, refer to Campaign Reports in Data360, Wingify’s upgraded statistical engine that offers deeper insights and improved control on statistics.
In this article, you’ll learn:
- Overview of A/B Test Report
- Viewing Report for a Date Range
- Viewing Report by Visitor Segments
- Warnings and Notifications
- Customizing the Report View
- Understanding the Reports Table
- Understanding the Graph View
Overview of A/B Test Report
Deriving meaningful insights from your test campaigns and applying them to improve your website conversion rate is your test campaigns' ultimate objective. The test campaign reports help you validate the hypothesis you test and derive a better understanding of your website's visitor behavior.
To access your test report, select the campaign for which you want to view the report and go to the REPORT tab.
To view the report corresponding to a goal, select one from the GOALS panel on the right. Additionally, you can also compare the data of different goals using the COMPARE GOALS option.
Note: While using the Date Range Filter, Segments Filter, or Changing the Base Variation, a loader appears on the page notifying you that the changes are being applied based on your selection. You are not allowed to perform any action until the results are fetched. A success message appears as soon as the changes are applied, and the report data is updated accordingly.
Viewing Report for a Date Range
Date filter enables you to filter data for the specified date range. To specify a date range, you can either use the existing options or can specify a custom date range using the calendar control.
Viewing Report by Visitor Segments
To filter the report by visitor segments, click the All Visitors drop-down menu and select the segment you want to apply. You can use the Segment Gallery (default Wingify segments) or filter the report per the segment you want to apply.
To learn how to create a custom segment, click here.
Warnings and Notifications
The notification box displays the status of your campaign and any warnings you must be aware of.
Note: To declare a variation as a winner or a smart decision, Wingify requires a minimum of 25 conversions per variation, 1500 visitors for the test, and the test must be running for at least a week.
- The notification section displays the campaign's status and any other issues with data collection in a variation or with conversion tracking. You can also view the Winner and Smart Decision notifications here.
- Warnings are displayed if errors were made when setting up the campaign, which can impact your report's data integrity. For example, if you modify a test after it starts running.
- Insights you derive from your test report, or even fresh ideas you may want to test.
To know more about Wingify warnings, refer to Warnings in Wingify.
Customizing the Report View
You can customize the layout of your report using the View Settings option. To access this option, click on the gear icon present next to the stats table.
- Layout Options: From this section, you can customize the view of your table and graph reports.
- Table Options: From this section, you can add or remove columns to the report table, customize table rows to display, and update the data you want to view in the report. You can also move the columns to change how these should appear in the table. For more details, refer to View Settings.
Understanding the Reports Table
The report table displays the data collected for each variation included in the campaign. By default, the control is set as the base for comparing that data. You can change the base to any other variation for calculations using the Use this as Baseline option and can disable or preview the variations using Disable variation and Preview variation options from the vertical ellipsis icon (⋮) present at the right end of the table row.
- Expected conversion rate/Expected revenue per visitor: This is the median conversion rate you can expect from the variation. The "best case" and "worst case" conversion rates represent the 99% credible interval where the conversion rate is likely to be contained.
- Improvement/Revenue improvement: This is the median improvement you can expect over the baseline if you implement the variation. The "best case" and "worst case" values represent the 99% credible interval where improvement is likely to be contained.
- Probability to beat control: The probability of a variation to perform better than the baseline. By default, control is the baseline.
- Probability to be best: This depicts the number of times a variation performed better than all other variations, including control.
- Expected revenue per conversion: This is the median revenue per visitor you can expect from the variation. The "best case" and "worst case" revenue per visitor values represent the 99% credible interval where they are most likely to be contained.
- Conversions/Visitors: The ratio of the number of actual goal conversions to the total number of page visitors.
- Absolute potential loss: This depicts the differences between all the sample sets where variation is not a winner. Absolute Potential Loss finds out how much you stand to lose in a scenario where the variation in question always loses to another variation.
Note: If your Absolute Potential loss is 2% and the expected conversion rate is 10%, you still have a chance to improve this conversion rate and increase it to 12%.
Understanding the Graph View
The graph provides information on the conversion rate, conversions, visitors, revenue per visitor, and revenue. You can view the graph in either of the following ways:
- Date Range
- Box Plot
- Probability Density
Date Range
In the Date Range view, you can view the Conversion Rate, Conversions, Visitors, and Revenue per visitors' data over a period of time. You can choose a custom date range using the Date Filter. For more information on Date Filter, refer to the View Report for a Date Range section.
Note: To get the most actual results, it is recommended to have a diverse set of data. For example, people tend to buy more mobile phones when the manufacturer announces a price cut and conversion rate for that month is not the best indicator of the overall picture.
Interpreting the Change in Conversions
The Graph below shows the comparison between the Control(C) and Variation(V1). The orange line denotes the Control, whereas the blue line denotes the Variation. Initially, both the Control and Variation are at mark zero. As soon as the inflow of visitor traffic starts, data collection begins, and the conversions are tracked. Based on the data collected over time, the change in graph pattern could be seen.
In the above graph, the x-axis represents the dates, and the y-axis represents the conversions. To identify the change between the two, let’s look at the graph pattern of June 21. It clearly shows the number of conversions in the Variation is well beyond the Control. Likewise, you can interpret the graph pattern for all other date ranges.
You can also turn on ranges and see how the conversion rate range for variations have evolved over time. Ideally, the ranges become smaller over time, and that should reflect in the graph. If the ranges go wide, it indicates fluctuations in the data. If the ranges overlap, it signifies uncertainty about which variation is performing better. Read more about uncertainty overlap in the Box Plot section.
Box Plot
Note: Box plot graphs are not applicable for campaigns that use the Enhanced Statistical Engine with Campaign Reports in Data360. They are applicable only for Bayesian campaigns that use the Classic Statistical Engine in both account types, Data360 and non-Data360. For more information on any of these features, contact Wingify Support.
In the Box Plot view, you can view Revenue, Revenue Per Visitor, Conversions, Conversion Rate, and Visitors as box plots to see any uncertainty overlap amongst Variations and Baseline.
Interpreting the Change in Box Plot
In this graph, the box plots of Control and Variations are shown along with the uncertainty overlap. The orange is the box plot of Control, and the other in blue is the box plot of Variation. Based on the data collected, the Best Case, Most Likely, and Worst Case are derived. Here, the Most Likely value represents the median value of the conversion rate. The Best Case and Worst Case values represent extreme cases. In the probability distribution curve below, you’ll see that Best Case and Worst Case scenarios are way less probable than the Most Likely scenario. The more the data, the higher the probability of conversion rate near the Most Likely(median) value. The area above the uncertainty overlap indicates that the conversions are much higher in the Variation compared to the Control.
Probability Density
The probability distribution curve tells us the probability of getting a conversion rate. In Wingify graphs, the x-axis represents the conversion rate, and the y-axis represents the probability. The top of the curve is the conversion rate with the highest probability of getting expected results, and the range of the graph represents the 99-percentile conversion rate range.
Info: By definition, the 99 percentile conversion rate range represents a range that we are 99% confident that the true conversion rate of your variation lies in this range. For instance, Wingify could have chosen to keep the conversion rate to 80 percentile or even lesser but, this would produce incorrect results. Thus, to get the most actual results, Wingify has kept the conversion rate range to 99 percentile.
Interpreting the Change in Probability Density curves
Our initial guess about the conversion rate of your website is anything between 0% and 100%, this indicates the probability of any conversion number is the same. For example, the probability of the conversion rate being 4% is the same as that of 32%, and so on. When you start collecting data, we use it to update our initial guess. The updated guess is what you see in the reports as Conversion Rate. With each data point, you come closer to the actual conversion rate. If you plot the Conversion Rate against its updated probability, you get a graph as above.
NOTE: The probability of the values that are not close to the expected conversion rate starts to drop off and the probability of the values near the expected conversion rate increases.
In the graph, the probability density of Control and Variations are shown along with the uncertainty overlap. The orange curve represents the Control, and the other one in blue represents the curve for Variation. The median value of Variation is 2.16%, whereas, for the control, it is 0.86%. This clearly indicates that the probability of conversion is higher in Variation. The more the data, the higher will be the probability of conversion rate to be near the median value.
INFO: We recommend you to go through this article to get a deeper understanding of the graphs in Wingify.