This article covers:
Overview of Experiment Vitals
Biases creep into experimentation for various reasons. While it is impossible to ensure that campaign results will always be trustworthy, it is far more effective to test for conditions that compromise their validity. Here’s where the Experiment Vitals feature helps you monitor certain checks that ensure your campaign results are trustworthy. Whenever a check is breached, it indicates a potential error in the campaign setup or the collected data. When you open a campaign, you can access Experiment Vitals as follows:
From the Report Header - Click the heart icon in the report header (as highlighted in the following image) to view key integrity checks such as Data Tracking, Conversion Tracking, Minimum Runtime Alert, Guardrails, Experimentation Conduct, and Sample Ratio Mismatch (SRM).
Wingify has multiple campaign health checks enabled in the campaign reports. These checks continuously run in the background and inform you if there is an issue with the campaign. The health checks are campaign-specific and metric-specific. Campaign-specific checks are common to the entire campaign, whereas metric-specific checks are applied separately to each metric.
Campaign-specific Checks
These are the essential configurational checks a campaign is expected to have. A red cross over these checks indicates any fault that arises, and the corresponding resolutions will also be provided. The checks are as follows:
| Data Tracking | It indicates whether data tracking is functioning correctly. If no visitors are tracked in the first three days, it may indicate an issue with the campaign setup. If the collected traffic is very low in the first seven days, it indicates insufficient traffic. Without proper data tracking, your test results may be inaccurate or inconclusive. In such cases, running the campaign becomes ineffective, and we strongly recommend: Verifying the setup - Double-check your targeting settings and tracking codes, and ensure variations are correctly implemented and active. Increasing the traffic - If a setup review doesn't resolve the issue, consider increasing the traffic allocated to the campaign. Wingify recommends a minimum of 500 visitors within a 7-day period to obtain statistically valid results. Extending the duration - Additionally, be cautious when interpreting data collected over a short period, as it may not capture normal fluctuations in user behavior. Although the ideal test duration depends on the specifics of the experiment, Wingify recommends running the campaign for a minimum of seven days to obtain stable and reliable insights. For more information, see Why a Minimum Test Duration Matters? |
| Minimum Runtime | It ensures campaigns run long enough to capture cyclic visitor behavior. Since user behavior often varies throughout the week, all A/B tests should run for at least 7 days. Wingify flags any conclusions drawn before this minimum runtime. |
| Guardrail Checks | It monitors critical user-defined metrics that should not be negatively affected during experimentation. If a guardrail metric shows adverse impact and is not addressed, Wingify highlights this in Experiment Vitals for corrective action. |
| Experimentation Conduct | It detects any changes made during a live campaign—such as updates to metric definitions, traffic allocation, targeting, or variation content—that may bias results. Wingify flags such instances to preserve the integrity of the experiment. |
| Sample Ratio Mismatch (SRM) | It occurs when the actual distribution of users in an A/B test deviates from the intended allocation. This can happen due to technical bugs that prevent certain variations from loading, users deleting or blocking cookies (which affects tracking), browser or device compatibility issues, regional or time-based traffic differences, or internal employee traffic (dogfooding) interfering with the experiment. If not addressed, these factors can lead to inaccurate data and unreliable test outcomes. To learn more about the possible reasons that cause SRM in a campaign, refer to sample ration mismatch. |
Metric-specific Check
The following metric-specific check is designed to check for anomalies in the data tracked for each metric individually:
Conversion Tracking: If no conversions are tracked in a campaign for the first three days, there may be an issue with the campaign setup or that the corresponding goal is not receiving any conversions. In either case, allowing the campaign to continue running is futile, and we strongly recommend checking the setup.
Attention: If one or more vital aspects of the campaign have been breached, it is strongly advised to analyze the results and scrutinize the conclusion.