|
This article covers the following: |
Overview of Experiment Vitals
Biases creep into experimentation for various reasons. While it is impossible to ensure that campaign results will always be trustworthy, it is far more effective to test for conditions that compromise their validity.
Here’s where the Experiment Vitals feature helps you monitor certain checks that ensure your campaign results are trustworthy. Whenever a check is breached, it indicates a potential error in the campaign setup or the collected data.
When you open a campaign, you can access Experiment Vitals as follows:
From the Report Header - Click the heart icon in the report header to view key integrity checks such as Data Tracking, Conversion Tracking, Minimum Runtime Alert, Guardrails, Experimentation Conduct, and Sample Ratio Mismatch (SRM).
Wingify has multiple campaign health checks enabled in the campaign reports. These checks continuously run in the background and inform you if there is an issue with the campaign. The health checks are campaign-specific and metric-specific. Campaign-specific checks are common to the entire campaign, whereas metric-specific checks are applied separately to each metric.
Campaign-specific Checks
These are the essential configurational checks a campaign is expected to have. A red cross over these checks indicates any fault that arises, and the corresponding resolutions will also be provided. The checks are as follows:
| Data Tracking |
It indicates whether data tracking is functioning correctly.
Without proper data tracking, your test results may be inaccurate or inconclusive. In such cases, running the campaign becomes ineffective, and we strongly recommend:
|
| Minimum Runtime | It ensures campaigns run long enough to capture cyclic visitor behavior. Since user behavior often varies throughout the week, all A/B tests should run for at least 7 days. Wingify flags any conclusions drawn before this minimum runtime. |
| Guardrail Checks | It monitors critical user-defined metrics that should not be negatively affected during experimentation. If a guardrail metric shows adverse impact and is not addressed, Wingify highlights this in Experiment Vitals for corrective action. |
| Experimentation Conduct | It detects any changes made during a live campaign—such as updates to metric definitions, traffic allocation, targeting, or variation content—that may bias results. Wingify flags such instances to preserve the integrity of the experiment. |
| Sample Ratio Mismatch (SRM) | It occurs when the actual distribution of users in an A/B test deviates from the intended allocation. This can happen due to technical bugs that prevent certain variations from loading, users deleting or blocking cookies (which affects tracking), browser or device compatibility issues, regional or time-based traffic differences, or internal employee traffic (dogfooding) interfering with the experiment. If not addressed, these factors can lead to inaccurate data and unreliable test outcomes. |
Metric-specific Check
The following metric-specific check is designed to check for anomalies in the data tracked for each metric individually:
Conversion Tracking: If no conversions are tracked in a campaign for the first three days, there may be an issue with the campaign setup or that the corresponding goal is not receiving any conversions. In either case, allowing the campaign to continue running is futile, and we strongly recommend checking the setup.
Attention: If one or more vital aspects of the campaign have been breached, it is strongly advised to analyze the results and scrutinize the conclusion.