What is the level at which I can do an A/B test?
An A/B test can be done at a journey level - a test is a comparison of two or more journeys. There is no limit to the number of variations allowed among journeys. A/B, A/B/n and Multivariate tests are supported on the tool.
Can I test more than 2 variations at a time?
Yes. However, in order to identify the true cause behind the results of a test, it is always advised that variations are kept to minimum.
Creating one variation and keeping everything else constant is the golden rule of A/B testing that helps gauge the impact of that one feature that was added/modified/removed.
The higher the number of variations/variants, the longer it will take to conclude the test with the expected confidence level.
How do I split the traffic? is it automatic or can I manually override?
Yes, once you've chosen the journeys you'd like to test, you can choose the traffic split. By default, the traffic is split equally among the variations. You can override it by manually entering the traffic split you prefer. The total traffic across journeys should sum to 100%.
Can I pause or stop tests at any point?
Yes, you can pause or stop an A/B test at any point.
To pause/resume a test
- Choose A/B tests from the top navigation bar
- Click
to pause an A/B test - On resuming an A/B test, the journey served to the audience during the paused state is stopped and the A/B test is resumed
To stop/restart a test
- Once the A/B test is conclusive or you wish to end the test at any point, you can do so by clicking on the
button. It is recommended that the test is stopped only when the confidence is reached and a winner is declared - If you wish to re-run a historic A/B test which was stopped, you can do so by duplicating it and starting a new test
What are the metrics that I can optimise for?
By default, there are 2 measurement criteria to be chosen from, based on business goals.
If you choose Clicks on Recommendation as the measurement criteria, then you can choose the following metrics from the drop-down:
- Click-Through Rate
- User Engagement Rate
If you choose Revenue as the measurement criteria, then you can choose the following metrics from the drop-down:
- Revenue Per Visit
- Average Order Value
- Conversion Rate
You will be able to track all other metrics regardless of the KPI selected in the Performance Metrics tab.
As a best practise, it is recommended that Conversion Rate is measured as KPI when No treatment journey is set as control. If all journeys contain Vue.ai experiences, it is recommended that User Engagement Rate is measured as the KPI.
Can I set different confidence levels?
The default confidence level set, also considered as a best practise is 95%. This input can be customised. Confidence allows you to define the required level of confidence with the A/B test result, thereby allowing you to make business decisions backed by statistics. The confidence % sets the threshold for the probability of a false positive. A 95% confidence means that there is a 5% chance for a false positive. Choose the confidence % for your A/B testing from the drop-down. Read more on best practises here.