Transcripted Summary
Many companies today use A/B testing in their web applications to test certain things such as which checkout flow results in more completed orders or which home page design results in more people signing up. Oftentimes, these variations are served randomly by the webserver so it can be difficult to write reliable tests in these cases.
With Applitools, you can associate multiple variations of your page with your baseline to represent each version of the webpage that could be served from the web server. Eyes will then consider a visual test to pass if it matches any of the variations that have been associated with that baseline.
In this chapter, we're going to show you how to set up and manage Baseline Variations.
A/B testing is used to serve different variations of a webpage back to the user at different rates. Sometimes this rate is random and other times it will depend on some other factors. For example a user may hit a web server and get variation 1 50% of the time and variation 2 the other 50% of the time. The question then becomes, how do we visually validate these variations if our baseline in Applitools shows variation 1 and our checkpoint shows variation 2?
Applitools has built in support for these types of scenarios by allowing you to associate up to 20 variations with a single baseline. Then, in order for your test to pass, the checkpoint image must match at least one of the variations that you've associated with this baseline.
Let's start out our example of setting up baseline variations over in our code. I've removed the JS injection from the previous section and also changed the test name to baselineVariations. Let's run this test and generate a new baseline.
public static void simulatePageVariations(JavascriptExecutor jsExec) {
Random rand = new Random();
int result = rand.nextInt(3);
System.out.println(result);
if (result == 1) {
// Left Justify the text
jsExec.executeScript("document.querySelector('.demo-page').setAttribute('class', 'demo-page'); document.querySelector('.demo-page').style.textAlign = 'left'");
} else if (result == 2) {
jsExec.executeScript("document.querySelector('div.section:nth-child(1)').style.backgroundColor = 'lightgray'");
}
}
To simulate baseline variations on the Applitools Hello World Page, I've created a function that will generate a random number and then perform a JS injection that alters the page based on this random number. There are 3 possible outcomes:
- The page is served as normal with no alteration, that's our current baseline
- The text is left justified instead of center
- A background color is added to the title
I then added a call to this function in our test. Let's go ahead and run this test until we get one of the variations as our checkpoint.
Looking at our test result, we've run into the issue where our baseline shows one of the variations and our checkpoint shows a different variation where the header has a gray background. Let's go ahead and add this variation to our baseline.
You can add a variation by clicking on the A/B button.
Then, Click on the Create New button.
And lastly, enter a name for the variation and click the Create button.
As a side note, once you hit create you may see this screen here that will automatically create this variation for the other steps in this test. Feel free to do so on your end. This is a feature known as auto maintenance and we'll be covering this later on this course.
After saving this checkpoint image as a baseline variation, we now have two variations and our test is no longer marked as Unresolved. The test now passes since our checkpoint image matches at least one of our baseline variations.
Before moving on, be sure to click the save button in the top right of our batch. Let's now run our test a few more times until we hit the other variation.
Once again, we notice that we have a mismatch between our checkpoint and all of the variations that we've saved so far. In this case, we can either mark this test as failed if this is a visual bug and not an actual variation or we can create another variation so that this version of the webpage will pass in the future as well.
Following the same steps as last time, I've created another variation for this case where the text is left justified.
Now, when our test passes, the Applitools Visual AI will automatically indicate which variation the checkpoint image was matched against.
When our test comes back as Unresolved, for each of our variations, we can choose to overwrite the variation with the checkpoint image just like we would in a normal test run. We can overwrite a variation even if it's not indicated as the main variation that the checkpoint was matched against. If we want to take a closer look, clicking on one of the variations will run the visual AI comparison against that variation specifically.
Alternatively, we can reject the checkpoint image and no updates will be made to our variations.
At any point in time we can perform certain actions to manage our variations. We can edit their names using the pencil icon.
We can delete variations using the trash can icon.
And if we no longer need these variations, we can use this special icon here to merge all of our variations back into one single baseline image.
In this chapter, we went over how Applitools supports A/B testing seamlessly by allowing you to create and manage baseline variations. Now, when your web server returns one of the variations of your page, your Applitools tests will not fail unnecessarily.
