Transcripted Summary
In this chapter, we're going to build off of the previous course and go over Batching more in depth. As a refresher, Batching's main purpose is to allow you to group together your test runs by creating a BatchInfo object and then applying this to your Configuration object. However, there are even more reasons to use batches that go beyond that core use case.
The first thing we're going to cover is Batch Sequences which allow you to track the performance of your batch over time. In this chapter we'll finally see why we added that sequence name way back at the beginning of chapter 1. After that, we'll set up an integration that allows you to send notifications over Slack, Microsoft Teams or Email that contains the test results for your batch. Let's go ahead and jump right in.
So, why did we add that mysterious batchSequence property to our Configuration way back in Chapter 1?
A Batch Sequence is simply a field that gets applied to a batch object and, when applied, lets the Applitools server know that we're interested in tracking this batch's performance over time. As we run tests, we'll be able to track our pass rate on various environments and we can then review all of these metrics on the Insights page within the dashboard.
On our dashboard, let's go ahead and navigate to the insights page by clicking on the dropdown and then selecting Insights.
The insights page has all sorts of interesting data about our tests in general that I highly recommend you check out. We can understand which tests we run are stable and which ones run into issues. For now though, go ahead and click on the Batch Statistics tab so we can check out the data for our batch Sequence.
A quick side note, the data presented on this screen is calculated only at regular intervals on the Applitools server. If you don't see any data just yet that doesn't mean you've done anything wrong, it just means that your batch statistics may not be calculated yet. I encourage you to check again after more time has passed and, for now, just follow along with me.
If we click on our batch Sequence Name, we're presented with a ton of data on how our batch has been performing. The most important metric by far is the Pass Rate in the top left. You want to aim for as high a pass rate as possible as this shows that your webpages are stable and few bugs are being introduced.
Scroll down a little further and you can also see the pass rates of individual browsers and environments. If you notice that your pass rate is suffering, you can use these metrics down here to tell if your tests are failing across the board or if there's a specific environment that appears to be problematic.
In summary, adding a batchSequence to your tests allows you to track all of this aggregate data over time and gain key insights into the performance of your tests. I highly recommend running all of your tests with a batchSequenceName so you don't miss out on this data.
The next topic I want to cover for advanced batching are Batch Notifications. You can set up an integration with Slack, Microsoft Teams or Email where you'll receive notifications when your batch finishes running. This notification will include the status of the tests within the batch as well as a link to the batch on your dashboard. In this section, I'm going to walk you through setting this feature up over email.
There is a two step process for setting up batch notifications. First, we need to set up the integration on our admin panel within the dashboard. Then, we need to make a simple change in our code.
Let's start with the first step by clicking on the dropdown option at the top of your dashboard and selecting Admin.
On this screen, select the Teams option.
Then, select the team that you want to set up the integration for.
Once you've selected your team, you can click on the Integrations tab at the top and this will show all the integrations that are available to you.
In this example, I'll be configuring Notifications over email. As I mentioned earlier, we also support notifications over Slack and Microsoft Teams. If you're interested in either of these I've placed some documentation showing how to set these up in the write up down below.
The first step is to click on the email option in this list of integrations.
And then we can set up our email subscription by clicking Add Subscription.
After clicking add subscription we can then configure our notifications. We're able to specify the apps we want to be notified about as well as the branches. For now I've left these as the default and we'll be notified for all apps and all branches.
I've also entered in my email and specified that I want to send notifications every single time a batch completes regardless of the status. That's it for step one! All that's left is to make a quick change in our code and we'll be ready to go.
The only code change we need to make is adding this call to setNotifyOnCompletion to our batchInfo object and passing in true. As this method indicates, we're toggling a flag that tells the Applitools server to forward a notification when the batch completes.
One thing to note, is that notifications are sent only when the batch gets marked as closed. When a test is running the batch is considered open until we make a call to runner.getAllTestResults(). This method, among other things, will close our batch and trigger the notifications to be sent.
There are also some other options that allow you to close out the batch and I've placed documentation on this in the write up down below.
Let's go ahead and run this test and then check out the notification sent to our email!
As we can see, we got notified that our batch has completed with 3 tests failed and 2 tests passed. If we click on the button at the bottom it will take us directly to this batch on our dashboard.
Batch Notifications are perfect for when you have a job that runs at regular intervals through a CI and allows you to be notified when that job completes. You can also configure the notifications to be sent only when you have unresolved tests. Then, you only need to check out the results on your dashboard when there's something that needs your attention.
In this chapter, we learned about some advanced features related to batching. We saw how we can track aggregate data about our batches performance over time by using a batchSequence name and then reviewing that data in the insights page.
We also set up Batch notifications over email by configuring our admin panel and then making a small change to our code. These notifications are great for getting alerted about the completion of a batch that was run by somebody else or a job that runs on a regular interval.
As always, please reach out to us on the support channels mentioned earlier if you have any questions about the topics covered in this chapter. Otherwise, I'll see you in the next chapter!
Resources
- Setting up Slack Notifications: https://applitools.com/docs/features/batch-completion-slack-integration.html
- Setting up Teams Notifications: https://applitools.com/docs/features/batch-completion-notifications-microsoft-teams-integration.html
- Close Batch Alternative: https://applitools.com/docs/api-ref/sdk-api/selenium/java/batchclose
