Transcripted Summary
Let's take a look at the application under test that we'll be using for for this course.
Note:
To view this application locally, get it at github under the website directory: Automation Bookstore Web App*
This is a single-page application. It represents an automation bookstore containing some of my favorite books. The way that it works is you can filter down these results based on the title or author of the book.
For example, let's say that we wanted to search for "Jason Arbon". We could simply type in his name, and we'll be shown the books that he's authored.
Likewise, we can search for the title of a book. So let's search for "Agile Testing", and that will narrow it down to this.
You could also do a substring. If we put in “Test” we'll see all of the books that have "Test" in either the title or the author field.
That's a quick overview of how the application works. Let's go to the automation framework.
Here we have a test that searches for a book by its full title and makes sure it is returned:
SearchTests.java
@Test
public void testSearchByFullTitle(){
String title = "Agile Testing";
page.search(title);
assertTrue("Book not found: " + title, page.isBookVisible(title));
}
# Let’s take a look at what this code does.
- We specify the title of the book is "Agile Testing"
- We search for this title
- We assert that the book with this title is visible
We could run this test, and it works. However, it's missing some key things.
# Let's take a closer look at the isBookVisible method and what it does.
public boolean isBookVisible(String title){
List<WebElement> books = findVisibleBooks();
for(WebElement book : books) {
if(title.equalsIgnoreCase(book.findElement(titleAttribute).getText())){
return true;
}
}
return false;
}
# Looking at the code, we notice that:
- The test finds all of the visible books on the page and then traverses through those books.
- If any of the books have a title attribute that matches the title that we passed into this method, we know that the book is present, and we exit.
- Otherwise, if we've gone through all of the visible books and have not found it, then we know that the book is not there.
# Let’s take a look at the actual application and do a search for "Agile Testing".
In our code, the only thing we're verifying is that one of the visible books has this title. Realistically, even if the search didn't work and the books were not filtered at all, our test would still pass because the book would still be visible amongst all of the other visible books.
So, this code is not really testing what we’re validating when we test this manually.
Also, there are other components to this book that are not being verified such as the price, authors, and book cover, which are really important aspects.
A lot of this can be covered with functional assertions, but we'll need to add a lot more to our testing framework - both in the framework as well as the test. Also, even if we added all of the functional assertions, we still wouldn't be able to make sure that the book cover is correct without visual validation.
WIth this example, we can see where visual validation is powerful and can really help us.
Note:
With visual testing, we can add one single assertion that would make sure that this is the only book that's present, that everything about the book is correct, and that there are no other exceptions or errors shown on the screen.
Instead of using the one weak assertion we had previously:
SearchTests.java
@Test
public void testSearchByFullTitle(){
String title = "Agile Testing";
page.search(title);
assertTrue("Book not found: " + title, page.isBookVisible(title));
}
We could add visual testing instead which will verify not only the title, but other components of the book as well as everything else about the screen.
Let's add the common elements of the visual testing code to the BaseTests.java file, so that we can use it in any of our tests later.
BaseTests.java
public void validateWindow(){
eyes.open(driver, "Automation Bookstore", Thread.currentThread().getStackTrace()[2].getMethodName());
eyes.checkWindow();
eyes.close();
}
# Let's take a look at what this code does:
Adds a method that validates the window
The
openmethod says to Applitools to "go ahead and get ready to start verifying."This code takes the Selenium WebDriver and also takes the name of the app under test, which we know is "Automation Bookstore." Finally, it takes the test name. The app name and the test name work together to specify a unique name for the test in our Applitools Test Manager dashboard, which we'll see later.
For the test name argument, since I have this in this common place, I don't know what the test name is going to be at this point. I can utilize the current thread to get the test's name at runtime. This code will get the name of whichever method is calling it, so that'll get the test method name. Optionally, you could always just add a parameter, and have your test pass its name in.
Adds a call to
checkWindowwhich will check the entire visible screen and everything on it to make sure that it looks exactly like we want it to.Finally, after the check, we need to add ode to close the session, aggregate any of the visual checks that we've specified, and produce a result for us.
This is enough to handle the test case, but there is another command that you can optionally add to your framework just to make sure that Eyes never hangs, for example, in the instance that there’s a test that doesn’t use the common validateWindow method and instead tries to open Eyes on its own and forgets to close it.
@AfterClass
public static void tearDown() {
driver.quit();
eyesManager.abort();
}
We can now go back to our test and replace the original assertion with a call to validateWindow and it will validate the entire screen.
The updated test now becomes:
SearchTests.java
@Test
public void testSearchByFullTitle(){
page.search("Agile Testing");
validateWindow();
}
When we run the test, the Bookstore app will open and do the specified search. Eyes will take a screenshot and send it to the Applitools cloud. They will validate the image using an AI comparison. Then, we will get our result.
After running this test, let’s go to the Applitools Test Manager dashboard.
The dashboard will show all of our visual tests and the results. As the image shows, we have our new test, indicated by the word “New”. By default, the baseline image for this test is captured and stored. So, we don't have to manually specify this. It's already captured.
When we open the test results, we can see the baseline image that is captured. Since this is the first time we have run this test, we want to make sure that the results are exactly what we expect. For the "Agile Testing" search test:
- we have only the one book returned
- the cover looks right
- the authors look right
- the price looks right
- the title is right
- there are no other exceptions
I'm happy with this so I can back out, and it will save this as the baseline image for future tests.
Let's run the test again so that we can see how it will compare the baseline image with the new image that is captured.
Notice that we have a new entry for the testSearchByFullTitle test; this time indicating that it's Passed. If we look at this, notice we'll see the baseline image and we'll see the new image. There was comparison done, they're the same, so everything is great. The test passed.
Now I'm going to change this test so that it fails because I want you to see how visual validation works in error cases.
I'm going to change our test to search for "Test" instead of "Agile Testing".
@Test
public void testSearchByFullTitle() {
page.search(“Test”);
validateWindow();
}
We already know that when we search for “Test,” the results window is going to look different.
There are five books instead of just the one. But we already have a baseline of just the Agile Testing book shown, so this visual test should fail. Let's go ahead and run it and look at the results.
The test fails within our project, and this is printed to the console:
com.applitools.eyes.exceptions.DiffsFoundException: Test 'testSearchByFullTitle' of 'Automation Bookstore' detected differences!
The DiffsFoundException indicates that there are visual differences, and it gives us a URL to review the differents on the dashboard.
Notice that it says "Unresolved", not "Failed." This allows for you, the tester, to review this yourself and determine if it's a failure or if the baseline just needs to be updated.
Let’s take a look at the details of the test result.
Not only does it show that the books are different, but also that the search was different: “Test” instead of “Agile Testing.”
Let’s say that you wanted to change your test in this way, with this being the correct result. You would simply press the “thumbs up” button and it would change your baseline. This lets Applitools know that this is now the new image that all subsequent tests should be compared against.
If this is not the case, you can simply fail it by clicking the “thumbs down” button and press "Save". And now, notice the test is marked as "Failed".
