Transcripted Summary
In this chapter, we'll write end-to-end tests that will eventually run in Azure Pipelines.
But before going down that road, let's understand why we should run tests in CI/CD pipelines?
Like we're continuously integrating code changes and continuously deploying the changes to various environments, we also need to continuously test our code.
Adding tests to CI/CD pipelines let us test continuously.
Continuous testing is basically running a repeatable set of tests whenever we make a commit to the code repository.
Continuous testing has a number of benefits.
It lets CI/CD pipelines provide fast feedback that the changes will not impact the product negatively.
Having fast feedback lets the team fix issues early in development and integration, which is faster, easier, and cheaper than fixing them later.
It gives you the confidence to move from continuous delivery to continuous deployment - that is, from deploying to production manually to deploying to production automatically.
Now that we're convinced continuous testing is the way to go, be aware that the path to achieving continuous testing has its own challenges.
Some common challenges include:
- Scalability issues - we do not want running end-to-end tests to be a bottleneck in the CI/CD pipeline. We desire fast execution and the ability to run tests in parallel. Selecting the right tool and framework is essential.
- Test data management - creating and maintaining multiple dataset versions for different environments and tests is challenging.
- Testing scope - as the product grows, so does the number of tests. Running tests may take hours, and so quick feedback isn't achieved. On the other hand, if you run only a subset of the test, then we may be compromising on the product quality. So choosing the right test and the right number of tests is essential.
All right, let's go ahead and write some tests.
We will be using Cypress as a test framework and we'll be writing some visual tests using Applitools.
If you're new to Cypress or Applitools, you may want to explore these amazing tools.
I've attached some resources that can help you get started.
I've cloned the GitHub repo to get a local copy of the project to work on.
Let's create a new folder to hold our test.
We'll call it e2e.
Let's now install Cypress.
We'll go to our test folder and install Cypress.
cd e2e
npm init -y
npm install cypress
npx cypress open
So Cypress has been installed.
We'll go ahead and remove the examples folder and add a new file search.spec.js to hold our test, and we'll add the test.
describe('Search for books', () => {
beforeEach(() => {
cy.visit('http://abookstore.azurewebsites.net/')
})
it('should return one book with title Agile Testing', () => {
const title = 'Agile Testing'
const expectedCount = 1
// Search
cy.get('#searchBar').type(title, { delay: 250 })
// Verify count
cy.get('li:not(.ui-screen-hidden)').should('have.length', expectedCount, `There should be exactly ${expectedCount} book(s) visible`)
// Verify book title
cy.get('h2').should('contain.text', title, `${title} should be visible`)
})
it('should return multiple books with title Test', () => {
const title = 'Test'
const expectedBooks = [
'Test Automation in the Real World',
'Experiences of Test Automation',
'Agile Testing',
'How Google Tests Software',
'Java For Testers']
const expectedCount = expectedBooks.length
// Search
cy.get('#searchBar').type(title, { delay: 250 })
// Verify count
cy.get('li:not(.ui-screen-hidden)').should('have.length', expectedCount, `There should be exactly ${expectedCount} book(s) visible`)
// Verify book titles
expectedBooks.forEach(b => cy.get('h2').should('contain.text', b, `${b} should be visible`))
})
})
Here, I have added two tests; the beforeEach() hook will launch our application.
The first test is to search for Agile Testing, and we expect to get back one book with title, "Agile Testing".
it('should return one book with title Agile Testing', () => {
const title = 'Agile Testing'
const expectedCount = 1
// Search
cy.get('#searchBar').type(title, { delay: 250 })
// Verify count
cy.get('li:not(.ui-screen-hidden)').should('have.length', expectedCount, `There should be exactly ${expectedCount} book(s) visible`)
// Verify book title
cy.get('h2').should('contain.text', title, `${title} should be visible`)
})
The second test is to search for Test, and these are the book titles that we expect to get back.
it('should return multiple books with title Test', () => {
const title = 'Test'
const expectedBooks = [
'Test Automation in the Real World',
'Experiences of Test Automation',
'Agile Testing',
'How Google Tests Software',
'Java For Testers']
const expectedCount = expectedBooks.length
// Search
cy.get('#searchBar').type(title, { delay: 250 })
// Verify count
cy.get('li:not(.ui-screen-hidden)').should('have.length', expectedCount, `There should be exactly ${expectedCount} book(s) visible`)
// Verify book titles
expectedBooks.forEach(b => cy.get('h2').should('contain.text', b, `${b} should be visible`))
})
Also, we'll verify the count of the books returned.
Now, let's add some visual tests to our project.
We'll be using Applitools Eyes.
Applitools Eyes is one of the products from Applitools that enables visual testing powered by Visual AI.
It isn't the old style of pixel by pixel comparison of images.
It says, "Tests infused with Visual AI are created 5.8x faster, run 3.8x more stable, and catch 45% more bugs versus traditional functional testing."
These visual tests are powerful.
Applitools Eyes integrates with all popular automation test frameworks like Selenium, Cypress, WebDriverIO, and more.
You can check out this page for more amazing features of Applitools Eyes.
Let's go back to VS Code and we'll add visual tests.
The first thing we need to do is install Eyes-Cypress.
npm install @applitools/eyes-cypress
To automatically configure the project to work with Applitools Eyes, we need to run this command:
npx eyes-setup
Now, in order to authenticate using the Applitools server, we need an Applitools API key.
To get the Applitools API key, you need to create an account on eyes.applitools.com.
Once you create an account and sign in, you can get the Applitools API key by clicking on the user icon and then "My API key".
Copy your API key.
Now, we'll create a file under the e2e folder - applitools.config.js.
This file will hold the Applitools configuration information.
module.exports = {
apiKey: process.env.APPLITOOLS_API_KEY,
appName:'Automation Bookstore'
}
So, here we have specified the apiKey and the appName.
Now, you could provide the API key right here, but the Applitools API key is considered a secret and we'll be pushing this code to GitHub so the secret shouldn't be there.
So, instead of providing the API key right here, we'll store it in Azure DevOps.
Our project is all set to use Applitools Eyes.
Let's create a new spec file search_visual.spec.js and add the visual test.
Let me walk you through the code.
The beforeEach hook will launch the application and we have a method eyesOpen() - this method initializes a test session.
Then in the afterEach loop, we are calling the eyesClose() method, which ends the test session and lets the server know it should display the results.
So here again, we have two tests.
In the first test, we are searching for 'Agile Testing', but notice we have no validations.
Instead, we have the eyesCheckWindow method, which will check the entire browser window.
And similarly, we have a second test where we are searching for 'Test'.
Now in our traditional test, we have two assertions - one for the count of the books and the other for the title.
If those two assertions pass, we assume all is good, but there's so much that could still go wrong - like the price could be inaccurate, the book placement could be wrong, the title could be running outside the book, and so many more things.
It is next to impossible to have assertions for every possible scenario, but these issues could easily be captured by Applitools Eyes.
And so, in addition to our traditional tests, having visual tests in the project is important.
All right, let's run a test now.
For that, in the package.json we will update the test script:
{
"name": "e2e",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "cypress run"
},
"keywords": [],
"author": "Bushra Alam",
"license": "ISC",
"dependencies": {
"@applitools/eyes-cypress": "^3.22.0",
"cypress": "^7.5.0"
}
}
Let's add the Applitools API key in applitools.config.js.
Now to run a test, we need to run the command npm test.
And also, note that we are running this command from within our e2e folder because that is where our package.json lies.
All our tests are passing and if you go to the Applitools dashboard, let's refresh it.
So these are our two test results.
This is the first one, and hence, it is marked as passed because there isn't a baseline available for it.
For the future runs, this will act as the baseline until you update it explicitly.
All right, let's push our code to GitHub.
But before that, let's remove the API key.
And also, we'll add the node_modules folder and screenshots and videos folder to get .gitignore.
e2e/node_modules/
e2e/cypress/screenshots
e2e/cypress/videos
Because CI/CD triggers are enabled, I expect our pipeline to run.
In the next chapter, we'll add steps to run these tests in our release pipeline. I'll see you there.
