Transcripted Summary
Now, we'll take a look at the advantages cloud infrastructure brings to speeding up our automated tests in our pipelines.
The State of DevOps Survey reports that using cloud infrastructure and containers is highly correlated to elite performing teams.
This technology helps us speed up feedback from automated tests. For example, we can run each test in our UI suite concurrently in its own container so that our test suite is only as long as our longest test. That means the test suite that maybe used to take 10 or 15 minutes, maybe takes 2 or 3 minutes.
It could be even faster.
Some team members at Blackboard documented their experience taking their UI test suites from an hour or more, within a year, they had it down to 16 minutes. And then in another month they had it down to 39 seconds. That is fast feedback.
This is all using the cloud infrastructure we have available today.
Think about what's slowing your testing down in your automated tests.
This lists some of the infrastructure problems I run into over the years.
We can't get a good build to deploy and test or we don't have any test environments available that we could deploy to. Virtual machines, when they came along helped a lot, but our test environments were still limited. As we were trying to create automated tests to run in our continuous integration, we didn't have any place to even do that work a lot of the time.
There were problems with getting the right kind of test data.
On my last team when we implemented Pivotal Cloud Foundry with Amazon Web Services and later Google Cloud Platform, we went from having a few test environments to dozens.
We were using GitFlow so that we were creating story branches for each story. Each commit spent up a cloud test environment for that particular branch where the automated test ran and then when the pipeline was finished and successful, we could do manual exploratory testing or security testing or anything else we needed.
Within a day, we were done with our testing and we could merge the pull request and delete that environment. Our master build automatically deployed to our permanent more production-like test environments.
We had so much flexibility and this is what enabled that team to move to continuous delivery.
Tools such as Terraform make it really easy to provision cloud infrastructure for both test and production environments. It's all version controlled. It's reliable. It's transparent. It's just like our code.
The State of DevOps survey showed teams using infrastructure as code are 1.8 times more likely to be elite performers.
This is definitely a direction that your team needs to consider going if you're going to embrace DevOps.
As I said, virtual machines back in the day were a big step forward, but containers are an even bigger one. This technology is accessible to everyone and it's affordable.
After getting tests in the cloud, the speed is just unbelievable.
From the Blackboard example, it wasn't expensive to do. It didn't cost much money or time to maintain. They implemented their Amazon Web Services Lambda in a day, and it let them execute their UI tests and parallel on a massive scale.
This is something that is possible for all of our teams.
We have to be careful when using the cloud.
If you use it improperly, you could have some unexpected costs. But generally, you can save a lot, both time and money.
Running tests in the cloud versus virtual machines is the same as running their apps in the clouds versus virtual machines â you get the same advantages.
The cloud provides an ability to easily wipe clean your environments and you can run cost effectively by only paying for the computers you're using.
Do watch out for maintaining long running environments. My previous team fell into the trap of doing feature branches and those ended up being around a lot longer than we thought they would be, and those cloud-based environments got dirty. They got kind of corrupted. The tests would no longer run reliably.
It's better to keep everything fresh running off master, from my experience.
It's easy to spin up a test environment for a given build, a story, or a feature branch. It's also easy to refresh a permanent test environment. On my last team, we had a test environment that was automatically refreshed with each new build or master, but we had 2 other permanent test environments where we can kick off a pipeline to deploy whatever build we wanted and we had scripts to refresh the databases quickly.
We could also deploy a test locally, so this gave us a lot more flexibility as we were creating our automated tests and testing them in anticipation of adding them to our pipeline. The modern tools give us all this flexibility.
Test data can still be a sore point. You want it to be realistic, but you want it to be quick to refresh and these days with privacy concerns we can't really use customer data. This is something again the diverse experience and skillsets on your team can solve problems like this.
DevOps uses a lot of infrastructure as code and code has to be tested.
Talk to your team about testing your infrastructure, such as your pipeline configuration, your deployment scripts, your failover testing.
We're going to talk more about monitoring and production in the next chapter.
Think about how you can help your team improve your infrastructure so that you can get faster feedback from automated tests.
This often makes the difference between being able to implement continuous delivery and not being able to do that.
If you're a tester, there are a lot of ways you can add value even if you're not an expert at configuring and maintaining the infrastructure. Testers, we're good at asking questions, we're good at spotting patterns and bringing up problems, and we always want to make problems a problem for the whole team to address.
We always want to be on the lookout for risks and looking for ways to improve.
There are some resources for more in-depth learning about infrastructure in the resources section for this chapter.
In the next chapter, we'll look at tools to help us learn from production use.
Resources
A Practical Guide to Testing in DevOps by Katrina Clokie (âPlatform Evolutionâ sectionâ)
âUI Testing at Scale with AWS Lambdaâ by Wes Couch and Kurt Waechter
âInfrastructure as Code: A Reason to Smileâ by Jafari Sitakange, Thoughtworks
More Agile Testing: Learning Journeys for the Whole Team by (Chapter 23) Testing in DevOps by Janet Gregory and Lisa Crispin â Chapter 23 pdf
