Chapter 1 - AI for Element Selection

AI for Element Selection: Erasing the Pain of Fragile Test Scripts

Chapter 3 - Training Your Own Element Classifier

Transcripted Summary

Hi, my name is Jason Arbon and I want to talk about how to use AI selectors for Appium. How do you get Appium to use AI and machine learning to find elements in your app for testing?

Let's look at an app like Walmart. Let's say we want to get Appium to find the shopping cart icon and click on it. Today, you have to write something that looks like this.

driver.find_element_by_xpath('//*[@id="mainbar"]/div[1]/div/div[1]/div[3]/img')

But these types of selectors are very brittle because as the application changes, you have to go in and re-edit these magic strings.

So how do we get AI to do this? Well, we start with a neural network. An empty neural network that's not really trained on anything too complicated. Then we show it a thousands of shopping carts from different apps and it learns and generalizes what a shopping cart looks like. Just like you, as a human, can see a shopping cart you've never seen before and you're still able to recognize it as a shopping cart.

Next, we essentially tell the API plugin for AI for Appium: "Hey, find a shopping cart on this page." And it looks at the screen just like your eyes do, and it finds something that looks like a shopping cart and returns the pointer back to you.

Now, I'm nowhere near as articulate as Jonathan Lipps, or as competent on this console, so I'll defer to him to kind of walk you through it.

(Narrator swithches to Jonathan Lipps)

This is Jonathan Lipps. I work on Appium and run an Appium consulting firm called Cloud Gray. For the last little while, I've been working with Jason on bringing test.ai's technology into Appium so that Appium users can benefit from their work for free.

What's presented in this course is a demonstration of what we've been able to do together. And what's cool is that everything I'm demonstrating is actually open source and already merged into Appium so you can use it today with the latest Appium Beta.

A word of warning though, it's all very early and experimental so it's likely you will run into issues, but it hopefully demonstrates that Jason's lofty vision of AI for testing can actually meet your reality. In this case with an Appium integration.

There's new kind of plugin for Appium, an element finding plugin that's powered by test.ai's machine learning model.

The basic idea is that by using this plugin, you can find an icon element by its semantic label. For example, you could find a shopping cart icon using the label cart, or an arrow button using the label arrow. It's a pretty cool idea.

So, let's see how it works. Here I've got an android emulator with a shopping app on it. In this case, Walmart. Like any good shopping app, you'll see that this app has a little cart icon that we can use to go to our cart and see what we've decided we want to purchase.

What we're going to do is use the new element finding plugin from test.ai to tap on this icon without needing to inspect our app hierarchy, defined selectors and without doing any kind of image processing ourselves. The end result will look like tapping on the cart and seeing our estimated total.

Okay, so let's load up our test. Here's the test code that I'm going to run.

const wd = require('wd');
const B = require('bluebird');

const APPIUM = "http://localhost:4723/wd/hub";

const ANDROID_CAPS = {
  platformName: 'Android',
  deviceName: 'Android Emulator',
  automationName: 'UiAutomator2',
  noReset: true,
  appPackage: 'com.walmart.android',
  appActivity: '.app.main.MainActivity',
  customFindModules: {'ai': 'test-ai-classifier'},
  shouldUseCompactResponses: false,
};

describe('Finding an Android element with machine learning magic', function () {
  let driver;

  before(async function () {
    driver = wd.promiseChainRemote(APPIUM);
    await driver.init(ANDROID_CAPS);
    await driver.setImplicitWaitTimeout(20000);
  });

  after(async function () {
    if (driver) {
      await driver.quit();
    }
  });

  it('should find the cart button', async function () {
    await driver.elementByAccessibilityId('Open navigation drawer');
    await driver.elementByCustom('ai:cart').click();
    await B.delay(6000); // for effect
  });
});

It's written in JavaSript using Mocha as the test runner. It's all straightforward Appium stuff, capabilities, before and after drivers, all that kind of stuff. Except there's two important new bits.

The first is a capability customFindModules. This is where we specify the plugin that I'm using called test-ai-classifier. I've already installed that in the appropriate place.
```
customFindModules: {'ai': 'test-ai-classifier'}
```
The second is a new find element command, elementByCustom, here in this library. This lets us find an element using the customElementFinding plugin that we registered above using the name that we registered it with, in this case ai. So, this command says we're looking for any cart elements using the ai element finding plugin.
```
await driver.elementByCustom('ai:cart').click();
```

When we run this test, we can see that Appium is able to successfully find the cart on the page just by us telling it what kind of thing it represented - in this case, a cart icon.

But this doesn't just work on Android, it works on IOS too.

So that's the basic idea. There's still a lot of work to be done. There are a lot of useful cases the machine learning model doesn't yet handle, for example, icons that are mixed with text, and of course, there's a lot of room for performance improvement.

But I'm super excited to be working with Jason to get some of this next generation technology into Appium and into all of your hands. So, don't forget to take the time to thank Jason and test.ai for opening up their machine learning model and giving us the ability to build this Appium integration. Back to you Jason.

(Narrator switches to Jason Arbon)

So, all of this is open source. The models are open source.

You can grab the plugin from Github: Test.ai Classifier Plugin for Appium

Just install it and then drop the files into the right folder in Appium.

Also note that Jonathan's work makes Appium extensible. So, if you have your own kind of a way to do classification or your own machine learning methods, or you want to take a variation of a riff on this one, you can do it. Play with it, and you can add it to Appium. You can customize Appium for elements selection.

And the rumor is there's work underway for Selenium, but more on that to come.

Resources

Add AI to your existing test scripts in minutes!