Transcripted Summary
In this chapter, I'd like to show you how you can train your own element classifier for your application-specific features.
We'll cover
- How to add new training data
- How much training data to add
- How to rebuild and retrain the model
- How to integrate this back into the Appium plugin and Appium itself
Our open source classifier on Github has about a hundred different types of common elements: menu buttons, shopping carts, search text boxes, those kinds of things.
But you will probably have things that are very specific to your app or just aren't in our list of elements and haven't been trained. So, if you want to add your own classifiers for things specific in your application, this is how you do it.
Download the project, like we showed the last session. Then save images of the element from your own app and as many pictures of things that look like what you're trying to train your AI on.
So say you have something with like three concentric circles in it. Get lots of pictures of things with three concentric circles. A good way to do that is go up to Google Image Search even, and just search by image. Then download a bunch of similar-looking images, and then drop them in one of these folders. In this case, we create a new folder in the project called Circles. You just put that in there, and you drop all these pictures of the thing that looks like the element in your app that you want the AI to recognize.
Then you go back, and you do what we did in the last session, which was go back and execute the retrain.py, pointing to the old sets of folders again. And then you can run the classifier, and your classifier, after it's been trained, if you run run_model.py, with a picture of your image, say those concentric circles in your app, it will recognize them.
If you want, you can also take those, rebuild that model, you can plug it back in to the test.ai plug-in up on Github, or just replace the model files with the ones you generated here. Then that can go back into Appium, so now Appium can use AI to classify and find the element selector, for example it will find the circles in your app.
So, there you go, that's how you can customize this stuff for your own application.
And again, if you can do this you're kind of a Machine Learning Engineer, so go ahead and change your title. 😉
Resources
- Classifier Builder
- Appium Classifier Plugin on Github
- List of Identifiers
- Classifier Builder Retrain
- Appium Plugin Architecture Doc - Element Finding Plugins
