Transcripted Summary
In this chapter, we are going to take a look at how we work with JSON, as well as use JSONPath to parse our JSON responses.
JSON is a very popular data format these days and most of the APIs use it as the request and the response structure.
It is also very useful for many test automation purposes, wherein you can use it as a data format to store your test data, even configs in some cases and whatnot, so it's very important to get a grasp of that.
If you're completely new to JSON, don't worry.
There are a bunch of articles that you can go through to form an understanding.
I've also written a blog on the same chapter that we are going to discuss today on my site.
You can take a look at JSON on the json.org website, or the Mozilla website, or even W3 schools.
All of these are a good resource and I'll provide a link to them in this chapter's resources.
For those of you who are very new to JSON, I'll just give a quick summary.
JSON is mostly a key-value data store and has support for some primitive data types like string, number, boolean, and also an array type.
It looks very similar to Python dictionaries and if that's something that you have worked with, then you already are familiar with how JSON would look.
Before we proceed, let's understand a couple of terms that come up quite frequently.
The process of converting a Python object to JSON is called a serialization and the reverse process of converting a JSON to a Python object is called deserialization.
JSON support in Python comes out of the box in the standard library and there is a JSON module that we can make use of.
Let's understand some common methods that this module offers before proceeding further.
All right, so there are a few methods that we need to know before proceeding further.
If you want to convert a Python dictionary to JSON, then you make use of dump or dumps.
The key difference between the two is dump can be used if you want to write directly to a file, but if you want to write to a Python string, you can make use of dumps.
Similarly, if you want to convert a JSON back to a Python dictionary, you make use of load if you want to read it directly from a file or loads if you want to read from a Python string.
Don't worry if these terms and these methods are a bit ambiguous, you'll get a hang of it as we go through.
Let's understand the typical test structure that we might have.
We might want to store our request body in a JSON, read it in our test, manipulate the JSON payload so that our requests could be different based on different cases, and finally, make the request and then verify that a person is created in our people-api.
We have seen this test created already, but we'll see how to work with JSON files and all of the above methods that we mentioned.
# Understanding a typical test structure
Coming back to the project that we have been working with so far, I have already taken the liberty of adding these files as well as these classes.
Let's go through and understand how they actually work, what their responsibilities are and whatnot.
Coming back to the project that we have been working with so far, let's make sure we check out the correct branch example/05_working_with_json.
Now that we have checked out our branch, let's see what the changes are.
If you notice under the tests directory, I have created a create_person.json file.
This should seem quite familiar to you as we use this to create a person in our persons database.
Let's take a look at the people_test.py file now.
I've added a new test called test_person_can_be_added_with_a_json_template and there are a couple of important things to note.
We are passing something called create_data.
If you go up, you can see that create_data is a @pytest.fixture, which is used to set up our person with whatever required characteristics we need.
This is a very good pattern to follow because you can separate your setup as well as your teardown logic in these pytest fixtures.
Here you can see that these lines of code are something that I'm doing to set up and finally yielding the value to the test function.
@pytest.fixture
def create_data():
payload = read_file('create_person.json')
random_no = random.randint(0, 1000)
last_name = f'Olabini{random_no}'
payload['lname'] = last_name
yield payload
Anything that you mention after the yield keyword is going to get executed when the test finishes its execution.
Let's see how this is structured.
We are reading a file called create_person.json with the function read_file.
Let's see what this function looks like.
import json
from pathlib import Path
BASE_PATH = Path.cwd().joinpath('..', 'tests', 'data')
def read_file(file_name):
path = get_file_with_json_extension(file_name)
with path.open(mode='r') as f:
return json.load(f)
def get_file_with_json_extension(file_name):
if '.json' in file_name:
path = BASE_PATH.joinpath(file_name)
else:
path = BASE_PATH.joinpath(f'{file_name}.json')
return path
I've introduced a file reader utility in the framework that we have, which has a very simple read_file method which expects a file name.
We have a function named get_file_with_json_extension which takes a file_name and sees if JSON is already present in the file_name.
If the file_name already has .json, then we are combining it with the BASE_PATH that we computed above and added the file name.
If not, we are adding the .json extension and then getting the path.
Here, path is basically coming from the pathlib module, wherein we are making use of get current working directory (.cwd()) and then joining the path to the actual data directory.
Making use of pathlib ensures that all the paths that you create are cross-platform and can work very easily.
Also, notice instead of using the open method directly, we are calling it on the path that we have received.
This is a very useful pattern because it ensures that open is able to understand the file path quite easily.
Finally, when we get a handle to the file in f, we are making use of json.load because we want to directly read from this file, and then return a Python object.
Now that you have your JSON request body in a Python object format back in create_data() in people_test.py, we are just getting a random number between 0 and 1000 and appending it to a prefix that we thought of, just to ensure that the username is always unique, though this value 1000 could be sufficiently high to avoid if you create more than a thousand users.
That's a problem for another day. We update it in the payload and finally return it, right?
We have made a couple of additional changes in the create_person method as well.
def create_person_with_unique_last_name(body=None):
if body is None:
unique_last_name = f'User {str(uuid4())}'
payload = dumps({
'fname': 'New',
'lname': unique_last_name
})
else:
unique_last_name = body['lname']
payload = dumps(body)
It now takes a body and if the body is not available, it still goes about the old business of generating the UUID and makes use of the dumps method to get a JSON string that we can use in our request body.
However, if the body is passed from outside, it will make use of that to get the request body.
So, this function create_person_with_unique_last_name should take care of making our POST request and inserting a person into the DB.
Now, we want to repeat the same assertion of whether the person that we just created is present in the database or not.
def test_person_can_be_added_with_a_json_template(create_data):
create_person_with_unique_last_name(create_data)
response = requests.get(BASE_URI)
peoples = loads(response.text)
After getting the response from the get API, we are making use of the loads method, which expects a JSON string to be passed, which we can get using the response.text method.
This is different from the way we did it earlier, wherein we directly called the .json method to get the JSON string, but showcases how you can actually use the loads method.
# Using JSONPath
Finally, now that we have a JSON string at hand, let's see how we can make use of JSONPath to parse this and use it for our test purposes.
We are going to make use of jsonpath-ng, which is a useful library to work with JSON responses, or requests for that matter, and gives us XMLPath-like capabilities.
If you have not already installed it, you can make use of pipenv install and install this on your environment, or if you're following along with this project, then you will find it already in the Pipfile.
A good way to start with JSONPath is to go through the PyPI documentation that the original author has created, but we'll take a simple example for our use case.
Let's say we want to do the same thing, wherein we want to get all the last names in the response to our JSON.
Previously, we parsed it manually and while that is a good idea for shorter JSONs, when your JSON structure grows and there is a heavy bit of nesting, then using an approach like JSONPath could be quite useful.
Observe that I'm making use of the parse method and I'm passing it a JSONPath expression.
jsonpath_expr = parse("$.[*].lname")
Here is the response we would get if you were to make a "Read all" request, as we are doing in this case.
We basically get an array of objects, wherein every object can have lname, or fname, or other properties.
So here, $.[*].lname, the dollar specifies that starting from the root, I want - for all the elements in the array - using this bracket and star notation - to get the lname value.
This will give me a JSONPath expression.
However, this is not evaluated right then and there.
To actually evaluate, we will use jsonpath_expr.find and give it a JSON string in the input parameters.
result = [match.value for match in jsonpath_expr.find(peoples)]
This is going to basically give us a list of last names.
What we are doing here is for every match that we found using this, we are getting the values.
Finally, we are asserting that the result list, which should have a list of all the last names, should contain our expected last name that we just created.
So, we can verify that the result is giving us the expected response using just the print command, so let's add that.
def test_person_can_be_added_with_a_json_template(create_data):
create_person_with_unique_last_name(create_data)
response = requests.get(BASE_URI)
peoples = loads(response.text)
# Get all last names for any object in the root array
# Here $ = root, [*] represents any element in the array
# Read full syntax: https://pypi.org/project/jsonpath-ng/
jsonpath_expr = parse("$.[*].lname")
result = [match.value for match in jsonpath_expr.find(peoples)]
print(result)
Let's run this test.
As we can see, the test passed and here is the list of persons in the entire database at this point of time.
If you want to understand how JSONPath expressions can be written, you can check out jsonpath_ng.
The maintainer has given good documentation on how to work with this library, as well as how the JSONPath's syntax looks.
It also gives a nice programmatic way to create JSONPaths so choose whatever works best for you, but the capability is there.
In this chapter, we learned how to work with JSON for our requests as well as the responses, and how to pass them using JSONPath library.
I hope this is useful and gives you an idea of how to work with these components in your framework.
That's it for this chapter, I'll see you in the next one.
