Warning!
This course was created with an older version of pytest-bdd.
Over the years, updates to the framework introduced backwards-incompatible changes.
The example project code and transcripts are updated, but the videos still show the old code.
Please review the pytest-bdd docs to see
specific syntax changes in newer versions.
Transcripted Summary
Our example test project now has many tests in it.
We could run them all at once using the pytest command; but if we were to continue adding more and more tests, that would take a lot more time. It's not unusual for test suites to take minutes to hours and possibly even days to run to completion.
So, how can we filter our tests to run only specific ones that we want to do?
One of the easiest answers is tags, which will be covering in this chapter.
I added tags to the feature files already in our test project.
If we look at cucumbers.feature I've put a tag at the Feature level called @cucumber-basket.
# cucumbers.feature
@cucumber-basket
Feature: Cucumber Basket
As a gardener,
I want to carry cucumbers in a basket,
So that I don't drop them all.
@add
Scenario Outline: Add cucumbers to a basket
Given the basket has "<initial>" cucumbers
When "<some>" cucumbers are added to the basket
Then the basket contains "<total>" cucumbers
Examples:
| initial | some | total |
| 0 | 3 | 3 |
| 2 | 4 | 6 |
| 5 | 5 | 10 |
@remove
Scenario Outline: Remove cucumbers from the basket
Given the basket has "<initial>" cucumbers
When "<some>" cucumbers are removed from the basket
Then the basket contains "<total>" cucumbers
Examples:
| initial | some | total |
| 8 | 3 | 5 |
| 10 | 4 | 6 |
| 7 | 0 | 7 |
I've also added two tags — @add and @remove — for the Add and Remove test scenarios.
Tags can be applied either to features, or to scenarios, or scenario outlines.
The format of a tag is fairly basic — it's prefixed by an @ symbol, and then it's a bunch of alphanumeric or dashes or underscore characters.
The key thing is that is an identifier, that it is one token.
A scenario or a feature can have an unlimited number of tags, and tags can be reused by different features and scenarios as well.
If we look at the other feature files:
To the
service.feature, I've added the 2 tags:@serviceand@duckduckgoAnd to the
web.feature, I've added:@weband@duckduckgo
If you take a closer look at this “web.feature” file, you'll notice that the tags are only at the feature level, they're not at the scenarios.
What's really nice about feature level tags, is they inherently apply to all the scenarios within the feature.
So, even though I don't specifically mark these 2 scenarios as being @web or @duckduckgo, because the feature bears those tags, so too, the scenarios effectively bear them as well.
You can use any pytest options from the command line with pytest-bdd, they just might work in slightly different ways.
To run all tests, use the standard python -m pytest [...] invocation.
But if you want to filter down and run only specific tests, you could say, "Run all the tests in a test module” by writing the path to that test module [pytest test_mod.py] or also a directory [pytest testing].
If you want to filter by tag, you can use the -k option: pytest -k "MyClass and not method"
You can also use the -m option if you would like.
If you want to run specific scenarios by name, you'll need to use the at scenario function instead of the scenarios function shortcut, and explicitly give that scenario a “test_” function name. And then you can reference it like this, double colons, by name of the function from the test module path [pytest test_mod.py::test_func]
Typically, though, for pytest-bdd tests, I find that to be a bit overkill, and I recommend sticking to tags just for simplicity and ease of invoking from the command line.
So now, let's run our pytest-bdd tests from the command line using different types of pytest invocation.
I'm already in my project, so I can run my tests using the basic command.
Since I'm using pipenv as my package manager, I'll need to start my command with pipenv run, so all the dependencies are loaded. Then I run the python command, and I attach -m pytest to say I'm going to run the pytest module. Without any other options, this will discover and run all tests.
pipenv run python -m pytest
And so, if I enter, we can see how it's discovering every single one of my step definition test modules, and now it's running.
Notice how the web tests are the slowest: That's the “rule of ones” in action. We give it just a moment. All right, cool.
All 14 tests were discovered, and they passed in about 18 seconds.
Now, instead of running all tests, let's run all the tests in 1 of our test modules.
If I provide the path to, let's say the “cucumbers module” —
pipenv run python -m pytest tests/step_defs/test_cucumbers_steps.py
and I hit run.
Notice how for this one, it only ran 6 tests. And it only run the 6 tests from the “test_cucumbers_steps” module. All the other tests were excluded.
Now, let's see instead of filtering by the test module path, I want to filter by tags.
I'll say pytest -k, and let's say I want to run the tests that have the cucumber basket tag [@cucumber-basket].
pipenv run python -k pytest “cucumber-basket”
Note here that I'm going to put it in double quotes just to make sure that any spaces are included in my tagging expression. And also note that there's no @ symbol in front, it's just the raw identifier name.
So, now when I run them, it again runs 6 tests, because all the tests in that feature file were tagged with @cucumber-basket.
But notice how also it mentions that 8 were deselected. That means that the other 8 tests of the 14, that did not have the specific tag, were explicitly excluded from the run.
We can also run tests by tags when the tag is used by multiple different feature files.
If you recall, we use the @duckduckgo tag in both the service and the web tests.
pipenv run python -k pytest “duckduckgo”
And here if I run, we can see that the “test_cucumbers_steps” test the module was excluded, but the “service” and the “web” steps modules were included. And so, 8 selected tests are running, and they all passed, nice.
We can be a little bit more advanced with our tagging expressions too.
Let's say that I want to run all tests that have the tags duckduckgo, as well as the service tag.
pipenv run python -k pytest “duckduckgo and service”
Now, this should filter all the scenarios that were in the test_service_steps
And exclude the ones from test_web_steps because even though the web steps once had the @ducduckgo tag, they did not have the @service tag.
And what do you know? That's exactly what happened.
We can use any sort of basic logical operator, such as “and”, “or”, and “not”, in our tagging expressions.
So, if we do it again, I'll show you an example with “or”. Let's say that I want to run all tests that have the tag @service, or @web.
pipenv run python -k pytest “service or web”
And again, it picks up all the “@service or @web” tests. Nice.
One more moment, web tests can be slow. There we go, cool.
Finally, let's show how to use “not”.
So, I can say, "Run not the web tests."
pipenv run python -k pytest “not web”
Boom! Lightning fast. All the tests except the web test were included in the run, awesome.
You can also filter by tags when running your tests through PyCharm.
What you'll need to do is go to the Run/Debug Configurations.
And for your Configuration for whichever tests you want to run, you can add the tag names as keywords.
Now, if you were to run them, you'll notice that it does the filtering.
So, if I run them real quick, we'll see here my test cucumbers steps, ran only 3 tests and it deselected the 3 because my run configuration included only the “@add” tag.
So, you can use it either through command line or through an IDE.
