This course was written with pytest-bdd version 3. When pytest-bdd updated to version 4, they introduced a backwards-incompatible change regarding "@given" decorators. You must now include a "fixture_target" parameter with the name of the method in order for other steps to use it as a fixture. The example project code is updated, but the videos and transcripts still show the old code.
Our example test project now has many tests in it.
We could run them all at once using the
pytest command; but if we were to continue adding more and more tests, that would take a lot more time. It's not unusual for test suites to take minutes to hours and possibly even days to run to completion.
So, how can we filter our tests to run only specific ones that we want to do?
One of the easiest answers is tags, which will be covering in this chapter.
I added tags to the feature files already in our test project.
If we look at
cucumbers.feature I've put a tag at the
Feature level called
@cucumber-basket Feature: Cucumber Basket As a gardener, I want to carry cucumbers in a basket, So that I don't drop them all. @add Scenario Outline: Add cucumbers to a basket Given the basket has "<initial>" cucumbers When "<some>" cucumbers are added to the basket Then the basket contains "<total>" cucumbers Examples: | initial | some | total | | 0 | 3 | 3 | | 2 | 4 | 6 | | 5 | 5 | 10 | @remove Scenario Outline: Remove cucumbers from the basket Given the basket has "<initial>" cucumbers When "<some>" cucumbers are removed from the basket Then the basket contains "<total>" cucumbers Examples: | initial | some | total | | 8 | 3 | 5 | | 10 | 4 | 6 | | 7 | 0 | 7 |
I've also added two tags —
@remove — for the Add and Remove test scenarios.
Tags can be applied either to features, or to scenarios, or scenario outlines.
The format of a tag is fairly basic — it's prefixed by an @ symbol, and then it's a bunch of alphanumeric or dashes or underscore characters.
The key thing is that is an identifier, that it is one token.
A scenario or a feature can have an unlimited number of tags, and tags can be reused by different features and scenarios as well.
If we look at the other feature files:
service.feature, I've added the 2 tags:
And to the
web.feature, I've added:
If you take a closer look at this “web.feature” file, you'll notice that the tags are only at the feature level, they're not at the scenarios.
What's really nice about feature level tags, is they inherently apply to all the scenarios within the feature.
So, even though I don't specifically mark these 2 scenarios as being
@duckduckgo, because the feature bears those tags, so too, the scenarios effectively bear them as well.
You can use any
pytest options from the command line with
pytest-bdd, they just might work in slightly different ways.
To run all tests, use the standard
python -m pytest [...] invocation.
But if you want to filter down and run only specific tests, you could say, "Run all the tests in a test module” by writing the path to that test module [
pytest test_mod.py] or also a directory [
If you want to filter by tag, you can use the
pytest -k "MyClass and not method"
You can also use the
-m option if you would like.
If you want to run specific scenarios by name, you'll need to use the at scenario function instead of the scenarios function shortcut, and explicitly give that scenario a “test_” function name. And then you can reference it like this, double colons, by name of the function from the test module path [
Typically, though, for
pytest-bdd tests, I find that to be a bit overkill, and I recommend sticking to tags just for simplicity and ease of invoking from the command line.
So now, let's run our
pytest-bdd tests from the command line using different types of
I'm already in my project, so I can run my tests using the basic command.
Since I'm using
pipenv as my package manager, I'll need to start my command with
pipenv run, so all the dependencies are loaded. Then I run the python command, and I attach
-m pytest to say I'm going to run the
pytest module. Without any other options, this will discover and run all tests.
pipenv run python -m pytest
And so, if I enter, we can see how it's discovering every single one of my step definition test modules, and now it's running.
Notice how the web tests are the slowest: That's the “rule of ones” in action. We give it just a moment. All right, cool.
All 14 tests were discovered, and they passed in about 18 seconds.
Now, instead of running all tests, let's run all the tests in 1 of our test modules.
If I provide the path to, let's say the “cucumbers module” —
pipenv run python -m pytest tests/step_defs/test_cucumbers_steps.py
Notice how for this one, it only ran 6 tests. And it only run the 6 tests from the “test_cucumbers_steps” module. All the other tests were excluded.
Now, let's see instead of filtering by the test module path, I want to filter by tags.
pytest -k, and let's say I want to run the tests that have the cucumber basket tag [@cucumber-basket].
pipenv run python -k pytest “cucumber-basket”
Note here that I'm going to put it in double quotes just to make sure that any spaces are included in my tagging expression. And also note that there's no @ symbol in front, it's just the raw identifier name.
So, now when I run them, it again runs 6 tests, because all the tests in that feature file were tagged with
But notice how also it mentions that 8 were deselected. That means that the other 8 tests of the 14, that did not have the specific tag, were explicitly excluded from the run.
We can also run tests by tags when the tag is used by multiple different feature files.
If you recall, we use the
@duckduckgo tag in both the
service and the
pipenv run python -k pytest “duckduckgo”
And here if I run, we can see that the “test_cucumbers_steps” test the module was excluded, but the “service” and the “web” steps modules were included. And so, 8 selected tests are running, and they all passed, nice.
We can be a little bit more advanced with our tagging expressions too.
Let's say that I want to run all tests that have the tags duckduckgo, as well as the service tag.
pipenv run python -k pytest “duckduckgo and service”
Now, this should filter all the scenarios that were in the
And exclude the ones from
test_web_steps because even though the web steps once had the
@ducduckgo tag, they did not have the
And what do you know? That's exactly what happened.
We can use any sort of basic logical operator, such as “and”, “or”, and “not”, in our tagging expressions.
So, if we do it again, I'll show you an example with “or”. Let's say that I want to run all tests that have the tag
pipenv run python -k pytest “service or web”
And again, it picks up all the “@service or @web” tests. Nice.
One more moment, web tests can be slow. There we go, cool.
Finally, let's show how to use “not”.
So, I can say, "Run not the web tests."
pipenv run python -k pytest “not web”
Boom! Lightning fast. All the tests except the web test were included in the run, awesome.
You can also filter by tags when running your tests through PyCharm.
What you'll need to do is go to the Run/Debug Configurations.
And for your Configuration for whichever tests you want to run, you can add the tag names as keywords.
Now, if you were to run them, you'll notice that it does the filtering.
So, if I run them real quick, we'll see here my test cucumbers steps, ran only 3 tests and it deselected the 3 because my run configuration included only the “@add” tag.
So, you can use it either through command line or through an IDE.