Test suite in the Functional Testing Framework
In the Functional Testing Framework (FTF), you can run a group of tests, which is called test suite. A test suite is a collection of tests, grouped by a specified rule, that is used to test a custom scope of functionality.
There are two rule types available:
- allow, specifying what must be included during the test run
- deny, specifying what must be excluded during the test run
The rules for a test case are defined in a separate
.xml file. (Recommended naming: use lowercase letters and underscore as a separator). One file contains rules for one test suite. All files are stored in the
<magento2_root_dir>/dev/tests/functional/testsuites/Magento/Mtf/TestSuite/InjectableTests directory by default. Only one test suite can be run at a time.
The example of the default test suite:
This set of rules selects functional tests that accepts the following criteria:
- WITH the tag
const test_type = 'acceptance_test'
- EXCEPT test cases with the tag
const stable = 'no'
AND runs variations
- WITH the tag
- EXCEPT variations with the tag
Learn more details in next topics.
Define the test suite to be run in the
The default test suite is
Run your test suite
To run a test suite enter the following commands from your terminal:
The only attribute of a rule node is the
scope, which enables you to use the following options:
scope = "testsuite". Enables you to filter the test cases by a namespace, a module, a class.
scope = "testcase". Enables you to select the test cases with a specified tag.
scope = "variation". Enables you to use in a test run only variations with a specified tag.
scope = "constraint". Enables you to run only assertions with a specified tag.
scope = "testsuite"
This scope enables you to filter functional tests using the following criteria:
||Apply a rule to the test case with the specified class name.||multiple||
||Apply a rule to all test cases from the specified module. Some test cases may refer to other modules using merging functionality of variations, fixtures etc. You can restrict such reference to other modules adding the
||Apply a rule to all test cases with the specified namespace.||multiple||
The namespace filter example:
The module filter example:
The class filter example:
scope = "testcase"
In this scope, you can group test cases using tags.
You can use
value parameters in the test case scope. In a test case, they are provided as a constant name and its value respectively. See the following example:
- Any tags that are used in the test case should be added to the beginning of a class definition:
- The tag in the rule:
A test case can contain multiple tag groups, and a group can have multiple values. For example:
scope = "variation"
You can assign a
tag node to a data set variation. This enables you to use customized sets of variations during the test run. You can allow to use in the test run only that variations which are specified with tag, or deny to use them.
You can use
value parameters in the variation scope. In a variation, they are provided in the following format:
For example, you have a data set with the following variation:
By using the
<allow> element, you can create a rule to use only the
or use all variations except the
scope = "constraint"
You can select constraints from the variation that will be run after a test flow using tags and rules with
A tag has two parameters:
value. In a constraint, they are provided as a constant name and its value respectively. See the following example:
- Any tags that are used in the constraint should be added to the beginning of a class definition:
- The rule that allows constraints with this tag only:
- The rule that allows all constraints except those having this tag: