How to use MFTF in CICD
To integrate MFTF tests into your CICD pipeline, it is best to start with the conceptual flow of the pipeline code.
The overall workflow that tests should follow is:
- Obtain a Magento instance + install pre-requisites.
- Generate the tests.
- Set options for single or parallel running.
- Delegate and run tests and gather test-run artifacts.
- Re-run options.
- Generate the Allure reports from the results.
Obtain a Magento instance
To start, we need a Magento instance to operate against for test generation and execution.
1 git clone https://github.com/magento/magento2
1 composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce
For more information on installing magento see Install Magento using Composer.
After installing the Magento instance, set a couple of configurations to the Magento instance:
1 2 3 4 bin/magento config:set general/locale/timezone America/Los_Angeles bin/magento config:set admin/security/admin_account_sharing 1 bin/magento config:set admin/security/use_form_key 0 bin/magento config:set cms/wysiwyg/enabled disabled
These set the default state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step to do it.
If your magento instance has Two-Factor Authentication enabled, see Configure 2FA to configure MFTF tests.
This is required for generating the report after your test runs. See Allure for details.
Generate tests based on what you want to run:
1 vendor/bin/mftf generate:tests
This will generate all tests and a single manifest file under
To generate all tests for use in parallel nodes:
1 vendor/bin/mftf generate:tests --config parallel
This generates a folder under
dev/tests/acceptance/tests/functional/Magento/_generated/groups. This folder contains several
group#.txt files that can be used later with the
mftf run:manifest command.
Delegate and run tests
If you are running on a single node, call:
1 vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt
You can optimize your pipeline by running tests in parallel across multiple nodes.
Tests can be split up into roughly equal running groups using
You do not want to perform installations on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes.
The groups can be then distributed on each of the nodes and run separately in an isolated environment.
- Stash artifacts from main node and un-stash on current node.
vendor/bin/mftf run:manifest <current_group.txt>on current node.
- Gather artifacts from
dev/tests/acceptance/tests/_outputfrom current node to main node.
In either single or parallel execution, to re-run failed tests, simply add the
run:failed command after executing a manifest:
1 vendor/bin/mftf run:failed
Generate Allure report
In the main node, generate reports using your
<path_to_results> into a desired output path:
1 allure generate <path_to_results> -c -o <path_to_output>