CLI commands: vendor/bin/mftf

The Magento Functional Testing Framework (MFTF) introduces the command line interface (CLI) tool vendor/bin/mftf to facilitate your interaction with the framework.

Note that mftf commands replace the robo commands that were used in previous releases.

Command format

In the project root directory (where you have installed the framework as a composer dependency), run commands using the following format:

1
vendor/bin/mftf command [options] [<arguments>] [--remove|-r]

Useful commands

Use the following commands to run commonly performed tasks.

Apply the configuration parameters

1
vendor/bin/mftf build:project

Upgrade the project

1
vendor/bin/mftf build:project --upgrade

Upgrades all installed MFTF tests after a major MFTF upgrade.

Generate all tests

1
vendor/bin/mftf generate:tests

Generate tests by test name

1
vendor/bin/mftf generate:tests AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest

Generate test by test and suite name

1
vendor/bin/mftf generate:tests WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest

Generate and run the tests for a specified group

1
vendor/bin/mftf run:group product -r

This command cleans up the previously generated tests; generates and runs tests for the product group (where group="product").

Generate and run particular tests

1
vendor/bin/mftf run:test AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest -r

This command cleans up the previously generated tests; generates and runs the AdminLoginSuccessfulTest and StorefrontPersistedCustomerLoginTest tests.

Generate and run particular test in a specific suite’s context

1
vendor/bin/mftf run:test WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest -r

This command cleans up previously generated tests; generates and run AdminCMSPageCreatePageTest within the context of the WYSIWYGDisabledSuite.

Generate and run a testManifest.txt file

1
vendor/bin/mftf run:manifest path/to/your/testManifest.txt

This command runs all tests specified in a testManifest.txt file. When you generate tests, a testManifest.txt file is also generated for you. You can pass this file directly to the run:manifest command and it will execute all listed tests. You can also create your own file in the same format to execute a subset of tests. Note: This command does not generate tests.

Generate previously failed tests

1
vendor/bin/mftf generate:failed

This command cleans up the previously generated tests; generates the tests listed in dev/tests/acceptance/tests/_output/failed. For more details about failed, refer to Reporting.

Run previously failed tests

1
vendor/bin/mftf run:failed

This command runs the tests listed in dev/tests/acceptance/tests/_output/failed. For more details about failed, refer to Reporting.

Error tolerance during generation

Starting from version 3.2.0, MFTF will not fail right away when encountering generation errors. Instead, MFTF will generate as many tests and suites as it can, log errors to mftf.log, and exit with a non-zero generation status.

Note:

• Not all errors are tolerable at generation. For example, schema validation errors, parser errors, and WebApi authentication errors will cause hard failures, with no tests or suites being generated.
• Error tolerance in generation is meant to help local test development and testing and is expected to be run locally. All generation errors must be fixed in order to use other framework functionality, pass static checks, and to deliver MFTF tests.

Reference

build:project

Description

Clone the example configuration files and build the Codeception project.

Usage

1
vendor/bin/mftf build:project [--upgrade] [config_param_options]

Options

You can include options to set configuration parameter values for your environment since the project build process also sets up the environment.

1
vendor/bin/mftf build:project --MAGENTO_BASE_URL=http://magento.local/ --MAGENTO_BACKEND_NAME=admin214365

doctor

Description

Diagnose MFTF configuration and setup. Currently this command will check the following:

• Verify admin credentials are valid. Allowing MFTF authenticates and runs API requests to Magento through cURL
• Verify that Selenium is up and running and available for MFTF
• Verify that new session of browser can open Magento admin and store front urls
• Verify that MFTF can run MagentoCLI commands

Usage

1
vendor/bin/mftf doctor

Options

generate:tests

Description

Perform XML schema validation and generate PHP code from the tests defined in XML files. The path is set in the TESTS_MODULE_PATH configuration parameter.

Usage

1
vendor/bin/mftf generate:tests [option] [<test name>] [<test name>] [--remove]

Options

Examples of the JSON configuration

The configuration to generate a single test with no suites:

1
2
3
4
5
6
{  
   "tests":[
      "general_test1"  //Generate the "general_test1" test.
      ],
   "suites": null
}

The configuration to generate a single test in the suite:

1
2
3
4
5
6
7
8
{  
   "tests": null,       // No tests outside the suite configuration will be generated.
   "suites":{  
      "sample":[        // The suite that contains the test.
         "suite_test1"  // The test to be generated.
      ]
   }
}

Complex configuration to generate a few non-suite tests, a single test in a suite, and an entire suite:

1
2
3
4
5
6
7
8
9
10
11
12
13
{  
   "tests":[  
      "general_test1",
      "general_test2",
      "general_test3"
   ],
   "suites":{           //Go to suites.
      "sample":[        //Go to the "sample" suite.
         "suite_test1"  //Generate the "suite_test1" test.
      ],
      "sample2":[]      //Generate all tests in the "sample2" suite.
   }
}

The command that encodes this complex configuration:

1
vendor/bin/mftf generate:tests --tests '{"tests":["general_test1","general_test2","general_test3"],"suites":{"sample":["suite_test1"],"sample2":null}}'

Note that the strings must be escaped and surrounded in quotes.

generate:suite

Description

Generates one or more suites based on XML declarations.

Usage

1
vendor/bin/mftf generate:suite <suite name> [<suite name>] [--remove]

Options

Example

1
vendor/bin/mftf generate:suite suite1 suite2

generate:urn-catalog

Description

Generates a URN catalog, enabling PhpStorm to recognize and highlight URNs. It also enables auto-completion in PhpStorm.

Usage

1
vendor/bin/mftf generate:urn-catalog [--force] [<path to misc.xml>]

misc.xml is typically located at <project root>/.idea/misc.xml.

Options

Example

1
vendor/bin/mftf generate:urn-catalog .idea/misc.xml

reset

Description

Cleans any configuration files and generated artifacts from the environment. The .env file is not affected.

Usage

1
vendor/bin/mftf reset [--hard]

Options

Example

1
vendor/bin/mftf reset --hard

run:group

Generates and executes the listed groups of tests using Codeception.

Usage

1
vendor/bin/mftf run:group [--skip-generate|--remove] [--] <group1> [<group2>]

Options

Examples

Clean up after the last test run; generate from XML and execute the tests with the annotations group="group1" and group="group2":

1
vendor/bin/mftf -r -- run:group group1 group2

Execute previously generated tests with the annotations group="group1" and group="group2", skipping the regeneration of the test:

1
vendor/bin/mftf run:group -k -- group1 group2

run:test

Generates and executes tests by name using Codeception.

Usage

1
vendor/bin/mftf run:test [--skip-generate|--remove] [--] <name1> [<name2>]

Options

Examples

Generate the LoginCustomerTest and StorefrontCreateCustomerTest tests from XML and execute all the generated tests:

1
vendor/bin/mftf run:test LoginCustomerTest StorefrontCreateCustomerTest

run:manifest

Runs a testManifest.txt file.

This command runs all tests specified in a testManifest.xml file. It does not generate tests for you. You must do that as first.

Usage

1
vendor/bin/mftf run:manifest path/to/your/testManifest.txt

Example testManifest.xml file

Each line should contain either: one test path or one group (-g) reference.

1
2
3
4
5
tests/functional/tests/MFTF/_generated/default/AdminLoginSuccessfulTestCest.php
-g PaypalTestSuite
tests/functional/tests/MFTF/_generated/default/SomeOtherTestCest.php
tests/functional/tests/MFTF/_generated/default/ThirdTestCest.php
-g SomeOtherSuite

run:failed

Regenerates and reruns tests that previously failed.

This command cleans up previously generated tests. It generates and runs the tests listed in dev/tests/acceptance/tests/_output/failed. For more details about failed, refer to Reporting.

Usage

1
vendor/bin/mftf run:failed

Options

Examples

Run the tests that failed in the previous run:

1
vendor/bin/mftf run:failed

setup:env

Updates the configuration parameter values in the [.env] file. Creates the .env file if it does not exist.

Usage

1
vendor/bin/mftf setup:env [config_param_option1=<value>] [config_param_option2=<value>]

config_param is a configuration parameter from the .env file. The command consumes the parameters in a format of options assigned with values, for example --MAGENTO_BASE_URL=http://magento.local/. If you specify a parameter that the .env file does not contain, the command returns an error.

You can also update configuration parameter values when you use the build:project command.

Examples

To change values for the MAGENTO_BASE_URL and BROWSER:

1
vendor/bin/mftf setup:env --MAGENTO_BASE_URL=http://magento.local/ --BROWSER=firefox

To create a .env file with example parameters:

1
vendor/bin/mftf setup:env

The example parameters are taken from the etc/config/.env.example file.

static-checks

Runs all or specific MFTF static-checks on the test codebase that MFTF is currently attached to. Behavior for determining what tests to run is as follows:

• If test names are specified, only those tests are run.
• If no test names are specified, tests are run according to staticRuleset.json.
• If no staticRuleset.json is found, all tests are run.

Static checks errors are written to *.txt files under TEST_BP/tests/_output/static-results/

Usage

1
vendor/bin/mftf static-checks [<names>]...

Options

Examples

To check what existing static check scripts are available

1
vendor/bin/mftf static-checks --help

To run all existing static check scripts

1
vendor/bin/mftf static-checks

To run specific static check scripts

1
vendor/bin/mftf static-checks testDependencies

1
vendor/bin/mftf static-checks actionGroupArguments

1
vendor/bin/mftf static-checks deprecatedEntityUsage

1
vendor/bin/mftf static-checks pauseActionUsage

1
vendor/bin/mftf static-checks annotations

1
vendor/bin/mftf static-checks deprecatedEntityUsage -p path/to/mftf/test/module

1
vendor/bin/mftf static-checks pauseActionUsage -p path/to/mftf/test/module

1
vendor/bin/mftf static-checks testDependencies actionGroupArguments

Existing static checks

Defining ruleset

The static-checks command will look for a staticRuleset.json file under either:

• dev/tests/acceptance/staticRuleset.json, if embedded with Magento2
• dev/staticRuleset.json, if standalone

This file works as the default configuration to easily allow for the integration of static-checks in a CI environment. Currently, the ruleset only defines the tests to run. Here is an example of the expected format:

1
2
3
4
5
6
{
  "tests": [
    "actionGroupArguments",
    "anotherTest"
  ]
}

upgrade:tests

When the path argument is specified, this upgrade command applies all the major version MFTF upgrade scripts to a Test Module in the given path. Otherwise, it will apply all the major version MFTF upgrade scripts to all installed test components.

Test Module should have the directory structure of ActionGroup, Data, Metadata, Page, Section, Test, and Suite.

Note:

The upgrade scripts are meant to be used for Test Modules under source code control. If you have old versions of test modules under vendor, those test modules will get upgraded

Usage

1
vendor/bin/mftf upgrade:tests [<path>]

<path> is the path to a MFTF Test Module that needs to be upgraded. The command searches recursively for any *.xml files to upgrade.

Examples

To upgrade all installed MFTF tests:

1
vendor/bin/mftf upgrade:tests

To upgrade all test components inside modules in the dev/tests/acceptance/tests/ directory:

1
vendor/bin/mftf upgrade:tests /Users/user/magento2/dev/tests/acceptance/tests/

To upgrade all test components inside the Catalog module:

1
vendor/bin/mftf upgrade:tests /Users/user/magento2/app/code/Magento/Catalog/Test/Mftf/

codecept:run

A MFTF wrapper command that invokes vendor/bin/codecept run. This command runs tests in functional suite. Tests must be generated before using this command.

Usage

See the Run Command.

1
vendor/bin/mftf codecept:run [<suite|test>] --[<option(s)>]

Examples

1
2
# Run all tests in functional suite
vendor/bin/mftf codecept:run functional

1
2
# Run all tests in functional suite with options
vendor/bin/mftf codecept:run functional --verbose --steps --debug

1
2
# Run one test
vendor/bin/mftf codecept:run functional Magento/_generated/default/AdminCreateCmsPageTestCest.php --debug

1
2
# Run all tests in default group
vendor/bin/mftf codecept:run functional --verbose --steps -g default