Reporting with the Functional Testing Framework
The Functional Testing Framework (FTF) provides a reporting tool, which logs failures or any other information for you during test run.
The following image demonstrates example of a general flow.
The event manager is a core component which:
- dispatches events
- gets a list of observers
- notifies observers depending on read configuration and preset
The event manager is defined in the
\Magento\Mtf\System\Event\EventManager class that:
- is an entry point to the event management system
- fetches configuration and observers
- handles events and passes them to observers
- notifies selected observers according to an event tags array and configuration
<magento_2_root_dir>/dev/tests/functional/phpunit.xml, you can set a preset to use and a directory to store reports.
Set a preset
<preset_name> with a name of preset that you want to use.
Set a reporting directory
Set the value of
<env name="log_directory"> to the directory in which to store your reports.
The default path is
An event preset specifies observers and dispatched events handled by them.
<magento_2_root_dir>/dev/tests/functional/etc/events.xml contains a list of event presets.
Format of a preset:
Explanation of the example:
Each time the
"failure" events are dispatched, HTML code and screen capture of the current web page are logged (see Observers section for details). If the
"curl_failed" event is dispatched, a corresponding cURL response is saved in an HTML file in the
magento/<module>/<test_case>/<variation>/curl-response directory inside the report directory.
Initially, event presets are defined in the FTF in
<magento_2_root_dir>/dev/tests/functional/vendor/magento/mtf/etc/events.xml (open the
events.xml on GitHub repository). It is not recommended to edit this file. You can extend the initial list or add new presets in
<magento_2_root_dir>/dev/tests/functional/etc/events.xml. All changes are merged automatically.
An observer is a PHP class which defines actions under Magento instance, browser, test run, and so on.
The list of ready-to-use observers is the following:
|Observer full class name||Description|
||Saves response into HTML file in
||Saves event message to the
||Sets a page URL parameter to the instance of the
||Captures a screenshot of a web page. Saves a PNG image to the
||Collects HTML code of a web page. Saves HTML code to the
A tag contains name of an event. When you want to process any event by a particular observer, you need to:
- Dispatch the event.
- Add a tag with name of the event to required observer in corresponding event preset.
In terms of XML, it is represented as an element
<tag /> in
<tag /> is a child element of an
<observer> element. See the following example:
As you can see, a tag contains one required attribute
name, where a name of event must be assigned.
A method that is used to dispatch events is defined in
\Magento\Mtf\System\Event\EventManagerInterface. The FTF uses its default implementation
It has two arguments:
- Array of event tags. Event tags specify the name of event that is dispatched. It is used as a tag in event preset.
- Input parameters. The parameters used by observers as input parameters. For example, a cURL response.
Example of use:
The following examples explain how to use the reporting tool on practice.
Create a preset
The following example shows how to add a
Task: Create a preset that logs only a web page HTML code and its screenshot when a test run is failed.
- HTML code
What is needed:
Step 1. Dispatch the
failure event in the test case. This event already exists and is added to the code where the FTF processes the test failure.
Step 2. Open
Step 3. Add a preset with required observers and tags.
Edit a preset
The following example shows how to edit the
Task: Take a screenshot before click, after click, and when a value is set.
What is needed:
base preset is stored in the FTF
<magento2>/dev/tests/functional/vendor/magento/mtf/etc/events.xml. To add or change any setting, edit
<magento2>/dev/tests/functional/etc/events.xml, which is merged with the one in the FTF.
Step 1. Dispatch the events
set_value in your code where applicable.
Step 2. In the
base preset, add required observer and event tags.
Create and apply a custom observer
You can create your own observer using existing examples.
General implementation rules:
- An observer must implement
- Put the class in
The following example shows how to use a custom observer in the example with the
Task: Create observer that logs WebAPI responses containing exceptions. Use the observer when a fixture is saved.
- Log in JSON
What is needed:
Step 1. Create an observer class
\Magento\Mtf\System\Observer\WebapiResponse that stores incoming events in JSON files.
Step 3. Add the observer and the tag to the
base preset in
<magento_2_root_dir>/dev/tests/functional/etc/events.xml, add to a preset
<preset name="base"> an observer
<observer class="Magento\Mtf\System\Observer\WebapiResponse"> with a tag
<tag name="webapi_failed" />: