Test results

Submissions go through several reviews before being released on the Magento Marketplace. You can use the following endpoints to get more information about the current status of a submission. See Get package details for details about these endpoints.

1
2
3
4
GET /rest/v1/products/packages/:submission_id/status
GET /rest/v1/products/packages/sku/:url_encoded_sku/status
GET /rest/v1/products/packages/item/:item_id/status
GET /rest/v1/products/packages/:submission_id/status/:tool_name/:tool_run_id

In this example, submission f4eacd72be failed the marketing review. Use this request to get a summary report of the review process:

Request:

1
2
3
curl -X GET \
     -H 'Authorization: Bearer baGXoStRuR9VCDFQGZNzgNqbqu5WUwlr.cAxZJ9m22Le7' \
     https://developer-stg-api.magento.com/rest/v1/products/packages/f4eacd72be/status

Response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[
    {
        "tool_run_id": "5",
        "name": "phpcpd",
        "status": "pass",
        "date": "2020-01-07 18:43:32"
    },
    {
        "tool_run_id": "5",
        "name": "phpcs",
        "status": "pass",
        "date": "2020-01-07 18:52:25"
    },
    {
        "tool_run_id": "5",
        "name": "varnish",
        "status": "pass",
        "date": "2020-01-07 18:45:03"
    },
    {
        "tool_run_id": "6",
        "name": "marketing",
        "status": "fail",
        "date": "2020-01-08 16:57:55"
    }
]

The status field indicates the overall result of the review, which can be pass, fail, error, or needs_review.

Additional details for each review can be found using the status API with the tool name and the tool run ID.

Request:

1
2
3
curl -X GET \
     -H 'Authorization: Bearer baGXoStRuR9VCDFQGZNzgNqbqu5WUwlr.cAxZJ9m22Le7' \
     https://developer-stg-api.magento.com/rest/v1/f4eacd72be/status/marketing/6

Response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "tool": "marketing",
    "status": "FAIL",
    "reports": {
        "tool": "marketing",
        "vendor": "acme",
        "extension": "one-click-checkout",
        "version" : "1.1.5",
        "status": "FAIL",
        "date": "2020-01-08 16:57:55",
        "details": {
            "reasons": ["Long Description: Your extension long description contains several errors. View marketing review guidelines..."],
            "message": "Please correct grammar and spelling issues. See attached screenshots for more information.",
            "attachment": "https://mp-stg-static.magento.com/user/attachments/ha/8d/ha8d83d9482un439d8s09xnc88/prv/jj/t1/jjt18m0u38ru0v4v8y76q5e2xre5vrecz/spelling.zip"
        }
    }
}
  • The top-level status field contains the final decision made for this test. All status values in the reports array must be PASS in order for the test to pass.
  • The reports array can contain multiple entries. For automated tests, there will be additional information about the environment used for testing.
  • The details array structure will be different for each test.

Test result structures

The following examples demonstrate failure reports for each test type.

Code Sniffer (PHPCS)

A code sniffer will be run against every PHP version the extension is compatible with. The reports array contains a separate entry for each environment. The details array contains the output generated by the Code Sniffer tool.

Additional fields:

Field Name Type Value
php_version string PHP version number

Request:

1
2
3
curl -X GET \
     -H 'Authorization: Bearer baGXoStRuR9VCDFQGZNzgNqbqu5WUwlr.cAxZJ9m22Le7' \
     https://developer-stg-api.magento.com/rest/v1/f4eacd72be/status/phpcs/8

Response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
{
    "tool": "phpcs",
    "status": "FAIL",
    "reports": [
        {
            "tool": "phpcs",
            "vendor": "acme",
            "extension": "one-click-checkout",
            "version" : "1.1.5",
            "php_version" : "7.1.16",
            "status": "FAIL",
            "date": "2020-01-07 18:52:25",
            "details": {
                "totals": {
                    "errors": 1,
                    "warnings": 0,
                    "fixable": 0
                },
                "files": {
                    "\/eqp-automation\/temp\/acme\/one-click-checkout\/acme-one-click-checkout-1.1.5\/registration.php": {
                        "errors": 1,
                        "warnings": 0,
                        "messages": [
                            {
                                "message": "\"resource\" is a reserved word in PHP 7",
                                "source": "MEQP2.NamingConventions.ReservedWords.FoundReservedWord",
                                "severity": 10,
                                "type": "ERROR",
                                "line": 3,
                                "column": 36,
                                "fixable": false
                            }
                        ]
                    }
                }
            }
        },
        {
            "tool": "phpcs",
            "vendor": "acme",
            "extension": "one-click-checkout",
            "version" : "1.1.5",
            "php_version" : "5.6.38",
            "status": "PASS",
            "date": "2020-01-07 18:52:25",
            "details": {
                "totals": {
                    "errors": 0,
                    "warnings": 0,
                    "fixable": 0
                },
                "files": {
                  "\/eqp-automation\/temp\/acme\/one-click-checkout\/acme-one-click-checkout-1.1.5\/registration.php": {
                        "errors": 0,
                        "warnings": 0,
                        "messages": []
                    }
                }
            }
        }
    ]
}

Copy/Paste Detector (PHPCPD)

CPD reports cannot be directly requested. Only the summary data is available through the API.

Varnish

The Varnish test installs the extension into a Magento instance and attempts to access several pages to determine if Varnish caching is working as expected. The test will be run against all combinations of Magento versions, Magento editions, and PHP versions.

Additional fields:

Field Name Type Value
php_version string PHP version number
magento_platform string M1 or M2
magento_edition string CE or EE
magento_version string Magento version number

Request:

1
2
3
curl -X GET \
     -H 'Authorization: Bearer baGXoStRuR9VCDFQGZNzgNqbqu5WUwlr.cAxZJ9m22Le7' \
     https://developer-stg-api.magento.com/rest/v1/f4eacd72be/status/varnish/8

Response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
    "tool" : "varnish",
    "status" : "FAIL",
    "reports" : [
        {
            "tool": "varnish",
            "vendor": "acme",
            "extension": "one-click-checkout",
            "version": "1.1.5",
            "magento_platform" : "M2",
            "magento_edition" : "CE",
            "magento_version" : "2.2.3",
            "php_version" : "7.1.16",
            "status": "FAIL",
            "date": "2020-01-07 19:45:42",
            "details" : [
                {
                    "command": "composer require acme\/one-click-checkout:1.1.5 2>&1",
                    "output" : "<string output of composer require command goes here>"
                },
                {
                    "command": "php bin\/magento setup:upgrade",
                    "output" : "<string output of setup:upgrade command goes here>"
                },
                ...
            ]
        },
        ...
    ]
}

The last command/output step in the details section is the command that failed to execute successfully.

Marketing and Manual QA

The Marketing and Manual QA reviews share the same report structure. There should only be one entry in the reports array for these reviews. The tool names are marketing and manual_qa respectively.

Request:

1
2
3
curl -X GET \
     -H 'Authorization: Bearer baGXoStRuR9VCDFQGZNzgNqbqu5WUwlr.cAxZJ9m22Le7' \
     https://developer-stg-api.magento.com/rest/v1/f4eacd72be/status/manual_qa/9

Response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
  "tool": "manual_qa",
  "status": "FAIL",
  "reports": {
    "tool": "manual_qa",
    "vendor": "acme",
    "extension": "one-click-checkout",
    "version": "1.1.5",
    "status": "FAIL",
    "date": "2020-01-14 07:21:57",
    "details": {
      "reasons": ["Tech: Your extension does not function as described in the user guide."],
      "message": "Missing feature - your extension was missing a feature when tested on Magento 2.2.3. See attached screenshots for more information.",
      "attachment": "https://mp-stg-static.magento.com/user/attachments/ha/8d/ha8d83d9482un439d8s09xnc88/prv/jj/t1/jjt18m0u38ru0v4v8y76q5e2xre5vrecz/missing_feature.zip"
    }
  }
}