Magento Enterprise Edition (EE) 1.13 Benchmarking Guide


Customer Needs and Expectations

The focus of the Magento Enterprise Edition 1.13 release is performance and scalability. The benchmarking results presented here demonstrate that we have addressed the following concerns:

  1. The eCommerce landscape is changing, and merchants must provide customers with an online shopping experience that meets their performance expectations
  2. As merchants grow their businesses and increasingly larger enterprise merchants are adopting Magento, Magento Enterprise Edition must scale to handle increased traffic volume and larger catalogs

Magento's Plan

We identified the following areas for enhancement in Magento Enterprise Edition 1.13:

What Magento Accomplished

Magento Enterprise Edition 1.13 delivers the following improvements over Magento Enterprise Edition 1.12:

Multi-Node Deployment Topology

Before we introduce you to the benchmarks, an overview of our multi-node benchmarking facility is in order. We started with a basic multi-node cluster with load balancer and caching and separate DB node, and installed Apache. This is a familiar hardware configuration for the Magento developer community. Once provisioned, we installed both Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12 on the cluster to compare their performance.

This is a representation of the multi-node Magento Enterprise Edition 1.12/1.13 installation used for performance testing. It consists of four physical nodes, three of which are virtualized.

We used HP ProLiant SL230s Gen8 servers with 8x16GB DDR3 and 2x Intel Xeon E5‐2660 CPUs for our benchmarking.

We used a physical disk of a usable 1.2 TB managed by a RAID10 controller. The load balancer we used is nginx.

The cluster resides in our Las Vegas data center, connected to the Internet via a gigabit connection. We tested the cluster using the popular Gatling suite from our engineering offices in Austin, TX.

Software Components


This section presents the merchant scenarios we simulated for our testing. These scenarios are based on real-world experience and industry standards with respect to shopper flows and catalog sizes.

Shopper Flows

We used real-world, established eCommerce metrics to simulate shopper flows.


The catalogs we simulated also reflect real-world, established eCommerce experience.

Target Merchant Profile

The simulated merchant profile we benchmarked against represents the profile of our enterprise customers. We established these metrics based on the day-to-day experience of large merchants operating a successful eCommerce business. Again, we used what we consider to be a typical hardware and software configuration.

Benchmarking Results

These benchmarks were generated during extensive testing of our multi-node configuration running Magento Enterprise Edition 1.13 described above. Our configuration was modeled after what a commercial hosting partner would put into production, and the results reflect accurate gains over Magento Enterprise Edition 1.12.

The duration of the test sessions is 72 seconds, which significantly stresses the multi-node configuration.

Incremental Re-indexing

In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index. Magento Enterprise Edition 1.13 introduces a new feature--incremental re-indexing. With incremental re-indexing, only those items that were changed or added will be re-indexed, reducing the processing time to a fraction of what was required before.

Take the example of a merchant with a catalog containing 500,000 products. In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index operation. In Magento Enterprise Edition 1.13, incremental re-indexing means the merchant will only re-index items that were changed. The test focused on measuring the improvements provided by the incremental re-indexing feature in Magento Enterprise Edition 1.13. The table below compares improvements to common admin actions, such as changing a product description, prices or inventory.

Full Re-indexing

As part of the benchmarking effort, we also measured the improvements in the full re-indexing feature on Magento Enterprise Edition 1.13, where indexing a 500,000-item catalog was 53% faster than Magento Enterprise Edition 1.12. Faster re-indexing means less load on the system and that changes to the catalog propagate faster to the storefront.

Individual Re-indexers

The Magento Enterprise Edition 1.13 indexer component contains a number of individual indexers such as Product Flat Data and Product Price. When a Magento admin changes the price of a product, it is only necessary to execute the Product Price indexer for pricing changes to propagate to the front end. The completion times of these individual indexers were measured in the benchmark environment for Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12.

This section presents the results for full re-index completion times for the individual indexers in Magento Enterprise Edition 1.13 compared to Magento Enterprise Edition 1.12.

Page Load Times

When Magento Enterprise Edition 1.12 and Magento Enterprise Edition 1.13 were compared, with both running on our multi-node benchmarking configuration, Magento Enterprise Edition 1.13 loaded pages 65% faster than Magento Enterprise Edition 1.12.

Guest checkout and registered user checkout are two flows that are crucial to storefront operation. This section presents the results of page load time measurements for these two flows.

Guest Checkout Flow Page Load Times

In the guest checkout flow pages, Magento Enterprise Edition 1.13 provides a substantial decrease in load times over its predecessor, most of the time more than twice as fast.

Registered Checkout Flow Page Load Times

The bar chart below presents the improvements in page load times for registered checkout flow.

Page Views and Orders

In addition to page load times, the benchmark also focused on measuring throughput improvements, particularly page views per day and orders per day.

During our testing, which simulated a storefront running at peak hours, EE 1.13 executed 33% more orders and 31% more page views than Magento Enterprise Edition 1.12 on the multi-node benchmarking configuration. Notably, Magento Enterprise Edition 1.13 served 47K pages during the test run (10 minutes).


What we noted during the benchmarking tests:


Magento Enterprise Edition 1.13 was engineered for performance--and clearly delivers on the goal as measured by important metrics.

In addition, we have focused on providing these benefits without the need to upgrade existing hardware, improving your return on assets and investment.


Page Load Times

This benchmark report presented analysis of the results produced by the Gatling load generator tool. In this appendix, you can see the actual results reported by Gatling. The results were obtained under the following test setup, which was designed to push Magento to its limit.
Concurrent session (users)1000
Peak load duration10 minutes
Session duration72 seconds
CPU utilization95%

CPU utilization is high because session duration was set to 72 seconds for the test. Session duration was set to a low value to increase the active load on Magento. A typical e-commerce site will see session durations of five minutes or more. Preliminary experiments by the benchmark team have confirmed that using a five-minute session duration reduces the CPU utilization to 50-60%.
Magento EE v1.13
RequestsTotalOKKOMinMaxMeanStd Dev95th Pct99th PctReq/s
Global Information06216162087748012910173518695780806075
product page0285732857308093907099962670492034
Click add to cart1104581045804407170186811394010498013
Click add to cart Redirect 121045810458096012910381622398050996013
click checkout334863486011001080034591965713089704
Registered Checkout:Login and Pick Address4336633660200738011551482487059004
Registered Checkout:Login and Pick Address Redirect 153366336607001144026051662564077104
Guest Checkout start612012003602490908487186024200
Registered:Pick Billing Address 17120120039042401457974351042100
Registered:Pick Billing Address 2-18120120041036901218685246030700
Registered:Pick Billing Address 2-29120120040036001201714270035100
Registered:Go to pick Shipping Method10120120034029001072633233027000
Guest:Pick Billing Address 111120120061046201688941358044400
Guest:Pick Billing Address 212120120038035301049589205027500
Guest:Go to pick Shipping Method1312012003002080796401153019600
Set Shipping Method (Flatrate) 1142402400510483017801066393047200
Set Shipping Method (Flatrate): Goto Payment1524024003003150905516194023000
Set Payment Method (Check/MO) 116240240067052901787987366049200
Set Payment Method (Check/MO) 21724024003002760904518201024400
Set Payment Method (Check/MO) 3182202200370818026961725581075900
/checkout/../success/ Redirect 120740741030835024731620559073100
Magento EE v1.12
RequestsTotalOKKOMinMaxMeanStd Dev95th Pct99th PctReq/s
Global Information04804447887157404058049853290106501387058
product page021327212864140200404557282097401235026
Click add to cart183408324164012130421622177520900010
Click add to cart Redirect 12832483204401650070223449117801350010
click checkout327802779192012350631128299760109503
Guest Checkout start49090039036902103795321034100
Registered Checkout:Login and Pick Address5269026900180191203298498415470169403
Guest:Pick Billing Address 1690900600775043481605643074000
Registered Checkout:Login and Pick Address Redirect 1726902688240243605433335011780140603
Guest:Pick Billing Address 2890900520824034121423548068100
Guest:Go to pick Shipping Method99090031047802334888330044500
Registered:Pick Billing Address 110909003701009036572453866096000
Registered:Pick Billing Address 2-11190900540176704148274110220125100
Registered:Pick Billing Address 2-2129090053012640398324118290115900
Registered:Go to pick Shipping Method1390900360629025461401505059800
Set Shipping Method (Flatrate) 1141801800450764039261884655074500
Set Shipping Method (Flatrate): Goto Payment151801800380629024511256427048300
Set Payment Method (Check/MO) 1161801800570829039231996646073000
Set Payment Method (Check/MO) 2171801800330523023181238421048100
Set Payment Method (Check/MO) 31818018009404058012673795725910295400
/checkout/../success/ Redirect 120930937801047050312251792088800

Estimating the Number of Visitors

Gatling does not report the number of users that cycled through during the test. However, it is possible to estimate the number using the following formula: