The focus of the Magento Enterprise Edition 1.13 release is performance and scalability. The benchmarking results presented here demonstrate that we have addressed the following concerns:
We identified the following areas for enhancement in Magento Enterprise Edition 1.13:
Magento Enterprise Edition 1.13 delivers the following improvements over Magento Enterprise Edition 1.12:
Before we introduce you to the benchmarks, an overview of our multi-node benchmarking facility is in order. We started with a basic multi-node cluster with load balancer and caching and separate DB node, and installed Apache. This is a familiar hardware configuration for the Magento developer community. Once provisioned, we installed both Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12 on the cluster to compare their performance.
This is a representation of the multi-node Magento Enterprise Edition 1.12/1.13 installation used for performance testing. It consists of four physical nodes, three of which are virtualized.
We used HP ProLiant SL230s Gen8 servers with 8x16GB DDR3 and 2x Intel Xeon E5‐2660 CPUs for our benchmarking.
We used a physical disk of a usable 1.2 TB managed by a RAID10 controller. The load balancer we used is nginx.
The cluster resides in our Las Vegas data center, connected to the Internet via a gigabit connection. We tested the cluster using the popular Gatling suite from our engineering offices in Austin, TX.
This section presents the merchant scenarios we simulated for our testing. These scenarios are based on real-world experience and industry standards with respect to shopper flows and catalog sizes.
We used real-world, established eCommerce metrics to simulate shopper flows.
The catalogs we simulated also reflect real-world, established eCommerce experience.
The simulated merchant profile we benchmarked against represents the profile of our enterprise customers. We established these metrics based on the day-to-day experience of large merchants operating a successful eCommerce business. Again, we used what we consider to be a typical hardware and software configuration.
These benchmarks were generated during extensive testing of our multi-node configuration running Magento Enterprise Edition 1.13 described above. Our configuration was modeled after what a commercial hosting partner would put into production, and the results reflect accurate gains over Magento Enterprise Edition 1.12.
The duration of the test sessions is 72 seconds, which significantly stresses the multi-node configuration.
In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index. Magento Enterprise Edition 1.13 introduces a new feature--incremental re-indexing. With incremental re-indexing, only those items that were changed or added will be re-indexed, reducing the processing time to a fraction of what was required before.
Take the example of a merchant with a catalog containing 500,000 products. In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index operation. In Magento Enterprise Edition 1.13, incremental re-indexing means the merchant will only re-index items that were changed. The test focused on measuring the improvements provided by the incremental re-indexing feature in Magento Enterprise Edition 1.13. The table below compares improvements to common admin actions, such as changing a product description, prices or inventory.
As part of the benchmarking effort, we also measured the improvements in the full re-indexing feature on Magento Enterprise Edition 1.13, where indexing a 500,000-item catalog was 53% faster than Magento Enterprise Edition 1.12. Faster re-indexing means less load on the system and that changes to the catalog propagate faster to the storefront.
The Magento Enterprise Edition 1.13 indexer component contains a number of individual indexers such as Product Flat Data and Product Price. When a Magento admin changes the price of a product, it is only necessary to execute the Product Price indexer for pricing changes to propagate to the front end. The completion times of these individual indexers were measured in the benchmark environment for Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12.
This section presents the results for full re-index completion times for the individual indexers in Magento Enterprise Edition 1.13 compared to Magento Enterprise Edition 1.12.
When Magento Enterprise Edition 1.12 and Magento Enterprise Edition 1.13 were compared, with both running on our multi-node benchmarking configuration, Magento Enterprise Edition 1.13 loaded pages 65% faster than Magento Enterprise Edition 1.12.
Guest checkout and registered user checkout are two flows that are crucial to storefront operation. This section presents the results of page load time measurements for these two flows.
In the guest checkout flow pages, Magento Enterprise Edition 1.13 provides a substantial decrease in load times over its predecessor, most of the time more than twice as fast.
The bar chart below presents the improvements in page load times for registered checkout flow.
In addition to page load times, the benchmark also focused on measuring throughput improvements, particularly page views per day and orders per day.
During our testing, which simulated a storefront running at peak hours, EE 1.13 executed 33% more orders and 31% more page views than Magento Enterprise Edition 1.12 on the multi-node benchmarking configuration. Notably, Magento Enterprise Edition 1.13 served 47K pages during the test run (10 minutes).
What we noted during the benchmarking tests:
Magento Enterprise Edition 1.13 was engineered for performance--and clearly delivers on the goal as measured by important metrics.
In addition, we have focused on providing these benefits without the need to upgrade existing hardware, improving your return on assets and investment.
|Concurrent session (users)||1000|
|Peak load duration||10 minutes|
|Session duration||72 seconds|
|Magento EE v1.13|
|Requests||Total||OK||KO||Min||Max||Mean||Std Dev||95th Pct||99th Pct||Req/s|
|Click add to cart1||10458||10458||0||440||7170||1868||1139||4010||4980||13|
|Click add to cart Redirect 12||10458||10458||0||960||12910||3816||2239||8050||9960||13|
|Registered Checkout:Login and Pick Address4||3366||3366||0||200||7380||1155||1482||4870||5900||4|
|Registered Checkout:Login and Pick Address Redirect 15||3366||3366||0||700||11440||2605||1662||5640||7710||4|
|Guest Checkout start6||120||120||0||360||2490||908||487||1860||2420||0|
|Registered:Pick Billing Address 17||120||120||0||390||4240||1457||974||3510||4210||0|
|Registered:Pick Billing Address 2-18||120||120||0||410||3690||1218||685||2460||3070||0|
|Registered:Pick Billing Address 2-29||120||120||0||400||3600||1201||714||2700||3510||0|
|Registered:Go to pick Shipping Method10||120||120||0||340||2900||1072||633||2330||2700||0|
|Guest:Pick Billing Address 111||120||120||0||610||4620||1688||941||3580||4440||0|
|Guest:Pick Billing Address 212||120||120||0||380||3530||1049||589||2050||2750||0|
|Guest:Go to pick Shipping Method13||120||120||0||300||2080||796||401||1530||1960||0|
|Set Shipping Method (Flatrate) 114||240||240||0||510||4830||1780||1066||3930||4720||0|
|Set Shipping Method (Flatrate): Goto Payment15||240||240||0||300||3150||905||516||1940||2300||0|
|Set Payment Method (Check/MO) 116||240||240||0||670||5290||1787||987||3660||4920||0|
|Set Payment Method (Check/MO) 217||240||240||0||300||2760||904||518||2010||2440||0|
|Set Payment Method (Check/MO) 318||220||220||0||370||8180||2696||1725||5810||7590||0|
|/checkout/../success/ Redirect 120||74||0||74||1030||8350||2473||1620||5590||7310||0|
|Magento EE v1.12|
|Requests||Total||OK||KO||Min||Max||Mean||Std Dev||95th Pct||99th Pct||Req/s|
|Click add to cart1||8340||8324||16||40||12130||4216||2217||7520||9000||10|
|Click add to cart Redirect 12||8324||8320||4||40||16500||7022||3449||11780||13500||10|
|Guest Checkout start4||90||90||0||390||3690||2103||795||3210||3410||0|
|Registered Checkout:Login and Pick Address5||2690||2690||0||180||19120||3298||4984||15470||16940||3|
|Guest:Pick Billing Address 16||90||90||0||600||7750||4348||1605||6430||7400||0|
|Registered Checkout:Login and Pick Address Redirect 17||2690||2688||2||40||24360||5433||3350||11780||14060||3|
|Guest:Pick Billing Address 28||90||90||0||520||8240||3412||1423||5480||6810||0|
|Guest:Go to pick Shipping Method9||90||90||0||310||4780||2334||888||3300||4450||0|
|Registered:Pick Billing Address 110||90||90||0||370||10090||3657||2453||8660||9600||0|
|Registered:Pick Billing Address 2-111||90||90||0||540||17670||4148||2741||10220||12510||0|
|Registered:Pick Billing Address 2-212||90||90||0||530||12640||3983||2411||8290||11590||0|
|Registered:Go to pick Shipping Method13||90||90||0||360||6290||2546||1401||5050||5980||0|
|Set Shipping Method (Flatrate) 114||180||180||0||450||7640||3926||1884||6550||7450||0|
|Set Shipping Method (Flatrate): Goto Payment15||180||180||0||380||6290||2451||1256||4270||4830||0|
|Set Payment Method (Check/MO) 116||180||180||0||570||8290||3923||1996||6460||7300||0|
|Set Payment Method (Check/MO) 217||180||180||0||330||5230||2318||1238||4210||4810||0|
|Set Payment Method (Check/MO) 318||180||180||0||940||40580||12673||7957||25910||29540||0|
|/checkout/../success/ Redirect 120||93||0||93||780||10470||5031||2251||7920||8880||0|