FAQ: Load testing a store demonstration

Now you've provided a free demonstration of our store on MageStack. We want to run some speed tests to supplement the report you have provided, what tools do you recommend? Would GTMetrix or LoadImpact be inappropriate?

The single most important thing to remember is that tests must represent the goal of the test.

Let me give you an example, if you were purchasing a dragster, would you to it to a race circuit to see how fast it went round the track? No – because that’s not how the dragster would ever be used. You would take it to a drag strip, and race it under a drag race – the condition for which it will be used.



Load testing with Load Impact

LoadImpact is to generate load, not provide a meaningful capacity test. Its purpose isn’t to provide comparative results, but instead highlight bottlenecks for a given test, so that further revisions can be made server-side; to repeat and generate more load.

The results themselves are almost meaningless, as they are meant for comparison against previous runs of the same infrastructure; not to compare two types of infrastructure.

Also during a demo, we use our smallest stack, with only a single 8-Core/16GB RAM physical machine in the stack. So in terms of judging capacity, this is unlikely to be the final configuration you would have – it will accurately show single-user, real-world performance - but being such a small stack, isn't going to reflect capacity.

Going back to the appropriateness of the test, unless your typical customer was going to launch a DOS attack against the store – LoadImpact isn’t going to represent what you customer would do. The test must reflect reality.

Performance testing with GT Metrix

With regards to GTMetrix, it’s a great tool – but again, it isn’t representative of what your customer’s would be doing. Its also a full-page test tool, which means you are downloading the full assets from the page; the CSS/JS/Images/External content. Server configuration cannot improve the performance of delivery of these assets, server configuration will dictate the TTFB (time to first byte), the time it takes the server to generate the page.

Beyond that, your template for the store will dictate the total page load time.

Example
If you’ve got a banner slider with 5x 10Mb images, nothing server side can be done to improve the extremely slow load time your customers will see. Improved full-page-load performance would be achieved from fixing the template, not the server.

Also, your customers aren’t going to visit your store, view a single page, then leave – all the while with a cold browser cache. This is what the synthetic test form GTMetrix does. It can’t account for sessions, multi-page views or browser cached content.



Tests must always reflect reality

Running the right tests will give you the right results; running the wrong tests will give inaccurate/meaningless results. We spend a lot of time doing proper, real-world load tests for our larger customers, so that we can provide guaranteed solutions.

The crux of what I’m trying to say, is that the most realistic tests you can run on the demonstration is the test we outlined in the demonstration report (that shows real server TTFB improvement) – or, simply browsing the website using your own web browser and seeing for yourself (just like a customer would).