With the many options in scaling up an Azure Web Site, I wanted to understand the different options and the type of performance I could expect from each option. I created a simple performance test using the Bakery site which is a basic ASP.NET web site template that has a sample store. It features a catalogue with a few items in it that you can then place an order.
The default out of the box implementation uses SQL Compact as its database, which has the database running through the file system. Using only a single user hitting the page, the average page load time is about 300-400 ms. I also tested migrating the database to a proper SQL Azure database and the average load time dropped to 30-50 ms, an almost 10x improvement in performance.
In order to test multiple users hitting the site, I created a performance test that hit the site using the open source tool JMeter. The test was a basic test to see how fast the pages could be retrieved under load. I tested different loads using 1, 2, 4, 8, 12, 20, 50, 100, 200 and 500 concurrent users. Tests were run from within an Azure VM so the latency was very low.
Here are the results using a number of different concurrent threads to test the scalability of each option. The summary of the results can be found here in this slideshare presentation.
Free
Free worked quite well running up to 20 concurrent users. However, at 50 threads, the response time doubled and within a couple minutes the Data Out quota was reached and the entire site was disabled!
Free is really only useful for testing – with only 165 MB per day in the Data Out quota, you’re going to run out even with the most basic web site under any reasonable load.
Shared
Using the SQL Compact edition, Shared still scaled up really well. Even at 50 concurrent users, the average response time was only 400 ms. However, at 100 threads we started to see slow down where response time increased to an average of 1290 ms. At 200 threads, the slow down was even more pronounced at 1755 ms.
If you look at the Azure dashboard, you can see as the tests run there were 8369 requests in one minute and it barely broke a sweat!
Running using Azure SQL improved scalability even further. It could easily handle 100 concurrent users without any noticeable change in performance.
However, one of the key limitations for Shared is the quota limits that are imposed on your site. While Shared can handle spikes in traffic quite nicely, you need to be careful about exceeding the quotas.
Running a simple ASP.NET page using the SQL Compact database, I was able to exceed my CPU quota in less than 5 minutes running at 50 concurrent users and then my site was disabled. Running the more optimized SQL Azure database, I could do the same at 200 concurrent users.
The key quota limit is CPU time – it resets every 5 minutes and limits your site to “2.5 CPU minutes”. Essentially, if you have too much traffic in a 5 minute period your site is disabled.
Shared is still the best option of all the tests for handling short spikes in performance as long as your site doesn’t exceed your usage quota.
Basic Small
Basic has 3 configurations – Small, Medium and Large. We tested each of these configurations under load.
Basic Small was significantly slower performance than Shared in all scenarios. If you look at the graph from the Azure monitor, you can see the difference in requests being handled from the time the site was in Shared vs. the time it is in Basic Small.
In Shared, we peaked at 8300+ requests per minute even running under SQL Compact, while in Basic Small the site was barely managing 250 requests per minute. By the time we load up to 50 users, the Basic Small site started generating errors from the SQL Compact database because it couldn’t handle the requests fast enough.
Running using a SQL Azure database, Basic Small works better but still suffers from scalability issues. At 20 concurrent users, the average load time was 93 ms compared to an unloaded 30 ms. At 50, 100 and 200 concurrent users, the performance got progressively worse.
Basic Medium
Basic Medium fairs much better than Basic Small – it consistently delivers reasonably good results even at higher loads. As you can see by the following graph, Basic Medium peaks at significantly higher requests per minute than Basic Small under all test scenarios.
As a result, Basic Medium performance is stable at around 400-450 ms when running the SQL Compact database to load on average until 20 concurrent users and then starts to increase slowly from there. When running SQL Azure, We could scale to 50 concurrent users without a significant decrease in performance.
Basic Large
Basic Large is the first instance that we tested that had faster average load times than Shared and Free. It also scales much better than Basic Small or Medium, where it stays at an average of 300 ms even with 20 concurrent users running under our SQL Compact database. Running under SQL Azure, Basic Large performed well up to 20 concurrent users and then performance slowly degraded as we increased to 50, 100, 200 and 500 concurrent users.
Standard Small
Switching to Standard provides you the same options as basic in terms of size of VM but with the additional ability to scale out the number of instances from 3 to 10 and the ability to auto scale up when your instance becomes bogged down.
The performance of Standard Small is a little bit better than Basic Small. In my test with Basic Small, we started seeing errors with 50 concurrent users. With Standard Small, the performance is very slow but manages to get through the test. However, at 100 concurrent users, Standard Small also fails by generating errors.
Standard Medium and Standard Large
Standard Medium did similar performance to Basic Medium. Standard Large is rock solid with the best scalability of all the options. At 200 concurrent users, the instance is still consistently delivering. Performance is about the same as Basic Large.
Shared X 3
Using Shared, you can increase the number of instances up to 10. What happens if you scale up to 3 instances – do you get 3x the performance?
Running multiple instances meant that only SQL Azure was supported – SQL Compact won’t work in this scenario at all because it is local on a file system. Running Shared on 3 instances, the speed to load a page was stable at 30 ms, even when running with 200 concurrent users!
At 500 concurrent users, I was able to again exceed by quota after a couple minutes of running at that speed. However, in that time I was able to generate almost 50,000 page views running on three shared instances.
Basic Medium X 3 and Standard Medium x 3
Basic Medium or Standard Medium running on 3 instances can handle a LOT of traffic. Both were rock solid running my test up to 50 concurrent users with an average load time of 33 ms. With 100 and 200 concurrent users, there was degraded performance but it was reasonable – about 50-60 ms. Even with 500 concurrent users, performance was still a respectable 124 ms.
Standard Medium running on 3 instances performed at about the same rate as Basic Medium.
Running 3 Basic Medium’s is only slightly more expensive than a single Standard large, but the performance is significantly better. At 500 concurrent users, the cluster of 3 Basic Medium’s was serving pages at 124 ms while at the same load, a Standard Large was taking 620 ms to serve a page.
Standard Large X 3
Standard large running on 3 instances is massive. It ran easily with 500 concurrent users with barely any decrease in performance!
As you can see by the graph, we peaked at almost 40K page views per minute!
Key Conclusions
In analyzing the performance of all the various hosting plans, here are my key conclusions:
- Underlying baseline performance makes a big difference in scalability. Optimizing your page rendering time can allow you to reduce hosting costs by allowing you to run under smaller or fewer instances.
- The best performing and most economical hosting plan is Shared. However, with imposed quotas, you have to be careful not to exceed your limits or your site is disabled.
- Azure SQL runs very fast and scales well. Even with 500 concurrent users, Azure SQL was never a noticeable bottleneck.
- Scaling out shared instances also scales out quota limits – 3 Shared instances @ $30 / month might be a better bet than 1 Basic Small at $60 / month.
- Basic / Standard Small are poor scalability choices – the potential cost savings compared with Medium is eroded quickly by degradation in performance under load.
- Scaling out (e.g. adding multiple instances) is generally more reliable, higher performing and cheaper than scaling up. For example, running 3 Basic Mediums provided superior performance to 1 Standard Large and cost is comparable.
- Autoscale is only available with Standard and only works horizontally – e.g. you cannot automatically scale up from a Small to a Medium to a Large.
Azure Web Sites as a platform is an incredible option especially for high volume web sites. It scales well and the various options for hosting plans mean you can can pay as little as $10 / month for your web site. As you need more capacity, changing the configuration can be done at any time and you pay only for the capacity you are using.
