The objective of load testing is to,
• identify and correct any issues created by simulating "live" conditions.
• determine the satisfactory working level of a system and it's components,
• determine the satisfactory working levels of different applications
The most important factor of load testing is to simulate the environment conditions as they would be when the system is actually used by real users.
Load testing is an extension of performance testing, where the baseline created for acceptable usage is tested against by simulating "live" conditions. This can be achieved by,
• increasing the number of users using the system concurrently,
• increasing the size of data used,
• increasing the time taken to run tests.
Any deviation from the baseline created during performance testing must be analysed for acceptability. Whilst a minor deviation may be acceptable, anything major may become a serious issue that requires addressing before any system is allowed to go into production.
Unfortunately for most, load testing is generally viewed just as a numbers game where the number of users that a Citrix server can safely handle without impacting the usability of the system is determined.
"We managed to get 45 users on the server".
Whilst this may be hold some truths, load testing is far more important than just a numbers game and is critical in determining usability and stability of a system in simulated "live" conditions.
For example, during performance testing, it was established that a transaction to populate a database from when the submit button was pressed on an application (installed on a Citrix server) took about 2 seconds.
The data entered into the application included fixed data which was entered by all users, such as names, addresses, personal details etc. There was also variable data which included free form text, the amount of this type of data input for this varied from user to user.
By increasing the number of users using the application on the Citrix server, it was found that the time taken to update the database remained constant up to 60 concurrent users but deteriorated after that, as shown below,
• 61 to 65 users = 16 seconds
• 66 to 75 users = 20 seconds
• 76 to 80 users = 32 seconds
How many users would it be safe to say that this Citrix server can support?
Well, the transaction time increased eight fold after the user load reached 61. Would that be acceptable and would users be able to live with this delay?
More importantly this is just one transaction, there would in reality be hundreds that the user would complete on a daily basis. An increase across the board on all transactions by a factor of eight, would have severe repercussions on the user's ability to actually work.
Based on initial findings it could stated that the system can handle 60 users. (More testing on the amount of data entered into free form text box would also need to be done, to assess the impact on usability).
So in terms of transaction timings, a key indicator of usability, load testing can determine quickly the most efficient number of users the Citrix system can support.
Performance testing is critical in determining the time taken to run a test is critical. Whereby longer test runs are required to accurately determine resource utilisation issues, such a memory leaks. Which are more likely to appear further into the operating cycle of a Citrix server (time between reboots).
With this mind, advocating 60 users from the above example might not be an accurate estimate of how many users the Citrix system can actually support during it's operating cycle. I prefer to run load tests over an extended period which cover the operating cycle of a typical Citrix server. This allows me to determine any issues caused with resource utilisation whilst running at load.
With the example above of 60 users and looking at the transaction times over a period of 5 days:
Day 1
1 to 60 users = 2 seconds
Day 2
1 to 60 users = 2 seconds
Day 3
1 to 60 users = 2 seconds
Day 4
1 to 42 users = 2 seconds
43 to 55 users = 3 seconds
56 to 60 users = 6 seconds
Day 5
1 to 38 users = 2 seconds
39 to 51 users = 5 seconds
52 to 57 users = 9 seconds
58 to 60 users = 16 seconds
The key day to look at is day 5 and here we can see that with up to 38 users the transaction time was as expected. However after 38 users, the transaction time increased markedly, resulting in the transaction time for 60 users increasing eight fold.
So by taking the results from Day 5, we could say that the system is only capable of supporting 38 users. However, we have a number of options to hand. We could reduce the operating cycle of the Citrix servers to 4 days, allowing us to get 42 users per server as expected. Or the better option of reducing the operating cycle to 3 days would allow 60 concurrent users whilst still retaining usability.
By running extended load and performance tests over the operating cycle of a Citrix server, not only can resource issues be quickly determined but the operating cycle can be adjusted to meet the expected requirements of the users. Instead of simply waiting until the systems are being used by real users and being barraged by support calls on usability deteriorating over the week.
In a worse case scenario what happens in the above example if the Citrix server runs out of resources by day 5 and stops responding? It dies as it is unable to cater any user requests, so the Citrix load balancing would move all new user requests to other Citrix servers. If the other Citrix servers also exhibit the same problems as the original; Citrix server then these could also become unresponsive and cause the Citrix farm to deteriorate.
The time taken to run the test is critical as the shorter the test period the greater the chance of gremlins that lurk underneath the surface not being found. Ideally the test period should be increased to match the operating cycle of the Citrix system, that is the time taken between server reboots, should dictate the time the some of the load tests should be run for.
This is an important part of load testing, to simulate how the real environment will be used. Therefore if a server is rebooted once a week, then a load test needs to be run for a week to see how the system behaves over this period. Simply running a load test for an hour or so and using the results produced could create serious issues further on down the line. As one of the great things that load testing can discover is resource leakage.
This inherently is an issue with Citrix based systems as the applications used are not specifically written for a multiuser environment as Citrix instead more for standalone environments such as desktops. Therefore any resource leakage is magnified, so whilst a memory leak of 20 megabytes on a desktop wouldn't be an issue, on a Citrix server this could potentially be magnified by the total number of users on the server, leading to 100's of megabytes of lost memory resource.
In the old days, Windows NT Terminal Server Edition which Citrix ran on was renowned for memory leaks and as such had to be rebooted at least once every day, but today's Terminal Server which has become Windows Terminal Services is far more stable and generally rebooting once a week will suffice.
In conclusion a clean system is vital to conduct Citrix load testing.