Load testing isn’t the easiest job invented. Depending on your business model, load testing can vary from important to absolutely critical. So despite the pains, every project at least makes a token gesture toward load testing. Unfortunately, either knowingly, or unknowingly, it’s often not much more than that, a token gesture.
The primary failure in the average load test is not creating a realistic load. There are plenty of excuses for this. There aren’t any servers comparable to the production servers. It’s too hard to produce test data that simulates true data. Or worse, you don’t even know what production load will look like. Those aren’t minor obstacles, calling them excuses isn’t meant to trivialize them, it’s more a reflection of the true importance of load testing, and knowing that when you do it, you do it right.
The less and less realistic the load you generate, the more your test becomes performance analysis. Performance analysis is great, but load testing and performance analysis are different animals. You are doing yourself a disservice if you use a fishing rod to catch a great white, or a harpoon for goldfish. If half your team is trying to do performance analysis and half is trying to load test then you will waste time you wouldn’t with a clear mission.
There are other things that distinguish performance analysis from load testing, but the number one is the type of load you generate. A performance analysis load may sometimes resemble a true load, but it usually should not. A performance analysis load should be structured to make it easy to pinpoint performance issues. A true load makes this more difficult by being too complex or too chaotic. So unless you’re tuning something that only performs badly in complex or chaotic scenarios simplify and isolate for performance analysis.
Load testing, by definition, needs a true load, or as close to it as you can approximate. Load testing is a validation. Load testing is developing reasons to be confident that under expected conditions, your system won’t fall over. Load testing is about giving assurances that it’s not a bad idea to depend upon the reliability of your system.
So all that said, how do you create a realistic load? If your software is like most, there is one, or probably many points where in the real world, a user takes some action. Since in the real world you have lots of users, you’d ideally want to automate all of those steps. Sometimes that’s not that difficult, and if it’s not, that’s the path to take. There are tools that can simulate clicking “submit” on a web form. Many of those same tools can simulate filling it with some data, or even using an AJAX control. But all of this has limits. If you’re within those limits, take the easy path. If you’re not, you’re either going to have to take the next step, or settle for a sub-par load test.
Moving work from the server to the client is sometimes an effective strategy to improve scalability, among other possible benefits, but it’s definitely going to complicate your ability to automate your steps with cookie cutter solutions. There are two paths you can take in that situation. One choice is to start from scratch and use your knowledge of your software to generate data from a template, substituting in values coming from preceding steps, and maybe some randomness. The second is to reuse code from your application and wire those pieces together.
Which choice you make is going to depend on your code. Using your application code has many things going for it, but you have to surmount several common challenges. First, if you have thousands of users (or millions), you’ll need your application code to simulate more than one. No matter how minimal your application, it’s very unlikely you can run hundreds or thousands of copies of it on a single box at a time.
Depending on how well you meet the first challenge, you may also need to make sure your application code can run in parallel on the same hardware. Usually this means multiple processes. Why wouldn’t you need multiple processes? If you can make one instance of your application code simulate a large number of users, and thus consume the overall load generating capacity of the hardware, then the second challenge can be skipped.
Either way, with a large number of users, you may find that a single machine isn’t going to have the ability to generate sufficient load. If one, two or three machines are necessary you may feel happy with manually starting instances. At the least do yourself the favor of using a tool like PsExec to let you do that from a batch file or something.
Whatever path you’ve taken, you are now generating a load. The challenges aren’t over though. Now need to validate you’ve met your load and that it hasn’t caused anything to topple over. That means monitoring and analyzing stats and logs. And if you do find a problem, you’ll need to switch hats back into performance analysis to hone in on the exact cause and find a solution. Since those aren’t topics to be taken lightly, I’m reserve them for a future date.