There are several reasons to test Web application software. Web application software consists of programs and data that are designed to operate across the World Wide Web, a combination of network and client-server software and hardware that enable teleoperations between computers local to and remote from a user. Since a Web software application is first and foremost a software application, the same reasons for testing application software apply to Web software.
There are also considerations for Web applications that are different than other software applications or simply do not apply to other applications. First and foremost among these is the fact that a Web application is as much an extension of a company’s or organization’s “storefront” or office space as it is a system of software that performs some set of tasks. One of the most self-serving tasks a Web application, particularly a commercial one, performs is to keep the user at the Web site.
One could make the argument that encouraging the prolonged use of any application may not be in the best interest of the application because it merely increases the chance that incorrect or poor functionality will be discovered, but that’s the price for success as well as the implied challenge for any software application. Any software application that performs poorly or malfunctions runs the risk of driving users to turn to alternative applications.
However, nowhere is this facet of application competition of more concern than in the world of the Web, where the user “is only one click away gone” – either exiting from the Web altogether or navigating off of the current Web application and onto another, possibly competing company’s Web site. [3] While this is of most concern to commercial Web application deployers, for any Web application: If the user loses interest in it, the functions that a Web application is to perform are superfluous.
While there should be no realistic expectation that even a perfectly functioning and performing Web application will hold the interest of all visitors to a Web site, you can make a darn good bet that a poorly and incorrectly functioning or performing Web application will drive away and discourage users faster than trying to catch flies with vinegar. Stress management consultants have identified a phenomena they’ve coined “Web rage”, which manifests itself when users become frustrated with poor Web service and performance.
Almost all of the users who were asked, in a survey concerning Web rage, what they were most likely to do when they didn’t like a company’s Web site indicated they’d move on to a competitor’s site. None of the respondents planned to spend any more time on a bad Web site, although a few said they’d try to contact the company for help. The same Web users described frustrating Web sites as being hard to read (due to font choices), poorly organized, or out-of-date, or making it difficult to find a telephone number to use to get in touch with the company.
The testing of Web applications in order to assure their deployers that users’ experiences with the Web applications will be pleasant is therefore of extreme importance. Even internally deployed Web applications, such as Web-based training or corporate sites are useless when employees refuse to use them. So Web applications should be tested as often and as thoroughly as time, money, and other resources allow.
Types of Performance Testing
Some of the more common types of tests are acceptance testing, conformance testing, usability testing, performance testing, reliability testing, and robustness testing. Acceptance testing demonstrates the applicability of an application to the automation, usability, performance, reliability, and robustness requirements of an organization to the satisfaction of the organization. Conformance testing matches the implementation of an application to its stated requirements. Usability testing determines the ability of the application to meet the needs of users in performing their tasks in an automated fashion.
Performance testing measures how much time and other system resources are required by an application to accomplish the tasks it performs. Reliability testing stresses the application in the face of system resource and other failures, including the ability of the application to continue to perform its tasks over a long period of time. Robustness testing investigates the application’s ability to perform tasks in ways not originally intended and the application’s potential and capacity to be modified and scaled beyond original requirements.
Performance Testing Strategy
Whittaker defines various types of testing and describes a clear and concise approach to performance testing software, treating tests as attacks on the software under analysis and describing a battle plan of sorts for testing software applications. [6] A similar approach and presentation format will be taken in this text, however the primary focus of this document is on those characteristics and approaches unique to software applications designed to operate over the Web. Like Whittaker’s text on “breaking” software applications, this paper will describe where and when Web applications should be tested but will focus on breaking Web applications from the client side.
Adept testers who test Web applications at this point and have achieved mastery of discerning situations and scenarios that might cause Web applications to, at best, perform inadequately may be referred to as “breakers”. A breaker might seem to be something out of a Web application’s developers’ worst nightmare, but I contend that a tester who can discover and report what any user might accidentally stumble across while using a software application will prove even more invaluable for a Web application’s developers than for developers of traditional software applications. This is because Web applications tend to be used more immediately and much more often than traditional software applications due to their being hosted on the Web. This text will also focus primarily on the areas of Web application software functionality and security, as they are most often viewed as the two primary foci of whether or not a Web application actually works correctly.
In contrast to a master’s thesis recently written by Jesper Rydén and Pär Svensson, titled “Web Application Testing”, this text will present examples and explanations of recommended attacks on Web applications. The tests and examples follow the general style of presentation established by Dr. Whittaker for describing and illustrating tests on software applications.
Information on Application Performance Testing
Several books and other technical publications explore application software testing. However, very few comprehensive texts on testing Web applications existed by the summer of 2002, although several have been announced for publication by the fall of 2002. Most of the material found in the subject areas of testing Web application software is available on the Web itself, as are many of the technical and popular journals and other, traditionally print, publications.
Although a number of articles, papers, and texts treating various aspects of performance testing Web applications are beginning to appear (and a few have existed for several years), the amount of material available in this subject area pales beside the staggering volumes describing traditional software application testing. This situation is anticipated when the subject area is as new as the testing of Web application software.
Understanding Web Software Behavior
In order to best determine how to test Web software, just as with any other software application, testers must understand the characteristics of Web software behavior in general and its interaction with other software and hardware. This means testers must study what a Web application does and the means it uses to operate properly. Arming themselves with this knowledge provides testers with loads of information on how to best spend their resources (time, money, and brainpower) to go after critical points in a Web application’s armor.
Web software, like any other software application, has particular potential trouble spots, which will result in application or system failure if the application fails to properly anticipate and handle trouble. To a software application, trouble manifests itself in many ways, including bad input data, unacceptable combinations of good input data, invalid sequencing of events, and failures in system resource availability and response. Applications cannot avoid these types of problems. How well a software application detects, protects itself from, and recovers from such occurrences is what separates sane applications from pathologic ones.
For Web applications, clumsy Web page and site presentation and navigation, insufficient handling of client-side, network, and server-side interface errors and fumbling data exchanges compound the opportunities to fail that software applications of any sort already provide. Good testers have either gained the experience, possess the amazing intuition, or both, to focus their attacks on software applications to devise tests that will cover the areas of most concern in the operation and performance of a software application as it attempts to perform its tasks. [6] Good Web application testers must also become well versed in the myriad of interfaces available to the Web application developer these days, due to the proliferation of various Web browsers, client systems, server systems, network systems, and client- and server-based Web software components.
To guide them in their bug quest, a fault model of a Web application is proposed. A fault model for a Web application describes in general terms the environment the Web application operates in, the capabilities of the Web application, how it is expected to be used by those whose requirements it was written to meet, and how it is expected to coexist with other applications in its world. The next section describes the general capabilities of Web applications and how they interact with other parts of the systems in which they reside. The following chapters describe strategies and tactics testers may follow to effectively explore aspects of Web applications most likely to expose faulty design and development of a Web application’s implementation of its requirements (bugs in the application).
A Web Application’s Environment
What is a Web application? And what makes it different in nature from any other kind of software application? Identifying the features of a Web application that are similar to other types of software applications and the features that distinguish a Web application from other classes of software applications is a crucial step in determining how and where to apply testing resources in order to make the best use of a tester’s time and energy. The types of interfaces and interactions with other components in the Web application’s universe, and the nature of those components themselves figure heavily into generating a model of the environment in which the Web application operates.
The Makeup of a Web Application
Several views on the makeup of a Web application have been expressed. According to Nguyen, a Web application consists of Web pages, custom software that runs on the client system, and custom software that runs on one or more server systems. Nguyen maintains that not only are the browser, Java interpreter, client operating system, networks, and complex third-party products, database servers, and Web servers not part of most Web applications, but the expenditure of major testing resources on finding bugs that are generated from the components and services a particular Web application uses detracts from the primary purpose of the tester: testing the unique parts of the Web application.
Stottlemyer views a Web application as a Web site whose functionality is intended to meet the business requirements, that is, the set of functions requested by the people with a vested interested in the development of the Web application. Splaine and Jaskiel characterize a Web application as a class of Web sites that is but one of the applications utilizing the part of the Internet that is the World Wide Web. Mosley views a Web application in a similar fashion, but also as an application belonging to the class of client-server applications in general.
These and other technical treatments on the nature of Web application operation differ slightly, but they do help us understand the environment of a Web application by the properties they have in common. In straightforward terms, a Web application begins with Web pages served up to a client system by a Web server. The Web server responds to requests from a Web browser that resides on the client system. The Web server interacts with Web application custom code, the client-side browser, other client-side third-party components and system services, server-side third-party components and system services, the networks over which these various pieces of software exchange data, and the Web navigation techniques employed by the Web application to perform its tasks.
Performance Testing: The Server Side
Following Whittaker’s conceptualization of a software application’s operating environment, each of the custom software components of the server side of a Web application is tested in four areas, each accessible to the application through its host operating system: the kernel, various APIs, the file system, and the User Interface (UI). Web application software performs the same basic operations that any other application software performs: reading data, processing data (performing calculations or translating input data to output data in other ways), writing data, and storing data in internal data structures. [6]
On the client machine, it makes very little sense to test most of these factors (you tend to end up testing how the Web browser reacts to system resource deprivation rather than how the Web application itself reacts), but on the server side these factors not only affect how the custom, server-side components perform but also the performance and reaction of the Web application’s client-side software. So although the client side of a Web application calls for a new model, testing the Web application for its reaction to server-side operations can mostly be accomplished using the familiar software application model applied on the server machine.
Performance Testing: The Client Side
The Web browser is more than just the client-side software application that displays the Web application’s Web pages and allows users to interact with the Web application through the use of various input devices such as mice and keyboards. It is the entire world to the Web application. Human users are in reality interacting with hardware input devices that communicate with the device services layer of the client system, which interacts with the Web browser through one or more set(s) of APIs (application programmer interfaces), including virtual machine interfaces, with the Web browser commanding sole access to the Web application through its Web pages. The Web application directs the browser to perform Web navigational and service tasks through agreed-upon protocols that are semi-standard across different Web browsers. In fact, the degree of compatibility between different Web browsers is an important consideration in the design and implementation of a Web application, because it determines how likely the Web application will be to perform its tasks on a given client system.
Note that testing the reaction of a Web application by invoking failures and near failures of system resources on the client side should be done to some extent, but a tester has to be very careful to not turn client-side testing into trying to break the Web browser instead of the Web application. Remember that the Web browser is a software application in its own right that runs on the client machine. Presumably, the Web browser has been tested by its manufacturer and at any rate is not under control of the developer of the Web application.
Performance Testing: The Human User
The human user portion of a Web’s operating environment includes navigation within the Web site and to other Web sites, the user interface portion of a Web application, and the compatibility of a Web application with a given Web browser. Browser compatibility testing for Web applications includes validating that the HTML presented by the Web application to the language is valid.
This is especially important because Web browsers are not consistent in the way they handle HTML, much less in the way they handle incorrectly written HTML. How the Web application allows the user to move around from object to object on the Web page is important because users expect the user interface to be consistent from application to application, even though Web applications available to any user on the Web may have been written by developers all over the world.
Contrast this situation with non-Web software applications. Although non-Web software may be developed all over the world, non-Web software is far more likely to have been written in closer proximity to the user than Web software due to the differences in distribution methods (although this, too is changing – non-Web software is increasingly being offered via download over the Web, which although being different than pulling up the Web site of a Web application logically shares the distribution method of Web applications).
Web page navigation also includes provisions for providing data input values to and viewing data output values from the Web application via objects on the current Web page, whose importance will be discussed in section 4. Besides HTML forms and client-side scripts, input and output on Web pages may also be handled by client-side plug-ins and Java.
Surfing from Web page to Web page or window or even to other points on the current page may be provided for via active links on the current Web page, client-side and server-side image maps, plug-ins, scripts, and Java. Even if the method of navigation is not entirely consistent from browser to browser for the same Web page, navigation should be as consistent as possible both within the same Web application and between Web applications hosted under the same browser.
As an example, one popular navigation technique is the use of a drop-down list box and a “go” button. The user expects from the visual layout of these items that some item in the drop-down list is to be selected, then the user is to click the “go” button or press the “enter” key. Some browsers jump to the Web location corresponding to the selected list item only after the user clicks the “go” button with a mouse or presses the “enter” key, but other browsers jump to the selected list item location upon selection of the item in the pull-down list.
Web applications should also be developed with the expectation that the user will not have the “latest and greatest” Web browser loaded on their system, nor the latest client-side plug-ins (including the latest Java applet plug-in), nor will they necessarily have enabled every Web interpretation option, such as “allow Java”, “allow all cookies”, and “allow JavaScript”. And the user’s loaded browser may not support frames. Web applications should both anticipate these situations and provide convenient and user-friendly means to handle them. A tester must consider and test how the Web application handles many different types of client-side software configurations.
One particularly annoying example of this problem is how Web applications handle ActiveX. While some Web applications, most notably Microsoft’s home site, actually do detect that the user does not have ActiveX capabilities, the most arrogant of these (like Microsoft’s home site and MSN) instruct the user to find another browser to use that supports ActiveX (What they really mean is: “You should be using Microsoft’s Internet Explorer”.) rather than providing a convenient means for the user to download a plug-in that will handle ActiveX.
Performance Testing: The Network
The Web application tester must be careful about testing network problems as part of testing the Web application because some network-based tests on the client side, such as strangling network resources, may not actually test the Web application’s behavior but that of the browser (depending on whether the Web application is redirecting to a URL or involved in some sort of messaging traffic, like HTTP, with the server-side part of the Web application). On the client side, more reasonable network testing includes garbling the information contained in or invalidating the incoming and outgoing network packets to see what the Web application will do in the face of bad input or output. The key is to devise tests that expose the behavior of the Web application itself rather than that of the Web browser.
The savvy Web application tester tend to ask the Web application questions like: “Okay, now what are you going to do if I change some of the data you’re expecting from your server?” or “Alright, what if I drop, scramble, or slightly change your message to your server?” A good tester does not generally ask the browser: “What will you do if I bog down your view of the network so you time out trying to load the new Web page the user just tried to go to via an active link on the current Web page?” unless the tester is testing the Web browser itself.
Performance Testing: Load Testing
Load testing, which essentially determines whether or not the components of a Web site (particularly the hardware) can handle a specified processing load, is an area of testing that is often misunderstood and, more importantly, incorrectly done. While load testing a Web site is crucial to ensuring the Web site’s success, this type of testing tests very little of the functionality of the Web application itself. Load testing is more for finding out how much load the Web site (its machines, systems, and software) can bear and still allow the Web application to perform in a timely fashion as advertised than for testing whether or not the Web application functions as required. [14]
One common misperception about load testing is that the testing is about what loads will break the Web site, particularly loads that place the Web site’s hardware and server-side components under stress. This type of testing is more along the lines of stress testing, which should be done in addition to load testing, but its purpose is to discover what the Web application does in reaction to having its resources become unavailable and broken due to duress. [14]
Another of the most common misperceptions regarding load testing is that the number of concurrent users equals the load on the Web site. As the number of users of a Web application increases, especially as the number of concurrent users increases, the load on the application’s site may very well increase, but it has been found that the two are not as tightly coupled as one would assume. [15]
Load testing is also generally performed by the Web site creators or their testers, because load testing is usually not considered valid unless the Web site layout, in terms of machines, software, and network components, are exactly duplicated in the test environment. Breakers will usually test Web applications from the perspective of the Web user on the client side of the Web site, which does not have the ability, resources, or access to exactly duplicate the server side of the Web application. [9][15]
In fact, Anderson claims that one of the top mistakes made in load testing Web applications is in starting too late. He observes that while many development projects schedule load testing after an application is complete, load testing should begin as soon as the processes that make up the application are running, even before the user interface has been developed. Although he advocates performing load testing just before putting a Web site into production and load testing during regression testing after changes have been made to a Web site, he explains that conducting load testing during implementation is also useful for choosing between alternatives when considering what hardware the Web site should contain.