Keywords

1 Introduction

The software testing process intends to distinguish all the imperfections existing in a software. Testing is no longer observed as a movement that begins simply after the coding stage, it ought to be inescapable all through the product advancement life cycle. Testing is a lot of exercises that can be arranged and directed deliberately [1]. It is the way toward accessing a framework or system parts by manual or programmed intends to confirm that it fulfills expected requirements or not. It additionally distinguishes the contrasts among expected and genuine outcomes.

Testing can be done by using automated software testing tools, which is also known as test automation tools [2] is when the tester writes scripts and uses another software to test the software. It can perform load, execution, practical and stress testing, and so on. It improves precision and sets aside time and cash in contrast with manual testing. Manual testing requires many labor-intensive tasks, running numerous program executions, and handling a great amount of information. Testing often consumes 40-50% of development efforts, and it consumes more effort for systems that require a higher level of reliability [3]. So, it will give rise to the demand for automated software testing tools. Appropriate tools can alleviate the burden of clerical, tedious operations and make them less error-prone. Sophisticated tools can support test design and test case generation, making it more effective. Direction to supervisors and analyzers on the best way to choose testing apparatuses that will be generally helpful to their association and experts is significant, as device determination extraordinarily influences testing productivity and viability.

We proceed with this paper with our writing study in which we have characterized and investigated many softwares (e.g., Selenium testing tool, Watir, QTP, TestComplete, WinRunner, Load Runner, SilkTest, Apache Jmeter, Wapt, Tellurium, Web Load, Neo Load, LoadUI, Appvance, rational performance tester, SahiPro, Telerik Test- Studio, Ranorex) dependent on different software quality attributes like understandability, modifiability, modularity, cost-effectiveness, extendibility, reliability, flexibility, reusability, browser compatibility, and OS compatibility [4,5,6,7,8].

Analysis of automated software testing tool gives administrators and analyzers understanding that can drive significant choices in regards to device determination for their undertaking. This data will give a superior comprehension of the apparatuses. As there are different automated testing tools which have different element yet despite the fact that they likewise have hardly any restrictions. This makes them monotonous to use for testing. There is a prerequisite of such a device that can be demonstrated its capability to do the entire testing process in less time, costs less cash, and needs less exertion and labor.

This paper will further contain Sect. 2 as a review methodology that describes the method adopted for the literature review. The literature review is described in Sect. 3. Further, the result of review work is presented in Sect. 4, and Sect. 5 presented the conclusion of all the above work.

2 Literature Review

The literature review was done to establish an understanding of the state of research in the area of automated testing tool, and the goal is to address the existing research done for automated testing tools and to present results related to these. The literature review represented in this paper is such that which identifies publications of relevance, evaluates, and selects publications for review and organizes them such that common concepts are identified and structured manuscript in different sections.

Anju Bansal [1] presented details of the flow of test information while testing an application. It also represents a different form of black box testing, white box testing, and gray box testing. Software testing can provide an independent view of the software to allow the business to appreciate and understand the risk of software implementation. This paper also provides the detailed insight into types of testing along with their advantages and disadvantages. With black box testing, one need not to have knowledge of programming language, testing can be done from user point of view, but test case is very hard to design, and some part of the backend is not tested at all. Apart from black box testing, white box testing focuses on the internal logic of code, it helps in uncovering the error from the code, but for this task, skilled testers are needed, and it is very tedious and time-consuming. While the third type, i.e., gray box testing generally combines the benefits of both types of black and white box testing.

A framework based on Selenium WebDriver and TestNG is proposed by Satish Gojare, Rahul Joshi, and Dhanashree Gaigaware [2]. As Selenium has a lack of customized report generation facility, therefore the author proposed a framework which is used to generate the customized report. The component of the proposed framework is object repository which performs the task of maintaining and repairing test cases, the input file will contain the input required by the web pages under test, and the utility section contains web page-related functions. Test suite can be formed by taking input from these three component object repository, input file, and utility file. Above all these, the test result after running the test suite can be fed to TestNG unit which will then generate a test report.

Quality of software system is defined based on quality attributes such as portability, reusability, interoperability, reliability, and performance. Software systems are checked based on their quality. Therefore, the quality characteristics are considered to be very important during the development of software. Therefore, Ritika Vern and Sanjay Kumar Dubey [4] present a study of few quality characteristics based on fuzzy logic technique. This will help the researcher, developer, and tester to decide their perspective in the area of software quality.

The quality attribute is required to capture how the functional requirement of an application is achieved. To trace the quality of the software, the quality attribute plays a very important role. Gorton, essential software architecture [6], gives a description of various quality attributes which helps to capture the quality of software, and it also sets a comparison criteria for comparing various software.

SWEBOK V3.0, [8] a guide to the software engineering body of knowledge Chap. 4 software testing, this chapter will represent the importance of software testing as a part of software development, and it also describes the basic terminologies and software quality-related attributes like maintainability, portability, usability, etc.

Nisha Gogna [9] talked about the most basic features of browser-based automation testing tool, i.e., Watir and Selenium. In this paper, the author describes mainly the details of components of Selenium such as Selenium IDE, Selenium RC, and the Selenium Grid. According to which, Selenium IDE is the integrated development environment used for preparing test cases and is used as Firefox add-ons. Firefox has many add-ons to be used with Selenium IDE which will increase its usability. The author also described briefly about Selenium RC, according to which the developer has to use programming language for writing test script to get maximum flexibility and extendibility. The Selenium Grid is another component of Selenium testing tool which scales the use of Selenium RC on multiple platforms and on multiple browsers. This paper also gives the details of the platform, browser, and languages supported by the Selenium testing tool. Also, Selenium has a deep learning curve to switch from one component to another as compared to Watir.

Based on the study of Selenium RC and Selenium WebDriver, the author Mahan Sunhare, Abhishek Tiwari [10] found that the architecture of Selenium RC is more complicated as compared to Selenium WebDriver and Selenium WebDriver has also friendlier API than Selenium RC. Software developers thought to build beautiful and attractive web pages, but they did not bother about their accessibility and presentation on various browsers. Therefore, the author describes the need of focusing on browser compatibility issue for a consistent look of web pages on all browsers.

In this paper, the author, Shaukat [11], presents the taxonomy of various automated software testing tools along with the importance of software testing methods as a part of the software development life cycle (SDLC). The author’s perspective is to present with a comparative study of various automated software testing tools such that one can get a brief overview of all the tools presented in the paper. The tools which are chosen for comparative study are Selenium, Watir, Sahi Pro, HP-QTP, TestComplete, Ranorex, NeoLoad, hp LoadRunner, FitNese, SilkTest, TestNG, Apache Jmeter, Soap UI, Tellurium, WebTest, Xebium, Wapt, WebLoad Testanywhere, LoadUI, Appvance, rational performance tester, Visual Studio, and Telerik Test Studio. All these tools are studied based on operating system support, browser support, application used for testing, language support, cost-effectiveness, etc.

A study on various open source automated testing tools and a brief comparison of the same based on application support, language support, platform support, and understandability is presented in the paper by the author Bhat and Chaudhary [12]. Based on the survey, it is found that some of open source tools have some limitations while a commercial one has some good features. Therefore, the selection of the tool is completely based on the project needs and testers knowledge. According to the study of this paper, among all studied tools, Selenium has support for vast programming languages and platforms.

Chandraprabha et al. [13] talked about the systematic study of Selenium testing tool with all its components, architecture, and limitations also. As Selenium is an open source automated Web testing tools and being component-based, it is used by most of the testers. According to the study, automated testing is very advantageous as it saves time, money, and effort. It is also reliable and improves the accuracy of the testing process. Along with the details of the Selenium testing tool’s component, the author also talked about the limitations of the component. According to which, the main feature of Selenium IDE is easy to record and playback, support autocomplete command, customization is allowed through plug-ins, allow to set breakpoints and debug the script, but apart from all these features, it has few limitations as it can be only used as a Firefox plug-ins, it has its own language Selenese. With Selenium RC, it is possible to run tests inside every JavaScript compatible browser using a wide range of programming language. On contrary to this feature, it is slow, struggles while running concurrent tests, and does not allow simultaneously tests across different OS and browsers. But with the help of Selenium Grid, one can run test cases across all browsers. Selenium WebDriver being a robust, open source, cost-effective and widely used tool cannot readily support new browsers.

Altaf et al. [14] specify all the pros and cons of every component of the Selenium testing tool. Selenium is a freeware open source testing tool. There are many challenges with it, and the main is that it has drawbacks, the user interface is slow for many reasons.

Mustafa et al. [15] had discussed about sets of software testing tool and classify them over the types of testing they can perform. Here, stress, load, regression, functional, unit, performance, and security testing are defined. The author had also represented the result of his/her study through the graph. One graphical representation specifies the type of application which is mostly tested, and the result shows that Web application widely tested by using software testing tool. There is another graph that shows a widely performed type of testing, and the result shows that functional testing is widely performed on applications.

3 Result of Literature Review

This section presents a brief study of most widely used testing tools based on literature review, which includes various testing tools. Individual tools such as Selenium testing tool, Watir, QTP, TestComplete, WinRunner, LoadRunner, SilkTest, Apache Jmeter, Wapt, Tellurium, Web Load, Neo Load, LoadUI, Appvance, rational performance tester, SahiPro, Telerik Test Studio, Ranorex are classified in Table 1 which represents the basic details of tools such as developer’ s name, year in which the tool is initially developed, OS support, browser support, application support, testing type, language support and whether the tools is open source or a commercial.

Table 1 Classification of automated software testing tools
Full size table

All the above-mentioned automated software testing tools are compared based on software quality attributes like understandability, modifiability, modularity, cost-effectiveness, extendibility, reliability, flexibility, browser compatibility, OS compatibility [4,5,6,7,8].

3.1 Software Quality Attributes

The detailed definition of all the above-mentioned software quality attributes is listed below, and their comparison based on these attributes is given in Table 2.

Table 2 Comparison of automated software testing tool based on software quality attributes
Full size table
  1. 1.

    Modifiability: This defines the ability of software testing tool of being modified according to user requirement [4, 6].

  2. 2.

    Understandability: It is the attribute that determines how easy is the tool to be used without the need for expertise [4].

  3. 3.

    Extendibility: It defines the degree to that software should be extended, i.e., functionalities should be added to the existing software [7].

  4. 4.

    Modularity: It is the degree to which a system’s component may be separated [7].

  5. 5.

    Compatibility: It is the usability of the same software in different environments (like different OS, browsers, etc.) [7].

  6. 6.

    Cost-effectiveness: It is the attribute that tells us whether the software is freely available to the user or there is purchasing cost associated with it [7].

  7. 7.

    Flexibility: It is the degree of ease with which a system can respond to any change [4].

  8. 8.

    Reliability: It is the degree to which the result of testing should be accurate or consistent. [4].

4 Conclusion

In total, from the survey and reviewed literature, we found that the tool selection depends on diverse evidence, such as development choices, evaluation objectives, execution facilities, and on so many parameters. In general, there may not be a unique tool that will satisfy particular needs, so maybe a suite of tools would be an appropriate choice. So, it gives us motivation to classify automated software testing tools and study them to explore their features and limitations. Analysis of automated software testing tool provides managers and testers insight that can drive important decisions regarding tool selection for their task. This information will provide a better understanding of the tools. As there are various automated software testing tools available which have various feature, but although they also have few limitations. This makes them tedious to use for testing. In today’s era, there is a requirement of such a tool which can be proved its capability to carry out the whole testing process in less time, costs less money, and needs less effort and man power. Also, during our study, we found few tools are widely used because of their features and performance, while some of them lag. So, in our further study, we give preference to such tools that are widely used.