|Validation Research||Tools investigated are novel and have not yet been implemented in practice. Tools used are for example experiments, i.e., work done in the lab.|
|Evaluation Research||Tools are implemented in practice and an evaluation of the tool is conducted. That means, it is shown how the tool is implemented in practice (solution implementation) and what are the consequences of the implementation in terms of benefits and drawbacks (implementation evaluation). This also includes identification of problems in industry.|
|Solution Proposal||A solution for a problem is proposed, the solution can be either novel or a significant extension of an existing tool. The potential benefits and the applicability of the solution is shown by a small example or a good line of argumentation.|
|Philosophical Paper||These papers sketch a new way of looking at existing things by structuring the field in form of a taxonomy or conceptual framework.|
|Opinion Paper||These papers express the personal opinion of somebody whether a certain tool is good or bad, or how things should be done. They do not rely on related work and research methodologies.|
|Experience Paper||Experience papers explain what and how something has been done in practice. It has to be the personal experience of the author.|
|Tool ||2004||The authors implemented a tool that can automatically generates test data to distinguish among models. Such automation is based on the ability of simulating or running models.|
|ScenTED DTCD ||2005||ScenTED-DTCD tool (Domain Test Case Scenario Derivation) was implemented in order to support the ScenTED technique (Scenario based TEst case Derivation), which aims at reducing effort in product family testing.|
|XUnit ||2005||It is a widely used open-source unit testing framework. It has been ported to most object-oriented programming languages and is integrated in many common IDEs.|
|ASADAL ||2006||It supports the entire life cycle of software development process based on a Product Line Software Engineering method called FORM (Feature-Oriented Reuse Method).|
|Objecteering ||2007||It is a prototype tool to automatically generates both functional and robustness test cases specific to a product from the Product Families requirements.|
|GATE ||2007||It is a prototype tool that implements an efficient mechanism of testing components with high automation. The framework facilitates the reusability and automatic creation of aspect test cases, with software product lines techniques.|
|ParTeG ||2008||It supports the automatic generation of boundary tests. It generates JUnit test suites for both configurations. The tool also allows the user to choose a coverage criterion, which should be satisfied as good as possible by the generated test suite.|
|Kesit ||2008||The prototype shows that incremental test generation provides significant performance improvements over conventional means of test input generation.|
|MoSo-PoLiTe||2011||It provides a test framework for SPL. It is a tool chain that contains a pairwise configuration selection component on the basis of a feature model.|
 C. Nebut, Y. Traon, and J. Jezequel, “System Testing of Product Lines: From Requirements to Test Cases,” Software Product Lines, pp. 447–477, 2007.
 E. Uzuncaova, D. Garcia, S. Khurshid, and D. Batory, “Testing Software Product Lines Using Incremental Test Generation,” 19th International Symposium on Software Reliability Engineering (ISSRE’08), pp. 249–258, 2008.
 A. Reuys, E. Kamsties, K. Pohl, and S. Reis, “Model-Based System Testing of Software Product Families,” International Conference on Advanced Information Systems Engineering CAiSE, pp. 519–534, 2005.
 Z. Stephenson, Y. Zhan, J. Clark, and J. McDermid, “Test Data Generation for Product Lines – A Mutation Testing Approach,” in 3rd International Workshop on Software Product Line Testing (SPLiT 2004). Citeseer, 2004, p. 13.
 M. Galli, O. Greevy, and O. Nierstrasz, “Composing Unit Tests,” in 4th International Workshop on Software Product Line Testing. Citeseer, 2005, pp. 16–22.
 K. Kim, H. Kim, M. Ahn, M. Seo, Y. Chang, and K. C. Kang, “ASADAL: A Tool System for Co-Development of Software and Test Environment based on Product Line Engineering,” 28th International Conference on Software Engineering (ICSE’06), pp. 783–786, 2006.
 Y. Feng, X. Liu, and J. Kerridge, “A product line based aspect-oriented generative unit testing approach to building quality components,” Proceedings of the 31st Annual International Computer Software and Applications Conference, pp. 403– 408, 2007.
 S. Weissleder, D. Sokenou, and B. Schlingloff, “Reusing State Machines for Automatic Test Generation in Product Lines,” 1st Workshop on Model-based Testing in Practice (MoTiP’08), p. 19, 2008.
 S. Oster, I. Zorcic, F. Markert, and M. Lochau, “MoSo- PoLiTe: Tool Support for Pairwise and Model-Based Software Product Line Testing,” in Proceedings of the 5th Workshop on Variability Modeling of Software-Intensive Systems, ser. VaMoS ’11. New York, NY, USA: ACM, 2011, pp. 79–82.
|IDATAG ||1998||It supports both the specification of the behavior of a user interface and the generation of two types of test case for GUI coverage and for checking the usability of the application.|
|SOSTEM ||2000||It is designed using SOFL and implemented using Java. It supports five functionalities: generation of proof obligations, interactive testing, batch testing, test results analysis, and test information management.|
|AsmL ||2003||It supports the generation of parameters, call sequences, and the conduction of conformance tests. The tool performs a semi-automatic approach, requiring a user to annotate models with information for generating tests.|
|DUTF ||2005||It assists in the construction of test cases for DSL programs. Failed test cases reported within the DUTF reveal the presence of a program fault.|
|STSC ||2005||The “STSC” prototype tool supports the statistical testing of software components. It supports a wide range of operational profiles and test oracles for test case generation and output evaluation.|
|JTest ||2006||It can automatically generates test inputs to perform white-box testing. When specifications are provided with the class, Jtest can make use of them to perform black-box testing.|
|COMPTest ||2006||It is a component test tool that can be used to automatically identify component-based API changes and impacts, as well as reusable test cases in a component test suite.|
|SimC ||2006||It is a prototype tool, which automatically generates test data for unit testing of C programs involving pointer and structure operations. The tool symbolically simulates the execution of the given program.|
|Taxi ||2007||It implements the XML-based Partition Testing approach for the automated generation of XML Instances conforming to a given XML Schema.|
|Websob ||2007||It is a tool for automatically generating and executing web-service requests given a service provider’s Web Service Description Language (WSDL) specification.|
|Korat ||2007||Korat is a tool for constraint-based generation of structurally complex test inputs for Java programs.|
|ATTEST ||2007||It is a toolkit to address problems in test automation and maintenance. It provides easy-to-use mechanisms for helping testers to write and maintain automated test scripts through describing system behaviors at a high abstract level.|
|TaRGeT ||2007||It is a tool for automatic test case generation from use case scenarios written in natural language. It automates a systematic approach for dealing with requirements and test artifacts in an integrated way.|
|CodeGenie ||2007||It is a tool that implements a test-driven approach to search and reuse of code available on large-scale code repositories. With it, developers design test cases for a desired feature first.|
|Jwalk ||2007||It is a unit-testing tool developed to address the need for systematic unit testing within the context of agile methods.|
|Smart ||2007||It is a novel approach for generating tests for web services. It enables non-programmers to generate unit test cases for web services by performing drag-and-drop operations on GUI elements.|
|UnITeD ||2008||It enables the automatic generation of optimized component test sets from UML behavioral description by state machines or activity diagrams according to structural model-based testing criteria.|
|AgitarOne ||2008||It is a comprehensive unit testing product for Java. It can use run-time feedback and limited on-the-fly static analysis to generate input data or creates regression tests using constraint solving and mock objects.|
|REST ||2008||It is a Reducing Effort in Script-based Testing for guiding test personnel through changes in test scripts so that they can use these modified scripts to test new versions of their respective GAPs.|
|JUnitMX ||2009||It is a unit testing tool that leverages a change model to assist developers in the creation of new unit tests. The tool provides quantitative feedback and detailed information about change effects.|
|WebDiff ||2010||It is a technique and tool for detecting cross-browser issues automatically and reporting the locations of such issues in the corresponding web page.|
|CoGenTe ||2010||It is a tool for meta-model based testing of code generators. It generates test cases to cover the syntatic aspects of a translation and complex semantic aspects.|
|TED ||2010||It enables integrated testing and debbuging of software stack and underlying hardware while providing more flexibility and control to user.|
|AutoBlackTest ||2011||It is a tool for automatic generation of test cases for interactive application. It interacts with the application though its GUI, and uses reinforcement learning techniques to understand the interaction modalities.|
 S. Liut, T. Fukuzakit, and K. Miyamoto, “A GUI and testing tool for SOFL,” 7th Asia-Pacific Software Engineering Conference (APSEC’00), no. 11694173, pp. 421–425, 2000.
 T. Xie and D. Notkin, “Tool-assisted unit-test generation and selection based on operational abstractions,” 21st IEEE International Conference on Automated Software Engineering (ASE 2006), vol. 13, no. 3, pp. 345–371, 2006.
 Z. Xu and J. Zhang, “A Test Data Generation Tool for Unit Testing of C Programs,” 6th International Conference on Quality Software (QSIC’06), pp. 107–116, 2006.
 A. J. H. Simons, “JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction,” 22nd IEEE International Conference on Automated Software Engineering (ASE 2006), vol. 14, no. 4, pp. 369–418, 2007.
 B. Daniel and M. Boshernitsan, “Predicting Effectiveness of Automatic Testing Tools,” 23rd IEEE/ACM International Conference on Automated Software Engineering, pp. 363– 366, 2008.
 S.Choudhary, H.Versee, and A.Orso,“ACross-browser Web Application Testing Tool,” in Software Maintenance (ICSM), 2010 IEEE International Conference on, 2010, pp. 1–6.
 A. C. Rajeev, P. Sampath, K. C. Shashidhar, and S. Ramesh, “CoGente: A Tool for Code Generator Testing,” in Proceedings of the IEEE/ACM international conference on Automated software engineering, ser. ASE ’10. ACM, 2010, pp. 349–350. New York, NY, USA:
 L. Mariani, M. Pezze`, O. Riganelli, and M. Santoro, “Auto- BlackTest: a tool for automatic black-box testing,” in Proceeding of the 33rd international conference on Software engineering, ser. ICSE ’11. 2011, pp. 1013–1015.
 Y. Ren and F. Chang, “ATTEST: A Testing Toolkit for Validating Software Properties,” 23rd IEEE International Conference on Software Maintenance, pp. 469–472, 2007.
 S. Nogueira, E. Cartaxo, D. Torres, E. Aranha, and R. Marques, “Model Based Test Generation: An Industrial Experience,” in 1st Brazilian Workshop on Systematic and Automated Software Testing, 2007.
 A. Beer, S. Mohacsi, and C. Stary, “IDATG: An Open Tool for Automated Testing of Interactive Software,” 22th Annual International Computer Software and Applications Conference (COMPSAC’98), pp. 6–11, 1998.
 M. Barnett, W. Grieskamp, W. Schulte, N. Tillmann, and M. Veanes, “Validating Use-Cases with the AsmL Test Tool,” 3rd International Conference on Quality Software, pp. 238 – 246, 2003.
 H. Wu and J. Gray, “Automated Generation of Testing Tools for Domain-Specific Languages,” in Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. ACM, 2005, p. 439.
 R. Shukla, P. Strooper, and D. Carrington, “Tool Support for Statistical Testing of Software Components,” in 12th Asia-Pacific Software Engineering Conference (APSEC’05). IEEE Computer Society, 2005, p. 8.
 J. Gao, D. Gopinathan, Q. Mai, and J. He, “A Systematic Regression Testing Method and Tool For Software Components,” 30th Annual International Computer Software and Applications Conference (COMPSAC’06), pp. 455–466, 2006.
 A. Bertolino, J. Gao, E. Marchetti, and A. Polini, “TAXI – A Tool for XML-Based Testing,” 29th International Conference on Software Engineering (ICSE’07 Companion), pp. 53–54, 2007.
 E. Martin, S. Basu, and T. Xie, “WebSob: A Tool for Robustness Testing of Web Services,” 29th International Conference on Software Engineering (ICSE’07 Companion), pp. 65–66, 2007.
 A. Milicevic, S. Misailovic, D. Marinov, and S. Khurshid, “Korat: A Tool for Generating Structurally Complex Test Inputs,” in 29th International Conference on Software Engineering (ICSE’07). IEEE Computer Society, 2007, pp. 771–774.
 L. Lemos, O. Augusto, S. Bajracharya, and J, “CodeGenie:: a Tool for Test-Driven Source Code Search,” Proceedings of the 22th annual ACM SIGPLAN conference on Object-Oriented Programming, Systems, Languages, and Applications, pp. 917–918, 2007.
 Q. Xie, M. Grechanik, and M. Hellige, “Smart: A Tool for Application Reference Testing,” in 22nd IEEE/ACM international conference on Automated software engineering. ACM, 2007, pp. 537–538.
 F. Pinte, N. Oster, and F. Saglietti, “Techniques and tools for the automatic generation of optimal test data at code, model and interface level,” 30th international conference on Software engineering – ICSE Companion ’08, p. 927, 2008.
 Q. Xie, M. Grechanik, and C. Fu, “REST: A Tool for Reducing Effort in Script-based Testing,” 24th IEEE International Conference onSoftware Maintenance, pp. 468–469, 2008.
 J. Wloka, B. G. Ryder, and F. Tip, “JUnitMX – A Change-aware Unit Testing Tool,” 31st International Conference on Software Engineering, pp. 567–570, 2009.
 E. Mishra and Y. Sonawane, “TED: Tool for Testing and Debugging uDAPL,” in Architectures for Networking and Communications Systems (ANCS), 2010 ACM/IEEE Sympo- sium on, 2010, pp. 1–2.
1. Application Testing Tools
Application testing tools are used to test software depending on the activity for which it was designed. Some application focus on a single task; others, offer several applications and each one must be tested properly. During this work, we analyzed 91 application testing tools organized as follows:
1.1. Source Testing Tools
Source Testing Tools groups everything related with source code (code coverage, static analysis and metric, identification of errors during compilation). We examined 22 source testing tools.
|AdaTEST||Coverage, static, and dynamic software testing for Ada Apps|
|AQtime||Profiling toolkit for applications with built in support for .NET, Java, VBScript and JScript code.|
|Bounds Checker||Run-time error detection and debugging tool for C++ developers. Supports C/C++,.net,ASP,ASP.net.|
|Bullseye||Bullseye Coverage find untested C++ code and measure testing completeness.|
|CTC++||Testwell CTC++ is a code coverage and dynamic analysis tool for C and C++ code. It can also be used to measure function execution costs (normally time).|
|Code Coverage||Code Coverage identifies untested portions of the source code.|
|CodeCheck||CodeCheck is a programmable tool for managing all C and C++ source code on a file or project basis|
|C\C++ Test||Static code, data flow, and metrics analysis. Peer code review process automation, unit test creation, execution, optimization and tracking. Run time error detection.|
|CTA++||Tool for unit testing C++ classes, libraries and subsystems.|
|CMT++||CMT++, Complexity Measures Tool for C/C++, is a code metrics tool for C and C++ languages.|
|devAdvantage||devAdvantage is an automated code review tool that analyzes .NET C# code looking for bugs and issues.|
|GlowCode||Memory and resource leak detection, code profiler, function call trace, and report tools for Win32 programs|
|Insure++||Runtime error detection tool for C and C++|
|LDRA Testbed||Static and code coverage analysis tool. Static analysis features include code visualisation, programming standards enforcement and complexity metrics.|
|Leak Check||Quality tool used for finding memory leaks. No recompile or relink is required! Application code and 3rd party code can be checked.|
|OSPC||Checks applications, written in C for conformance to company coding standards, International standards, and conformance to the C standard|
|Panorama++||Test Coverage Analysis & Metrics Testing for C & C++|
|McCabe IQ||McCabe IQ Test Team Edition provides comprehensive test/code coverage, (from LOC to the Path Level).|
|Coverity Static Analysis||It helps developers find hard-to-spot, yet potentially crash-causing defects early in the software development life-cycle, reducing the cost, time, and risk of software errors|
|Rational Purify||Pinpoints hard to find memory errors such as uninitialized memory access, buffer overflow and improper freeing of memory.|
|TCAT C/C++||It is a fully automated tool for static analysis and code coverage at the unit, sub-system, system and integration testing levels.|
|Test Coverage||Family of test coverage tools for a wide variety of languages (C, C++, C#, COBOL, Java, more…). Very low probe overhed. Independent of compiler|
1.2. Functional Testing Tools
Functional Testing Tools validate the functions by feeding them input and examining the output. These tools verify a program by checking it against design documents or specifications. We investigated 27 functional testing tools.
|dotTEST||Unit-testing tool that automatically tests classes written on Microsoft’s .NET Framework without requiring developers to write a single test scenario or stub.|
|ATD||Automated Test Designer, (ATD) is an easy to use windows client/server tool designed to create Test Cases, Test Data anda Automated Test Scripts based on Requirements.|
|Avignon||Avignon is an acceptance test system that allows the user to write executable tests. Is uses XML to define the syntax of the language.|
|CAPBAK/X||It allows the user to record mouse mouvements, keyboard activities, widget calls and verification information into test scripts.|
|Certify||Worksoft Certify is an automated functional testing solution for SAP lifecycle management and cross-platform business process validation.|
|CitraTest||It is a software test automation solution for automating functional testing and automating any manual process for any application, that is accessible from a Windows client.|
|Eggplant||It is a QA automation tool that is easy to use. It acts as a virtual user to generate and run test scripts and to automate other tasks on the remote machine.|
|GUITAR||It is a model-based system for automated GUI Testing. It generates new test cases using event-flow graphs. Pre/postconditions are used to generate expected response.|
|Holodeck||It is a testing tool that uses fault simulation to emulate real-world application and system errors. It allows users to analyze and debug error-handling code in hostile environments.|
|Jenny||It is a tool for generating regression tests. It covers the interactions using pairwise testing.|
|JPdfUnit||It is a framework for testing a generated pdf document with JUnit test framework so JPdfUnit is a high level API. It is designed for an easy access to the PDFBox library.|
|JSystem||It is a open-source framework for writing and managing automated system tests. It allows the user to create and run test-scenarios using drag and drop approach.|
|MITS.GUI||Has an intelligent state machine engine that makes real-time decisions for navigating through the GUI portion of an application without scripts.|
|PETA||It is a platform for automated software testing of distributed software systems like client-server or service oriented architectures (SOA) on the integration or system layer.|
|PyUnit||It is a testing framework. It is a Python language version of JUnit.|
|Repro||It is a tool that records the software process execution at multiple levels to provide unmatched visibility into process execution.|
|RFT||Rational Functional Tester is an automated functional testing and regression tool. It enables testers to automate tests resilient to frequent application user interface changes.|
|ScriptTech||It generates executable scripts for QTP, TetPartner, Open Source as well as Java, C++, VB, C#, TCL or other custom scripting. It also creates scripts prior to code availability.|
|SilkTest||It is a tool for automated function and regression testing. It creates test automation across a broad range of application technologies to identify quality problems.|
|SilverMark’s Test Mentor||It is a functional test and test modeling tool for Java developers to use as they develop their Java classes, clusters, subsystems, frameworks, and other components.|
|Squish||It is a cross platform/cross technology GUI test automation tool for functional GUI regression tests. It is possible to test the application using recorded or manually written test scripts.|
|TestArchitect||It is a test automation framework. It includes automation support for the most common platforms, along with version-control features to support team-based test automation.|
|TestComplete||It is an automated test manager to create, manage and run tests for any Windows, Web, or Rich Client software.|
|TestWorks||Integrated suite of Automated Regression Test and Coverage Analyzers for UNIX, Windows with multiple reports and re-usable test scripts in C, C++ and JAVA.|
|Vermont HighTest Plus||It is a regression-testing tool for testing stand-alone or browser-based applications. It supports full scripting language with variable, loops and conditionals.|
|T-Plan Robot||It is a black box automated testing tool, also known as VNCRobot. It provides human-like approach to software testing.|
|HP UFTS||HP Unified Functional Testing software accelerates automated software testing by simplifying test design and maintenance. It also validates integrated test scenarios.|
1.3. Performance Testing Tools
Performance Testing Tools are used to determine the speed or effectiveness of the application involving quantitative tests and qualitative attributes such as reliability, scalability, and interoperability. We studied 12 Performance testing tools.
|BugTimer||It is a timer application designed to streamline the entire process of timing and documenting performance test results. It also records, displays, saves, sorts, and prints the test results.|
|CPT||Certify Performance Test provides an intuitive yet powerful approach to assuring SAP applications operate efficiently and reliably.|
|DTM DB Stress||It is a utility for stress testing the server parts of information systems and applications, as well as DBMSs and servers themselves.|
|IxChariot||It is a test tool for simulating real-world applications to predict device and system performance under realistic load conditions.|
|IxLoad||It is a tool for running performance testing applications that can emulate millions of users. It emulates data, voice, and video subscribes.|
|liSA||It offers direct functional and performance validation of component of distributed applications. It also identifies performance bottlenecks with metrics.|
|LoadeaTest||It is a stress testing application that aims to test the functionality of a system when this system is heavily loaded for a given period of time.|
|LoadRunner||It helps to prevent performance problems by detecting and reducing bottlenecks before a system deployment or upgrade.|
|Shunra||It enables the deployment of new applications and infrastructure changes. It provides graphically reports and drill-down transaction.|
|SilkPerformer||It is a performance testing solution for optimizing the performance of business applications. It enables performance tests that simulate peak demand loads.|
|Performance PRO||It isolates performance bottle-necks. It checks every form, combo and list box, and reviews SQL statements. It also reports exactly what to fix to improve the speed of the application.|
|XtremeLoad||It is a software framework for load testing of distributed software systems. It provides an extensible, scalable, and easy-to-use foundation to build load testing.|
1.4. Java Testing Tools
Java Testing Tools ensure that the problems in source code are addressed during the development. Java Testing tools also ensure functional test classes that understand the application and test the source files thoroughly. Sixteen Java Testing Tools were verified.
|Abbot||It is a JUnit extension for testing Java GUIs. It supports recording and playback at both low and high levels of abstraction.|
|adaptiveCells/J||It is an open source J2EE benchmarking system. It enables the development of complex artificial J2EE test-beds without requiring a single line of code.|
|Agitator||It provides the understanding of code behavior as developer writes or modifies Java classes or methods. It helps the user to validate the expected behavior.|
|Bugkilla||It is a set of java tools for the functional test of J2EE Web Applications.|
|Cactus||It is a test framework for unit testing server-side java code (Servlets, EJBs, Tag Libs, Filters, etc). It lower the cost of writing tests for server-side code. It uses JUnit and extends it.|
|GUIdancer||It offers support for Code Coverage using the JaCoCo code coverage tool for Java.|
|JCover||It is a code coverage analyzer for Java programs. It shows coverage overlap or disjoint.|
|Jemmy||It is an open source library that can be used to create tests for Java GUI applications.|
|JMeter||It is an open source software designed to load test functional behavior and measure performance. It can be used to simulate a heavy load on server, network or object.|
|JStyle||It is an automated Java code review tool. It performs Java code inspection as part of every build. It also computes several object-oriented metrics.|
|Jtest||It is an solution for automating a borad range of practices proven to improve development team productivity and software quality.|
|JUnit||It is an open source testing framework for users who implements unit tests in Java.|
|JVerify||It is a Java class/API testing tool that supports a unique invasive testing model.|
|Marathon||It is a testing framework for GUI applications developed using Java/Swing. It composes of recorder, runner and editor. The testscripts are composed of python code.|
|QF-Test||It is a tool for automated testing of Java and Web applications with graphical user interface. It supports regression and load testing.|
|TCAT/Java||It is a code coverage analyzer for Java applications and applets.|
1.5. Embedded Testing Tools
Embedded Testing Tools are responsible for test embedded (real time) systems. Embedded testing tools allow detection of timing issues and fault insertion. We looked over six embedded testing tools.
|Reactis Tester||It generates comprehensive test suites from Simulink/Stateflow models. The test suites exercise large portions of the software under test while avoiding redundancy.|
|TBrun||It is an automated unit testing tool. Its functionality includes automatic test harness generation, automatic test stub generation and test documentation reporting.|
|Tessy||It offers automated module testing of C and C++ code directly on the target system using debugging technology. It supports the unit testing cycle and works on debuggers.|
|TestCast||It tests embedded software components, processes, subsystems, etc. It contains tools for visualization of messages traffic between the component and their simulated counterparts.|
|TestQuest||It is a test automation solution that tests the end-user experience, and allows testing of multiple devices, enabling end-to-end testing.|
|VectorCAST||It is a world-class integrated software test solution that automates the tasks associated with testing software components.|
1.6. Database Testing Tools
Database Testing Tools focus on test the referential integrity, security (including permissions and privileges), interfaces, and web services of the database. We evaluated 8 database testing tools.
|Data Generator||It is an automated testing and data generation tool, which enables the test data creation for software quality assurance, performance testing, and database load testing.|
|DTM Data Generator||It is a tool that generates data, tables (views, procedures etc) for database testing (performance testing, QA testing, load testing or usability testing) purposes.|
|Datatect||It generates test data to ASCII flat files or directly to RDBMS including Oracle, Sybase, SQL Server, and Informix, using ODBC.|
|DTM DB Stress||It is an utility for stress testing the server parts of information systems and applications, as well as DBMSs and servers themselves. It allows the creation of a set of requests to the server.|
|ER/Datagen||It generates test data in support of Allfusion ERwin and ER/Studio (enterprise editions).|
|SQL Profiler||It is an module that allows the user to identify, isolate, and correct coding mishaps and performance bottlenecks that could lead to costly performance problems.|
|TurboData||It is a test data generator tool. It also generates Select, Update and Delete SQL. TurboData includes an SQL Builder, Grid Data Editor and has full Version Control.|
|utPLSQL||It is an unit-testing framework for users using Oracle PL/SQL language. It allows the automated testing of packages, functions and procedures.|
2. Web Testing Tools
Web Testing Tools are responsible for testing applications that are accessed over a network. Web applications are popular due to the convenience of using a web browser as a client. For this reason, is necessary test the web applications in different browsers and operational systems. In our work, we found 92 web testing tools organized as follows:
2.1. Link and HTML Testing Tools
Link and HTML Testing Tools consist of verifing if the connections with hyperlinks are correct and the identification of the HTML and services related are working. We discovered 17 link and HTML testing tools.
|AccVerify||It verifies, corrects, monitors and manages websites and web based applications.|
|ChangeAgent||It is a web site repair and maintenance application that makes fixing problems simple.|
|CSE HTML Validator||It eliminates website problems increasing the productivity. HTML, XHTML, CSS, link, spelling, and accessibility checker available.|
|Cyber Spyder||CyberSpyder Link Test is a web site management program to be used for verifying that the URLs on a site are not broken and for analyzing site content.|
|HPT||HTML PowerTools is a suite of Windows tools for HTML checking, spelling, etc.|
|HTML Tidy||It is a tool that fixes common errors and prints HTML.|
|InFocus||It is an integrated solution that includes testing and analysis tools, a rules builder, a checking engine, and content spidering.|
|Link Checker||Link Checker Pro is a link checking tool for websites. It can export results in a number of formats and provide a graphical view of the website structure.|
|LinkRunner||Alert Link Runner is a dead link detector tool.|
|LinkScan||It is a dead link checking and website management tool.|
|LinkSleuth||It checks web sites for broken links. Link verification is done on links, images, frames, plug-ins, backgrounds, local image maps, style sheets, scripts and java applets.|
|Link Validator||Web Link Validator is a link checker and site management tool for webmasters to check links for accuracy and availability, find broken links and links containing syntactic errors.|
|MOMspider||It is a web-roaming robot that specializes in the maintenance of distributed hypertext infrastructure.|
|Ramp Ascend||It is an accessibility testing tool that tests and remediates websites and web based applications for integrated accessibility for Section 508 and WCAG compliance.|
|Real Validator||It is an html syntax checker that uses a SGML parser. It works offline, provides convenience, speed and flexibility.|
|Truwex||It is a tool for evaluating website accessibility, privacy, performance, quality, broken links. It shows issues on a web page map and in the HTML code.|
|WebLight||It finds broken links and markup problems, and identifies relationships between URL’s.|
2.2. On-line Link and Testing Services
On-line Link and Testing Services consist of verifing if the connections with hyperlinks are correct and the identification of the HTML and services related are working. We discovered 17 on-line link and testing services.
|AETG Web Service||It generates test cases from a model of your requirements. It uses combinatorial design techniques to find a minimal test-case set that covers all pairwise interactions|
|Audit Blossom||It is a web site audit service that finds improperly formatted URLs and references to documents that have either moved or no longer exist.|
|CSSCheck||It is an on-line cascading Style sheet (CSS and CSS2) checker.|
|CSS Validation||It is a service that checks Cascading Style Sheets (CSS) in (X)HTML documents or standalone for conformance to W3C recommendations|
|Dr. Watson||It is a program error debugger that gathers information about the computer when an error (or user-mode fault) occurs with a program.|
|HTML Toolbox||It is a service which searches the site to find broken links, spot bad HTML tags, and rates the server’s response time.|
|Link Alarm||It is an online service for detecting broken links.|
|Link Checker||It looks for issues in links, anchors and referenced objects in a Web page, CSS style sheet, or recursively on a whole web site.|
|Link Cheching||It identifies broken links, analyzes accessibility, reports on search engine optimization, monitors page load times, and helps to manage changes.|
|HTML Validator||It is an online service that validates HTML documents.|
|Markup Validation||It is an online service that checks HTML documents for conformance to W3C HTML and XHTML Recommendations and other HTML standards.|
|SDT||The Software Development Technologies (SDT) software testing tools consulting service help the user to establish or improve the use of software testing tools.|
|Site Check||It runs HTML and stylesheet validators, accessibility assessment, link check, load time check, and more using online web testing tools.|
|Weblint Gateway||It performs an HTML error scan and creates an HTML listing with error messages in context and tags, URLs, and error messages highlighted in color.|
|WPBCV||Web Page Backward Compatibility Viewer is an on-line utility that serves a page based on how various browsers would render it.|
|Web Page Purifier||It is an on-line utility that maps a page to HTML 2.0, HTML 3.2, HTML 4.0, or WebTV 1.1. standards.|
|XMLV||XML Validation tool validates XML documents.|
2.3. Functional Testing Tools
Functional Testing Tools focus on the functionalities from the web systems and its particularities. We detected 32 functional Testing Tools for web applications.
|actiWATE||It is a Java-based software platform for writing test scripts.|
|Badboy||It is a tool designed to aid in testing and development of complex dynamic applications. It allows the user to capture CGI parameters, pages and framesets.|
|Canoo WebTest||It is a open source tool for testing of web applications. It calls pages and verifies the results, giving comprehensive reports on success and failure.|
|eValid||It provides Browser-Based Client-Side WebSite quality checking for Functional Testing, LoadTest, Site Analysis (SiteMap), Performance check and page tuning.|
|IeUnit||It is a framework to test logical behaviors of web pages. It helps users to create, organize and execute functional unit tests.|
|Imprimatur||It is a web application functional testing tool. It sends http requests to the application, and then validates the responses using regular expressions.|
|iMacros||It is used to capture and replay multi-step business transactions that require repeated form fills on a continuous basis.|
|HTTP::Recorder||It is a browser-independent recorder that records interactions with web sites and produces scripts for automated playback.|
|ITP||It is a web application testing harness that can test the site from the user’s perspective.|
|liSA||It allows the user to fully test dynamic web sites and Rich Internet Application (RIA) browser interfaces, as well as directly testing the building blocks below the UI.|
|Netvantage||Netvantage Functional Tester, functional and regression automated web testing tool.|
|PesterCat||It is a web testing tool designed to perform functional testing for web applications. It features an proxy recorder that enables to record scripts using web browser.|
|QA Wizard||It automates the functional and regression testing of Web, Windows, and Java applications, and load testing of Web applications.|
|Ranorex||It is a GUI test and automation framework for testing applications.|
|Rational Robot||It is a test automation tool for functional testing of client/server applications. It enables defect detection, includes test cases and test management.|
|Sahi||It is an automation and testing tool for web applications, with the facility to record and playback scripts.|
|SAMIE||It is a simple automated module for Internet Explorer that allows a user to automate actions.|
|Selenium||It is a test tool for web applications that tests run directly in a browser, just as real users do.|
|SilkTest||It is a tool for automated function and regression testing, delivering advanced test automation capabilities that can be used by many projects.|
|SOAtest||It automates web application testing, message/protocol testing, cloud testing and security testing.|
|soapUI||It is a application for inspecting, invoking and testing (functional and load) of web services.|
|Solex||It is an open source web application testing tool built as a plug-in for the Eclipse IDE. It provides functions to record a client session, adjust it according to various parameters.|
|Squish||It is a test tool which allows to create, debug and run automated GUI tests for Qt, Tk, XView and HTML/Web applications.|
|swete||It provides cross platform console based tools for regression testing of web applications. It allows to record XML test scripts using any web browser.|
|TestDrive||It is a automated software quality solution designed to test browser and legacy applications, as well as GUIs, including Java, Ajax, Flex and Silverlight.|
|Test Studio||It is a tool for build automated tests for web and desktop applications. It runs tests against all browsers at once. It bundles functional and performance tests.|
|vTest||It is a an automated functional and regression testing tool for web applications. It generates automated test scripts, replays both automated and custom test scripts.|
|WatiN||Is is an open source C# framework for automating functional web testing.|
|Watir||It is an open-source functional testing tool for automating browser-based tests of web applications. It clicks links, fills in forms, presses buttons.|
|WebInject||It is a tool for testing of web applications and services. It is used to test any individual system component with an HTTP interface (JSP, ASP, CGI, PHP, Servlets, HTML Forms).|
|WET||It is a framework for Web automation testing. It has many features like multiple parameter based object identification, support for XML object repository, better popup handling, etc.|
|Yawet||It is a Java/swing tool to create, run and debug functional web tests. It can verify Html, Pdf and Xml documents. Yawet offers reports, step libraries, parametrisation and custom extensions.|
2.4. Security Testing Tools
Security Testing Tools determine that a system protects data and maintains functionality as intended.
|Application Security||It includes tools for automating and managing application security testing, including static testing of source code in development and dynamic testing of applications.|
2.5. Performance Testing Tools and Services
Performance Testing Tools are used to measure the speed or effectiveness of the application. We observed 25 performance testing tools for web applications.
|ANTS||ANTS Performance Profiler identifies bottlenecks and optimizes .NET application performance.|
|AppLoader||It is a tool that simulate user actions on applications. It allows to load stress applications or to test the quality of the applications. It also generates performance reports.|
|CloudBurst||It is a cloud-based extension to SilkPerformer, the performance testing offering that enables software quality teams to launch size peak-load performance test.|
|Dotcom-Monitor||It simulates real users by using its remote agents positioned around the globe. It allows the end-user to setup the conditions for the web site or web application to realize stress test.|
|Forecast||It is a tool suite for system load testing, performance measurement and multi-user functional testing. It supports nested and asynchronous transactions.|
|http_load||It is an Unix-Based load load generator for web servers. It runs multiple http fetches in parallel, to test the throughput of a web server.|
|LoadTracer||It is a load-testing tool, which also used for stress and performance testing. It uses various technique to create load on web server as real user does.|
|MACT||Microsoft Application Center Test is designed to stress test Web servers and analyze performance and scalability problems with Web applications, including ASP and components.|
|NeoLoad||The NetLoad Cloud Testing solution simulates hundreds of virtual users on the web site, getting performance statistics and revealing errors under stress. .|
|OpenLoad||It is a web testing tool used to simulate hundreds or even thousands of users simultaneously accessing a web site or application.|
|OpenSTA||It is a distributed software testing architecture based on CORBA. It is designed to be used by Performance Testing Consultants or other technically proficient individuals.|
|Proxy Sniffer||It is a performance web load and stress testing tool. It permits to analyze the performance characteristics and the stability of a Web application under various load conditions.|
|QEngine||It is a test automation tool which supports load testing and functional testing for Web Applications. It performance testing helps at the identification of bottlenecks.|
|Siege||It is an http load testing and benchmarking utility. It was designed to measure code under duress. It supports basic authentication, cookies, http and https protocols.|
|StressIT||It is a robust stress testing application that simulates the number of users the site can maintain. It charts the breaking point at which performance is no longer acceptable.|
|SiteStress||Webmetrics Full-Service Load Testing is a managed solution that gather requirements, define scenarios, run a series of load tests, and review the results.|
|Site Tester||It is a load-testing utility designed to test web servers and web applications. It simulates concurrent access of various users from different hosts to a web/application server.|
|TestMaker||It is an open source framework to build intelligent test agents that test Web services for scalability and performance. It surface performance bottlenecks and functional issues.|
|Wbox||It checks if the redirections are working correctly. It tests if the http compression is working.|
|WebLOAD||It is a software for performance testing of internet applications. It helps to meet the performance requirements by simulating traffic of up to tens of thousands of users.|
|Web Performance||Web Performance Load Tester is a web testing tool that analyze how many users a website can handle. It automatically detects and configures the test cases.|
|Web Polygraph||It is a performance testing tool for caching proxies, origin server accelerators, content filters, and other web intermediaries.|
|WAPT||It is a load and stress testing tool that provides a way of testing web sites, web servers, and intranet applications with web interfaces.|
|Web Server Stress Tool||It is a load test application for web servers that tests the web server performance under normal and excessive loads to ensure if the information and services are available.|
|WebSizr||It is a performance analysis and sizing framework for http based servers and applications. It is a server measurement tool that simulate client behavior.|
3. Bug Tracking Tools
Bug Tracking Tools are designed to report and keep track of defects in software programs. Divided into Web Based and Bug Tracking Applications. We recognized 38 Bug tracking tools.
3.1. Web Based Bug Tracking
We analysed 28 Web Based Bug Tracking Tools.
|AceProject||It can be used as an issue management program and bug tracking software.|
|AdminiTrack||It is an issue and task tracking application.|
|ADT Web||Advanced Defect Tracking Web Edition is a defect-tracking solution. The software allows you to track defects, feature requests and suggestions by version, customer etc.|
|ALMComplete||It tracks status and resolution progress of defects and issues across releases. It coordinates development and remediation activities as bugs are found.|
|BugAware||It is a Bug tracking solution. Features include: Email alert notification, dynamic reporting, team management, user discussion threads, file attachment, searching.|
|Bugcentral||It is a web based defect tracking service.|
|BUGtrack||It is a web based bug tracking system.|
|BugHost||It is a feature-complete hosted defect tracking system. The system is easy to use and customizable to meet the users needs.|
|BugStation||It is a web-based application with many enhancements designed to make Bugzilla easier and more secure. A centralized system for entering, assigning and tracking defects.|
|Bug Tracker||It is an open-source, web based bug tracking and general purpose issue tracking application. It is configurable with email integration, search, screen capture, etc.|
|Bug Tracking||It is a web based detect tracking software. It offers email notification, file attachment, tracking history, bilingual pages, etc.|
|Bugzero||It is a bug tracking system for software detect tracking, and a general management tool for customer and sales tracking and helpdesk customer support.|
|Bugzilla||It is a server software designed to help the software development management. It is an open source defect tracking system developed originally for the Mozilla project.|
|Census||It is a bug tracking and defect tracking tool. It includes VSS integration, notifications, workflow, reporting and change history.|
|Dragonfly||It is a web-based, cross-browser, cross-platform issue tracking and change management for software development, testing, debugging, and documentation.|
|FogBugz||It is a web-based bug tracking system.|
|JIRA||It is a issue tracking and project management application.|
|JTrac||It is an open-source and customizable issue-tracking web-application.|
|Mantis||It is a web-based bug tracking system.|
|Ozibug||It is a platform-independent defect tracking system. It can be used to manage the reporting of bugs and tracking the progress.|
|ProblemTracker||It is a web-based collaboration software for issue tracking, automated support, and workflow, process, and change management.|
|PR Tracker||It is an enterprise level problem tracking system designed for bug tracking.|
|Squish||It is a web-based management tool that keep track of product development, client request, software bugs, help desk tickets, tasks, and more.|
|teamatic||It is a web-based bug tracking system.|
|TrackStudio||It is a configurable issue tracking system, workflow engine and document management system that can be used to track the progress of any asset.|
|VisionProject||It is a web based issue tracking and project collaboration tool designed to make projects more efficient and profitable.|
|Woodpecker IT||It is a web-based request, issue or bug tracking tool. It can be used to manage business processes, requirement management or bug tracking.|
|yKAP||It is a web based Bug/Defect tracking, Issue Management and Messaging product. It allows the user to add custom values for system parameters.|
3.2. Bug Tracking Applications
We analysed 10 Bug Tracking Tools.
|BugRat||It is a software that provides a defect reporting and tracking system.|
|Bug Trail||It captures and tracks software bugs. It allows the user to attach screenshots, automatically capture system parameters and create output reports.|
|Defect Manager||It is a grade bug and issue tracking system. It tracks bugs, defects, calls, tasks and enhancements through th software development life cycle.|
|FIT BugTrack||It implements a bug tracking solution using a system designed for the management of software bugs. It eliminates unnecessary duplication of effort and resources.|
|GNATS||It is a defect tracking software.|
|IssueNet||IssueNet platform provides an extensible framework to handle the many issue management needs. It has several pre-build solutions.|
|IssueView||It is an application that allows the management of bugs.|
|TestTrack||It is a scalable issue management tool.|
|Trac||It is an issue tracking system for software development projects. It provides an interface to version control systems.|
|ZeroDefect||It is an issue management system.|
4. Testing Management Tools
Test management tools are used to structure automated tests and manual test processes, and manage multiple environments. Quality assurance teams use these tools for managing test cases, environments, automated tests, defects and project tasks. We encountered 10 Testing Management Tools.
|ApTest Manager||It is a tool for managing testing of hardware and software products. It defines test requirements and specifications, establishing test plans, and executing tests.|
|Concerto Test Center||It generates, assigns/distributes, and facilitates specifics tasks. It also correlates test results to policy and functional requirements.|
|HP Quality Center||It is a scalable, unified platform for managing and automating the delivery of secure, reliable, quality applications. It enables the implementation of a management infrastructure.|
|RQM||Rational Quality Manager is a centralized test management environment that helps increase the efficiency and quality of software delivery.|
|SilkCentral||It is a test management solution for agile and traditional development projects. It provides an integrated framework for improving productivity and traceability for software testing.|
|SQS-TEST Test Center||It is a tool for test management, test specification and test execution for quality assurance in software development projects.|
|T-Plan||It allows the user to manage the test assets from planning to execution.|
|Test Manager||It enables the user to use requirements management tools to create and manage requirements. Borland CaliberRM Adapter, Telelogic DOORS Adapter, Rational ClearQuest Adapter.|
|TestLog||It is an integrated test case management software solution. It provides a tool for software test teams to create and update effective test plans and test cases.|
|QAComplete||It is an application for test and QA management including requirements management, test case management, and defect tracking.|
5. API Testing Tools
API (application programming interface) testing tools are used to evaluate the application using public and private APIs. It focus on the validation of interfaces. We analysed 4 API Testing tools.
|ADL project repository||The Assertion Definition Language (ADL) is an automated test generation system. It is a formal grammar for describing the behavior of interfaces.|
|MITS.Comm||It simulates third party systems over any type of interface for testing anything from electronics to web-related applications.|
|DejaGNU||It is a framework for testing interactive programs that provides a single front end for tests.|
|TET||The Test Environment Toolkit (TET) is provided as an open source product.|
6. Communications Testing Tools
Communications Testing Tools group the tools that test network communication between applications, network performance, and multiple concurrency. We investigated 11 Communications testing tools. For example, WebMNS Simulation Toolkit is a network simulator for testing the devices communication.
|CWS||The Cellogic Wireless System (CWS) is a tool for introduction, optimization and maintainance of wireless communication services.|
|Fault Factory||It is a active debugging solution for applications that communicate with other applications over the network. It utilizes API-Level Fault Injection to imitate faults.|
|IxANVL||IxANVL (Automated Network Validation Library) is a system for testing of network products. It enables network product to testing processes faster and more thorough.|
|LANTraffic||It is a traffic generator for IP networks using different protocols (TCP, ICMP, SCTP, and UDP). It is independent of any transmission or telecom line.|
|Maxwell||It enables the user to modify, distort, and corrupt the flow of network traffic. It also operates in real time and modifies traffic while the protocol discussion continues.|
|NetHawk EAST||It is a simulator used by equipment manufacturers and operators in load testing, regression testing and functional testing. It is available for VoIP, SS7, and wireless.|
|nGenius||nGenius Performance Manager provides mobile and fixed operators with a unified view into subscriber, service and network performance for IP networks.|
|NuStreams 2000||It is a multichannel gigabit Ethernet test systems for LAN, WAN, network, switch, router and IP device testing. Test, measure, certify and stress your networks and devices.|
|Silvercreek||Silvercreek SNMP Test Suite verifies protocol compliance, error and exception handling, boundary conditions, and performance under heavy network loads.|
|SNAsim||It is a SNA testing and traffic generation tool. It provides a PC-based environment for simulating the 3270 and 5250 data streams.|
|WebNMS||It is a complete device simulation and network simulation software for testing and demonstration of NMS/EMS in multiple operating systems.|