Research Type Facet


Classes Description
Validation Research Tools investigated are novel and have not yet been implemented in practice. Tools used are for example experiments, i.e., work done in the lab.
Evaluation Research Tools are implemented in practice and an evaluation of the tool is conducted. That means, it is shown how the tool is implemented in practice (solution implementation) and what are the consequences of the implementation in terms of benefits and drawbacks (implementation evaluation). This also includes identification of problems in industry.
Solution Proposal A solution for a problem is proposed, the solution can be either novel or a significant extension of an existing tool. The potential benefits and the applicability of the solution is shown by a small example or a good line of argumentation.
Philosophical Paper These papers sketch a new way of looking at existing things by structuring the field in form of a taxonomy or conceptual framework.
Opinion Paper These papers express the personal opinion of somebody whether a certain tool is good or bad, or how things should be done. They do not rely on related work and research methodologies.
Experience Paper Experience papers explain what and how something has been done in practice. It has to be the personal experience of the author.
Advertisements

Names and descriptions of the SPL Testing Tools


Tool Name Year Description
Tool [40] 2004 The authors implemented a tool that can automatically generates test data to distinguish among models. Such automation is based on the ability of simulating or running models.
ScenTED DTCD [39] 2005 ScenTED-DTCD tool (Domain Test Case Scenario Derivation) was implemented in order to support the ScenTED technique (Scenario based TEst case Derivation), which aims at reducing effort in product family testing.
XUnit [41] 2005 It is a widely used open-source unit testing framework. It has been ported to most object-oriented programming languages and is integrated in many common IDEs.
ASADAL [42] 2006 It supports the entire life cycle of software development process based on a Product Line Software Engineering method called FORM (Feature-Oriented Reuse Method).
Objecteering  [37] 2007 It is a prototype tool to automatically generates both functional and robustness test cases specific to a product from the Product Families requirements.
GATE [43] 2007 It is a prototype tool that implements an efficient mechanism of testing components with high automation. The framework facilitates the reusability and automatic creation of aspect test cases, with software product lines techniques.
ParTeG [44] 2008 It supports the automatic generation of boundary tests. It generates JUnit test suites for both configurations. The tool also allows the user to choose a coverage criterion, which should be satisfied as good as possible by the generated test suite.
Kesit [38] 2008 The prototype shows that incremental test generation provides significant performance improvements over conventional means of test input generation.
MoSo-PoLiTe[45] 2011 It provides a test framework for SPL. It is a tool chain that contains a pairwise configuration selection component on the basis of a feature model.

 References

[37] C. Nebut, Y. Traon, and J. Jezequel, “System Testing of Product Lines: From Requirements to Test Cases,” Software Product Lines, pp. 447–477, 2007.

[38] E. Uzuncaova, D. Garcia, S. Khurshid, and D. Batory, “Testing Software Product Lines Using Incremental Test Generation,” 19th International Symposium on Software Reliability Engineering (ISSRE’08), pp. 249–258, 2008.

[39] A. Reuys, E. Kamsties, K. Pohl, and S. Reis, “Model-Based System Testing of Software Product Families,” International Conference on Advanced Information Systems Engineering CAiSE, pp. 519–534, 2005.

[40] Z. Stephenson, Y. Zhan, J. Clark, and J. McDermid, “Test Data Generation for Product Lines – A Mutation Testing Approach,” in 3rd International Workshop on Software Product Line Testing (SPLiT 2004). Citeseer, 2004, p. 13.

[41] M. Galli, O. Greevy, and O. Nierstrasz, “Composing Unit Tests,” in 4th International Workshop on Software Product Line Testing. Citeseer, 2005, pp. 16–22.

[42] K. Kim, H. Kim, M. Ahn, M. Seo, Y. Chang, and K. C. Kang, “ASADAL: A Tool System for Co-Development of Software and Test Environment based on Product Line Engineering,” 28th International Conference on Software Engineering (ICSE’06), pp. 783–786, 2006.

[42] Y. Feng, X. Liu, and J. Kerridge, “A product line based aspect-oriented generative unit testing approach to building quality components,” Proceedings of the 31st Annual International Computer Software and Applications Conference, pp. 403– 408, 2007.

[44] S. Weissleder, D. Sokenou, and B. Schlingloff, “Reusing State Machines for Automatic Test Generation in Product Lines,” 1st Workshop on Model-based Testing in Practice (MoTiP’08), p. 19, 2008.

[45] S. Oster, I. Zorcic, F. Markert, and M. Lochau, “MoSo- PoLiTe: Tool Support for Pairwise and Model-Based Software Product Line Testing,” in Proceedings of the 5th Workshop on Variability Modeling of Software-Intensive Systems, ser. VaMoS ’11. New York, NY, USA: ACM, 2011, pp. 79–82.

Names and descriptions of the Single System Testing Tools


Tool Name Year Description
IDATAG [23] 1998 It supports both the specification of the behavior of a user interface and the generation of two types of test case for GUI coverage and for checking the usability of the application.
SOSTEM [13] 2000 It is designed using SOFL and implemented using Java. It supports five functionalities: generation of proof obligations, interactive testing, batch testing, test results analysis, and test information management.
AsmL [24] 2003 It supports the generation of parameters, call sequences, and the conduction of conformance tests. The tool performs a semi-automatic approach, requiring a user to annotate models with information for generating tests.
DUTF [25] 2005 It assists in the construction of test cases for DSL programs. Failed test cases reported within the DUTF reveal the presence of a program fault.
STSC [26] 2005 The “STSC” prototype tool supports the statistical testing of software components. It supports a wide range of operational profiles and test oracles for test case generation and output evaluation.
JTest [14] 2006 It can automatically generates test inputs to perform white-box testing. When specifications are provided with the class, Jtest can make use of them to perform black-box testing.
COMPTest [27] 2006 It is a component test tool that can be used to automatically identify component-based API changes and impacts, as well as reusable test cases in a component test suite.
SimC [15] 2006 It is a prototype tool, which automatically generates test data for unit testing of C programs involving pointer and structure operations. The tool symbolically simulates the execution of the given program.
Taxi [28] 2007 It implements the XML-based Partition Testing approach for the automated generation of XML Instances conforming to a given XML Schema.
Websob [29] 2007 It is a tool for automatically generating and executing web-service requests given a service provider’s Web Service Description Language (WSDL) specification.
Korat [30] 2007 Korat is a tool for constraint-based generation of structurally complex test inputs for Java programs.
ATTEST [21] 2007 It is a toolkit to address problems in test automation and maintenance. It provides easy-to-use mechanisms for helping testers to write and maintain automated test scripts through describing system behaviors at a high abstract level.
TaRGeT [22] 2007 It is a tool for automatic test case generation from use case scenarios written in natural language. It automates a systematic approach for dealing with requirements and test artifacts in an integrated way.
CodeGenie [31] 2007 It is a tool that implements a test-driven approach to search and reuse of code available on large-scale code repositories. With it, developers design test cases for a desired feature first.
Jwalk [16] 2007 It is a unit-testing tool developed to address the need for systematic unit testing within the context of agile methods.
Smart [32] 2007 It is a novel approach for generating tests for web services. It enables non-programmers to generate unit test cases for web services by performing drag-and-drop operations on GUI elements.
UnITeD [33] 2008 It enables the automatic generation of optimized component test sets from UML behavioral description by state machines or activity diagrams according to structural model-based testing criteria.
AgitarOne [17] 2008 It is a comprehensive unit testing product for Java. It can use run-time feedback and limited on-the-fly static analysis to generate input data or creates regression tests using constraint solving and mock objects.
REST [34] 2008 It is a Reducing Effort in Script-based Testing for guiding test personnel through changes in test scripts so that they can use these modified scripts to test new versions of their respective GAPs.
JUnitMX [35] 2009 It is a unit testing tool that leverages a change model to assist developers in the creation of new unit tests. The tool provides quantitative feedback and detailed information about change effects.
WebDiff [18] 2010 It is a technique and tool for detecting cross-browser issues automatically and reporting the locations of such issues in the corresponding web page.
CoGenTe [19] 2010 It is a tool for meta-model based testing of code generators. It generates test cases to cover the syntatic aspects of a translation and complex semantic aspects.
TED [36] 2010 It enables integrated testing and debbuging of software stack and underlying hardware while providing more flexibility and control to user.
AutoBlackTest [20] 2011 It is a tool for automatic generation of test cases for interactive application. It interacts with the application though its GUI, and uses reinforcement learning techniques to understand the interaction modalities.

 References

[13] S. Liut, T. Fukuzakit, and K. Miyamoto, “A GUI and testing tool for SOFL,” 7th Asia-Pacific Software Engineering Conference (APSEC’00), no. 11694173, pp. 421–425, 2000.

[14] T. Xie and D. Notkin, “Tool-assisted unit-test generation and selection based on operational abstractions,” 21st IEEE International Conference on Automated Software Engineering (ASE 2006), vol. 13, no. 3, pp. 345–371, 2006.

[15] Z. Xu and J. Zhang, “A Test Data Generation Tool for Unit Testing of C Programs,” 6th International Conference on Quality Software (QSIC’06), pp. 107–116, 2006.

[16] A. J. H. Simons, “JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction,” 22nd IEEE International Conference on Automated Software Engineering (ASE 2006), vol. 14, no. 4, pp. 369–418, 2007.

[17] B. Daniel and M. Boshernitsan, “Predicting Effectiveness of Automatic Testing Tools,” 23rd IEEE/ACM International Conference on Automated Software Engineering, pp. 363– 366, 2008.

[18] S.Choudhary, H.Versee, and A.Orso,“ACross-browser Web Application Testing Tool,” in Software Maintenance (ICSM), 2010 IEEE International Conference on, 2010, pp. 1–6.

[19] A. C. Rajeev, P. Sampath, K. C. Shashidhar, and S. Ramesh, “CoGente: A Tool for Code Generator Testing,” in Proceedings of the IEEE/ACM international conference on Automated software engineering, ser. ASE ’10. ACM, 2010, pp. 349–350. New York, NY, USA:

[20] L. Mariani, M. Pezze`, O. Riganelli, and M. Santoro, “Auto- BlackTest: a tool for automatic black-box testing,” in Proceeding of the 33rd international conference on Software engineering, ser. ICSE ’11. 2011, pp. 1013–1015.

[21] Y. Ren and F. Chang, “ATTEST: A Testing Toolkit for Validating Software Properties,” 23rd IEEE International Conference on Software Maintenance, pp. 469–472, 2007.

[22] S. Nogueira, E. Cartaxo, D. Torres, E. Aranha, and R. Marques, “Model Based Test Generation: An Industrial Experience,” in 1st Brazilian Workshop on Systematic and Automated Software Testing, 2007.

[23] A. Beer, S. Mohacsi, and C. Stary, “IDATG: An Open Tool for Automated Testing of Interactive Software,” 22th Annual International Computer Software and Applications Conference (COMPSAC’98), pp. 6–11, 1998.

[24] M. Barnett, W. Grieskamp, W. Schulte, N. Tillmann, and M. Veanes, “Validating Use-Cases with the AsmL Test Tool,” 3rd International Conference on Quality Software, pp. 238 – 246, 2003.

[25] H. Wu and J. Gray, “Automated Generation of Testing Tools for Domain-Specific Languages,” in Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. ACM, 2005, p. 439.

[26] R. Shukla, P. Strooper, and D. Carrington, “Tool Support for Statistical Testing of Software Components,” in 12th Asia-Pacific Software Engineering Conference (APSEC’05). IEEE Computer Society, 2005, p. 8.

[27] J. Gao, D. Gopinathan, Q. Mai, and J. He, “A Systematic Regression Testing Method and Tool For Software Components,” 30th Annual International Computer Software and Applications Conference (COMPSAC’06), pp. 455–466, 2006.

[28] A. Bertolino, J. Gao, E. Marchetti, and A. Polini, “TAXI – A Tool for XML-Based Testing,” 29th International Conference on Software Engineering (ICSE’07 Companion), pp. 53–54, 2007.

[29] E. Martin, S. Basu, and T. Xie, “WebSob: A Tool for Robustness Testing of Web Services,” 29th International Conference on Software Engineering (ICSE’07 Companion), pp. 65–66, 2007.

[30] A. Milicevic, S. Misailovic, D. Marinov, and S. Khurshid, “Korat: A Tool for Generating Structurally Complex Test Inputs,” in 29th International Conference on Software Engineering (ICSE’07). IEEE Computer Society, 2007, pp. 771–774.

[31] L. Lemos, O. Augusto, S. Bajracharya, and J, “CodeGenie:: a Tool for Test-Driven Source Code Search,” Proceedings of the 22th annual ACM SIGPLAN conference on Object-Oriented Programming, Systems, Languages, and Applications, pp. 917–918, 2007.

[32] Q. Xie, M. Grechanik, and M. Hellige, “Smart: A Tool for Application Reference Testing,” in 22nd IEEE/ACM international conference on Automated software engineering. ACM, 2007, pp. 537–538.

[33] F. Pinte, N. Oster, and F. Saglietti, “Techniques and tools for the automatic generation of optimal test data at code, model and interface level,” 30th international conference on Software engineering – ICSE Companion ’08, p. 927, 2008.

[34] Q. Xie, M. Grechanik, and C. Fu, “REST: A Tool for Reducing Effort in Script-based Testing,” 24th IEEE International Conference onSoftware Maintenance, pp. 468–469, 2008.

[35] J. Wloka, B. G. Ryder, and F. Tip, “JUnitMX – A Change-aware Unit Testing Tool,” 31st International Conference on Software Engineering, pp. 567–570, 2009.

[36] E. Mishra and Y. Sonawane, “TED: Tool for Testing and Debugging uDAPL,” in Architectures for Networking and Communications Systems (ANCS), 2010 ACM/IEEE Sympo- sium on, 2010, pp. 1–2.

Result of the Site Search of Industry Tools


1.    Application Testing Tools

Application testing tools are used to test software depending on the activity for which it was designed. Some application focus on a single task; others, offer several applications and each one must be tested properly. During this work, we analyzed 91 application testing tools organized as follows:

1.1.        Source Testing Tools

Source Testing Tools groups everything related with source code (code coverage, static analysis and metric, identification of errors during compilation). We examined 22 source testing tools.

Name

Description

AdaTEST Coverage, static, and dynamic software testing for Ada Apps
AQtime Profiling toolkit for applications with built in support for .NET, Java, VBScript and JScript code.
Bounds Checker Run-time error detection and debugging tool for C++ developers. Supports C/C++,.net,ASP,ASP.net.
Bullseye Bullseye Coverage find untested C++ code and measure testing completeness.
CTC++ Testwell CTC++ is a code coverage and dynamic analysis tool for C and C++ code. It can also be used to measure function execution costs (normally time).
Code Coverage Code Coverage identifies untested portions of the source code.
CodeCheck CodeCheck is a programmable tool for managing all C and C++ source code on a file or project basis
C\C++ Test Static code, data flow, and metrics analysis. Peer code review process automation, unit test creation, execution, optimization and tracking. Run time error detection.
CTA++ Tool for unit testing C++ classes, libraries and subsystems.
CMT++ CMT++, Complexity Measures Tool for C/C++, is a code metrics tool for C and C++ languages.
devAdvantage devAdvantage is an automated code review tool that analyzes .NET C# code looking for bugs and issues.
GlowCode Memory and resource leak detection, code profiler, function call trace, and report tools for Win32 programs
Insure++ Runtime error detection tool for C and C++
LDRA Testbed Static and code coverage analysis tool. Static analysis features include code visualisation, programming standards enforcement and complexity metrics.
Leak Check Quality tool used for finding memory leaks. No recompile or relink is required! Application code and 3rd party code can be checked.
OSPC Checks applications, written in C for conformance to company coding standards, International standards, and conformance to the C standard
Panorama++ Test Coverage Analysis & Metrics Testing for C & C++
McCabe IQ McCabe IQ Test Team Edition provides comprehensive test/code coverage, (from LOC to the Path Level).
Coverity Static Analysis It helps developers find hard-to-spot, yet potentially crash-causing defects early in the software development life-cycle, reducing the cost, time, and risk of software errors
Rational Purify Pinpoints hard to find memory errors such as uninitialized memory access, buffer overflow and improper freeing of memory.
TCAT C/C++ It is a fully automated tool for static analysis and code coverage at the unit, sub-system, system and integration testing levels.
Test Coverage Family of test coverage tools for a wide variety of languages (C, C++, C#, COBOL, Java, more…). Very low probe overhed. Independent of compiler

1.2.        Functional Testing Tools

Functional Testing Tools validate the functions by feeding them input and examining the output. These tools verify a program by checking it against design documents or specifications. We investigated 27 functional testing tools.

Name

Description

dotTEST Unit-testing tool that automatically tests classes written on Microsoft’s .NET Framework without requiring developers to write a single test scenario or stub.
ATD Automated Test Designer, (ATD) is an easy to use windows client/server tool designed to create Test Cases, Test Data anda Automated Test Scripts based on Requirements.
Avignon Avignon is an acceptance test system that allows the user to write executable tests. Is uses XML to define the syntax of the language.
CAPBAK/X It allows the user to record mouse mouvements, keyboard activities, widget calls and verification information into test scripts.
Certify Worksoft Certify is an automated functional testing solution for SAP lifecycle management and cross-platform business process validation.
CitraTest It is a software test automation solution for automating functional testing and automating any manual process for any application, that is accessible from a Windows client.
Eggplant It is a QA automation tool that is easy to use. It acts as a virtual user to generate and run test scripts and to automate other tasks on the remote machine.
GUITAR It is a model-based system for automated GUI Testing. It generates new test cases using event-flow graphs. Pre/postconditions are used to generate expected response.
Holodeck It is a testing tool that uses fault simulation to emulate real-world application and system errors. It allows users to analyze and debug error-handling code in hostile environments.
Jenny It is a tool for generating regression tests. It covers the interactions using pairwise testing.
JPdfUnit It is a framework for testing a generated pdf document with JUnit test framework so JPdfUnit is a high level API. It is designed for an easy access to the PDFBox library.
JSystem It is a open-source framework for writing and managing automated system tests. It allows the user to create and run test-scenarios using drag and drop approach.
MITS.GUI Has an intelligent state machine engine that makes real-time decisions for navigating through the GUI portion of an application without scripts.
PETA It is a platform for automated software testing of distributed software systems like client-server or service oriented architectures (SOA) on the integration or system layer.
PyUnit It is a testing framework. It is a Python language version of JUnit.
Repro It is a tool that records the software process execution at multiple levels to provide unmatched visibility into process execution.
RFT Rational Functional Tester is an automated functional testing and regression tool. It enables testers to automate tests resilient to frequent application user interface changes.
ScriptTech It generates executable scripts for QTP, TetPartner, Open Source as well as Java, C++, VB, C#, TCL or other custom scripting. It also creates scripts prior to code availability.
SilkTest It is a tool for automated function and regression testing. It creates test automation across a broad range of application technologies to identify quality problems.
SilverMark’s Test Mentor It is a functional test and test modeling tool for Java developers to use as they develop their Java classes, clusters, subsystems, frameworks, and other components.
Squish It is a cross platform/cross technology GUI test automation tool for functional GUI regression tests. It is possible to test the application using recorded or manually written test scripts.
TestArchitect It is a test automation framework. It includes automation support for the most common platforms, along with version-control features to support team-based test automation.
TestComplete It is an automated test manager to create, manage and run tests for any Windows, Web, or Rich Client software.
TestWorks Integrated suite of Automated Regression Test and Coverage Analyzers for UNIX, Windows with multiple reports and re-usable test scripts in C, C++ and JAVA.
Vermont HighTest Plus It is a regression-testing tool for testing stand-alone or browser-based applications. It supports full scripting language with variable, loops and conditionals.
T-Plan Robot It is a black box automated testing tool, also known as VNCRobot. It provides human-like approach to software testing.
HP UFTS HP Unified Functional Testing software accelerates automated software testing by simplifying test design and maintenance. It also validates integrated test scenarios.

1.3.        Performance Testing Tools

Performance Testing Tools  are used to determine the speed or effectiveness of the application involving quantitative tests and qualitative attributes such as reliability, scalability, and interoperability. We studied 12 Performance testing tools.

Name

Description

BugTimer It is a timer application designed to streamline the entire process of timing and documenting performance test results. It also records, displays, saves, sorts, and prints the test results.
CPT Certify Performance Test provides an intuitive yet powerful approach to assuring SAP applications operate efficiently and reliably.
DTM DB Stress It is a utility for stress testing the server parts of information systems and applications, as well as DBMSs and servers themselves.
IxChariot It is a test tool for simulating real-world applications to predict device and system performance under realistic load conditions.
IxLoad It is a tool for running performance testing applications that can emulate millions of users. It emulates data, voice, and video subscribes.
liSA It offers direct functional and performance validation of component of distributed applications. It also identifies performance bottlenecks with metrics.
LoadeaTest It is a stress testing application that aims to test the functionality of a system when this system is heavily loaded for a given period of time.
LoadRunner It helps to prevent performance problems by detecting and reducing bottlenecks before a system deployment or upgrade.
Shunra It enables the deployment of new applications and infrastructure changes. It provides graphically reports and drill-down transaction.
SilkPerformer It is a performance testing solution for optimizing the performance of business applications. It enables performance tests that simulate peak demand loads.
Performance PRO It isolates performance bottle-necks. It checks every form, combo and list box, and reviews SQL statements. It also reports exactly what to fix to improve the speed of the application.
XtremeLoad It is a software framework for load testing of distributed software systems. It provides an extensible, scalable, and easy-to-use foundation to build load testing. 

1.4.         Java Testing Tools

Java Testing Tools ensure that the problems in source code are addressed during the development. Java Testing tools also ensure functional test classes that understand the application and test the source files thoroughly. Sixteen Java Testing Tools were verified.

Name

Description

Abbot It is a JUnit extension for testing Java GUIs. It supports recording and playback at both low and high levels of abstraction.
adaptiveCells/J It is an open source J2EE benchmarking system. It enables the development of complex artificial J2EE test-beds without requiring a single line of code.
Agitator It provides the understanding of code behavior as developer writes or modifies Java classes or methods. It helps the user to validate the expected behavior.
Bugkilla It is a set of java tools for the functional test of J2EE Web Applications.
Cactus It is a test framework for unit testing server-side java code (Servlets, EJBs, Tag Libs, Filters, etc). It lower the cost of writing tests for server-side code. It uses JUnit and extends it.
GUIdancer It offers support for Code Coverage using the JaCoCo code coverage tool for Java.
JCover It is a code coverage analyzer for Java programs.  It shows coverage overlap or disjoint.
Jemmy It is an open source library that can be used to create tests for Java GUI applications.
JMeter It is an open source software designed to load test functional behavior and measure performance. It can be used to simulate a heavy load on server, network or object.
JStyle It is an automated Java code review tool. It performs Java code inspection as part of every build. It also computes several object-oriented metrics.
Jtest It is an solution for automating a borad range of practices proven to improve development team productivity and software quality.
JUnit It is an open source testing framework for users who implements unit tests in Java.
JVerify It is a Java class/API testing tool that supports a unique invasive testing model.
Marathon It is a testing framework for GUI applications developed using Java/Swing. It composes of recorder, runner and editor. The testscripts are composed of python code.
QF-Test It is a tool for automated testing of Java and Web applications with graphical user interface. It supports regression and load testing.
TCAT/Java It is a code coverage analyzer for Java applications and applets.

1.5.         Embedded Testing Tools

Embedded Testing Tools are responsible for test embedded (real time) systems. Embedded testing tools allow detection of timing issues and fault insertion. We looked over six embedded testing tools.

Name

Description

Reactis Tester It generates comprehensive test suites from Simulink/Stateflow models. The test suites exercise large portions of the software under test while avoiding redundancy.
TBrun It is an automated unit testing tool. Its functionality includes automatic test harness generation, automatic test stub generation and test documentation reporting.
Tessy It offers automated module testing of C and C++ code directly on the target system using debugging technology. It supports the unit testing cycle and works on debuggers.
TestCast It tests embedded software components, processes, subsystems, etc. It contains tools for visualization of messages traffic between the component and their simulated counterparts.
TestQuest It is a test automation solution that tests the end-user experience, and allows testing of multiple devices, enabling end-to-end testing.
VectorCAST It is a world-class integrated software test solution that automates the tasks associated with testing software components.

1.6.         Database Testing Tools

Database Testing Tools focus on test the referential integrity, security (including permissions and privileges), interfaces, and web services of the database. We evaluated 8 database testing tools.

Name

Description

Data Generator It is an automated testing and data generation tool, which enables the test data creation for software quality assurance, performance testing, and database load testing.
DTM Data Generator It is a tool that generates data, tables (views, procedures etc) for database testing (performance testing, QA testing, load testing or usability testing) purposes.
Datatect It generates test data to ASCII flat files or directly to RDBMS including Oracle, Sybase, SQL Server, and Informix, using ODBC.
DTM DB Stress It is an utility for stress testing the server parts of information systems and applications, as well as DBMSs and servers themselves. It allows the creation of a set of requests to the server.
ER/Datagen It generates test data in support of Allfusion ERwin and ER/Studio (enterprise editions).
SQL Profiler It is an module that allows the user to identify, isolate, and correct coding mishaps and performance bottlenecks that could lead to costly performance problems.
TurboData It is a test data generator tool. It also generates Select, Update and Delete SQL. TurboData includes an SQL Builder, Grid Data Editor and has full Version Control.
utPLSQL It is an unit-testing framework for users using Oracle PL/SQL language. It allows the automated testing of packages, functions and procedures.

2.    Web Testing Tools

Web Testing Tools are responsible for testing applications that are accessed over a network. Web applications are popular due to the convenience of using a web browser as a client. For this reason, is necessary test the web applications in different browsers and operational systems. In our work, we found 92 web testing tools organized as follows:

2.1.         Link and HTML Testing Tools

Link and HTML Testing Tools consist of verifing if the connections with hyperlinks are correct and the identification of the HTML and services related are  working. We discovered 17 link and HTML testing tools.

Name

Description

AccVerify It verifies, corrects, monitors and manages websites and web based applications.
ChangeAgent It is a web site repair and maintenance application that makes fixing problems simple.
CSE HTML Validator It eliminates website problems increasing the productivity. HTML, XHTML, CSS, link, spelling, and accessibility checker available.
Cyber Spyder CyberSpyder Link Test is a web site management program to be used for verifying that the URLs on a site are not broken and for analyzing site content.
HPT HTML PowerTools is a suite of Windows tools for HTML checking, spelling, etc.
HTML Tidy It is a tool that fixes common errors and prints HTML.
InFocus It is an integrated solution that includes testing and analysis tools, a rules builder, a checking engine, and content spidering.
Link Checker Link Checker Pro is a link checking tool for websites. It can export results in a number of formats and provide a graphical view of the website structure.
LinkRunner Alert Link Runner is a dead link detector tool.
LinkScan It is a dead link checking and website management tool.
LinkSleuth It checks web sites for broken links. Link verification is done on links, images, frames, plug-ins, backgrounds, local image maps, style sheets, scripts and java applets.
Link Validator Web Link Validator is a link checker and site management tool for webmasters to check links for accuracy and availability, find broken links and links containing syntactic errors.
MOMspider It is a web-roaming robot that specializes in the maintenance of distributed hypertext infrastructure.
Ramp Ascend It is an accessibility testing tool that tests and remediates websites and web based applications for integrated accessibility for Section 508 and WCAG compliance.
Real Validator It is an html syntax checker that uses a SGML parser. It works offline, provides convenience, speed and flexibility.
Truwex It is a tool for evaluating website accessibility, privacy, performance, quality, broken links. It shows issues on a web page map and in the HTML code.
WebLight It finds broken links and markup problems, and identifies relationships between URL’s.

2.2.         On-line Link and Testing Services

On-line Link and Testing Services consist of verifing if the connections with hyperlinks are correct and the identification of the HTML and services related are working. We discovered 17 on-line link and testing services.

Name

Description

AETG Web Service It generates test cases from a model of your requirements. It uses combinatorial design techniques to find a minimal test-case set that covers all pairwise interactions
Audit Blossom It is a web site audit service that finds improperly formatted URLs and references to documents that have either moved or no longer exist.
CSSCheck It is an on-line cascading Style sheet (CSS and CSS2) checker.
CSS Validation It is a service that checks Cascading Style Sheets (CSS) in (X)HTML documents or standalone for conformance to W3C recommendations
Dr. Watson It is a program error debugger that gathers information about the computer when an error (or user-mode fault) occurs with a program.
HTML Toolbox It is a service which searches the site to find broken links, spot bad HTML tags, and rates the server’s response time.
Link Alarm It is an online service for detecting broken links.
Link Checker It looks for issues in links, anchors and referenced objects in a Web page, CSS style sheet, or recursively on a whole web site.
Link Cheching It identifies broken links, analyzes accessibility, reports on search engine optimization, monitors page load times, and helps to manage changes.
HTML Validator It is an online service that validates HTML documents.
Markup Validation It is an online service that checks HTML documents for conformance to W3C HTML and XHTML Recommendations and other HTML standards.
SDT The Software Development Technologies (SDT) software testing tools consulting service help the user to establish or improve the use of software testing tools.
Site Check It runs HTML and stylesheet validators, accessibility assessment, link check, load time check, and more using online web testing tools.
Weblint Gateway It performs an HTML error scan and creates an HTML listing with error messages in context and tags, URLs, and error messages highlighted in color.
WPBCV Web Page Backward Compatibility Viewer is an on-line utility that serves a page based on how various browsers would render it.
Web Page Purifier It is an on-line utility that maps a page to HTML 2.0, HTML 3.2, HTML 4.0, or WebTV 1.1. standards.
XMLV XML Validation tool validates XML documents.

2.3.         Functional Testing Tools

Functional Testing Tools focus on the functionalities from the web systems and its particularities. We detected 32 functional Testing Tools for web applications.

Name

Description

actiWATE It is a Java-based software platform for writing test scripts.
Badboy It is a tool designed to aid in testing and development of complex dynamic applications. It allows the user to capture CGI parameters, pages and framesets.
Canoo WebTest It is a open source tool for testing of web applications. It calls pages and verifies the results, giving comprehensive reports on success and failure.
eValid It provides Browser-Based Client-Side WebSite quality checking for Functional Testing, LoadTest, Site Analysis (SiteMap), Performance check and page tuning.
IeUnit It is a framework to test logical behaviors of web pages. It helps users to create, organize and execute functional unit tests.
Imprimatur It is a web application functional testing tool. It sends http requests to the application, and then validates the responses using regular expressions.
iMacros It is used to capture and replay multi-step business transactions that require repeated form fills on a continuous basis.
HTTP::Recorder It is a browser-independent recorder that records interactions with web sites and produces scripts for automated playback.
ITP It is a web application testing harness that can test the site from the user’s perspective.
liSA It allows the user to fully test dynamic web sites and Rich Internet Application (RIA) browser interfaces, as well as directly testing the building blocks below the UI.
Netvantage Netvantage Functional Tester, functional and regression automated web testing tool.
PesterCat It is a web testing tool designed to perform functional testing for web applications. It features an proxy recorder that enables to record scripts using web browser.
QA Wizard It automates the functional and regression testing of Web, Windows, and Java applications, and load testing of Web applications.
Ranorex It is a GUI test and automation framework for testing applications.
Rational Robot It is a test automation tool for functional testing of client/server applications. It enables defect detection, includes test cases and test management.
Sahi It is an automation and testing tool for web applications, with the facility to record and playback scripts.
SAMIE It is a simple automated module for Internet Explorer that allows a user to automate actions.
Selenium It is a test tool for web applications that tests run directly in a browser, just as real users do.
SilkTest It is a tool for automated function and regression testing, delivering advanced test automation capabilities that can be used by many projects.
SOAtest It automates web application testing, message/protocol testing, cloud testing and security testing.
soapUI It is a application for inspecting, invoking and testing (functional and load) of web services.
Solex It is an open source web application testing tool built as a plug-in for the Eclipse IDE. It provides functions to record a client session, adjust it according to various parameters.
Squish It is a test tool which allows to create, debug and run automated GUI tests for Qt, Tk, XView and HTML/Web applications.
swete It provides cross platform console based tools for regression testing of web applications. It allows to record XML test scripts using any web browser.
TestDrive It is a automated software quality solution designed to test browser and legacy applications, as well as GUIs, including Java, Ajax, Flex and Silverlight.
Test Studio It is a tool for build automated tests for web and desktop applications. It runs tests against all browsers at once. It bundles functional and performance tests.
vTest It is a an automated functional and regression testing tool for web applications. It generates automated test scripts, replays both automated and custom test scripts.
WatiN Is is an open source C# framework for automating functional web testing.
Watir It is an open-source functional testing tool for automating browser-based tests of web applications. It clicks links, fills in forms, presses buttons.
WebInject It is a tool for testing of web applications and services. It is used to test any individual system component with an HTTP interface (JSP, ASP, CGI, PHP, Servlets, HTML Forms).
WET It is a framework for Web automation testing. It has many features like multiple parameter based object identification, support for XML object repository, better popup handling, etc.
Yawet It is a Java/swing tool to create, run and debug functional web tests. It can verify Html, Pdf and Xml documents. Yawet offers reports, step libraries, parametrisation and custom extensions.

2.4.         Security Testing Tools

Security Testing Tools determine that a system protects data and maintains functionality as intended.

Name

Description

Application Security It includes tools for automating and managing application security testing, including static testing of source code in development and dynamic testing of applications.

2.5.         Performance Testing Tools and Services

Performance Testing Tools are used to measure the speed or effectiveness of the application. We observed 25 performance testing tools for web applications.

Name

Description

ANTS ANTS Performance Profiler identifies bottlenecks and optimizes .NET application performance.
AppLoader It is a tool that simulate user actions on applications. It allows to load stress applications or to test the quality of the applications. It also generates performance reports.
CloudBurst It is a cloud-based extension to SilkPerformer, the performance testing offering that enables software quality teams to launch size peak-load performance test.
Dotcom-Monitor It simulates real users by using its remote agents positioned around the globe. It allows the end-user to setup the conditions for the web site or web application to realize stress test.
Forecast It is a tool suite for system load testing, performance measurement and multi-user functional testing. It supports nested and asynchronous transactions.
http_load It is an Unix-Based load load generator for web servers. It runs multiple http fetches in parallel, to test the throughput of a web server.
LoadTracer It is a load-testing tool, which also used for stress and performance testing. It uses various technique to create load on web server as real user does.
MACT Microsoft Application Center Test is designed to stress test Web servers and analyze performance and scalability problems with Web applications, including ASP and components.
NeoLoad The NetLoad Cloud Testing solution simulates hundreds of virtual users on the web site, getting performance statistics and revealing errors under stress. .
OpenLoad It is a web testing tool used to simulate hundreds or even thousands of users simultaneously accessing a web site or application.
OpenSTA It is a distributed software testing architecture based on CORBA. It is designed to be used by Performance Testing Consultants or other technically proficient individuals.
Proxy Sniffer It is a performance web load and stress testing tool. It permits to analyze the performance characteristics and the stability of a Web application under various load conditions.
QEngine It is a test automation tool which supports load testing and functional testing for Web Applications. It performance testing helps at the identification of bottlenecks.
Siege It is an http load testing and benchmarking utility. It was designed to measure code under duress. It supports basic authentication, cookies, http and https protocols.
StressIT It is a robust stress testing application that simulates the number of users the site can maintain. It charts the breaking point at which performance is no longer acceptable.
SiteStress Webmetrics Full-Service Load Testing is a managed solution that gather requirements, define scenarios, run a series of load tests, and review the results.
Site Tester It is a load-testing utility designed to test web servers and web applications. It simulates concurrent access of various users from different hosts to a web/application server.
TestMaker It is an open source framework to build intelligent test agents that test Web services for scalability and performance. It surface performance bottlenecks and functional issues.
Wbox It checks if the redirections are working correctly. It tests if the http compression is working.
WebLOAD It is a software for performance testing of internet applications. It helps to meet the performance requirements by simulating traffic of up to tens of thousands of users.
Web Performance Web Performance Load Tester is a web testing tool that analyze how many users a website can handle. It automatically detects and configures the test cases.
Web Polygraph It is a performance testing tool for caching proxies, origin server accelerators, content filters, and other web intermediaries.
WAPT It is a load and stress testing tool that provides a way of testing web sites, web servers, and intranet applications with web interfaces.
Web Server Stress Tool It is a load test application for web servers that tests the web server performance under normal and excessive loads to ensure if the information and services are available.
WebSizr It is a performance analysis and sizing framework for http based servers and applications. It is a server measurement tool that simulate client behavior.


3.    Bug Tracking Tools

Bug Tracking Tools are designed to report and keep track of defects in software programs. Divided into Web Based and Bug Tracking Applications. We recognized 38 Bug tracking tools.

3.1.         Web Based Bug Tracking

We analysed 28 Web Based Bug Tracking Tools.

Name

Description

AceProject It can be used as an issue management program and bug tracking software.
AdminiTrack It is an issue and task tracking application.
ADT Web Advanced Defect Tracking Web Edition is a defect-tracking solution. The software allows you to track defects, feature requests and suggestions by version, customer etc.
ALMComplete It tracks status and resolution progress of defects and issues across releases. It coordinates development and remediation activities as bugs are found.
BugAware It is a Bug tracking solution. Features include: Email alert notification, dynamic reporting, team management, user discussion threads, file attachment, searching.
Bugcentral It is a web based defect tracking service.
BUGtrack It is a web based bug tracking system.
BugHost It is a feature-complete hosted defect tracking system. The system is easy to use and customizable to meet the users needs.
BugStation It is a web-based application with many enhancements designed to make Bugzilla easier and more secure. A centralized system for entering, assigning and tracking defects.
Bug Tracker It is an open-source, web based bug tracking and general purpose issue tracking application. It is configurable with email integration, search, screen capture, etc.
Bug Tracking It is a web based detect tracking software. It offers email notification, file attachment, tracking history, bilingual pages, etc.
Bugzero It is a bug tracking system for software detect tracking, and a general management tool for customer and sales tracking and helpdesk customer support.
Bugzilla It is a server software designed to help the software development management. It is an open source defect tracking system developed originally for the Mozilla project.
Census It is a bug tracking and defect tracking tool. It includes VSS integration, notifications, workflow, reporting and change history.
Dragonfly It is a web-based, cross-browser, cross-platform issue tracking and change management for software development, testing, debugging, and documentation.
FogBugz It is a web-based bug tracking system.
JIRA It is a issue tracking and project management application.
JTrac It is an open-source and customizable issue-tracking web-application.
Mantis It is a web-based bug tracking system.
Ozibug It is a platform-independent defect tracking system. It can be used to manage the reporting of bugs and tracking the progress.
ProblemTracker It is a web-based collaboration software for issue tracking, automated support, and workflow, process, and change management.
PR Tracker It is an enterprise level problem tracking system designed for bug tracking.
Squish It is a web-based management tool that keep track of product development, client request, software bugs, help desk tickets, tasks, and more.
teamatic It is a web-based bug tracking system.
TrackStudio It is a configurable issue tracking system, workflow engine and document management system that can be used to track the progress of any asset.
VisionProject It is a web based issue tracking and project collaboration tool designed to make projects more efficient and profitable.
Woodpecker IT It is a web-based request, issue or bug tracking tool. It can be used to manage business processes, requirement management or bug tracking.
yKAP It is a web based Bug/Defect tracking, Issue Management and Messaging product. It allows the user to add custom values for system parameters.

3.2.         Bug Tracking Applications

We analysed 10 Bug Tracking Tools.

Name

Description

BugRat It is a software that provides a defect reporting and tracking system.
Bug Trail It captures and tracks software bugs. It allows the user to attach screenshots, automatically capture system parameters and create output reports.
Defect Manager It is a grade bug and issue tracking system. It tracks bugs, defects, calls, tasks and enhancements through th software development life cycle.
FIT BugTrack It implements a bug tracking solution using a system designed for the management of software bugs. It eliminates unnecessary duplication of effort and resources.
GNATS It is a defect tracking software.
IssueNet IssueNet platform provides an extensible framework to handle the many issue management needs. It has several pre-build solutions.
IssueView It is an application that allows the management of bugs.
TestTrack It is a scalable issue management tool.
Trac It is an issue tracking system for software development projects. It provides an interface to version control systems.
ZeroDefect It is an issue management system.

4.    Testing Management Tools

Test management tools are used to structure automated tests and manual test processes, and manage multiple environments. Quality assurance teams use these tools for managing test cases, environments, automated tests, defects and project tasks. We encountered 10 Testing Management Tools.

Name

Description

ApTest Manager It is a tool for managing testing of hardware and software products. It defines test requirements and specifications, establishing test plans, and executing tests.
Concerto Test Center It generates, assigns/distributes, and facilitates specifics tasks. It also correlates test results to policy and functional requirements.
HP Quality Center It is a scalable, unified platform for managing and automating the delivery of secure, reliable, quality applications. It enables the implementation of a management infrastructure.
RQM Rational Quality Manager is a centralized test management environment that helps increase the efficiency and quality of software delivery.
SilkCentral It is a test management solution for agile and traditional development projects. It provides an integrated framework for improving productivity and traceability for software testing.
SQS-TEST Test Center It is a tool for test management, test specification and test execution for quality assurance in software development projects.
T-Plan It allows the user to manage the test assets from planning to execution.
Test Manager It enables the user to use requirements management tools to create and manage requirements. Borland CaliberRM Adapter, Telelogic DOORS Adapter, Rational ClearQuest Adapter.
TestLog It is an integrated test case management software solution. It provides a tool for software test teams to create and update effective test plans and test cases.
QAComplete It is an application for test and QA management including requirements management, test case management, and defect tracking.

5.    API Testing Tools

API (application programming interface) testing tools are used to evaluate the application using public and private APIs. It focus on the validation of interfaces. We analysed 4 API Testing tools.

Name

Description

ADL project repository The Assertion Definition Language (ADL) is an automated test generation system. It is a formal grammar for describing the behavior of interfaces.
MITS.Comm It simulates third party systems over any type of interface for testing anything from electronics to web-related applications.
DejaGNU It is a framework for testing interactive programs that provides a single front end for tests.
TET The Test Environment Toolkit (TET) is provided as an open source product.

6.    Communications Testing Tools

Communications Testing Tools group the tools that test network communication between applications, network performance, and multiple concurrency. We investigated 11 Communications testing tools. For example, WebMNS Simulation Toolkit is a network simulator for testing the devices communication.

Name

Comments

CWS The Cellogic Wireless System (CWS) is a tool for introduction, optimization and maintainance of wireless communication services.
Fault Factory It is a active debugging solution for applications that communicate with other applications over the network. It utilizes API-Level Fault Injection to imitate faults.
IxANVL IxANVL (Automated Network Validation Library) is a system for testing of network products. It enables network product to testing processes faster and more thorough.
LANTraffic It is a traffic generator for IP networks using different protocols (TCP, ICMP, SCTP, and UDP). It is independent of any transmission or telecom line.
Maxwell It enables the user to modify, distort, and corrupt the flow of network traffic. It also operates in real time and modifies traffic while the protocol discussion continues.
NetHawk EAST It is a simulator used by equipment manufacturers and operators in load testing, regression testing and functional testing. It is available for VoIP, SS7, and wireless.
nGenius nGenius Performance Manager provides mobile and fixed operators with a unified view into subscriber, service and network performance for IP networks.
NuStreams 2000 It is a multichannel gigabit Ethernet test systems for LAN, WAN, network, switch, router and IP device testing. Test, measure, certify and stress your networks and devices.
Silvercreek  Silvercreek SNMP Test Suite verifies protocol compliance, error and exception handling, boundary conditions, and performance under heavy network loads.
SNAsim It is a SNA testing and traffic generation tool. It provides a PC-based environment for simulating the 3270 and 5250 data streams.
WebNMS It is a complete device simulation and network simulation software for testing and demonstration of NMS/EMS in multiple operating systems.