Random testing is a black-box software testing technique where programs are tested by generating random inputs. Results of the output are compared against software specifications to verify that the test output is pass or fail. In case of absence of specifications the exceptions of the language are used which means if an exception arises during test execution then it means there is a fault in the program.
Overview
Consider the following C++ function:
int myAbs(int x) {
if (x>0) {
return x;
}
else {
return x; // bug: should be '-x'
}
}
Now the random tests for this function could be {123, 36, -35, 48, 0}. Only the value '-35' triggers the bug. If there is no reference implementation to check the result, the bug still could be noticed. However, an assertion could be added to check the results, like:
void testAbs(int n) {
for (int i=0; i<n; i++) {
int x = getRandomInput();
int result = myAbs(x);
assert(result>=0);
}
}
The reference implementation is sometimes available, e.g. when implementing a simple algorithm in a much more complex way for better performance. For example, to test implementation of Schönhage–Strassen algorithm algorithm, standard "*" operation on integers can be used:
int getRandomInput() {
. . .
}
void testFastMultiplication(int n) {
for (int i=0; i<n; i++) {
long x = getRandomInput();
long y = getRandomInput();
long result = fastMultiplication(x, y);
assert(x*y==result);
}
}
On randomness
According to the seminal paper on random testing by D. Hamlet
[..] the technical, mathematical meaning of "random testing" refers to an explicit lack of "system" in the choice of test data, so that there is no correlation among different tests.[1]
Types of random testing
With respect to the input
- Random input sequence generation (i.e. a sequence of method calls)
- Random sequence of data inputs (sometimes called stochastic testing) - f.ex. a random sequence of method calls
- Random data selection from existing database
Guided vs. unguided
- undirected random test generation - with no heuristics to guide its search
- directed random test generation - f.ex. "feedback-directed random test generation"[2]
Implementations
Some tools implementing random testing:
- QuickCheck - a famous test tool, originally developed for Haskell but ported to many other languages, that generates random sequences of API calls based on a model and verifies system properties that should hold true after each run. Check this QuviQ QuickCheck flyer for a quick overview.
- Randoop - generates sequences of methods and constructor invocations for the classes under test and creates JUnit tests from these
- Simulant - a Clojure tool that runs simulations of various agents (f.ex. users with different behavioral profiles) based on a statistical model of their behavior, recording all the actions and results into a database for later exploration and verification
See also
- Fuzz testing - a kind of random testing when invalid input is provided to tested program.
- Lazy systematic unit testing#Systematic Testing - a systematic way of exploring "all" method calls, as implemented e.g. by NASA's Java Path Finder (which blands testing with model checking by limiting the state space to a reasonable size by various means)
- Constrained random generation in SystemVerilog
References
- ^ Richard Hamlet (1994). "Random Testing". In John J. Marciniak (ed.). Encyclopedia of Software Engineering (PDF) (1 ed.). John Wiley and Sons. ISBN 0471540021. Retrieved 16 June 2013.
- ^ Pacheco, Carlos (2007). "Feedback-directed random test generation" (PDF). ICSE '07: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society: 75–84. ISSN 0270-5257.
{{cite journal}}: Unknown parameter|coauthors=ignored (|author=suggested) (help); Unknown parameter|month=ignored (help)
External links
- Random testing by Andrea Arcuri.
- Random testing by Richard Hamlet, professor emeritus at Portland State University; a valuable list of resources at the end of the paper
- Random Testing wiki at Cunningham & Cunningham, Inc.