Content deleted Content added
→Strengths and weaknesses: note problems for CI |
Citation bot (talk | contribs) Alter: title, template type. Add: chapter-url, chapter. Removed or converted URL. Removed parameters. Some additions/deletions were parameter name changes. | Use this bot. Report bugs. | Suggested by Headbomb | Linked from Wikipedia:WikiProject_Academic_Journals/Journals_cited_by_Wikipedia/Sandbox | #UCB_webform_linked 686/1156 |
||
| (40 intermediate revisions by 24 users not shown) | |||
Line 1:
{{
'''Random testing''' is a black-box [[software testing]] technique where programs are tested by [[random number generation|generating]] random, independent inputs. Results of the output are compared against software specifications to verify that the test output is pass or fail.<ref name="Hamlet94"/> In case of absence of specifications the exceptions of the language are used which means if an exception arises during test execution then it means there is a fault in the program, it is also used as a way to avoid biased testing.
==
Random testing for hardware was first examined by [[Melvin Breuer]] in 1971 and initial effort to evaluate its effectiveness was done by Pratima and [[Vishwani Agrawal]] in 1975.<ref>
In software, Duran and Ntafos had examined random testing in 1984.<ref>{{cite journal|title=An Evaluation of Random Testing|first1=J. W.|last1=Duran|first2=S. C.|last2=Ntafos|date=1 July 1984|journal=IEEE Transactions on Software Engineering|volume=SE-10|issue=4|pages=438–444|doi=10.1109/TSE.1984.5010257}}</ref>
▲Random testing for hardware was first examined by [[Melvin Breuer]] in 1971 and initial effort to evaluate its effectiveness was done by Pratima and [[Vishwani Agrawal]] in 1975.<ref>[http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1672882&tag=1 Agrawal, P.; Agrawal, V.D., "Probabilistic Analysis of Random Test Generation Method for Irredundant Combinational Logic Networks," Computers, IEEE Transactions on , vol.C-24, no.7, pp.691,695, July 1975]</ref>
The use of hypothesis testing as a theoretical basis for random testing was described by Howden in ''Functional Testing and Analysis''. The book also contained the development of a simple formula for estimating the number of tests ''n'' that are needed to have confidence at least 1-1/''n'' in a failure rate of no larger than 1/n. The formula is the lower bound ''n''log''n'', which indicates the large number of failure-free tests needed to have even modest confidence in a modest failure rate bound.<ref name=":0">{{Cite book|last=Howden|first=William|title=Functional Program Testing and Analysis|publisher=McGraw Hill|year=1987|isbn=0-07-030550-1|___location=New York|pages=51–53}}</ref>
==
Consider the following C++ function:
<syntaxhighlight lang="cpp">
int myAbs(int x) {
if (x > 0) {
return x;
}
Line 22 ⟶ 23:
</syntaxhighlight>
Now the random tests for this function could be {123, 36, -35, 48, 0}. Only the value '-35' triggers the bug. If there is no reference implementation to check the result, the bug still could go unnoticed. However, an [[assertion (software development)|assertion]] could be added to check the results, like:
<syntaxhighlight lang="cpp">
Line 29 ⟶ 30:
int x = getRandomInput();
int result = myAbs(x);
assert(result >= 0);
}
}
Line 37 ⟶ 38:
<syntaxhighlight lang="cpp">
int getRandomInput() {
}
Line 45 ⟶ 46:
long y = getRandomInput();
long result = fastMultiplication(x, y);
assert(x * y == result);
}
}
Line 52 ⟶ 53:
While this example is limited to simple types (for which a simple random generator can be used), tools targeting object-oriented languages typically explore the program to test and find generators (constructors or methods returning objects of that type) and call them using random inputs (either themselves generated the same way or generated using a pseudo-random generator if possible). Such approaches then maintain a pool of randomly generated objects and use a probability for either reusing a generated object or creating a new one.<ref name="AutoTest"/>
==
According to the seminal paper on random testing by D. Hamlet
<blockquote>[..] the technical, mathematical meaning of "random testing" refers to an explicit lack of "system" in the choice of test data, so that there is no correlation among different tests.<ref name=Hamlet94>{{cite book|title=Encyclopedia of Software Engineering|year=1994|publisher=John Wiley and Sons|isbn=978-0471540021|author=Richard Hamlet|edition=
==
{{
Random testing is
*
*
*
*
The following weaknesses
*
*
*
*
*Some argue that it would be better to thoughtfully cover all relevant cases with manually constructed tests in a white-box fashion, than to rely on randomness.<ref name="so" />
*It may require a very large number of tests for modest levels of confidence in modest failure rates. For example, it will require 459 failure-free tests to have at least 99% confidence that the probability of failure is less than 1/100.<ref name=":0" />
==
===
*
*
*
===
*
*
== Implementations ==
Some tools implementing random testing:▼
* [[QuickCheck]] - a famous test tool, originally developed for [[Haskell (programming language)|Haskell]] but ported to many other languages, that generates random sequences of API calls based on a model and verifies system properties that should hold true after each run. Check this [http://www.quviq.com/documents/QuviqFlyer.pdf QuviQ QuickCheck flyer] for a quick overview.▼
* [https://randoop.github.io/randoop/ Randoop] - generates sequences of methods and constructor invocations for the classes under test and creates [[JUnit]] tests from these▼
* [https://github.com/Datomic/simulant/wiki/Overview Simulant] - a [[Clojure]] tool that runs simulations of various agents (f.ex. users with different behavioral profiles) based on a statistical model of their behavior, recording all the actions and results into a database for later exploration and verification▼
* [https://docs.eiffel.com/book/eiffelstudio/autotest AutoTest] - a tool integrated to EiffelStudio testing automatically Eiffel code with contracts based on the eponymous research prototype.<ref name="AutoTest"/>·▼
* [https://code.google.com/p/yeti-test/ York Extensible Testing Infrastructure (YETI)] - a language agnostic tool which targets various programming languages (Java, JML, CoFoJa, .NET, C, Kermeta).▼
* [https://github.com/codelion/gramtest GramTest] - a grammar based random testing tool written in Java, it uses BNF notation to specify input grammars.▼
▲Some tools implementing random testing:
== Critique ==▼
▲* [[QuickCheck]] - a famous test tool, originally developed for [[Haskell (programming language)|Haskell]] but ported to many other languages, that generates random sequences of API calls based on a model and verifies system properties that should hold true after each run
▲*
▲*
▲*
▲*
▲*
<blockquote>Random testing has only a specialized niche in practice, mostly because an effective oracle is seldom available, but also because of
For programming languages and platforms which have contracts (
==
*
*
*
*[[Corner case]]
*[[Edge case]]
*[[Concolic testing]]
==
{{Reflist}}<!--<ref name=":0" />-->
==
*
*
*
{{software testing}}
[[Category:Software testing]]
| |||