Random testing: Difference between revisions

Revision as of 05:54, 16 February 2021

Latest revision as of 10:04, 9 February 2025

(7 intermediate revisions by 5 users not shown)

Line 1:

{{Refimprove|date=September 2014}}

'''Random testing''' is a black-box [[software testing]] technique where programs are tested by [[random number generation|generating]] random, independent inputs. Results of the output are compared against software specifications to verify that the test output is pass or fail.<ref name="Hamlet94"/> In case of absence of specifications the exceptions of the language are used which means if an exception arises during test execution then it means there is a fault in the program, it is also used as a way to avoid biased testing.

==History of random testing==

Line 7:

In software, Duran and Ntafos had examined random testing in 1984.<ref>{{cite journal|title=An Evaluation of Random Testing|first1=J. W.|last1=Duran|first2=S. C.|last2=Ntafos|date=1 July 1984|journal=IEEE Transactions on Software Engineering|volume=SE-10|issue=4|pages=438–444|doi=10.1109/TSE.1984.5010257}}</ref>

The use of hypothesis testing as a theoretical basis for random testing was described by Howden in ''Functional Testing and Analysis''. The book also contained the development of a simple formula for estimating the number of tests ''n'' that are needed to have confidence at least 1-1/''n'' in a failure rate of no larger than 1/n. The formula is the lower bound ''n''log''n'', which indicates the large number of failure-free tests needed to have even modest confidence in a modest failure rate bound.<ref name=":0">{{Cite book|last=Howden|first=William|title=Functional Program Testing and Analysis|publisher=McGraw Hill|year=1987|isbn=0-07-030550-1|___location=New York|pages=51-5351–53}}</ref>

==Overview==

Line 66:

The following weaknesses have been described :

*It only finds basic bugs (fe.exg. [[null pointer]] dereferencing).

*It is only as precise as the specification and specifications are typically imprecise.

*It compares poorly with other techniques to find bugs (fe.exg. [[static program analysis]]).

*If different inputs are randomly selected on each test run, this can create problems for [[continuous integration]] because the same tests will pass or fail randomly.<ref name="so">{{cite web|url=https://stackoverflow.com/q/636353 |title=Is it a bad practice to randomly-generate test data?|website=stackoverflow.com|accessdate=15 November 2017}}</ref>

*Some argue that it would be better to thoughtfully cover all relevant cases with manually constructed tests in a white-box fashion, than to rely on randomness.<ref name="so" />

Line 77:

===With respect to the input===

*Random input sequence generation (i.e. a sequence of method calls)

*Random sequence of data inputs (sometimes called stochastic testing) - fe.exg. a random sequence of method calls

*Random data selection from existing database

===Guided vs. unguided===

*undirected random test generation - with no heuristics to guide its search

*directed random test generation - fe.exg. "feedback-directed random test generation"<ref name="PachecoLET2007">{{cite journalbook|last=Pacheco|first=Carlos|author2=Shuvendu K. Lahiri |author3=Michael D. Ernst |author4=Thomas Ball |titlechapter=Feedback-directedDirected randomRandom testTest Generation generation|journaltitle=ICSE '07: Proceedings of the 29th International Conference on Software Engineering (ICSE'07)|date=May 2007|pages=75–84|doi=10.1109/ICSE.2007.37 |isbn=978-0-7695-2828-1 |chapter-url=http://people.csail.mit.edu/cpacheco/publications/feedback-random.pdf|issn=0270-5257}}</ref> and "adaptive random testing" <ref name="ART">{{citation |last1=T.Y. Chen |last2=F.-C. Kuo |last3=R.G. Merkel |last4=T.H. Tse |title=Adaptive random testing: The ART of test case diversity |journal=Journal of Systems and Software |volume=83 |issue=1 |pages=60-6660–66 |date=2010 |doi=10.1016/j.jss.2009.02.022|hdl=10722/89054 |url=https://figshare.com/articles/journal_contribution/26243711 |hdl-access=free }}</ref>

== Implementations ==

Line 89:

* [[QuickCheck]] - a famous test tool, originally developed for [[Haskell (programming language)|Haskell]] but ported to many other languages, that generates random sequences of API calls based on a model and verifies system properties that should hold true after each run.

* Randoop - generates sequences of methods and constructor invocations for the classes under test and creates [[JUnit]] tests from these

* Simulant - a [[Clojure]] tool that runs simulations of various agents (fe.exg. users with different behavioral profiles) based on a statistical model of their behavior, recording all the actions and results into a database for later exploration and verification

* AutoTest - a tool integrated to EiffelStudio testing automatically Eiffel code with contracts based on the eponymous research prototype.<ref name="AutoTest"/>·

* York Extensible Testing Infrastructure (YETI) - a language agnostic tool which targets various programming languages (Java, JML, CoFoJa, .NET, C, Kermeta).

Line 99:

A [[test oracle]] is an instrument for verifying whether the outcomes match the program specification or not. An operation profile is knowledge about usage patterns of the program and thus which parts are more important.

For programming languages and platforms which have contracts (for examplee.g. Eiffel. .NET or various extensions of Java like JML, CoFoJa...) contracts act as natural oracles and the approach has been applied successfully.<ref name="AutoTest">{{cite web|url=http://se.inf.ethz.ch/research/autotest/|title=AutoTest - Chair of Software Engineering|website=se.inf.ethz.ch|accessdate=15 November 2017}}</ref> In particular, random testing finds more bugs than manual inspections or user reports (albeit different ones).<ref name="ManualvsRandom">{{cite journal|title=On the number and nature of faults found by random testing|year=2009|author=Ilinca Ciupa|author2=Alexander Pretschner|author3=Manuel Oriol|author4=Andreas Leitner|author5=Bertrand Meyer|journal=Software Testing, Verification and Reliability|doi=10.1002/stvr.415|volume=21|pages=3–28}}</ref>

==See also==

*[[Fuzz testing]] - a kind of random testing which provides invalid input to the tested program

*[[Lazy systematic unit testing#Systematic Testing]] - a systematic way of exploring "all" method calls, as implemented e.g. by NASA's [https://web.archive.org/web/20110107010258/http://babelfish.arc.nasa.gov/trac/jpf/wiki/intro/what_is_jpf Java Path Finder] (which blends [https://web.archive.org/web/20110106040136/http://babelfish.arc.nasa.gov/trac/jpf/wiki/intro/testing_vs_model_checking testing with model checking] by limiting the state space to a reasonable size by various means)

*[[SystemVerilog#Constrained random generation|Constrained random generation in SystemVerilog]]

*[[Corner case]]

Line 117:

*[http://c2.com/cgi/wiki?RandomTesting Random Testing wiki] at Cunningham & Cunningham, Inc.

{{software testing}}

[[Category:Software testing]]