|
===Continuous testing===
[[Continuous testing]] is the process of executing automated tests as part of the software delivery pipeline to assesassess the business risk of releasing the SUT.<ref name="essential">[https://www.techwell.com/techwell-insights/2015/08/part-pipeline-why-continuous-testing-essential Part of the Pipeline: Why Continuous Testing Is Essential], by Adam Auerbach, TechWell Insights August 2015</ref><ref name="stickym">[http://www.stickyminds.com/interview/relationship-between-risk-and-continuous-testing-interview-wayne-ariola The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola], by Cameron Philipp-Edmonds, Stickyminds December 2015</ref> The scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.<ref name="pnsqc">[http://uploads.pnsqc.org/2015/papers/t-007_Ariola_paper.pdf DevOps: Are You Pushing Bugs to Clients Faster], by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015</ref>
===Model-based testing===
===Test-driven development===
[[Test-driven development]] (TDD) inherently includes the generation of automation test code. [[Unit test]] code is written while the SUT code is written. When the code is complete, the tests are complete as well.<ref name="Learning TDD">{{cite journal|doi=10.1109/ms.2007.80|title=Learning Test-Driven Development by Counting Lines|year=2007|last1=Vodde|first1=Bas|last2=Koskela|first2=Lasse|journal=IEEE Software|volume=24|issue=3|pages=74–79|s2cid=30671391}}</ref>
===Other===
Other test automation techniques include:
* [[Data-driven testing]]
* [[Modularity-driven testing]]
* [[Keyword-driven testing]]
* [[Hybrid testing]]
* [[Behavior driven development]]
==Considerations==
==Roles==
To support coded automated testing, the [[test engineer]] or [[software quality assurance]] person must have software coding ability. Some testing techniques such as table-driven and no-code can lessen or alleviate the need for programming skill.
===Test automation tools===
Test automation tools can be expensive and are usually employed in combination with manual testing. Test automation can be made cost-effective in the long term, especially when used repeatedly in [[regression testing]]. A good candidate for test automation is a test case for common flow of an application, as it is required to be executed (regression testing) every time an enhancement is made in the application. Test automation reduces the effort associated with manual testing. Manual effort is needed to develop and maintain automated checks, as well as reviewing test results.
===Test engineer=Framework==
A test automation [[Software framework |framework]] provides a programming environment that integrates test logic, test data, and other resources. The framework provides the basis of test automation and simplifies the automation effort. Using a [[Software framework |framework]] can lower the cost of test development and [[Software maintenance |maintenance]]. If there is change to any [[Test case (software)|test case]] then only the test case file needs to be updated and the driver script and startup script will remain the same.
{{Main|Test engineer}}
In automated testing, the [[test engineer]] or [[software quality assurance]] person must have software coding ability since the test cases are written in the form of source code which when run produce output according to the [[Assertion (computing)|assertion]]s that are a part of it. Some test automation tools allow for test authoring to be done by keywords instead of coding, which do not require programming.
A framework is responsible for defining the format in which to express expectations, providing a mechanism to hook into or drive the SUT, executing the tests, and reporting results.<ref>{{cite web
== Testing at different levels ==
A strategy to decide the amount of tests to automate is the test automation pyramid. This strategy suggests to write three types of tests with different granularity. The higher the level, less is the amount of tests to write.<ref name=":0" />
=== Unit, service, and user interface levels ===
[[File:The test automation pyramid.png|thumb|The test automation pyramid proposed by Mike Cohn<ref name=":0" />]]
* As a solid foundation, [[unit testing]] provides robustness to the software products. Testing individual parts of the code makes it easy to write and run the tests. Developers write unit tests as a part of each story and integrate them with CI.<ref>{{Cite web |title=Full Stack Testing by Gayathri Mohan |url=https://www.thoughtworks.com/en-us/insights/books/full-stack-testing |access-date=2022-09-13 |website=www.thoughtworks.com |language=en-US}}</ref>
* The service layer refers to testing the services of an application separately from its user interface, these services are anything that the application does in response to some input or set of inputs.
* At the top level we have [[UI Testing|UI testing]] which has fewer tests due to the different attributes that make it more complex to run, for example the fragility of the tests, where a small change in the user interface can break a lot of tests and adds maintenance effort.<ref name=":0">{{cite book | author = Mike Cohn | title = Succeeding with Agile | year = 2010 | publisher = Raina Chrobak | isbn = 978-0-321-57936-2}}</ref><ref>[https://martinfowler.com/articles/practical-test-pyramid.html The Practical Test Pyramid], by Ham Vocke</ref>
=== Unit, integration, and end-to-end levels ===
[[File:Testing Pyramid.png|alt=A triangular diagram depicting Google's "testing pyramid". Progresses from the smallest section "E2E" at the top, to "Integration" in the middle, to the largest section "Unit" at the bottom.|thumb|Google's testing pyramid<ref name=":1">{{Cite web |title=Just Say No to More End-to-End Tests |url=https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html |access-date=2023-02-11 |website=Google Testing Blog |language=en}}</ref>]]
One conception of the testing pyramid contains unit, integration, and end-to-end unit tests. According to [[Google]]'s testing blog, unit tests should make up the majority of your testing strategy, with fewer integration tests and only a small amount of end-to-end tests.<ref name=":1" />
* Unit tests: These are tests that test individual components or units of code in isolation. They are fast, reliable, and isolate failures to small units of code.
* Integration tests: These tests check how different units of code work together. Although individual units may function properly on their own, integration tests ensure that they operate together coherently.
* End-to-end tests: These test the system as a whole, simulating real-world usage scenarios. They are the slowest and most complex tests.
==Framework approach in automation==
A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort.
The main advantage of a [[Software framework|framework]] of assumptions, concepts and tools that provide support for automated software testing is the low cost for [[Software maintenance|maintenance]]. If there is change to any [[test case]] then only the test case file needs to be updated and the [[driver Script]] and [[startup script]] will remain the same. Ideally, there is no need to update the scripts in case of changes to the application.
Choosing the right framework/scripting technique helps in maintaining lower costs. The costs associated with test scripting are due to development and maintenance efforts. The approach of scripting used during test automation has effect on costs.
Various framework/scripting techniques are generally used:
# Linear (procedural code, possibly generated by tools like those that use record and playback)
# Structured (uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements)
#[[Data-driven testing|Data-driven]] (data is persisted outside of tests in a database, spreadsheet, or other mechanism)
#[[Keyword-driven testing|Keyword-driven]]
# Hybrid (two or more of the patterns above are used)
# Agile automation framework
The Testing framework is responsible for:<ref>{{cite web
| url = https://www.youtube.com/watch?v=qf2i-xQ3LoY
| title = Selenium Meet-Up 4/20/2010 Elisabeth Hendrickson on Robot Framework 1of2
}}</ref>
Various types of frameworks are available:
# defining the format in which to express expectations
# creating a mechanism to hook into or drive the application under test
# executing the tests
# reporting results
* Linear {{endash}} procedural code, possibly generated by tools like those that use record and playback
===Unit testing frameworks===
* Structured {{endash}} uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements
A growing trend in software development is the use of [[unit testing]] frameworks such as the [[xUnit]] frameworks (for example, [[JUnit]] and [[NUnit]]) that allow the execution of unit tests to determine whether various sections of the [[code]] are acting as expected under various circumstances. [[Test case]]s describe tests that need to be run on the program to verify that the program runs as expected.
* [[Data-driven testing |Data-driven]] {{endash}} data is persisted outside of tests in a database, spreadsheet, or other mechanism
* [[Keyword-driven testing |Keyword-driven]]
* Hybrid {{endash}} multiple types are used
* Agile automation framework
* Unit testing {{endash}} some frameworks are intended primarily for [[unit testing]] such as [[xUnit]], [[JUnit]] and [[NUnit]]
===Test automation interface===
TestA test automation interfacesinterface areis a platformsplatform that provideprovides a single [[workspace]] for incorporating multiple testing tools and frameworks for [[system testing |Systemsystem/Integrationintegration testing]]. of application underA test. Theautomation goalinterface of Test Automation Interface is tomay simplify the process of mapping tests to business criteria without coding. comingA in the way of the process. Testtest automation interface are expected tomay improve the efficiency and flexibility of maintaining test scriptstests.<ref name="Interface">{{cite web
| url = http://www.qualitycow.com/Docs/ConquestInterface.pdf
| title = Conquest: Interface for Test Automation Design
[[File:Test Automation Interface.png|thumb|Test Automation Interface Model]]
TestA Automationtest Interfaceautomation interface consists of the following core modulesaspects:
; Interface engine: Consists of a [[parser]] and a test runner. The parser is present to parse the object files coming from the object repository into the test specific scripting language. The test runner executes the test scripts using a [[test harness]].<ref name="Interface" />
* Interface Engine
* Interface Environment
* Object Repository
====Interface engine====
Interface engines are built on top of Interface Environment. Interface engine consists of a [[parser]] and a test runner. The parser is present to parse the object files coming from the object repository into the test specific scripting language. The test runner executes the test scripts using a [[test harness]].<ref name="Interface" />
====Object repository====
Object repositories are a collection of UI/Application object data recorded by the testing tool while exploring the application under test.<ref name="Interface" />
==Defining boundaries between automation framework and a testing tool==
Tools are specifically designed to target some particular test environment, such as Windows and web automation tools, etc. Tools serve as a driving agent for an automation process. However, an automation framework is not a tool to perform a specific task, but rather infrastructure that provides the solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer.
There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are:
#[[Data-driven testing]]
#[[Modularity-driven testing]]
#[[Keyword-driven testing]]
#[[Hybrid testing]]
#[[Model-based testing]]
# Code-driven testing
#[[Behavior driven development]]
===Data-driven testing===
{{Excerpt|Data-driven testing|only=paragraph}}
===Modularity-driven testing===
{{Excerpt|Modularity-driven testing|only=paragraph}}
===Keyword-driven testing===
{{Excerpt|Keyword-driven testing|only=paragraphs}}
===Hybrid testing===
{{Excerpt|Hybrid testing|only=paragraphs}}
===Model-based testing===
{{Excerpt|Model-based testing|paragraph=1}}
; Object repository: Collection of UI/Application object data recorded by the testing tool while exploring the SUT.<ref name="Interface" />
===Behavior driven development===
{{Excerpt|Behavior driven development|only=paragraph}}
== See also ==
* {{Annotated link |List of web testing tools}}
* {{Annotated link |Fuzzing}}
* {{Annotated link |Headless browser}}
* {{Annotated link |Software testing}}
* {{Annotated link |System testing}}
* {{Annotated link |Unit test}}
==References==
| 