Content deleted Content added
Added {{Unreliable sources}} tag to article (TW) |
Citation bot (talk | contribs) Removed URL that duplicated identifier. Removed parameters. | Use this bot. Report bugs. | Suggested by Headbomb | Linked from Wikipedia:WikiProject_Academic_Journals/Journals_cited_by_Wikipedia/Sandbox | #UCB_webform_linked 509/967 |
||
| (21 intermediate revisions by 14 users not shown) | |||
Line 1:
{{Short description|Type of software bug}}
A '''software regression''' is a type of [[software bug]]
* ''Local'' – a change introduces a new bug in the changed module or component.▼
* ''Remote'' – a change in one part of the software breaks functionality in another module or component.▼
* ''Unmasked'' – a change unmasks an already existing bug that had no effect before the change.▼
Regressions are often caused by [[Hotfix|encompassed bug fixes]] included in [[software patch]]es. One approach to avoiding this kind of problem is [[regression testing]]. A properly designed [[test plan]] aims at preventing this possibility before releasing any software.<ref>{{cite book |last=Richardson |first=Jared |author2=Gwaltney, William Jr |title=Ship It! A Practical Guide to Successful Software Projects |url=https://archive.org/details/shipitpracticalg0000rich/page/32 |year=2006 |publisher=The Pragmatic Bookshelf |___location=Raleigh, NC |pages=[https://archive.org/details/shipitpracticalg0000rich/page/32 32, 193] |isbn=978-0-9745140-4-8 }}</ref> [[Automated testing]] and well-written [[Test case (software)|test case]]s can reduce the likelihood of a regression.
==Prevention and detection==
Techniques have been proposed that try to prevent regressions from being introduced into software at various stages of development, as outlined below.
▲* Local – a change introduces a new bug in the changed module or component.
▲* Remote – a change in one part of the software breaks functionality in another module or component.
===Prior to release===
▲* Unmasked – a change unmasks an already existing bug that had no effect before the change.
{{main|Regression testing}}
In order to avoid regressions being seen by the [[end-user]] after release, developers regularly run [[regression tests]] after changes are introduced to the software. These tests can include [[unit tests]] to catch local regressions as well as [[integration tests]] to catch remote regressions.<ref>{{cite book |last1=Leung |first1=Hareton K.N. |last2=White |first2=Lee |title=Proceedings of the International Conference on Software Maintenance |date=November 1990 |publisher=IEEE |___location=San Diego, CA, USA |isbn=0-8186-2091-9 |chapter=A study of integration testing and software regression at the integration level|doi=10.1109/ICSM.1990.131377 |s2cid=62583582 }}</ref> Regression testing techniques often leverage existing test cases to minimize the effort involved in creating them.<ref>{{cite journal |last1=Rothermel |first1=Gregg |last2=Harrold |first2=Mary Jean |last3=Dedhia |first3=Jeinay |title=Regression test selection for C++ software |journal=Software Testing, Verification and Reliability |date=2000 |volume=10 |issue=2 |pages=77–109 |doi=10.1002/1099-1689(200006)10:2<77::AID-STVR197>3.0.CO;2-E |url=https://onlinelibrary.wiley.com/doi/abs/10.1002/1099-1689(200006)10:2%3C77::AID-STVR197%3E3.0.CO;2-E |language=en |issn=1099-1689|url-access=subscription }}</ref> However, due to the volume of these existing tests, it is often necessary to select a representative subset, using techniques such as [[Regression_testing#Test_case_prioritization|test-case prioritization]].
For detecting performance regressions, [[software performance testing|software performance tests]] are run on a regular basis, to monitor the response time and resource usage metrics of the software after subsequent changes.<ref>{{cite journal |last1=Weyuker |first1=E.J. |last2=Vokolos |first2=F.I. |title=Experience with performance testing of software systems: issues, an approach, and case study |journal=IEEE Transactions on Software Engineering |date=December 2000 |volume=26 |issue=12 |pages=1147–1156 |doi=10.1109/32.888628 |issn=1939-3520}}</ref> Unlike functional regression tests, the results of performance tests are subject to [[variance]] - that is, results can differ between tests due to variance in performance measurements; as a result, a decision must be made on whether a change in performance numbers constitutes a regression, based on experience and end-user demands. Approaches such as [[statistical significance test|statistical significance testing]] and [[change point detection]] are sometimes used to aid in this decision.<ref>{{cite book |last1=Daly |first1=David |last2=Brown |first2=William |last3=Ingo |first3=Henrik |last4=O'Leary |first4=Jim |last5=Bradford |first5=David |title=Proceedings of the International Conference on Performance Engineering |date=20 April 2020 |publisher=Association for Computing Machinery |isbn=978-1-4503-6991-6 |pages=67–75 |url=https://dl.acm.org/doi/abs/10.1145/3358960.3375791 |chapter=The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System|doi=10.1145/3358960.3375791 |s2cid=211677818 }}</ref>
===Prior to commit===
Since [[debugging]] and localizing the root cause of a software regression can be expensive,<ref>{{cite book |last1=Nistor |first1=Adrian |last2=Jiang |first2=Tian |last3=Tan |first3=Lin |title=Proceedings of the Working Conference on Mining Software Repositories (MSR) |date=May 2013 |pages=237–246 |chapter=Discovering, reporting, and fixing performance bugs|doi=10.1109/MSR.2013.6624035 |isbn=978-1-4673-2936-1 |s2cid=12773088 }}</ref><ref>{{cite journal |last1=Agarwal |first1=Pragya |last2=Agrawal |first2=Arun Prakash |title=Fault-localization techniques for software systems: a literature review |journal=ACM SIGSOFT Software Engineering Notes |date=17 September 2014 |volume=39 |issue=5 |pages=1–8 |doi=10.1145/2659118.2659125 |s2cid=12101263 |url=https://dl.acm.org/doi/abs/10.1145/2659118.2659125 |issn=0163-5948|url-access=subscription }}</ref> there also exist some methods that try to prevent regressions from being committed into the [[code repository]] in the first place. For example, [[Git]] Hooks enable developers to run test scripts before code changes are committed or pushed to the code repository.<ref>{{cite web |title=Git - Git Hooks |url=https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks |website=git-scm.com |access-date=7 November 2021}}</ref> In addition, [[change impact analysis]] has been applied to software to predict the impact of a code change on various components of the program, and to supplement test case selection and prioritization.<ref>{{cite journal |last1=Orso |first1=Alessandro |last2=Apiwattanapong |first2=Taweesup |last3=Harrold |first3=Mary Jean |title=Leveraging field data for impact analysis and regression testing |journal=ACM SIGSOFT Software Engineering Notes |date=1 September 2003 |volume=28 |issue=5 |pages=128–137 |doi=10.1145/949952.940089 |url=https://dl.acm.org/doi/abs/10.1145/949952.940089 |issn=0163-5948|url-access=subscription }}</ref><ref>{{cite book |last1=Qu |first1=Xiao |last2=Acharya |first2=Mithun |last3=Robinson |first3=Brian |title=Proceedings of the International Conference on Software Maintenance |date=September 2012 |pages=129–138 |chapter=Configuration selection using code change impact analysis for regression testing|doi=10.1109/ICSM.2012.6405263 |isbn=978-1-4673-2312-3 |s2cid=14928793 }}</ref> [[Lint (software)|Software linters]] are also often added to commit hooks to ensure consistent coding style, thereby minimizing stylistic issues that can make the software prone to regressions.<ref>{{cite book |last1=Tómasdóttir |first1=Kristín Fjóla |last2=Aniche |first2=Mauricio |last3=van Deursen |first3=Arie |title=Proceedings of the International Conference on Automated Software Engineering |date=October 2017 |pages=578–589 |chapter=Why and how JavaScript developers use linters|doi=10.1109/ASE.2017.8115668 |isbn=978-1-5386-2684-9 |s2cid=215750004 }}</ref>
==Localization==
Many of the techniques used to find the root cause of non-regression software bugs can also be used to debug software regressions, including [[breakpoint|breakpoint debugging]], print debugging, and [[program slicing]]. The techniques described below are often used specifically to debug software regressions.
===Functional regressions===
A common technique used to localize functional regressions is [[Bisection (software engineering)|bisection]], which takes both a buggy commit and a previously working commit as input, and tries to find the root cause by doing a binary search on the commits in between.<ref>{{cite book |last1=Gross |first1=Thomas |title=Proceedings of the International Workshop on Automatic Debugging |date=10 September 1997 |publisher=Linkøping University Electronic Press |pages=185–191 |url=https://ep.liu.se/en/conference-article.aspx?series=ecp&issue=1&Article_No=15 |language=en |chapter=Bisection Debugging}}</ref> [[Version control]] systems such as Git and [[Mercurial]] provide built-in ways to perform bisection on a given pair of commits.<ref>{{cite web |title=Git - git-bisect Documentation |url=https://git-scm.com/docs/git-bisect |website=git-scm.com |access-date=7 November 2021}}</ref><ref>{{cite web |title=hg - bisect |url=https://www.selenic.com/mercurial/hg.1.html |website=www.selenic.com |publisher=Mercurial |access-date=7 November 2021}}</ref>
Other options include directly associating the result of a regression test with code changes;<ref>{{cite web |title=Reading 11: Debugging |url=https://web.mit.edu/6.005/www/fa15/classes/11-debugging/ |website=web.mit.edu |publisher=MIT}}</ref> setting divergence breakpoints;<ref>{{cite book |last1=Buhse |first1=Ben |last2=Wei |first2=Thomas |last3=Zang |first3=Zhiqiang |last4=Milicevic |first4=Aleksandar |last5=Gligoric |first5=Milos |title=Proceedings of the International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) |date=May 2019 |pages=15–18 |chapter=VeDebug: Regression Debugging Tool for Java|doi=10.1109/ICSE-Companion.2019.00027 |isbn=978-1-7281-1764-5 |s2cid=174799830 }}</ref> or using incremental [[data-flow analysis]], which identifies test cases - including failing ones - that are relevant to a set of code changes,<ref>{{cite book |last1=Taha |first1=A.-B. |last2=Thebaut |first2=S.M. |last3=Liu |first3=S.-S. |title=Proceedings of the Annual International Computer Software & Applications Conference |date=September 1989 |publisher=IEEE |pages=527–534 |chapter=An approach to software fault localization and revalidation based on incremental data flow analysis|doi=10.1109/CMPSAC.1989.65142 |isbn=0-8186-1964-3 |s2cid=41978046 }}</ref> among others.
===Performance regressions===
[[Profiling (computer programming)|Profiling]] measures the performance and resource usage of various components of a program, and is used to generate data useful in debugging performance issues. In the context of software performance regressions, developers often compare the [[call stack|call trees]] (also known as "timelines") generated by profilers for both the buggy version and the previously working version, and mechanisms exist to simplify this comparison.<ref>{{cite journal |last1=Ocariza |first1=Frolin S. |last2=Zhao |first2=Boyang |title=Localizing software performance regressions in web applications by comparing execution timelines |journal=Software Testing, Verification and Reliability |date=2021 |volume=31 |issue=5 |pages=e1750 |doi=10.1002/stvr.1750 |s2cid=225416138 |url=https://onlinelibrary.wiley.com/doi/abs/10.1002/stvr.1750 |language=en |issn=1099-1689|url-access=subscription }}</ref> [[Web development tools]] typically provide developers the ability to record these performance profiles.<ref>{{cite web |title=Analyze runtime performance |url=https://developer.chrome.com/docs/devtools/evaluate-performance/ |website=Chrome Developers |publisher=Google |access-date=7 November 2021 |language=en}}</ref><ref>{{cite web |title=Performance analysis reference - Microsoft Edge Development |url=https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/evaluate-performance/reference |website=docs.microsoft.com |publisher=Microsoft |access-date=7 November 2021 |language=en-us}}</ref>
Logging also helps with performance regression localization, and similar to call trees, developers can compare systematically-placed performance logs of multiple versions of the same software.<ref>{{cite book |last1=Yao |first1=Kundi |last2=B. de Pádua |first2=Guilherme |last3=Shang |first3=Weiyi |last4=Sporea |first4=Steve |last5=Toma |first5=Andrei |last6=Sajedi |first6=Sarah |title=Proceedings of the International Conference on Performance Engineering |date=30 March 2018 |publisher=Association for Computing Machinery |isbn=978-1-4503-5095-2 |pages=127–138 |url=https://dl.acm.org/doi/abs/10.1145/3184407.3184416 |chapter=Log4Perf: Suggesting Logging Locations for Web-based Systems' Performance Monitoring|doi=10.1145/3184407.3184416 |s2cid=4557038 }}</ref> A tradeoff exists when adding these performance logs, as adding many logs can help developers pinpoint which portions of the software are regressing at smaller granularities, while adding only a few logs will also reduce overhead when executing the program.<ref>{{cite journal |title=A Qualitative Study of the Benefits and Costs of Logging from Developers' Perspectives |journal=IEEE Transactions on Software Engineering |date=30 January 2020 |doi=10.1109/TSE.2020.2970422 |last1=Li |first1=Heng |last2=Shang |first2=Weiyi |last3=Adams |first3=Bram |last4=Sayagh |first4=Mohammed |last5=Hassan |first5=Ahmed E. |volume=47 |issue=12 |pages=2858–2873 |s2cid=213679706 }}</ref>
Additional approaches include writing performance-aware unit tests to help with localization,<ref>{{cite book |last1=Heger |first1=Christoph |last2=Happe |first2=Jens |last3=Farahbod |first3=Roozbeh |title=Proceedings of the International Conference on Performance Engineering |date=21 April 2013 |publisher=Association for Computing Machinery |isbn=978-1-4503-1636-1 |pages=27–38 |url=https://dl.acm.org/doi/abs/10.1145/2479871.2479879 |chapter=Automated root cause isolation of performance regressions during software development|doi=10.1145/2479871.2479879 |s2cid=2593603 }}</ref> and ranking subsystems based on performance counter deviations.<ref>{{cite book |last1=Malik |first1=Haroon |last2=Adams |first2=Bram |last3=Hassan |first3=Ahmed E. |title=Proceedings of the International Symposium on Software Reliability Engineering |date=November 2010 |pages=201–210 |chapter=Pinpointing the Subsystems Responsible for the Performance Deviations in a Load Test|doi=10.1109/ISSRE.2010.43 |isbn=978-1-4244-9056-1 |s2cid=17306870 }}</ref> Bisection can also be repurposed for performance regressions by considering commits that perform below (or above) a certain baseline value as buggy, and taking either the left or the right side of the commits based on the results of this comparison.
==See also==
* [[Software rot]]
* [[Software aging]]
==References==
Line 19 ⟶ 51:
{{DEFAULTSORT:Software Regression}}
[[Category:Software bugs]]
| |||