Software development effort estimation: Difference between revisions

Revision as of 16:37, 3 August 2008

Latest revision as of 10:52, 27 August 2025

(296 intermediate revisions by more than 100 users not shown)

Line 1:

{{Short description|Process in software development}}

[[Software development]] effort [[estimation]] is the process of predicting the most realistic use of effort required to develop or maintain [[software]] based on incomplete, uncertain and/or noisy input. Effort estimates may be used as input to project plans, iteration plans, budgets, investment analyses, pricing processes and bidding rounds.

In [[software development]], '''effort estimation''' is the process of predicting the most realistic amount of effort (expressed in terms of person-hours or money) required to develop or maintain [[software]] based on incomplete, uncertain and noisy input. Effort [[estimation|estimates]] may be used as input to project plans, iteration plans, budgets, investment analyses, pricing processes and bidding rounds.<ref>{{cite web | url=http://www.infoq.com/articles/software-development-effort-estimation | title=What We do and Don't Know about Software Development Effort Estimation}}</ref><ref>{{cite web|title=Cost Estimating And Assessment Guide GAO-09-3SP Best Practices for developing and managing Capital Program Costs|date=2009|publisher=US Government Accountability Office|url=https://www.gao.gov/new.items/d093sp.pdf }}</ref>

== State-of-practice ==

Published surveys on estimation practice suggest that expert estimation is the dominant strategy when estimating software development effort.<ref>{{cite journal

Published surveys on estimation practice suggest that expert estimation is the dominant strategy when estimating software development effort<ref>{{cite web

| author = Jørgensen, M.

| title = A Review of Studies on Expert Estimation of Software Development Effort

| journal = Journal of Systems and Software

| url = http://simula.no/research/engineering/publications/SE.4.Joergensen.2004.c

| volume = 70

}}</ref>.

| issue = 1–2

| pages = 37–60

Typically, effort estimates are over-optimistic and there is a strong over-confidence in their accuracy. The mean effort overrun seems to be about 30% and not decreasing over time. For a review of effort estimation error surveys, see <ref>{{cite web

| url = http://simula.no/publications/review-studies-expert-estimation-software-development-effort

| author = Molokken, K. Jorgensen, M.

| year = 2004

| title = A review of software surveys on software effort estimation

| doi = 10.1016/S0164-1212(02)00156-5

| url = http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1237981

}}</ref>

}}</ref>. However, the measurement of estimation error is not unproblematic, see [[#Assessing and interpreting the_accuracy of effort estimates|Assessing and interpreting the_accuracy of effort estimates]].

The strong over-confidence in the accuracy of the effort estimates is illustrated by the finding that, on average, if a software professional is 90% confident or “almost sure” to include the actual effort in a minimum-maximum interval, the observed frequency of including the actual effort is only 60-70% <ref>{{cite web

| author = Jørgensen, M. Teigen, K.H. Ribu, K.

| title = Better sure than safe? Over-confidence in judgement based software development effort prediction intervals

| url = http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V0N-49N06GS-5&_user=674998&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000036598&_version=1&_urlVersion=0&_userid=674998&md5=36c6383445cf481447d06cb30c1ccb63 }}</ref>.

Currently the term “effort estimate” is used to denote as different concepts as most likely use of effort (modal value), the effort that corresponds to a probability of 50% of not exceeding (median), the planned effort, the budgeted effort or the effort used to propose a bid or price to the client. This is believed to be unfortunate, because communication problems may occur and because the concepts serve different goals

<ref>Edwards, J.S. Moores, T.T. (1994), "A conflict between the use of estimating and planning tools in the management of information systems.". European Journal of Information Systems 3(2): 139-147.</ref>

<ref>Goodwin, P. (1998). Enhancing judgmental sales forecasting: The role of laboratory research. Forecasting with judgment. G. Wright and P. Goodwin. New York, John Wiley & Sons: 91-112.</ref>.

== History ==

Typically, effort estimates are over-optimistic and there is a strong over-confidence in their accuracy. The mean effort overrun seems to be about 30% and not decreasing over time. For a review of effort estimation error surveys, see.<ref>{{cite book

Software researchers and practitioners have been addressing the problems of effort estimation for software development projects since at least the 1960s; see, e.g., work by Farr <ref>{{cite web

| author = FarrMolokken, LK. Nanus Jorgensen, BM.

| title = 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings

| title = Factors that affect the cost of computer programming

| pages = 223–230

| url = http://stinet.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=AD0603707

| doi = 10.1109/ISESE.2003.1237981

}}</ref> and Nelson <ref>Nelson, E. A. (1966). Management Handbook for the Estimation of Computer Programming Costs. AD-A648750, Systems Development Corp.</ref>.

| year = 2003

| isbn = 978-0-7695-2002-5

| chapter = A review of software surveys on software effort estimation

| s2cid = 15471986

}}</ref> However, the measurement of estimation error is problematic, see [[#Assessing the accuracy of estimates|Assessing the accuracy of estimates]].

The strong overconfidence in the accuracy of the effort estimates is illustrated by the finding that, on average, if a software professional is 90% confident or "almost sure" to include the actual effort in a minimum-maximum interval, the observed frequency of including the actual effort is only 60-70%.<ref>{{cite journal

| author = Jørgensen, M. Teigen, K.H. Ribu, K.

| title = Better sure than safe? Over-confidence in judgement based software development effort prediction intervals

| doi=10.1016/S0164-1212(02)00160-7

| volume=70

| issue = 1–2

| year=2004

| journal=Journal of Systems and Software

| pages=79–93}}</ref>

Currently the term "effort estimate" is used to denote as different concepts such as most likely use of effort (modal value), the effort that corresponds to a probability of 50% of not exceeding (median), the planned effort, the budgeted effort or the effort used to propose a bid or price to the client. This is believed to be unfortunate, because communication problems may occur and because the concepts serve different goals.<ref>{{cite journal | last1 = Edwards | first1 = J.S. Moores | year = 1994 | title = A conflict between the use of estimating and planning tools in the management of information systems | journal = [[European Journal of Information Systems]] | volume = 3 | issue = 2| pages = 139–147 | doi=10.1057/ejis.1994.14| s2cid = 62582672 }}</ref><ref>Goodwin, P. (1998). Enhancing judgmental sales forecasting: The role of laboratory research. Forecasting with judgment. G. Wright and P. Goodwin. New York, John Wiley & Sons: 91-112. Hi</ref>

Most of the research has focused on the construction of formal software effort estimation models. The early models were typically based on [[regression analysis]] or mathematically derived from theories from other domains. Since then a high number of model building approaches have been evaluated, such as approaches founded on [[case-based reasoning]], classification and [[regression trees]], [[simulation]], [[neural networks]], [[Bayesian statistics]], [[lexical analysis]] of requirement specifications, [[genetic programming]], [[linear programming]], economic production models, [[soft computing]], [[fuzzy logic]] modeling, statistical [[bootstrapping]], and combinations of one or more of these models. The perhaps most common estimation products today, e.g., the formal estimation models [[COCOMO]] and SLIM have their basis in estimation research conducted in the 1970s and 1980s. The estimation approaches based on functionality-based size measures, e.g., [[function points]], is also based on research conducted in the 1970s and 1980s, but are re-appearing with modified size measures under different labels, such as “use case points” <ref>{{cite web

| author = Anda, B. Angelvik, E. Ribu, K.

| title = Improving Estimation Practices by Applying Use Case Models

| url = http://www.springerlink.com/content/7lpyel912m5cr654/

}}</ref> in the 1990s and 2000s.

==History==

== Estimation approaches ==

Software researchers and practitioners have been addressing the problems of effort estimation for software development projects since at least the 1960s; see, e.g., work by Farr<ref>{{cite web

| author = Farr, L. Nanus, B.

| title = Factors that affect the cost of computer programming, volume I

| url = http://www.dtic.mil/dtic/tr/fulltext/u2/603707.pdf

| archive-url = https://web.archive.org/web/20170221211529/http://www.dtic.mil/dtic/tr/fulltext/u2/603707.pdf

| url-status = dead

| archive-date = February 21, 2017

}}</ref><ref>{{cite web

| author = Farr, L. Nanus, B.

| title = Factors that affect the cost of computer programming, volume II

| url = http://www.dtic.mil/dtic/tr/fulltext/u2/607546.pdf

| archive-url = https://web.archive.org/web/20180728134658/http://www.dtic.mil/dtic/tr/fulltext/u2/607546.pdf

| url-status = dead

| archive-date = July 28, 2018

}}</ref> and Nelson.<ref>Nelson, E. A. (1966). Management Handbook for the Estimation of Computer Programming Costs. AD-A648750, Systems Development Corp.</ref>

Most of the research has focused on the construction of formal software effort estimation models. The early models were typically based on [[regression analysis]] or mathematically derived from theories from other domains. Since then a high number of model building approaches have been evaluated, such as approaches founded on [[case-based reasoning]], classification and [[regression trees]], [[simulation]], [[neural networks]], [[Bayesian statistics]], [[lexical analysis]] of requirement specifications, [[genetic programming]], [[linear programming]], economic production models, [[soft computing]], [[fuzzy logic]] modeling, statistical [[bootstrapping]], and combinations of two or more of these models. The perhaps most common estimation methods today are the parametric estimation models [[COCOMO]], [[SEER-SEM]] and SLIM. They have their basis in estimation research conducted in the 1970s and 1980s and are since then updated with new calibration data, with the last major release being COCOMO II in the year 2000. The estimation approaches based on functionality-based size measures, e.g., [[function points]], is also based on research conducted in the 1970s and 1980s, but are re-calibrated with modified size measures and different counting approaches, such as the [[Use Case Points|use case points]]<ref>{{cite book

There are many ways of categorizing estimation approaches, see for example <ref>Briand, L. C. and I. Wieczorek (2002). Resource estimation in software engineering. Encyclopedia of software engineering. J. J. Marcinak. New York, John Wiley & Sons: 1160-1196.</ref><ref>{{cite web

| author = JørgensenAnda, MB. Shepperd Angelvik, E. Ribu, MK.

| title = Product A Systematic Review ofFocused Software Development Cost EstimationProcess StudiesImprovement

| chapter = Improving Estimation Practices by Applying Use Case Models

| url = http://simula.no/research/engineering/publications/Jorgensen.2007.1 }}</ref>. The top level categories are the following:

| series = Lecture Notes in Computer Science

| doi=10.1007/3-540-36209-6_32

| year=2002

| volume = 2559

| pages=383–397

| isbn = 978-3-540-00234-5

| citeseerx = 10.1.1.546.112

}} {{isbn|9783540002345|9783540362098}}.</ref> or [[object point]]s and [[COSMIC_functional_size_measurement|COSMIC Function Points]] in the 1990s.

==Estimation approaches==

* Expert estimation: The quantification step, i.e., the step where the estimate is produced, is based on judgmental processes.

There are many ways of categorizing estimation approaches, see for example.<ref>Briand, L. C. and Wieczorek, I. (2002). "Resource estimation in software engineering". ''Encyclopedia of software engineering''. J. J. Marcinak. New York, John Wiley & Sons: 1160–1196.</ref><ref>{{cite web

| author = Jørgensen, M. Shepperd, M.

| title = A Systematic Review of Software Development Cost Estimation Studies

| url = http://simula.no/research/engineering/publications/Jorgensen.2007.1 }}</ref> The top level categories are the following:

* Expert estimation: The quantification step, i.e., the step where the estimate is produced based on judgmental processes.<ref>{{cite web | url=http://www.oxagile.com/services/custom-software-design-and-development/ | title=Custom Software Development Services – Custom App Development – Oxagile}}</ref>

* Formal estimation model: The quantification step is based on mechanical processes, e.g., the use of a formula derived from historical data.

* Combination-based estimation: The quantification step is based on a judgmental orand mechanical combination of estimates from different sources.

Below are examples of estimation approaches within each category.

Line 58 ⟶ 86:

| [[Analogy]]-based estimation

| Formal estimation model

| ANGEL, [[Weighted Micro Function Points]]

| ANGEL

|-

| [[Work_breakdown_structureWork breakdown structure|WBS-based]] (bottom up) estimation

| Expert estimation

| [[MS Project management software]], company specific activity templates

|-

| Parametric models

| Formal estimation model

| [[COCOMO]], [[Putnam model|SLIM]], [[SEER-SEM]], [[TruePlanning for Software]]

|-

| Size-based estimation models<ref>Hill Peter (ISBSG) – Estimation Workbook 2 – published by International Software Benchmarking Standards Group [http://www.isbsg.org/ISBSGnew.nsf/WebPages/~GBL~Practical%20Project%20Estimation%202nd%20Edition ISBSG - Estimation and Benchmarking Resource Centre] {{Webarchive|url=https://web.archive.org/web/20080829172340/https://www.isbsg.org/isbsgnew.nsf/WebPages/~GBL~Practical%20Project%20Estimation%202nd%20Edition |date=2008-08-29 }}</ref>

| Size-based -based estimation models

| Formal estimation model

| [[Function Point Analysis]],<ref>Morris Pam&nbsp;— Overview of Function Point Analysis, [http://www.totalmetrics.com/function-point-resources/what-are-function-points Total Metrics - Function Point Resource Centre]</ref> [[Use Case]] Analysis, [[Use Case Points]], SSU (Software Size Unit), [[Story pointspoint]]s-based estimation in [[Agile software development]], [[Object point|Object Points]]

|-

| Group estimation

| Expert estimation

| [[Planning poker]], [[Wideband Delphidelphi]]

|-

| Mechanical combination

| Combination-based estimation

| Average of an analogy-based and a [[Work breakdown structure]]-based effort estimate<ref>Srinivasa Gopal and Meenakshi D'Souza. 2012. Improving estimation accuracy by using case based reasoning and a combined estimation approach. In ''Proceedings of the 5th India Software Engineering Conference'' (ISEC '12). ACM, New York, USA, 75-78. {{doi|10.1145/2134254.2134267}}</ref>

|-

| Judgmental combination

Line 85 ⟶ 113:

|}

== Selection of estimation approach approaches==

The evidence on differences in estimation accuracy of different estimation approaches and models suggest that there is no "best approach" and that the relative accuracy of one approach or model in comparison to another depends strongly on the context

.<ref>{{cite journal

| author = Shepperd, M. Kadoda, G.

| title = Comparing software prediction techniques using simulation

| journal = IEEE Transactions on Software Engineering

| volume = 27

| issue = 11

| pages = 1014–1022

| doi = 10.1109/32.965341

| year = 2001

| bibcode = 2001ITSEn..27.1014S

| url = http://bura.brunel.ac.uk/handle/2438/1102

}}

</ref> This implies that different organizations benefit from different estimation approaches. Findings<ref name="Jørgensen, M">{{cite web

| author = Jørgensen, M.

| title = Estimation of Software Development Work Effort:Evidence on Expert Judgment and Formal Models

| url = http://simula.no/research/engineering/publications/Jorgensen.2007.2 }}</ref> that may support the selection of estimation approach based on the expected accuracy of an approach include:

* Expert estimation is on average at least as accurate as model-based effort estimation. In particular, situations with unstable relationships and information of high importance not included in the model may suggest use of expert estimation. This assumes, of course, that experts with relevant experience are available.

* Formal estimation models not tailored to a particular organization's own context, may be very inaccurate. Use of own historical data is consequently crucial if one cannot be sure that the estimation model's core relationships (e.g., formula parameters) are based on similar project contexts.

* Formal estimation models may be particularly useful in situations where the model is tailored to the organization's context (either through use of own historical data or that the model is derived from similar projects and contexts), and it is likely that the experts’ estimates will be subject to a strong degree of wishful thinking.

The most robust finding, in many forecasting domains, is that combination of estimates from independent sources, preferable applying different approaches, will on average improve the estimation accuracy.<ref name="Jørgensen, M"/><ref>{{cite journal

The evidence on differences in estimation accuracy of different estimation approaches and models suggest that there is no “best approach” and that the relative accuracy of one approach or model in comparison to another depends strongly on the context

| author = Winkler, R.L.

<ref>{{cite web

| title = Combining forecasts: A philosophical basis and some current issues Manager

| author = Shepperd, M. Kadoda, G.

| doi=10.1016/0169-2070(89)90018-6

| title = Comparing software prediction techniques using simulation

| volume=5

| url = http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/32/20846/00965341.pdf?arnumber=965341 }}

| issue = 4

</ref>. This implies that different organizations benefit from different estimation approaches. Findings, summarized in <ref>{{cite web

| year=1989

| author = Jørgensen, M.

| journal=International Journal of Forecasting

| title = Estimation of Software Development Work Effort:Evidence on Expert Judgment and Formal Models

| pages=605–609}}

| url = http://simula.no/research/engineering/publications/Jorgensen.2007.2 }}

</ref><ref>{{cite journal

</ref>, that may support the selection of estimation approach based on the expected accuracy of an approach include:

| author = Blattberg, R.C. Hoch, S.J.

| title = Database Models and Managerial Intuition: 50% Model + 50% Manager

| jstor = 2632364

| volume=36

| issue = 8

| pages=887–899

| doi=10.1287/mnsc.36.8.887

| journal=Management Science

| year=1990}}

</ref>

It is important to be aware of the limitations of each traditional approach to measuring software development productivity.<ref>{{cite web

* Expert estimation is on average at least as accurate as model-based effort estimation. In particular, situations with unstable relationships and information of high importance not included in the model may suggest use of expert estimation. This assumes, of course, that experts with relevant experience are available.

| author = BlueOptima

| title = Identifying Reliable, Objective Software Development Metrics

| url = https://www.blueoptima.com/blog/identifying-reliable-objective-software-development-metrics | date = 2019-10-29

}}

</ref>

In addition, other factors such as ease of understanding and communicating the results of an approach, ease of use of an approach, and cost of introduction of an approach should be considered in a selection process.

* Formal estimation models not tailored to a particular organization’s own context, may be very inaccurate. Use of own historical data is consequently crucial if one cannot be sure that the estimation model’s core relationships (e.g., formula parameters) are based on similar project contexts.

==Assessing the accuracy of estimates==

* Formal estimation models may be particularly useful in situations where the model is tailored to the organization’s context (either through use of own historical data or that the model is derived from similar projects and contexts), and/or it is likely that the experts’ estimates will be subject to a strong degree of [[wishful thinking]].

The most common measure of the average estimation accuracy is the MMRE (Mean Magnitude of Relative Error), where the MRE of each estimate is defined as:

: {{math|1=''MRE'' = {{sfrac|{{mabs|(actual effort) - (estimated effort)}}|(actual effort)}}}}

The most robust finding, in many forecasting domains, is that combination of estimates from independent sources, preferable applying different approaches, will on average improve the estimation accuracy <ref>{{cite web

| author = Winkler, R.L.

This measure has been criticized <ref>{{cite journal

| title = Combining forecasts: A philosophical basis and some current issues Manager

| author = Shepperd, M. Cartwright, M. Kadoda, G.

| url = http://www.sciencedirect.com/science/article/B6V92-45P4G7H-2B/2/d05dc6c369ab173c5792a05ea1be21d9 }}

| title = On Building Prediction Systems for Software Engineers

| doi=10.1023/A:1026582314146

| volume=5

| issue = 3

| year=2000

| journal=Empirical Software Engineering

| pages=175–182| s2cid = 1293988

}}

</ref>

<ref>{{cite web

| author = Blattberg[[Barbara Kitchenham|Kitchenham, RB.C]], Pickard, L.M. Hoch, MacDonell, S.JG. Shepperd

| title = What accuracy statistics really measure

| title = Database Models and Managerial Intuition: 50% Model + 50% Manager

| url = http://scitation.aip.org/getabs/servlet/GetabsServlet?prog=normal&id=IPSEFU000148000003000081000001&idtype=cvips&gifs=yes }}

| url = http://www.jstor.org/pss/2632364 }}

</ref>

<ref>{{cite web journal

| author = Foss, T., Stensrud, E., [[Barbara Kitchenham|Kitchenham, B.]], Myrtveit, I.

| author = Jørgensen, M.

| title = A Simulation Study of the Model Evaluation Criterion MMRE

| title = Estimation of Software Development Work Effort:Evidence on Expert Judgment and Formal Models

| journal = IEEE Transactions on Software Engineering

| url = http://simula.no/research/engineering/publications/Jorgensen.2007.2 }}

| volume = 29

</ref>.

| issue = 11

| pages = 985–995

| url = http://portal.acm.org/citation.cfm?id=951936 | doi = 10.1109/TSE.2003.1245300

| year = 2003

| bibcode = 2003ITSEn..29..985F

| citeseerx = 10.1.1.101.5792

}}

</ref> and there are several alternative measures, such as more symmetric measures,<ref>{{cite journal

| author = Miyazaki, Y. Terakado, M. Ozaki, K. Nozaki, H.

| title = Robust regression for developing software estimation models

| journal = Journal of Systems and Software

| volume = 27

| pages = 3–16

| url = http://portal.acm.org/citation.cfm?id=198684 | doi = 10.1016/0164-1212(94)90110-4

| year = 1994

}}

</ref> Weighted Mean of Quartiles of relative errors (WMQ)

<ref>{{cite web

| author = Lo, B. Gao, X.

| title = Assessing Software Cost Estimation Models: criteria for accuracy, consistency and regression

| url = http://dl.acs.org.au/index.php/ajis/article/view/348 }}</ref> and Mean Variation from Estimate (MVFE).<ref>{{cite journal

| author = Hughes, R.T. Cunliffe, A. Young-Martos, F.

| title = Evaluating software development effort model-building techniquesfor application in a real-time telecommunications environment

| journal = IEE Proceedings - Software

| volume = 145

| page = 29

| url = https://ieeexplore.ieee.org/document/689296 | archive-url = https://web.archive.org/web/20170920055746/http://ieeexplore.ieee.org/document/689296/ | url-status = dead | archive-date = September 20, 2017 | doi = 10.1049/ip-sen:19983370

| year = 1998

| doi-broken-date = 12 July 2025

}}</ref>

MRE is not reliable if the individual items are skewed. PRED(25) is preferred as a measure of estimation accuracy. PRED(25) measures the percentage of predicted values that are within 25 percent of the actual value.

In addition, other factors such as ease of understanding and communicating the results of an approach, ease of use of an approach, cost of introduction of an approach should be considered in a selection process.

A high estimation error cannot automatically be interpreted as an indicator of low estimation ability. Alternative, competing or complementing, reasons include low cost control of project, high complexity of development work, and more delivered functionality than originally estimated. A framework for improved use and interpretation of estimation error measurement is included in.<ref>{{cite web

== Uncertainty assessment approaches ==

| author = Grimstad, S. Jørgensen, M.

| title = A Framework for the Analysis of Software Cost Estimation Accuracy

| url = http://simula.no/publications/framework-analysis-software-cost-estimation-accuracy | year = 2006

}}</ref>

==Psychological issues==

The uncertainty of an effort estimate can be described through a prediction interval (PI). An effort PI is based on a stated certainty level and contains a minimum and a maximum effort value. For example, a project leader may estimate that the most likely effort of a project is 1000 work-hours and that it is 90% certain that the actual effort will be between 500 and 2000 work-hours. Then, the interval [500, 2000] work-hours is the 90% PI of the effort estimate of 1000 work-hours. Frequently, other terms are used instead of PI, e.g., prediction bounds, prediction limits, interval prediction, prediction region and, unfortunately, confidence interval. An important difference between confidence interval and PI is that PI refers to the uncertainty of an estimate, while confidence interval usually refers to the uncertainty associated with the parameters of an estimation model or distribution, e.g., the uncertainty of the mean value of a distribution of effort values. The confidence level of a PI refers to the expected (or subjective) probability that the real value is within the predicted interval<ref>{{cite web

There are many psychological factors potentially explaining the strong tendency towards over-optimistic effort estimates. These factors are essential to consider even when using formal estimation models, because much of the input to these models is judgment-based. Factors that have been demonstrated to be important are [[wishful thinking]], [[Anchoring (cognitive bias)|anchoring]], [[planning fallacy]] and [[cognitive dissonance]].<ref>{{cite journal

| author = Armstrong, J. S.

| author = Jørgensen, M. Grimstad, S.

| title = Principles of forecasting: A handbook for researchers and practitioners

| title = How to Avoid Impact from Irrelevant and Misleading Information When Estimating Software Development Effort

| url = http://www.forecastingprinciples.com }}</ref>.

| journal = IEEE Software

| date = 2008

| pages = 78–83

| url = https://www.simula.no/publications/avoiding-irrelevant-and-misleading-information-when-estimating-development-effort }}

</ref>

* It's easy to estimate what is known.

* It's hard to estimate what is known to be unknown. (known unknowns)

* It's very hard to estimate what is not known to be unknown. (unknown unknowns)

==Humor==

There are several possible approaches to calculate effort PIs, e.g., formal approaches based on regression or bootstrapping <ref>{{cite web

The chronic underestimation of development effort has led to the coinage and popularity of numerous humorous adages, such as ironically referring to a task as a "[[small matter of programming]]" (when much effort is likely required), and citing laws about underestimation:

| author = Angelis, L. Stamelos, I.

* [[Ninety–ninety rule]]:

| title = A simulation tool for efficient analogy based cost estimation

{{blockquote |<!-- PLEASE DON'T CHANGE THIS QUOTE WITHOUT CHECKING THE ORIGINAL SOURCE! THE %'S ARE NOT SUPPOSED TO TOTAL 100%! -->The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90<!-- YES THIS TOTALS 180%! THIS IS NOT AN ERROR! --> percent of the development time.<ref name="Bentley1985">{{cite journal|last= Bentley|first= Jon|year= 1985|title= Programming pearls|journal= Communications of the ACM|volume= 28|issue= 9|pages= 896–901|issn= 0001-0782|doi= 10.1145/4284.315122|s2cid= 5832776|format= fee required|doi-access= free}}</ref> |Tom Cargill|[[Bell Labs]]}}

| url = http://portal.acm.org/citation.cfm?id=594467&dl=ACM&coll=portal }}</ref>, formal or judgmental approaches based on the distribution of previous estimation error

* [[Hofstadter's law]]:

<ref>{{cite web

{{blockquote|Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.|[[Douglas Hofstadter]]| ''[[Gödel, Escher, Bach: An Eternal Golden Braid]]''<ref>

| author = Jørgensen, M. Sjøberg, D.I.K.

''Gödel, Escher, Bach: An Eternal Golden Braid''. 20th anniversary ed., 1999, p. 152. {{ISBN|0-465-02656-7}}.

| title = An effort prediction interval approach based on the empirical distribution of previous estimation accuracy

| url = http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V0B-47HC6S5-1&_user=674998&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000036598&_version=1&_urlVersion=0&_userid=674998&md5=6cb917a379855c79eebe9f18ca9ac424 }}</ref>, and pure expert judgment of minimum-maximum effort for a given level of confidence. Expert judgments based on the distribution of previous estimation error has been found to systematically lead to more realistic uncertainty assessment than the traditional minimum-maximum effort intervals in several studies, see for example <ref>{{cite web

| author = Jørgensen, M.

| title = Realism in assessment of effort estimation uncertainty: It matters how you ask

| url = http://simula.no/research/engineering/publications/SE.4.Joergensen.2004.e }}</ref>.

== Assessing and interpreting the accuracy of effort estimates ==

The most common measures of the average estimation accuracy is the MMRE (Mean Magnitude of Relative Error), where MRE is defined as:

''MRE'' = |actual effort − estimated effort| / |actual effort|

This measure has been criticized <ref>{{cite web

| author = Shepperd, M. Cartwright, M. Kadoda, G.

| title = On Building Prediction Systems for Software Engineers

| url = http://www.ingentaconnect.com/content/klu/emse/2000/00000005/00000003/00278191 }}

</ref>

}}

<ref>{{cite web

* [[Brooks's law|Fred Brooks' law]]:

| author = Kitchenham, B. Pickard, L.M. MacDonell, S.G. Shepperd,

{{blockquote|What one programmer can do in one month, two programmers can do in two months.|[[Fred Brooks]]|{{Citation needed|date=November 2024|reason=Where did this exact quote come from?}}

| title = What accuracy statistics really measure

}}

| url = http://scitation.aip.org/getabs/servlet/GetabsServlet?prog=normal&id=IPSEFU000148000003000081000001&idtype=cvips&gifs=yes }}

</ref>

<ref>{{cite web

| author = Foss, T. Stensrud, E. Kitchenham, B. Myrtveit, I.

| title = A Simulation Study of the Model Evaluation Criterion MMRE

| publisher = [[IEEE]]

| url = http://portal.acm.org/citation.cfm?id=951936 }}

</ref> and there are several alternative measures, such as more symmetric measures <ref>{{cite web

| author = Miyazaki, Y. Terakado, M. Ozaki, K. Nozaki, H.

| title = Robust regression for developing software estimation models

| url = http://portal.acm.org/citation.cfm?id=198684 }}

</ref>

, Weighted Mean of Quartiles of relative errors (WMQ)

<ref>{{cite web

| author = Lo, B. Gao, X.

| title = Assessing Software Cost Estimation Models: criteria for accuracy, consistency and regression

| url = http://dl.acs.org.au/index.php/ajis/article/view/348 }}</ref> and Mean Variation from Estimate (MVFE) <ref>{{cite web

| author = Hughes, R.T. Cunliffe, A. Young-Martos, F.

| title = Evaluating software development effort model-building techniquesfor application in a real-time telecommunications environment

| url = http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=689296 }}</ref>.

==Comparison of development estimation software==

A high estimation error cannot automatically be interpreted as an indicator of low estimation ability. Alternative, competing or complementing, reasons include low cost control of project, high complexity of development work, and more delivered functionality than originally estimated. A framework for improved use and interpretation of estimation error measurement is included in <ref>{{cite web

<!-- PLEASE NOTE - Please only place entries here that describe development estimation software - meaning they can can do at least either Schedule or Cost estimate. If you have questions, use the talk page. Please try to keep entries in alphabetical order. Thanks.

| author = Grimstad, S. Jørgensen, M.

-->

| title = A Framework for the Analysis of Software Cost Estimation Accuracy

{| class="wikitable sortable" style="text-align: center; width:100%;"

| url = http://simula.no/research/engineering/publications/Grimstad.2006.2/simula_pdf_file }}</ref>.

|-

! '''Software'''

! Schedule estimate

! Cost estimate

! Cost Models

! Input

! Report Output Format

! [[Programming language|Supported Programming Languages]]

! [[Computing platform|Platforms]]

! Cost

! [[Software license|License]]

|-

|[[AFCAA REVIC]]<ref>AFCAA Revic 9.2 manual [http://sites.google.com/site/revic92 Revic memorial site]</ref>

| {{yes}}

| {{yes}}

| REVIC

| [[Source lines of code|KLOC]], Scale Factors, Cost Drivers

| proprietary, Text

| {{any}}

| DOS

| {{Free}}

| Proprietary<br/>/ Free for public distribution

|-

|[[SEER-SEM|Seer for Software]]

| {{yes}}

| {{yes}}

| [[SEER-SEM]]

| [[Source lines of code|SLOC]], [[Function points]], use cases, bottoms-up, object, features

| proprietary, Excel, Microsoft Project, IBM Rational, Oracle Crystal Ball

| {{any}}

| Windows, Any ([[Web application|Web-based]])

| Commercial

| {{Proprietary}}

|-

|[[Putnam model|SLIM]]<ref>{{cite web|url=http://www.qsm.com/tools |title=SLIM Suite Overview |publisher=Qsm.com |access-date=2019-08-27}}</ref>

| {{yes}}

| {{yes}}

| [[Putnam model|SLIM]]

| Size ([[Source lines of code|SLOC]], [[Function points]], Use Cases, etc.), constraints (size, duration, effort, staff), scale factors, historical projects, historical trends

| proprietary, Excel, Microsoft Project, Microsoft PowerPoint, IBM Rational, text, HTML

| {{any}}

| Windows, Any ([[Web application|Web-based]])<ref>{{cite web|url=http://www.qsm.com/tools/slim-webservices |title=SLIM-WebServices |publisher=Qsm.com |access-date=2019-08-27}}</ref>

| Commercial

| {{Proprietary}}

|-

|[[PRICE Systems|TruePlanning]]<ref>TruePlanning Integrated Cost Models [http://www.pricesystems.com/en-us/offerings/pricecostmodels.aspx#167514-integrated-models PRICE Systems site] {{Webarchive|url=https://web.archive.org/web/20151105111619/http://www.pricesystems.com/en-us/offerings/pricecostmodels.aspx#167514-integrated-models |date=2015-11-05 }}</ref>

| {{yes}}

| {{yes}}

| PRICE

| Components, Structures, Activities, Cost drivers, Processes, Functional Software Size (Source Lines of Code (SLOC), Function Points, Use Case Conversion Points (UCCP), Predictive Object Points (POPs) etc.)

| Excel, CAD

| {{any}}

| Windows

| Commercial

| {{Proprietary}}

|}

==See also==

== Psychological issues related to effort estimation ==

{{columns-list|colwidth=22em|

* [[Cone of uncertainty]]

There are many psychological factors potentially explaining the strong tendency towards over-optimistic effort estimates that need to be dealt with to increase accuracy of effort estimates. These factors are essential even when using formal estimation models, because much of the input to these models is judgment-based. Factors that have been demonstrated to be important are: [[Wishful thinking]], [[anchoring]], [[planning fallacy]] and [[cognitive dissonance]]. A discussion on these and other factors can be found in work by Jørgensen and Grimstad <ref>{{cite web

* [[Cost estimation in software engineering]]

| author = Jørgensen, M. Grimstad, S.

* [[Cost estimation models]]

| title = How to Avoid Impact from Irrelevant and Misleading Information When Estimating Software Development Effort

| url = http://simula.no/research/engineering/publications/Simula.SE.112 }}

</ref>.

== See also ==

* [[Parametric estimating]]

* [[Estimation in software engineering]]

* [[Wideband Delphi]]

* [[Project management]]

* [[Planning poker]]

* [[Cost overrun]]

* [[COCOMO]]

* [[SEER-SEM]]

* [[Function points]]

* [[PROBEPlanning fallacy]]

* [[PERTProxy-based estimating]]

* [[Putnam model]]

* [[Software metric]]

== External links ==

* [[Software parametric models]]

}}

* Special Interest Group on Software Effort Estimation: http://www.forecastingprinciples.com/Software_Estimation/index.html

* General forecasting principles: http://www.forecastingprinciples.com

* Estimation resources: http://www.itprojectestimation.com/estrefs.htm

* Downloadable research papers on effort estimation: http://simula.no/research/engineering/projects/best

* Mike Cohn's Estimating With Use Case Points from article from Methods & Tools: http://www.methodsandtools.com/archive/archive.php?id=25

* Resources on Software Estimation from Steve McConnell: http://www.construx.com/Page.aspx?nid=297

== References ==

{{Reflist}}

[[Category:Software project management]]

{{reflist}}

[[Category:Software engineering costs]]

[[Category: Software project management]]