Content deleted Content added
→Accuracy: removed redundant sentence (if the method's goal is to predict accurately, then it's inability to do so isn't a specific weakness, but rather the result of other weaknesses, which is the focus of this section) |
m Open access bot: doi updated in citation with #oabot. |
||
| (31 intermediate revisions by 15 users not shown) | |||
Line 1:
{{Short description|Interactive forecasting method}}{{Futures studies}}The '''Delphi method''' or '''Delphi technique''' ({{IPAc-en|ˈ|d|ɛ|l|f|aɪ}} {{respell|DEL|fy}}; also known as '''Estimate-Talk-Estimate''' or '''ETE''') is a structured communication technique or method, originally developed as a systematic, interactive [[forecasting]] method
▲The '''Delphi method''' or '''Delphi technique''' ({{IPAc-en|ˈ|d|ɛ|l|f|aɪ}} {{respell|DEL|fy}}; also known as '''Estimate-Talk-Estimate''' or '''ETE''') is a structured communication technique or method, originally developed as a systematic, interactive [[forecasting]] method which relies on a panel of experts.<ref>{{cite journal | vauthors = Dalkey N, Helmer O | year = 1963 | title = An Experimental Application of the Delphi Method to the use of experts | journal = Management Science | volume = 9 | issue = 3| pages = 458–467 | doi=10.1287/mnsc.9.3.458| hdl = 2027/inu.30000029301680 | hdl-access = free }}</ref><ref>{{cite report | vauthors = Brown BB | title = Delphi Process: A Methodology Used for the Elicitation of Opinions of Experts. | publisher = Rand Corp | ___location = Santa Monica CA | date = September 1968 | id = P-3925 | url = https://apps.dtic.mil/sti/citations/AD0675981 }}</ref><ref>{{cite report | vauthors = Sackman H | title = Delphi assessment: Expert opinion, forecasting, and group process. | publisher = The Rand Corporation | ___location = Santa Monica CA | id = R-1283-PR | date = 1974 | url =https://apps.dtic.mil/sti/citations/AD0786878 }}</ref><ref>{{cite report | vauthors = Brown T | title = An Experiment in Probabilistic Forecasting | publisher = The Rand Corporation | ___location = Santa Monica CA | id = R-944-ARPA | date = 1972 | url = https://apps.dtic.mil/sti/citations/AD0777061 }}</ref><ref>{{cite book |title=The Delphi Method: Techniques and Applications |url=http://is.njit.edu/pubs/delphibook/ |year=1975 | veditors = Linstone HA, Turoff M |archive-url=https://web.archive.org/web/20080520015240/http://is.njit.edu/pubs/delphibook/ |___location=Reading, Mass. |publisher=Addison-Wesley |isbn=978-0-201-04294-8 |archive-date=2008-05-20 |url-status=dead}}</ref> The technique can also be adapted for use in face-to-face meetings, and is then called '''mini-Delphi'''. Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach, [[prediction markets]].<ref name="Green_2008" />
Delphi can also be used to help reach expert consensus and develop professional guidelines.<ref name="Taylor2020"/> It is used for such purposes in many health-related fields, including clinical medicine, public health, and research.<ref name="Taylor2020"/><ref name=Moher2010/>
Delphi is based on the principle that forecasts (or decisions) from a structured group of individuals are more accurate than those from unstructured groups.<ref name="rw2001">{{cite book | vauthors = Rowe G, Wright G | author-link2 = George Wright (psychologist) | date = 2001 | chapter = Expert Opinions in Forecasting
Special attention has to be paid to the formulation of the Delphi theses and the definition and selection of the experts in order to avoid methodological weaknesses that severely threaten the validity and reliability of the results.<ref>{{cite journal | vauthors = Markmann C, Spickermann A, von der Gracht HA, Brem A | title = Improving the question formulation in
Ensuring that the participants have requisite expertise and that more domineering participants do not overwhelm weaker-willed participants, as the first group tends to be less inclined to change their minds and the second group is more motivated to fit in, can be a barrier to reaching true consensus. <ref>{{cite journal | vauthors = Shang, Zshida| title = Use of Delphi in health sciences research: A narrative review. | journal = Medicine (Baltimore)| date = Feb 2023 | volume = 102 | issue = 7 | pages = e32829 | doi = 10.1097/MD.0000000000032829 | pmid = 36800594| pmc = 9936053 }}</ref>
▲Special attention has to be paid to the formulation of the Delphi theses and the definition and selection of the experts in order to avoid methodological weaknesses that severely threaten the validity and reliability of the results.<ref>{{cite journal | vauthors = Markmann C, Spickermann A, von der Gracht HA, Brem A | title = Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior. | journal = Futures & Foresight Science | date = March 2021 | volume = 3 | issue = 1 | pages = e56 | doi = 10.1002/ffo2.56 | s2cid = 225273393 | doi-access = free }}</ref><ref>{{cite journal | vauthors = Mauksch S, Heiko A, Gordon TJ | title = Who is an expert for foresight? A review of identification methods. | journal = Technological Forecasting and Social Change | date = May 2020 | volume = 154 | pages = 119982 | doi = 10.1016/j.techfore.2020.119982 | s2cid = 216161197 }} </ref>
==History==
Line 20 ⟶ 18:
Experts were asked to give their opinion on the probability, frequency, and intensity of possible enemy attacks. Other experts could anonymously give feedback. This process was repeated several times until a consensus emerged.
In 2021, a cross-disciplinary study by Beiderbeck et al. focused on new directions and advancements of the Delphi method, including [[Real-time Delphi]] formats. The authors provide a methodological toolbox for designing Delphi surveys including among others sentiment analyses of the field of psychology.<ref name="Beiderbeck_2021">{{cite journal | vauthors = Beiderbeck D, Frevel N, von der Gracht HA, Schmidt SL, Schweitzer VM | title = Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements | journal = MethodsX | volume = 8 | issue = |
▲In 2021, a cross-disciplinary study by Beiderbeck et al. focused on new directions and advancements of the Delphi method, including [[Real-time Delphi]] formats. The authors provide a methodological toolbox for designing Delphi surveys including among others sentiment analyses of the field of psychology.<ref name="Beiderbeck_2021">{{cite journal | vauthors = Beiderbeck D, Frevel N, von der Gracht HA, Schmidt SL, Schweitzer VM | title = Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements | journal = MethodsX | volume = 8 | issue = | pages = 101401 | date = 2021 | pmid = 34430297 | pmc = 8374446 | doi = 10.1016/j.mex.2021.101401 }}</ref>
==Key characteristics==
Line 70 ⟶ 66:
=== Use in health settings ===
The Delphi technique is widely used to help reach expert consensus in health-related settings.<ref name="Taylor2020"/> For example, it is frequently employed in the development of [[medical guideline]]s and [[Protocol (science)|protocol]]s.<ref name="Taylor2020">{{cite journal |vauthors=Taylor E |title=We Agree, Don't We? The Delphi Method for Health Environments Research |journal=HERD |volume=13 |issue=1 |pages=11–23 |date=2020 |pmid=31887097 |doi=10.1177/1937586719887709 |s2cid=209519275 |url=https://journals.sagepub.com/doi/pdf/10.1177/1937586719887709|url-access=subscription }}</ref>
====
Some examples of its application in [[public health]] contexts include non-alcoholic fatty liver disease,<ref>{{cite journal | vauthors = Lazarus JV, Mark HE, Anstee QM, Arab JP, Batterham RL, Castera L, Cortez-Pinto H, Crespo J, Cusi K, Dirac MA, Francque S, George J, Hagström H, Huang TT, Ismail MH, Kautz A, Sarin SK, Loomba R, Miller V, Newsome PN, Ninburg M, Ocama P, Ratziu V, Rinella M, Romero D, Romero-Gómez M, Schattenberg JM, Tsochatzis EA, Valenti L, Wong VW, Yilmaz Y, Younossi ZM, Zelber-Sagi S | display-authors = 6 | title = Advancing the global public health agenda for NAFLD: a consensus statement | journal = Nature Reviews. Gastroenterology & Hepatology | volume = 19 | issue = 1 | pages = 60–78 | date = January 2022 | pmid = 34707258 | doi = 10.1038/s41575-021-00523-4 | s2cid = 239891445 | doi-access = free }}</ref> iodine deficiency disorders,<ref>{{cite journal | vauthors = Schaffner M, Rochau U, Stojkov I, Qerimi Rushaj V, Völzke H, Marckmann G, Lazarus JH, Oberaigner W, Siebert U | display-authors = 6 | title = Barriers Against Prevention Programs for Iodine Deficiency Disorders in Europe: A Delphi Study | journal = Thyroid | volume = 31 | issue = 4 | pages = 649–657 | date = April 2021 | pmid = 32912084 | doi = 10.1089/thy.2020.0065 | s2cid = 221622474 }}</ref> building responsive health systems for communities affected by migration,<ref>{{cite journal | vauthors = Pottie K, Hui C, Rahman P, Ingleby D, Akl EA, Russell G, Ling L, Wickramage K, Mosca D, Brindis CD | display-authors = 6 | title = Building Responsive Health Systems to Help Communities Affected by Migration: An International Delphi Consensus | journal = International Journal of Environmental Research and Public Health | volume = 14 | issue = 2 | pages = 144 | date = February 2017 | pmid = 28165380 | pmc = 5334698 | doi = 10.3390/ijerph14020144 | doi-access = free }}</ref> the role of health systems in advancing well-being for those living with HIV,<ref>{{cite journal | vauthors = Lazarus JV, Safreed-Harmon K, Kamarulzaman A, Anderson J, Leite RB, Behrens G, Bekker LG, Bhagani S, Brown D, Brown G, Buchbinder S, Caceres C, Cahn PE, Carrieri P, Caswell G, Cooke GS, Monforte AD, Dedes N, Del Amo J, Elliott R, El-Sadr WM, Fuster-Ruiz de Apodaca MJ, Guaraldi G, Hallett T, Harding R, Hellard M, Jaffar S, Kall M, Klein M, Lewin SR, Mayer K, Pérez-Molina JA, Moraa D, Naniche D, Nash D, Noori T, Pozniak A, Rajasuriar R, Reiss P, Rizk N, Rockstroh J, Romero D, Sabin C, Serwadda D, Waters L | display-authors = 6 | title = Consensus statement on the role of health systems in advancing the long-term well-being of people living with HIV | journal = Nature Communications | volume = 12 | issue = 1 | pages = 4450 | date = July 2021 | pmid = 34272399 | pmc = 8285468 | doi = 10.1038/s41467-021-24673-w }}</ref> and in creating a 2022 paper on recommendations to end the [[COVID-19 pandemic]].<ref name=":0">{{cite journal | vauthors = Lazarus JV, Romero D, Kopka CJ, Karim SA, Abu-Raddad LJ, Almeida G, Baptista-Leite R, Barocas JA, Barreto ML, Bar-Yam Y, Bassat Q, Batista C, Bazilian M, Chiou ST, Del Rio C, Dore GJ, Gao GF, Gostin LO, Hellard M, Jimenez JL, Kang G, Lee N, Matičič M, McKee M, Nsanzimana S, Oliu-Barton M, Pradelski B, Pyzik O, Rabin K, Raina S, Rashid SF, Rathe M, Saenz R, Singh S, Trock-Hempler M, Villapol S, Yap P, Binagwaho A, Kamarulzaman A, El-Mohandes A | display-authors = 6 | title = A multinational Delphi consensus to end the COVID-19 public health threat | journal = Nature | volume = 611 | issue = 7935 | pages = 332–345 | date = November 2022 | pmid = 36329272 | pmc = 9646517 | doi = 10.1038/s41586-022-05398-2 }}</ref>▼
▲Some examples of its application in [[public health]] contexts include [[metabolic dysfunction–associated steatotic liver disease|non-alcoholic fatty liver disease]],<ref>{{cite journal | vauthors = Lazarus JV, Mark HE, Anstee QM, Arab JP, Batterham RL, Castera L, Cortez-Pinto H, Crespo J, Cusi K, Dirac MA, Francque S, George J, Hagström H, Huang TT, Ismail MH, Kautz A, Sarin SK, Loomba R, Miller V, Newsome PN, Ninburg M, Ocama P, Ratziu V, Rinella M, Romero D, Romero-Gómez M, Schattenberg JM, Tsochatzis EA, Valenti L, Wong VW, Yilmaz Y, Younossi ZM, Zelber-Sagi S | display-authors = 6 | title = Advancing the global public health agenda for NAFLD: a consensus statement | journal = Nature Reviews. Gastroenterology & Hepatology | volume = 19 | issue = 1 | pages = 60–78 | date = January 2022 | pmid = 34707258 | doi = 10.1038/s41575-021-00523-4 | s2cid = 239891445 | doi-access = free | hdl = 11424/243850 | hdl-access = free }}</ref> iodine deficiency disorders,<ref>{{cite journal | vauthors = Schaffner M, Rochau U, Stojkov I, Qerimi Rushaj V, Völzke H, Marckmann G, Lazarus JH, Oberaigner W, Siebert U | display-authors = 6 | title = Barriers Against Prevention Programs for Iodine Deficiency Disorders in Europe: A Delphi Study | journal = Thyroid | volume = 31 | issue = 4 | pages = 649–657 | date = April 2021 | pmid = 32912084 | doi = 10.1089/thy.2020.0065 | s2cid = 221622474 }}</ref> building responsive health systems for communities affected by migration,<ref>{{cite journal | vauthors = Pottie K, Hui C, Rahman P, Ingleby D, Akl EA, Russell G, Ling L, Wickramage K, Mosca D, Brindis CD | display-authors = 6 | title = Building Responsive Health Systems to Help Communities Affected by Migration: An International Delphi Consensus | journal = International Journal of Environmental Research and Public Health | volume = 14 | issue = 2 | pages = 144 | date = February 2017 | pmid = 28165380 | pmc = 5334698 | doi = 10.3390/ijerph14020144 | doi-access = free }}</ref> the role of health systems in advancing well-being for those living with HIV,<ref>{{cite journal | vauthors = Lazarus JV, Safreed-Harmon K, Kamarulzaman A, Anderson J, Leite RB, Behrens G, Bekker LG, Bhagani S, Brown D, Brown G, Buchbinder S, Caceres C, Cahn PE, Carrieri P, Caswell G, Cooke GS, Monforte AD, Dedes N, Del Amo J, Elliott R, El-Sadr WM, Fuster-Ruiz de Apodaca MJ, Guaraldi G, Hallett T, Harding R, Hellard M, Jaffar S, Kall M, Klein M, Lewin SR, Mayer K, Pérez-Molina JA, Moraa D, Naniche D, Nash D, Noori T, Pozniak A, Rajasuriar R, Reiss P, Rizk N, Rockstroh J, Romero D, Sabin C, Serwadda D, Waters L | display-authors = 6 | title = Consensus statement on the role of health systems in advancing the long-term well-being of people living with HIV | journal = Nature Communications | volume = 12 | issue = 1 |
==== Use in reporting guidelines====▼
Use of the Delphi method in the development of guidelines for the reporting of health research<ref name=Moher2010>{{cite journal | vauthors = Moher D, Schulz KF, Simera I, Altman DG | title = Guidance for developers of health research reporting guidelines | journal = PLOS Medicine | volume = 7 | issue = 2 | pages = e1000217 | date = February 2010 | pmid = 20169112 | pmc = 2821895 | doi = 10.1371/journal.pmed.1000217 }}</ref> is recommended, especially for experienced developers.<ref name="EQUATOR">{{cite web |title=Developing your reporting guideline |url=https://www.equator-network.org/toolkits/developing-a-reporting-guideline/developing-your-reporting-guideline/ |website=www.equator-network.org |publisher=[[EQUATOR Network]] |archive-url=https://web.archive.org/web/20220929172737/https://www.equator-network.org/toolkits/developing-a-reporting-guideline/developing-your-reporting-guideline/ |archive-date=29 September 2022 |language=en |url-status=live}}</ref> Since this advice was made in 2010, two systematic reviews have found that fewer than 30% of published reporting guidelines incorporated Delphi methods into the development process.<ref>{{cite journal | vauthors = Wang X, Chen Y, Yang N, Deng W, Wang Q, Li N, Yao L, Wei D, Chen G, Yang K | display-authors = 6 | title = Methodology and reporting quality of reporting guidelines: systematic review | journal = BMC Medical Research Methodology | volume = 15 | issue = 74 | pages = 74 | date = September 2015 | pmid = 26395179 | pmc = 4579604 | doi = 10.1186/s12874-015-0069-z }}</ref><ref>{{cite journal | vauthors = Banno M, Tsujimoto Y, Kataoka Y | title = The majority of reporting guidelines are not developed with the Delphi method: a systematic review of reporting guidelines | journal = Journal of Clinical Epidemiology | volume = 124 | pages = 50–57 | date = August 2020 | pmid = 32302679 | doi = 10.1016/j.jclinepi.2020.04.010 | s2cid = 215809096 }}</ref>▼
▲Use of the Delphi method in the development of guidelines for the reporting of health research<ref name=Moher2010>{{cite journal | vauthors = Moher D, Schulz KF, Simera I, Altman DG | title = Guidance for developers of health research reporting guidelines | journal = PLOS Medicine | volume = 7 | issue = 2 | pages = e1000217 | date = February 2010 | pmid = 20169112 | pmc = 2821895 | doi = 10.1371/journal.pmed.1000217 | doi-access = free }}</ref> is recommended, especially for experienced developers.<ref name="EQUATOR">{{cite web |title=Developing your reporting guideline |url=https://www.equator-network.org/toolkits/developing-a-reporting-guideline/developing-your-reporting-guideline/ |website=www.equator-network.org |publisher=[[EQUATOR Network]] |archive-url=https://web.archive.org/web/20220929172737/https://www.equator-network.org/toolkits/developing-a-reporting-guideline/developing-your-reporting-guideline/ |archive-date=29 September 2022 |language=en |url-status=live}}</ref> Since this advice was made in 2010, two systematic reviews have found that fewer than 30% of published reporting guidelines incorporated Delphi methods into the development process.<ref>{{cite journal | vauthors = Wang X, Chen Y, Yang N, Deng W, Wang Q, Li N, Yao L, Wei D, Chen G, Yang K | display-authors = 6 | title = Methodology and reporting quality of reporting guidelines: systematic review | journal = BMC Medical Research Methodology | volume = 15 | issue = 74 |
=== Online Delphi systems ===
Line 83 ⟶ 80:
==Variations==
Traditionally the Delphi method has aimed at a consensus of the most probable future by iteration. Other versions, such as the Policy Delphi,<ref>{{cite journal | vauthors = Turoff M | title = The Design of a Policy Delphi | url = http://is.njit.edu/pubs/delphibook/ch3b1.pdf | journal = Technological Forecasting and Social Change | volume = 2 | issue = 2| date = January 1970 | pages = 149–171 | doi = 10.1016/0040-1625(70)90161-7 }}</ref><ref>{{citation |author= |title=Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health |year=1996 | veditors = Adler M, Ziglio E |___location=London |publisher=Kingsley Publishers |isbn=978-1-85302-104-6 }}</ref>
In Europe, more recent web-based experiments have used the Delphi method as a communication technique for interactive [[decision-making]] and [[e-democracy]].<ref>{{cite book |title= Democrazia elettronica. Metodo Delphi e politiche pubbliche | trans-title = Electronic Democracy. Delphi Method and Public Policy-Making |year=2001| vauthors = Bolognini M |language=it |publisher=Carocci Editore |___location=Rome |isbn=978-88-430-2035-5}}. An example of e-democracy application is DEMOS (Delphi Mediation Online System), whose prototype was presented at the 3rd Worldwide Forum on Electronic Democracy, in 2002.</ref>
The Argument Delphi, developed by Osmo Kuusi, focuses on ongoing discussion and finding relevant arguments rather than focusing on the output. The Disaggregative Policy Delphi, developed by Petri Tapio, uses cluster analysis as a systematic tool to construct various scenarios of the future in the latest Delphi round.<ref>{{cite journal | vauthors = Tapio P | year = 2003 | title = Disaggregative Policy Delphi: Using cluster analysis as a tool for systematic scenario formation | journal = Technological Forecasting and Social Change | volume = 70 | issue = 1| pages = 83–101 | doi = 10.1016/S0040-1625(01)00177-9 | s2cid = 53516828 }}</ref> The respondent's view on the probable and the preferable future are dealt with as separate cases. The computerization of Argument Delphi is relatively difficult because of several problems like argument resolution, argument aggregation and argument evaluation. The computerization of Argument Delphi, developed by [[Sadi Evren Seker]], proposes solutions to such problems.<ref>{{cite journal | vauthors = Seker SE | year = 2015 | title = Computerized Argument Delphi Technique | journal = IEEE Access | volume = 3 | issue = 2| pages = 368–380 | doi = 10.1109/ACCESS.2015.2424703 | doi-access = free | bibcode = 2015IEEEA...3..368S }}</ref>
A fast-track Delphi was developed to provide consensual expert opinion on the state of scientific knowledge in public health crises.<ref name=fast-track/> It can provide results within three weeks, while the conventional Delphi can take several months (sometimes years).<ref name=fast-track>{{cite journal |last1=Velarde Crézé |first1=Camille |last2=Duperrex |first2=Olivier |last3=Lebon |first3=Luc |last4=Faivre |first4=Vincent |last5=Pasche |first5=Myriam |last6=Cornuz |first6=Jacques |date=18 December 2024 |title=A multi-stage approach to support timely health policy decisions during crisis: the fast-track Delphi |journal=[[BMC Public Health]] |volume=24 |issue=1 |page=3412 |doi=10.1186/s12889-024-20903-0 |doi-access=free |pmid=39696176 |pmc=11653821 }}</ref>
==Accuracy==
Today the Delphi method is a widely accepted forecasting tool and has been used successfully for thousands of studies in areas varying from technology forecasting to [[drug abuse]].<ref>{{cite book | vauthors = Jillson IA | chapter = II.B.3. The National Drug-Abuse Policy Delphi: Progress Report and Findings to Date | veditors = Turoff M, Linstone HA | title = The Delphi Method: Techniques and Applications | date = 2002 | chapter-url = http://is.njit.edu/pubs/delphibook/ch3b3.html | archive-url = https://web.archive.org/web/20120206170836/http://is.njit.edu/pubs/delphibook/ch3b3.html | archive-date = 6 February 2012 }}</ref>
Overall the track record of the Delphi method is mixed.<ref name=":1">Khodyakov, D., Grant, S., Kroger, J., Bauman, M. (2023). ''RAND methodological guidance for conducting and critically appraising Delphi panels.'' RAND Corporation. www.rand.org/t/TLA3082-1 https://doi.org/10.7249/tla3082-1</ref> There have been many cases when the method produced poor results. Still, some authors attribute this to poor application of the method and not to the weaknesses of the method itself.
It must also be realized that in areas such as science and technology forecasting, the degree of uncertainty is so great that exact and always correct predictions are impossible, so a high degree of error is to be expected. An important challenge for the method is ensuring sufficiently knowledgeable panelists. If panelists are misinformed about a topic, the use of Delphi may only add confidence to their ignorance.<ref name = "Green_2008">{{cite journal | vauthors = Green KC, Armstrong JS, Graefe A | title = Methods to elicit forecasts from groups: Delphi and prediction markets compared. | journal = Foresight: The International Journal of Applied Forecasting | date = June 2008 | volume = 8 | pages = 17–20 | doi = 10.2139/ssrn.1153124 | url = https://repository.upenn.edu/cgi/viewcontent.cgi?article=1168&context=marketing_papers | doi-access = free }}</ref>
One of the initial problems of the method was its inability to make complex forecasts with multiple factors. Potential future outcomes were usually considered as if they had no effect on each other. Later on, several extensions to the Delphi method were developed to address this problem, such as [[cross impact analysis]], that takes into consideration the possibility that the occurrence of one event may change probabilities of other events covered in the survey. Still the Delphi method can be used most successfully in forecasting single scalar indicators.
Line 136:
{{Qualitative forecasting methods}}
{{Risk management}}
{{Authority control}}
| |||