Content deleted Content added
→Evaluating collective impact: fix broken link and add archive url for Guide to Evaluating Collective Impact |
|||
| (108 intermediate revisions by 59 users not shown) | |||
Line 1:
{{Short description|Collecting, analyzing, and using information to assess projects}}
{{Confuse|program analysis}}
{{multiple issues|{{Very long|date=March 2022}}
{{More citations needed|date=March 2022}}}}
'''Program evaluation''' is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and [[program (management)|programs]],<ref>Administration for Children and Families (2010) ''[https://web.archive.org/web/20051105021406/http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/index.html The Program Manager's Guide to Evaluation]. Chapter 2: What is program evaluation?''.</ref> particularly about their [[effectiveness]] (whether they do what they are intended to do) and [[efficiency]] (whether they are good value for money).
In the [[public sector|public]], [[private sector|private]], and [[voluntary sector]], [[project stakeholder|stakeholders]] might be required to assess—under law or charter—or want to know whether the programs they are funding, implementing, voting for, receiving or opposing are producing the promised effect. To some degree, program evaluation falls under traditional [[cost–benefit analysis]], concerning fair returns on the outlay of economic and other assets; however, social outcomes can be more complex to assess than market outcomes, and a different skillset is required. Considerations include how much the program costs per participant, program [[impact evaluation|impact]], how the program could be improved, whether there are better alternatives, if there are [[unintended consequences|unforeseen consequences]], and whether the program goals are appropriate and useful.<ref>{{cite report|last=Shackman|first=Gene|title=What Is Program Evaluation: A Beginner's Guide|publisher=The Global Social Change Research Project|ssrn=3060080}}</ref> Evaluators help to answer these questions. Best practice is for the evaluation to be a joint project between evaluators and stakeholders.<ref>{{cite web|title=Hints for Conducting Strong Evaluations|url=https://www.cdc.gov/eval/strongevaluations/index.htm|work=Program Evaluation|publisher=CDC – Office of the Associate Director for Program – Program Evaluation|access-date=April 8, 2012}}</ref>
A wide range of different titles are applied to program evaluators, perhaps haphazardly at times, but there are some established usages: those who regularly use program evaluation skills and techniques on the job are known as '''program analysts'''<!-- only this title is bolded, having been made a target of a #REDIRECT page from [[Program Analyst (administration)]], as a primary search term to land here, parallel to Policy Analyst in the related profession; please do NOT bold others without talk page discussion -->; those whose positions combine [[administrative assistant]] or [[secretary]] duties with program evaluation are known as program assistants, program clerks (United Kingdom), program support specialists, or program associates; those whose positions add lower-level [[project management]] duties are known as Program Coordinators.
The process of evaluation is considered to be a relatively recent phenomenon. However, planned social evaluation has been documented as dating as far back as 2200 BC.<ref>{{cite book|author=Shadish, W. R.|author2=Cook, T. D.|author3=Leviton, L. C.|title=Foundations of program evaluation: Theories of practice.|year=1991|publisher=Sage.|___location=Newbury Park, CA}}</ref> Evaluation became particularly relevant in the United States in the 1960s during the period of the [[Great Society]] social programs associated with the [[John F. Kennedy|Kennedy]] and [[Lyndon Johnson|Johnson]] administrations.<ref>{{cite web|url=http://www.dol.gov/oasam/programs/history/dolchp06.htm|archive-url=https://web.archive.org/web/20080413170714/http://www.dol.gov/oasam/programs/history/dolchp06.htm|url-status=dead|archive-date=April 13, 2008|title=U.S. Department of Labor – Brief History of DOL – Eras of the New Frontier and the Great Society, 1961-1969|publisher=dol.gov}}</ref><ref>[https://www.archives.gov/research/guide-fed-records/groups/051.html National Archives, Records of the Office of Management and Budget (1995) ''51.8.8 Records of the Office of Program Evaluation'']</ref>
Program evaluations can involve both [[quantitative method|quantitative]] and [[qualitative method]]s of [[social research]]. People who do program evaluation come from many different backgrounds, such as [[sociology]], [[psychology]], [[economics]], [[social work]], as well as [[political science]] subfields such as [[public policy]] and [[public administration]] who have studied a similar methodology known as [[policy analysis]]. Some universities also have specific training programs, especially at the [[Postgraduate education|postgraduate]] level in program evaluation, for those who studied an [[Undergraduate education|undergraduate]] subject area lacking in program evaluation skills.<ref>American Evaluation Association's [https://www.eval.org/Education-Programs/University-Programs list of university programs].</ref>
==Conducting an evaluation==
Program evaluation may be conducted at several stages during a program's lifetime. Each of these stages raises different questions to be answered by the evaluator, and correspondingly different evaluation approaches are needed. Rossi, Lipsey and Freeman (2004) suggest the following kinds of assessment, which may be appropriate at these different stages:
Line 18 ⟶ 26:
A needs assessment examines the population that the program intends to target, to see whether the need as conceptualized in the program actually exists in the population; whether it is, in fact, a problem; and if so, how it might best be dealt with. This includes identifying and diagnosing the actual problem the program is trying to address, who or what is affected by the problem, how widespread the problem is, and what are the measurable effects that are caused by the problem. For example, for a housing program aimed at mitigating homelessness, a program evaluator may want to find out how many people are homeless in a given geographic area and what their demographics are. Rossi, Lipsey and Freeman (2004) caution against undertaking an intervention without properly assessing the need for one, because this might result in a great deal of wasted funds if the need did not exist or was misconceived.
Needs assessment involves the processes or methods used by evaluators to describe and diagnose social needs<ref name="Rossi"/
This is essential for evaluators because they need to identify whether programs are effective and they cannot do this unless they have identified what the problem/need is. Programs that do not do a needs assessment can have the illusion that they have eradicated the problem/need when in fact there was no need in the first place. Needs assessment involves research and regular consultation with community stakeholders and with the people that will benefit from the project before the program can be developed and implemented. Hence it should be a bottom-up approach. In this way potential problems can be realized early because the process would have involved the community in identifying the need and thereby allowed the opportunity to identify potential barriers.
Line 27 ⟶ 35:
Having clearly identified what the problem is, evaluators need to then assess the extent of the problem. They need to answer the ‘where’ and ‘how big’ questions. Evaluators need to work out where the problem is located and how big it is. Pointing out that a problem exists is much easier than having to specify where it is located and how rife it is. Rossi, Lipsey & Freeman (2004) gave an example that: a person identifying some battered children may be enough evidence to persuade one that child abuse exists. But indicating how many children it affects and where it is located geographically and socially would require knowledge about abused children, the characteristics of perpetrators and the impact of the problem throughout the political authority in question.
This can be difficult considering that child abuse is not a public behavior, also keeping in mind that estimates of the rates on private behavior are usually not possible because of factors like unreported cases. In this case evaluators would have to use data from several sources and apply different approaches in order to estimate incidence rates. There are two more questions that need to be answered:<ref name="Barbazette">{{cite
Evaluators need to also answer the ’how’ and ‘what’ questions<ref name=Barbazette/> The ‘how’ question requires that evaluators determine how the need will be addressed. Having identified the need and having familiarized oneself with the community evaluators should conduct a performance analysis to identify whether the proposed plan in the program will actually be able to eliminate the need. The ‘what’ question requires that evaluators conduct a [[task analysis]] to find out what the best way to perform would be. For example, whether the job performance standards are set by an organization or whether some governmental rules need to be considered when undertaking the task.<ref name=Barbazette/>
Third, define and identify the target of interventions and accurately describe the nature of the service needs of that population<ref name=Rossi/>
It is important to know what/who the target population is/are – it might be individuals, groups, communities, etc. There are three units of the population: population at risk, population in need and population in demand<ref name=Rossi/>
* Population at risk: are people with a significant probability of developing the risk e.g. the population at risk for birth control programs are women of child
* Population in need: are people with the condition that the program seeks to address; e.g. the population in need for a program that aims to provide
* Population in demand: that part of the population in need that agrees to be having the need and are willing to take part in what the program has to offer e.g. not all HIV positive people will be willing to take
Being able to specify what/who the target is will assist in establishing appropriate boundaries, so that interventions can correctly address the target population and be feasible to apply<<ref name=Rossi/>
Line 41 ⟶ 49:
#:Evaluators need to compare current situation to the desired or necessary situation. The difference or the gap between the two situations will help identify the need, purpose and aims of the program.
#Identify priorities and importance
#:In the first step above, evaluators would have identified a number of interventions that could potentially address the need e.g. training and development, [[organization development]] etc. These must now be examined in view of their significance to the
# Identify causes of performance problems and/or opportunities
#:When the needs have been prioritized the next step is to identify specific problem areas within the need to be addressed. And to also assess the skills of the people that will be carrying out the interventions.
Line 49 ⟶ 57:
===Assessing program theory===
The program theory, also called a [[logic model]], knowledge map,<ref>Wright, B. & Wallis, S.E. (2019). ''Practical Mapping for Applied Research and Program Evaluation''. SAGE Publications.</ref> or impact pathway,<ref>Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).</ref> is an assumption, implicit in the way the program is designed, about how the program's actions are supposed to achieve the outcomes it intends. This 'logic model' is often not stated explicitly by people who run programs, it is simply assumed, and so an evaluator will need to draw out from the program staff how exactly the program is supposed to achieve its aims and assess whether this logic is plausible. For example, in an HIV prevention program, it may be assumed that educating people about HIV/AIDS transmission, risk and safe sex practices will result in safer sex being practiced. However, research in South Africa increasingly shows that in spite of increased education and knowledge, people still often do not practice safe sex.<ref>Van der Riet, M. (2009). 'The production of context: using activity theory to understand behaviour change in response to HIV and AIDS.' Unpublished doctoral dissertation. University of KwaZulu-Natal, Pietermaritzburg.</ref> Therefore, the logic of a program which relies on education as a means to get people to use condoms may be faulty. This is why it is important to read research that has been done in the area.
Explicating this logic can also reveal unintended or unforeseen consequences of a program, both positive and negative. The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders about what the program is actually supposed to do and how it is supposed to do it, which is often lacking (see [[Participatory impact pathways analysis]]). Of course, it is also possible that during the process of trying to elicit the logic model behind a program the evaluators may discover that such a model is either incompletely developed, internally contradictory, or (in worst cases) essentially nonexisistent. This decidedly limits the effectiveness of the evaluation, although it does not necessarily reduce or eliminate the program.<ref>Eveland, JD. (1986) "Small Business Innovation Research Programs: Solutions Seeking
Creating a logic model is a wonderful way to help visualize important aspects of programs, especially when preparing for an evaluation. An evaluator should create a logic model with input from many different stake holders. Logic Models have 5 major components: Resources or Inputs, Activities, Outputs, Short-term outcomes, and Long-term outcomes <ref name="McLaughlin, J. A. 1999">McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: a tool for telling your programs performance story. Evaluation and program planning, 22(1), 65-72.</ref> Creating a logic model helps articulate the problem, the resources and capacity that are currently being used to address the problem, and the measurable outcomes from the program. Looking at the different components of a program in relation to the overall short-term and long-term goals allows for illumination of potential misalignments. Creating an actual logic model is particularly important because it helps clarify for all stakeholders: the definition of the problem, the overarching goals, and the capacity and outputs of the program.<ref name="McLaughlin, J. A. 1999"/>
Line 58 ⟶ 66:
This entails assessing the program theory by relating it to the needs of the target population the program is intended to serve. If the program theory fails to address the needs of the target population it will be rendered ineffective even when if it is well implemented.<ref name=Rossi/>
* Assessment of logic and plausibility<ref name=Rossi/>
This form of assessment involves asking a panel of expert reviewers to critically review the logic and plausibility of the assumptions and expectations inherent in the program's design.<ref name=Rossi/> The review process is unstructured and open ended so as to address certain issues on the program design. Rutman (1980), Smith (1989), and
:Are the program goals and objectives well defined?
:Are the program goals and objectives feasible?
:Is the change process presumed in the program theory feasible?
:Are the procedures for identifying members of the target population, delivering service to them, and sustaining that service through completion well defined and
:Are the constituent components, activities, and functions of the program well defined and sufficient?
:Are the resources allocated to the program and its various activities adequate?
Line 73 ⟶ 81:
These different forms of assessment of program theory can be conducted to ensure that the program theory is sound.
Wright and Wallis (2019) described an additional technique for assessing a program theory based on the theory's ''structure''. This approach, known as integrative propositional analysis (IPA), is based on research streams finding that theories were more likely to work as expected when they had better structure (in addition meaning and data). IPA involves, first, identifying the propositions (statements of cause-and-effect) and creating a visual diagram of those propositions. Then, the researcher examines the number of concepts and causal relationships between them (circles and arrows on the diagram) to measure the breadth and depth of understanding reflected in the theory's structure. The measure for ''breadth'' is the number of concepts. This is based on the idea that real-world programs involve a lot of interconnected parts, therefore a theory that shows a larger number of concepts shows greater breadth of understanding of the program. The ''depth'' is the percentage of concepts that are the result of more than one other concept. This is based on the idea that, in real-world programs, things have more than one cause. Hence, a concept that is the result of more than one other concept in the theory shows better understanding of that concept; a theory with a higher percentage of better-understood concepts shows a greater depth of understanding of the program.
===Assessing implementation===
Process analysis looks beyond the theory of what the program is supposed to do and instead evaluates how the program is being implemented. This evaluation determines whether the components identified as critical to the success of the program are being implemented. The evaluation determines whether target populations are being reached, people are receiving the intended services, staff are adequately qualified. Process evaluation is an ongoing process in which repeated measures may be used to evaluate whether the program is being implemented effectively. This problem is particularly critical because many innovations, particularly in areas like education and public policy, consist of fairly complex chains of action. For example, process evaluation can be used in public health research.<ref>{{Cite journal|last1=Mclaughlin|first1=Matthew|last2=Duff|first2=Jed|last3=Sutherland|first3=Rachel|last4=Campbell|first4=Elizabeth|last5=Wolfenden|first5=Luke|last6=Wiggers|first6=John|date=December 2020|title=Protocol for a mixed methods process evaluation of a hybrid implementation-effectiveness trial of a scaled-up whole-school physical activity program for adolescents: Physical Activity 4 Everyone (PA4E1)|url= |journal=Trials|language=en|volume=21|issue=1|pages=268|doi=10.1186/s13063-020-4187-5|issn=1745-6215|pmc=7077014|pmid=32183902 |doi-access=free }}</ref> Many of which these elements rely on the prior correct implementation of other elements, and will fail if the prior implementation was not done correctly. This was conclusively demonstrated by [[Gene V. Glass]] and many others during the 1980s. Since incorrect or ineffective implementation will produce the same kind of neutral or negative results that would be produced by correct implementation of a poor innovation, it is essential that evaluation research assess the implementation process itself.<ref>Eveland, JD (1986)
===Assessing the impact (effectiveness)===
The impact evaluation determines the causal effects of the program. This involves trying to measure if the program has achieved its intended outcomes, i.e. program outcomes.
====Program
An outcome is the state of the target population or the social conditions that a program is expected to have changed.<ref name=Rossi/> Program outcomes are the observed characteristics of the target population or social conditions, not of the program. Thus the concept of an outcome does not necessarily mean that the program targets have actually changed or that the program has caused them to change in any way.<ref name=Rossi/>
Line 88 ⟶ 98:
*'''Program effect''' refers to that portion of an outcome change that can be attributed uniquely to a program as opposed to the influence of some other factor.
====Measuring
Outcome measurement is a matter of representing the circumstances defined as the outcome by means of observable indicators that vary systematically with changes or differences in those circumstances.<ref name=Rossi/> Outcome measurement is a systematic way to assess the extent to which a program has achieved its intended outcomes.<ref>The Intermediary Development Series. '' Measuring Outcomes. Dare Mighty Things, INC''.</ref> According to Mouton (2009) measuring the impact of a program means demonstrating or estimating the accumulated differentiated proximate and emergent effect, some of which might be unintended and therefore unforeseen.<ref name="Mouton, J 2009">{{cite journal | last1 = Mouton
Outcome measurement serves to help
This can involve using sophisticated statistical techniques in order to measure the effect of the program and to find causal relationship between the program and the various outcomes
===Assessing efficiency===
Finally, cost-benefit or cost-
<ref>{{cite journal|last1=Shahab|first1=Sina|last2=Clinch|first2=J Peter|last3=O’Neill|first3=Eoin|title=Impact-based planning evaluation: Advancing normative criteria for policy analysis|journal=Environment and Planning B: Urban Analytics and City Science|volume=46|issue=3|date=21 July 2017|pages=534–550|doi=10.1177/2399808317720446|url=https://researchrepository.ucd.ie/bitstream/10197/9594/4/Final%20-%20Impact-based%20Evaluation.pdf|hdl=10197/9594|s2cid=54510005 |hdl-access=free}}</ref>
==Determining causation==
Line 109 ⟶ 120:
If the program is fairly large, and there are enough data, statistical analysis can be used to make a reasonable case for the program by showing, for example, that other causes are unlikely.
==Reliability, validity and sensitivity
It is important to ensure that the instruments (for example, tests, questionnaires, etc.) used in program evaluation are as reliable, valid and sensitive as possible. According to Rossi et al. (2004, p. 222),<ref name = "Rossi">Rossi, P. Lipsey, M. W., & Freeman, H.E. (2004). ''Evaluation: A systematic approach'' (7th ed.). Thousand Oaks, CA: Sage.</ref> 'a measure that is poorly chosen or poorly conceived can completely undermine the worth of an impact assessment by producing misleading estimates. Only if outcome measures are valid, reliable and appropriately sensitive can impact assessments be regarded as credible'.
===Reliability===
Line 121 ⟶ 132:
The principal purpose of the evaluation process is to measure whether the program has an effect on the social problem it seeks to redress; hence, the measurement instrument must be sensitive enough to discern these potential changes (Rossi et al., 2004).<ref name="Rossi" /> A measurement instrument may be insensitive if it contains items measuring outcomes which the program couldn't possibly effect, or if the instrument was originally developed for applications to individuals (for example standardized psychological measures) rather than to a group setting (Rossi et al., 2004).<ref name="Rossi" /> These factors may result in 'noise' which may obscure any effect the program may have had.
Only measures which adequately achieve the benchmarks of reliability, validity and sensitivity can be said to be credible evaluations. It is the duty of evaluators to produce credible evaluations, as their findings may have far reaching effects. A discreditable evaluation which is unable to show that a program is achieving its purpose when it is in fact creating positive change may cause the program to lose its funding undeservedly.{{synthesis
==Steps to
The Center for Disease Control (CDC) delineates six steps to a complete program evaluation. The steps described are: engage stakeholder, describe the program, focus the evaluation design, gather credible evidence, justify conclusions, and ensure use and share lessons learned.<ref>{{cite web|last=Center for Disease Control|title=Evaluation Framework|url=https://www.cdc.gov/eval/framework/index.htm|publisher=CDC|access-date=20 September 2012}}</ref> These steps can happen in a cycle framework to represent the continuing process of evaluation.
==Evaluating collective impact==
Though program evaluation processes mentioned here are appropriate for most programs, highly complex non-linear initiatives, such as those using the [[collective impact]] (CI) model, require a dynamic approach to evaluation. Collective impact is "the commitment of a group of important actors from different sectors to a common agenda for solving a specific social problem"<ref>{{cite web|last1=Kania|first1=John|last2=Kramer|first2=Mark|title=Collective Impact|url=http://www.ssireview.org/articles/entry/collective_impact|website=Stanford Social Innovation Review|publisher=Stanford Center on Philanthropy and Civil Society at Stanford University|access-date=19 September 2014}}</ref> and typically involves three stages, each with a different recommended evaluation approach:
* '''Early phase:''' CI participants are exploring possible strategies and developing plans for action. Characterized by uncertainty.
''Recommended evaluation approach:'' Developmental evaluation to help CI partners understand the context of the initiative and its development:<ref>{{cite web
| last1 = Preskill
| first1 = Hallie
| last2 = Parkhurst
| first2 = Marcie
| last3 = Juster
| first3 = Jennifer Splansky
| title = Guide to Evaluating Collective Impact
| url = https://www.fsg.org/resource/guide-evaluating-collective-impact/#resource-downloads
| date = 2014-05-05
| access-date = 2025-08-19
| website = www.fsg.org
| publisher = FSG, Inc.
| url-status = live
| archive-url = https://web.archive.org/web/20250502000322/https://www.fsg.org/resource/guide-evaluating-collective-impact/#resource-downloads
| archive-date = 2025-05-02
}}
</ref> "Developmental evaluation involves real time feedback about what is emerging in complex dynamic systems as innovators seek to bring about systems change."<ref>{{cite web|last1=Patton|first1=Michael|title=Evaluation Approaches and Techniques|url=http://tei.gwu.edu/evaluation-approaches-and-techniques#developmental-evaluation|website=The Evaluators' Institute|publisher=George Washington University|access-date=2014-09-19}}</ref>
* '''Middle phase:''' CI partners implement agreed upon strategies. Some outcomes become easier to anticipate.
''Recommended evaluation approach:'' Formative evaluation to refine and improve upon the progress, as well as continued developmental evaluation to explore new elements as they emerge. Formative evaluation involves "careful monitoring of processes in order to respond to emergent properties and any unexpected outcomes."<ref>{{cite web|title=Formative Evaluation|url=http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=24:formative-evaluation&catid=17:formative-evaluation&Itemid=125|website=Community Sustainability Engagement Evaluation Toolbox|
* '''Later phase:''' Activities achieve stability and are no longer in formation. Experience informs knowledge about which activities may be effective.
''Recommended evaluation approach:'' Summative evaluation
==Planning a program evaluation==
Planning a program evaluation can be broken up into four parts: focusing the evaluation, collecting the information, using the information, and managing the evaluation.<ref>{{cite web|last1=Taylor-Powell|first1=Ellen|last2=Steele|first2=Sarah|last3=Douglah|first3=Mohammad|title=Planning a Program Evaluation|url=http://learningstore.uwex.edu/assets/pdfs/g3658-1.pdf|website=University of Wisconsin Extension|
*What am I going to evaluate?
*What is the purpose of this evaluation?
Line 152 ⟶ 179:
==Methodological constraints and challenges==
===The shoestring approach===
The
===Budget constraints===
Frequently, programs are faced with budget constraints because most original projects do not include a budget to conduct an evaluation (Bamberger et al., 2004). Therefore, this automatically results in evaluations being allocated smaller budgets that are inadequate for a rigorous evaluation. Due to the budget constraints it might be difficult to effectively apply the most appropriate methodological instruments. These constraints may consequently affect the time available in which to do the evaluation (Bamberger et al., 2004).<ref name="Bamberger" /> Budget constraints may be addressed by simplifying the evaluation design, revising the sample size, exploring economical data collection methods (such as using volunteers to collect data, shortening surveys, or using focus groups and key informants) or looking for reliable [[secondary data]] (Bamberger et al., 2004).<ref name="Bamberger" />
===Time constraints===
Line 169 ⟶ 195:
===Five-tiered approach===
The five-tiered approach to evaluation further develops the strategies that the shoestring approach to evaluation is based upon.<ref name="F.H.J">{{Cite journal | last1 = Jacobs | first1 = F. H. | title = Child and Family Program Evaluation: Learning to Enjoy Complexity | doi = 10.1207/S1532480XADS0702_3 | journal = Applied Developmental Science | volume = 7 | issue = 2 | pages = 62–75 | year = 2003 |
The earlier tiers (1-3) generate descriptive and process-oriented information while the later tiers (4-5) determine both the short-term and the long-term effects of the program.<ref name="NPEN">{{cite web|publisher=National Parenting Education Network (n.d.)|title=Five-tiered approach to evaluation|
* Tier 1: needs assessment (sometimes referred to as pre-implementation)<ref name="B&D">{{cite web|first1=S. J.|last1=Bailey|first2=M. Y.|last2=Deen|year=2002|title=A Framework for introducing program evaluation to extension faculty and staff|website=Journal of Extension|
* Tier 2: monitoring and accountability
* Tier 3: quality review and program clarification (sometimes referred to as understanding and refining)<ref name="CYF">{{cite web|title=Five-tiered approach to program evaluation|
* Tier 4: achieving outcomes
* Tier 5: establishing impact
Line 185 ⟶ 211:
===Methodological challenges presented by language and culture===
The purpose of this section is to draw attention to some of the methodological challenges and dilemmas evaluators are potentially faced with when conducting a program evaluation in a developing country. In many developing countries the major sponsors of evaluation are donor agencies from the developed world, and these agencies require regular evaluation reports in order to maintain accountability and control of resources, as well as generate evidence for the
Culture is defined by Ebbutt (1998, p. 416) as a
Language also plays an important part in the evaluation process, as language is tied closely to culture.<ref name="Bulmer & Warwick, 1993" /> Language can be a major barrier to communicating concepts which the evaluator is trying to access, and translation is often required.<ref name="Ebbutt, 1998" /> There are a multitude of problems with translation, including the loss of meaning as well as the exaggeration or enhancement of meaning by translators.<ref name="Ebbutt, 1998" /> For example, terms which are contextually specific may not translate into another language with the same weight or meaning. In particular, data collection instruments need to take meaning into account as the subject matter may not be considered sensitive in a particular context might prove to be sensitive in the context in which the evaluation is taking place.<ref name="Bulmer & Warwick, 1993" /> Thus, evaluators need to take into account two important concepts when administering data collection tools: lexical equivalence and conceptual equivalence.<ref name="Bulmer & Warwick, 1993" /> Lexical equivalence asks the question: how does one phrase a question in two languages using the same words? This is a difficult task to accomplish, and uses of techniques such as back-translation may aid the evaluator but may not result in perfect transference of meaning.<ref name="Bulmer & Warwick, 1993" /> This leads to the next point, conceptual equivalence. It is not a common occurrence for concepts to transfer unambiguously from one culture to another.<ref name="Bulmer & Warwick, 1993" /> Data collection instruments which have not undergone adequate testing and piloting may therefore render results which are not useful as the concepts which are measured by the instrument may have taken on a different meaning and thus rendered the instrument unreliable and invalid.<ref name="Bulmer & Warwick, 1993" />
Line 244 ⟶ 270:
===Positivist===
Potter (2006)<ref>Potter, C. (2006). Program Evaluation. In M. Terre Blanche, K. Durrheim & D. Painter (Eds.), ''Research in practice: Applied methods for the social sciences'' (2nd ed.) (pp. 410-428). Cape Town: UCT Press.</ref> identifies and describes three broad paradigms within program evaluation . The first, and probably most common, is the [[positivist]] approach, in which evaluation can only occur where there are
A detailed example of the positivist approach is a study conducted by the Public Policy Institute of California report titled "Evaluating Academic Programs in California's Community Colleges", in which the evaluators examine measurable activities (i.e. enrollment data) and conduct quantitive assessments like factor analysis.<ref>{{cite web|last=Gill|first=Andrew|title=Evaluating Academic Programs in California's Community Colleges|url=http://www.ppic.org/main/publication.asp?i=322|publisher=PPIC}}</ref>
===Interpretive===
The second paradigm identified by Potter (2006) is that of interpretive approaches, where it is argued that it is essential that the evaluator develops an understanding of the perspective, experiences and expectations of all stakeholders. This would lead to a better understanding of the various meanings and needs held by stakeholders, which is crucial before one is able to make judgments about the merit or value of a program. The
A report commissioned by the World Bank details 8 approaches in which qualitative and quantitative methods can be integrated and perhaps yield insights not achievable through only one method.<ref>{{cite web|last=Woolcock|first=Michael|title=Integrating Qualitative and Quantitative Approaches in Program Evaluation|url=http://siteresources.worldbank.org/SOCIALANALYSIS/1104890-1120158274352/20566665/Integratingqualitativeandquantapproachesraoandwoolcock.pdf|
===Critical-emancipatory===
Potter (2006) also identifies critical-emancipatory approaches to program evaluation, which are largely based on [[action research]] for the purposes of social transformation. This type of approach is much more ideological and often includes a greater degree of social activism on the part of the evaluator. This approach would be appropriate for qualitative and participative evaluations. Because of its critical focus on societal power structures and its emphasis on participation and empowerment, Potter argues this type of evaluation can be particularly useful in developing countries.
Despite the paradigm which is used in any program evaluation, whether it be positivist, interpretive or critical-emancipatory, it is essential to acknowledge that evaluation takes place in specific socio-political contexts. Evaluation does not exist in a vacuum and all evaluations, whether they are aware of it or not, are influenced by socio-political factors. It is important to recognize the evaluations and the findings which result from this kind of evaluation process can be used in favour or against particular ideological, social and political agendas (Weiss, 1999).<ref>[[Carol Weiss|Weiss, C.H.]] (1999). Research-policy linkages: How much influence does social science research have? World Social Science Report, pp. 194-205.</ref> This is especially true in an age when resources are limited and there is competition between organizations for certain projects to be prioritised over others (Louw, 1999).<ref>Louw, J. (1999). Improving practice through evaluation. In D. Donald, A. Dawes & J. Louw (Eds.), ''Addressing childhood adversity'' (pp. 60-73). Cape Town: David Philip.</ref>
==Empowerment evaluation==
{{howto|date=May 2020}}
{{Main|Empowerment evaluation}}
Empowerment evaluation makes use of evaluation concepts, techniques, and findings to foster improvement and self-determination of a particular program aimed at a specific target population/program participants.<ref name="F.D.M">{{cite journal | last1 = Fetterman
According to Fetterman (2002)<ref name="F.D.M"/> empowerment evaluation has three steps;
Line 268 ⟶ 294:
===Establishing a mission===
The first step involves evaluators asking the program participants and staff members (of the program) to define the mission of the program. Evaluators may opt to carry this step out by bringing such parties together and asking them to generate and discuss the mission of the program. The logic behind this approach is to show each party that there may be divergent views of what the program mission actually is.
===Taking stock===
Taking stock as the second step consists of two important tasks. The first task is concerned with program participants and program staff generating a list of current key activities that are crucial to the functioning of the program. The second task is concerned with rating the identified key activities, also known as ''prioritization''. For example, each party member may be asked to rate each key activity on a scale from 1 to 10, where 10 is the most important and 1 the least important. The role of the evaluator during this task is to facilitate interactive discussion amongst members in an attempt to establish some baseline of shared meaning and understanding pertaining to the key activities. In addition, relevant documentation (such as financial reports and curriculum information) may be brought into the discussion when considering some of the key activities.
===Planning for the future===
After prioritizing the key activities the next step is to plan for the future. Here the evaluator asks program participants and program staff how they would like to improve the program in relation to the key activities listed. The objective is to create a thread of coherence whereby the mission generated (step 1) guides the stock take (step 2) which forms the basis for the plans for the future (step 3). Thus, in planning for the future specific goals are aligned with relevant key activities. In addition to this it is also important for program participants and program staff to identify possible forms of evidence (measurable indicators) which can be used to monitor progress towards specific goals. Goals must be related to the program's activities, talents, resources and scope of capability- in short the goals formulated must be realistic.
These three steps of empowerment evaluation produce the potential for a program to run more effectively and more in touch with the needs of the target population. Empowerment evaluation as a process which is facilitated by a skilled evaluator equips as well as empowers participants by providing them with a 'new' way of critically thinking and reflecting on programs. Furthermore, it empowers program participants and staff to recognize their own capacity to bring about program change through collective action.<ref name="F.D.M2">Fetterman, D.M. (2005). Empowerment evaluation principles in action: Assessing levels of commitment. In D.M. Fetterman & A. Vandersman (Eds.), ''Empowerment evaluation principles in practice''(pp.27-41). New York: Guilford Press</ref>
==Transformative
The transformative paradigm is integral in incorporating social justice in evaluation. Donna Mertens, primary researcher in this field, states that the transformative paradigm,
Both the [[American Evaluation Association]] and [[National Association of Social Workers]] call attention to the ethical duty to possess [[cultural competence]] when conducting evaluations. Cultural competence in evaluation can be broadly defined as a systemic, response inquiry that is actively cognizant, understanding, and appreciative of the cultural context in which the evaluation takes place; that frames and articulates epistemology of the evaluation endeavor; that employs culturally and contextually appropriate methodology; and that uses stakeholder-generated, interpretive means to arrive at the results and further use of the findings.<ref name=SenGupta>{{cite book|last=SenGupta|first=S., Hopson, R., & Thompson-Robinson, M.|title=Cultural competence in evaluation:an overview. New Directions in Evaluation, 102.|year=2004|pages=5–19}}</ref> Many health and evaluation leaders are careful to point out that cultural competence cannot be determined by a simple checklist, but rather it is an attribute that develops over time. The root of cultural competency in evaluation is a genuine respect for communities being studied and openness to seek depth in understanding different cultural contexts, practices and paradigms of thinking. This includes being creative and flexible to capture different cultural contexts, and heightened awareness of power differentials that exist in an evaluation context. Important skills include: ability to build rapport across difference, gain the trust of the community members, and self-reflect and recognize
===Paradigms===
Line 290 ⟶ 313:
====Axiology (Values and Value Judgements)====
The transformative
* The importance of being culturally respectful
* The promotion of social justice
Line 297 ⟶ 320:
====Ontology (Reality)====
Differences in perspectives on what is real are determined by diverse values and life experiences. In turn these values and life experiences are often associated with differences in access to privilege, based on such characteristics as disability, gender, sexual identity, religion, race/ethnicity, national origins, political party, income level,
====Epistemology (Knowledge)====
Knowledge is constructed within the context of power and privilege with consequences attached to which version of knowledge is given privilege.<ref name="Mertens (2012)" />
====Methodology (Systematic Inquiry)====
Line 306 ⟶ 329:
===Lenses===
While operating through social justice, it is imperative to be able to view the world through the lens of those who experience injustices. Critical Race Theory, Feminist Theory, and Queer/LGBTQ Theory are frameworks for how we think others should think about providing justice for marginalized groups. These lenses create opportunity to make each theory priority in addressing inequality.
====Critical Race Theory====
[[Critical Race Theory]](CRT)is an extension of critical theory that is focused in inequities based on race and ethnicity. Daniel Solorzano describes the role of CRT as providing a framework to investigate and make visible those systemic aspects of society that allow the discriminatory and oppressive status quo of racism to continue.<ref name=Solorzano>{{cite journal|last=Solorzano|first=D.|title=Images and Words that wound: Critical race theory, racial stereotyping and teacher education.|journal=Teacher Education Quarterly|year=1997|volume=24|pages=5–19}}</ref>
====Feminist
The essence of [[feminist theories]] is to
====Queer/LGBTQ
[[Queer theory|Queer/LGBTQ theorists]] question the heterosexist bias that pervades society in terms of power over and discrimination toward sexual orientation minorities. Because of the sensitivity of issues surrounding LGBTQ status, evaluators need to be aware of safe ways to protect such individuals’ identities and ensure that discriminatory practices are brought to light in order to bring about a more just society.<ref name="Mertens (2012)" />
==Government requirements==
[[File:CDC Evaluation framework.gif|alt=Framework for program evaluation in public health|thumb|Framework for program evaluation in public health]]
Given the Federal budget deficit, the Obama Administration moved to apply an "evidence-based approach" to government spending, including rigorous methods of program evaluation. The President's 2011 Budget earmarked funding for 19 government program evaluations for agencies such as the Department of Education and the United States Agency for International Development (USAID). An inter-agency group delivers the goal of increasing transparency and accountability by creating effective evaluation networks and drawing on best practices.<ref>{{cite web|title=Program Evaluation|url=https://obamawhitehouse.archives.gov/sites/default/files/omb/performance/chapter8-2012.pdf|via=[[NARA|National Archives]]|work=[[Office of Management and Budget]]|access-date=20 September 2011}}</ref>
A six-step framework for conducting evaluation of public health programs, published by the [[Centers for Disease Control and Prevention]] (CDC), initially increased the emphasis on program evaluation of government programs in the US. The framework is as follows:
# Engage [[Project stakeholder|stakeholders]]
Line 326 ⟶ 350:
# Justify conclusions.
# Ensure use and share lessons learned.
In January 2019, the [[Foundations for Evidence-Based Policymaking Act]] introduced new requirements for federal agencies, such as naming a Chief Evaluation Officer. Guidance published by the [[Office of Management and Budget]] on implementing this law requires agencies to develop a multi-year learning agenda, which has specific questions the agency wants to answer to improve strategic and operational outcomes.<ref>{{Cite web|url=https://trumpwhitehouse.archives.gov/wp-content/uploads/2019/07/M-19-23.pdf|title=Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance|date=2019-07-10|website=Office of Management and Budget}}</ref> Agencies must also complete an annual evaluation plan summarizing the specific evaluations the agency plans to undertake to address the questions in the learning agenda.
==Types of evaluation==
There are many different approaches to program evaluation. Each serves a different purpose.
* [[Utilization-Focused Evaluation]]
* CIPP Model of evaluation
* Formative Evaluation
* Summative Evaluation
* Developmental Evaluation
* Principles-Focused Evaluation
* [[Theory-Driven Evaluation]]
* Realist-Driven Evaluation
==CIPP Model of evaluation==
Line 331 ⟶ 368:
===History of the CIPP model===
The CIPP model of evaluation was developed by Daniel Stufflebeam and colleagues in the 1960s.CIPP is an acronym for Context, Input, Process and Product. CIPP is an evaluation model that requires the evaluation of '''context''', '''input''', '''process''' and '''product''' in judging a
===CIPP model===
The CIPP framework was developed as a means of linking evaluation with programme [[decision-making]]. It aims to provide an analytic and rational basis for programme decision-making, based on a cycle of planning, structuring, implementing and reviewing and revising decisions, each examined through a different aspect of evaluation –context, input, process and product evaluation.<ref name = Robinson/>
The CIPP model is an attempt to make evaluation directly relevant to the needs of decision-makers during the phases and activities of a programme.<ref name = Robinson/>
===Four aspects of CIPP evaluation===
Line 350 ⟶ 387:
===Using CIPP in the different stages of the evaluation===
CIPP model allows
*'''Context''': What needs to be done? Vs. Were important needs addressed?
*'''Input''': How should it be done? Vs. Was a defensible design employed?
Line 371 ⟶ 408:
{{refbegin|30em}}
* Borland, J., & Tseng, Y. P. (2011). A Primer on Doing Evaluation of Social Programs. Parity, 24(7), 8.
* Boulmetis, John and Dutwin, Phyllis. ''The ABCs of Evaluation'' (2005)
* Cook, Thomas D. and Campbell, Donald T. ''Quasi-Experimentation: Design & Analysis for Field Settings'' (1979)
* Khaidukov, Danil; Tasalov, Kirill; Schepetina, Ekaterina; Chueva, Ekaterina (2016). Improvement of methodological approaches to enhancing the efficiency of state programs of the Russian Federation // Lomonosov Moscow State University Science Conference «Effective management», Poligraf Service, Moscow, pp. 65–73
* Preskill, Hallie, Parkhurst, Marcie, and Juster, Jennifer Splansky. "Guide to Evaluating Collective Impact" (2014)
* [https://web.archive.org/web/20150528212346/http://www.acf.hhs.gov/sites/default/files/opre/program_managers_guide_to_eval2010.pdf The Program Manager's Guide to Evaluation, Second Edition], Office of Planning, Research & Evaluation, Administration for Children and Families.* Rivlin, Alice M. ''Systematic Thinking for Social Action'' (1971)
* Rossi, Peter H., Freeman, Howard A. and Lipsey, Mark W.. ''Evaluation. A Systematic Approach'' (1999)
* Suchman, Edward A. ''Evaluative Research: Principles and Practice in Public Service & Social Action Programs'' (1967)
* [[Carol Weiss|Weiss, Carol H.]] ''Evaluative Research: Methods of Assessing Program Effectiveness'' (1972)
* {{cite book |vauthors=Wholey Joseph S, Hatry Harry P, Newcomer Kathryn E, etal |year=2010 |title=Handbook of Practical Program Evaluation Third Edition |___location=San Francisco |publisher=Jossey-Bass |isbn=978-0-470-52247-9 }}
{{refend}}
==External links==
* [https://web.archive.org/web/20090117185946/http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/pmguide_toc.html Administration for Children and Families] The Program Manager's Guide to Evaluation. Discussion of evaluation, includes chapters on Why evaluate, What is evaluation.
* [http://www.betterevaluation.org/ BetterEvaluation] BetterEvaluation: Sharing information to improve evaluation
* [http://www.
* [http://www.evaluationcanada.ca/ Canadian Evaluation Society] Includes a link to [https://web.archive.org/web/20060417183901/http://evaluationcanada.ca/site.cgi?section=1&ssection=1&_lang=an Evaluation information] such as services, professional development, resources, organizations, regional chapters
* [https://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm CDC six-step framework]. Also available here [https://www.cdc.gov/eval/framework.htm Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11)]. Includes a description of logic models in the Steps section.
* [https://books.google.com/books?id=sgoHv5ZP6dcC&dq=self+selection++experimental+design&pg=PA82 Handbook of Research Design & Social Measurement]. Delbert Charles Miller, Neil J. Salkind (2002) Edition: 6, revised. Published by SAGE.
* [http://www.evalpartners.org/ EvalPartners (EvalYouth)] International Evaluation Network and Training tools
*[https://web.archive.org/web/20070330104852/http://www.evaluationwiki.org/ The EvaluationWiki] - The mission of EvaluationWiki is to make freely available a compendium of up-to-date information and resources to everyone involved in the science and practice of evaluation. The EvaluationWiki is presented by the non-profit [https://web.archive.org/web/20070311150127/http://www.evaluationwiki.org/wiki/index.php/Evaluation_Resource_Institute Evaluation Resource Institute].
* [https://sites.google.com/site/gsocialchange/ Free Resources for Program Evaluation and Social Research Methods] This is a gateway to resources on program evaluation, how to, online guides, manuals, books on methods of evaluation and free software related to evaluation.
* [http://www.innonet.org Innovation Network] A nonprofit organization working to share planning and evaluation tools and know-how. The organization provides online tools, consulting, and training, for nonprofits and funders.
* [https://web.archive.org/web/20060222134254/http://www.education.purdue.edu/AssessmentCouncil/Links/Index.htm Links to Assessment and Evaluation Resources] List of links to resources on several topics, including: centers, community building, education and training in evaluation; Foundations; Indiana government & organizations; Links collected by...; Logic models; performance assessment & electronic portfolios, political & private groups or companies, professional assns, orgs & pubs, Purdue University, United States Government, web searches for publications by author & topic, and Vivisimo topical meta searches.
* [http://www.maine.gov/legis/opega/ Maine Legislature's Office of Program Evaluation & Government Accountability] An excellent example of a governmental Program Evaluation office with links to several detailed reports which include methodology, evaluations results, recommendations and action plans.
* [https://web.archive.org/web/20070927210340/http://www.ncsl.org/nlpes/ National Legislative Program Evaluation Society] Includes links to state offices of program evaluation and/or performance auditing in the US.
* [http://www.uwex.edu/ces/4h/evaluation/Evaluating 4-H Youth Development Programs]{{Dead link|date=November 2018 |bot=InternetArchiveBot |fix-attempted=yes }} This is a teaching website and provides help wishing to evaluate their programs.
* [http://www.urban.org/sites/default/files/alfresco/publication-pdfs/310776-Key-Steps-in-Outcome-Management.PDF Program evaluation and Outcome management - The Urban Institute]
* [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3060080 What is Program Evaluation: a Beginner's Guide] Gene Shackman
{{DEFAULTSORT:Program Evaluation}}
[[Category:Impact assessment]]
[[Category:
[[Category:Public policy research]]
| |||