In addition, we contacted experts in the field and sent requests for papers to two existing global mailing lists for evaluators: MandENEWS and Pelican. Following the search of databases of peer-reviewed journal articles, the titles and abstracts of the search results from peer-reviewed papers were exported into Endnote [ 23 ] where duplicates and irrelevant titles were removed. The peer-reviewed journal articles found through contact with experts were added to this. The titles and abstracts were double screened by EB and LL against the inclusion and exclusion criteria.
Once the abstracts were screened, the full papers or reports of the included abstracts were obtained and assessed for eligibility by both reviewers. Following the grey literature search as described above, all potentially relevant results were saved into Evernote [ 24 ]. These were double screened by both reviewers against the inclusion and exclusion criteria. The data from the papers were extracted by the first author EB onto a data extraction form. This included information on authors, publication dates, the type of interventions and outcomes, the development of ToC, the use of ToC in the design, implementation and evaluation of the intervention and the influence of context.
The data collection form also included key principles of theory-driven evaluation proposed by Coryn et al. These included how the programme theory was a formulated, b used to formulate and prioritise evaluation questions, c plan and conduct evaluations, d inform the measurement of constructs in the programme theory and e provide a causal explanation. Where a paper described or showed a ToC, we assessed what elements of ToC they presented.
- Institutions and the Path to the Modern Economy: Lessons from Medieval Trade (Political Economy of Institutions and Decisions).
- Cybernaut (ISBN 974-830-87-21)!
- Change Models - Kotter!
However, as there is no agreed upon assessment of quality for papers reporting ToC, we did not asses the quality of the included papers. We did not contact authors for additional information. The papers were compared, evaluated and summarised narratively in relation to review questions. Due to the heterogeneity of the study designs, interventions and outcomes included in this review, a meta-analysis was not conducted.
In total, abstracts were screened, resulting in full text peer-reviewed articles which were assessed for eligibility. An additional 65 records were identified from the grey literature search and screened for eligibility. A total of 62 papers were included [ 1 , 12 — 14 , 16 , 26 — 82 ]. The publication dates of the papers range between and , with a steady increase in papers over time Fig.
The majority were published in English in peer-reviewed journals, but we also included PhD theses, presentations and NGO reports from the grey literature. Four pairs of papers are reported on the same public health interventions [ 1 , 13 , 42 , 43 , 54 , 60 , 81 , 82 ]. However, as the primary interest of this paper is how the use of ToC is described in reports and peer-reviewed journal articles, we have included them as separate papers.
These included systems of care for adolescents with behavioural and emotional difficulties [ 12 , 26 , 50 , 53 , 54 , 56 , 61 , 65 , 80 — 82 ], substance use interventions [ 27 , 49 ], domestic violence interventions [ 29 ], comprehensive community initiatives [ 13 , 16 , 35 , 62 , 81 ], medication supply among community health workers [ 40 ] and integrated district level mental healthcare plans in low- and middle-income countries [ 55 ].
Characteristics of studies included in the review and reported aspects of the ToC process. The ToCs were developed using workshops [ 28 , 34 , 47 , 48 , 55 , 63 , 64 , 72 , 76 ] and working groups [ 12 , 53 , 54 , 61 , 68 , 69 , 82 ], document reviews [ 16 , 35 , 44 , 56 , 67 , 71 ], interviews and discussions [ 16 , 27 , 29 , 35 , 40 , 44 , 47 , 56 , 57 , 62 , 65 , 66 , 73 , 80 ], surveys [ 31 , 67 ], programme observation [ 16 , 44 , 45 , 56 , 67 ], literature reviews [ 33 , 40 , 68 , 69 , 80 ] and existing conceptual frameworks or theory [ 33 , 40 , 42 — 44 , 51 , 64 , 68 , 69 ].
The ToC development included consultations or interviews with the following stakeholders: programme staff [ 27 , 38 , 40 , 44 , 45 , 52 , 54 , 57 , 63 , 65 — 67 , 72 , 73 , 82 ], management [ 12 , 57 , 61 , 66 , 70 , 77 , 82 ], families [ 12 , 26 , 54 , 65 , 77 , 82 ], service users [ 39 , 47 , 50 , 61 , 65 ], experts [ 40 , 64 ] and evaluators [ 13 , 14 , 38 , 44 , 52 , 58 , 61 , 70 , 75 , 77 , 81 ]. Many used multiple methods, for example, Mookheriji and Lafond used immunisation programme theory and discussion with programme stakeholders, including immunisation experts, to develop a ToC of routine immunisation performance [ 64 ].
They used a case study approach to evaluate immunisation performance and then refined the ToC based on the results of this evaluation and a stakeholder workshop. In one case, a table was used. Almost all of the ToCs outlined the long-term outcome required, and the majority described the process or sequence of change.
- The Law of Internal Armed Conflict!
- Technical History of the Beginnings of Radar (Radar, Sonar, Navigation and Avionics).
- Symmetries in Science IX;
However, assumptions and indicators were displayed or described infrequently. Components of ToC in the papers where a ToC was displayed or described. Essential and additional components adapted from Vogel [ 8 ]. The majority of these reported that they used the ToC as a framework for the intervention [ 12 , 31 , 42 , 43 , 70 ] or as a basis for a strategic plan [ 61 , 68 , 69 , 76 , 82 ]. Some examples of how ToCs were used to design public health interventions follow. Basson et al. Lund et al. A few presentations and papers reporting the development of systems of care for children with behavioural difficulties used the ToC as an outline of their public health intervention and as a basis for their strategic plan [ 12 , 61 , 65 ].
Chandani et al. This includes the development of indicators, the overall evaluation design and data analysis. The development of indicators used in the ToC was described in 28 papers. The indicators were often developed from the short-, medium- or long-term outcomes described in the ToC [ 27 , 35 , 38 , 58 , 65 , 74 , 81 , 84 ]. Thirty-two Only two papers [ 12 , 82 ] explicitly described the use of ToC to identify indicators for ongoing monitoring of the implementation of the intervention. The majority of papers However, the papers varied in the amount of detail they provided on this process.
A common description was that the ToC was used to provide a framework for the evaluation [ 27 , 32 , 33 , 48 , 64 , 72 , 74 , 78 , 79 , 81 , 82 ]. Others reported that they used the evaluation to develop [ 39 ], refine [ 40 ] or validate the ToC [ 64 ]. Two papers reported that their evaluation was guided by testing the assumptions in the ToC [ 29 , 34 ]. The data collection and analysis methods used varied greatly across papers. Data collected for the evaluation included routinely collected data [ 33 , 44 ], custom-designed surveys [ 13 , 16 , 32 , 72 , 76 , 79 ] and qualitative data.
Qualitative data collection methods included interviews [ 13 , 27 , 35 , 47 , 71 , 73 , 75 , 79 ], programme observation [ 13 , 27 , 35 ], programme documentation [ 13 , 35 , 71 , 75 , 79 ] and visual evidence [ 32 ]. The quantitative data analysis methods were strongly linked to the types of data collected and included descriptive statistics [ 33 ], inferential statistics [ 27 , 40 , 42 , 43 , 74 , 78 ], multilevel modelling [ 16 ] and path analysis [ 41 ].
Other methods included case study approaches [ 16 , 33 , 36 , 64 ] and iterative thematic analysis [ 71 ] whereas others did not explicitly state their specific data analysis approach [ 14 , 77 ]. Few papers explicitly explored the influence of context of the intervention in relation to ToC. Although some ToCs mentioned context, particularly those with a realist evaluation focus, there was little description of how context affected the interpretation of the evaluation.
There were some exceptions [ 40 , 56 , 64 , 72 ]. Mookherji and LaFond used a case study approach to explore what worked within and between immunisation programme contexts to identify common factors influencing immunisation performance in Ghana, Ethiopia and Cameroon [ 64 ]. For example, political and social commitment to routine immunisation was seen as a key factor in influencing immunisation performance although it was described slightly differently for each context.
Similarly, Chandani et al. They compared whether each of the preconditions and the outcome was achieved in each setting [ 40 ].
These differences were then explained based on the contextual factors in each setting such as types of medication provided by the health workers, standard operating procedures and data availability and means of transport and travel times. Secker et al. Few papers reported on the identification of breakdowns and side effects, effectiveness or efficacy and causal explanation as described by Coryn et al.
Looking for other ways to read this?
Only four 6. In this systematic review, we provide an overview of how ToCs have been developed and used to develop and evaluate public health interventions.
As expected, there is variation in how ToCs are developed and used in evaluation although the papers report very little detail about the ToC process. We have shown that the ToC approach has been in use since at least with 62 papers found in peer-reviewed journals and grey literature. This was significantly more than expected, given that Coryn et al. However, Coryn et al.
'Theories of change': An expanding resource list - Research to Action
In this review, many papers provided little detail in relation to the process of ToC development and how the ToC was used to design the intervention or conduct the evaluation. For example, Bonner [ 36 ] describes the ToC approach in detail but provides only a short example of the Health Action Zones experience of using ToC to evaluate an intervention to reduce drug taking.
Brown et al. The only description of ToC was found in the abstract and then mentioned briefly in the discussion. There was no clarity on how the ToC was developed or any explicit mention of how it was used to inform the analysis.