Citation: Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen J-A, et al. (2012) Guidance for Evidence-Informed Policies about Health Systems: Rationale for and Challenges of Guidance Development. PLoS Med 9(3): e1001185. doi:10.1371/journal.pmed.1001185
Published: March 6, 2012
Copyright: © 2012 Bosch-Capblanch et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The development of the Handbook was supported by a grant to the WHO by the Rockefeller Foundation, which had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The paper represents the views of the authors and neither WHO nor the Rockefeller Foundation.
Competing interests: The Swiss Tropical and Public Health Institute and the Norwegian Knowledge Centre for the Health Services received funds from the WHO for the contributions of DD, LB, SL, and XBC to developing a Handbook to produce health systems guidance, and some of this work is reported in this article. All other authors declare no competing interests.
Abbreviations: AGREE, Appraisal of Guidelines Research and Evaluation; LMIC, low- and middle-income country; MDG, Millennium Development Goal; WHO, World Health Organization
Provenance: Not commissioned; externally peer reviewed.
- Weak health systems hinder the implementation of effective interventions; policies to strengthen such systems need to draw on the best available evidence.
- Health systems evidence is best delivered in the form of guidance embedded in policy formulation processes, but health systems guidance is poorly developed at present.
- The translation of research on problems, interventions, and implementation into decisions and policies that affect how systems are organised is one challenge facing the development of health systems guidance.
- The development of guidance that is timely and usable by the broad range of health systems stakeholders, and of methods to appraise the quality of health systems guidance, are additional challenges.
- Further research is needed to adapt existing approaches (e.g., those used in clinical guidelines) to produce meaningful advice that accounts for the complexity of health systems, political systems, and contexts.
- This is the first paper in a three-part series in PLoS Medicine on health systems guidance.
Present trends suggest that many of the poorest countries in the world, including many in sub-Saharan Africa, will not meet the health-related Millennium Development Goals  (MDGs), especially MDG 4 (reducing under-five mortality) and MDG 5 (reducing maternal mortality) . Even in those countries that are on track to meet health MDGs, striking inequities exist among countries and among socioeconomic groups within them , despite effective and cost-effective interventions being available to improve population health, including that of vulnerable groups . Such interventions are delivered through health systems, which consist of “all organisations, people and actions whose primary intent is to promote, restore or maintain health” , but, in many settings, interactions between weakened health systems and the sometimes conflicting demands of single-disease intervention programmes are hindering the uptake and implementation of life-saving interventions –. A growing number of governments, international institutions, and funding agencies have therefore recognised the urgent need to coordinate and harmonise investments in health systems strengthening in low- and middle-income countries (LMICs) to provide universal social protection and effective coverage of essential health interventions .
Investments in health systems aim to “enhance [their] performance…for meeting the needs of patients and populations in an equitable and efficient manner”  while reducing the risk of impoverishment due to the costs of care . However, although a number of broad principles have been proposed , there is no wide agreement on the operational definition of health systems strengthening , and it remains unclear how health systems can best be strengthened. Because the evidence base addressing this issue is patchy , health systems research has recently been identified as a priority  and its definition and scope have been outlined . Indeed, the need for greater capacity to produce evidence to inform health systems strengthening was one of the drivers that led to the first global symposium on health systems research (Montreux, Switzerland, November 2010) , at which some of the issues developed in this article were presented and discussed.
Importantly, to be useful to policy makers, research evidence needs to be retrieved, its quality appraised, and the recommended options properly framed in the form of guidance. In an analogy with clinical practice guidelines (“systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances” ), we define health systems guidance as systematically developed statements produced at global or national levels to assist decisions about appropriate options for addressing a health systems challenge in a range of settings and to assist with the implementation of these options and their monitoring and evaluation (Box 1). We use the term “guidance” rather than “guidelines”, as health systems and the evidence on health systems are highly context sensitive. Health systems guidance statements refer to policy options that are accompanied by assessments of the quality of evidence supporting them and the potential for unintended harms, and by discussions of implementation and contextual issues.
Box 1. Health Systems Guidance and Knowledge Translation
Evidence-informed health systems guidance tackles health systems problems by:
- Framing health systems problems;
- Systematically retrieving, translating, and packaging the best available evidence on health systems interventions and implementation issues;
- Using this evidence to recommend and formulate—in a deliberative process—options to solve these problems and to inform policy-making, the level of decision-making where different courses of action are considered;
- Providing insights on the strategies that can be followed in order to implement and evaluate a given health systems policy.
Guidance needs to be transparent, systematic, and adapted to the local contexts. It therefore needs to use validated approaches, to consider all the available evidence and to assess its quality. It also needs to take into account local factors that may influence the effects of all options recommended and to address their feasibility.
Several global institutions have legitimate roles in producing guidance, often in response to requests from national decision makers (see the second paper in this series ). Thus, to avoid unnecessary duplication of the enormous efforts of guidance development, coordination is needed at the global level. However, as decisions on the options recommended are taken at national level, global guidance needs adaptation, ideally through national deliberative processes.
The production of guidance is largely based on knowledge translation approaches that bridge the gap between research evidence and its application to policy-making ,. Figure 1 schematically represents these approaches across the research, policy, managerial, and societal domains.
The need for evidence-informed guidance on policies that impact health systems performance is widely accepted  and is one of the six priorities of the director-general of the World Health Organization (WHO) . Moreover, a World Health Assembly resolution  recently urged member states to use evidence-based approaches to assess “country's health and health systems challenges” and to develop “evidence-based responses to evolving challenges and opportunities, and to involve all relevant stakeholders”. However, although well-established methods exist to develop clinical guidelines , there is little experience in developing health systems guidance and the process poses conceptual and methodological challenges related to the different types of evidence to be considered, the complexity of health systems and the pre-eminence of contextual issues. The current experience is mainly related to the development of different by-products of research syntheses and decision aids  targeted towards policy makers, rather than systematically and transparently developed guidance.
- Assess to what extent the need for health systems guidance is part of national policies and plans and how guidance is currently formulated by analyzing strategic health sector documents from LMICs;
- Describe the methodological challenges in outlining the approaches to produce health systems guidance and to suggest ways to address these challenges.
The second article in this series explores the challenge of linking guidance development and policy development at global and national levels and examines the range of factors that can influence policy development , and the third article explores the challenge of assessing how much confidence to place in evidence on health systems interventions .
The Need for Health Systems Guidance in LMICs
To assess the need for guidance and how it is formulated, we scrutinised the use of guidance-related terms in national health policy and strategic documents from LMICs (Box 2). We found that the terms “guidance” or “guidelines” frequently appeared in strategic documents but were more often related to clinical matters than to health systems (Box 3).
Box 2. Assessing Demand for Health Systems Guidance: Methods
To assess the need of guidance and how it is formulated:
- Two authors examined all available documents written in English and French in the Country Planning Cycle Database , which has gathered documents that describe national health policies or strategies since 2005, for the following LMICs: Afghanistan, Bangladesh, Benin, Burkina Faso, Burundi, Cambodia, Central African Republic, Chad, Comoros, Congo DR, Eritrea, Ethiopia, The Gambia, Ghana, Guinea, Guinea Bissau, Haiti, Kenya, Korea DPR, Kyrgyzstan, Lao People's Democratic Republic, Liberia, Madagascar, Malawi, Mali, Mauritania, Myanmar, Nepal, Niger, Rwanda, Sierra Leone, Solomon Islands, Somalia, Tanzania, Togo, Uganda, Zambia, and Zimbabwe.
- The documents were searched using the terms “guideline(s)”, “guidance”, “guide(s)”, “recommendation(s)”, and their French equivalents and sentences or paragraphs where the terms appeared were extracted.
- One of the authors classified the statements extracted from the documents according to the area of interest (e.g., clinical, public health, health systems), health system component, the purpose of guidance (e.g., for setting standards, to guide policy decisions), the decisional scope of guidance (e.g., national, sub-national), guidance developers and sources (e.g., Ministries of Health, donor, United Nations agencies), guidance production, and the topics of guidance according to the WHO Health Systems Framework  (SI1).
- Of the 195 documents retrieved from the database, 157 dealing with specific programmes or strategies were excluded.
- Of the 661 statements retrieved in the 38 remaining documents, 161 were excluded because the term “guidance” was not used in the sense of decision-support (relevant for our analysis), but rather as generic advice or because the contexts where the terms appeared did not fit into any of the areas described above.
Box 3. Assessing the Demand for Guidance: Findings
The scope of guidance could be discerned in 63 statements from the included documents:
- 35 statements (56%) were national (i.e., supported decisions or activities implemented consistently across a country)
- 14 (22%) were sub-national
- 4 (6%) referred to international guidance
- 20 (32%) related to stakeholders, certain types of health facilities, or all levels of care.
The area of guidance could be identified in 407 statements:
- 201 statements (49.3%) referred to clinical issues
- 63 (15%) referred to public health issues
- 143 referred to miscellaneous areas (e.g., laboratory, management).
In 283 statements (either related to clinical, public health, or miscellaneous issues), references to one or more WHO health systems building blocks  included:
- 83 (29%) on governance and leadership (e.g., roles of governing bodies in producing or implementing guidance)
- 20 (7%) on financing (e.g., drug revolving funds)
- 53 (19%) on health workforce (e.g., training)
- 67 (24%) on medical supplies (including traditional medicines)
- 31 (11%) on information systems (e.g., data for measuring performance)
- 29 (10%) on service delivery (e.g., basic health care package)
Documents from eight countries had explicit statements suggesting guidance as a strategy to improve the “quality of care”. Guidance statements related to traditional medicine were found in The Gambia, Liberia, Rwanda, and Zimbabwe.
The main bodies developing guidance were international organisations (e.g., WHO), ministries of health, and special national committees usually linked to the ministries of health. Guidance production and use was mainly referred to in connection with high-level health sector entities, or at the sub-national level in decentralised structures, although similar terms were used in relation to implementation strategies or operational instructions. There were also statements emphasising the need for producing guidance, for making it accessible to users, and for reinforcing adherence. Finally, some documents explicitly related the development of guidance to the concepts of evidence-based health care.
Our search had several limitations. First, we may have missed some relevant documents. Second, statements do not necessarily reflect the status that governments or stakeholders ascribe to guidance. Finally, the terms used to refer to health systems guidance can differ between countries and across languages. Our findings nevertheless show that the term “guidance” appeared frequently in strategic documents. In its “technical” meaning (i.e., systematically developed statements to assist decisions), the term seems to be more often related to the clinical field rather than to health systems. This could be because the concept of formal guidance applied to health systems decisions is not yet well-established among policy makers or because there are only few examples of such guidance to draw from. However, the statements linking guidance in a more generic sense to health systems actions were plentiful in relation to the functions typically ascribed to ministries of health.
Challenges in Outlining the Approaches to Produce Health Systems Guidance
To improve how WHO responds to requests for guidance on health systems and the quality of the guidance produced, WHO recently commissioned the production of a Handbook outlining approaches to develop health systems guidance . A Task Force on Developing Health Systems Guidance (listed in the Acknowledgments section) was formed from WHO expert panels and from experts working in the field of health systems research at international and academic institutions and oversaw the drafting of the Handbook for Developing Health Systems Guidance: Supporting Informed Judgements for Health Systems Policies  (which was based on existing best practices for the development of clinical guidelines, on approaches and tools described in the literature, and on the expertise of the Task Force members) by reviewing and commenting on its content by email and through regular teleconferences and meetings in person. During the production of the Handbook, because evidence from research (for example, on the effectiveness and acceptability of health systems interventions) is not guidance and is insufficient for optimal decision-making , several conceptual and methodological challenges associated with the production of health systems guidance were identified. The writing group for this paper further considered these issues and produced this manuscript, which was finalised after several iterations of comments by the Task Force and external reviewers. Here, we discuss four specific challenges that were encountered during the production of the Handbook, namely:
- Research on effectiveness is typically articulated around health interventions, but policy decisions often relate to arrangements of the health system, services, and programmes and encompass multiple interventions packaged into a particular policy;
- Users and producers of guidance include a broad range of stakeholders who may not all be familiar with research methods;
- The production of guidance has to be timely in relation to the need and available capacity to implement it and consistent with countries' priorities;
- The quality of guidance needs to be appraised using transparent criteria; guidance then needs to be disseminated and promoted actively to facilitate its uptake.
Translating Research on Problems, Interventions, and Implementation into Decisions on Policies and Services
Many processes, frequently involving iterations from research evidence into policy formulation and from policy evaluation into research prioritisation, are needed to bridge the gaps between research, policy, and practice. The complexity of these processes demands a dynamic framework  that is comprehensive and incorporates current thinking about evidence, policy formulation, and health systems such as the one shown in Figure 1.
Figure 1. A generic knowledge translation framework across the research, policy, managerial, and societal domains.
Vertical rectangles contain the methods or approaches to bridging each phase, the frames at the bottom indicate the products for each phase, and the concepts in between the vertical rectangles represent the different forms of knowledge. Systematic reviews are summarised into a unified body of knowledge that links priority problems with the effects of interventions and implementation strategies. Knowledge summaries support the deliberative process in which stakeholders develop guidance products that, in turn, result in policies for services and programmes arrangements. The outcomes of programmes and services are evaluated to ascertain the extent to which the needs of the population have been met. Evaluation should inform further research, in an iterative process.doi:10.1371/journal.pmed.1001185.g001
Research is articulated around research questions that can be addressed using particular methods and is driven by research opportunities (e.g., funding), researchers' interests, and feasibility. By contrast, policy decisions and the managerial arrangements needed to put them into practice have to be responsive to population needs and integrated into complex health systems that go beyond the mere aggregation of single interventions. For example, the research question of whether lay health workers are effective in delivering specific health care interventions will become fully relevant for policy only when the essential components of the intervention (e.g., training of lay health workers), related actions (e.g., adaptations needed in the distributions of tasks of cadres), implementation issues (e.g., the preferences of potential clients), and the implications across other health system building blocks (e.g., adaptations to the health information sub-system that may be needed to capture the tasks undertaken by such workers) are all considered.
Ideally, implementation and contextual issues should be considered when conducting the systematic reviews of effectiveness that are needed to translate research into health systems guidance or in other types of reviews addressing them ,. Although there are frameworks to analyse barriers to implementation  and the applicability of evidence , many systematic reviews fail to consider issues pertaining to applicability and equity, or the complexity of interventions in relation to their technical feasibility . This is partially due to the lack of this type of information in the primary research that underlies systematic reviews. Where this evidence is missing or scarce, efforts to search, appraise, and synthesise additional evidence on implementation issues will have to be made. It will also be important to consider how to address inequities in coverage of health care , which are pervasive in many countries.
The production of guidance not only entails putting research findings into context, but also entails taking a health systems perspective. New trends in health systems thinking advocate the construction of conceptual pathways that look at the interventions from a systems perspective, making explicit how interventions may trigger reactions in related components of the health system that may produce unintended consequences ,. For example, increasing the salary of health workers in HIV/AIDS treatment programmes may result in a reduction of the health workforce addressing other conditions who do not receive the same salary benefits.
Finally, it is important to note that the study designs for research (and the methods used to synthesise research findings) that explore different health systems components vary. For example, research exploring the governance and leadership components of health systems is likely to involve qualitative methods, whereas research on health care delivery could utilise both qualitative and quantitative methods (see also the third paper in this series ). Qualitative synthesis methods have addressed the nature of problems such as patient adherence to treatment for tuberculosis . Syntheses of mixed method research can address policy-oriented questions using framework synthesis, a method that has been applied to qualitative and mixed methods research addressing research management , public health ,, and workforce management .
Producers and Users of Guidance
Although research evidence to inform guidance is generated through research synthesis (i.e., systematic reviews) and often uses complex statistical methods, guidance is typically produced through a deliberative process where evidence is interpreted and contextualised (Figure 1, Policy domain). During the deliberative process, which can include a wide range of stakeholders, potential users, and beneficiaries of guidance, knowledge is exchanged, filtered (stakeholders decide on the relevance of evidence), and amplified (stakeholders stress evidence consistent with their views). Colloquial (non-research) evidence (e.g., tacit or experiential knowledge) helps to interpret or contextualise research evidence and addresses issues for which research evidence is not available but that may play a role in the decisions (e.g., considerations about the political implications of decisions, or about potential vested interests of stakeholders) . Making colloquial evidence explicit adds transparency to the guidance development process.
Typically, users and producers of guidance, who may include economists, managers, administrators, social scientists and other professional groups as well as elected politicians, come from very different research traditions or none at all and have very different approaches to research and decision-making . Therefore, it is probably unrealistic to expect that they will all be familiar with issues such as assessing the quality or appropriateness of systematic reviews . Furthermore, even in the ideal situation where policy makers are familiar with research methods, they may choose to ignore evidence that creates uncertainty, questions conventional knowledge, ignores local context , or that is not consistent with pre-determined ideas of preferred policy options.
Although evidence on the effects of health systems interventions is relatively scarce, there are renewed efforts to increase availability and accessibility of this evidence ,,. Such evidence should be presented in user-friendly formats (e.g., minimising technicalities such as complex statistical outputs) to increase its accessibility when producing guidance and to facilitate appropriate interpretation by those without a strong research background. Finally, the preferences of different decision makers for how evidence and guidance should be presented should be taken into account.
Timeliness in the Production of Guidance
Timeliness in the production of evidence and guidance is critical, but systematic reviews can take between one and two years to complete depending on the availability of new evidence, and guidance development can take a year or more . The bottom of Figure 1 also shows the potentially long time frames for the policy decision processes and the evaluation of policies. Furthermore, guidance needs to be regularly updated in the light of new evidence or changes and modifications to existing, previously accepted evidence.
Without timely production of research synthesis and guidance, the gap between research, policy, and practice cannot be bridged, even if primary research is available. Alternative approaches and methods are therefore needed to reduce the lengthy time frames needed for producing systematic reviews and for translating the findings of reviews into guidance. Approaches that may be useful include rapid assessment methods , the use of text mining to speed the search for publications , and the adaptation of existing guidance that can be institutionally endorsed by the organisations producing or using it .
Appraisal, Dissemination, and Implementation of Guidance
The quality of guidance has to be assessed to ascertain the extent to which the guidance used state-of-the-art and validated methods during its development and is, therefore, balanced and reliable in relation to the evidence that informs it. The AGREE instrument (Appraisal of Guidelines Research and Evaluation) and its recent revision (AGREE II) is used to assess the quality of clinical guidelines but could be adapted to assess health systems guidance. The AGREE instrument, which consists of 23 items, aims to assess the quality of guidelines, to provide a methodological strategy for the development of guidelines, and to inform guideline developers on what and how information ought to be reported in guidelines . Many of its items are applicable to both clinical guidelines and health systems guidance, but some need minor adaptation for the latter. For example, item 16 in the AGREE II instrument mentions the “different options for the management of the condition or health issue are clearly presented” whereas in health systems guidance this would need to be reworded to indicate “different options for addressing the health system topic are clearly presented”. In addition, there should be specific reference to the need for health systems guidance to refer to the contextual factors that would determine the extent to which research evidence is applicable in specific circumstances.
Passive dissemination of guidance will not ensure its uptake by potential users , particularly if it lacks relevance to local or national situations. There are several proven approaches for the effective dissemination of research evidence and the promotion of clinical guidelines (e.g., distribution of educational materials, educational meetings and outreach visits, involvement of local opinion leaders, audit and feedback) that have moderate effects on outcomes . Some of these approaches may also be relevant to health systems guidance, although the challenges relating to dissemination and uptake for these two types of guidance are different. At a national level, a relatively small number of decision makers may need to be influenced to integrate guidance into national health policies and plans. Alternatively, sub-national and local decision makers may be in a better position to influence guidance implementation, particularly in more decentralised systems. “Policy briefs” and “policy dialogues” are examples of tools used to improve the contextualisation and utilisation of guidance  (discussed further in the second article in this series ). Although uptake of guidance depends on how it is presented and disseminated, additional factors may increase the utilisation of scientific evidence by policy-makers. These include early, informal interactions between researchers and policy makers, the relevance and timeliness of evidence, and the consistency between the evidence and the recommended options on the one hand, and the beliefs, values, interests, or political goals of policy makers on the other .
We found that high-level policy and strategic health sector documents frequently mention the need for the development of health systems guidance but suggest that there may still be limited awareness that health systems guidance can provide key inputs into policy-making and that the development of such guidance requires systematic and transparent approaches, inspired by the development of clinical guidelines. However, given the growing amount of evidence on health systems and the current initiatives to address methodological challenges to produce health systems guidance (e.g., the Handbook for Developing Health Systems Guidance ), the international health community at all levels is likely to be faced with increasing demands for guidance on health systems issues.
The approaches we identified for producing guidance for health systems typically mirror validated methods developed for clinical guidelines but with some important differences. Specifically, to develop health systems guidance that is relevant and useful to decision makers, it is necessary to acknowledge and to find ways of incorporating the complex interrelations of the system components, and the numerous contextual factors that may influence the effectiveness of interventions, particularly their effects on disadvantaged populations. These issues are discussed further in the second paper in this series . The uptake of guidance by decision makers and the additional complexity of the decision-making process itself are also important.
Health systems guidance has the potential to improve decision-making and enable more efficient use of resources with consequent improvements in the health of populations. However, such guidance needs rigorous and transparent processes of production and evidence-based approaches to ensure its dissemination and uptake. These processes and approaches are still at a rudimentary stage of development. Importantly, the availability of health systems guidance should encourage better and more comprehensive health systems research. This, in turn, should foster more cross-disciplinary approaches to studying the dynamic interactions within complex health systems and thus help to develop new methods to effectively translate health systems evidence into usable policy guidance that is relevant to stakeholders at global and national levels, that takes into account social and ethical imperatives, and that recognises the complexity of health systems and political systems within which such guidance is introduced.
Translation of the Summary Points into Spanish by Xavier Bosch-Capblanch
Translation of the Summary Points into French Bruno Clary, William Lenoir, and Lise Beck
Translation of the Summary Points into Portuguese by Bruno Viana
Translation of the Summary Points into Arabic by Fadi El-Jardali
Characteristics of guidance in health policy or strategy documents analysed with examples of quotations
We acknowledge other members of the Task Force on Developing Health Systems Guidance, who include (with their affiliations at the time when the Task Force was initiated): Abdul Ghaffar, Alliance for Health Policy and Systems Research, World Health Organization (Switzerland); Timothy Evans, Assistant Director-General, Information, Evidence and Research, World Health Organization (Switzerland); Davina Ghersi, Department of Research Policy and Cooperation, Information, Evidence and Research, World Health Organization (Switzerland); Metin Gulmezoglu, UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction, World Health Organization (Switzerland); Pierre Ongolo-Zogo, Centre for the Development of Best Practices in Health, Yaoundé Central Hospital, and University of Yaoundé, Yaoundé (Cameroon); Ulysses Panisset, Department of Research Policy and Cooperation, Information, Evidence and Research, World Health Organization (Switzerland); Gunn Vist, Norwegian Knowledge Centre (Norway).
Thanks to Don de Savigny for fruitful exchanges around health systems and guidance.
Conceived and designed the experiments: XBC. Performed the experiments: XBC. Analyzed the data: XBC. Wrote the first draft of the manuscript: XBC. Contributed to the writing of the manuscript: XBC JL SL RA JAR DD LB AB FEJ LG SO KW PT RK TP AH. ICMJE criteria for authorship read and met: XBC JL SL RA JAR DD LB AB FEJ LG SO KW PT RK TP AH. Agree with manuscript results and conclusions: XBC JL SL RA JAR DD LB AB FEJ LG SO KW PT RK TP AH.
- 1. United Nations (2000) Millennium Development Goals (MDG). Ref: United Nations Millennium Declaration. United Nations General Assembly Resolution 55/2. New York: United Nations. Available: . http://www.un.org/millennium/declaration/ares552e.pdf. Accessed 2 February 2012.
- 2. UNDP (2010) Millennium Development Goals: 2010 progress chart. Available: http://unstats.un.org/unsd/mdg/Resources/Static/Products/Progress2010/MDG_Report_2010_Progress_Chart_En.pdf. Accessed 30 January 2012.
- 3. UN (2010) The Millennium Development Goals Report 2010. Available: http://www.un.org/millenniumgoals/pdf/MDG%20Report%202010%20En%20r15%20-low%20res%2020100615%20-.pdf. Accessed 30 January 2012.
- 4. Darmstadt GL, Bhutta ZA, Cousens S, Adam T, Walker N, et al. (2005) Evidence-based, cost-effective interventions: how many newborn babies can we save? Lancet 365: 977–988.
- 5. WHO (2007) Everybody's business: strengthening health systems to improve health outcomes: WHO's framework for action. Geneva: WHO.
- 6. Biesma RG, Brugha R, Harmer A, Walsh A, Spicer N, et al. (2009) The effects of global health initiatives on country health systems: a review of the evidence from HIV/AIDS control. Health Policy Plan 24: 239–252.
- 7. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, et al. (2004) Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet 364: 900–906.
- 8. Bhutta ZA, Chopra M, Axelson H, Berman P, Boerma T, et al. (2010) Countdown to 2015 decade report (2000–10): taking stock of maternal, newborn, and child survival. Lancet 375: 2032–2044.
- 9. Reich MR, Takemi K (2009) G8 and strengthening of health systems: follow-up to the Toyako summit. Lancet 373: 508–515.
- 10. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, et al. (2010) Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 376: 1923–1958.
- 11. WHO (2010) The world health report - health systems financing: the path to universal coverage. Geneva: WHO.
- 12. Swanson RC, Bongiovanni A, Bradley E, Murugan V, Sundewall J, et al. (2010) Toward a consensus on guiding principles for health systems strengthening. PLoS Med 7: e1000385. doi:10.1371/journal.pmed.1000385.
- 13. Sundewall J, Swanson RC, Betigeri A, Sanders D, Collins TE, et al. (2011) Health-systems strengthening: current and future activities (comment). Lancet 377: 1222–1223.
- 14. Lewin S, Lavis JN, Oxman AD, et al. (2008) Supporting the delivery of cost-effective interventions in primary health-care systems in low-income and middle-income countries: an overview of systematic reviews. Lancet 372: 928–939.
- 15. [No authors listed] (2004) The Mexico Statement on Health Research. Knowledge for better health: strengthening health systems. From the ministerial summit on health research; Mexico City; 16–20 November 2004.
- 16. Remme JHF, Adam T, Becerra-Posada F, D'Arcangues C, Devlin M, et al. (2010) Defining research to improve health systems. PLoS Med 7: e1001000. doi:10.1371/journal.pmed.1001000.
- 17. Steering Committee of the First Global Symposium on Health Systems Research (2010) Montreux Statement from the Steering Committee of the First Global Symposium on Health Systems Research. Available: http://www.hsr-symposium.org/index.php/montreux-statement. Accessed 30 January 2012.
- 18. Institute of Medicine (1990) Clinical practice guidelines: directions for a new program. Washington (D.C.): National Academy Press.
- 19. Oxman AD, Fretheim A, Schünemann HJ, SURE (2006) Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst 4: 12.
- 20. World Health Assembly (24 May 2011) Strengthening national policy dialogue to build more robust health policies, strategies and plans. Sixty-fourth World Health Assembly WHA64.8. Agenda item 13.4. Available: http://apps.who.int/gb/ebwha/pdf_files/WHA64/A64_R8-en.pdf. Accessed 30 January 2012.
- 21. Grimshaw JM, Russell IT (1993) Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 342: 1317–1322.
- 22. SUPPPORT (n.d.) Supporting Policy relevant Reviews and Trials. Available: http://www.support-collaboration.org/index.htm. Accessed 30 January 2012.
- 23. Lavis JN, Røttingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, et al. (2012) Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med 9: e1001186. doi:10.1371/journal.pmed.1001186.
- 24. Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis J, Ghersi D, Røttingen JA, Steinmann P, Gulmezoglu M, Tugwell P, El-Jardali F, Haines A (2012) Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med 9: e1001187. doi:10.1371/journal.pmed.1001187.
- 25. Bosch-Capblanch X and Project Team (2011) Handbook for developing health systems guidance: supporting informed judgements for health systems policies. Basel: Swiss Tropical and Public Health Institute. Available: http://www.swisstph.ch/?id=53; http://www.swisstph.ch/fileadmin/user_upload/Pdfs/SCIH/WHOHSG_Handbook_v04.pdf. Accessed 30 January 2012.
- 26. Lomas J, Culyer T, McCutcheon C, McAuley L, Law S (2005) Conceptualising and combining evidence for health systems guidance. Final report. Ottawa: Canadian Health Services Research Foundation.
- 27. Plsek PE, Greenhalgh T (2001) The challenge of complexity in health care. BMJ 323: 625–628.
- 28. Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK (2005) Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Ann Intern Med 142: 1056–1065.
- 29. Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, et al. (2010) Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health. BMJ 341: c4739.
- 30. Fretheim A, Munabi-Babigumira S, Oxman AD, Lavis JN, Lewin L (2009) SUPPORT tools for evidence-informed policymaking in health 6: using research evidence to address how an option will be implemented. Health Res Policy Syst 7(Suppl 1): S6.
- 31. Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, et al. (2009) SUPPORT tools for evidence-informed health policymaking (STP) 9: assessing the applicability of the findings of a systematic review. Health Res Policy Syst 7(Suppl 1): S9.
- 32. Gericke CA, Kurowski C, Ranson MK, Mills A (2005) Intervention complexity—a conceptual framework to inform priority-setting in health. Bull World Health Organ 83: 285–293.
- 33. De Savigny D, Adam T, editors. (2009) Systems thinking for health systems strengthening. Geneva: Alliance for Health Policy and Systems Research.
- 34. Atun R, Menabde N (2008) Health systems and systems thinking. In: Coker R, Atun R, McKee M, editors. Health systems and the challenge of communicable diseases: experiences from Europe and Latin America. Berkshire (United Kingdom): Open University Press, McGraw Hill Education. pp. 121–140.
- 35. Munro SA, Lewin SA, Smith HJ, Engel ME, Fretheim A, et al. (2007) Patient adherence to tuberculosis treatment: a systematic review of qualitative research. PLoS Med 4: e238. doi:10.1371/journal.pmed.0040238.
- 36. Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley A, et al. (2008) A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect 11: 72–84.
- 37. Lorenc T, Brunton G, Oliver S, Oliver K, Oakley A (2008) Attitudes to walking and cycling among children, young people and parents: a systematic review. J Epidemiol Community Health 62: 852–857.
- 38. Carroll C, Booth A, Cooper K (2011) A worked example of “best fit” framework synthesis: A systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol 11: 29.
- 39. Kiwanuka SN, Kinengyere AA, Rutebemberwa E, Nalwadda C, Ssengooba F, et al. (2011) Dual practice regulatory mechanisms in the health sector: a systematic review of approaches and implementation. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. ISBN: 978-1-907345-08-1.
- 40. Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A (2009) SUPPORT tools for evidence-informed health policymaking (STP) 14: organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst 7(Suppl 1): S14.
- 41. Walshe K, Rundall TG (2001) Evidence-based management: from theory to practice in health care. Milbank Q 79: 429–457.
- 42. Lavis JN, Oxman AD, Lewin S, Fretheim A (2009) SUPPORT tools for evidence-informed health policymaking (STP). Health Res Policy Syst 7(Suppl 1): I1.
- 43. Green A, Bennett S, editors. (2007) Sound choices. Enhancing capacity for evidence-informed health policy. WHO-AHPSR.
- 44. McMaster University (n.d.) Health systems evidence. Available: http://www.healthsystemsevidence.org. Accessed 30 January 2012.
- 45. National Library of Medicine (2011) Health services research (HSR) PubMed queries. http://www.nlm.nih.gov/nichsr/hedges/search.html. Accessed 30 January 2012.
- 46. Raine R, Sanderson C, Black N (2005) Developing clinical practice guidelines: a challenge to current methods. BMJ 331: 631–633.
- 47. UK Civil Service (n.d.) Rapid Evidence Assessment Toolkit. Available: http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance. Accessed 30 January 2012.
- 48. Thomas J, McNaught J, Ananiadou S (2011) Applications of text mining within systematic reviews. Research Synthesis Methods 2(1): 1–14.
- 49. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, et al. (2010) AGREE II: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 18: E839–E842.
- 50. Fretheim A, Schünemann HJ, Oxman AD (2006) Improving the use of research evidence in guideline development: 15. Disseminating and implementing guidelines. Health Res Policy Syst 4: 27.
- 51. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, et al. (2004) Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 8: iii–iv, 1–72.
- 52. Lavis JN, Davies H, Oxman A, Denis J-L, Golden-Biddle K, et al. (2005) Towards systematic reviews that inform healthcare management and policy-making. J Health Serv Res Policy 10(supplement 1): 35–48.
- 53. Tricco AC, Tetzlaffa J, Moher D (2010) The art and science of knowledge synthesis. J Clin Epidemiol 64: 11–20.
- 54. International Development Centre (IDRC) (n.d.) What is knowledge translation? Available: http://web.idrc.ca/en/ev-125530-201-1-DO_TOPIC.html. Accessed 30 January 2012.
- 55. International Health Partnership (IHP+) (n.d.) Country Planning Cycle Database: a WHO resource. Available: http://creator.zoho.com/downeyc/countryplanningdb/view-perma/Country_Planning_Cycle_DB. Accessed 30 January 2012.