Key messages:
- Evaluation is the last stage of the political process, but it is also the moment when the cycle restarts with new demands, based on the results achieved;
- It is the least adopted phase, due to the difficulties inherent to the evaluation process. And it is also the phase where society's participation is lowest, but still possible;
- Examples of society's participation in the evaluation stage are: involvement in research projects and data generation, to help elucidate issues related to a certain policy; partnerships with the government or use of governmental research resources in preparing evaluation studies; support in disseminating studies conducted by the government;
- The most common evaluation at the end of the cycle is the impact evaluation, which seeks to measure the effectiveness of a policy;
- Ideally, the evaluation process of a public policy should be incorporated into governmental practice so that decisions are based on evidence. In this sense, a successful example is the case of Espírito Santo, which has very clear governance for policy evaluation, with well-established guidelines, guides, and recommendations.
The final stage of the policy cycle covered in this series of five articles is evaluation. It represents the moment when the policy cycle ends, but also begins again based on the results of this process. Generally speaking, it is the least adopted phase, due to the difficulties inherent to the evaluation process.
What is evaluation and how does it fit into the policy cycle?
Evaluation can be understood as a process of analyzing the performance of a particular political process, for example, a program or a policy. Ultimately, the goal is to evaluate whether the policy produced the desired results in the population, according to the objectives set out in its formulation. Although evaluation and research are distinct things, the fact that evaluations use research methods to analyze the effectiveness of a policy ends up associating these two terms. For this reason, although research is frequently associated with the evaluation process, it can be used throughout the cycle.
The figure below illustrates the dynamics of the use of research in the policy cycle, based on the results obtained from the evaluations. As the policy cycle is an ongoing process, the evaluation phase does not represent the end, as stated earlier. Conversely, the results obtained can trigger the start of a new cycle, new questions, the definition of new issues on the governmental agenda, and the need for policy formulation - to solve potential problems encountered or adjust the policy to new needs.
The most common evaluation at the end of the cycle is the impact evaluation, which seeks to measure the effectiveness of the policy. However, evaluations can be done throughout the entire policy process. In this case, different types of analyses are used. For each of them, there are several guiding questions of the evaluative process, and the use of distinct research methods, which will not be discussed here as they fall outside the scope of this article.
Let's use the Mais Médicos Program in Brazil as an example, which was created with the objective of increasing the provision of medical services in regions where there is a shortage or absence of these professionals, in addition to fostering the training of medical specialists in family and community medicine. This program was redesigned based on the identification of a problem - the lack of doctors in primary care in remote locations or with high social vulnerability. Based on the identified need and the policy designed to solve this problem, we would have the following evaluations at each stage:
· Content evaluation: which municipalities need doctors, what is the capacity and need for qualification of professionals, among others.
· Implementation evaluation: how the implementation of the program is being carried out, for example, regarding the hiring of doctors, municipalities' adhesion, doctors' adherence to the program, user satisfaction with the service, etc.
· Impact evaluation: we can divide this evaluation into two stages.
Ø Short term:
o Increase in primary care coverage in municipalities
o Increase in the number of consultations
o Increase in the number of home visits
Ø Medium and long term:
o Improvement in health indicators; for example, high blood pressure and diabetes control.
o Reduction in hospitalizations due to conditions sensitive to outpatient care, among others.
Ideally, the evaluation process of a public policy should be incorporated into governmental practice so that decisions are based on evidence. The success of monitoring and evaluation programs occurs when there is recognition by the government of the importance of using this type of program in its governance, incentives, and a proximity of such systems to political decision-makers.
Some units of the Federation have very clear governance for evaluating policies, with well-established guidelines, guides, and recommendations for evaluation, such as Espírito Santo. In this state, there are partnerships with other actors in society, such as non-governmental organizations and educational institutes, forming a network that works together for the evaluation process. More detailed information about the model adopted in Espírito Santo can be accessed on the website of the Jones dos Santos Neves Institute, linked to the state's Secretariat of Economic Development and dedicated to producing knowledge to subsidize public policies at its various levels.
Evaluation – including a continuous monitoring system for the entire process here – contributes to governance, as the information produced can support decision-making, assist in the development and improvement of policies and programs, aid in activity management, and encourage greater transparency and accountability. Furthermore, it should be oriented towards the stakeholders and involve, whenever possible, the government, civil society, and other important actors in this process.
How can society participate in the evaluation process of a policy?
Despite the advantages and some good examples, the government does not always have an organized structure with the participation of diverse spheres of society. Furthermore, evidence is just another component in decision-making, but it is not the only one, as discussed in the previous articles. Considering this perspective, more recent literature has worked with the issue of social participation and argumentative approach in the logic of evaluation. Generally, social participation is seen as a process more common to the other stages of the cycle, especially agenda setting and policy formulation, but not in the evaluation process.
This new literature assumes that evaluation rarely follows a rational logic, where an independent body performs the entire evaluative process (scoping, needs assessment, data collection and analysis) and presents the results of the policy. In practice, there is a multiplicity of criteria and perspectives on a given issue that should be considered and that contribute to a shared understanding of the various angles that can subsidize an evaluation.
In this approach, participatory political evaluation ideally involves engaging a diversity of actors who affect, or are affected by, a policy or program in the evaluation process. The goal would then be to facilitate dialogue among various actors in society working in advocacy in choosing evaluation criteria. But beyond choosing the analysis criteria, society can contribute to the evaluation and monitoring process of policies in a few ways. I point out some examples from civil society that can serve as inspiration:
· Involvement in research projects and data generation that can contribute to elucidating issues relating to a certain policy.
Ø The Health and Sustainability Institute's mission is to transform knowledge into action to improve life in cities, with a special focus on the issue of air pollution. It acts in the development of its own research in conjunction with academic institutions, and acts in the dissemination of these studies through various media (print media, television, among others) and in the use of this information to subsidize advocacy actions, especially for changes in legislation regarding air quality.
Ø The creation of pathology observatories represents an important instrument for patient associations to monitor and evaluate policies, in addition to conducting studies that can be used as advocacy instruments in other phases of the policy cycle. Here we can cite the Oncology and Multiple Sclerosis Observatory, created by the Brazilian Lymphoma and Leukemia Association (Abrale) and Friends with Multiple Sclerosis (AME), respectively.
· Partnerships with the government or the use of governmental research resources in preparing evaluation studies or that can subsidize the evaluation.
Ø Many institutions work directly with the government or via research grants for program evaluation, such as the Ayrton Senna Institute (in partnership with the Espírito Santo State Government and Insper), or the Maria Cecília Souto Vidigal Foundation, which works with Fapesp grant funds, for example, for program evaluation, and generation of scientific knowledge that helps to influence policies in Brazil and "promote a national research agenda that addresses under-explored areas."
Ø The Abrale Oncology Observatory, created independently of the government in 2015, ended up becoming a reference for data on oncology for the government itself, being cited on the federal government's Brazilian Open Data Portal as a source of information on cancer.
· Support in disseminating studies carried out by the government as support in advocacy actions to improve the living conditions of the population served.
Ø Civil society groups can use documents and reports produced by the government itself and its auditing bodies to disseminate information and foster discussions that can contribute to improvements in public policies. Here we can cite the example of the Brazilian Hemophilia Federation (FBH) which, in 2009, held a public hearing at the Senate Human Rights Commission to discuss changes in the legislation on the purchase of medicines. In this hearing, FBH used data from an operational audit report conducted by the Federal Court of Accounts (TCU) in 2008 on the processes of purchasing, deciding, and executing the budget for acquiring coagulation factors. This audit was prompted by the shortage of the medication used for treating patients with hemophilia A in 2008. The advocacy actions originating from this report (such as holding the public hearing in the senate) contributed to the creation of a specific budget for hemophilia starting in 2011, ensuring prophylactic treatment for all patients.
Ø Another example of advocacy action arising from the use of documents prepared by the government itself can be given for the case of oncology. In 2011, the publication of an operational audit also carried out by the TCU on the Oncology Care Policy in Brazil found that waiting times for cancer diagnosis and treatment were very high, which contributed to the high mortality of patients. The publication of this report boosted the work of patient associations such as the Brazilian Federation of Philanthropic Institutions to Support Breast Health (Femama), as the support and mobilization of its affiliates, to the creation of a bill (the 60-Day Law, mentioned in the article on implementation in this series) in conjunction with parliamentarians to reduce the times between cancer diagnosis and treatment.
Civil society can also request the public authorities to conduct policy evaluations, or work in partnership with academic institutions with suggestions for possible evaluations, based on the problems faced by their audiences and the provision of data that can assist the analysis of programs and policies. For this, it is important that they are capable of generating data and information about their audiences.
There is a diversity of forms of social participation in all phases of the policy cycle, as seen throughout the five articles of this series. It is up to groups in society to understand the functioning of the political process, its actors, and what the objective to be achieved is. With this, it becomes easier to identify where the issues of interest lie in the policy cycle, what types of strategies can be used in each context, what must be done to contribute to a fairer, more egalitarian, and more participatory society.
