Terms of Reference: Final Evaluation of the Improved Utilisation and Opportunities for Reproductive Health for Women and Youth Project
Nkhotakota District, Malawi
Background / Context
Malawi has one of the highest maternal mortality rates in the world, where one out of every 36 women dies due to pregnancy-related complications. Despite having made significant progress over the last 25 years, the 2015 maternal mortality ratio is 510 / 100,000 live births; significantly higher than the Millennium Development Goal target of 280. While evidence-based, reproductive health interventions to address maternal mortality are being implemented in Malawi, the quality, availability, and accessibility of these services – especially for poor and vulnerable populations – remains a challenge. Adolescent pregnancy is of particular concern in Malawi. By the age of 18, 50% of girls have married, and one quarter has had their first child. Adolescents are estimated to account 20% of maternal mortality in Malawi.
Introduction
With funding from the Scottish Government, Merck for Mothers, and the Elizabeth Taylor AIDS Foundation, Concern Worldwide is implementing a maternal and reproductive health project in Nkhotakota district, Malawi from November 2012 – March 2016. The purpose of the project is to reduce maternal morbidity and mortality through the following strategic objectives:
Improve utilization of high quality maternal and reproductive health services through increased availability and accessibility of services
Improve utilization of high quality community-based family planning services, including youth-friendly reproductive health, through increased availability and accessibility of services
Improve capacity of government and traditional leadership structures to plan, manage, support, and monitor key maternal and reproductive health activities
A total of 33,515 women of reproductive age are targeted to benefit from a full package of facility and community-based interventions in three of seven Traditional Areas (TAs) in the district: Malengachanzi, Mwadzama, and Mwansambo. An additional 34,378 women will benefit from improved health services throughout all seven TAs in district.
As detailed in the attached logical framework (Annex 1), the project has implemented a range of activities at the district, facility, and community levels; both through direct implementation and partnership. The principal partner is the Ministry of Health, specifically the Nkhotakota District Health Office. Other partners include VillageReach, Grassroot Soccer, Interchurch Organization for Development Cooperation (ICCO), and Banja La Mtsogolo (local affiliate of Marie Stopes International).
Purpose
As the project nears its completion, Concern seeks to hire an independent consultant to conduct an external evaluation of the project. The final evaluation will take stock of project achievements against targets and identify key successes and challenges associated with the implementation of the project from the perspectives of stakeholders at all levels including beneficiaries, volunteers, community leaders, health workers, district officials, project staff and partners.
The final evaluation report will be used as a key learning document to inform the design of Concern’s future maternal, reproductive, and adolescent health initiatives in Malawi and beyond.
Evaluation Questions
The overall evaluation questions are organized by the DAC criteria, as follows:
DAC Criteria
Evaluation Question
Effectiveness
To what extent were the three strategic objectives achieved?
Was the programme logic well thought through and how did the project interventions contribute to the achievement of the three strategic objectives?
What were the key successes and challenges that lead to the achievement or non-achievement of the objectives?
What steps were taken to address issues of inequality and ensure the interests of the most marginalised were taken on board during programme planning, implementation and monitoring? How effective was this?
Efficiency
How did the implementation of key activities through partners affect project implementation and achievement of project objectives?
How did collaboration with partners and other district-level stakeholders contribute to opportunities for further scale-up of project approaches within a broader context?
Were financial resources well used? Could things have been done differently and how?
Was the programme M&E system fit for purpose? What evidence is there of effective Results Based Management in the programme?
Relevance
Were the project activities designed to the unique needs of target population based on appropriate contextual analysis or formative research?
To what extent were the strategic objectives and associated activities relevant to and in support of district and national-level Ministry of Health strategies and priorities?
Sustainability
Is there evidence that project outputs and outcomes will contribute to lasting benefits beyond the life of the project?
What are the key factors which may positively or negatively influence the likelihood of the project’s overall sustainability?
Impact
Overall, what real difference has the project made to its intended beneficiaries (women of reproductive age)?
What has changed as a direct result of the project?
While the evaluation will consider the overall project implementation and achievements, emphasis will be placed on assessing key elements of the project for which Concern has particular interest. These include the adolescent health interventions under Objective 2, as well as the social and behaviour change (SBC) activities implemented under Objective 1 (see log frame) and the overall impact of those specific interventions on the target population.
Conversely, several project approaches have already been evaluated. The Health Institution Capacity Assessment Process (HICAP) tool – a key methodology under Objective 3 - was evaluated in August, 2015. The hotline service (Chipatala Cha Pa Foni) being scaled-up under this project was extensively evaluated in 2013. The consultant is therefore expected to review these documents and incorporate the relevant findings of these previous evaluations into the overall report.
Methodology
The consultant will lead a participatory evaluation. This means that project staff, district stakeholders, and community representatives will be involved in the implementation of final evaluation activities and make meaningful contributions to the analysis of findings. The evaluation will utilize a mixed-methods approach incorporating both quantitative and qualitative data, as well as the review of secondary data sources.
The consultant will furnish a written methodology for the final evaluation (including the proposed number of key informant interviews, focus groups discussions, observations, and locations) to Concern for comment and planning purposes prior to the evaluation.
Secondary Data: The consultant will review project documents (proposals, donor reports, mid-term review) to assess progress of project activities in relation to the project design and targets set. The consultant will also review key tools and curricula designed and utilized by the project, including SKILLZ Malawi and SKILLZ Girl curricula, the HICAP tool and HICAP evaluation, and key social and behaviour change tools developed.
Quantitative Data: All quantitative data will be collected and analysed before the final evaluation begins. The consultant will review the project’s completed results framework, which will provide baseline and endline data against targets for the key project indicators[1]. This should form the basis from which the effectiveness questions are answered.
Qualitative Data: As part of the final evaluation field work, in-depth interviews or focus group discussions will be conducted with project staff and partners, district-level stakeholders, health facility staff, community volunteers, and project beneficiaries.
Outputs
Lead an in-country debriefing meeting with Concern project staff, senior management, and other stakeholders prior to departure from Malawi
Evaluation report of no more than 25 pages to be submitted at the end of the assignment. Report should summarize findings against each of the evaluation questions and provide specific recommendations for future reproductive and maternal health programmes
Stand-alone 2-3 page executive summary of evaluation report
Timeframe
The consultancy must take place between February and March 2016. All deliverables must be received and invoices must be paid no later than 31st March 2016.
Lines of Communication
Briefing and de-briefing with the Country Director and Programmes Director; daily coordination with the Health and Nutrition Coordinator.
Budget
The total budget for the evaluation is in the range of 10,000-12,000 EURO as a maximum. This includes the consultant’s daily rate, international travel, visa / immunization expenses, lodging, and per diem based on Concern’s standard policy.
Person Specifications
Essential:
Expertise in community-based reproductive and maternal health in sub-Saharan Africa
Demonstrated experience conducting mixed-method evaluations of community-based health programmes
Strong analytical abilities; capable of synthesizing and making sense of information across a range of sources
Ability to lead and facilitate learning among an evaluation team consisting of project staff and district stakeholders
Excellent writing skills in English
Desirable:
Expertise in adolescent health and youth programming
Expertise in social and behaviour change
Familiarity with the reproductive, maternal, and adolescent health context in Malawi
[1] Baseline data is mainly from the 2010 DHS (district level DHS data was used where possible). A midterm survey was conducted in May 2015 to provide updated, project-specific data against key indicators as well.
