Dominican Republic: Program Mid-Term Evaluator Consultant

Organization: International Executive Service Corps
Country: Dominican Republic
Closing date: 12 Sep 2017

Exporting Quality and Safety (EQS) Program

Midterm Performance Evaluation Scope of Work

International Executive Service Corps (IESC) invites proposals from qualified, independent third-party individuals or firms to conduct a midterm performance evaluation of the EQS Program funded under the United States Department of Agriculture (USDA) Food for Progress mechanism.

Background

IESC is currently implementing EQS in the Dominican Republic, a USD $15 million program which runs from October 2015 to September 2019. IESC is the lead implementer with sub-partners Centro para el Desarrollo Agropecuario y Forestal (CEDAF), Florida Agricultural and Mechanical University (FAMU), and the World Food Logistics Organization / Global Cold Chain Alliance (WFLO/GCCA). The program aims to increase productivity and sales for domestic and export markets of high-value fruit and vegetable global value chains: avocado, cacao, pineapple, and greenhouse and oriental vegetables. The program will improve product quality, increase production and efficiency, increase the value of post-harvest products, and improve market and marketing linkages.

EQS is organized under the following five activity areas:

Capacity Building: Producer Groups and Cooperatives;
Capacity Building: Trade Associations;
Cold Chain Improvement;
In-kind Grants: Equipment; and
Public Information Campaign: Disperse Improved Market Information.

EQS activities are designed to work primarily with value chain aggregators, such as producer organizations (clusters and associations), cooperatives, packinghouses, processors, and trade associations to reach middlemen traders and producers. The program looks at end markets to determine growth-related demand and requirements, in turn focusing on technical solutions related to productivity losses (e.g., orchard underperformance, technology gaps) and product losses (e.g. spoilage, cold chain mismanagement, point of entry refusal, and food safety). The program’s interventions have focused on productivity and product losses of 30 percent or higher in the first two years in order to achieve the end-of-year-two target results related to volume (40,000 MT from a baseline of 32, 688 MT) and sales ($55 million USD from a baseline of $33.7 million USD). Losses can also be defined as not having access to key markets due to certain product bans (e.g. Medfly, Thrips, and other contamination) or refusals based on import inspections. The program has endeavored to work with a significant number of beneficiaries in each value chain to achieve the end-of-year-two targets (3,440 direct and 17,200 indirect beneficiaries), which should then allow the transformation needed for continued and sustained growth as further market-driven scaling and sector-wide replication occur.

EQS is monitoring a total of 36 result and activity indicators, of which 32 had a baseline of zero and four had baseline data established within six months of program start-up. We will provide full results framework, indicator table, and baseline study to interested applicants.

EQS is carrying out activities in the areas of Santo Domingo, La Vega, San Juan, Santiago, San Cristóbal, San José de Ocoa, Monte Plata, Monseñor Nouel, Duarte, Samaná, Puerto Plata, Peravia, Sánchez Ramírez, and Azua provinces.

Purpose of Midterm Performance Evaluation

Per USDA’s Food Aid Division (FAD) M&E Policy, the purpose is to critically and objectively review and take stock of EQS’ program implementation, assess relevance of interventions, provide early signal of effectiveness of interventions (or lack of), document lessons learned, assess sustainability efforts to date, and discuss/recommend mid-course corrections as necessary.

Objectives and Evaluation Questions

Specifically, the evaluator/evaluation team will assess the following:

Program design, implementation, and efficiency
1. Is the program on track to reach specified targets at midpoint and by end of the program?
2. Specifically, what percent is on track and what percent is not on track?
3. For targets, not on track to be met, specify changes (if any) to the initial assumptions or within the operating context that warrant a review and recommended revision of the targets.
4. What have been the main reasons behind current achievement against result and activity indicator targets?
5. Specifically, what should be maintained and/or scaled up?
6. What have been the major challenges to program implementation and how has EQS responded to these challenges?
7. Specifically, to what extent has the Med-fly ban affected program implementation?
8. To what extent did unexpected heavy rains in the Fall of 2016 affect oriental vegetable exports?
9. What, if any, unintended but important direct results (positive or negative) have occurred?
10. How should these be assessed in the overall program context?
11. How has EQS responded to these?
12. To what extent is staffing, management, and oversight costs suitable given the number/scope of activities carried out?
Relevance of interventions
1. To what extent do EQS activities address the core issues of target beneficiaries?
Effectiveness of interventions
1. Which interventions/activities have the highest potential in being the most effective approach to achieve the program’s higher-level results (increased agricultural productivity and more market transactions)
2. How effective is EQS at reaching women beneficiaries? What could be done to improve women’s participation?
3. How effective has the training approach been to build and/or enhance capacity of target beneficiaries?
4. To what extent do public authorities and private sector partners have an influence (positive or negative) on the achievement of program objectives?
Sustainability efforts to date
1. What factors contribute towards sustainability of program results and how has EQS focused on these?
2. Is there a well-developed exit strategy?

Methodology Guideline

The evaluator/evaluation team should use a mixed methods approach, including but not limited to quantitative surveys, focus group discussions with target beneficiaries, and key informant interviews with government officials and relevant public/private stakeholders including EQS staff and USDA.

The data collection tools for collection of key data should be similar to the tools used during the baseline study or during routine monitoring (to the extent possible) so that results may be comparable.

The quantitative sample size should ensure a 95% confidence level with a 5% margin of error. The total direct beneficiaries as of March2017 is 698, which could be used for the sampling frame.

The evaluator/evaluation team will meet with IESC and USDA to discuss proposed methodology prior to initiating evaluation activities.

Key Activities

The evaluator/evaluation team will undertake the following key activities:

Desk review
1. Review program-related documents including:
2. EQS agreement and any modifications
3. Approved M&E plan
4. Baseline study report
5. Semi-annual reports 1, 2, 3, and 4
6. Approved work plan
7. USDA M&E policy
8. Value chain assessment reports
9. Beneficiary mapping report
10. Any other relevant program documents provided by IESC
11. Review other relevant documents to understand the operating environment in which EQS operates:
12. Biweekly communiques
13. Program Facebook Page
Develop and finalize evaluation methodology
1. In collaboration with IESC and USDA,
2. Confirm sampling frame and finalize sampling technique and sample size;
3. Develop and propose survey design and data collection tools for quantitative survey, follow-up focus group discussions, and key informant interviews. Refine or create any tools necessary to answer the evaluation questions; and
4. Based on the desk review, propose any additional topics or issues for analysis in the final evaluation at the end of the program.
Field data collection
1. Plan and coordinate all necessary logistics to conduct evaluation.
2. In terms of office space, the evaluator/evaluation team will co-locate with the EQS program in Santo Domingo. The evaluator/evaluation team will also receive assistance from EQS in locating relevant beneficiaries. Other than this, all resources (HR and material) should be arranged by the evaluator/evaluation team.
3. Translate, pre-test, and finalize data collection tools including survey instruments and focus group questionnaire.
4. Train enumerators, as needed.
5. Carry out fieldwork with liaison support from EQS staff, particularly M&E manager and analyst.
Data analysis and reporting
1. Scrub, synthesize, analyze, and interpret data from survey, focus groups, and interviews.
2. Prepare and submit datasets with relevant documents to EQS and USDA.
3. Prepare a draft evaluation report addressing the objectives and questions of this midterm evaluation as well as propose course correction or scale-up recommendations for review by EQS.
4. Present evaluation findings in-person to EQS, and relevant stakeholders
5. Submit final report.

Timeframe and Proposed Level of Effort

This contract should take about eight weeks starting on or about October 16, 2017. An illustrative timeline is presented below.

Activity

Level of Effort

Due Date

Comments

Desk review

2 days

Does not have to be consecutive days

Kick-off meeting with USDA and IESC staff to discuss expectations, protocol, methodology, and timeline

1 day

TBD

IESC/EQS staff to coordinate.

Develop draft evaluation work plan (including methodology and draft data collection tools)

5 days

TBD

Present draft evaluation work plan to USDA and IESC. Refine based on comments received.

2 days

TBD

IESC/EQS staff to coordinate presentation date and location.

Field work/data collection

15 days

Includes training of enumerators, if needed.

Data entry and scrubbing

3 days

Data analysis and report writing

8 days

Submit draft report to IESC for comments and clarifications

Within 7 weeks of contract signing

EQS and USDA will review and provide comments within 5 days of receiving draft report

Prepare oral presentation of findings to USDA, IESC/EQS, and relevant stakeholders and conduct presentation.

2 days

TBD

IESC/EQS staff to coordinate presentation date, location, and required attendees.

Finalize report and submit to IESC.

3 days

Within 8 weeks of contract signing

Submit to IESC and USDA

Deliverables

Evaluation work plan that describes the following:
- Understanding of the program based on desk review and kick-off meeting
- Final evaluation methodology, including detailed sampling plan, field work plan, and any limitations of the proposed approach
- Description of planned quality control measures
- Description of communication protocol
- Final timeline
Electronic copies of all clean and final versions of data collection tools, both in English and Spanish
Clean and final versions (both English and Spanish) of quantitative datasets and qualitative transcripts in agreed upon format
Electronic draft midterm evaluation report in English, addressing all evaluation objectives and questions
Oral presentation materials of evaluation findings in agreed upon format
15 – 20 high quality pictures of the process
Electronic English version of the final evaluation report in PDF and Word, as well as two printed copies in color (one for USDA/DR and one for the EQS office). The final report should include, but not limited to:
- List of acronyms/abbreviations
- Table of contents
- Executive summary
- Background
- Detailed evaluation methodology
- Findings
- Recommendations for the remainder of the program
- Annexed scope of work
- Annexed data collection instruments

Qualificatons and Selection Criteria

All interested parties will be assessed based on the following:

Experience conducting evaluations of economic growth programs
Knowledge of USDA Food for Progress programs
Experience using various quantitative and qualitative methodologies
Fluency in both English and Spanish required
Clarity of thought process aand writing style, as evidenced in technical proposal
Previous experience in the DR highly preferred

How to apply:

Submission Requirements

All interested parties should submit the following:

Technical proposal (not to exceed 8 pages on typed single-space 8 ½ by 11 paper) that includes:
- Proposed approach to the evaluation
- CV(s) of evaluator/evaluation team and specific roles/responsibilities of each
- At least 2 references of other clients for which similar evaluation assignments were undertaken with contact information for each one
- If bidder is a firm, submit organizational capacity statement.
Itemized budget with narrative explanation of line items.