16: Program Evaluation

Standard of Practice: The mentoring program creates and implements a formal evaluation and data collection plan that addresses tracking of implementation fidelity, mentoring relationship quality, relevant participant outcomes, and program costs.

Download Element 16
Download Checklist

Practices Supporting this Standard

The program has a written evaluation plan.

This plan will look different in variety of program models and settings, but it is important that every program create a written plan that outlines efforts they will take to better understand their adherence to policies and procedures, the quality and consistency with which mentoring services are delivered, the costs of those services, and the relationships and outcomes experienced as part of the program. Effective evaluation plans include information on:

  • What important questions the program is trying to answer, such as testing key aspects of the theory of change (see Element 2) and measuring short-, intermediate-, and long-term outcomes, implementation fidelity, mentoring relationship quality, and program costs.
  • Who will provide information to address those questions.
  • When each piece of information will be collected.
  • How information will be collected, both in terms of method (e.g., survey, focus group) and process (e.g., which staff members are responsible and how they will implement data collection processes, such as preparing participants and generating buy-in for data collection, obtaining consent from respondents, and scheduling survey administration).
  • How the program plans to analyze and secure the data, for example, who will be responsible, what types of resources will be needed, and how the evaluation will address each question.
  • How the program will share findings with stakeholders (e.g., funders, community partners, board members) and participants.
  • How staff will obtain feedback on those findings and integrate it into program operations.
  • Information about any data sharing agreements needed for accessing data from external sources (e.g., academic records, juvenile justice data).

The program engages in consistent, ongoing data collection and analysis to address the questions outlined in its evaluation plan.

The program should use the evaluation plan to guide its collection, analysis, and sharing of data on an established timeline. To accomplish this, programs will need to dedicate resources to data collection and analysis, including training and supporting staff as needed.

The program shares evaluation findings with stakeholders.

This includes program participants, staff, board members, funders, and other community partners. When sharing findings, programs should consider: 

  • generating a formal report that is accessible to all stakeholders;
  • creating other summaries of the results or infographics that can be shared broadly; and
  • creating a process for program participants and staff to reflect on the findings and offer suggestions for program improvements.

The program uses findings to make improvements in its services on a regular basis.

Programs should determine how findings and feedback from key stakeholders will be used to improve the program and more effectively meet client expectations and needs. Ideally these research-to-practice improvement efforts will be led by the program’s advisory committee or other ad hoc group with authority to recommend program changes to leadership. Regular and consistent reviews of new data collected should be implemented to ensure the responsiveness of program improvement efforts.

Because there is tremendous diversity in how and where mentoring is delivered to young people, here we offer additional practices and recommendations related to this Element for some common mentoring contexts. Readers should note that there may be overlap in the following categories below (e.g., a peer mentoring program in a school or a Boys & Girls Club offering a group mentoring program on-site) and read all that may be relevant to their work. The next recommendations can support program evaluation and continuous improvement efforts for some typical mentoring models and settings.

GROUP MENTORING MODELS

Group mentoring programs should largely follow the previously noted evaluation practices, but there are a few nuances they may wish to consider when building their evaluation strategy:

It will likely be important to measure group concepts, such as group cohesion and belonging, that may be drivers of outcomes. If part of the program’s theory of change is that youth will benefit from being welcomed into a new and positive peer context, measuring their feelings of belonging and perceptions of group functioning and support may be important inputs into understanding whether the program is working as intended.

Group programs may benefit from including observational activities in their evaluation. This involves having someone from the program or evaluation team observe the groups in action, looking for signs of healthy group interaction and noting the situations or circumstances in which groups tend to struggle. Important group dynamics and insights into groups’ overall functioning may be well captured through this approach and highlight implementation successes and challenges.

If programs are using a team approach, in which multiple mentors work collaboratively with a group of youth, they should measure the quality of their mentor-mentor interactions and relationships. If a team of mentors is having conflict, chances are their work with the group will be negatively impacted in some way.

Because group mentoring programs tend to emphasize the use of structured activities or a prescribed curriculum to guide group discussions, these programs will want to measure the completion of these activities or progression through the required curriculum. Completion of certain activities may be essential to the program’s theory of change and the progress of each group may be a predictor of the benefits youth ultimately experience.

PEER MENTORING MODELS

As noted above for group models, peer-to-peer mentoring models will likely want to track the completion of program activities or curriculum as part of their evaluations. Additionally, peer programs are encouraged to:

While potentially meaningful in any program, the outcomes experienced by mentors take on increased relevance in programs such as these where the mentors are also young people. The program’s theory of change should highlight the benefits for the older peer mentors, as well as those for their younger mentees. Peer mentors may grow in areas such as their leadership skills, confidence in helping others, or their sense of identity and self-esteem.  

Surveys asking peer mentors about their experiences should be simplified for an appropriate reading level and specifically framed around collecting the unique perspectives of a young person in the mentoring role. Questions about their personal growth, or the growth they have seen in their mentee, should be simple and easy for them to understand conceptually.

While the perspectives of others outside the mentor-youth dyad can be helpful in any program evaluation, it may be especially important in peer programs where other school staff are directly involved in the relationships. It can be difficult for both peer mentors and young mentees to express their perceptions of how the mentoring experience is affecting them. Those changes might be seen and reported more easily by others who know the youth participating in the program. Their reports of positive change can make meaningful contributions to demonstrating program impact.

E-MENTORING MODELS

E-mentoring models are another example of programs in which participants’ progression through a set of prescribed activities can be an important marker of engagement and a likely driver of program outcomes. In fact, online mentoring programs should emphasize several markers of participation in their data collection and analysis. Data points such as the number of logins or messages sent, the average word count of messages, the frequency of interactions between mentors and youth, the average response time between participants, and the total time spent engaged in the program platform may all be important predictors of how impactful the program is for participants. Additionally, e-mentoring programs may wish to:

(e.g., texts with links to brief surveys that note, “After your next interaction, click on this link.”). This can make data collection an integrated part of the experience of engaging in an activity and takes advantage of their time in the platform where they communicate.

It can be frustrating for participants if technological glitches, bugs, connection issues, or other software or hardware barriers are hindering their communication. They may also have feedback about how the communication methods used by the program are helping or could help their mentoring experience (e.g., may prefer video calls to email correspondence).

These approaches can be helpful in any mentoring program but may take on increased importance for purely virtual mentoring models where so much of the interaction and participation is typing or text based. Using alternative data collection methods that allow for verbal responses can help accommodate many individuals with physical or learning disabilities. In fact, it can be difficult for any young person to provide a lot of information via typing. Video interviews, focus group chats, phone calls, and other methods where participants can verbally share their thoughts can be helpful in generating richer, more complete data.

SCHOOL- AND OTHER FORMAL SITE-BASED MODELS

School- and site-based models are also encouraged, as noted above for group, peer, and e-mentoring programs, to track participant completion of key activities related to the program’s theory of change. And as noted for peer programs, teachers, counselors, caregivers, and other faculty or site staff may be important sources of information about program impact.

When set in broader youth-serving organizations and institutions, formal school- and site-based mentoring programs are additionally encouraged to: 

(e.g., principals in school-based programs). School districts, in particular, may have a formal process for vetting and approving any data collection involving students and district evaluation staff or institutional review boards (IRBs) may need to examine the plan before evaluation activities can begin.

(e.g., close to other major tests or right before long breaks when some students may be absent). Noting the other evaluation activities happening at the site can also help avoid redundant data collection.

(e.g., provide staff with a quiet, private place to conduct interviews or administer surveys).

Others in the organization may have unique insights into how well the program is functioning in the broader context.

Schools may have a particular advantage here, as they have access to information about other youth that most nonprofit providers do not. This information can be invaluable in helping illustrate the value that the program is bringing to mentored students, compared to a similar group not participating in the program.

INFORMAL MENTORING MODELS

Organizations that are offering informal mentoring via staff may choose not to evaluate mentoring separately, given that mentoring relationships may not be experienced by all youth and the connection to other organization services or outcomes may be less clear. But, when possible, youth and caregivers should be asked about their mentoring experiences (if any) as part of the overall evaluation of the organization’s services. Programs can ask about their mentoring experiences, the benefits they received, and how the presence of mentoring may have bolstered or enhanced their overall experience in the organization.

  • Why do we want to evaluate our program? What questions do we have and why are they relevant? What value will the answers bring to our work or to the community we serve?
  • What metrics would indicate that we are implementing our program as intended? What could we measure to demonstrate that our program is functioning as intended?
  • What is the story we’d like to tell about our program? What story do our participants want to tell? And how can we best tell it? What methods or activities can best show the hard work and outcomes of our program and participants?
  • How can youth, caregivers, and mentors help us design our evaluation and collect meaningful data?
  • How can we create buy-in and ownership of this evaluation among those who need to provide data? How can we incentivize their participation to make sure we have a good response rate?
  • What questions do we have about how our participants experience the program? What would we like to know that we don’t already?
  • How can we ensure that information is collected consistently and with as little bias as possible?
  • What outcomes are reasonable for us to examine based on our theory of change? How can we ensure that we are measuring the right things and not being overly ambitious (or focusing too narrowly)?
  • What types of data collection tools (e.g., surveys, records, interviews) can provide evidence that is most convincing/valued by our funders and stakeholders?
  • How will our evaluation approach reflect our core values as a program?
  • How will we handle results that are surprising, unexpected, or disappointing? How will we explain surprising successes? How can we share disappointing results with an emphasis on how we will build on them as part of continuous improvement?
  • What steps will we take to make our evaluation findings actionable? Who on our staff can assess what’s working and lead course corrections in practices that are not?
  • Youth, caregivers, and mentors can help design the evaluation plan, helping you understand what might engage other participants in sharing their experiences, how you might frame questions to those groups, how you could generate buy-in, and how best to share your evaluation findings with those participant groups. A participatory action research approach can maximize their engagement in evaluation activities.
  • If there are certain groups of participants who reported relatively low levels of satisfaction with a given practice or their broader program experience, you might consider holding a listening session or individual interviews with participants in those groups to elicit ideas for program changes that could improve their experiences.
  • Participants can support program staff in making important changes based on feedback (e.g., helping to revamp mentor training to better address challenges in communication with youth revealed in the evaluation findings), especially through the work of a formal advisory committee or ad hoc group of stakeholders tasked with translating evaluation results into actionable program improvement efforts.
  • Youth, caregivers, and mentors can help share findings with key stakeholders by sharing their own mentoring stories and providing rich examples of what the findings mean in their own relationship.

Programs may want to set benchmarks and track progress around metrics such as:

  • Response rate on evaluation surveys (and whether those responses are coming from a broad representation of your participants).
  • Proportion of data collection strategies implemented as planned in a program cycle.
  • Dissemination metrics of evaluation report/infographics.

A Brief Primer on Youth Participatory Action Research for Mentoring Programs. National Mentoring Resource Center.
This brief primer provides an overview of youth participatory action research (YPAR). YPAR is a promising approach for elevating youth voices in mentoring programs to create positive change.

The Community Builder’s Approach to Theory of Change: A Practical Guide to Theory Development. Anderson, A., Aspen Institute.
This guide provides a basic overview of the major concepts that define theories of change along with guidance and a resource toolbox to support development of theories of change.

Measurement Guidance Toolkit. National Mentoring Resource Center.
A mentoring-focused collection of measurement tools for examining participant outcomes. Includes a Selected Reading and Resources page that contains several valuable resources on general evaluation, survey design, and data sharing.  

From Soft Skills to Hard Data: Measuring Youth Program Outcomes. Wilson-Ahlstrom, A., Yohalem, N., DuBois, D., & Ji, P., The Forum for Youth Investment. This compendium describes scales in a wide range of areas used to measure youth outcomes.