ASDP Innovators in Monitoring, Evaluation and Learning (MEL)

nov 2016 action sports, development, multi-sited
ASDP Innovators in Monitoring, Evaluation and Learning (MEL)

There are a lot of pressures on established and emerging ASDP organisations to develop MEL methods to measure and prove the effectiveness of their programmes. In our second 'Over the Gap' feature we are delighted to share interviews with two innovators in MEL: Lana Rolfe of Waves for Change and Sylvia Lim of Capoeira4Refugees. Hear about their individual and organisational journeys towards developing culturally appropriate and effective approaches towards monitoring, evaluation and learning, and feel free to post your questions to Lana, Sylvia or the ASDP community more broadly to open up this discussion about the struggles and strategies of MEL in the field of ASDP.

Lana Rolfe: Monitoring, Evaluation and Learning Manager for Waves for Change (South Africa)

Waves for Change (W4C), a sport for development organisation in Cape Town, South Africa, uses surfing and mentoring to promote wellbeing in high-risk children from local disadvantaged communities. The organisation, founded in 2011 with 15 children and two coaches, has grown to 250 children per week and 18 coaches across three sites. Lana Rolfe is their full-time Monitoring, Evaluation and Learning (MEL) Manager.

W4C MEL training 1

Lana Rolfe leads a MEL activity for Waves for Change in South Africa

 

1. When did you start working for Waves for Change as the MEL Manager? What is your background in MEL and how did this inform what you ended up doing at Waves for Change?

I started here in January 2016 after completing my Master’s degree in Program Evaluation at the University of Cape Town. My background is in research psychology and the postgraduate degree in evaluation was a great fit for someone like me who has always wanted to improve the lives of others in a sustainable way.  As part of the degree I had to conduct an evaluation of an existing program and that is how my relationship with Waves for Change came about. I conducted a formative evaluation on their coach traineeship and upon completion of my degree heard that they were recruiting for a MEL Manager. Based on our existing relationship and the strength of my findings, they offered me the role.

2. How important is MEL for Waves for Change? What are some of the more standard MEL approaches that you use, and what are some of the more innovative? 

The only thing higher on the list of priorities than MEL is our child protection and safeguarding policy. Seriously, MEL permeates every aspect here. From grant writing to informing how we contact parents to how we recruit our site managers.

We use MEL to track standard things like attendance and fidelity and we are always looking at innovative ways of learning more about our participants, how they interact with the program and what effect this has on them. 

3. What are some of the key aims (or philosophies) underpinning how Waves for Change are using MEL?

Learning, learning, learning and of course, sharing. We believe in being candid about what we find challenging and being serious about finding solutions. We encourage other organisations to do the same. In fact, we are committed to hosting two learning networks in 2017. In March, we are hosting “experts” in the MEL, mentoring, mindfulness, and action sports for development and peace sectors with the goal of learning from each other and sharing that learning later in the year with other, less experienced organisations that make up the second network. Anyone interested in being part of either network is welcome to contact us at info@isiqalo.org.

4. What are some of the internal and external uses of the data gathered via MEL methods? Do you ever feel that there are particular pressures to use some methods/approaches over others? 

We use MEL as a reporting tool but first and foremost for troubleshooting. If you have a decent MEL system in place it can provide you with invaluable and rapid feedback on program delivery and performance. When you are working with a young high-risk population this is imperative.

One of our most effective systems is the tracking of the performance of our coaches through a caseload system. Every child is allocated to a coach. Coaches have certain responsibilities toward each child in their caseload and these all get tracked. Since implementing this we have seen improvements in attendance, dosage, attrition, session delivery and program effects.

Regarding whether we feel pressures: the short answer is yes. Anyone in this field knows that statistics ‘rule’ and that impact is ‘best’ measured through validated psychological scales. There are some partners who ask for case studies and stories of impact but these are still regarded as less ‘rigorous’.

We have started to voice our discomfort of these methods and are realising we are not alone.  Once you broach the subject (rather self-consciously at first) you often find that others have the same concerns.

5. Do you adapt particular MEL methods for the local context or particular groups? If so, could you give us an example of two of how you have modified methods for the local context? 

We realised long ago that self-reported data, especially in children who are answering in a way they believe you want them to, is highly unreliable. To strengthen our data, we now collect observational data from our coaches and caregivers and triangulate this with what the children are saying. 

We are also trialling a combination of participatory methods such as community mapping, ranking and daily schedules. We are particularly interested in those providing data for our funder reports as well as our internal drive for highly reliable data.

 lana 1 

Lana Rolfe welcomes the ASDP community to engage in a constructive dialogue about MEL in our field.

6. What do you find are the most challenging and rewarding aspects of your job in MEL? 

It is always challenging to introduce new methods and systems. And we have done a lot of that this year! Fortunately, our site managers and coaches have bought into the importance of MEL and have been great at adopting whatever we have thrown at them.

The most rewarding? Definitely when I can show the coaches the fruits of their labour. When what feels like tedious data collection and capturing is transformed into something visual that explains how their daily inputs lead to improvements in the program and ultimately the lives of the children.

7. Any advice you might give to the ASDP community, many of whom are just starting to develop MEL methods for their organizations?

It doesn’t need to be perfect, just get it started. Many people are intimidated by MEL. When people talk about “systems”, they sound so professional and unobtainable for those starting from scratch but once you delve a bit deeper you’ll find that the “system” started with a simple idea that has been tweaked over time to suit the organisation’s needs.

We use Google Forms to track most of our data. It is easy to set up, easy to train others to use and once you have data from a couple of forms rolling in, viola, you have a system!

Most importantly, we need to get more MEL conversations going.  Organisations should be encouraged to share their learning instead of promoting their achievements and downplaying their failures. We encourage our MEL counterparts to criticize and question our results, found here.

 

Sylvia Lim: M&E Co-ordinator, Capoeira4Refugees (Amman, Jordan)

To read more about the story of Capoeira4Refugees, please check out our first 'Over the Gap' feature in which founder Tarek Alsaleh was interviewed.

1.When did you start working on MEL with Capoeira4Refugees? What is your background in MEL and how did this inform what you ended up doing at Capoeira4Refugees?

I started working for Capoeira4Refugees (C4R) in Amman, Jordan, in 2015. I was hired as both the M&E officer and a capoeira trainer. I have a pretty diverse background – I was worked as a journalist for years before obtaining graduate degrees in anthropology and public health. I have played capoeira since 2004, so when I came across this job, I felt it was the perfect fit. Using capoeira as a way to reach out to conflict-impacted children and youth really touched me.

C4R 1 1

Capoeira4Refugees head trainer Daniel 'Arame' Vallejo working with a student in a community center in Irbid, Jordan.

2. How important is MEL for Capoeira4Refugees? What are some of the more standard MEL approaches that you use, and what are some of the more innovative ones?

MEL helps us see whether we are making an impact on the young people we work with, and how. We use semi-structured interviews and program evaluation surveys as our standard MEL approaches, and we have refined two other methods for our organization. We adapted the Most Significant Change technique (see Question 5), and changed the way we count students in the countries we work in. In the past year, we have standardized variables for our quantitative data collection, and implemented a software suite (Dharma Platform) to streamline attendance data gathering, analysis and management.

3. What are some of the key aims (or philosophies) underpinning how Capoeira4Refugees is using MEL?

At C4R, we take the empowernment approach in our work. This approach emphasizes self-evaluation, community participation, and open discussions in the evaluation process. It is crucial for us to adopt an approach that aligns with our organization’s values of strengthening communities and youth leadership. Ideally, our evaluation process should contribute to increasing the skills and capacities of the young people we work with. We also have very clear purposes as to why we perform specific evaluations – we collect relevant data using accessible methods and try to be as unintrusive as possible. This is important as many of the children and youth we work with have experienced traumatic events. Therefore, we try to avoid using tools that would trigger negative emotions or subject them to lengthy evaluations.   

4. What are some of the internal and external uses of the data gathered via MEL methods? Do you ever feel that there are particular pressures to use some methods/approaches over others?

Internally, we use the data we collected to help improve program delivery and identify areas for professional development (for example: training on child protection, gender-based violence, dealing with secondary trauma, etc.). More importantly, we share the information with our team as a way to reinforce team spirit and remind everyone why we do what we do, and how we can improve. Externally, we include data in our donor reporting, on our website, and on social media to share MEL data publicly. 

Over the years, there have been pressures by some partners to use standard, validated surveys or questionnaires. However, after considering our students’ context and background, we decided to utilize simple, accessible, and child-friendly data-gathering methods as some of the “standard” instruments can be intimidating and difficult to deploy. Afterall, our organization teaches capoeira to young people, and our trainers are not case managers or psychology experts. We need to be mindful of how we interact with our students to avoid causing unintended consequences.

5. Do you adapt particular MEL methods for the local context or particular groups? If so, could you give us an example of two of how you have modified methods for the local context?

The Most Significant Change (MSC) is a storytelling technique we use to track changes among our students over time. Instead of just using it as progress-monitoring tool, C4R has adapted this method to encourage our trainers to think more critically about their work and to allow them to change the way they work as they see fit. We typically do a baseline session after starting a new project, and meet once a month so they can share stories about students they have been observing over time. This process encourages trainers to become more reflexive about their work, and enables them to implement strategies to improve the quality of our programming. 

For example, we work in Jordanian refugee camps with partner organizations. Our work in the camps is sometimes dictated by our partners’ funding cycles and contracts. In one of the MSC sessions we held, a trainer shared a story about an incident at the camp that had impacted her deeply. The incident took place shortly after a break between programs, and our trainers did not get a chance to tell the students that we had to temporarily stop classes. When this trainer returned to the camp, she bumped into a student who told her: “I thought you had died.” This story is a huge reminder of how some of our students understand or internalize absences of the people around them. After the story was shared in a MSC session, we implemented two major changes. First, we would let our students know several weeks in advance of any breaks in our programs and hold a small celebration at the end as a form of closure. Second C4R has made it a priority to seek out longer-time partnerships to minimize programmatic disruptions. Our work centers on building relationships with our students, and having multiple breaks can jeopardize the trust that our trainers have established with them.

C4R 2

Capoeira4Refugees trainers, Hussein 'Palhaco' Alzaben and Daniel 'Arame' Vallejo, lead a capoeira music session in a community center in Irbid, Jordan 

6. What do you find are the most challenging and rewarding aspects of your job in MEL?

Working directly with young people who have experienced conflict can be a challenge due to many reasons. As a former trainer and a MEL staff member, field work can take an emotional toll. However, seeing the way our students’ faces light up when they successfully learn a new movement or share with you about how capoeira makes them feel less isolated and more connected with their peers, can be very rewarding.  

7. Any advice you might give to the ASDP community, many of whom are just starting to develop MEL methods for their organizations?

Be clear about why you are doing the evaluation, and keep processes simple because it helps with continuity and sustainability.

 

A: responses

Administrator

#3

ASDP Community, this is your opportunity to ask questions of Lana, Sylvia or the community more widely about MEL, or to share your own experiences of developing MEL methods and approaches. I know we would love to hear your experiences and insights, so please do 'drop in' to this dialogue and share your questions and wisdom.

post a comment
0 comments