It’s been a few months since my last guest post, so today I’d like to introduce you to one of the savviest data divas I know, Laura Budzyna (see full bio below). Laura and I first met in graduate school where we led a joint research trip to the Dominican Republic, working with my current employer, The DREAM Project, for which she had previously volunteered. Laura is especially perceptive when it comes to both the technical and interpersonal aspects of M&E, which makes her an asset to any team. As she prepared to embark on a recent consulting gig, she put together a stakeholder analysis to help her manage the cast of characters often found on the M&E stage.
Here’s her story!
“Oh, so you’re like an auditor.”
“Well, to them you are.”
I stopped mid-sip of my smoothie. I had just explained to a friend of mine that I was going to be visiting a few microfinance institutions in South America and taking a look at their monitoring and evaluation systems. My plan was to hang out with their IT teams and their loan officers, make some recommendations on how to make their collection and input processes more efficient, and offer a few data analysis tips to boot. But an auditor?
“Sure. They probably feel threatened. They’re certainly not anxious to set a date for your arrival.”
Huh. He was right – we had gone back and forth about the date of my visit for weeks now. Well, now, let’s think about this a second. Sure, the MFIs didn’t actually invite me to come – an investor of theirs was paying for my ticket and my time. An investor who…hmm…funds MFIs based on their social impact track record. And now, here I was, a foreigner sent by an investor, asking to rifle around their databases. I suddenly saw it from their perspective: who knows what I might find?
If I found inefficiencies? Okay, those would be helpful enough to point out, if not a little embarrassing for the organization to share with their funder. If I found inaccuracies? Even more awkward. And if I found that their clients’ wellbeing was not improving? Well, I might as well just pull the plug on their funding altogether.
I was at best a disruption, and at worst a threat. I hadn’t quite thought this through.
Monitoring and evaluation, like any other specialty, is criss-crossed with complex relationships. Who are the stakeholders? Who are the influencers, and what are their agendas? Who is the information for, really? And perhaps more importantly…who should it be for? To answer these questions, many development practitioners use a tool called stakeholder analysis. This technique is featured in the handbooks of many of the heavyweights in the industry (see: DFID, UNDP, EuropeAid and CARE). The name rings tinny with jargon, I know. But if you can get past the dry stakeholder matrices prescribed by DFID and their ilk (you know, the ones languishing in the appendices of too many UN reports), the actual thought process behind stakeholder analysis is extremely helpful. It’s one of the first tools that we teach to the Masters in Development Practice students at SIPA, and in solidarity with them, I’m going to take a stab at one of my own.
I’ll begin with the caveat that my analysis is not specific to any one organization, and therefore it makes a few generalizations. It’s also a tad tongue-in-cheek: the summaries read a little like trading cards or Myers-Briggs profiles (if I had artistic leanings, I would have happily adorned each with an avatar, too). Still, I think they offer a useful starting point for an outsider trying to understand the daily drama of the M&E world. So, without further ado…
Who’s Who in M&E?
Most M&E systems have a fairly established cast of characters. Within the organization, it’s the management, the IT/data or admin team, and the client-facing staff that make M&E (especially M) happen on a daily basis. Outside of the organization, there are the funders who demand data and the consultants and academics who vet it. And then of course, there are the clients themselves who provide this valuable data. Let’s take a closer look:
Funders: These are donors and investors who, like the rest of us, have gotten pretty excited about the word “impact” and now want ever more convincing evidence of it in exchange for their dough. As evaluations have gotten fancier over the years, so have their demands. In fairness to them, many also provide technical assistance (see: Evaluation Consultants) and other resources to help their grantees meet these demands.
- Influence: High
- Motivation: To see that their money is well spent. And since many funding organizations have their own funders and investors, they want to be able to prove their own impact.
- Accountable to: their own funders, industry peers, policymakers
- Use results: to make funding decisions, inform policymakers and other thought leaders, set agendas
Managers: These leaders are constantly balancing the need to produce impact for their clients with the need to prove it to their funders. Some are excited about measuring impact for the organization’s own sake; others are only doing it reluctantly at the behest of their partners. To tell the difference, it helps to find out how the current M&E system kicked off: did the managers initiate and champion the effort, or was it an external request?
- Influence: High
- Motivation: To keep the ship afloat. And that means keeping investors happy while trying to focus on their core mission…and keeping the minions (i.e. me) at bay.
- Accountable to: In mission, clients. In practice, funders.
- Use results: To plan strategically, to change course, to report to funders, to benchmark against other similar or competing organizations
Evaluation Consultants: These folks are paid to poke around other people’s kitchens. (Okay, they also design instruments, implement studies and analyze data.)
- Influence: Medium
- Motivation: To get tons of information in a (usually) short amount of time, so they can make effective recommendations to their client.
- Accountable to: Whoever is paying them – could be the investor/funder, another consulting company, a research initiative, or the management. (If the management isn’t paying them, managers don’t have any control over how the findings are shared – another scary thought.)
- Use results: To produce shiny reports for someone else to use, to add to their own knowledge base so they can carry it along to the next project.
Academic Researchers: In the same category as evaluation consultants, these outsiders bring scientific rigor to the evaluation effort. While there is more and more talk of successful partnerships between academic institutions and NGOs, the scrupulous academics and the pragmatic practitioners don’t always see eye to eye on how M&E should be carried out.
- Influence: Medium
- Motivation: Rigor and recognition, usually in the form of a peer-reviewed publication.
- Accountable to: The academic community. And whoever’s on their tenure committee.
- Use results: To publish, and to add new knowledge to their field
The IT/Data/Admin Team: From the database managers to the data entry wonks, this is actually where the magic happens. In many organizations, these employees are revered, as they’re the wardens and curators of the organizations’ data. Usually, they’re the only ones who can (or know how to) access the raw data, while management can only pull standardized reports. Depending on leadership’s enthusiasm for the M&E effort, this may or may not afford the IT team a good deal of influence in the organization.
- Influence: Low-to-medium
- Motivation: To do well at – and be recognized for – their challenging and specialized job
- Accountable to: Management
- Use results: To assess data validity
The Data Collectors: Your results are only as good as the data you gather. Without these worker bees, the whole system would collapse. Often, data collectors are client-facing workers who take on data collection on top of an already towering task list (think community health workers, microfinance loan officers, etc.). Others work for local research companies and are contracted for short-term data collection efforts, like an annual survey or an external evaluation. They spend time with the clients, they speak the language, and they populate our spreadsheets with hours and hours of conversations.
- Influence: Low (Pro tip: if you can engage your data collectors during the survey design process and pilot, they will be your most helpful critics.)
- Motivation: If they’re exclusively a data collector: per survey compensation. If they’re an employee with 27 tasks to complete other than data collection: to finish as quickly as possible. The data collectors’ time is one of the top considerations when designing surveys or deciding on which indicators to measure on a regular basis.
- Accountable to: Supervisors
- Use results: Indirectly. They may hear the highlights, but they don’t usually interact with results firsthand unless they give rise to major operational changes. Still, many of the workers who interact with clients regularly are very excited to learn about their progress, and it’s my opinion that they should be among the first to see the results.
The Client/Beneficiary: Oh yeah, remember them?
- Influence: Low
- Motivation: To improve their quality of life. And to get finished with this damn survey. (In fact, clients in a position to choose between different organizations may choose the one with fewer bothersome questionnaires.)
- Accountable to: Their own families, their own jobs, and certainly not this data collector.
- Use results: When was the last time you showed an impact report to the client? Imagine the power shift if you did! Clients holding organizations accountable for producing results? The implications are tremendous, and worthy of another blog post altogether.
Looking at these profiles, my own challenges with this MFI project are beginning to make a lot more sense. No one is actually accountable to me, the evaluation consultant, except for the fact that I’m linked to a funder. Still, my recommendations have influence and may well change their jobs substantially. And if I don’t understand how my suggestions will affect all the stakeholders, then my whole project will fall flat.
It’s only the first piece of the puzzle – I’ve got a few weeks of meetings ahead to understand the full picture. But at the very least, I’m glad I did this before tomorrow morning, when I’ll walk into my meetings and (knock on wood!) convince these organizations that I’m on their side.
I’d love to hear whether any of this holds up against your experiences. Where do you fit into the mix? What are your motivations, whom are you accountable to, and how does this affect how you interact with the other stakeholders?