To Nurture Organizational Learning, Be a Learner. Show More Than Tell.

Last week, I got to join a Community of Practice of Program Officers and Measurement & Evaluation Specialists from the Department of State's Bureau of Educational and Cultural Affairs. We talked about 10 tactics to build organizational learning, based on a blog post I’d written for the American Evaluation Association, which the group’s leader had read.

As I prepared my talk, I realized that the 10 tactics could be distilled into 5 takeaways:

• Build relationships

• Empathy: Seek to understand roles, needs, interests, processes, contexts

• Share the “why” and “how” of your work

• Share examples

• Give people a way in

This weekend, it then occurred to me that if you want to promote learning, be a learner. Show more than tell.

In our conversation last week, I was reminded how evaluators can feel tensions around: having our training and expertise valued, wanting to maintain impartiality, and trying to be efficient and scientific. So, a few questions came up about how my 10 tactics bump up against these feelings and experiences.

Without getting into the weeds, I believe there are ways to balance these concerns. Of course, I want my own expertise to be valued, and I’ve experienced times when I’ve felt disrespected and disempowered. The solution, in my opinion, is not to double-down on “I’m the expert” (and sadly, sometimes the solution is to walk away). Ultimately, without relationships and empathy and honoring the expertise of our communities and colleagues, our science will be bad. If we aren’t measuring the right things, using the right language, in ways that make sense for the context and people involved, without meaning-making and interpreting findings together, then our methods will lack validity, clarity, and utility---we may have “faux” rigor. We’ll continue to perpetuate inequities, and we’ll lose out on the opportunity to learn ourselves and enjoy our work. There are many cases when “starting slow to go fast” makes sense in evaluation---if we spend time up front, getting on the same page and buy-in with our involved colleagues and communities, we can set up processes and collect data that are more efficient, effective, complete, and meaningful than if we don’t.

My “show, don’t just tell,” thinking is informed by my background in developmental psychology too. There are numerous studies that demonstrate how important modeling is to kids’ learning. Here’s just one study from 2018 that examined how parents influence children’s attitudes toward reading. Among the findings, the researchers noted that ”…children are keenly aware of parental hypocrisy, providing unanticipated direct comment on it regarding parental encouragement. This would suggest that parents could actively attempt to foster interest, attainment, and utility value, but if they are not readers themselves, it may not yield any positive influence on student attitudes.” (p. 692). I would argue the same is true for those of us trying to foster learning within our organizations.

Learning to ride a bike as adult—something I feel vulnerable sharing, and something I want to get back to.

Train for the Research and the Results

Sometimes when a project or goal feels too big, I remind myself about Atomic Habits—a book I love and an approach to focusing on tiny steps, process, and habits rather than where anxiety wants me to go—focusing on results.

Lately, this has worked well for me to start running more again (and enjoying it). For the past 12 weeks, I’ve focused on running 4 days a week, any amount, any speed, with as much walking as I want. I’m focusing on rebuilding a habit, getting outside, having fun, and time on my legs. I’m not worrying about results. As I get stronger, and the habit sticks and feels good, I can layer on other pieces (speedwork, hills, getting ready for a 10 mile race I love).

Most summers, including this one, I’m helping 4-5 AmeriCorps programs design their next evaluation studies. We talk about where they are in their evaluation and evidence history/journey, what they want to learn next, before going deep into deciding what type of study they should do (a process study, an outcomes study, a quasi-experimental study) and how (methods, doing it internally or hiring an external evaluator).

One thing worth considering is not just what kinds of results or evidence they hope to build, but what their experiences have been (or not been) in collecting and using data, in doing a study internally, or hiring and working with an external evaluator. Do they have time on their legs and some habits established before they sign up for the marathon?

As a developmental psychologist (and evaluator), I think about how needs, interests, abilities change over time and how contexts support growth. This includes whether and how an evaluation study is the right fit to where the program is at. Would I advise a new runner who just started this week to run 10 miles tomorrow? If they managed to run 10 miles without training, would it be an actual reflection of how well they could run? Would it cause harm vs. feeling good and strengthening them?

I was also trained in using the 5-Tiered Approach to Evaluation and ways that a program might learn and build evidence over time, adapting their approaches as their program gained experience, felt confident in the quality and consistency of implementation, etc. before focusing too narrowly on outcomes or certainly a comparison group.

So, consider, before leaping headfirst into a marathon, I mean, a complex study, how can you train not just for demonstrating positive results but how you can train for the research process itself? What are some ways you can collect and use some high-quality data? How can you get to know what it’s like to hire and work (well) with an external evaluator? What initial analyses or smaller, focused studies might you do that help you learn what’s involved in different aspects of evaluation? Are there ways I can help or point you to useful resources?

Here's a blog post I wrote about getting ready for an evaluation study in 2015.

What’s worked for you? Are you watching the Olympics? Getting outside this summer? Let’s inspire each other.

Cross Pollination and Play

This is your call to talk to someone new and different. Explore “third places.” Take a new route. Mix it all up with some friends who are different than you. Some of my best ideas and most fun moments come from flowers of a different bunch.

It makes so much sense that I should love cross pollination…

In nature, insects (often bees!), wind, and certain birds and bats apply pollen from one flower to another. According to Brittanica, they “combine the hereditary traits of both parents, and the resulting offspring generally are more varied than would be the case after self-pollination. In a changing environment, the genetic variability within a cross-pollinated population may enable some individuals to be adapted to their new situation, ensuring survival of the species….” Don’t we all need to adapt?

Especially since I left academia, I’ve loved the opportunities to collaborate across sectors and disciplines. At City Year, I got to learn and work with members of the marketing and communications team, for instance. I discovered that we took similar approaches to research and statistics—we just used different terminology. They segmented data and created personas and profiles, and I used person-oriented approaches and cluster analyses. Tomato/tomahto. At one point, I started a City Year data analysts’ affinity group, when I realized that multiple departments had folks in data analyst roles who all shared common challenges. We shared solutions and resources and felt stronger together.

This is one reason why I love my coworking space, The Village Works (TVW). I’m surrounded by coworkers in different fields, and through proximity (and the welcoming environment that TVW creates), we support each other (e.g., sharing feedback and resources on presentation slides, logic models, ideas for teaching, and ice breakers; offering tips about ChatGPT and other software; and exchanging advice on how to handle relationships with clients and job searches). So many of us work or volunteer within or with nonprofits, that we started a monthly Social Impact Lunch group. Sometimes we chit chat and get to know new TVW members, and sometimes members offer talks about their work.

And friends! Friends are among the best bees. I’m constantly delighted to find unexpected connections. Who knew how much internal evaluators and lawyers serving as in-house counsel had in common, or how much I’d admire my real estate agent as a woman in business, building lasting relationships in an authentic way, or how one of my dearest friends, who worked in banking and finance, could fully commiserate about spreadsheets, data integrity, data viz, and so much more?!

Not to mention how much I love parallel play/work, buzzing near others like this NYT article describes. That’s not quite cross pollination but sometimes leads to it. Which reminds me that tomorrow is the first International Day of Play!

How will you play, connect, and cross pollinate?

a bee visits a flower while I catch my breath during a run

Ten Tactics Internal Evaluators Can Use to Build Organizational Learning within Nonprofits

[I wrote this piece for the American Evaluation Association’s AEA365 (a daily blog with tips about evaluation from evaluators) during a special Organizational Learning-Evaluation Capacity Building week in April 2024; it was published April 15, 2024].

Most of my work focuses on evaluation capacity-building with nonprofits. I’ve been immersed deeply in building culture and practices for organizational learning for 16 years, both as an internal evaluator and as a coach to many internal evaluators. Based on my experiences in a range of settings, here are ten (among many more) tactics to build organizational learning:

1. Form a data committee that includes representatives from different departments and roles. The purpose of this committee might vary depending on where your organization is in its data journey, but some functions might include: inspiring champions in non-evaluation roles to bring learning and information to others in your organization; data governance; and identifying areas of overlap and opportunity between evaluation and other projects.

2. Fold data/results and reflection into existing meetings and structures. To help reduce the silo-ing of evaluation, look for opportunities to bring a data point naturally into staff and other regular meetings.

3. Use consistent reflection questions and/or protocols. Consider using some predictable questions and structures that support non-data folks to have a “way in” to interpret data during longer data reviews.

4. Set, share, and keep to a data schedule. To help avoid that “black box” feeling, create and share a data schedule with your organization that includes when tools will be ready (e.g., when a survey will open), when data will be collected, and when results will be available in which format(s).

5. Be proactive. Share updates, findings, check in about data needs in your organization. People will appreciate hearing small wins or challenges you are working on and that you have their interests in mind.

6. Express appreciation. Honor those who contribute to evaluation work (e.g., who help collect data) or who use your data. Not only will this contribute to a positive data culture and make people feel good, but it will also foster learning through examples.

7. Share examples from other organizations/nonprofits. Our colleagues, who are not evaluators, put a lot of faith in us. Finding and sharing examples of how similar organizations have done what you propose can build confidence.

8. Ask leaders in a range of roles to feature/share how they use evaluation in their work. This helps people see a broader vision of how they could leverage data and inspires data use.

9. Learn by doing together. Work on tools like surveys, and get input on types of analyses, with folks who are not evaluators. Describe the why behind your thinking and ask about their rationales too. You’ll learn from each other, and the work will be more meaningful and fun.

10. Add capacity with an evaluation advisory board. Bringing some outside evaluators and community members in can help advance learning, adding diverse perspectives, experiences, and wisdom.

Hot Tips

Curious about how to set up an evaluation advisory board? Here’s an article about it in the Journal of Youth Development that I co-wrote with fellow advisory board colleagues. Although we focus on the context of youth development programs, the tips and examples are applicable to other settings.

Rad Resources

Here’s a helpful protocol that includes questions for data reviews from the Network for College Success.

I’ve found The Facilitator’s Guide to Participatory Decision-Making to be a powerful and practical book to help me frame meetings, data reviews, and other gatherings effectively. You can find more information about it, as well as related publications, here.

Flowers blooming from a recent visit to the Oak Park Conservatory with a bestie who has been a teacher for 30 years—she is always learning and promoting learning.

Let’s Get Physical (with Data and in Nature)

For quite some time, I’ve been fascinated by data physicalization—collecting, visualizing, engaging with data using tactile and physical forms. This is a story of bringing hands-on data collection to a hands-on event, which maybe will inspire you to try something old-school too.

What am I even talking about? Here are some beautiful examples of data physicalization from Jon Schwabish and The Urban Institute.

We weren’t sure how people were learning about our events….

After volunteering for over a year, I recently joined the Board of The Somerville Growing Center—a green space in my neighborhood where community and plants grow. The Center is a space for local preschool children and people of all ages to learn about the natural world and urban agriculture and to simply play outdoors. I’ve loved participating in and helping to host yoga classes, concerts, and the most adorable annual camp-out for stuffed animals, among other events. Last spring, I grew sunflower plants from seeds at home, which then grew in the garden—it was so fun to watch them unfold, grow, and survive bunny snack attacks. The Growing Center is a true oasis in Somerville-the most densely populated municipality in New England, with over 80,000 people living in just 4 square miles.

In March, The Somerville Growing Center hosted its 25th annual Maple Boil Down.  A team of volunteers tapped maple trees on the Tufts University (my Ph.D. alma mater) campus from January to March. Then, at the Boil Down, volunteers boiled it all down into real, true 100% hyper-local maple syrup. Kids and adults got to learn about the process, taste sap and syrup, and engage in hands-on activities.

As we planned outreach and advertising for the event, a fellow board member who was going to put up flyers wondered how effective they were, and a group of us discussed a survey that the Growing Center had used in the past to ask folks how they found out about the event. The group wondered if we might use such a survey again and which outreach categories we should ask about. I learned that at last year’s Boil Down, where they posted a survey using a QR code, they only got two responses….Always eager to learn and improve, the team was ready to try something new.

I thought about the event, and all the hands-on activities that were planned (e.g., getting to touch and use some of the kinds of tools involved in tapping sap), and I wondered if we could use a physical survey. We quickly identified outreach categories/responses, and the team made jars for each category. As parties entered the event, volunteers gave them each a small tile, which attendees placed in a relevant jar, before engaging in other activities. As a result, we got 410 responses! (out of around 620 attendees, but for instance, my friend’s 6-month-old baby didn’t answer). We learned that friends and family/word of mouth was the most powerful source, followed by online event listings and social media. It was easy, hands-on, and an approach that matched the flow and vibe of the day. We’re excited to use this approach again.

By the way, the Growing Center is celebrating its 30th anniversary right now. If you like their mission, and feel so inspired, please donate $30 (or any amount) here.

Also, I will love to hear about any ways you use data physicalization. Reach out if you want to get crafty and make something together.

Happy Spring!

Our physicalized survey with some responses early in the day

Shhh...Listening and Using Scientific Tools to Solve Problems

Today, I heard a fascinating interview with Saul Permutter, a Nobel-winning physicist, on WBUR’s Here and Now radio show. Perlmutter discussed his new book that shares ideas of how we can all apply the scientific method to decision-making. He noted how Americans tend to have a bias toward valuing people with strong opinions.

Of course, this raised feelings, thoughts, and fears about political divisiveness, but it also reminded me of my work. I’ve noticed a lot of evaluation consultants positioning themselves as “experts'“ and that some who express strong opinions (e.g., “always do this” or “never ever do that”) get a lot of attention. I understand this—as a human with my own biases and attention span, etc.—and also because of programs’ and policymakers’ frustration with experts, particular those in academia. As scientists and academics, we are trained to consider nuances, limitations, and caveats. We get a question, and we say, “well on the one hand….but on the other….” There was a famous joke that I’ve seen attributed to different sources about some type of expert/academic having too many hands because they always answered questions with “on the one hand, on the other, on the other." Sometimes you just want a simple answer!

So, on the one hand, I appreciate how some experts in my field have cut through the noise with clear, compelling communications, and on the other…. (hahahahaha, see what I did there?!). But it’s true—I value expertise, and I am proud of all I’ve learned and can share from a 30+ year career in research and evaluation. And yet experts don’t know everything, and I tire of that positioning at times, especially when we need to listen more to voices who haven’t been heard. I aspire, as I wrote in a post yesterday, to communicate about research and evaluation and theory clearly, in ways that can be understood and used by a range of folks. And yes, I have strong opinions about some things (e.g., pie charts with too many wedges, right?!), and there are times when I want my expertise and experiences to be heard and considered. But on the other hand, I want to do more listening, and you might find, that when we work together, I’ll say, “There’s rarely one right way to do this kind of thing” and “you are the expert about your own program [or experience].” I hope I listen well, and together we generate multiple solutions that we can consider and test together.

To make a sound decision, take a meaningful action, or solve a problem — whether as individuals, in groups, or as a society — we need first to understand reality. But when reality is not easy to discern, and we’re not sure which experts to trust to clarify the matter, we adopt other strategies for navigating the clutter. We “go with our gut”; decide what we “believe” and look for evidence to reaffirm whatever that is; adopt positions based on our affiliations with people we know; even find reassurance in belittling the people who disagree with us. We choose to consult experts who tell us what we like to hear; or bond in shared mistrust of people providing or communicating the information that confuses us, whether they are scientists, scholars, journalists, community leaders, policymakers, or other experts. These coping strategies may help us get by in our personal or professional lives; they may provide a consoling sense of identity or belonging. But they do not actually help us see clearly or make good decisions.
Book excerpt: 'Third Millennium Thinking: Creating Sense in a World of Nonsense'

By Saul Perlmutter, John Campbell and Robert MacCoun

A post-it someone stuck on the window of my bus stop!

Translations and Little Passions

We all need more small delights, and I love getting to hear about people’s little passions, as you may have deduced from my recent post here.

One of my passions is yet another quirky podcast that is so full of joy and silliness that I wish someone, anyone?! I know would listen to it and talk about it with me…it’s Dr. Gameshow, hosted by the comedian Jo Firestone. Listeners write in with games they design, and on each show, callers (adults and sometimes their kids) play 3 of these games. A recent game was “Do You Know What My Roommate is Talking About?” Two listeners designed it based on conversations with their roommate who forgets words and uses phrases like “centipede car (it takes the children)” to mean other things (in this case, school bus!). Callers get a word or phrase from the roommate and guess what they were talking about.

I relate to this game as a lifelong “translator.” Growing up, my grandmother lived on the first floor of our house, and my mother and I lived on the top two floors. They did not get along well, and I spent a lot of time up and down the stairs, carrying and translating messages between them. Was this healthy for a kid? Big NOPE. But sometimes difficult experiences and the ways we learn to cope lead to some skills we can use in positive ways. (I also seemed to love interpreting the communications of my 2 year-old neighbor, when I was 5 years old, so maybe this was just something innate within me too). Anyway, across my career, I’ve been fascinated by and drawn to ways to improve science communication (check out the Alda Center for some cool stuff about this) and in particular how to translate research and evaluation for use among parents, programs, and policy makers (among others). This led me to my incredible Applied Child Development doctoral program at Tufts. Through an internship and part-time job at The Children’s Trust across 5 years of my program, I got to translate research and evaluation needs and findings between state legislators, academics, parent educators, and nonprofit leaders. I learned so much from everyone there.

For the past 16 years, I’ve been immersed in the world of AmeriCorps, first as an internal evaluator and then as a consultant to programs and state commissions across the country. I’ve been called a unicorn by one program officer because I know evaluation and I know AmeriCorps, and I can speak both languages. I’m still always learning, and I don’t want to be the rare unicorn!

If you know an evaluator who wants to become more familiar with AmeriCorps, I’m offering this orientation webinar on April 5 and would love for you to share this.

What are your small passions these days?

An image from Dr. Gameshow’s Instagram

Evaluation Inspiration is Everywhere: Sleeping with Celebrities

If you want to be a good evaluator, or to be really good at so many professions actually, be sure to keep your eyes open in the world outside of evaluation for ideas. Sometimes you’ll find inspiration and approaches to emulate in surprising places.

One of my favorite podcasts to listen to is called Sleeping with Celebrities. Most of the guests are not well-known celebrities, in my opinion, but that’s no matter: “Each week on the slyly humorous and reassuring Sleeping with Celebrities, host John Moe talks with a different guest from the world of entertainment about something they know a lot about.”

It's a show you can listen to as a relaxing escape or to help feel sleepy (“we don’t want guests to bring their A game, we want them to bring their Zzz game”). The host is from the world of comedy and throws in some deadpan humor I just love and that makes me suddenly giggle while sleepy.

What are these somethings the guests know a lot about?” Well among my favorite episodes guests have gone deep deep into such passions and topics as: porridge; describing Melrose Place; ten (surprising) favorite candies; how to build a home music recording studio; and a guy’s setlist of covers from his college open-mic days. (By the way, in one episode, I learned the shocking fact that carrots are bad for rabbits, and that we only think of carrots and bunnies as going together because of Bugs Bunny and the movie It Happened One Night!).

How does this relate to evaluation? We can apply a lot from this show to our skills in designing interviews and focus groups and how we facilitate them.

• Conveying curiosity, warmth, and empathy: John Moe conveys genuine interest in his guests’ sometimes most ordinary of topics, through his tone of voice, his questions and paraphrases, and another way he does this is by….

• Listening well and using meaningful probes: Although he has planned some questions, he listens well and asks follow-up questions that are truly interesting and detailed in response to what he’s heard from his guest.

• Slowing things way down and keeping them simple: Too many times, we try to cram too many questions into an hour-long (or even shorter!) interview or focus group. We rush through. Let’s cut our questions way back and leave space to breathe, slow down, allow for surprises, and to really dig in.

• Appreciating differences, quirks, and discovery: The topics covered are rarely something the host knows about (he had never seen Melrose Place, but he asked follow-up questions about characters and plot points his guest had mentioned with great delight). Isn’t this why we choose to interview the people we do?—we want to learn from them. Let’s keep our minds open to listen for new discoveries and insights, big and small.

Of course, these are good reminders for our everyday conversations and listening too. At one evaluators’ meetup, after I discovered this podcast, I found myself chatting with a colleague about his new hobby of fly fishing. Instead of rushing off from a quick chat with him to talk to someone else, I found myself asking more questions about fly fishing, and the more I asked, the more fun we had. (And, did you know there are free little flybraries!!! to borrow fly ties just like Free Little Libraries?! Mind blown).

Sweet evaluation-y dreams,

Gretchen

Data Inspiration from Geena Davis

This is about how Geena Davis’s use of data inspired me, and some of my takeaways that maybe you will find useful too.

I belong to a celebrity memoir book club through my local library (CelebriTea! We drink and spill the tea!). I’ve loved it—it gets me to read some fun books I might not otherwise read with a group of women from different backgrounds, who exchange thoughts and questions and ideas in a remarkably enjoyable and inclusive way. I don’t know how it happens, but I’ve never seen a group who takes turns so well, builds on each other’s ideas, and shares dissenting opinions so warmly and naturally. The stakes are low, of course, but we often have strong opinions, and the dynamic has been this way with members new and old joining, and varied sizes of attendance. I’d love to bottle the nature of these exchanges and be able to bring them to more spaces.

One month, we read Geena Davis’s memoir, which was fun and compelling. I was aware of her work to bring gender parity to Hollywood, but listening to her audiobook, I learned more details. I was struck by how she, who is not a data scientist or program evaluator, valued research and recognized that collecting data would help her learn and make her case to Disney and others. She invested in research, and her work has led to improvements, though gradual, in both roles and representation of women and girls as well as in measurement tools themselves.

You can read more about her work here, among other places:

https://www.bu.edu/articles/2020/geena-davis/ and this is the book we read: https://www.harpercollins.com/products/dying-of-politeness-geena-davis?variant=40404274806818

Inspiring takeaways/reminders

• Great research ideas come from the people involved in the issue (I hope you already know this!). Geena Davis was a woman acting in Hollywood, seeing roles diminish for women as they aged, and a mother who noticed the lack of female representation in what her kids watched. These experiences inspired her to take action and fund groundbreaking research.

• Results are powerful when shared within relationships/Don’t weaponize data. In her memoir, she describes bringing the results of their initial research to Disney animators, who were surprised and dismayed to see the unconscious biases they brought to their work. One animator noted that he had just drawn a crowd scene and filled it with all male characters, without even thinking about it. She comments on how these animators got into their work in order to bring joy to children and families. As quoted in the BU article, “I never bust anyone publicly.” Instead, Ms. Davis assumes best intentions and leverages a range of meeting types to share results.

• Iterate! She started her research center in the early 2000s, and now 20 years later, they’ve continued to create new innovations in science and measurement. Here’s an example of software they developed: https://seejane.org/research-informs-empowers/data/ . Before developing this software, most studies of representation had been conducted manually.

From this rainy winter day in Boston, I hope you are staying warm tonight, maybe with a cup of tea and a book, and finding some inspiration. The Geena Davis Institute on Gender in Media says, “If she can see it, she can be it®” If we can share examples of effective data use, we can learn and make the world better. Also, please support your local library and librarians!

Tiny Habits

While big moves have their place, I am often fascinated by working small and how tiny habits, interventions, and movements can lead to large impacts.
I loved the book Atomic Habits--it inspired me and gave me practical ideas to use.
When stuck on some big, hard tasks, I return again and again to the Pomodoro Method 🍅 to help me focus and break them down into smaller bites.

What are some ways you like to work small? #atomichabits #pomodorotechnique #eval

A bee hotel at The Somerville Growing Center with a QR code to learn more and make your own.