Jay Rothman

President of the ARIA Group, Inc.

Interviewed by Julian Portilla, 2003

This rough transcript provides a text alternative to audio. We apologize for occasional errors and unintelligible sections (which are marked with ???).

The other process I have is called the action evaluation process. This came out of an effort to try to build an appropriate technology for the field in evaluation. Evaluation is a real problem in so many fields, and especially so in ours, where success is so squishy and each project is so contextual. How can we define success in ways that are relevant to our projects, but have some generalizability to our field? We began this project of developing a methodology called action evaluation about eleven years ago. It's turned out to be an intervention tool as well as an assessment tool.

And what we do with it is we ask people who are prospectively going to be in some kind of a project to define success as a hypothesis. We ask them three simple questions. What are your goals for this project? Why do you care about these goals? What are your values? And this gets back to the identity question. Who are you that makes your goals meaningful to you, that gives you passion about what you want to do in the world? Finally, given your what and why, how do you think this could happen and what could you contribute to making this effective? So this methodology is also incorporating the participation that's so core to our field, I believe, the sense of co-creation, the dynamism where we are making success be self-fulfilling by articulating what it is that it should become.

So a project that we had was with a foster care agency. Not a conflict project, but a visioning project. It was with Berkshire Farms Institute, one of the largest foster care agencies in New York state, a 15 million dollar agency with sites all over the state. We were doing a pilot 300,000-dollar project over a few years, a teeny little project in the scale of that institute, where we were trying to create a model program for helping the youth in the program transition into independence. It's a very treacherous time for kids who don't have family structures when they max out of the system. They can't stay beyond the age of 21, sometimes 18. So how do we help them do that? How do we help them become adults when they've been dependent kids, and they don't have structures to support them?

So, we used action evaluation, and what we did is we asked the three stake-holding groups to define success. The first stake-holding group was the foundation, and this is one of our theories in practice, is that we need to ask foundations who fund projects to be explicit about what they want, and then we can negotiate with them. If they're implicit and they don't articulate it, then the grant recipients have to always guess. What is it that I have to do to satisfy my funders? If the funders can be explicit, then the recipients can say, well, I appreciate those goals and here's how they match up with mine and I'll do that. Here's how mine diverge, will that work for you? Can I do that? So we have a real value in trying to get everybody to be transparent and explicit about what they want to accomplish in these conflict resolution projects.

So we had the foundation define success, and that was at this point only two people, the founder/executive director and the program officer. They did it independently. Each of them answered our questionnaire online, on our website. Answering what, why, how, up to three answers. We then do an analysis from wherever we are, because it's web-based, and we look for what's shared, unique, and contrasting, in this case between the two members of the stake-holding group. We then come back and we do a feedback session with them where we say, this is what we got from your data, that you two share these goals, you have this unique goal, and here's a goal that you two are going in different directions about, and we want to put this on the table. You might need to work on it, you might need to just acknowledge it, or maybe I didn't get it right.

Q: Is that the part about surfacing the tension, surfacing the conflict?

A: That's right, and making it available for them to make a decision about, which could be as simple as, "now we know it, let's move on," or "we better not move on until we figure out what we're going to do with this." But it's engaging it, making it so it's in front of us and not in back of us, so it doesn't hit us over the head when we're not looking. So then we did the same thing with the staff. In this case the staff was fifteen people at the foster care agency. What are your goals for this new program? Why do you care about these goals? And how do you want to accomplish them?

Then we did the same thing as we did with the foundation, this time with fifteen people, getting about 45 answers, three from each, using our database to redact them down to a half dozen goals that are shared, a few that are unique, maybe a couple that are contrasting. We come back to a feedback session with them. The first thing we do in the feedback session is we have them talk about their "why" responses. So we are grounding them in their own identities, in their own theories of practice, their own values.

Q: Before you talk about the what, you talk about the why?

A: Exactly, and usually their written "whys" are often times their "whats" in disguise. People are not used to talking about "whys." So there's a lot of coaching here. Why do you really care about that? That sounds to me like what you want, and that's really wonderful. Why do you want that? And you keep asking. One of my stock phrases is, and it's perhaps obvious and maybe almost a silly question, but why do you care so much about the kids becoming independent. It's often tearful, I care about the kids being independent because we've worked so hard to help build them up and I'm terrified that they're going to go out into the world and they're not going to make it. And, they're like my children. You know, it's very passionate, very powerful. And then having done that with everybody, and there's no debate, this is not about debate.

This is about everybody understanding where their colleagues are coming from, and they always learn about each other, whether it's the first time they've been together or they've been together for a decade as staff members. They learn some deep insights and it's real team building. They help me inquire, you know, help me ask questions to understand where their colleague is coming from. You don't have to agree, it's not about that. It's about understanding their reality. So we do that, and then we negotiate the "what" goals.

So I've taken their 45 "what" goals, I've redacted them down to eight or ten categories, perhaps, and then I say to them, now that you've had your value discussions, your why discussions, we want you now to reach consensus on the goals that you all share for this project. I think they came up with about five of them, and after the session they have this real sense of a team, they have a sense of values, and they're unified as the foundation officers are unified. Now we go to the youth, do the same thing. In this case there were ten youth. We do the exact same process.