AMCHP 2005 ANNUAL CONFERENCE
DELIVERING RESULTS, IMPROVING PREGNANCY & BIRTH
February 19-23, 2005
KAREN BURSTEIN: Good morning. I'm Karen Burstein and I'm a part of this group in a way that's kind of interesting to me. I tend to service the evaluator and to find out how did the system work and what was the benefit and so what's it all mean? What's it all mean when at the end of the day when we put these things together and we've had activities and we've had a new project, such as the first responders, or we have a pharmacy in a place that wasn't there before or around a medical home or around all the six core performance indicators and one of the things that I think I've been fortunate enough to do over the last couple of years is kind of make a shift from being kind of a hardcore quantitative person to being able to look at some of the other more nuanced data that tells a different kind of a story. So today instead of talking about what are the kinds of ways that we go and quickly find how it worked and what worked I'd like to share with you a couple of other ideas that I've come up with and I'd like to share with you--I have a handout that's around here and it's--does everybody have my handout?
And I have a separate little handout of a form. I'd like to refer you all and I didn't--before I give this PowerPoint presentation over to AMCHP for distribution, I need to get a citation on it. I've recently been engaged with this work called Communication for Social Change. The Rockefeller Center published this document two years ago. It's a marvelous document. It's available online. I will have it in my handout as a reference for you, but it is well worth your time to find this document and to read about the history of social change and the impact of communication. It's truly a wonderful tool and some of the things that I'm going to share with you today, I will be citing as part of this. The guiding principles around social change: Well, the biggest thing it would be the sustainability of social change and it's more likely that any project is going to be sustained if it's owned by the constituents that implement it, so it's not an outside issue.
Change is an internal issue. Communities should be the agents and the evaluators of their own change and that's what happened very clearly with the First Responders Project. When they identified the issues they identified the things that they would measure across the process. Emphasis on the outcome needed to go beyond the individual's behavior and move to change in social norms, policy, culture, and the supporting environment. So the social norm went from the hysteria around the circus is coming to town to the social norm becoming now we all expect to work together as a team and that's some of the change that we need to be looking at measuring. And one of the most important things, that I believe, that I've been able to find out of my research of recent, is that no gap should exist between that issue that originally gets defined. The outcomes that get measured and clearly evaluation starts the day that the issue is identified and the two have to go hand in hand.
Evaluation cannot be a post talk thing that we do. Oops, we forgot to say, now the grants over so we've got to say what happened. It has to be measured all across the time. And when looking for change, and this is kind of my new mantra, instead of just looking for the big changes, some of the more subtle changes are truly very, very important and those changes in social norm are what I'm trying to get to now. So how do we measure this stuff? Well, we do a couple of types of evaluation. We do these things called formative evaluation and that's that stuff that we measure across the life of the project. Is it getting up and going this participative evaluation? Who are our leaders? Who are our participants? Who are our constituents? What are the goals in identifying? Do we have well articulated goals? Are our short-term goals are they measurable, are they observable? Do we have time lines and are the timelines realistic for us as a team? What activities do we implement in order to get to our big goal at the end? Then we do this thing called summative evaluation and that's big stuff at the back end when we say, what are the outcomes? Did our activities work? Did we conduct professional development and did the First Responders change across time? Can we see differences? Well, I would contend that, yes, that when we measure these people for the professional development that they undertake we will see differences.
Whether we use something like a process of change model to measure their change and the way they integrate new knowledge, depending on what we us, but will we be able to measure it and I contend, yes, we will. We look and pre and post tests. You're developing pre and post tests. We also have the satisfaction indicators, were the constituents happy? Were the First Responders happier? Were they able to do their job better? And do we ultimately live in a better community as a result? So then determine the impact of this project on a group. Well those are things that are this is this new data. This is this how did we change as people, so we measure the sense of community ownership. I would contend in your community it is well owned. It's not an outside project. It is definitely an internal project. We measure social cohesion and we measure these changes in the norms, participation, and leadership around specific issues. Who conducts the assessment?
Well, there's basically three people that conduct assessment. They're the members of the community, the people that are doing this, who want to know how well their effort met their objectives and they want to be able to say can I share these results with the rest of my community? So we self-access on the participation process. Secondly, they're external change agents. Lee's an external change agent. She's involved in the process of and needs to document this change to support to our funding agent so we can say, yeah we met the guidance in our grant. We said we did what we're going to do, that's a really important thing if we ever want to get funded again. That's when we look at pre and post levels of participation, we look at feedback that people give us. And then there's this third area, and I think I kind of fall into the social science area, and I want to conduct an analysis of the relationships between the process and the outcome and I want to find out how to inform other scholars of what happened. So is this a model that can moved to another community? I contend yes.
I think that based on the work that OCEAN's been doing and now the implementation that's happening out in the communities that these are ways that we can then move this to another level. And then I just want to make a quick comment about this thing called Participatory Evaluation. I've sent out some forms that I think might be an interesting thing for each of you to consider as you put together groups. How was the evaluation conducted? So when you think about how do you conduct this. Who participates in the evaluation? We all know we can make results do almost anything we want and you know those people that used to say, "Oh, I can make statistics say anything." I used to say, "Oh, no numbers never lie." Well you know, depending on who you sample, the numbers might not lie, but the sample can be really biased. So who participated in your evaluation? And how are you disseminating these results to a broader community? Are you keeping it in your own community? I don't think so. I heard that the big goal for this was what, Kim? What's the big goal for First Responders to go?
KIM STAMPER: We want to go nationwide. (Inaudible) statewide (inaudible).
KAREN BURSTEIN: Yeah. And ultimately what we ultimately do at the end of the day is we'll reflect on this process and figure out what did we learn from the process? So we have the outcomes from First Responders, but then what did we learn from the process? And I believe that that's something that OCEAN is very concerned about and that we are very concerned about, about how did this process work? Is the process individually dependent? Is it dependent on those individuals in the verve around that they develop around their goal, or can this be moved someplace else? And think that becomes a very key issue when we look at community development and what are those tenants that we can then replicate in other communities nationwide? And I think that's it.