MCHB ALL GRANTEE MEETING

Six IOM Teams Present Highlights on

Critical MCH Issues and Recommended Strategies

October 4-7, 2004

RICHARD ROBERTS: Our Institute of Medicine item was Evidence-Based Decision-Making and Measuring Success. And what’s so striking, I think, about what I’ve been hearing up to this point, is that you could take many of the points that were made by all of the other groups, and just put data into there instead of whatever word they might be using and still the issues are very, very similar. Really the issue is systems-building and using all of the parts of a CQI model to get that system to work the way that it should. And that was the issue that our group was grappling with as we worked. It was really quite a great group. I mean it was just so much fun going from one session to the next and listening to the conversations and spirited debates and grappling with some issues that for many of the people may not have been as top on their minds as it was as the session went on. I’m reminded of a poster that I used to have in my office when I was a graduate student in Clinical Psychology at the University of Hawaii, and it said, “Just because you’re paranoid doesn’t mean they’re not out to get you!” And sometimes when we think about data, that particular sentence applies.

So when our group began I think that they were very, very cautious actually within the context of, “Well, what is evidence-based decision-making mean? How is this going to change what we’re doing? Why are people making us do this now? Why are we thinking about these kinds of issues?” And that lasted for about a minute and a half, and as the conversation went on they became very, very engaged in the way that data matter, and how it’s important to the overall process and for what they need to know in order to be able to do their jobs. So, there was a real, very quick paradigm shift as people kind of really “got into” the topic and started to work with it. One of the things that people recognized, I think, as they started to talk about data and that little bit of paranoia that was creeping in occasionally, had to do with the way that data are used. And as you all know, there are data and then there are data; and it’s really how it’s played together in an overall model of continuously proving a system that became kind of one of the themes that everyone spoke of. There are cultural groups who have had data used both to their advantage and their disadvantage, and the issue of cultural competence in data management is no less than any of the other areas of systems development and very important, and something that we don’t know very much about.

So the overall plea, or the overall finding I think of Work Group Five in its three parts was that this really had to be a learning community, within the MCH community, on how data actually work within our system and how we get it, how we use it, and how we feed it back to other people. And that it had to be not so much as edicts coming down from the top, but rather that all of the community would embrace data as a way of the decisions we make and the way that we understand whether we’re on track or not. And that the partnerships be formed at each of the different levels, both vertically and horizontally to make data an important component of the whole system. And that many of the recommendations had to do with how that partnership could be developed so that they weren’t feeling like the data could be both honestly placed out and could be used to better the system as opposed to it being a hammer. And with that as kind of a start, let’s go into the recommendations of the groups. The first group, one of the critical issues was the disconnect between performance measures and outcome measures and the resources required. That is—(could we slip back to that first for a second? I’m sorry.)

What they were talking about was that it was sometimes the understanding at a programmatic level between the performance measures and what that’s supposed to lead to in outcomes, and the amount of resources that are required both to do that work, that system change work, and to collect the data to see whether you’re on track. There’s a big disconnect between those parts. That is that there’s dollars many times just enough for doing the systems change activities or the work of the day, and not much is left over frequently for being able to measure what you’re doing and how you’re doing it. Okay. So that one of the things that they would find helpful from the Bureau in this respect is to clarify and enhance the guidance in RFP’s that people respond to and programs respond to, so that it’s very clear the way data should be used within the system. And what types of resources should be going toward the data collection in evidence-based models over time. The second issue had to do with leadership and technical assistance support. The recognition was that at all of the different levels within the system, there are still major gaps within our understanding of how to use data, how to use it appropriately, and how to use it again in this Continuous Quality Improvement model to make the system work better.

So both leadership training in that component and technical assistance and support to programs on how data could be used appropriately and what this whole model looks like. And how it can be felt at the community level and at the state level becomes a very important issue. It wasn’t a matter of not wanting to do it; it was a matter of not knowing how to do it sometimes and needing some support at each of the levels of the system so that they could do it well and use the information in ways that are supportive of families and programs. Okay. The second issue had to do with... I’m sorry, and I forgot to say this! The two very hard-working people who kept corralling Group One and keeping them on task and getting them to the end were Debra Roebuck and Joanne Dotson. In Group Two, Marilyn Seabrooke and Amy Brynn took that role on and (inaudible) the fact that, or not. We all got somewhere, never quite sure where. A critical issue for Group Two was the lack of definition or process for applying evidence-based decision-making in MCH. One of the conversations that most of the group started with was, “How does evidence-based decision-making fit into the model and our understanding of what Maternal Child Health and its programs are about?”

And that was a very vigorous conversation in all of the groups, and getting a sense of the connection between the two, and that’s an incomplete discussion at this point. People still have feelings about the connectivity between data, evidence-based decision-making and kind of the “warm” feeling that MCH has as a program, and not wanting to lose that component of MCH as they think about using data and using measurement strategies to measure our progress along the way. In all of our work at the Early Intervention Research Institute around these kinds of issues, we’ve found them to be very compatible. But you really have to get in there and get your fingers dirty on both sides of this in order to kind of feel that they can fit together; and many people have not yet had that experience. So, the lack of definition or process for applying it, like, “How is this going to work? What is it, and how does it look in my daily activities at the community level, at the program level, at the state level when I’m working with a family in a clinic? How does this work and how does it feel to me to have that as part of what I’m about?

Is it going to change me? Is it going to change the way that I do business?” And of course the answer to that is, “Yes.” But hopefully, it will be in a good way. So, I’m sorry, let’s go back. Let me give a couple more examples of Number Two. There were no little things on the back? Okay. So the feeling was that evidence-based decision-making needed to be defined in the MCH context, and defined in ways that again have consensus across all of the different levels within the system, and that that process is one that, though it will take some time, will serve the MCH community very well in the long run because then people will be on board in a different kind of way as this is put in place or as this continues to put in place. Okay. Critical issue number three, which was led by Rich Havaguarte and Joyce Brooks: No uniform standards for data collection in Maternal Child Health, and no protocols for interpreting the data. So again, feeling not so much that they couldn’t do it, but that they needed the structure to be built in such a way with them, so that they were able to do this appropriately and do it well, and use it in a way that can move the system forward. (Is there anything I haven’t covered? What happens if you flip it? Okay, good.

So some of the strategies that came forward in Orange Three had to do with coordinating communication within Maternal Child Health, which will lead toward joint decision-making among all the key players. Again, that issue of a participatory approach in putting this into place as opposed to a mandated dictatorial one. Secondly, MCH should establish relationships with external groups to share data and foster relationships, so it’s not the data, the information that we collect, in measuring our success should be also those that we share with our partners, so that at each of the different levels so that we’re able to tell the story of MCH in a very efficient way. Which is one of the things, of course, that data do so well. That Maternal and Child Health should establish guidelines for interpreting the data so that people understand again the need for what data mean and how you can use the data in your programs. I think that one of the major themes that kept coming over was, we just need to know how this fits into what we do and we need some help in getting that to be a good comfortable fit for us at all of these different levels. How do we use data? How do we interpret it? What’s the right way? What’s the wrong way to do that? What are the limits of what we should be doing with it? And getting a good feel for it in ways that many people do not have at this point.

And then establishing guidelines for the data collection, that is so that there is some uniformity and that the data can be pooled at several levels and that it could be used in ways that show the whole picture across the country with respect to what we’re doing and how we’re doing it. Is that the end? Yeah. So the bottom line from the Evidence-Based Decision-Making and Measuring Success Group was that this is a very exciting time and that it could be something that really helps the Maternal Child Health family in moving forward. It’s not quite clear to many of the people how to articulate some of the things that we’re doing now and that we need to kind of start the conversation in a more significant way as to how that happens, so that the parts of it are felt cleanly and that people feel very confident in being able to do it. And that they’re hoping that the conversation is one that’s really a bilateral one and that people have an understanding and have an opportunity to have input into the way that the systems are put into place so that they’re most useful for them at their family, community, state and regional levels.

There was one point, that people were really bemoaning the loss of the regional centers. They were thinking, “Gosh, this would be a great thing for the regional centers to help us on.” So, that was one issue that they asked me to bring forward, and that was the Orange Group.