AMCHP 2006 ANNUAL CONFERENCE
DELIVERING RESULTS, IMPROVING PREGNANCY & BIRTH
March 4-8, 2006
MARGO ROSENBACH: Well thanks everybody for sitting through and making it to the end. We’re in the home stretch and I’m really delighted to be here. Thank you very much for inviting me. I’m going to talk about a project that Jack introduced, our assessment of the MCHB performance measures. There is a handout in the back of the room. I hope everybody had a chance to get a copy and if not come up at the end and we’ll get you a copy through e-mail.
So first I want to acknowledge that is a collaborative team effort as most efforts in MCHB are. I wanted to acknowledge our project officer Jack Tenenbaum and also a group that he has assembled and facilitated the MCH Bureau Evaluation Coordinating Committee. And this is comprised of a number of MCHB staff, who meet with us on a periodic basis and provide input to our study. And I think without whose input the study would definitely not be as relevant and as useful. And then we also have a couple of consultants, Sue Ginsburg and Donna Strobino who many of you might know from their many years of working with MCHB and I’m also assisted by three staff members back at MPR, Solimpa Amarra, Sybil Day and Davine Wright. And they also are very valuable in this effort.
So let me just give you a very quick roadmap of what I’m going to do. I’m going to tell you about the goals and objectives of this project, the conceptual framework that we’ve developed to guide this project, our assessment approach and then hopefully we’ll have some time at the end for a discussion and questions and answers.
So starting with the goals and objectives. We started off this project with a few over arching goals and this is really an opportunity in some ways for all of you to step back and think about performance measurement more broadly. We started off the day today with some real detailed presentation about TBIS, which is really the foundation for a lot of the work that we’re doing. We’re also looking broadly at the discretionary grant programs as well. And so what we’re trying to do, as part of this project, is to assess the current performance measures, what’s working and what’s not working, and then to make some recommendations for improvements. And again I want to emphasize that this is very much a collaborative process. It’s not a project where some outside consultant is going to come in and tell you what’s working and what’s not. This is really a bottom up process where we’re engaging the program, and hopefully also grantees, to tell us what’s working and what’s not. And then we’ll develop a synthesis of what we’re hearing.
Now to give you an overview of the specific objectives we’re looking at the individual performance measures used by each MCHB program. I’ve mentioned the block grant, the discretionary grants, other MCHB programs, such as Healthy Start. And then the other thing that we’re doing and this is really, I think, part of Jack’s vision for this whole project is not just looking at the performance measures for each program individually, but also looking across all of the MCHB programs so that we look at the totality of MCHB and tell a story about how all MCHB programs are working. And again, I think as you’ve heard from Jack, the context for this is, in large part, budget allocations, resource allocations, in a really constrained environment. So what we’re trying to do is to anticipate GPRA and part assessments to really look at how MCHB as a whole is doing.
And as I mentioned, we developed a conceptual framework for this project and this was very helpful to us because, like many of you, we were fairly new to the performance measurement process within MCHB and really trying to get our arms around the total set of measures, the process that’s being used. And so, what this, I think this conceptual framework is designed to do is to give a sense of the context and the process that’s being used within MCHB for performance measurement, and also bring in the efforts that states and other grantees are using. So we refer to the MCHB Strategic Plan, which is a five-year plan that's developed by MCHB. And it, what we’ve tried to do is to highlight the inputs and the outcomes, the outcomes being both process outcomes as well as intermediate outcomes and then ultimately the long-term outcomes, which as you all know are harder to measure. Again this is a work in progress. It does reflect preliminary input from the Beck numbers, but we’re working on this and as you probably know if you’ve ever done logic models, logic models are never, ever final. It’s a very dynamic process. And for those of you who have the handout, I apologize that it looks microscopic within the handout. If you want a copy, come on up and I can get you a copy through e-mail. Okay.
So what I’ve done here is to layout the various components of the logic model, to roll that out so you can get a sense of, of what it looks like in its various components. So the first component is the performance measurement context. And you all know from the needs assessments that you do regularly, and I think as we heard, needs assessments aren’t just done every few years. They’re done on an ongoing basis. You have a variety of measures that go into the needs assessment process. The health indicators, Healthy people 2010 and many of you have individual state Healthy people 2010 legislative initiatives, priorities that come to you through your partners. And then, as was also mentioned, the external environmental factors that can influence MCH. And so then that leads to setting goals.
And those can often be reflected through a strategic plan. And what we’ve listed here are the five MCHB goals that are in MCHB’s strategic plan. So for example, providing national leadership for MCH, promoting an environment that supports MCH, eliminating health barriers and disparities, improving health infrastructure and systems appear, and assuring quality of care. And again, that comes directly out of the MCHB strategic plan.
And then that leads to allocation of MCHB resources and what we’ve listed here are the various MCHB program components. And then you have the outputs or the outcomes. And we’ve depicted it here in a linear framework. Obviously, there is an iterative process very often, but what we wanted to do was try and layout in a somewhat more ideal sense, not necessarily where we are now but, I think, hopefully, where we’ll migrate toward in thinking about process outcomes, which tend to be more oriented toward the health systems capacity indicators, intermediate outcomes, which would be the performance measures, and the long-term outcomes, which are health status indicators with the ultimate outcome being to improve MCH broadly. So again, looking across all of the programs, thinking about MCHB’s goals overall, the overall goal is to improve MCH. And what we know from having started off this process is that not all of the performance measures are actually intermediate outcomes. Some of them tend to be more process oriented. And as you saw when Chris was talking about some of the changes that were being made in the performance measures for this cycle, a lot of the effort is really meant to move more toward intermediate outcomes and less toward process outcomes, because again, what’s driving it largely is the resource allocation process, the pressures from PART and GPRA, which lead more toward focusing on outcomes.
And this last part of the conceptual framework shows a set of arrows leading back toward the performance measurement context. And again, what we’re trying to emphasize here is that this is an ongoing process. There’s a need to periodically review the measures. Again you saw that, that something that was done for the Title V block grant measures and to keep feeding in what we know about the context.
So let me tell you a little bit about the approach that we’re taking. Again, the scope is broad and it’s actually broadened quite a bit, I think, since we started this project and we learned more about the system of measures. So not only are we looking at the Title V block grant core measures, the 18 core measures, but we’ve also started looking at the state’s specific measures. And we’ve classified them according to the primary focus of each of those measures. So we’ve tended to look at how many states have obesity types of measures or nutrition and physical activity measures and so on and it’s very broad. And I think what we see is that this is reflecting what state priorities are and could, I think, help us to step back and figure out how responsive the current performance measures are to state priorities and state assessment of needs.
For the Title V discretionary grants, there are 36 core measures and some of the programs have designated additional data elements. One of the things that we’ve learned very early on in our assessment is that we have some programs without measures and some measures without programs. So I think that will be very useful to try and look at which programs need measures and then where the measures are not being used, does that mean that they’re just not relevant, they’re not useful or should some of the programs be adopting those measures. And then we have health systems capacity indicators and health status indicators, which we’ve already talked about. And we have some counts in here but it tends to be a moving target particularly as some of these are being switched from category to category. And with the A’s and the B’s it’s hard to know how to count, but this is the current count as we have come up with it.
So the four components of this project are first to conduct a lit review of the evolution of performance measurement. And I think that’s been put in the context of GPRA and PART, again with the evolution to establish criteria for assessing the utility of the performance measures, to conduct a comprehensive assessment and then to develop a process for periodic review. This is a two-year project. We started in October, so we’re about five months into the project. And so at the point we’re trying to wrap up the lit review and the establishment of the criteria and we’re just beginning to conduct the comprehensive assessment.
So let me give you a very brief overview of the lit review. The first step in this was to synthesize the Health Services literature to provide a theoretical basis for the assessment. And I think what we have found is that most of the literature is about 10 years old. So in 1993 or so, when GPRA was coming in, there was a lot of focus about performance measurement and, you know, in the theoretical sense what programs should be doing, how agencies should be structuring themselves, but very little since then. A little bit more around the time that PART came in, but not very much. And I think as most of you would probably recognize, it’s not published in the peer-reviewed literature. So a lot of it is in the gray literature, particularly from Washington think tanks and from government agencies.
The next thing that we’re doing is providing an overview of how government agencies use performance measures. And here again not a lot has been published. So the synthesis is trying to get a sense of how it’s being used, but again, there’s not that much to report on.
Next, we’re describing features of effective performance measurement systems and I think what this has done is it’s really helped us to think about the MCHB performance measures as a system, because I think when you look at each of them individually, you don’t really get a sense of that totality. So we’re really trying to think about it as a system to represent what MCHB is doing.
And then finally we’re summarizing the criteria that are typically used to select performance measures. So here’s a list of the preliminary assessment criteria. And I think the first category, the usefulness for program planning, is very consistent with the title of this session, of Making MCH Data Count. So I think in terms of usefulness for program planning, resource allocation, budget justification, program improvement, those are some of the uses that we’ve seen in the literature and so we’ll be looking at whether the measures are relevant, you know, whether they’re linked to the program objectives, whether they’re accessible, meaning are they interpretable to policy makers and are they responsive?
Do they meet the needs of GPRA and PART and other assessment efforts? Data availability and quality, I know this is probably something you struggle with all the time. Are the data complete? Are they accurate? Are they comproable over time or across states? Are they reliable and valid? And then finally, again something that I know is of great importance to you all is a level of burden. And not just with data collection but also data processing, for example, if you have to link Medicaid data and vital statistics data that can be very challenging. So looking at level of burden on both of those dimensions.
So the third component is to conduct a comprehensive assessment of the measures. And we’ve just started getting into this component of the project. And the kinds of things that we’re going to be looking at are what are the strengths of the current measures, what are their weaknesses and what are their gaps? How well do the current measures capture program outcomes? And what we’ve been hearing from some of the programs is that they don’t really capture program outcomes very well. And this can be very important when you’re going into a PART assessment, to make sure that the outcomes that you’re trying to measure are really the outcomes that are linked to the program. Otherwise, it can be fairly problematic, I think as Jack mentioned in terms of justifying the funding. And I know some programs have tended to be zeroed out for not having justifiable outcomes. We’re also asking programs how the current measures have been used? Some programs, particularly the discretionary grant programs, are not as far along in using the measures because they’re relatively new. And so I think what we’re trying to get a better sense is how the measures can be used and how we can we make them more useful?
And then finally, what activities are underway to assess existing performance measures? Some of the programs have little effort underway and some are actually looking quite closely at their measures. And so, what we want to do is build on the foundation of what these programs are doing so that we make sure that it’s part of our assessment as well. We have a variety of sources that we’re going to be using for the assessment, interviews with MCHB staff, which we’ve already started, meetings with grantees and other stakeholders. This would be an example of a meeting with Title V agencies and we hope to have many more meetings with other grantees to get a sense of how the performance measures are working. I know we won’t have time today to talk about that, but hopefully some other time we will. And then there are a variety of secondary sources. The HRSA guidance, the Title V needs assessment, the TBIS and DGIS and then data used to populate grantee reports.
Now the fourth component is the periodic review process. And again this I think was part of Jack’s vision for the project. And as many of you know, Jack will be retiring in another month and a half or so.
JACK TENENBAUM: Two months.
MARGO ROSENBACH: Two months. He doesn’t let us forget. But I think, I hope, I think this could well be his legacy, or one of his many legacies within MCHB, is really trying to get the agency to be thinking about how two periodically review the performance measures. This is not a stagnant process by any means. I think that for the block grant the process seems to be fairly well established for the core block grant measures. And Chris did talk about some of the changes earlier today. But our goal is to develop a similar process for reviewing and updating the discretionary grant measures. Again, to respond to GPRA and PART assessments and also to address emerging MCH needs.
And some of the components of this periodic review process would be the assessment criteria that would be used, how to get stakeholder input again, don’t want this to be a top down process. Want this to involve a bottom up process. How to field, test the changes and then thinking about this in the context of a three year timeline for review, given that the OMB process generally is on a three year cycle. Wanting to make sure that the process starts late enough that there’s been enough experience using the existing performance measures and, but early enough to allow enough time for developing and vetting the performance measures.
In terms of our future plans, we’re going to continue to conduct the assessments with MCHB program staff, review current performance measurement data, provide recommendations for improving the performance measurement system. Again, this will be a very collaborative process, not something that we would do without substantial input from MCHB program staff and stakeholders. Propose a process for the periodic review of discretionary grant measures and then make sure that on an ongoing basis we obtain feedback from the Beck members and other stakeholders as appropriate. So with that, I’ll open it up for discussion, comments, questions?
UNKNOWN SPEAKER: Questions for either Margo or Vivian. There might be none.
MARGO ROSENBACH: It’s getting late. Thank you all very much for your time.