MCHB EPI Atlanta Conference
 
December 5 - 7, 2006

MCH Epidemiology: Mapping the Future

DEBORAH ROSENBERG: Well, for those of you who don’t know who I am, as Bao-Ping said, I’m Deborah Rosenberg. And I’m here to talk to you about a project that we’ve been doing at University of Illinois at Chicago with some help from others including Roger Rochat from Emory University. So my colleagues are Amy Herman-Rullof, Joan Kennelley, and Arden Handler. Two of the three authors of that “Definition of Effective MCH Epidemiology,” that Bao-Ping showed from 1999.

So the first and maybe most important thing is many of you in this room were part of being interviewed by us. We have interviewed all 50 states in the District of Columbia, and we have no missing data. So the biggest thank you goes to the states in D.C. for participating with us in this project. And then, of course, our funders, CDC, as well as being a data partner with us, and HRSA, and CSTE, and AMCHP, and Florida State University. And then we also have put together an advisory group and a reviewer group of folks out in the field from both state and local health agencies, the Federal Government as well as academe to help us with what we think is a really hard and complex but really important project that we’re doing. So thank you to all of you.

So what is our project? And as I say, we’re funded through a cooperative agreement; that’s one of the ASPH-CDC cooperative agreements. And our goal was to delineate alternative pathways to effective MCH Epidemiology, and what this means to us because Bill even talked about -- there’ve been lots of other capacity assessment initiatives, and what we see is a little bit different about our project is that we are really focusing on organizational competencies. So rather than the competencies of the individuals who are the MCH epidemiologists and others who work in the field, we’re focusing on organizational competencies, or what we call the MCH Epidemiology effort.

And we’re also, being epidemiologists, trying to do this in a multi variable way. So we would like to be able to assess the joint impact of many factors that we’ve collected in our data set.

So I am also going to ask more questions, then I’d provide the answers because we really want you guys to come up to the microphone when the three of us are done and have a dialogue about this.

So here are some of the questions we’ve been asking ourselves as we work on the project. Is it essential to have a named MCH Epidemiology unit in order to have effective MCH Epidemiology in a state? Is it essential to have a lead MCH epidemiologist? Is it essential to have a CDC assignee? Is it essential to have direct access to raw data, and to every data set, to only some data sets? Is it essential to have linked and integrated data? And is it essential to do sophisticated data analysis? And then we could add to these questions, but we’ve been trying to go at this, really challenging our own biases because we all have them, about what makes for effective MCH Epidemiology.

Here’s our over arching conceptual model, and you’ll recognize it because it’s really based on the (inaudible) model of structure process and outcome, modified also by Turnock and Handler specifically for Public Health practice.

So we start with the big picture, the macro context, and you can see that we’ve divided out -- for structural factors, the structure of the state health agency overall, and then separately the structure of the MCH Epidemiology effort. And then within that, we move down to process which is the work that gets done, the activities, what people are actually engaged in. And then we have outputs, or the results of that activity. And then, actually, our advisory group kind of suggested this to us, that we add intermediate outcomes before we even go to health status outcomes. Those being much -- things like, for instance, has legislation happened? Has change occurred? Have policies at the state level really been affected by MCH Epidemiology?

So what is in the data set that I’m going to begin to report to you? Some very preliminary results today. We have a really rich data set and what that means is we have 2,200 or so variables, and even with our no missing data, we have a sample size of 51; the 50 states and the District of Columbia. So this is a challenge for us in terms of doing the analysis. We did, as you know, primary data collection. We talked to all 50 states. We also asked every state in the District of Columbia to send us a packet of documentation that was really about the outputs of their MCH Epidemiology and some other materials, but primarily about outputs. And then we have lots of secondary data as well. We have data from CSTE. We entered into a data sharing agreement with them. We have the 2002 MCH Epi capacity assessment, and we have the 2004 version of the ECA; Bill is telling us that the 2006 version is going to come out soon. We have the 2004 data. We have the Title V information system data, thanks to HRSA. We have some data that was -- I've given you sort of a half citation here. An article appeared in the American Journal of Public Health in January of 2006 by Les Beitsch et. al., and they graciously shared their data with us that is really about the functioning at the state health agency level, not about Maternal and Child Health Epidemiology; so we have those data. We have data from this conference, like some of the data that Bao-Ping was showing you, which I have a beef with what Bao-Ping showed you. Anyway, we have -- well, as epidemiologist, the abstract submission has gone up because the attendance has gone up, so we want to have rates of abstracts submission, don’t we? Okay. Which we have created in our data set.

So we have CD assignee history data, thanks to Roger Rochat, who really has helped us -- who really has put together for us all the historical information about which states got assignees, when, for how long, Fellows, all the sort of different mobilization of resources and infusion of resources that states have gotten in terms of personnel. We also have some data about who goes to training; in particular the training that AMCHP sponsors before this conference, and also the national training course in MCH Epidemiology that HRSA supports. And finally we’ve, of course, used some macro context data, census data, et cetera. So it's a big, big, unwieldy data set.

So for today, what are we talking about in terms of analysis and what are we going to use as an outcome measure? And so our outcome is effective MCH Epidemiology. What is that?

I want to tell you where we are now. And of course we want to hear from you in terms of maybe how we can do this better. But we did two qualitative assessments. Our study team assessed the phone interview we did with each state in D.C., and each member of the study team did this independently; so we had four independent reviews of that data. And then the states also sent us material, as I mentioned; and we did another assessment, the study team did. And for those documentation packets that you mailed to us, at least two of us reviewed each one of those. So there were two assessments and multiple reviewers of each assessment. And what we've done for today is created an average across all of those assessments and we've actually -- from that average to find a three-level ordinal variable that we're going to be using as our outcome measure, some measure of level of effective MCH Epidemiology.

This is really small but we wanted to give you a sense of the pool of independent variables we're working with. And you'll see most of these in -- create the ones that we want to highlight for at least right now on the subsequent slides; so I apologize if you can't read all of this.

But also I want you to notice that we've shown you some of the macro context variables, the usual suspects, population, population density, whether a state had lots of what are designated as frontier counties as sort of a measure of rurality. We have other measures of population of the capital city, et cetera. We have lots of structure variables; things about how's the agenda for the work of MCH Epi set? What's the level of the MCH Epidemiologist working in state health agencies? Who’s getting training? Are there barriers? Those kinds of things. And for process we have, what kinds of analysis do you do? Do you do data linkage and integration? And what you notice here is we haven’t even listed the outputs or the intermediate outcomes because those are really more going to be used as validation for our qualitative assessment of what is effective MCH Epidemiology. And one thing I was going to say, for instance, we’ve had some interesting discussion in the study team. Bao-Ping had the slide about abstract submission to this conference. So just think about, is submitting an abstract to (inaudible) meeting, what is that? Is that an output that’s a result of having an effective MCH Epi program? Or is it a process? It’s part of the activities of having an MCH Epi program? Should it be an independent variable? Not be an independent variable? Even attending this meeting; is that a marker that an effective MCH Epi program already exists in the states? Or is it the way you build that program by sending junior staff here, for instance? So we struggle with these issues all the time. Okay. What are -- I put limitations right up front because we have a lot of them and I want you to think about them and talk to us about them. So one is, I already mentioned, the small sample size. We only have a sample size and we’ll never get a bigger sample size than 51. We have lots of multi co-linearity because, of course, we’re measuring (inaudible) versions of all of these features of -- to try and get at what is this subtle thing called effective MCH Epidemiology. So we’re having to figure out how to handle that. What about missing data? And, in fact, today, we’re only going to be reporting results from our primary data collection from the TVIS data which you all have to fill out and the states have to in terms of submitting the Block Grant application, and the data about the MCH assignees and, for instance -- but, for instance, we’re not right now incorporating the CSTE data because as Bill mentioned, the end there is usually around 37 or 40; and so for right now, we’re just going to present you where we don’t have big gaps in the data. Then I wanted to talk to you about some other analytic issues, potential misclassification. What does it mean if a state describes to us on the phone, for instance, its MCH Epidemiology effort as diffuse. Well, that could mean that it’s a really strong, coordinated, collaborative effort, there are lots of people involved in the state agency in doing MCH Epidemiology and they’re coordinated across the agency, but it also might imply the opposite of that. It might mean the program is weak, there’s only a few individuals working in isolation, and the word diffuse might be used to describe that too. So we’re also struggling with that issue. Another example, what does it mean if a state describes the level of staffing? We asked about how adequate is staffing to do the work of MCH Epidemiology. What if we get the answer inadequate or what if we get the answer adequate? Well, we’ve realized that adequacy is sensitive to a state’s expectations; already what you all have think about at least implicitly, what is effective MCH Epidemiology? So what do you consider the (inaudible) and depth of the work? What are your expectations for what you should be doing? If you have really high expectations, you have a very broad vision of MCH Epidemiology, then there might have been overreporting of inadequate staffing because you want to do more, and more, and more, and more that you think is essential to the work of MCH Epidemiology. If you have lower expectations, a slightly narrower view of what effective MCH Epi is, maybe there was overreporting of adequate staffing. Yeah, we can do this work because what we view is the work we have to do; we have enough to do that. Another analytic issue, complex directionality I’ve called it. And I’m going to just read these to you because I want you to answer these questions for us. So does institutional vision and (inaudible) promote effective MCH Epi leadership, or does the presence of MCH Epi leadership promote effective institutional functioning, or both? Does the state of the art technical environment promote more sophisticated data analysis, or does the recognition of the need for more sophisticated analysis promote development of the required technical capacity, or both? Does the quantity and quality of the workload determine staffing levels -- back to staffing, or do staffing levels determine the quantity and quality of the work, or both?

One more analytic issue and then I promise we’ll get to a few results.

If there’s no association between a variable that we’ve collected and the outcome measures we’ve defined it of effective MCH Epi, what’s the interpretation? So, of course, there could be measurements here and misclassification, and I’ve just gone through some examples of that. There might just be no variability. And this doesn’t really tell us if there’s no variability across the 50 states and D.C. Is that because there’s low capacity or high capacity? If everybody says we have adequate staffing, what’s the -- and so adequate staffing doesn’t predict effective EPI, but what’s the interpretation of that? Of course lack of statistical power, we’ve talked about that already. And then of course there is the variable just not in the pathway to effective MCH Epi.

So here’s a couple of examples. And you don’t -- I don’t even -- you don’t have to look at the odds ratios or anything. It’s just an example of no variability. So one -- we said, “How often do you do -- do you produce -- do descriptive analysis and produce descriptive reports?” Well, of course, basically, everybody does that. So there’s no variability. It’s not associated with effective MCH Epi. It doesn’t have any discriminating power.

Conversely we asked questions about barriers to hiring both senior and junior EPI staff. And basically, everybody does have barriers of one type or another, so there’s no variability. So we don’t get that having any barriers is associated -- or in this case not having barriers because no one didn’t have barriers, it doesn’t look associated with effective MCH Epi. So we have to be really careful with how we interpret our results.

Okay. So now you are looking at a little matrix that we made. And the rows of the matrix are also another assessment that we got from our -- the primary data collection from the interview we conducted with the 50 states and D.C. And we classified states as having either an underdeveloped to partially developed, or a highly developed MCH Epidemiology effort. So the effort is a whole. It wasn’t about the individuals per se in the effort, but our assessment of the MCH Epi environment itself. And the columns are three-category measure of effective Epi.

So what we would expect is that states with high capacity, a highly developed MCH Epi effort, are going to have effective MCH Epidemiology. And states with underdeveloped capacity, an underdeveloped MCH Epi effort, are going to show less effective MCH Epidemiology.

And so I’ve highlighted for you the diagonal where we would expect concordance between these two measures. And as you can see, most of the states and D.C. -- I actually don’t know where D.C. is in this. It could be in this -- are on the diagonal. So there is concordance between having a highly developed or a more developed MCH Epi effort and evidence of effective MCH Epidemiology at the other end.

But you can also see that not all states or D.C. fell on that diagonal. So my questions are, “What factors keep some states from fully utilizing capacity?” In other words, some states who are in the lower left triangle here, have a highly developed MCH Epidemiology effort, but they still didn’t look like they were doing effective MCH Epidemiology. And conversely, what factors enable some states who it doesn’t appear to have a lot of capacity, and are doing great stuff, and are very effective in their MCH Epi work?

More questions for you to help us answer.

And then this also came out of our meetings and our discussions with our advisory group measures -- members. What about translation? So we all know EPIs, we should talk about rates and proportions and do good analytic work, and blah, blah, blah; but what about translation? And one of our -- I think it was a conference call with our advisory group. There was a lot of disagreement about this. Yes, MCH epidemiologists also have the role of translating data, and others said, “No, MCH Epis do the work, they do a good job, they provide maybe some interpretation, initial interpretation,” but then there’s what got called -- what was being termed “the hand-off.”

MCH Epis hand the data off to the policymakers or the program folks. And those are the folks that do translation. So then we ask the question when we talked to the states, “Is translation a function of the MCH Epi effort?” And you can see that 41 out of the 51 states and D.C. said, “Yeah, most EPI staff should -- most or all EPI staff should be doing translation.” Another nine said, “At least certain select EPI staff should be doing translation.” And only one, I’ve no idea who it is, said, “Translation is not a function of the MCH Epi effort.”

Well, if that’s true, it looks like there’s pretty good consensus that translation is part of MCH Epidemiology. Well, if it is, is it actually being done? And in the bottom table here we get a different picture where 21 of the 51, or 41 approximately percent, states said, “Well, they’re not really doing much translation work.” And I was asking Amy this morning, we’ve looked at the Bivariate Association here too, and it’s not as though all those nine states and the one who said it shouldn’t be part of MCH Epi or part of the 21, these two are actually not correlated so -- or they’re not highly correlated, I should say. So lots of states who are saying, “Yeah, it’s part of our jobs to do this work,” aren’t doing it. So the question is, “Why aren’t they doing it?”

Okay. So here’s a summary of what you’re going to see on the next two slides. We thought it would help in just walking through this. We’ve begun to do some model building. The fact there’s six variables in each of these models I’m going to show you is somewhat arbitrary. We have small sample size, but the whole point here is to have a multivariable perspective, so we didn’t want to just put two variables in the model. So this is very preliminary work here. I can’t emphasize that enough.

So what we’re showing you today is two six-variable models. We’ve done cumulative logistic regression because we’ve got a three-category outcome variable of affective MCH Epidemiology. And what we want to highlight for you here is that in both models -- there are common variables in both models and we’ve called them the core variables here. And that includes a way that the MCH -- the agenda for the MCH Epi work is set, and that’s collaborative and by a consensus building process whether or not there’s actually a named MCH Epi unit. And also where there’s some mobilization of other resources. So looking at whether there’s interns in Fellows that have been working in the states along with the regular state staff, that also keeps showing up no matter how we run these models.

And then we’ve tried to put some categories here. And these categories are meaningful. So you can see personnel, data infrastructure, and technical capacity. Because we have a lot of variables, what our next step is -- we wanted to show you the actual variables today. We’re going to do some methods such as principle component analysis or factor analysis to try and reduce the data burden and kind of define dimensions that are part of -- that are predictive of effective MCH Epi. So these might end up being the labels we put on those dimensions, personnel, data infrastructure, technical capacity.

And you can see that we have two slightly different -- we can construct models that put together slightly different variables in each model. So pathway one has adequate personnel to do data collection. There was variability on states that said, “Yeah, we have enough staff to do data collection,” not just data analysis. Or we asked several questions about adequacy of staffing. That only appears in pathway one. And likewise below that you can see having CDC assignee actually appears in pathway two, the way we’ve defined it so far, but it didn’t necessarily appear in pathway one.

So this is the summary. Let me just show you the actual modeling results. And be aware here that we let variables be -- we built these models and let variables stay in them even with P values greater than 0.05. So you’ll see P values as high as -- in this model we got 0.18. Because again, we’ve got really low power, really small sample size, and we are going to have to address this; but at the stage we’re at right now, we’re not throwing out variables if they don’t reach P less than 0.05.

So here’s a potential pathway constellation of factors that might lead to effective MCH Epi. Again, having a named -- the first three are those core variables having a named MCH Epi unit whether it sits in Title V or it sits in the statewide Epi unit, that’s not so important; but a named MCH Epi unit. Agenda set by consensus building, and this was on a scale. So it was asking things like, “Who sets the agenda for MCH Epi? Is it the Title V director all alone? Is it the Title V director with just the senior MCH Epi? Is it those two plus other MCH Epi staff?” All the way out to, “Is it a broader ray of folks working on MCH Epi and external stakeholders?” So again, we have an ordinal variable here about how strong the consensus building process is. Adequate staffing for data collection, data sharing is being expected and data linkage being routine. And by the way, that variable came from the TVIS data where you all have to answer in the capacity indicators. In TVIS, do you have access to and it’s everything from, of course, vital records data, but to medicate and hospital discharge, and PRAMS, and newborn screening, et cetera, et cetera, et cetera; and MCH Epi staff that participate in the actual technical work of data linkage when that goes on.  So you can -- right now, to me, the point estimates and confidence intervals are interesting, but this is just the beginning.

The second pathway has those same three core variables, but then it has -- having had a CDC assignee. Historically, it has access -- I’m sorry, this is the TVIS variable. I misspoke on the previous slide. This is the one with the TVIS variable and it -- what was on the previous slide? What did I miss? I guess I just added a variable without realizing it to our model.

So okay, the TVIS variable about access to data is in this model. And then the frequency of doing multivariable analysis is in this model, but not in the other. So we’re starting to ask questions of sort of necessary and sufficient. What’s necessary, what’s sufficient, what different variables should be in the models?

So, I’m done. I know I’ve probably taken up more time and I’ve been the one bugging my co-speakers about not taking a lot of time. So this is just the statistical analysis.  I’ve told you upfront what all the limitations are and we’d love to hear ideas about helping us with the analytic work. The statistical analysis will inform development of alternative pathways, but the statistical analysis is the empirical data. We also have and want to hear from you sort of the theoretical and conceptual framework that’s going to be interwoven with those empirical results. So the statistical analysis is not the be-all and end-all, it’s just a tool for us getting there.  And we will be building profiles of states, anonymous profiles of states, that are performing at a high level, and we’re going to be detailing those approaches and you’ll see those profiles. We’re going to be producing a book, a handbook, of either if you want to call it “Best Practices” or I think we all sort of prefer “Pathways to Effective Epi.” And I wanted to say one -- and then I’m done. And then I just wanted to say one other thing because it was striking, but we don’t have it quantitatively; that in our interviews with all of you, those of you that are from states, we heard over and over again about how you were able to do the work -- a lot of what you’d been able to do recently had to -- came from the SSDI initiative. And then of course you’ve seen here also the impact of the CDC assignees being in states and interns and Fellows. So it’s really clear -- and again, we can’t exactly quantify this, but it’s really clear that those initiatives at the federal level have been really important in moving MCH Epidemiology forward in the states, and so I’d like us also to have discussion on what now? For instance, maybe we need help with data translation or something; I don’t know. But it’s clear that that effort, those initiatives from the federal government to the states, are really crucial to the work that you all do. I’m going to stop, and hopefully you’ll -- as soon as -- when Violanda is finished, we’ll have discussion.