MCHB/EPI Miami Conference — December 7 - 9, 2005

Methods Can Help — Transcript

BARBARA SUTER: Hello, I'm Barbara Suter from the University of Rochester . I work in the Division of Community and Preventive Medicine, actually the Department of Community and Preventive Medicine, Division of Public Health. I work with birth certificate data, people that code birth certificates, people that use the data, as well as other tasks. I'm a senior analyst there. Today, I'm very happy to be here to be talking to you about a method that we've used to increase the validity of abstracted data.

I'd like to start by telling you a little bit about what I mean by abstracted data. It's a process that's used to produce information that is usable and represents the nature of the situation for which the information is obtained. Data abstraction involves reviewing data in its original source, whether it's a medical record, maybe clinic data, maybe its interview data. It involves taking the information that you've gained in these documents or interviews and putting them into another format so that you can use them in your analysis. It may involve coding. If you have interview data, you may want to try to quantify the responses of long-ended questions, so you'll want to code them, so you can do that sort of analysis with it.

I want to just briefly to look at this health policy we have modeled after one by Richmond and Kotelchuck. The knowledge base is the data. It's what informs your action. The actions will fall within the political will and social strategy are necessary to implement an action. You need political will. You need the policy makers and the funders to be behind you, and the social strategy to implement your action.

Behind all of this, though, is the knowledge base, and that is what we want to talk about today. We want to talk about making sure that your data is accurate that you're using. And we're going to be concentrating on birth certificate data when we talk about data.

Okay. Why do we need accurate data? We need it because we use it to allocate scarce resources sometimes. We use it for policy development, and we use it for research. If we're going to use data to indicate what action we need to take, that data needs to be accurate. You've seen lots of uses of data so far today and hopefully the data is accurate. I've listed many databases here that you're probably familiar with and many of them do involve birth certificate information, either right up front or maybe later are linked with birth certificate information in order to make them more useful. Okay.

When we're looking at data, we're looking at wanting to have accurate data, and we're looking at sources here of inaccuracy. Sometimes it can come at the reporting stage. You have a woman who comes in, tells her doctor when her last menstrual period is, and it's not the right date. When you're doing interviews, you might ask a woman how many times--how many cigarettes she smoked in the last month. That's probably something she can't give you very accurately. If you're able to ask her within the last 24 hours, you'll get more accurate information. So sometimes errors occur in the reporting of information.

They also can occur with record documentation. Physicians are known for their poor penmanship, and if they are recording some information in a medical record, it may be hard for somebody using that record to abstract that information correctly.

Also, sometimes information you expect to be in the medical record isn't there, or whatever document you're looking at. There can be equipment errors: blood pressure machines that aren't functioning properly, scales that aren't calibrated correctly. And this will impact your data.

Abstraction needs to be done following guidelines, established guidelines and protocols. If it's not then it doesn't produce uniform collection of data. Data entry also needs to follow guidelines.

Analysis is important if you're going to be using data for analysis that you know how the analysis was collected. You can interject error into your analysis if you don't understand all the ramifications of the data that you're using.

As an example, we're going to look at data collection in up-state New York , the Finger Lakes region. Here we show the hospital-based birth registrar, who has her own office. She goes to the OB Unit. She gets medical information from the mother's chart, the baby's chart, the prenatal record. She comes back and she enters the data into an internet-based system. It's already gone now to New York State , who tweaks it, does things with it, sends it off to the National Center for Health Statistics, and then you'll find it on the internet when you want to compare your state's data with our state's data. There it is. How accurate is that data? Who is the person there that is most responsible for the accuracy of the data? And that's the birth registrar that's sitting there in her office doing her thing. Ultimately the responsibility for the accuracy of the data rests with the regional perinatal center, who works directly with the coders with the hope of increasing the accuracy of this data.

So, we talked about various sources of error. I wanted to concentrate today because our method deals with this with abstraction error. I mentioned the birth registrar sitting in the hospital doing her thing. Some of the bigger hospitals have one person designated or two people to collect birth certificate data. Many of the hospitals are smaller, have two or three hundred births a year, so they don't have a dedicated person. They may have someone from medical records that they coerce into doing this, or perhaps the unit secretary on OB will do it. Sometimes we have a nurse that will do the abstraction, and nurses certainly have more background in what they're doing, but they may see it as just an added task and responsibility that they don't really care too much about, don't see the importance of, so they may not do, you know, a super-duper job with it.

When you talk about abstraction error, there are various things that can occur. First of all, if we have--woops, lack of knowledge. If we have a birth registrar that has very little training, she may be the unit secretary who's been working on the unit for six months and suddenly has this job in her lap. You know, how knowledgeable is she going to be? I heard one report that there's a question on alcohol use during pregnancy on the birth certificate, and the doctor had written it as ETOH used during pregnancy, and the birth certifier didn't recognize that as being alcohol intake. So they're abbreviations and terminology that are used all the time that birth registrars need to be aware of.

I wanted to demonstrate something, and I want you just to think quickly and jot down on a piece of paper the number of people in your family. If you could take one second to do that, the number of people in your family, jot it down. I'm watching. Okay. I'm wondering how many of you included just your partner and/or spouse and children. Can we see a raise of hands? Did some of you include maybe your parents and siblings? We have some of you that did that. You may include people that are in your household that weren't even really blood relatives, but they live in your household. You consider them to be your family. You can see that I gave you no definition of what your family was so you were free to interpret it in any way you wanted to. In much the same way, there are definitions that go along with birth certificate coding, but there's two to three hundred variables and knowing all of the definitions is difficult.

And also some of the definitions may be inadequate. If I gave you, you know, the definition of family is being people that you live with, then maybe you would interpret that still to mean blood relatives, whereas some others wouldn't. So sometimes definitions just aren't tight enough and allow for error.

Errors can occur also with decision making. We may have a mother that had a placental abruption. The placenta pulled away from the uterus before it was supposed to, before the baby was delivered. And then the birth certifier has to decide is that a maternal condition that necessitated a cesarean section, or is it a fetus at risk? So she needs to have some knowledge. Things aren't in black and white always in the medical record.

And random errors will always occur. We're human beings. We make mistakes, and birth certifiers are no exception to that rule. We all transpose numbers from time to time. Certainly you don't want somebody that's doing a lot of that in this job, but that does happen. We saw that with the medical records numbers in the last presentation, or the one before that. Sorry. There can be omissions or carelessness. Things are missing. If a birth certifier doesn't feel her job is very important, then she doesn't take care in what she does.

So what can we do? We can train all new coders. That's really a no-brainer, and yet lots of times it doesn't happen. We'll find a month later that we have a new birth certifier at one of our 14 hospitals that we didn't even know that the other one had left due to illness or retired or whatever. Has that birth certifier been trained? If she's trained, she may have been trained by the person that left, and if that person wasn't particularly good, then she's been trained and she will probably perpetuate the same errors.

We need to provide ongoing training. If a person has been trained five years ago, can we really expect them to be still be adhering to the same criteria that we taught them? We learn as we go along. We modify what we do, and we don't always modify it in a positive direction.

We have tried to help coders understand how important their data is. We'll invite them to come to outreach meetings to see the data that they've actually put into the system, how it can be used to compare their hospital with others in the region. I think before we did this, they felt as though they entered data, it went off into the field, and nobody ever saw it again so what difference did it make.

And we also have tried to let the people, the document care, the clinicians, know how important their documentation is. We were at one hospital and we were talking about early screening--maternal screening for defects, and one hospital had a very low rate. And they said, but we do them all the time. Well, as it comes out--you know, came about we found that they were documenting it on a chart in their office, not on the prenatal that came to the hospital. So the birth certifier had no way of knowing that these women had been screened. So bringing issues out helps to make the data better.

Okay. So what we're going to concentrate on for the rest of my talk is ongoing training of coders. We--and by we, I mean my coauthor Ann Dosier, Dr. Chris Glance, who's the principal investigator for our perinatal data system, and myself, trying to find ways to improve the data quality. For a long time--well, we have monthly meetings with the data coders, and that really helps. It brings out issues that they have found in trying to code the data. But it was mostly driven from their ideas of what they thought was a problem. And how many times are there problems that we're not even aware of that we're causing? You need somebody from the outside looking in to tell you what you need to really be looking at. So we tried having them abstract medical record material, give them a whole record, give them the birth certificate forms, ask them to do it. Well, first of all, they didn't do it. They didn't send them back to us. It was too much work. It was just like what they were doing all day. They didn't want to do it. We tried doing it at our coder meetings, so they'd just come; we ply them with snacks and ask them to do this. They didn't come to the meetings then, I think partly because it was boring, partly because they didn't want to be identified as making errors. So that didn't work.

So we then developed what we call the coder fax. And it's coder because it involves coders--birth certifiers also known as coders. It involves coding of information. And it was a process that involved faxing. So that's how it got its name. Okay.

So what is the coder fax? It's a training tool for coders to improve the quality of abstracted data, and it consists of a page or two of medical record material and then an excerpt from the medical record form. These two pieces of information were faxed to the birth certifiers in the hospitals. They were to complete the forms and fax them back to us where we could see how they did, and then we could provide them feedback by faxing back to them the answers that were right and the guidelines that were applicable. It helped--has helped us to understand as data users how they're interpreting the data definitions, and so it helps us eventually too in our analysis. Okay.

So we started this process in September 2003, and at that time we faxed medical record material and the form, and I did include one of the forms on the back page of my presentation so you could see it. I faxed it to 23 people at 14 hospitals. We hoped to reach everybody that did anything with birth certificates, so even if they weren't are key contact people, even if they didn't attend coder meetings, we were hoping that they would take part in this exercise to try to improve their ability to correctly code birth certificates. We received 74 percent back. To the one question that was asked on the first coder fax, and that was in regard to gestational age, 59 percent of the respondents answered that one question correctly; 41 percent got it wrong, something that seems as simple as gestational age. What we found was that there are different gestational age depending on who is coding it. A Ballard exam, a physical exam of the infant will give you one gestational age, whereas the obstetrician's best estimate, something that's calibrated during pregnancy, is another estimate. The guidelines didn't really specify which was to be used. We as a region had decided that we were going to concentrate on the obstetrician's best estimate. So the 41 percent that got it wrong went with the Ballard, and that was the reason why they got it wrong.

We have since repeated the gestational age variable. I think five months later we did it, and everyone but one person got it right. And that one person took the documented gestational age was like 35 weeks, 6 days, so she reported it as 36 weeks. We are supposed to just report gestational age by weeks, just not rounding but disregarding the days. She, you know--we're all taught to round, so it was a perfectly natural mistake, and it made us then re-emphasize the definition.

A year after the initial gestational age variable was sent in the coder fax, we did it again and we got--everybody was correct. I think that was a fluke. I'm sure we had different birth registrars involved, but the process is ongoing. You can't stop because there's always a turnover and everybody always needs to be refreshed. Okay.

Two more minutes. Okay, so who are we reaching? We--from September 2003 to 2005, we've sent out 21 coder faxes. We don't do them generally in the summer. The coder faxes were put online actually in January 2005. I've given you the URL down below if you wanted to look. We developed a perinatal programs web site, and we have a link there to the coder fax. An e-mail is now sent to coders to remind them that a new coder fax has been posted. They can go online, download the information. They still need to fax it back to us, and we still--and then we go online and we post the answers. We send them another e-mail to tell them that the answers are posted.

Participation rates have varied from 14 to 90 percent, and we've had two to five participants from each hospital. This has worked well, especially with hospitals that are far away, too far to come to coder meetings, have participated. Our low-end hospital unfortunately is one of those that isn't able to make it to coder meetings, but we still feel as though we're reaching them. We're getting them some information.

The ease of data abstraction depends on how easily the fields can be abstracted. Some fields that you know they know just where to go to get the APGAR on the chart, and they know just where to enter it; the same with newborn screening and birth weight. But when we get to one where the information has to be processed, where we have to consider whether the mother received Pitocin for induction of labor or augmentation, sometimes the birth registrar has to make that decision. That's a little more difficult to accurately do.

And then in the last instance, sometimes there's lots of contingencies to a diagnosis. One of those regards prenatal visits, the number of prenatal visits. Is a visit to an endocrinologist a prenatal visit? How about one to a dietician for somebody that's a diabetic? There's lots of different contingencies. We've spent lots of time trying to decide is it really a prenatal visit or not. This is important to know because adequacy of prenatal care depends on these number of visits, and when you realize that there is some flex in that, then you need to consider that when you're using that in your analysis.

All right. So what have we learned? We've learned that coder training and detailed data definitions are at the core of data integrity. We've learned that some fields are more prone to error than others. You need to understand the data before you analyze it. And the process of training must be ongoing. Thank you.