Ninth Annual Maternal and Child Health Epidemiology Conference / December 10-12, 2003
SCOTT D. GROSSE: Good Morning. I’m going to start by talking about principles of economics, economic evaluation, and then discuss two case studies. The two case studies, I selected because I tried to touch with different groups that are involved with Maternal and Child Health, one is prenatal care/low birth weight, the other is birth defects.
Economics is not accounting. I am not an accountant. Economics is about getting value for money. Getting the best outcomes we can get, given scarce resources, not just our budgets, but also the time and energy of providers and families. We have scarcities, we need to be careful and focus our energies on those things which get us the best outcomes. Everything we do has an opportunity cost. We have a given block grant funding. If you spent that money on one activity, you are not going to have that money to spend on another activity. The cost of the activity you choose to fund is not the dollars, but what you could have gotten with those same funds if you had used them in a different place.
Economic evaluation always involves comparing alternatives. You should never look at just one program, one policy, and say this is cost effective or not. You have to consider, what are the alternate uses? What are the options? Different options for achieving the same goal, or given the same resources, trying to attack a different goal. Economic evaluation is important to inform policy decisions, but it should not be thought of as a decision rule, that is, it should not be a requirement, in my opinion, that you show that your cost effectiveness ratio is below fifty thousand dollars per quality in order to justify the program, or, even if you do have a low cost ratio that does not mean that intervention necessarily should be funded. You have to consider all the alternatives.
Economic evaluation is not just about economics. It is also about epidemiology. You cannot do good economic evaluation without having good epidemiology. And, no matter how good your epidemiology is, you always should be prepared to test your assumptions. We can always refine our understanding of the underlining processes. I’m going to discuss, primarily, two different methods of economic evaluations. First, Cost and Illness studies, which is often not considered part of economic evaluation because you’re not looking at an intervention…but I consider this the first step of economic evaluation. You have to be able to measure the costs that are associated with health outcomes in order to be able to determine the benefits of interventions that prevent those outcomes. Cost effectiveness analysis and cost benefit analysis are two different methods, Russ is absolutely correct, that are not interchangeable, and I will show you how they differ.
Cost effectiveness analysis, which is the method primarily used in the health field, leaves health outcomes in either physical, natural units, or converts them to a standard metric, such as a quality adjusted life, a quality that combines both mortality and morbidity in a single measure. Cost benefit analysis converts health into dollars. It is used especially in the field of regulation.
So, cost of illness is not just direct medical cost. Often when people calculate the costs of prematurity or the cost of other health outcomes, they start with hospitalization costs, which is a good place to start; as long as you don’t stop there. Besides hospitalizations, you have outpatient care, medications…there are other direct costs. Children who have disabling conditions require early intervention services. Many of them will require special educational services. If you don’t have information on the use of those services, you’re going to be substantially understating the costs of those conditions. Besides direct costs, there are indirect costs, which is not what you think of in budget terms, markup, or overhead. Economists used to refer to these as productivity losses, that is, rather than actual expenditures, are the foregone economic production, the results of someone being unable to work, either because they are dead or disabled.
When you’re doing cost/illness studies, you have to be able to take multiple datasets into account. Start with hospital discharges. Medicaid data is great if you have access to it. There are National surveys that can give you information on utilization for population-based samples. The medical expenditure panel surveys are great, especially for more common conditions such as asthma. For the less common conditions that you don’t have enough observations. Survey income and program participation from the Census Bureau is important for looking at disability, work disability for looking at people with different conditions. There are a number of claims databases that can be used to supplement the Medicaid data, for the low income population. Special Education records. If you want to look at Special Education cost you need to have access to those records, which I understand with HIPAA that’s a serious issue.
The problem with these datasets is that they usually don’t have the exposures that we’re interested in. For example, if you’re looking at the cost of prematurity, you need to know what someone’s gestational age is. To get that, you’re going to have to be able to link data sets. You have to find datasets that have the exposure to things such as vital statistics. Or for birth defects, your birth defects surveillance registries. And you have to be able to link them with the datasets with healthcare costs and utilization.
Here’s an example of cost of illness estimates, this is from a book chapter that was just published, ‘A New Estimate of Cost of Cerebral Palsy Along with Mental Retardation, Vision Loss and Hearing Impairment’. ‘The direct medical costs by eighty-three thousand dollars over the lifetime discounted back to present value using 3% discount rate is by eighty-three thousand dollars which is only 10% of the total cost estimate of the lifetime. 80% of the cost is the indirect cost. The fact that children with cerebral palsy are more likely to die and are much more likely to be unable to work or limited in the amount or type of work they are able to perform. Special education costs are just as important as the medical costs. There are some other costs, which I haven’t included there.
Once you have your cost illness estimates, how do you use them to do an evaluation of interventions? You have to decide whether you are doing a cost effectiveness or a cost benefit analysis. Cost effectiveness analysis includes direct costs, and usually only direct costs, whereas the cost benefit analysis include both direct and indirect costs. Depending upon the type of intervention you’re looking at, you may need to take into account multiple outcomes. For example prematurity is not just having a diagnosis of pre-term birth or low birth weight. There are sequeli, which you need to take into account. For example, premature infants are at an elevated risk for cerebral palsy, mental retardation, hearing loss, and vision impairment, so you need to take into account the weighted estimates, if the probability that each of those outcomes by the cost of those outcomes.
The economic evaluation can inform Public Health Policy at multiple steps. You start in the purple pie there with research. Prior to having an intervention, economic evaluation can be important helping to find the agenda through cost illness studies of where the more costly outcomes are, and which outcomes contribute to them. Doing evaluation of pilot studies, epidemiologic research is essential for evaluating the effectiveness of interventions or potential effectiveness. Once you’ve conducted a number of research studies, you have to pull it together. Systematic reviews are absolutely critical…and unfortunately not done nearly often enough. When people sit down to do analysis they often do a quick mid-line search, pick one or two studies, then draw the numbers from those studies without considering the full range of studies. Once you have the reviews, then you can make recommendations for policy makers. Policy makers make their policy decisions at that stage, some policy makers will ask for the results of economic analyses. Many will not, there is a lot of variability.
Once policy is made and a program is being implemented, economic analysis can contribute to decisions by the program managers on how to implement the program. That is, the economic analysis can help you decide how to target your resources. Not just whether to do an intervention, but how to implement it. And, of course, once the intervention is in place, you need to do economic evaluation to look at the outcomes and decide, ‘is what we are doing working? Is it worth the resources we are using, or should we be doing something differently?’ It is not a ‘one off’. There is a continuing cycle. Once you have the intervention in place you need to continue doing research to look at potential alternatives.
So, you start with cost of illness studies, review the effectiveness data, then you go on to cost effectiveness analysis, and cost effectiveness analysis can be done before intervention, or afterwards. The extent to which you use assumptions or data depends upon whether you actually have an intervention to evaluate. Often, policy makers ask for economic evaluations before the intervention. The case studies that I’m going to be talking about are case studies of prospective economic evaluations, and what I want to do, is go back afterwards and see how were those, how accurate were those analysis at the time. I’m not going to tell you how to do an economic evaluation, just a two minute overview. First you have to define the study question, you have to say ‘what is the hypothesis that you are trying to evaluate, and who are you doing it for?’ Because you have to define what costs to include depending on whether it’s from a societal perspective, the State Health department perspective, the healthcare system perspective. You have to calculate the unit costs, not charges. You need to either estimate the change of outcomes or model them based on your understanding of the processes. Then, calculate the costs of the intervention.
And once you have the costs of the intervention, the change of outcomes and the cost of those outcomes, you can calculate what is the expected ‘net’ cost, that is, the cost of delivering the intervention, subtracted at the cost, the averted cost of care, that results from the changes in outcomes. If your net cost is negative, your intervention is said to be cost saving. It’s true that cost saving is not all that matters, but you do have a cost saving intervention then its pretty easy to sell that to the policy makers. If net costs are positive, you then calculate the cost effectiveness ratio, where your numerator is your net cost, your denominator consists of the change of health outcomes. That may be number of deaths prevented, numbers of qualities.
Cost effectiveness is not the same as cost saving. Cost effective means you’re getting good value for the money that you are spending compared to alternative uses. Cost savings states that prevention actually saves payers money. The cost savings argument is often misused. I don’t know how many times you’ve read or heard statements that every dollar spent, we will save so many dollars. Those claims are usually wrong. Averted costs usually do not exceed intervention costs in maternal and child health except in the area of immunizations. Even there, the argument may be misused because even if something saves money on average, that does not mean that additional money spent is going to be cost saving, in fact, usually, it’s not. The cost required to get the last 5% of children immunized is much greater per child than the cost getting the first 90% immunized. Public Health advocates need to learn how to make the case for cost effective interventions.
Ok, to give you an example of the fallacy of the cost savings argument, here is an actual quote I took from our local newspaper, ‘For every dollar you spend on screening, you’ll save ten to twenty dollars on long term costs.’ What is the empirical evidence? The last broad study of this subject in the United States, was done in 1988 by the Austin Technology Assessment. They concluded that the screening of all babies, once for each child, for PKU congenital hypothyroidism, would save almost three dollars per dollar spent. However, other screening interventions, including screening a second specimen for every child, which many states do, was not cost saving. Screening for other disorders, was not cost saving. So even if the basic screening for PKU hypothyroidism is cost saving, you can not generalize that to additional resources.
There are a couple of recently published articles on tandem spectrometry, expanded screening for MKI deficiency and other disorders. Both concluded that the averted cost would be about 20 to 25 cents on the dollar. It’s nice, but it’s not cost saving. That’s not why we should be pushing or promoting that kind of screening. As I mentioned earlier, cost effectiveness should not be considered a decision rule. It informs decisions, it should not dictate them. Why? First of all, important outcomes are almost always left out of the models. When you do an intervention, there are often outcomes, such with prenatal care. Prenatal care is not just about preventing low birth weight, or reducing infant mortality, there are many other outcomes, which are not included.
Second, fairness issues may trump costs. For example, newborn screening for sickle cell disease, a number of analyses have consistently shown it’s not terribly cost effective to screen Caucasian babies, yet almost all states have chosen to screen all infants. There is an equity issue, a social justice issue, about segregating public health programs by race or ethnicity. So states have chosen to do that despite the evidence of economic analysis.
Third, estimates are always uncertain. We need to be careful when we do or read economic analysis to take the results with multiple grains of salt. First you should always be careful that the sensitivity analyses have been done carefully, to see how sensitive are the results to variations in the assumptions. And even if the sensitivity analyses indicated the results are a bust, they may not be. There is uncertainty within the model where you have known variability in the parameters, and there are other things that we ‘don’t know what we don’t know’.
As I said, epidemiology matters. In order to do an economic evaluation, you need to project how many cases of disease or death are going to occur without the intervention, and how many are going to occur with the intervention. We often lack sufficient data. Expert opinion may be biased. Because of that, we need to have good quality evaluations, rigorous evaluations that control for potential biases. Your epidemiologists, I don’t need to tell you about, are sources of bias and how to control for them. And, we also need evidence based reviews.
Here’s an example: Newborn screening for MKI deficiency…I’m currently working on an evidence based review for this, which is why I thought I’d mention… we know that, or we believe that, a number of children with this disorder will die or will experience serious neurological disability in the absence of early detection. The question is, how often do those outcomes occur without screening. To do cost effectiveness analysis you need to give numbers, you need to say ‘these are the numbers of deaths, and the numbers of cases of disabilities that we are preventing’. You cannot do a cost effective analysis without making explicit assumptions. Which is actually one of the advantages of doing economic analysis, it requires you to make your assumptions explicit. Instead of simply saying ‘well, we know it’s a good thing because it saves lives’, you need to say how many lives are being saved. I’d like to take one example of one outcome of three published economic evaluations, one came out last month in ‘Pediatrics’, and look at the outcome of cognitive disability. How many survivors of acute episodes of MKI will develop serious neurological impairment? One analysis assumed zero. One assumed 10%, and one assumed 32%.
Based on my reading of the literature, the 10% is pretty well on target. The other two were not based on careful reading of the literature. So, just because an economic evaluation is published, in a peer reviewed journal, it doesn’t mean that the numbers are reasonable. There is a quality control issue.
So, how well do prospective economic evaluations predict outcomes? Economists or economy models are very good at explaining the past. Predicting the future is much harder. So, I mentioned two maternal and child health case studies where economists did make predictions, prior to policies being adopted and implemented. One was the proposed expansion of Medicaid funding to provide coverage of prenatal care to women who were not only below the poverty line, but at low incomes, but still in that sort of in-between group. Second was the fortification of enriched grain product with folic acid to prevent nueral tube defects. All the economic evaluations that were done showed both interventions to be cost saving.
So the question is, have those interventions actually reduced costs as predicted? Prenatal care and low birth weight…I’m relying heavily on articles published by Huntington and Connell, in the New England Journal of Medicine in 1994, I’m sure a number of you have read that article, it’s a great article…they reviewed the economic studies and epi studies that had been published which suggested that prenatal care leads to fewer low birth weight births, and noticed that these studies did not adequately control for confounding self-selection or even reverse causation bias from the gestational age issue, obviously if you give birth at 24 weeks it’s hard to have a large number of prenatal care visits. The results of these analyses were used to project reductions in births at costs.
The Institute of Medicine came out with a report of low birth weight in 1985, every additional dollar spent on prenatal care within the target group would save $3.38 and the total cost caring for low birth weight infants. There’s an article from New Hampshire, ‘For each additional $1.00 spent on prenatal care, $2.57 in medical care costs would be saved.’ These cost-saving arguments were influential, they were used by advocates, health policy makers accepted those arguments, and helped lead to a dramatic expansion in funding or coverage for prenatal care during this period between 1986 and 1991, Medicaid coverage for prenatal care and delivery more than doubled in this country. However, evaluations have shown there were much smaller increases in actual prenatal care use, early prenatal care especially. And, almost no evidence of any reduction in low birth weight births. However, despite the lack of change of low birth weight, there is evidence of a decline in infant mortality that was associated with the Medicaid expansion, and this was concentrated in the very low birth weight births. It is a dramatic reduction in mortality for the most fragile infants born to low income mothers, and is probably due to increased Medicaid funding of nicuse which is not exactly cost saving…cost effective, perhaps, but not cost saving.
So what are the implications of this case study? First, it’s very difficult to predict behavioral responses. The assumption was, if you provide access to prenatal care, women would accept it, and take advantage of it. You can’t assume compliance. You have to look at non-financial barriers. Second, this is something I’ve heard a number of times at this conference, routine prenatal care is probably not enough. We need to consider the quality and content of the care. Providing care throughout the life course, certainly before the onset of pregnancy, and we need well designed experiments and evaluations of such interventions to test their effectiveness in achieving the outcomes that we desire. And finally, maternal and child health researchers need to be cautious, in inferring causation from observational data, and assuming that the interventions will have the effects that we want them to have in the absence of data demonstrating that with controlled studies. Remember, you need to show effectiveness before you show cost effectiveness.
Last case study: Folic acid fortification. In 1996 the Food and Drug Administration mandated that all manufacturers of enriched grain products, add 140 micrograms of folic acid per 100 gram of grain product by January 1st, 1998. This was based on a number of studies including two randomized controlled trials, showing that multi-vitamins containing 400 micrograms of the folic acid were effective at preventing at least 50% or more of all neuro-tube defects in pregnancy.
It was also based on the fact that the recommendation in 1992 by the Public Health Service that all women of reproductive age consume folic acid had very little effect on behavior. Given the difficulty of change of behavior through voluntary supplement use, the decision was made to provide population coverage through fortification. The downside, the concern, which delayed the decision was that people who have vitamin B12 deficiency…B12 and folic acid have the same metabolic pathway…that the additional folic acid could mask the hemotologic signs of vitamin B12 deficiency, postpone clinical diagnosis which could lead to neurological damage primarily in elderly people who are at risk of pernicious anemia, vitamin B12 deficiency.
So, two economic evaluations were published, in 1995 and 1996, there is an article in The American Journal of Public Health by Patrick Romano, Norm Weitzman, and others, from California. There is a book chapter by Alison Kelly, Anne Haddox and others from the CDC. These different methods, the California study was a cost benefit analysis that included direct cost and indirect costs. The projected net benefit of 94 million dollars per year, the benefit cost ratio 4.3 to 1. Now if you look at just the direct cost, which is what is done for cost savings, it is much more modest. It’s still slightly cost saving, that is, the averted cost slightly exceeded the actual cost of fortification, and the assumed complications of neurological damage from masked vitamin B12 deficiency. The cost effectiveness analysis concluded the net savings were 4.7 million dollars per year with a cost saving ratio of 1.4 to 1.
However, that included a fairly large estimate of the time cost to families with children with spina bifida, if you excluded that complement which is not typically included in cost effectiveness analyses, the cost savings was less than the cost of fortification 6 to 1. Both studies concluded that higher fortification levels than the FDA adopted would have resulted in substantially greater net benefits. The CDC study actually suggested a level 5 times higher. The FDA did not base their decision on either of those analyses. FDA, unlike EPA or other federal agencies does not use cost benefit or cost effectiveness analysis, they use a safety principle. You have to first show that what you are doing is not going to cause harm, so they were concerned with a level that would minimize any potential risk of damage from masked B12 deficiency.
Here’s an interesting slide, what was the number of NTDs that were projected to decline? Romano projected a 10% decline in spina bifida anencephaly. The CDC predicted only a 2% decline. The evidence from birth defect surveillance programs is much greater. The middle estimate, the 15 surveillance programs without prenatal ascertainment, that excludes pregnancy terminations resulting from prenatal diagnosis of an NTD is a 28% drop in spina bifida, a 13% decline in anencephaly. The birth certificate data would show much lower numbers because of substantial understatement, under recording of birth defects on birth certificates showed a 21 to 24% decline in both spina bifida and anencephaly. During this period of time there was very little change in the use of vitamin supplements by women. Maybe a 2% increase, so that would account for virtually none of the draw. And very little of the draw appears to be due to changes in prenatal diagnosis determination.
So why did these two economic models not foresee the rates of decline in neuro-tube defects which have been observed since fortification? First, the actual level of folic acid in the food supply is probably greater than was projected. The FDA sets a minimum level of folic acid, but does not set a maximum, so manufacturers have an incentive to err on the side of caution by putting in substantially more than they are required to put because the monitoring is only to make sure that each batch has a sufficient amount. So they are penalized if they go too low, not penalized if they go higher. And folic acid is extremely cheap. It’s the cheapest nutrient that you can add to food. So, we’re getting more folic acid in our food than we expected there would be, that’s part of the reason.
More importantly, the epidemeologic modeling that was used was extremely conservative. It assumed that if you did not get 400 micrograms per day, there would be no benefit. All or nothing. They did not assume a dose response curve, and so they assumed once you got to 400 micrograms per day there is about a 50 to 60% reduction, below that level, no benefit. The two studies use different assumptions about dietary folate and NTDs. Dietary folate is the naturally occurring folate in food, it’s different from the synthetic folic acid. The bio-availability of dietary folate is only about half that of synthetic folic acid. The one study assumed that folate, folic acid, ahh, they’re all the same. So it assumed the folate you get from food, you would just add that to folic acid fortification or from supplements. The CDC study assumed the opposite extreme, that folate had no benefit, that only the synthetic folic acid did. So the additional percent of women who had reached that 400 microgram per day level of folic acid or folic acid plus folate, was 17% in the study that assumed that folate counted, and 3% in the study that assumed it did not count.
Our current understanding is that first there is indeed a dose response curve between folic acid intake and neuro-tube defects. The more folic acid you consume, the higher your red cell folate level. There’s a nice dose response curve between your folate level and the risk of neuro-tube defect. And also, as I mentioned, we now know that dietary folate is not as effective as folic acid, but it’s not completely ineffective, it contributes to prevention as well.
What about the effect of folic acid on masking vitamin B12 deficiency? Here’s where the two studies differed dramatically. The Romano study assumed about 6 times as many people would be affected, and that the cost would be 10 times as much as the other study. So we have a difference of about 50 times greater assumed cost in one article than in the other. Why the difference in assumptions? Lack of data! We don’t know how many people have masked vitamin B12 deficiency because of folic acid, and we really don’t know the cost is. It’s not a condition that you’re going to find with an ICD9 code, that you’re going to be able to look up in one of your standard databases. So they rely on expert opinion. If you get different groups of experts, you’re going to get different opinions.
So what is the economic impact of folic acid fortification? These are some tentative calculations that I’ve been working on with Norm Weitzman and Patrick Romano, so these are not definitive, the numbers are going to change, but I just thought you might be interested to see a rough sense. Based on the birth defects surveillance data, we’re getting about 500 fewer births with spina bifida per year than before fortification. And encephala, not quite 100 fewer births per year. New estimates of the direct costs per case, current dollars, using 3% discount rate is almost three hundred thousand dollars for spina bifida, just a few thousand dollars for anencephaly, that’s the cost for about 3 days in the hospital, because that’s the average amount of time a birth with anencephaly survives so, total averted costs multiplied by 500 births a year by three hundred thousand dollars, that’s roughly a hundred and fifty million dollars a year in averted cost of care.
If you include the indirect costs, it’s over four hundred million dollars per year. The cost of fortification is about 10 million dollars per year. It’s extremely cheap. Now, I’ve not included any of the costs associated with masking vitamin B12 deficiency because we don’t have data, whether it’s 1 million or 10 million per year it’s not clear, but it’s certainly much smaller than the amounts we’re averting. I won’t calculate a benefit costs ratio for you because we don’t have the complete costs, but this is clearly one of the most exceptionally efficient health policies that has been adopted in terms of cost savings.
So what are the implications? One, modeling biological relationships may be almost as challenging as behavioral relationships. The economic studies were correct to err on the side of caution. When you’re doing an economic evaluation of a proposed intervention it is always better to be conservative in the numbers of outcomes that you think will be prevented. It is much better afterwards to have been shown to be too conservative, that the outcomes were better than you expected, than the other way. Your credibility is greater if you’re conservative. Third, it’s very difficult to model risks of harms where you don’t have data on those harms.
Finally, the calculations I’ve shown do not include other outcomes. The intervention was designed to prevent nuero-tube defects. There are studies now being done of other outcomes in children that also are changing. There was just an article published recently shown reporting a 60% decline in cases of nueroblastoma, a childhood cancer.
So, conclusions. Economic evaluation is a partnership of economics and epidemiology.
I know there was a discussion yesterday as to whether economic evaluation is a part of maternal and child epidemiology, I don’t venture to say whether it’s part of epidemiology, but I do know that epidemiology is an important part of economic evaluation. Economic estimates are no better than the epidemiology that they are based on. Both the economic and the epidemiological assumptions must be constantly tested by new data.
And, if anyone is interested in contacting me, here is my contact information.