Ninth Annual Maternal and Child Health Epidemiology Conference / December 10-12, 2003
MARION CARTER: Okay, great. Well, good morning. Great. Well, my talk today is about the California Women’s Health Survey that’s going to be denoted hereafter as CWHS. Let’s see, page down? Yeah. Now, my presentation about the CWHS will touch upon the topics listed here. And throughout I will evaluate this woman’s health survey in terms of the key attributes outlined by the CDC as central to assessing any surveillance system. Now, in the United States, states have central authority and responsibility for their population’s public health and obviously to guide decision making about public health budgets and programs states rely at least in part on health data. Now, these data come from sources such as vital statistics, insurance records, disease reporting systems and of course population based surveys. Population based surveys hold a special place among data sources. These surveys can provide population prevalence estimates about health knowledge, attitudes, and behaviors.
And this kind of information is critical to identifying public health priorities. Now, there are numerous sources of high quality national level population based data such as the national survey of family growth. But knowing for example that 75 percent of U.S. women are currently using contraception is not particularly useful to state level family planning programs and policies. National level health statistics mask considerable heterogeneity among U.S. states. But states continue to face a dearth of such data. The CDC led behavioral risk factor surveillance system, or BRFSS and the pregnancy risk factor surveillance system or PRAMS, are two key responses to this gap in state level data. But gaps remain, including data about women’s health. In fact, only six states have ever fielded a women’s health survey, and all but one have done so at only one point in time, thus most are not a surveillance system in the true sense. So as a part of it’s strategic planning, the Division of Reproductive Health, or DRH, is exploring ways to strengthen state level population based data collection about women’s health. And one strategy under consideration is to help other states conduct their own women’s health surveys on an ongoing basis. Now a logical part of this effort is to examine existing state level surveys.
And at the moment there is one candidate, California, the only state with an ongoing effort to collect such data through the California women’s health survey. So the objective of my investigation was to consider how does the CWHS work? And then from there, what can we learn from the survey system? Note that my talk here will revolve around this first question. Okay. So how did the survey get started? Three things converged that together seemed to launch this survey. First, an Office of Women’s Health was established at a high level within the health department. The momentum from this development, combined with leadership shown on the part of some of the data users in the state health department who advocated for more state level data on women’s health. These efforts were further supported by the model provided by Georgia, which had just successfully fielded its own state level women’s health survey. Now, from there it was not difficult to martial more support from other programs and eventually a survey group formed. In California the survey group, denoted in the center, is comprised of a number of state and non governmental agency.
This slide presents a typical range of primary stake holders. This survey is characterized by group collaboration and consensus. This is not a hierarchy. The Office of Women’s Health at the top of the circle has played a key coordinating role, but each group that participates in the survey has similar responsibilities and opportunities with the CWHS. The survey research group marked in yellow at two o’clock is the non governmental organization that carries out the field work. All others listed here are state programs, though in some years private partners have been involved as well. This group modeled the survey design on the BRFS. It is a telephone survey based on a random digit dialing or RDD sampling method. It began in 1997 and has been conducted every year since then.
Each year, about 4,000 women, ages 18 or older, have been interviewed. And the survey, which has roughly 200 questions, takes about 35 minutes, though this time can range widely. The survey is fielded in both English and Spanish. This slide presents the topic areas that the CWHS has covered over the years. Some topics have been included on all or most years of the survey, while other topics have appeared less frequently. Thus, the ways the survey together represent a yearly surveillance system for some aspects of women’s health, but not all. And from glancing at this you can see that the survey aims for breadth as opposed to depth. Give you a minute to look that over. So the CWHS is a telephone survey and covers a range of health topics. A key question to ask is, well, how is the response rate? Now, there are actually many ways to calculate response rates. CASRO is the name of the strictest calculation method while Upperbound is the name of the most liberal method. You can see here the comparative response rates from another state level telephone survey, BRFS, all from 1999. These levels are not high and those low CASRO rates are of some concern, but it’s hard to assess how much bias was introduced as a result of these low rates without having an objective standard to measure against.
Now, those response rates pertain to the entire survey, but you can also think about them from a perspective of individual questions. We worry about this for more sensitive questions. In this survey, refusals for most questions are rare, however, a number of women each year decline to answer the entire domestic violence module. For example, in 2001, about 100 women out of a sample of 4,000 did. Okay. So how does the CWHS shape up in terms of the CDCs key attributes of surveillance systems? This table presents a full list of the attributes. Based on what I’ve presented so far, we can speak to a few attributes which are bolded in yellow. I’ll evaluate each attribute on a three point scale of good, fair, or poor, just general impressions based on my investigation.
Now, in terms of representativeness, the survey excludes women under 18 and women who do not speak either English or Spanish, and of course those without a phone. These exclusions, while understandable from a logistical and a cost point of view compromise the survey’s representativeness. Furthermore, the response rates are not high, but the bias is unknown. Thus, I rated the survey fair with a question mark on this attribute. We saw the survey’s content had changed frequently over the years. This change reflects good flexibility on the part of the survey to respond to each year’s policy and program environment. Thus I rated it good for flexibility. Acceptability in this case speaks to the acceptability of the survey to both stakeholders and survey respondents.
The fact that so many groups are willing to participate in the survey is a strong sign of survey acceptability. The vast majority of respondents find the survey acceptable too, so overall I deemed acceptability of the survey good. Okay. Now, the survey costs about $400,000 each year. About $2,000 per question. But this amount does not come from a central source of funding. Rather, each program that participates in the survey contributes funds according to the number of questions it can and wants to support. The entire group shares the cost of core questions, which include for example, those about socioeconomic background. Questioner development is also handled through a shared collaborative process. Each program or organization develops and submits to the group a list of questions it would like to include on the survey. But interest and funding alone will not get a question on the survey. Each and every question must be approved by the consensus of the entire survey group. So group consensus trumps all other concerns.
The decisions about what to include in the survey is guided by a set of predetermined criteria developed early in the survey’s history. The criteria include 15 questions asked of each question considered for inclusion in the survey. This slide presents just four of those 15 as an example. And as you can sense from these examples, the criteria plays highest priority in question utility and data quality. We can now speak to a few more of the key attributes, specifically to data quality and sensitivity. Now, thanks to those criteria and the scrutiny each question receives, data quality and sensitivity of the survey seem good. Nevertheless, it’s difficult to know exactly how many women each year, for example, answer the questions about domestic violence truthfully. It’s hard to assess validity on these sorts of things. Now the data become available within a few months of each year’s data collection.
All analysis is done internally, within each participating program or organization. Done however they wish and to whatever extent they are willing and able to do. Now, while a complete inventory of data utilization was unavailable, most evidence points to uneven analysis and utilization across programs. Now, the main mode of public dissemination of results is a one page focused briefing sheet called data points. And these are developed primarily by participating programs. Drafts go before a volunteer editorial board, comprised of members of the survey group. This board insures scientific quality as well as consistency in format and tone. Then sets of data points are published periodically on the web and or in hard copy. And being relatively simple, these data points have been generated fairly quickly. To date, this was as of some months ago, over 60 data points had been published on a huge variety of topics. However, to date there are no summary reports from the CWHS that cut across topics. The survey group has wanted to produce this but has faced administrative and funding hurdles to doing so. But the utilization of these data extend well beyond the data points. Here I’ve listed other key ways the data have been used. They’ve been used for internal program review, prevalence assessment and other research. For example, within the department of social services, the data help guide program development. For example, providing data about mental health among the welfare population. These data have showed--have allowed the domestic violence group to stop relying as much on prevalence data from the Department of Justice, which seriously underestimates prevalence. Various groups have produced reports, presentations, journal articles with these data.
The data have been used to evaluate interventions also. for example, a group in Maternal and Child Health working on folic acid, used trend data in this survey to track knowledge of the benefits of folic acid during pregnancy in the face of an informational campaign in movie theatres. They’ve also been used to support legislation. One example from the STD group which used data--one example is from the STD group which used data about knowledge of Chlamydia to help expand a state program to promote STD awareness. Again, let’s return to the key attributes given the new information I’ve presented. Now, in terms of stability, I rated the survey fair. Part of the reason is that the survey relies on consider volunteer time and energy from participating programs and organizations in every aspect of the survey. Furthermore, the budget is a patchwork and therefore somewhat unstable, especially given that program’s budgets can change radically from year to year. And I rate timeliness as good. The survey is done annually, data analysis is prompt, and data points are published rapidly. In terms of the simplicity of this system, I mean carrying out multiple waves of this survey is not simple, no population based survey is.
But for what it is, the process is straightforward and transparent. So I rated simplicity as good. Useful; absolutely. The various ways the data have been used reminds us again of the gaps in state level data that existed prior to the survey’s inception. But the data could be further analyzed and more broadly utilized and promoted. Now, to conclude on this and the next slide, I’ve summarized some key lessons from this evaluation. These are some of the issues that I believe that the Division of Reproductive Health can keep in mind as it considers ways to help other states successfully implement population based surveillance of women’s health. First, various factors have facilitated this survey. These factors include high level political support, especially at the start. The presence of a very motivated and capable individuals behind the survey.
Speedy public dissemination of at least some results. And the presence of an excellent expert local survey research organization. The survey also shows how local ownership and collaboration can be real assets. The collaboration that characterizes every aspect of the survey helps insure ongoing usefulness of the survey and its results. It spreads costs and energy and it strengthens the feasibility of the survey and has a somewhat unintended benefit of strengthening ties among programs. But the system has disadvantages. In particular, it lacks stability because the system relies so heavily on programs, staff, and funding which themselves are so vulnerable to change even year to year. So finally I’d like to thank some of the people that facilitated the evaluation and this presentation. And thanks to you all for your attention.