Quasi-Experimental Experiments Insurance

The quasi-experimental approach to solving the “evaluation problem” relies, as the name suggests, on a situation in the real world that approximates what might be achieved in a social experiment.

In the context we have been discussing, such opportunities may arise when a “natural experiment” causes health insurance coverage to vary for some measurable reason or reasons not related to an individual’s health status; when this variation is not correlated with other, unobserved determinants of health such as income; and when there are identifiable individuals whose coverage is not affected who can be used as a control group to pick up any secular (i.e. unrelated to the insurance changes) changes in health outcomes, such as those due to improvements in medical technology.

In this section we discuss all the natural experiments of which we are aware that provide credible evidence on the causal effect of insurance coverage on health. We classify natural experiments into two groups: small and large.

In discussing these studies, we pay some attention to effects on medical care utilization, but place greater emphasis on studies that seek to identify direct effects on health.

We do so for the same reasons that others who have reviewed the effects of health insurance on health have done so: it is difficult to know whether increases in utilization will translate into improvements in health. The results of some of the studies we examine reinforce this point.

An alternative justification for examining effects on utilization is to explain the absence of effects on health; if insurance affects health only through its effects on medical care and we do not observe effects on medical care, we should not expect effects on health.

In the studies we examine, this is not relevant because we generally do find effects on health. In these cases, we try, where possible, to use results on utilization to better understand the mechanism by which health insurance affects health. Small-scale natural experiments (case studies)

Lurie et al.: Medi-Cal cutbacks

Lurie et al. (1986) report that in 1982, California terminated Medi-Cal benefits for all 270,000 “medically indigent” beneficiaries, defined as those with “economic or medical need but … not eligible for assistance from a federal program for the aged, blind or disabled for families with dependent children.”

The authors examine changes in health outcomes for 186 patients at a Los Angeles clinic whose Medi-Cal benefits were terminated and compare them with changes in outcomes for a comparison group of 109 patients at the same clinic who were continuously covered by Medi-Cal.

Those who lost benefits experienced on average a statistically significant increase in diastolic blood pressure (9 mm Hg six months after benefit termination, 6 mm Hg one year after termination), while the comparison group experienced no significant change in blood pressure over this period.

Self-reported health status also declined significantly for treatments but not controls. Lurie et al. do not focus much attention on the mechanism by which the loss of insurance may have effects, but do note a 45% decline in the use of outpatient services among those who lost benefits that might plausibly contribute to these declines in health outcomes.

The results in this study may be biased by the fact that the authors, alarmed at the increases in blood pressure observed at the six-month follow-up, intervened to help some of the subjects regain insurance coverage.

But this would be expected to bias the results toward zero, and the authors nonetheless find significant increases in blood pressure one year after the termination of benefits.

Since the termination of benefits was motivated by financial pressures on the state, it is possible that the state simultaneously cut back on other welfare programs that may have affected the treatment group (who were not categorically eligible for any Federal assistance programs) but not the control group.

Though this hypothesis is plausible, we are aware of no specific evidence that such cutbacks occurred. It is also possible that whatever criteria led individuals to be excluded from Medicaid are also correlated with less favorable outcomes over time.

For example, people who were continuously insured may have had more stable living circumstances and perhaps had a great interest in maintaining coverage and being compliant with medical advice; those persons whose benefits were cut might not have continued to be enrolled even without cuts.

Nevertheless, overall this case study offers evidence that losing health insurance coverage is associated with declines in health status.

Fihn and Wicher: VA cutbacks

Fihn and Wicher (1988) report the results of a natural experiment involving the cancellation of Veterans’ health benefits for a group of Seattle area beneficiaries in 1983.

Because of a budget shortfall, regular outpatient services at the Seattle VA Medical Center (VAMC) were terminated for veterans who had no “service-connected disability”, had not been admitted to the VAMC during the previous year, and had not had a scheduled outpatient visit in the past three months.

Physicians could appeal these terminations on a case-by-case basis and, if they could demonstrate the “medical instability” of a given patient, his benefits would not be cancelled. As a result, 89 of the original 360 patients targeted for cancellation in fact retained their eligibility for outpatient services.

These 89 patients were treated as the “control group.” Twenty patients initially retained were later discharged and were excluded from the analysis; the remaining 251 individuals form the “treatment group” whose benefits were terminated.

The authors obtained follow-up data 16 months after termination on 69% (n=172) of the treatment group and 91% (n=82) of the control group. This does not include the 6% of the treatment group and 8% of the control group who had died.

In addition to questions about access to medical care and general health status, the authors measured the subjects’ blood pressure.

Both systolic and diastolic blood pressure appear very similar for the treatment and control groups before the termination of coverage (the authors do not report a test of the hypothesis that the before-termination means differ across groups).

At the 16-month follow up, the treatment group had increased statistically significant increases in both systolic (+11.2 mm Hg, p<.001) and diastolic (+5.6 mm Hg, p<.001) blood pressure. In contrast, the control group had experienced insignificant changes to both systolic (+0.5 mm Hg) and diastolic (-2.5 mm Hg) blood pressure.

In addition, a significantly higher fraction of treatments than controls reported at the 16-month follow up that their health was “much worse” than it had been at baseline (41% vs. 8%, p<0.001).

At follow-up, the treatment group was also substantially less likely than the control group to identify a usual source of care (70% vs. 100%, p<0.001) and to be satisfied with their present medical care (41% vs. 100%, p<0.001).

The treatment group was also substantially more likely to report having reduced the number of prescribed medications (including antihypertensive medications) (47% vs. 25%, p<0.002). Several of these effects were greater for persons with lower incomes.

The combination of worsened outcomes and declines in utilization that are especially prominent among lower income persons who lose coverage is certainly suggestive of a true effect of insurance on health, but there are also some clear problems with this study.

In addition to the very small sample size, the treatment and control groups were not truly randomized, since they were determined by the selective exemption of some patients from benefit termination because of doctors’ efforts on their behalf.

One might expect that this would result in the control group representing sicker patients than the treatments, since they were those for whom doctors demonstrated “medical instability.”

This type of selection might result in a conservative estimate of the treatment effect if sicker patients would be independently more likely to experience declines in health status. In fact this seems not to have been the case; some comorbid conditions, such as coronary artery disease, were more prevalent among the treatment group.

The authors attribute this “to the fact that physicians in the busy cardiology clinic allowed almost all targeted patients to be discharged and rarely appealed the decision” (p. 359). This points to the possibility that patients themselves may have played a role in advocating for maintenance of their benefits in some cases.

It is not unlikely that patients who are more concerned about their own health will be more likely both to advocate for maintenance of their insurance and to be compliant with a treatment regimen. If so, the relationship between continued coverage and health status could well reflect unobservable patient characteristics rather than the effects of health insurance.

This feature of the study makes it very unclear how successful the “randomization” to treatment or control groups was at eliminating any correlation of treatment with unobservable determinants of health.

Another thing to note about the Fihn and Wicher study is that the natural experiment on which it is based may be more appropriate for studying the impact of medical care access, rather than insurance, on health.

Eligibility for VA outpatient services functioned as a form of insurance, but in practice it may have been as if the usual source of care for these men had shut down.

Thus, these men not only lost their previous insurance coverage, but also access to their usual set of health care providers with whom at least some of them had presumably established meaningful relationships.

The impact of such events on health is interesting in its own right but may be fundamentally different from the impact of a change in insurance on health, and it is the latter effect that we are concerned with here.

Haas et al.: Healthy Start expansions

Haas et al. (1993a, 1993b) examine the impact of the Massachusetts Healthy Start program on maternal health. This program, begun in December 1985, provided health insurance coverage for pregnant women with incomes up to 185% of the poverty line.

Medicaid coverage at that time in Massachusetts covered pregnant women up to 100% of the poverty line. In 1987, according to Haas et al., 54% of women who gave birth and had neither private coverage nor Medicaid were covered by Healthy Start.

The data consist of hospital discharge data merged to vital statistics records for nearly all live in-hospital births in Massachusetts in fiscal year 1984 (final n= 57,257) in fiscal year 1987 (final n = 64,346).

The research strategy of Haas et al. consists of comparing changes in medical care use and maternal and infant health for a treatment group consisting of women with neither private insurance nor Medicaid (“the uninsured”) to changes in these outcomes for Medicaid recipients and for the privately insured.\

Any change in these outcomes for the treatment group compared to either privately insured patients or Medicaid recipients is attributed by the authors to the expansions of insurance coverage among the treatment group that occurred between 1984 and 1987.

Haas et al. find no statistically significant changes in the following outcomes for the treatments compared to either privately insured or Medicaid controls: the incidence of adverse birth outcomes (low birth weight or prematurity), the fraction of women receiving satisfactory prenatal care, the fraction of women initiating care before the third trimester, and adverse maternal health outcomes (pregnancy-related hypertension, placental abruption, and a hospital stay longer than the infant’s).

In fact, the only outcome to show any significant change between 1984 and 1987 in the uninsured/insured differential is cesarean section rates, which increased for women in the treatment group from 17.2% to 22.4% (+5.2 percentage points) and for privately insured women from 23.0% to 25.9% (+2.9 percentage points).

However, as the authors note, there is no change in either maternal or infant outcomes corresponding to this change in procedure use.

One feature of the Haas et al. studies is that they assume that all the women newly insured by Healthy Start had been uninsured, and do not consider the possibilitythat some of the women might have had private insurance.

To the extent that some of them had been covered previously by private insurance – and there is evidence for the subsequent Medicaid expansions that approximately one-third of newly eligible recipients had previously been covered by private insurance (Cutler and Gruber 1996) – the measured effect on birth outcomes may be smaller than if the expansions had truly reached a group of previously uninsured women.

This is, however, not so much a problem with a design of the study as a feature of the expansions themselves. If the expansions did not result in net increases in insurance, then it would not be surprising that there was no improvement in health outcomes.

We will discuss this in more detail below in the context of the Medicaid expansions. Taken as a whole, these three case studies provide mixed evidence on the effect of insurance on health.

The Lurie et al. and Fihn and Wicher studies strongly suggest that cutting back on insurance coverage in a vulnerable, low-income population has the potential to increase blood pressure significantly.

On the other hand, the Haas et al. studies suggest that expanding coverage to pregnant women may not affect health outcomes for them or their infants, even though it may result in changes in medical care utilization.

The literature contains studies relying on five large-scale natural experiments: the passage of Medicare in 1965, expansions of Medicaid eligibility in the 1980s and 1990s, the passage of National Health Insurance in Canada, the variation across states in the generosity of insurance coverage for HIV patients, and the much lower rates of health insurance coverage among selfemployed workers than among wage-and-salary workers. In this section we discuss each of these studies in detail.

Lichtenberg: The enactment of Medicare

Lichtenberg (2001) uses data from U.S. Vital Statistics, the National Hospital Discharge Survey, the National Health Interview Survey, the National Ambulatory Medical Care Survey to examine the effects of Medicare on the health of older Americans by looking for evidence of abrupt discontinuities in health care utilization and outcomes at age 65, when people typically first become eligible for Medicare.

He finds evidence that utilization of ambulatory care and, to a smaller extent, inpatient care, increases abruptly at age 65. Lichtenberg then examines whether there is a reduction in morbidity and mortality at age 65 relative to the trends in outcomes prior to that age.

The results show a reduction in days spent in bed of about 13% as well as a 13% reduction in the probability of death after age 65 compared to what they would have been in the absence of Medicare.

Lichtenberg also examines whether the increase in health care utilization and the improvements in outcomes around age 65 over time are associated with each other.

Indeed, he finds that conditional on age and the death rate in the previous year, the short-run elasticity of the death rate with respect to the number of physician visits is -0.095, and the long-run elasticity is -0.497 so that a sustained 10% increase in the number of visits will reduce the death rate by 5%.

Some further insight into this association may be provided by the fact that the number of physician visits in which at least one drug is prescribed also increases suddenly at age 65.

Better characterizing which drugs are prescribed might be particularly useful in understanding how these additional visits might result in improved health.

Another interesting finding is that the increase in the consumption of hospital services at age 65 is preceded by a decline in hospital utilization at ages 63 and 64, suggesting that at least some of this increase results from postponement of hospitalization in the prior two years.

Lichtenberg’s findings suggest a powerful effect of Medicare on both utilization and health outcomes, but alternative interpretations are possible. One is that 65 is also a common age of retirement, and retirement may result more time available for health care and thus improved health.

To address this argument, Lichtenberg points out that 62% of workers have already retired by age 64, but this does not rule out the possibility of a spike in retirement at age 65 that might result in a (negative) spike in mortality.

This possibility could be tested definitively by determining whether the spike in utilization and decline in mortality at age 65 are present for people who remain employed.

Lichtenberg does not perform this test, however, he does examine whether there is a difference in the discontinuity of mortality rates at age 65 prior to the year 1965 (when Medicare began) compared to after 1965. He finds no evidence of a discontinuity prior to age 65, but strong evidence after 1965.

It seems likely that this reflects a change due to the implementation of Medicare, but it is also possible that it reflects a change in the spike in retirement at age 65, which might also have intensified with the establishment of Medicare.

A related set of tests might also examine whether discontinuities in outcomes might differ for people depending on whether they have health insurance prior to age 65, and this is a valuable area for future work. Lichtenberg’s preliminary findings suggest there is a significant effect of health insurance on health for persons at retirement age.

Location: Indonesia