Reproducibility and its discontents

A widely cited (but unreproducible) therapeutic study caused the most difficult moral choice I ever faced in decades of neurologic practice. Lack of reproducibility is currently in the news big time thanks to the work of 270 or so volunteer psychologists [ Science vol. 349 pp. 910 – 911, 943, aa47161 to 8 ’15 (28 August) ]. They tried to reproduce 100 significant studies appearing in 3 high level psychology journals in 2008. They had the cooperation of the original authors. Sadly, they were only replicate slightly over 1/3, and the effect sizes of the original 100 were half that on replication, and in a few cases, the effects were opposite.

In what follows you will encounter reasoning of almost talmudic intricacy. Don’t give up. The real world effects were far from trivial, and undoubtedly some died of therapeutically induced intracerebral hemorrhage.

The study was large and done under NIH auspices [ New England J. Med. vol. 333 pp. 1581 – 1587 ’95 ] and had an extremely logical basis — if some strokes are due to blood clots blocking vessels, administer an agent to break up the clot and restore circulation, before the part of the brain lacking circulation irreversibly dies. The agent was Tissue Plasminogen Activator (tpa).

624 patients were studied (so it had significant statistical power to detect an effect) . Treatment had to be initiated within 3 hours (about this more later) As compared with patients receiving placebo, patients with tPA were 30% more likely to have minimal or no disability at 3 months (the absolute increase in good outcome was 11 – 13%) — e.g. without tpa roughly 30% of the placebo group had good outcomes at 3 months, with tpa the figure was 41 – 43%. So the therapy was no guarantee of success.

There was a huge amount of interest in the paper and letters to the NEJM soon followed. One from Mass. General noted that rapid neurologic improvement expected after recanalization was rare in the treated group (there were few differences at 24 hours between the two groups ). This surprised my cardiologist friends, long used to giving clot busting (thrombolytic) agents to patients with acute heart attacks (myocardial infarctions) due to coronary occlusion and watching EKG changes vanish before their eyes along with improved cardiac function.

Followup of the 624 was impressive (609), and results one year out showed that the 11 – 13% absolute increase in good outcome persisted [ New England J. Med. vol. 340 pp. 1781 -1787 ’99 ]. Of interest was the risk of another stroke during this time — was 6% in both treated and untreated. Hopefully that figure is lower now (I don’t know as I’m no longer in practice and reading the literature).

There are a variety of scales to measure stroke severity and stroke outcome. The tpa and placebo groups had the same average National Institutes of Health Stroke Severity scores (NIHSS scores) on entry to the study — the higher the number the more severe the stroke. The authors used something called the modified Rankin score to measure outcome (here lower is better).

There was a huge push to get everyone to use tpa for stroke hitting the ER in the first few hours. The evidence seemed pretty flimsy and the effect pretty minimal to me (and to most clinical neurologists that I talked to). Hospitals used it as an advertising tool. The legal profession geared up to sue docs who didn’t use it. Fortunately the criteria for use were quite strict (within 3 hours of stroke, no hypertension, etc. etc.). The academic industrial complex weighed in, with the author of one of the subsequent studies threatening to testify in malpractice cases against docs who didn’t use it.

There was a lot of resistance on the part of clinicians and ER docs to using the therapy, but we had no blogosphere then.

Results when the therapy was adopted in the community were not happy ones. Here’s just one example [ Stroke vol. 30 pp. 34 – 39 ’99 ] A study of 138 patients treated in Houston and Calgary from 12/95 to 7/98 showed a much higher rate of symptomatic intracerebral hemorrhage (9 % == 13/138) and the any hemorrhage rate was also high (30% — 42/138). The only predictor was blood glucose. 7/13 symptomatic hemorrhage patients died and the rest were severely disabled.

This was not a therapy to be undertaken lightly, particularly if you thought the evidence was flimsy.

A technique was available (TransCranial Doppler) to actually look at blood flow inside the head in someone with a stroke, and this was later applied to patients receiving tpa. It involved continuous ultrasound monitoring, which one plucky neurologist actually thought was breaking up the clot (rather than the tpa) Robinson, L. Clototripsy? Stroke 2000; 31 2024. Clototripsy is a play on lithotripsy a therapy in widespread use to focus sound energy on kidney stones to break them up.

It was suggested that a 3 pronged study be done on patients with acute stroke due to arterial occlusion (1) tpa alone (2) TransCranial Doppler alone (3) Tpa + TransCranial doppler. This was never done.

A later study [4] Alexandrov AV, Molina CA, Grotta JC, Garami Z, Ford, SR, Alvarez-Sabin J, Montaner J, Saqqur M, Demchuk AM, Moye LA, Hill MD, Wojner AW. Ultrasound-Enhanced Systemic Thrombolysis for Acute Ischemic StrokeN. Engl. J. Med. 2004; 351: 2170 – 2178
was devastating (although the authors didn’t realize it). Why?

29% of patients receiving tPA alone had a modified Rankin score of 0 – 1 at 3 months (e.g. a good result). But 29% is practically the same as 26% — the percentage of good outcome in patients in the original 1995 study who received no tPA whatsoever. A mere difference of 3 points on the initial NIHSS score between the two groups is not explanatory.  The 41 – 43% good outcome of the original study was nowhere in sight.

So tpa was actually no better than placebo when retested 9 years later. The authors had failed to replicate the original work.

I tried to get the following letter in the New England Journal back then.

To the editor:

The initial results of continuous 2 MHz transcranial Doppler (TCD) monitoring during tPA infusion for acute ischemic stroke were exciting and unprecedented. [1] Since no study up to then had replicated the NINDS results [2], I argued that the dramatic results with continuous TCD were probably real, and quite likely independent of tPA [3]. I urged that a three-pronged study (tPA alone, TCD alone, and tPA plus TCD) should be carried out.

The recent two-pronged study [4] also fails to replicate the NINDS study. Only 29% of patients receiving tPA alone had a modified Rankin score of 0 – 1 at 3 months. Even worse, 26% of patients in the NINDS study receiving no tPA whatsoever had this outcome. A mere difference of 3 points on the initial NIHSS score between the two groups is not explanatory. After 9 years our diagnostic techniques have improved and we are presumably smarter about whom to treat. It’s time to do the study I suggested and to stop regarding tPA for stroke as ‘standard of care’.

[1] Alexandrov AV, Demchuk AM, Felberg RA, Christou I, Barber PA, Burgin WS, Malkoff M, Wojner AW, Grotta JC. High rate of complete recanalization and dramatic clinical recovery during tPA infusion when continuously monitored with 2-MHz transcranial Doppler monitoring. Stroke. 2000; 31: 610 – 614.

[2] The National Institutes of Neurological Disorders and Stroke rt-PA Stroke Study Group. Tissue plasminogen activator for acute ischemic stroke. N. Engl. J. Med. 1995; 333: 1581 – 1587

[3] Robinson, L. Clototripsy? Stroke 2000; 31 2024

[4] Alexandrov AV, Molina CA, Grotta JC, Garami Z, Ford, SR, Alvarez-Sabin J, Montaner J, Saqqur M, Demchuk AM, Moye LA, Hill MD, Wojner AW. Ultrasound-Enhanced Systemic Thrombolysis for Acute Ischemic Stroke
N. Engl. J. Med. 2004; 351: 2170 – 2178

It was not published.

Thank God for the blogosphere. Often irritating and irascible but extremely useful.

So what do you think I did faced with a patient satisfying the criteria for tpa, knowing that the therapy could (and had) killed, and not thinking that it was much good. Satisfy my conscience and my responsibility to the patient by not giving it and risk financial destruction, or do what I thought was wrong and  protect my family with the full weight of medical legal precident to back me up.

What would you do in this situation?

It’s important to realize, that all this took place a decade ago, and that new information and studies may have come out, showing that tpa actually is of use in acute stroke.  I wouldn’t rely on meta-analyses of the old studies however.

Nonetheless, the above reasoning is still valid.  Docs do the best they can with the information available to them at the time (which is NEVER enough) and this was the state of play when I was in practice.

Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: