For years, clinical practice guidelines promulgated by prominent health care organizations have been hailed with accolades as received wisdom. However, there is increasing reason to be skeptical of such guidelines. Many guidelines are not based on rigorous application of the principles of evidence-based medicine, and often seem to arise from the personal opinions of their authors. This is particularly troublesome when those authors have conflicts of interest, and when the organizations that sponsor guideline development have institutional conflicts of interest. Back in 2011, an Institute of Medicine panel advocated standards for guideline development, including strict limits on conflicted panel members, to make their results more trustworthy. However, as we noted here, those standards have been largely ignored.
Therefore, it is good news that the just released, long awaited guidelines on the treatment of blood cholesterol from the American College of Cardiology (ACC) and American Heart Association (AHA)(1) they provoked controversy rather than adulation. However, connecting some dots reveals that the guideline development process and the defenses of the guidelines by its developers were even more confused than they first seemed. That confusion may be explained by conflicts of interest affecting guideline development of the sort that the IOM report wanted eliminated. .
The New Cholesterol Guidelines - Background
A striking feature of the guidelines was a new approach to drug treatment for primary prevention, that is, for patients who do not already have heart disease or other atherosclerosis. Such drug based primary prevention has been controversial, although drug treatment for people with high cholesterol who also have documented coronary heart disease, secondary prevention, is well-established.
The new cholesterol treatment guideline suggested drug treatment, essentially limited to statin medications, for patients believed to be at elevated risk of developing coronary artery disease or other forms of atherosclerotic disease, even in the absence of elevated cholesterol levels.
Also,
Previously, guidelines and other recommendations for the treatment of cholesterol for primary prevention, that is, for patients without known atherosclerotic cardiovascular disease, suggested treatments according to the level of cholesterol or its components.
Less controversially, the guidelines recommended treatment for patients with existing ASCVD, very high LDL-C, and diabetes.
The new guidelines raise some questions:
- Given the past controversy, how good is the evidence supporting cholesterol lowering drug treatment in primary prevention?
- What is the evidence supporting deciding on drug treatment for primary prevention on predicted risk of atherosclerotic cardiovascular disease?
- Can physicians make good enough risk predictions to use this approach?
Evidence Supporting Drug Treatment of Cholesterol in Primary Prevention - Do Benefits Outweigh Harms?
The reason that cholesterol lowering drug use for primary prevention has been controversial is the lack of clear evidence that such drug use leads to benefits to patients that outweigh its harms. A central principle of evidence based medicine is that only treatments whose benefits clearly outweigh their harms should be prescribed. .
A recent commentary by Abramson et al in the British Medical Journal in October, 2013 outlines the issues.(2) There is no good evidence that statins used in primary prevention increase overall survival, or decrease overall incidence of adverse events, defined as death, hospital admission, prolongation of admission, cancer or permanent disability.
Individual trials and meta-analyses do show that statins lead to a small reduction in the rate of cardiovascular events. For example, the authors' re-analysis of data from a patient level meta-analysis showed that of 140 low-risk primary prevention patients treated for five years, one patient would avoid a major coronary event or stroke.
However, it is not clear that this small likelihood of benefit offsets the likelihood of adverse events due to treatment. The data about the harms of statins in primary prevention is not very clear, partially because the relevant randomized controlled trials featured "reporting of adverse events ... [that was] generally poor, 'with failure to provide details of severity and type of adverse events or to report on health-related quality of life."
In summary, Abramson et al wrote,
However, the new guideline focused on the ability of statins to prevent ASCVD,
Furthermore, the guideline seemed unreasonably optimistic about the harms of statins. In particular, it ignored evidence about harms other than diabetes, myopathy (serious muscle disease), and stroke. Yet there is evidence suggesting that statins at least might cause "liver dysfunction, acute renal failure, and cataracts; cognitive symptoms, neuropathy, and sexual dsyfunction; decreased energy and exertional fatigue; and psychiatric symptoms, including depression, memory loss, confusion, and aggressive reactions" as summarized by Abramson and colleagues.
Thus the new guidelines did not make a new and improved case that statin treatment for primary prevention has benefits that outweigh its harms. One could argue that this is their fatal flaw and thus all the rest of the guidelines' discussion about which patients should get primary prevention was pointless.
Dr Abramson and his co-author, Dr Rita Redberg, did get to repeat their arguments about why statin use for primary prevention is not justified in a NY Times op-ed, but otherwise the fundamental problem with the guideline's argument for aggressive use of statins went unnoticed.
Evidence Supporting Making Decisions about Statin Treatment as Primary Prevention According to Patients' Risks of Developing Atherosclerotic Cardiovascular Disease
The guidelines suggested statin therapy for patients judged to have at least a 7.5% risk of ASCVD. As noted above, there is no evidence that statin therapy in primary prevention in general increases survival, reduces serious events overall, improves health status, quality of life, or function, or has benefits that clearly outweigh its harms. I could not find any reference in the guidelines to clear evidence about the effects of statin treatment in primary prevention for this sub-group of patients.
In the supplemental guideline on risk assessment,(3) there was this
The wording is confusing, but might be stating that a strategy of using statins only for patients predicted to have a risk greater than 7.5% has not been assessed in a clinical trial. (It might also, however, be stating that the specific method recommended by the guideline to assess this risk has never been tested in a clinical trial, see below.)
The apparent lack of evidence in support of the specific strategy advocated by the guideline to use statins for primary prevention in patients whose risk of ASCVD exceeded the designated threshold did not otherwise attract any notice.
Evidence Supporting the Ability of Physicians to Assess Risk of ASCVD Accurately Enough to Implement the Recommended Strategy
The supplemental guideline stated that previously published methods of risk prediction would not be suitable for the new proposed approach,
Since the recommended strategy requires an estimation of risk, the lack of availability of acceptable risk assessment methods would appear to be another fatal flaw. However,
So the question now becomes: did the guideline provide sufficient evidence that the new risk prediction tool developed as part of guideline development predicts sufficiently well to be used to make decisions as recommended by the guidelines?
Note that developing, validating, and publishing a new risk score normally would be considered to be tasks that are part of research, not guideline development. Nonetheless, the guideline developers decided to take on these tasks as part of guideline development, which precluded independent publication of the the results of this research after peer review. This makes it more difficult for others to critically evaluate the results of the research concerning the new risk score. But let me attempt to do so.
The work group went ahead to develop a new multivariate prediction model for ASCVD. This means they used statistical methods to find variables, that is, patients' clinical or demographic characteristics, that independently could predict the outcome of interest, and then combined these variables in an equation (or algorithm) to make risk predictions for individual patients. Such multivariate models to make diagnoses or, as in this case, prognostic predictions have been the subject of considerable research since at least the 1970s. However, they have not been as useful as their initial advocates hoped.
The issues turn out to be a bit complex. I must digress into an area in which I previously did research.
It is quite easy to develop multivariate diagnostic or prediction models. The statistical analytic tools can almost always find multiple patient characteristics that independently correlate with the outcome of interest. The problem is that such correlations can be based on random associations, or biases produced by idiosyncrasies in the particular data set used for model development. That a new model can diagnose or predict accurately for patients who were not in the original data set thus is not assured, and should be considered to be merely a hypothesis.
Models developed on one group of patients often do not work when tried prospectively on new patients, probably because the initial development group was somehow not completely representative of all the patients of interest. For example, a model may have been developed using patients from a particular hospital which attracts different sorts of patients than found in other hospitals in which the model might be used. .Therefore, before one has confidence in such a multivariate model, one must verify that the model predicted or diagnosed well not only in the group of patients from which it was derived, but prospectively on other patients like those on whom it would be used in clinical practice.
The supplemental guidelines did assert that the new model was prospectively tested,
However, it did not specify what these cohorts were.
There is some data buried in Appendix 4 of the supplemental guideline on the performance of the model. They did not distinguish results from derived from prospective validation from those from model derivation.
These two sentences do not provide strong evidence that the model would predict well when used prospectively.
The C-statistic is an overall measure of the ability of the model to discriminate between patients who will go on to develop the diseases of interest and those who will not. The statistic is formally equal to the probability that were two patients, one who developed disease, and another who did not, selected randomly, the model would predict disease more strongly for the patient who actually got it. Thus, at best, for 20% of random pairs of African-American women, one who got disease, one who did not, the model would give a more pessimistic prediction for the women who would not get disease. This is good, but not great discrimination ability. (If this result referred to data from model derivation, not prospective validation, it is likely to be over-optimistic.)
Furthermore, since the guideline would be used to make decisions based on the absolute risk, its calibration is also important. Calibration is the measure of whether the model predictions of risk are close to reality. To assess calibration, one ought to assess the whole range of predictions made by the model. Given that the guidelines suggest a 7.5% risk threshold, it would be particularly important to determine whether patients given predictions above and below that value really have risks above and below that value.
Unfortunately, the two sentences above are not helpful in this regard. The chi-square statistic presented is a measure of overall calibration, but does not show calibration for groups of patients given predictions with particularly interesting values, like above or below 7.5%
Note that the supplemental guideline on risk assessment includes a statement that details about the model validation done "internally and externally" are in a Full Panel Report Data Supplement. The download of that supplement so far does not seem to work
Setting that aside for a moment, the published guidelines do not provide good evidence that the risk prediction tool the guidelines recommend should be used to assess ASCVD risk in order to make decisions on statin use for individual patients in primary prevention. The lack of a sufficiently accurate method to predict risk seems to a fatal flaw for a strategy that requires risk prediction.
Controversy in the Media
It turns out that I was not the first person to identify this problem. In fact, it appears that during the internal review process for the guidelines, major questions had already been raised about the risk prediction model's calibration. However, these questions did not seem to have been conveyed to the guideline authors, and hence were never addressed.
As reported by the New York Times on November 17, 2013,
The article by Ridker and Cook was indeed published. on November 19, 2013(4). It suggested major problems with the calibration of the risk assessment model,
Note that Figure 1 of the Ridker and Cook article showed the calibration of the model in the patient cohorts newly tested by these authors, focusing in particular on whether patients predicted to have risks of cardiovascular disease just over the 7.5% threshold actually had rates of such disease greater than 7.5%. They clearly did not Note also that Figure 2 showed calibration of the model when the guideline authors attempted to test it on new patient cohorts, apparently the data found in the so far inaccessible Full Panel Report Data Supplement. Again, the model overestimated risk for patients just over the crucial threshold.
So the guideline developers knew that their model overestimated risk, buried this information in supplemental data, did not admit or perhaps appreciate how it threatened the credibility of their guidelines, and somehow were not given the internal review that suggested this was a fatal flaw of the proposed guidelines .
The Guideline Developers' Responses
The media controversy over the accuracy of the prediction model incorporated into the new guidelines provoked responses from the guideline developers, but these responses were at best confused. .
First, as reported by the NY times,
Thus, after promulgating guidelines that seemed hardly based on evidence, when challenged, the guideline developers' response was confused and illogical.
What About the Issue of Benefits versus Harms?
The other problems that I identified above, lack of evidence that primary prevention provides benefits that clearly outweigh harms, and lack of evidence that a strategy based on risk assessment would provide benefits that outweighs harms, did not get media coverage, and so did not provoke a response from the developers. However, I did find that one of the members of the guideline panel implied her approach to the benefits versus harms issue in a Medscape news article. Dr Noel Bairey Merz gave a talk at the 2013 American Heart Association meeting about primary prevention for women, first saying
Also,
So Dr Merz seemed to assume that statins work for particular patients in the absence of evidence that they do not work, maybe implying a general assumption that all treatments are beneficial until proven otherwise. This stands a central precept of evidence-based medicine, and perhaps the ancient dictum to physicians to do no harm, on their heads.
References
1. Stone NJ, Robinson J, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2013. Link here.
2. Abramson JD, Rosenberg HG, Jewell N et al. Should statins be prescribed to people at low risk of cardiovascular disease. Brit Med J 2013; 347: 15-17. Link here.
3. Goff Jr DC, Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA Guideline on the Assessment of Cardiovascular
Risk. Journal of the American College of Cardiology (2013), doi: 10.1016/j.jacc.2013.11.005. Link here.
4. Rikder PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013; Link here.
5. Collier J. The price of independence. Br Med J 2006; 332: 1447-9. Link here
Therefore, it is good news that the just released, long awaited guidelines on the treatment of blood cholesterol from the American College of Cardiology (ACC) and American Heart Association (AHA)(1) they provoked controversy rather than adulation. However, connecting some dots reveals that the guideline development process and the defenses of the guidelines by its developers were even more confused than they first seemed. That confusion may be explained by conflicts of interest affecting guideline development of the sort that the IOM report wanted eliminated. .
The New Cholesterol Guidelines - Background
A striking feature of the guidelines was a new approach to drug treatment for primary prevention, that is, for patients who do not already have heart disease or other atherosclerosis. Such drug based primary prevention has been controversial, although drug treatment for people with high cholesterol who also have documented coronary heart disease, secondary prevention, is well-established.
The new cholesterol treatment guideline suggested drug treatment, essentially limited to statin medications, for patients believed to be at elevated risk of developing coronary artery disease or other forms of atherosclerotic disease, even in the absence of elevated cholesterol levels.
Adults 40 to 75 years of age with LDL–C [so called "bad cholesterol"] ]70 to 189 mg/dL, without clinical ASCVD* [atherosclerotic cardiovascular disease] or diabetes and an estimated 10-year ASCVD risk ≥7.5% should be treated with moderate- to high-intensity statin therapy.
Also,
It is reasonable to offer treatment with a moderate intensity statin to adults 40 to 75 years of age, with LDL–C 70 to 189 mg/dL, without clinical ASCVD* or diabetes and an estimated 10-year ASCVD risk of 5% to [less than] ... 7.5%
Previously, guidelines and other recommendations for the treatment of cholesterol for primary prevention, that is, for patients without known atherosclerotic cardiovascular disease, suggested treatments according to the level of cholesterol or its components.
Less controversially, the guidelines recommended treatment for patients with existing ASCVD, very high LDL-C, and diabetes.
The new guidelines raise some questions:
- Given the past controversy, how good is the evidence supporting cholesterol lowering drug treatment in primary prevention?
- What is the evidence supporting deciding on drug treatment for primary prevention on predicted risk of atherosclerotic cardiovascular disease?
- Can physicians make good enough risk predictions to use this approach?
Evidence Supporting Drug Treatment of Cholesterol in Primary Prevention - Do Benefits Outweigh Harms?
The reason that cholesterol lowering drug use for primary prevention has been controversial is the lack of clear evidence that such drug use leads to benefits to patients that outweigh its harms. A central principle of evidence based medicine is that only treatments whose benefits clearly outweigh their harms should be prescribed. .
A recent commentary by Abramson et al in the British Medical Journal in October, 2013 outlines the issues.(2) There is no good evidence that statins used in primary prevention increase overall survival, or decrease overall incidence of adverse events, defined as death, hospital admission, prolongation of admission, cancer or permanent disability.
Individual trials and meta-analyses do show that statins lead to a small reduction in the rate of cardiovascular events. For example, the authors' re-analysis of data from a patient level meta-analysis showed that of 140 low-risk primary prevention patients treated for five years, one patient would avoid a major coronary event or stroke.
However, it is not clear that this small likelihood of benefit offsets the likelihood of adverse events due to treatment. The data about the harms of statins in primary prevention is not very clear, partially because the relevant randomized controlled trials featured "reporting of adverse events ... [that was] generally poor, 'with failure to provide details of severity and type of adverse events or to report on health-related quality of life."
In summary, Abramson et al wrote,
statin therapy prevents one serious cardiovascular event per 140 low risk people (five year risk ... [less than] 10%) treated for five years. Statin therapy in low risk people does not reduce all cause mortality or serious illness and has about an 18% risk of causing side effects that range from minor and reversible to serious and irreversible. Broadening the recommendations in cholesterol lowering guidelines to include statin therapy for low risk individuals will unnecessarily increase the incidence of adverse effects without providing overall health benefit.
However, the new guideline focused on the ability of statins to prevent ASCVD,
The RCTs identified in the systematic evidence review indicated a consistent reduction in ASCVD eventsHowever, the new guideline ignored the absence of evidence that statin treatment prolonged life, prevented serious illness overall, or improved health status, function or quality of life.
from 3-hydroxy-3-methylglutaryl-coenzyme A reductase inhibitors (statins) therapy in secondary and primary prevention populations....
Furthermore, the guideline seemed unreasonably optimistic about the harms of statins. In particular, it ignored evidence about harms other than diabetes, myopathy (serious muscle disease), and stroke. Yet there is evidence suggesting that statins at least might cause "liver dysfunction, acute renal failure, and cataracts; cognitive symptoms, neuropathy, and sexual dsyfunction; decreased energy and exertional fatigue; and psychiatric symptoms, including depression, memory loss, confusion, and aggressive reactions" as summarized by Abramson and colleagues.
Thus the new guidelines did not make a new and improved case that statin treatment for primary prevention has benefits that outweigh its harms. One could argue that this is their fatal flaw and thus all the rest of the guidelines' discussion about which patients should get primary prevention was pointless.
Dr Abramson and his co-author, Dr Rita Redberg, did get to repeat their arguments about why statin use for primary prevention is not justified in a NY Times op-ed, but otherwise the fundamental problem with the guideline's argument for aggressive use of statins went unnoticed.
Evidence Supporting Making Decisions about Statin Treatment as Primary Prevention According to Patients' Risks of Developing Atherosclerotic Cardiovascular Disease
The guidelines suggested statin therapy for patients judged to have at least a 7.5% risk of ASCVD. As noted above, there is no evidence that statin therapy in primary prevention in general increases survival, reduces serious events overall, improves health status, quality of life, or function, or has benefits that clearly outweigh its harms. I could not find any reference in the guidelines to clear evidence about the effects of statin treatment in primary prevention for this sub-group of patients.
In the supplemental guideline on risk assessment,(3) there was this
After deliberation, the Work Group endorsed the existing and widely employed paradigm of matching the intensity of preventive efforts with the individual’s absolute risk.. The Work Group acknowledges that none of the risk assessment tools or novel risk markers examined in the present document have been formally evaluated in randomized controlled trials of screening strategies with clinical events as outcomes.
The wording is confusing, but might be stating that a strategy of using statins only for patients predicted to have a risk greater than 7.5% has not been assessed in a clinical trial. (It might also, however, be stating that the specific method recommended by the guideline to assess this risk has never been tested in a clinical trial, see below.)
The apparent lack of evidence in support of the specific strategy advocated by the guideline to use statins for primary prevention in patients whose risk of ASCVD exceeded the designated threshold did not otherwise attract any notice.
Evidence Supporting the Ability of Physicians to Assess Risk of ASCVD Accurately Enough to Implement the Recommended Strategy
The supplemental guideline stated that previously published methods of risk prediction would not be suitable for the new proposed approach,
As part of its deliberations, the Work Group considered previously published risk scores with validation in NHLBI cohort data as 1 possible approach. However, a number of persistent concerns with existing risk equations were identified including nonrepresentative or historically dated populations, limited ethnic diversity, narrowly defined endpoints, endpoints influenced by provider preferences (e.g., revascularizations), and endpoints with poor reliability (e.g., angina and heart failure [HF]).
Since the recommended strategy requires an estimation of risk, the lack of availability of acceptable risk assessment methods would appear to be another fatal flaw. However,
Given the inherent limitations of existing scores, the Work Group judged that a new risk score was needed to address some of the deficiencies of existing scores, such as utilizing a population sample that approaches, to the degree possible, the ideal sample for algorithm development and closely represents the U.S. population.
So the question now becomes: did the guideline provide sufficient evidence that the new risk prediction tool developed as part of guideline development predicts sufficiently well to be used to make decisions as recommended by the guidelines?
Note that developing, validating, and publishing a new risk score normally would be considered to be tasks that are part of research, not guideline development. Nonetheless, the guideline developers decided to take on these tasks as part of guideline development, which precluded independent publication of the the results of this research after peer review. This makes it more difficult for others to critically evaluate the results of the research concerning the new risk score. But let me attempt to do so.
The work group went ahead to develop a new multivariate prediction model for ASCVD. This means they used statistical methods to find variables, that is, patients' clinical or demographic characteristics, that independently could predict the outcome of interest, and then combined these variables in an equation (or algorithm) to make risk predictions for individual patients. Such multivariate models to make diagnoses or, as in this case, prognostic predictions have been the subject of considerable research since at least the 1970s. However, they have not been as useful as their initial advocates hoped.
The issues turn out to be a bit complex. I must digress into an area in which I previously did research.
It is quite easy to develop multivariate diagnostic or prediction models. The statistical analytic tools can almost always find multiple patient characteristics that independently correlate with the outcome of interest. The problem is that such correlations can be based on random associations, or biases produced by idiosyncrasies in the particular data set used for model development. That a new model can diagnose or predict accurately for patients who were not in the original data set thus is not assured, and should be considered to be merely a hypothesis.
Models developed on one group of patients often do not work when tried prospectively on new patients, probably because the initial development group was somehow not completely representative of all the patients of interest. For example, a model may have been developed using patients from a particular hospital which attracts different sorts of patients than found in other hospitals in which the model might be used. .Therefore, before one has confidence in such a multivariate model, one must verify that the model predicted or diagnosed well not only in the group of patients from which it was derived, but prospectively on other patients like those on whom it would be used in clinical practice.
The supplemental guidelines did assert that the new model was prospectively tested,
The equations were also assessed in external validation studies using data from other available cohorts
However, it did not specify what these cohorts were.
There is some data buried in Appendix 4 of the supplemental guideline on the performance of the model. They did not distinguish results from derived from prospective validation from those from model derivation.
In summary, discrimination and calibration of the models were very good. C statistics ranged from a low of 0.713 (African-American men) to a high of 0.818 (African-American women). Calibration chi-square statistics ranged from a low of 4.86 (nonHispanic White men) to a high of 7.25 (African-American women).
These two sentences do not provide strong evidence that the model would predict well when used prospectively.
The C-statistic is an overall measure of the ability of the model to discriminate between patients who will go on to develop the diseases of interest and those who will not. The statistic is formally equal to the probability that were two patients, one who developed disease, and another who did not, selected randomly, the model would predict disease more strongly for the patient who actually got it. Thus, at best, for 20% of random pairs of African-American women, one who got disease, one who did not, the model would give a more pessimistic prediction for the women who would not get disease. This is good, but not great discrimination ability. (If this result referred to data from model derivation, not prospective validation, it is likely to be over-optimistic.)
Furthermore, since the guideline would be used to make decisions based on the absolute risk, its calibration is also important. Calibration is the measure of whether the model predictions of risk are close to reality. To assess calibration, one ought to assess the whole range of predictions made by the model. Given that the guidelines suggest a 7.5% risk threshold, it would be particularly important to determine whether patients given predictions above and below that value really have risks above and below that value.
Unfortunately, the two sentences above are not helpful in this regard. The chi-square statistic presented is a measure of overall calibration, but does not show calibration for groups of patients given predictions with particularly interesting values, like above or below 7.5%
Note that the supplemental guideline on risk assessment includes a statement that details about the model validation done "internally and externally" are in a Full Panel Report Data Supplement. The download of that supplement so far does not seem to work
Setting that aside for a moment, the published guidelines do not provide good evidence that the risk prediction tool the guidelines recommend should be used to assess ASCVD risk in order to make decisions on statin use for individual patients in primary prevention. The lack of a sufficiently accurate method to predict risk seems to a fatal flaw for a strategy that requires risk prediction.
Controversy in the Media
It turns out that I was not the first person to identify this problem. In fact, it appears that during the internal review process for the guidelines, major questions had already been raised about the risk prediction model's calibration. However, these questions did not seem to have been conveyed to the guideline authors, and hence were never addressed.
As reported by the New York Times on November 17, 2013,
The problems were identified by two Harvard Medical School professors whose findings will be published Tuesday in a commentary in The Lancet, a major medical journal. The professors, Dr. Paul M. Ridker and Dr. Nancy Cook, had pointed out the problems a year earlier when the National Institutes of Health’s National Heart, Lung, and Blood Institute, which originally was developing the guidelines, sent a draft to each professor independently to review. Both reported back that the calculator was not working among the populations it was tested on by the guideline makers.
That was unfortunate because the committee thought the researchers had been given the professors’ responses, said Dr. Donald Lloyd-Jones, co-chairman of the guidelines task force and chairman of the department of preventive medicine at Northwestern University.
The article by Ridker and Cook was indeed published. on November 19, 2013(4). It suggested major problems with the calibration of the risk assessment model,
Another concern for clinicians is whether the new prediction algorithm created by the ACC/AHA correctly assesses the level of vascular risk. To be useful, prediction models must not only discriminate between individuals with and without disease, but must also calibrate well so that predicted risk estimates match as closely as possible the observed risk in external populations. We calculated predicted 10-year risks of the same atherosclerotic events using the new ACC/AHA risk prediction algorithm and compared these estimates with observed event rates in three large-scale primary prevention cohorts, the Women's Health Study, the Physicians' Health Study, and the Women's Health Initiative Observational Study.
As shown in figure 1, in all three of these primary prevention cohorts, the new ACC/AHA risk prediction algorithm systematically overestimated observed risks by 75–150%, roughly doubling the actual observed risk. As shown in figure 2, similar overestimation of risk was observed in two external validation cohorts used by the guideline developers themselves, an issue readily acknowledged in the report. Thus, on the basis of data from these five external validation cohorts, it is possible that as many as 40–50% of the 33 million middle-aged Americans targeted by the new ACC/AHA guidelines for statin therapy do not actually have risk thresholds that exceed the 7·5% threshold suggested for treatment. Miscalibration to this extent should be reconciled and addressed in additional external validation cohorts before these new prediction models are widely implemented. It is possible, for example, that the five external validation cohorts are more contemporary than the cohorts used in the risk prediction algorithm and thus reflect secular improvements in overall health and lifestyle patterns in the USA over the past 25 years.
Note that Figure 1 of the Ridker and Cook article showed the calibration of the model in the patient cohorts newly tested by these authors, focusing in particular on whether patients predicted to have risks of cardiovascular disease just over the 7.5% threshold actually had rates of such disease greater than 7.5%. They clearly did not Note also that Figure 2 showed calibration of the model when the guideline authors attempted to test it on new patient cohorts, apparently the data found in the so far inaccessible Full Panel Report Data Supplement. Again, the model overestimated risk for patients just over the crucial threshold.
So the guideline developers knew that their model overestimated risk, buried this information in supplemental data, did not admit or perhaps appreciate how it threatened the credibility of their guidelines, and somehow were not given the internal review that suggested this was a fatal flaw of the proposed guidelines .
The Guideline Developers' Responses
The media controversy over the accuracy of the prediction model incorporated into the new guidelines provoked responses from the guideline developers, but these responses were at best confused. .
First, as reported by the NY times,
In a response on Sunday, Dr. [Sidney C] Smith of the guidelines committee said the concerns raised by Dr. Cook and Dr. Ridker 'merit attention.'
But, he continued, 'a lot of people put a lot of thought into how can we identify people who can benefit from therapy.' Further, said Dr. Smith, who is also a professor of medicine at the University of North Carolina and a past president of the American Heart Association, 'What we have come forward with represents the best efforts of people who have been working for five years.'
Note that this response includes two logical fallacies. First, it contained a straw man argument. It appeared to respond to accusations no one made. Nobody accused the guideline developers of being lazy, not putting in much effort, or not devoting much thought to the effort. The response also included an implied appeal to authority: big experts came up with these guidelines so their opinions should be credited, even in the presence of data to the contrary.
Also, as reported by CNN,
However, 'I can't speak to whether the calculator is valid or not,' Dr. Robert Eckel, co-chair of the American Heart Association committee that wrote the new guidelines and the association's past president, told CNN. 'That needs to be determined.''We trusted that the calculator worked,' he said. 'We trusted that the calculator is valid.'
This was confusing. Maybe Dr Eckel meant it to be another appeal to authority, the authority in question being that of the work group that developed the risk prediction model.
Furthermore, again according to CNN,
Researchers apparently did not receive the professors' responses, Dr. Donald Lloyd-Jones, chairman of the committee that developed the equation, told the Times.But Lloyd-Jones told reporters Monday, 'There's nothing wrong with these equations.'Committee members were aware there could be 'overestimation of risk in some populations,' he said.
In addition,
'Our risk assessment guideline doesn't tell you what to do. ... It just evaluates risk,' he said.
I am not sure this even rises to the level of a logical fallacy. It appears to be pure denial. The whole point of the risk assessment tool was to determine whether a patient's risk is above or below the thresholds suggested by the treatment guideline, and thus to tell you what to do. There clearly is something wrong with "these equations." Using them appears to vastly overestimate risk, and thus implementation of the guideline would probably lead to vast over treatment of real people .
What About the Issue of Benefits versus Harms?
The other problems that I identified above, lack of evidence that primary prevention provides benefits that clearly outweigh harms, and lack of evidence that a strategy based on risk assessment would provide benefits that outweighs harms, did not get media coverage, and so did not provoke a response from the developers. However, I did find that one of the members of the guideline panel implied her approach to the benefits versus harms issue in a Medscape news article. Dr Noel Bairey Merz gave a talk at the 2013 American Heart Association meeting about primary prevention for women, first saying
Although the randomized clinical trial evidence supporting primary prevention with statin therapy in women is not perfect, 'the absence of data means negative data.'
That is the argument formulated by Dr Noel Bairey Merz (Cedars Sinai Medical Center, Los Angeles, CA), who spoke today here at the American Heart Association 2013 Scientific Sessions.
'How confident are we that statins do not save lives in the week before a heart attack, but they do save lives the week after a heart attack, for women and men?'
Also,
in the overall JUPITER study of 18 000 patients, there was no treatment benefit when women were studied as a subgroup. Merz argues that JUPITER is powered for the total sample size only, not for women alone. In addition, the statistical test for heterogeneity revealed the interaction by sex was not statistically significant.
'Pretty much all the subgroups fall beyond the statistically significant range,' said Merz. 'So should we withhold treatment for women, who now are the majority of victims of cardiovascular disease, because of low precision and a trial that was not designed to address or answer this question?'
So Dr Merz seemed to assume that statins work for particular patients in the absence of evidence that they do not work, maybe implying a general assumption that all treatments are beneficial until proven otherwise. This stands a central precept of evidence-based medicine, and perhaps the ancient dictum to physicians to do no harm, on their heads.
Unwarranted Enthusiasm for (Over) Treatment and Conflicts of Interest
The new cholesterol guidelines, and those who developed them, seem enthused about the treatment of cholesterol with statin drugs for primary prevention in the absence of evidence that such treatment produces benefits that outweigh its harms. They also seem enthused about basing treatment decisions on a statistical prognostic model that has not been shown to be accurate, and in fact which appears to be biased towards promoting excess treatment in the context in which it would be used. The excess enthusiasm occurred in spite of evidence, and at times in spite of logic.
One possible reason that the guideline developers got so enthused that they seemed unable to think straight appears to be their own conflicts of interest, as first publicly noted in a post on Pharmalot. Reviewing the disclosure forms provided with the guidelines revealed more detail.
Of the 13 people on the main treatment guideline panel who were not NHLBI staffers serving ex-officio, 7 had financial relationships with pharmaceutical companies that manufacture statins:
- Jennifer Robinson, co-chair, research funded by AstraZeneca and Merck;
- C Noel Bairey Merz, consulting for Abbott, Bristol-Myers Squib Novartis, and Pfizer;
- Robert H Eckel, consulting for Merck, Pfizer and Abbott;
- Anne Carol Goldberg, consulting for Abbott and Merck, research funded by Abbott, Merck, and Novartis;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer;
- Karol Watson, consulting for Abbott, AstraZeneca, Merck and Pfizer, research funded by Merck;
- Peter W F Wilson, consulting for and research funded by Merck.
Of the 10 expert reviewers for this panel, 3 had financial relationships with pharmaceutical companies that manufacture statins
- William Virgin Brown, consultant for Abbott, Bristol-Myers Squibb, and Pfizer;
- Matthew Ito, consultant for Kowa;
- Robert S Rosenson, consultant for Novartis and Pfizer.
Of the 11 people on the risk prediction panel who were not NHLBI staffers serving ex-officio, 5 had financial relationships with pharmaceutical companies that manufacture statins
-David C Goff Jr, co-chair, research funded by Merck;
- Raymond Gibbons, consultant for AstraZeneca;
- Jennifer Robinson, research funded by AstraZeneca, and Merck;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer [although these relationships were not listed as relevant to this panel, but found in the listing for the panel above];
- Peter W.F Wilson, consultant for and research funded by Merck.
Also, in the Abramson and Redberg NY Times op-ed, the authors noted
As noted on Pharmalot, the prevalence of conflicted panel members did not appear to conform to the standards for the development of trustworthy guidelines recently published by the Institute of Medicine:
Also, noted by the Los Angeles Times was this comment from Dr John Abramson, lead author of the commentary on statins in primary prevention(4),
As Joe Collier observed, "people who have conflicts of interest often find giving clear advice (or opinions) particularly difficult."(5)
This difficulty giving clear advice, when amplified by a guideline for a common problem supported by prestigious non-profit organizations, and promoted by vigorous public relations, could lead to "more than 45 million middle-aged Americans who do not have cardiovascular disease being recommended for consideration of statin therapy" (per Ridker and Cook[4]) unnecessarily, likely resulting in millions suffering unneeded side effects, and billions in costs.
Summary
Guidelines for management of a very common problem promulgated by a major medical society and a major disease oriented non-profit organization suggested a strategy that would vastly increase drug treatment of currently healthy patients. The strategy appears not to have been based on good evidence. When some of the problems with this evidence were pointed out, the guideline developers responded with illogic. Apparently many of the guideline developers have financial relationships with the drug companies that would most profit from increases in drug treatment as recommended by the guidelines Implementation of the new guidelines might results in millions of people in the US receiving unneeded drugs, with resultant side effects and costs. .
Do we need more examples of how conflicts of interest are causing the poor outcomes and excess costs that are wrecking our health care system? Do we need more excuses not to eliminate conflicts of interest from guideline development? Do we need more delay implementing the standards provided by the Institute of Medicine report on trustworthy guidelines? Do we need more excuses not to drastically reduce conflicts of interest affecting academic medicine, medical societies, and disease specific non-profits, specifically starting with the earlier (and so far generally disregarded) Institute of Medicine report on conflicts of interest in medicine?
While we in the US argue incessantly in the details of minor reforms of our supposed free health care market, we ignore the rot at its foundations. True health care reform would attack the conflicts of interest that have put money, not patients at the center of health care.
The new cholesterol guidelines, and those who developed them, seem enthused about the treatment of cholesterol with statin drugs for primary prevention in the absence of evidence that such treatment produces benefits that outweigh its harms. They also seem enthused about basing treatment decisions on a statistical prognostic model that has not been shown to be accurate, and in fact which appears to be biased towards promoting excess treatment in the context in which it would be used. The excess enthusiasm occurred in spite of evidence, and at times in spite of logic.
One possible reason that the guideline developers got so enthused that they seemed unable to think straight appears to be their own conflicts of interest, as first publicly noted in a post on Pharmalot. Reviewing the disclosure forms provided with the guidelines revealed more detail.
Of the 13 people on the main treatment guideline panel who were not NHLBI staffers serving ex-officio, 7 had financial relationships with pharmaceutical companies that manufacture statins:
- Jennifer Robinson, co-chair, research funded by AstraZeneca and Merck;
- C Noel Bairey Merz, consulting for Abbott, Bristol-Myers Squib Novartis, and Pfizer;
- Robert H Eckel, consulting for Merck, Pfizer and Abbott;
- Anne Carol Goldberg, consulting for Abbott and Merck, research funded by Abbott, Merck, and Novartis;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer;
- Karol Watson, consulting for Abbott, AstraZeneca, Merck and Pfizer, research funded by Merck;
- Peter W F Wilson, consulting for and research funded by Merck.
Of the 10 expert reviewers for this panel, 3 had financial relationships with pharmaceutical companies that manufacture statins
- William Virgin Brown, consultant for Abbott, Bristol-Myers Squibb, and Pfizer;
- Matthew Ito, consultant for Kowa;
- Robert S Rosenson, consultant for Novartis and Pfizer.
Of the 11 people on the risk prediction panel who were not NHLBI staffers serving ex-officio, 5 had financial relationships with pharmaceutical companies that manufacture statins
-David C Goff Jr, co-chair, research funded by Merck;
- Raymond Gibbons, consultant for AstraZeneca;
- Jennifer Robinson, research funded by AstraZeneca, and Merck;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer [although these relationships were not listed as relevant to this panel, but found in the listing for the panel above];
- Peter W.F Wilson, consultant for and research funded by Merck.
Also, in the Abramson and Redberg NY Times op-ed, the authors noted
both the American Heart Association and the American College of Cardiology, while nonprofit entities, are heavily supported by drug companies
As noted on Pharmalot, the prevalence of conflicted panel members did not appear to conform to the standards for the development of trustworthy guidelines recently published by the Institute of Medicine:
whenever possible, guideline development group members should not have conflicts of interest… and the chair or co-chairs should not be a person(s) with conflicts of interest.
Also, noted by the Los Angeles Times was this comment from Dr John Abramson, lead author of the commentary on statins in primary prevention(4),
'There is overtreatment that’s been built into the risk calculator, and this is a warning sign about the overtreatment that’s built into the guidelines themselves and the conflicts of interest in the organizations that are overseeing the production of these guidelines,' said Dr. John Abramson, a Harvard University cardiologist who has argued that statins offer little value for people with a 10-year risk level of heart attack or stroke of less than 20%. 'There aren’t brakes being put on the enthusiasm and overreaching of the experts.'
'There are statin believers, and when you hear these experts talk, they’re talking emotionally, not scientifically,' Abramson added. 'The experts are using emotion, not science.'
As Joe Collier observed, "people who have conflicts of interest often find giving clear advice (or opinions) particularly difficult."(5)
This difficulty giving clear advice, when amplified by a guideline for a common problem supported by prestigious non-profit organizations, and promoted by vigorous public relations, could lead to "more than 45 million middle-aged Americans who do not have cardiovascular disease being recommended for consideration of statin therapy" (per Ridker and Cook[4]) unnecessarily, likely resulting in millions suffering unneeded side effects, and billions in costs.
Summary
Guidelines for management of a very common problem promulgated by a major medical society and a major disease oriented non-profit organization suggested a strategy that would vastly increase drug treatment of currently healthy patients. The strategy appears not to have been based on good evidence. When some of the problems with this evidence were pointed out, the guideline developers responded with illogic. Apparently many of the guideline developers have financial relationships with the drug companies that would most profit from increases in drug treatment as recommended by the guidelines Implementation of the new guidelines might results in millions of people in the US receiving unneeded drugs, with resultant side effects and costs. .
Do we need more examples of how conflicts of interest are causing the poor outcomes and excess costs that are wrecking our health care system? Do we need more excuses not to eliminate conflicts of interest from guideline development? Do we need more delay implementing the standards provided by the Institute of Medicine report on trustworthy guidelines? Do we need more excuses not to drastically reduce conflicts of interest affecting academic medicine, medical societies, and disease specific non-profits, specifically starting with the earlier (and so far generally disregarded) Institute of Medicine report on conflicts of interest in medicine?
While we in the US argue incessantly in the details of minor reforms of our supposed free health care market, we ignore the rot at its foundations. True health care reform would attack the conflicts of interest that have put money, not patients at the center of health care.
References
1. Stone NJ, Robinson J, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2013. Link here.
2. Abramson JD, Rosenberg HG, Jewell N et al. Should statins be prescribed to people at low risk of cardiovascular disease. Brit Med J 2013; 347: 15-17. Link here.
3. Goff Jr DC, Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA Guideline on the Assessment of Cardiovascular
Risk. Journal of the American College of Cardiology (2013), doi: 10.1016/j.jacc.2013.11.005. Link here.
4. Rikder PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013; Link here.
5. Collier J. The price of independence. Br Med J 2006; 332: 1447-9. Link here
Confused Thinking about New Cholesterol Guidelines - Were Conflicts of Interest to Blame?
Reviewed by MCH
on
November 22, 2013
Rating:
No comments: