Reporting bias: biomarkers

DefinitionBiomarkers or biological markers are typically as a surrogate for a disease process; for example the number of active lesions on MRI as a marker of disease activity or neurofilament levels in the spinal fluid as a marker of nerve damage.

Background: Many biomarkers are proposed in highly cited studies as determinants of disease risk, prognosis, or response to treatment, but few eventually transform clinical practice. The field of MS is no exception.

Objective: This study examined whether the magnitude of the effect sizes of biomarkers proposed in highly cited studies is accurate or overestimated.
Methods: The researchers included biomarker studies that had a relative risk presented in their abstract and had received more than 400 citations (referenced in other articles) using the ISI Web of Science database and that had been published in any of 24 highly top biomedical journals.

Results: 86% of the most highly cited studies had a stronger effect estimate than the largest study. For 83% of the most highly cited studies, the corresponding meta-analysis (combining all the results together) found a smaller effect estimate. 

Conclusions:  Highly cited biomarker studies often report larger effect estimates for postulated associations than are reported in subsequent meta-analyses evaluating the same associations.

Ioannidis and Panagiotou. Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA. 2011 Jun 1;305(21):2200-10.

"This study illustrates that researchers are subject to an inherent or built in bias that favours positive studies. Why this occurs is an interesting social phenomenon. We prefer to focus on the positive rather than the negative. This is phenomenon that occurs in the lay press as well."

"This is why experienced researchers are usually very sceptical about positive studies; they need them to be confirmed and repeated many times by independent investigators. In fact most good research studies include both exploratory and validation cohorts within their study to design; by doing this the researchers try and validate their own data."

"This is why the regulatory agencies require two large pivotal phase 3 studies before the license a drug. By doing this they reduce the chance of reporting bias or a chance positive finding in the first study. The only exception to this is for orphan diseases (rare diseases) and when the unmet need is massive or life saving; the regulatory agencies are prepared to license the drug before the second study is completed."

"One of the biggest problems in science is reporting bias; i.e. journals tend to only accept and publish positive studies and research tend not to write up and submit negative studies for publication. This is the curse of the biomarker field and explains why so many initial positive studies fail to deliver on their promise."

Extra reading: biomarkers, meta-analysis, reporting bias