How to Understand Scientific Studies

Posted on by

From John Pohl’s Twitter Feed:

Twenty tips for interpreting scientific claims http://bit.ly/1hY3nD5. Referenced article: Nature 503, 335–337 (21 November 2013) doi:10.1038/503335a

An excerpt:

we suggest 20 concepts that should be part of the education of civil servants, politicians, policy advisers and journalists — and anyone else who may have to interact with science or scientists. Politicians with a healthy scepticism of scientific advocates might simply prefer to arm themselves with this critical set of knowledge…

Differences and chance cause variation…

No measurement is exact. Practically all measurements have some error…Results should be presented with a precision that is appropriate for the associated error, to avoid implying an unjustified degree of accuracy…

Bias is rife. Experimental design or measuring devices may produce atypical results in a given direction….

Bigger is usually better for sample size…

Correlation does not imply causation. It is tempting to assume that one pattern causes another. However, the correlation might be coincidental, or it might be a result of both patterns being caused by a third factor — a ‘confounding’ or ‘lurking’ variable…

Regression to the mean can mislead. Extreme patterns in data are likely to be, at least in part, anomalies attributable to chance or error…

Extrapolating beyond the data is risky. Patterns found within a given range do not necessarily apply outside that range…

Beware the base-rate fallacy. The ability of an imperfect test to identify a condition depends upon the likelihood of that condition occurring (the base rate). For example, a person might have a blood test that is ‘99% accurate’ for a rare disease and test positive, yet they might be unlikely to have the disease. If 10,001 people have the test, of whom just one has the disease, that person will almost certainly have a positive test, but so too will a further 100 people (1%) even though they do not have the disease.

Controls are important. A control group is dealt with in exactly the same way as the experimental group, except that the treatment is not applied. Without a control, it is difficult to determine whether a given treatment really had an effect…

Randomization avoids bias. Experiments should, wherever possible, allocate individuals or groups to interventions randomly…

Seek replication, not pseudoreplication. Results consistent across many studies, replicated on independent populations, are more likely to be solid…

Scientists are human. Scientists have a vested interest in promoting their work, often for status and further research funding, although sometimes for direct financial gain. This can lead to selective reporting of results and occasionally, exaggeration. Peer review is not infallible: journal editors might favour positive findings and newsworthiness. Multiple, independent sources of evidence and replication are much more convincing.

Significance is significant. Expressed as P, statistical significance is a measure of how likely a result is to occur by chance. Thus P = 0.01 means there is a 1-in-100 probability that what looks like an effect of the treatment could have occurred randomly, and in truth there was no effect at all. Typically, scientists report results as significant when the P-value of the test is less than 0.05 (1 in 20).

Separate no effect from non-significance. The lack of a statistically significant result (say a P-value > 0.05) does not mean that there was no underlying effect: it means that no effect was detected. A small study may not have the power to detect a real difference…

Effect size matters. Small responses are less likely to be detected…

Study relevance limits generalizations. The relevance of a study depends on how much the conditions under which it is done resemble the conditions of the issue under consideration…

Feelings influence risk perception. Broadly, risk can be thought of as the likelihood of an event occurring in some time frame, multiplied by the consequences should the event occur…

Dependencies change the risks. It is possible to calculate the consequences of individual events, such as an extreme tide, heavy rainfall and key workers being absent. However, if the events are interrelated, (for example a storm causes a high tide, or heavy rain prevents workers from accessing the site) then the probability of their co-occurrence is much higher than might be expected…

Data can be dredged or cherry picked. Evidence can be arranged to support one point of view…

Extreme measurements may mislead…

Comment: This is a really good reference to provide context for understanding scientific studies and sources of error.

Crisis or Not -Tough to Change Bad Habits

Posted on by

For those of you with new resolutions, a sobering study from JAMA shows how difficult it is to change bad habits, even after a heart attack or a stroke.

From Sanjay Gupta’s twitter feed, link and excerpt: Video (1:48) http://ow.ly/sdS7R 

Even a brush with death is often not enough to get us to make better choices. Researchers studied more than 150,000 people from all around the world, rich and poor, urban and rural. Participants answered questions about exercise, diet, and smoking.

Because the group was so large, there were almost 8,000 participants who had survived either a heart attack or a stroke. The health habits of this group were startling. Only 39 percent reported improving their diet, and just 35 percent increased their physical activity. Of those who were smokers, only 52 percent quit. Just 4 percent of those 8,000 people improved their habits in all three areas: smoking, diet, and exercise.

Moving to All Oral Therapy for Hepatitis C

Posted on by

Two more studies show the promise of all oral treatment for Hepatitis C virus:

  • NEJM 2014; 370: 211-21.
  • NEJM 2014; 370: 222-32.

A summary of these articles is available at the following link: http://t.co/Z8jMPKoLGz.

Here is an excerpt:

Hepatitis C treatment isn’t pretty, but the dark days of weekly injections, rough side effects and no guarantee of full recovery from the liver-damaging disease may soon be over, researchers report.

Two studies, both published in the Jan. 16 issue of the New England Journal of Medicine, involved giving various combinations of antiviral pill cocktails to patients with hepatitis C. Some had failed to respond to standard treatments, and some had not received treatment yet. Yet, the cocktails cleared the virus in both studies for between 93 percent and 98 percent of the patients…

The first study, conducted by Johns Hopkins researchers, included 211 men and women with hepatitis C who took two pill-form antiviral medications, daclatasvir and sofosbuvir. The patients were treated at 18 medical centers in the United States and Puerto Rico. They took 60 milligrams of daclatasvir and 400 milligrams of sofosbuvir for either 12 or 24 weeks, with or without a third drug, ribavirin….

98 percent of the 126 previously untreated patients and 98 percent of 41 patients whose infections had not cleared despite treatment with standard hepatitis C therapy, were considered cured. “There was no detectable virus in their blood three months after the treatment stopped,” he noted.

The second study, headed up by researchers at Virginia Mason Medical Center in Seattle, involved more than eight medical centers in the United States and internationally. It included 571 patients with hepatitis C, some of whom had not received treatment previously and others who had previously received standard treatments with interferon injections and ribavirin — an antiviral drug that when given reduces relapses — but had not responded to them.

The participants were randomly assigned to take any of three combinations of antiviral pills — medications called ABT-450, ABT-267, and ABT-333 — for eight, 12 or 24 weeks…

Almost all of the patients (more than 93 percent in both groups) saw the virus cleared from their systems within 24 weeks.

Bottomline: Once daily treatment with a combination of medicines will be an effective and safe cure for more than 90% of individuals with HCV. Whether these agents will be affordable remains in doubt.

Related blog posts:

 

Science That Matters -Especially for Teenage Drivers

Posted on by

Animated overview of distracted driving study narrated by Editor-in-Chief Jeffrey Drazen. http://nej.md/1hVzO1t (2:43 video from NEJM)

More explanation of this study from NY Times: http://nyti.ms/1dX4lcp 

“An inexperienced driver who reaches for a cellphone increases the risk for a crash by more than 700 percent, a new study found.”

Micronutrient Monitoring in Intestinal Failure

Posted on by

J Pediatr 2013; 163: 1692-6.  This retrospective study of prospectively collected data from 178 children provides data with regard to micronutrient deficiency among intestinal failure patients transitioning to enteral feeds. Figures 1 and 2 along with Table 2 provide the prevalence of micronutrient deficiency while receiving supplemental parenteral nutrition (PN) and while on full enteral nutrition (FEN).  Iron deficiency was most common in both situations with prevalence of 84% and 61% respectively. With the exception of folate (0%), all of the vitamins and micronutrients had fairly high rates of deficiency.  While on FEN,  deficiencies were  the following:

  • Vitamin A        19%
  • Vitamin B12    6.5%
  • Vitamin D        30%
  • Vitamin E          6%
  • Copper            8%
  • Iron                61%
  • Selenium         4%
  • Zinc               23%

The study does not indicate that the deficiency values were adjusted based on CRP values.  Instead, “low serum levels were used to define deficiencies.”  This is likely to lead to numerous errors.  Nevertheless, it is clear that these deficiencies are common.  Another finding of the study was that normal anthropometrics did not reduce the frequency of these deficiencies.  In their patient population, 57 of 136 (42%) with sufficient height and weight data had a height-for-age z-scores of <-2 by the time of FEN; where as 52 of 139 patients (37%) had weight-for-age z-scores of <-2.

A recent post on The Pediatric Nutritionist blog provides a suggested approach to the monitoring of vitamins and micronutrients based on the need for parenteral nutrition and on the need to consider inflammatory markers in the interpretation of these lab values: The Importance of Nutrition Lab Monitoring Protocols Featuring …

Bottomline: Vitamin and micronutrient deficiencies are common among intestinal failure patients.  In addition, a large percentage of these kids are not large at all.

Related blog post:

What happens to micronutrient levels in the hospital setting …

22Q on Video

Posted on by

A couple recent youtube videos on 22Q deletion provide helpful information:

1. An Introduction to 22q112 Deletion Syndrome – YouTube –~20 minute video which provides an overview of the various problems and presentations of patients with 22Q deletion.

2. http://www.youtube.com/watch?v=34YhIYhNrdo -~ 5 minute video from the Netherlands conveys the message that kids with 22Q wanted to be treated like everyone else.

Related blog post:

Immunoglobulin deficiencies with DiGeorge Syndrome …  This post lists a number of references and resources related to 22Q (aka. DiGeorge Syndrome or Velocardiac Facial Syndrome).

Sanjay Gupta is Wrong… about Stem Cell Therapy

Posted on by

According to a 3 min video (and article) publicized on twitter by Dr. Sanjay Gupta, Stem Cell therapy for Crohn disease is 97% effective. ow.ly/smrJM I sent him a tweet asking for data to support this figure but have not heard back.  That being said, there are very few treatments that work in 97% of patients with any chronic disease.

The context of the video regards a model who has had 75 hospitalizations for Crohn disease and is unable to tolerate standard treatment.  “Thus far, there is a 97 percent success rate with this procedure, but it’s not fully covered by insurance, so Jocelyn must find the money for the procedure, her travel, and the long recovery.”

To my knowledge, stem cell therapy, while promising, for Crohn disease remains an experimental treatment without any large studies proving its effectiveness. 

An abstract at DDW last year (Stem Cell Transplantation Halts Crohn’s Disease – Medscape) reported that among the 22 patients in the stem cell treatment group (who were refractory to multiple other medications), 40% had mucosal healing and 58% had segmental healing. The presenting physician, Dr. Christopher Hawkey, noted ‘there were serious adverse events and many patients were not cured…. We need controlled trials showing a long-term risk/benefit ratio.’

Another study (Blood. 2010;116:6123-6132) with 24 patients, reported that “the percentage of clinical relapse-free survival defined as the percent free of restarting CD medical therapy after transplantation is 91% at 1 year, 63% at 2 years, 57% at 3 years, 39% at 4 years, and 19% at 5 years.”

Bottomline: I think the information in the video is not accurate.  Inevitably, it will lead to a lot of ill-informed questions by families.  When a respected physician posts this type of unsupported information, it has the potential to undermine not just his credibility but other physicians as well. Perhaps, Dr. Gupta will consider revising this information.

Related article:

Clin Gastroenterol Hepatol 2014; 12: 64-71: “A phase 2 study of allogeneic mesenchymal stromal cells for luminal Crohn’s disease refractory to biologic therapy.” Results: among the 15 patients (of 16) who completed the study, the mean CDAI score was reduced from 370 to 203.  Twelve patients had a clinical response and eight had clinical remission.  All patients received 4 weekly infusions of mesenchymal stromal cells.  One patient had a stage 1 adenocarcinoma (colon) but the authors think that this was likely present prior to the infusions. Why this study is important? If shown effective in larger studies, mesenchymal stromal cells  are much safer than allogeneic stem cells as donor to recipient matching is not needed nor chemotherapeutic marrow conditioning.

Timing of Solid-Food Introduction

Posted on by

The “DAISY” (diabetes autoimmunity  study in the young) study indicates that the timing of solid-food introduction can influence the likelihood of developing type 1 diabetes (T1DM) (JAMA Pediatr 2013; 167: 808-15).

The participants were 1853 children at increased genetic risk for T1DM who were enrolled in a longitudinal observational cohort study in Denver. Early solid-food exposure was considered <4 months of age and late >6 months of age.

Results:

  • “Both early and late first exposure to any solid food predicted development of T1DM.”  For early exposure, the Hazard Ratio was 1.91 and for late HR was 3.02.
  • Breastfeeding at the time of introduction to wheat/barley conferred protection (HR 0.47)

The study has several limitations, particularly the relatively low numbers of children who developed T1DM (n=53).

A second study (Pediatrics 2013 [doi: 10.1542/peds2012-3692]) –thanks to Ben Gold for this reference –showed that “solid foods were introduced significantly earlier among the infants with allergies, with 35% of them receiving their first solids before and including 16 weeks, compared with 14% of control infants (P=.011).”   (Solid foods before 17 weeks linked to food allergy)

Bottomline: As with celiac disease (“Gluten–Related Disorders” (Part 1) | gutsandgrowth), current science suggests the introduction of solid foods between 4-6 months of age may diminish the risk of developing T1DM as well as food allergies.

 

Why I have always liked Arthur Caplan…

Posted on by

I have been a fan of Arthur Caplan for a long time.  In medical school, I had the good fortune to study closely with several brilliant bioethicists including Baruch Brody and Tristram Engelhardt.  Since that time, I have remained interested in bioethics.  Arthur Caplan has been a leading voice in bioethics for a long time and often approaches topics in a no-nonsense manner.

A recent link (from John Pohl’s twitter feed) chronicle.com/blogs/conversation/2014/01/07/distinguishing-science-from-nonsense/ … and excerpt from Arthur Caplan addresses the need for a better appreciation of science; this short opinion piece ties together brain death, vaccines, evolution, and nutritional supplements:

A key reason for the poor performance of our children with respect to science is that American culture is both ignorant of and disrespectful to science.

As I write this, two women in ICUs in the United States are on life support despite having been pronounced dead by medical experts. These women, a teenager in Oakland, Calif., and a young woman in Fort Worth carrying a 14-week-old fetus when she died, were found to be dead on the basis of brain death. Both had their bodies maintained by machines (in Oakland it was with the support of her family; in Texas it was against family wishes). Neither the news media nor the medical profession seemed to be able to explain that brain death is truly death. Nor did the public seem inclined to listen, believing that somehow a miracle might occur.

At the same time as those cases emerged, a poll was released by the Pew Research Center showing that a third of Americans do not believe in evolution. They think that “humans and other living things have existed in their present form since the beginning of time.” Twenty-four percent acknowledge evolution but believe that a Supreme Being has directly guided life on earth.

And as I write this, flu season has begun. The federal Centers for Disease Control and Prevention estimates that last year 381,000 Americans were hospitalized because of the flu. They also estimate that the flu vaccine prevented 79,000 hospitalizations and 6.6 million illnesses. Yet a tiny cabal of kooks and know-nothings has gotten so much attention that barely half of all Americans get a flu shot.

The problem does not end there. The multibillion-dollar nutritional-supplements industry has no solid evidence for the efficacy of its products, while there are plenty of instances in which death and disability have been linked to poorly manufactured or mislabeled supplements. Yet our airwaves are full of ads and endorsements for this cornucopia of malarkey.

The point ought to be clear. Children are not going to flourish at science in a society that treats science either as something you can believe in selectively, something that is simply one point of view, or something about which anyone can have a credible opinion no matter how ill-qualified, dumb, or misinformed.

If we want to have a brighter economic future, then we need to start thinking about science education outside of our schools. We need editors who refuse to put fringe points of view on the air. We need scientists who see it as their duty to engage broader audiences—not just their peers—about their work. We need the training of scholars in the public understanding of science so that more voices are heard respecting science and the scientific method.

We need our courts to better vet who can speak for science. And we need more scientists as role models rather than the athletes and entertainers so put before the eyes of kids who may find it a bit hard to take chemistry, ecology, epidemiology, statistics, and geology seriously when their home life is filled with the musings of the casts of Duck Dynasty, Here Comes Honey Boo Boo, and Long Island Medium.

And we wonder why Johnny and Jane can’t distinguish science from nonsense.

Arthur L. Caplan is a professor of bioethics and director of the division of medical ethics at NYU Langone Medical Center.

What you might not know about anti-TNF monitoring…

Posted on by

At a recent group dinner meeting, we had the opportunity to review therapeutic anti-TNF monitoring. In addition, we discussed emerging treatments for inflammatory bowel disease, like golimumab, tofacintinib and vedolizumab.

As noted in previous blog entries (see below), therapeutic anti-TNF monitoring can help adjust treatment.  Namely, if a patient loses response to therapy and has low trough levels of anti-TNF (Infliximab ❤ μg/mL, Adalimumab <8 μg/mL, or certolizumab <27.5 μg/mL) without antidrug antibodies (ADAs), then increasing the dose is likely to be effective.  However, if a patient has a therapeutic level and is not responding, changing to another agent and/or further investigation is worthwhile.

So, what information is new?

  • Only about 20% of patients who lose clinical response develop ADAs.  So, drug level, rather than ADAs, is most helpful.
  • For infliximab, adjusting dose 14 weeks into therapy to achieve a target trough level between 3-7 mcg/mL may be helpful.
  • Severe colitis patients may need higher initial doses (?as high as 20 mg/kg) due to potential for ‘antigen sink.’  This is due to notably higher clearance in the presence of low albumin, and high CRP.  Other factors that increase clearance include higher BMI and male gender.
  • About 1/2 of patients who receive higher doses due to severe disease may be able to deescalate dosage when improved. (?which half)
  • Currently, a reactive approach to checking levels is common in U.S. in part due to costs associated with checking trough levels and ADAs (as much as $2500).  That is, most commonly checking levels is undertaken in patients with suboptimal clinical response.  A proactive approach to achieve target levels may be shown to be helpful.
  • While studies have not shown higher adverse reactions with higher trough levels, there are a few clinical situations in which lower trough levels can be important.  In patients with psoriatic skin lesions and arthralgias, if trough levels are elevated, lowering the dose may be helpful.

Outstanding questions?

  • Should patients have drug levels checked when they are asymptomatic?
  • How does a practitioner account for variability among different laboratory assays?
  • What is the optimal target level for each anti-TNF agent? Is this different in Crohn disease compared with ulcerative colitis? Is the trough target level different in adults than children?
  • Is there a toxic level?
  • If a rapid test response were available, would checking drug levels be needed for hospitalized patients to assess anti-TNF rescue therapy?

Related blog links:

Disclaimer: These blog posts are for educational purposes only. Specific dosing of medications (along with potential adverse effects) should be confirmed by prescribing physician.  This content is not a substitute for medical advice, diagnosis or treatment provided by a qualified healthcare provider. Always seek the advice of your physician or other qualified health provider with any questions you may have regarding a condition