Checklists -Helpful? Overhyped? Hawthorne Effect?

Over the last few years, adoption of surgical safety checklists has taken hold due to the promise of improving outcomes with a simple intervention.  While the concept of checklists is sound, new data indicate that in practical experience these checklists may not result in any significant reductions in meaningful outcomes (NEJM 2014; 370: 1029-38).

The study of surgical safety checklists in Ontario (13 million people) examined the 3-month periods before and after adoption of a surgical safety checklist (around July 2010) at a total of 101 hospitals.  In the period prior to adoption, there were 109,341 procedures and afterwards 106,370.

Key findings:

  • Adjusted risk of death during a hospital stay or within 30 days of surgery was 0.71 prior and 0.65 afterwards.  This produced an odds ratio of 0.91 with 95% confidence limits of 0.80 to 1.03.
  • Adjusted risk of surgical complications was 3.86% and 3.82% respectively, yielding an odds ratio of 0.97 with 95% confidence limits of 0.90 to 1.03.
  • Checklists did not reduce emergency room visits or hospital readmissions within 30 days after discharge.

These findings contradict previous WHO estimates that at least 500,000 deaths per year could be prevented through worldwide implementation of checklists.  A previous meta-analysis of three other before-and-after checklist studies determined that the checklists were associated with a pooled relative risk of operative death of 0.57 (confidence intervals 0.42 to 0.76). Yet, this Ontario study had a similar implementation without this reduction.

So, how can this study show no significant reduction in operative mortality or complications?

Possible explanations:

  • Hawthorne effect could explain some of the previous results.  This effect refers to “the tendency for some people to perform better when they perceive that their work is under scrutiny.”
  • Publication bias.  Because these checklists are utilized in thousands of hospitals, “many will have improvements in the outcomes by chance alone.”

The authors note that there has never been a controlled trial with randomization to determine conclusively the effectiveness of checklists.  However, the current study is less susceptible to biases than single center studies and no other confounding variables were identified.

Take-home message: In this study, checklists did not result any striking improvements.  Nevertheless, “there may be value in the use of surgical safety checklists, such as enhanced communication and teamwork and the promotion of a hospital culture in which safety is a high priority.”

On a side note, this study reinforced my view that many quality initiatives are well-meaning but sometimes overhyped with regard to their effectiveness.

Related blog posts:

3 thoughts on “Checklists -Helpful? Overhyped? Hawthorne Effect?

  1. Interesting. I would like to see the effectiveness compared between hospitals who already had reasonably low complication rates and ones who had more mistakes/complications/readmissions. This is pure speculation, but in WHO terms, checklists may very well help in second world and third world settings, and in North American hospitals that are lagging behind their peers.

    Increase study power to infinity and one will find a difference between groups. So often such a difference is clinically irrelevant, but in terms of unnecessary death from surgery, isn’t any gain from a checklist beneficial? I think we should be careful poo pooing an intervention that is effectively free, takes a few seconds, has no side effects, and that on the scale of millions of cases, could save one preventable surgical morbidity or mortality.

    • Thanks for your comments. To be clear, I’m not recommending getting rid of checklists. I think their effectiveness has been overstated. See tomorrow’s post for more on this subject.

  2. Pingback: “The future of gastrointestinal disease and symptom monitoring: biosensor, E-portal, and social media” | gutsandgrowth

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.