Over the last few years, adoption of surgical safety checklists has taken hold due to the promise of improving outcomes with a simple intervention. While the concept of checklists is sound, new data indicate that in practical experience these checklists may not result in any significant reductions in meaningful outcomes (NEJM 2014; 370: 1029-38).
The study of surgical safety checklists in Ontario (13 million people) examined the 3-month periods before and after adoption of a surgical safety checklist (around July 2010) at a total of 101 hospitals. In the period prior to adoption, there were 109,341 procedures and afterwards 106,370.
- Adjusted risk of death during a hospital stay or within 30 days of surgery was 0.71 prior and 0.65 afterwards. This produced an odds ratio of 0.91 with 95% confidence limits of 0.80 to 1.03.
- Adjusted risk of surgical complications was 3.86% and 3.82% respectively, yielding an odds ratio of 0.97 with 95% confidence limits of 0.90 to 1.03.
- Checklists did not reduce emergency room visits or hospital readmissions within 30 days after discharge.
These findings contradict previous WHO estimates that at least 500,000 deaths per year could be prevented through worldwide implementation of checklists. A previous meta-analysis of three other before-and-after checklist studies determined that the checklists were associated with a pooled relative risk of operative death of 0.57 (confidence intervals 0.42 to 0.76). Yet, this Ontario study had a similar implementation without this reduction.
So, how can this study show no significant reduction in operative mortality or complications?
- Hawthorne effect could explain some of the previous results. This effect refers to “the tendency for some people to perform better when they perceive that their work is under scrutiny.”
- Publication bias. Because these checklists are utilized in thousands of hospitals, “many will have improvements in the outcomes by chance alone.”
The authors note that there has never been a controlled trial with randomization to determine conclusively the effectiveness of checklists. However, the current study is less susceptible to biases than single center studies and no other confounding variables were identified.
Take-home message: In this study, checklists did not result any striking improvements. Nevertheless, “there may be value in the use of surgical safety checklists, such as enhanced communication and teamwork and the promotion of a hospital culture in which safety is a high priority.”
On a side note, this study reinforced my view that many quality initiatives are well-meaning but sometimes overhyped with regard to their effectiveness.
Related blog posts: