What can be done about bias in peer review?
by Kym Morrissey, BA, CPHQ, CNMT, RT(N), peer review coordinator at St. Anthony Hospital in Lakewood, Colorado
A number of articles have been written about bias in peer review?what it is, how it affects the overall peer review process, and types of bias, to name a few. Bias is understandably the stumbling block to effective peer review. It is the one factor that can take a well-meaning committee that is truly focused on improvement and make it appear as if it is practicing sham peer review.
At St. Anthony Hospital, our professional review committee is a multidisciplinary committee that represents the most active specialties of the medical staff; it also has representation from internal medicine and primary care. Over the years, changes to committee scoring have been implemented to help score more fairly and with less bias.
To assess whether these changes have made an impact, we conduct a biannual survey to assess the perception of the peer review committee members. This has been done since 2009. For the past seven years, we have asked the same questions to allow for comparison across time as new members join, old members step down, and efforts toward improvement are implemented. Two questions specifically designed to assess bias have consistently been included in the survey:
- Do you feel that the cases are reviewed in a fair and impartial manner by the committee members?
- Do you feel that the action taken at the meeting is appropriate?
The results of those two questions reflect improvements that have been made and the impact of those improvements on our survey results. (See the chart at bottom-left.)
In 2010, multi-level scoring was implemented but included patient outcome, which inherently biases the case review, particularly if the outcome isn’t good. In 2012, the committee moved to a multilevel scoring system where overall practitioner care, issue identification, and documentation comprised the final case assessment. The perception of bias is slowly improving with the change to the multi-level scoring.
In 2013, one of the committee members suggested blind voting to increase members’ ability to vote with their conscience without the pressure of a show of hands. Initially this was done with a voting sheet, and the scores were tallied and reported during the meeting, but this method proved too onerous. The committee then started utilizing an audience response system to allow the members to vote privately. The voting results are displayed immediately so that the members are aware of the case level assessment.
It has been interesting to watch the voting reflect the opinions of the members. Previously a show of hands would be unanimous; it would be difficult to say that members were voting according to their conscience. Group pressure would prevail, and hands would go up as members looked around the table. With an audience response system, the results are more telling?rarely is there a unanimous vote. A simple majority determines the level assigned. In the case of a tie, the committee may discuss a few of the salient points again and then revote the question. The voting results are displayed immediately so that the members are aware of the case level assessment.
In reviewing the survey, the 2014 results marked the first time a unanimous response was registered to the question of whether the actions taken by the committee were appropriate. From a low of 41% in 2011 to 100% in 2014, we may say that anonymous voting has given the committee the freedom to vote truthfully and the peace of mind that actions are appropriate. The verbatim comments from the most recent survey of peer review committee members bear this out:
- "The electronic voting has made final determinations more consistent and fair."
- "I feel like I can express my opinion without risk of comment during the meeting because of electronic voting."
- "Originally thought the voting took too long; now I appreciate the anonymity."
In summary, from much of the literature that exists on professional peer review, there is a general opinion that bias is one of its inherent enemies. Even small attempts to reduce bias can add value. Will we ever be able to overcome all bias? In all honesty, no, but we should not give up the battle to reduce it.