Automated suicide risk prediction models operating on electronic health records could help clinical teams efficiently identify patients for face-to-face suicide risk screening and prevention.
In a retrospective cohort study using electronic health records, a predictive model developed at Vanderbilt University Medical Center accurately predicted short-term risk of suicidal behaviors — that is, suicidal thoughts or attempts — among patients at a group of U.S. Navy primary care clinics.
Reported Nov. 8 in JAMA Network Open, the VUMC-led study indicates civilian suicide risk tools could potentially be implemented in military health systems, avoiding costly new development.
Colin Walsh, MD, MA, associate professor of Biomedical Informatics, Medicine and Psychiatry, application developer Michael Ripperger, and colleagues focused on records of 260,583 active-duty Navy service members who received care from 2007 to 2017 at clinics affiliated with Naval Medical Center Portsmouth in Virginia.
They applied a suicide risk model originally created using electronic health records from VUMC. The model was trained to predict suicide attempt within 30 days based on factors like past diagnoses, medications, demographics and health care utilization.
In gauging the VUMC model’s prospective clinical usefulness for the Navy, the team used several performance metrics, including one called the area under the receiver operating characteristic curve, or AUC, which can be interpreted as measuring how well a model ranks patients according to risk. Straight out of the box, the VUMC model achieved an AUC of 77% on Naval records, which the authors characterized as feasible accuracy for clinical deployment. (For comparison, the model’s enterprise-wide AUC at VUMC had been 84%.)
Next, the team retrained and recalibrated the VUMC model on Naval data and updated it with Naval-specific health variables and demographics, achieving an outstanding AUC of 92% on the Navy’s primary care records.
The study found that if Navy primary care teams were to limit face-to-face screening to the 10% of patients at greatest risk per the model, with the VUMC model they would stand to encounter one would-be case of 30-day suicidal behavior per every 366 patients screened, and with the retrained model, one per every 200 patients screened.
“For comparison, both these ‘numbers needed to screen’ are in the neighborhood of numbers needed in universal age- and gender-based screenings to prevent cases of common forms of cancer, such as colorectal cancer and breast cancer,” Walsh said.
Walsh said the findings suggest that, with some additional testing and design, externally developed suicide risk tools like the one from VUMC might help guide the Navy’s prevention efforts. “Adapting an external model may provide needed performance with less development time and cost,” he said.
Others on the study from VUMC include Jhansi Kolli, BS, Drew Wilimitis, BS, Katelyn Robinson, BA, Carrie Reale, MSN, RN, and Laurie Novak, PhD, MHSA. They were joined by researchers from the U.S. Navy and from Florida State University.
The study was funded by the Military Suicide Research Consortium and the Evelyn Selby Stead Fund for Innovation.