Development and Validation of Electronic Health Record Measures of Safety Planning Practices As Part of Zero Suicide Implementation

ObjectiveSafety planning for suicide prevention is an important quality metric for Zero Suicide implementation. We describe the development, validation, and application of electronic health record (EHR) programs to measure uptake of safety planning practices across six integrated healthcare systems as part of a Zero Suicide evaluation study.MethodsSafety planning was documented in narrative notes and structured EHR templates using the Stanley Brown Safety Planning Intervention (SBSPI) in response to a high-risk cutoff score on the Columbia Suicide Severity Rating Scale (CSSRS). Natural Language Processing (NLP) metrics were developed and validated using chart review to characterize practices documented in narrative notes. We applied NLP to measure frequency of documentation in the narrative text and standard programming methods to examine structured SBSPI templates from 2010-2022.ResultsChart reviews found three safety planning practices documented in narrative notes that were delivered to at least half of patients at risk: professional contacts, lethal means counseling for firearms, and lethal means counseling for medication access/storage. NLP methods were developed to identify these practices in clinical text with high levels of accuracy (Sensitivity, Specificity, & PPV >= 82%). Among visits with a high-risk CSSRS, 40% (Range 2-73% by health system) had an SBSPI template within 1 year of implementation.ConclusionsThis is one of the first reports describing development of measures that leverage electronic health records to track use of suicide prevention safety plans. There are opportunities to use the methods developed here in future evaluations of safety planning. Measuring safety planning delivery in real-world systems to understand quality of suicide prevention care is challenging.Natural Language Processing (NLP) methods effectively identified some safety planning practices in electronic health records (EHR) from all notes ensuring a comprehensive measurement, but NLP will require updates/testing for local documentation practices.Structured safety planning templates in the EHR using the Stanley Brown Safety Planning Intervention improve ease and accuracy of measurement but may be less comprehensive than NLP for capturing all instances of safety planning documentation.