August 23, 2011 — Analysis of electronic medical records (EMRs) with natural language processing shows an improved ability to identify postoperative surgical complications compared with the standard method of relying on administrative data codes, according to a new study published in the August 24/31 issue of JAMA.
In efforts to improve patient safety, hospital administrative data are typically screened for codes that may reflect potential adverse events during hospitalization, and a quality surveillance tool developed by the Agency for Healthcare Research and Quality has refined that process to focus on a set of 20 patient safety indicators used in screening the data.
However, the system has some drawbacks, including some uncertainty about the validity of administrative codes and the inability of discharge codes to distinguish whether a disease existed before a patient's admission or was acquired during hospitalization, according to Harvey J. Murff, MD, MPH, lead author of the study from Tennessee Valley Healthcare System, Veterans Affairs Medical Center, and Vanderbilt University, Nashville, TN, and colleagues.
The emergence of EMRs, combined with the development of automated systems such as natural language processing, however, allows for screening of more extensive medical data and documents and extraction of specific medical concepts, as opposed to simply searching for potentially unreliable discharge codes.
In an effort to compare the 2 approaches, the researchers evaluated data on 2974 patients undergoing inpatient surgical procedures at 6 Veterans Health Administration medical centers from 1999 to 2006.
During this period, percentages of patients with postoperative acute renal failure requiring dialysis was 2% (39/1924 patients); pulmonary embolism, 0.7% (18/2327 patients); deep vein thrombosis, 1% (29/2327 patients); sepsis, 7% (61/866 patients); pneumonia, 16% (222/1405 patients); and myocardial infarction, 2% (35/1822 patients).
Natural language processing was able to correctly identify 82% (95% confidence interval [CI], 67% - 91%) of acute renal failure cases, whereas screening using patient safety indicators only correctly identified 38% (95% CI, 25% - 54%).
The results were also more accurate for natural language processing compared with patient safety indicators for venous thromboembolism (59% [95% CI, 44% - 72%] vs 46% [95% CI, 32% - 60%]), pneumonia (64% [95% CI, 58% - 70%] vs 5% [95% CI, 3% - 9%]), and sepsis (89% [95% CI, 78% - 94%] vs 34% [95% CI, 24% - 47%]). Comparison of the 2 methods for postoperative myocardial infarction, however, showed similar results (91% [95% CI, 78% - 97%] vs 89% [95% CI, 74% - 96%]).
Both approaches were highly specific for the diagnoses.
"In general, using a natural language processing–based approach had higher sensitivities and lower specificities than did the patient safety indicator," the authors write.
"The increase in sensitivity of the natural language processing–based approach compared with the patient safety indicator was more than 2-fold for acute renal failure and sepsis and over 12-fold for pneumonia. Specificities were 4% to 7% higher with the patient safety indicator method than the natural language processing approach."
The authors noted that a greater ability to refine and vary search strategies allowed for greater sensitivities with natural language processing, with only a small reduction in specificities in all areas except postoperative myocardial infarction.
"In contrast to the patient safety indicator approach, for which test characteristics are fixed, the natural language processing approach offered a wide array of search strategies with varying test characteristics," they add.
"Nevertheless in some cases, specifically postoperative myocardial infarction, the patient safety indicator algorithm had excellent test characteristics that were not improved through the natural language processing approach."
The study's limitations include that patient safety indicators were not originally designed for Veterans Health Administration data; however, the researchers noted that the patient safety indicator rates appeared similar between the Veterans Health Administration and non–Veterans Health Administration populations. In addition, institutions that have not adopted EMRs yet would obviously not benefit from the approach.
The results suggest, however, that natural language processing systems should be considered as EMRs become more widely implemented and evolve.
"As additional institutions develop fully integrated EMR, electronic chart reviews for quality purposes should be further developed and evaluated," the authors state in their conclusion.
In an accompanying editorial, Ashish K Jha, MD, MPH, from the Department of Health Policy and Management, Harvard School of Public Health, Division of General Medicine, Brigham and Women's Hospital, and the VA Boston Healthcare System, Boston, Massachusetts, states that "[d]espite the promise of [EHRs], recent data on their benefits have been disappointing." The data to date, Dr. Jha explains, have shown that EHRs can be a useful tool to help clinicians adhere to guideline-based care and to reduce medication errors. "[B]eyond these narrow benefits, there is little evidence that EHRs improve patient outcomes and even less evidence that they improve the efficiency of care," he states.
However, "in this sea of disappointing data about EHRs comes some good news."
Dr. Murff and colleagues "push beyond the traditional uses of the EHR by demonstrating that natural language processing, when applied to electronic data, can help clinicians track adverse events after surgery," writes Dr .Jha. Although this benefit may seem "esoteric," its value should not be "underestimated," according to the editorialist. "Their value as quality measurement tools will improve substantially when EHRs can automatically generate quality measures that account for the reasons guideline-driven care is adhered to or, if not, why not.
"Currently, the EHR remains a tool with vast potential but a limited set of current capabilities. Natural language process has the potential for many new applications such as automated quality assessment to assisting in the performance of comparative effectiveness research," concludes Dr. Jha. "The study by Murff et al suggests that these benefits may be closer than ever, but only if the power of computing is harnessed to understand the vast amount of written data that currently needs a pair of eyes and a human brain to comprehend." He emphasizes, however, that federal funding is needed to propel research in this area forward.
The study was supported by a grant from the Department of Veterans Affairs. One author is supported by the Veterans Health Administration Career Development Award, and 2 authors by Veterans Health Consortium for Health Informatics Research awards. Dr. Jha reports that he serves on the scientific advisory board of Humedica Inc, which pools clinical data to provide clinical intelligence to physicians and hospitals.
No comments:
Post a Comment