Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... 

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.

Enhancing Precision in Detecting Severe Immune-Related Adverse Events: Comparative Analysis of Large Language Models and International Classification of Disease Codes in Patient Records

Author(s): Virginia H. Sun, MD1,2; Julius C. Heemelaar, MD1,2,3; Ibrahim Hadzic, MSc1,4,5,6; Vineet K. Raghu, PhD1,2; Chia-Yun Wu, MD7,8; Leyre Zubiri, MD, PhD1,7; Azin Ghamari, MD1,2; Nicole R. LeBoeuf, MD, MPH1,9,10; Osama Abu-Shawer, MD, MS11; Kenneth L. Kehl, MD, MPH1,12,13; Shilpa Grover, MD, MPH1,14; Prabhsimranjot Singh, MD1,13; Giselle A. Suero-Abreu, MD, PhD, MSc1,2,15; Jessica Wu, BA1,2; Ayo S. Falade, MD, MBA, APGD16; Kelley Grealish, MSN, NP7; Molly F. Thomas, MD, PhD17,18,19; Nora Hathaway, MSN, NP7; Benjamin D. Medoff, MD1,20; Hannah K. Gilman, BS1,2; Alexandra-Chloe Villani, PhD1,21,22; Jor Sam Ho, MPH1,2; Meghan J. Mooradian, MD1,7; Meghan E. Sise, MD1,23; Daniel A. Zlotoff, MD, PhD1,15; Steven M. Blum, MD1,7,21,22; Michael Dougan, MD, PhD1,24; Ryan J. Sullivan, MD1,7; Tomas G. Neilan, MD, MPH1,2,15; Kerry L. Reynolds, MD1,7
Source: https://doi.org/10.1200/JCO.24.00326

Dr. Maen Hussein's Thoughts

Using AI to detect immunotherapy adverse events in hospital admissions was better than using ICD codes, another application for AI. We will see more AI studies in the next couple of years.

PURPOSE

Current approaches to accurately identify immune-related adverse events (irAEs) in large retrospective studies are limited. Large language models (LLMs) offer a potential solution to this challenge, given their high performance in natural language comprehension tasks. Therefore, we investigated the use of an LLM to identify irAEs among hospitalized patients, comparing its performance with manual adjudication and International Classification of Disease (ICD) codes.

METHODS

Hospital admissions of patients receiving immune checkpoint inhibitor (ICI) therapy at a single institution from February 5, 2011, to September 5, 2023, were individually reviewed and adjudicated for the presence of irAEs. ICD codes and an LLM with retrieval-augmented generation were applied to detect frequent irAEs (ICI-induced colitis, hepatitis, and pneumonitis) and the most fatal irAE (ICI-myocarditis) from electronic health records. The performance between ICD codes and LLM was compared via sensitivity and specificity with an α = .05, relative to the gold standard of manual adjudication. External validation was performed using a data set of hospital admissions from June 1, 2018, to May 31, 2019, from a second institution.

RESULTS

Of the 7,555 admissions for patients on ICI therapy in the initial cohort, 2.0% were adjudicated to be due to ICI-colitis, 1.1% ICI-hepatitis, 0.7% ICI-pneumonitis, and 0.8% ICI-myocarditis. The LLM demonstrated higher sensitivity than ICD codes (94.7% v 68.7%), achieving significance for ICI-hepatitis (P < .001), myocarditis (P < .001), and pneumonitis (P = .003) while yielding similar specificities (93.7% v 92.4%). The LLM spent an average of 9.53 seconds/chart in comparison with an estimated 15 minutes for adjudication. In the validation cohort (N = 1,270), the mean LLM sensitivity and specificity were 98.1% and 95.7%, respectively.

CONCLUSION

LLMs are a useful tool for the detection of irAEs, outperforming ICD codes in sensitivity and adjudication in efficiency.

Author Affiliations

1Harvard Medical School, Boston, MA; 2Cardiovascular Imaging Research Center, Massachusetts General Hospital, Boston, MA; 3Leiden University Medical Center, Leiden, the Netherlands; 4Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Boston, MA; 5Brigham and Women’s Hospital, Boston, MA; 6Maastricht University, Maastricht, the Netherlands; 7Division of Hematology and Oncology, Department of Medicine, Massachusetts General Hospital, Boston, MA; 8Far Eastern Memorial Hospital, New Taipei City, Taiwan; 9Department of Dermatology, Brigham and Women’s Hospital, Boston, MA; 10Center for Cutaneous Oncology, Dana-Farber Cancer Institute, Boston, MA; 11Department of Internal Medicine, Cleveland Clinic, Cleveland, OH; 12Division of Population Sciences, Dana-Farber Cancer Institute, Boston, MA; 13Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA; 14Division of Gastroenterology, Hepatology, and Endoscopy, Brigham and Women’s Hospital, Boston, MA; 15Division of Cardiology, Massachusetts General Hospital, Boston, MA; 16Internal Medicine Department, Massachusetts General Brigham Salem Hospital, Salem, MA; 17Division of Gastroenterology, Oregon Health and Science University, Portland, OR; 18Department of Medicine, Oregon Health and Science University, Portland, OR; 19Department of Cell, Developmental, and Cancer Biology, Oregon Health and Science University, Portland, OR; 20Division of Pulmonary and Critical Care Medicine, Massachusetts General Hospital, Boston, MA; 21Center for Immunology and Inflammatory Diseases (CIID), Massachusetts General Hospital Krantz Family Center for Cancer Research, Boston, MA; 22Broad Institute of MIT and Harvard, Cambridge, MA; 23Division of Nephrology, Massachusetts General Hospital, Boston, MA; 24Division of Gastroenterology, Massachusetts General Hospital, Boston, MA;

Leave a Comment

Your email address will not be published. Required fields are marked *

Related Articles