Skip to content
Back to Blog
4 min read

MSK's AI System Reviews Patient Safety Incidents 29x Faster

Memorial Sloan Kettering's AI-ILS uses large language models to classify patient safety incidents 29 times faster, agreeing with experts 88% of the time.

  • AI in Healthcare
  • Patient Safety
  • LLM Applications
  • Medical AI

Memorial Sloan Kettering Cancer Center (MSK) has built an AI system that classifies patient safety incidents 29 times faster than human reviewers. The research, published this week in npj Digital Medicine, applies large language models to one of the slowest parts of hospital safety work: learning from what went wrong.

A researcher working in an MSK lab
A researcher working in an MSK lab

The bottleneck in incident review

Hospitals file thousands of incident reports every year, from near misses to adverse events. Each one may hold a lesson, but someone has to read it, classify it, identify the root cause and decide what to change. Experts do this by hand. The more thorough the review, the longer it takes, and while it waits, the same error can happen again.

Dr. Jean Moran, Director of the Division of Radiotherapy Physics at MSK, put the aim this way: "AI-assisted review and classification of incidents can accelerate learning to improve patient safety, allowing teams to shift focus to designing safer systems in support of patients."

How AI-ILS works

The Artificial Intelligence-based Incident Analysis and Learning System (AI-ILS) was developed by a team led by medical physics resident Dr. Abbas Jinia, supervised by Drs. Jean Moran and Anyi Li. It borrows its structure from aviation safety, using the Human Factors Analysis and Classification System (HFACS).

HFACS sorts the causes of an incident into four levels: unsafe acts, preconditions for unsafe acts, unsafe supervision, and organisational influences. Aviation has used it for decades. The MSK team adapted it for healthcare.

The model was trained on more than 1,500 synthetic incident reports that experts had curated and categorised. The team then tested it on 350 real clinical incidents. The results:

  • 29 times faster than traditional human review
  • 88% agreement with expert classifications
  • 0.92 AUROC (area under the receiver operating characteristic curve, a standard measure of how well a model separates categories)
  • 79% overall accuracy in classifying causes

Reviewers can see the reasoning

The design choice that matters most for adoption is transparency. Reviewers interact with the tool and can question how it reached a classification, so they can check it and overrule it when it is wrong. In a clinical setting, accuracy figures alone rarely earn trust. Staff need to be able to follow and verify the output, and the MSK team built that into the system from the start.

The system does not remove human review. It takes over the first pass of classification, which frees the safety team to work on what follows: designing system-level fixes, spotting trends across many incidents, and putting preventive measures in place.

Lessons for anyone building something similar

Four details in MSK's approach are worth copying:

  • Curated training data. The model learned from expert-reviewed reports, not raw, unreviewed ones.
  • An established framework. HFACS gave the model a clear structure to classify against. Another taxonomy may suit a different setting.
  • Testing on real cases. Validation against real clinical incidents, not only synthetic data, was essential.
  • Visible reasoning. The interactive explanations helped clinical users trust the tool.

The pattern extends beyond hospitals. Incident review needs domain knowledge, follows an established taxonomy, and gains a great deal from speed. Similar work exists in legal document review, financial compliance monitoring and scientific literature triage: large volumes of text, classified against a known framework, with experts left to handle the unusual cases and the decisions. For healthcare providers in the UAE with their own incident reporting systems, MSK's study is a useful model of how to scope such a project and how to test it before anyone relies on it.