DOP13 Artificial Intelligence (AI) in endoscopy - Deep learning for detection and scoring of Ulcerative Colitis (UC) disease activity under multiple scoring systems

Byrne, M.(1);East, J.(2);Iacucci, M.(3);Panaccione, R.(4);Kalapala, R.(5);Duvvur, N.(5);Rughwani, H.(5);Singh, A.(5);Henkel, M.(6);Canaran, L.(7);Laage, G.(7);St-Denis, L.(7);Nikfal, S.(7);Asselin, J.(7);Monsurate, R.(8);Cremonese, E.(7);Soudan, F.(7);Travis, S.(9)

(1)Vancouver General Hospital, Division of Gastroenterology, Vancouver, Canada;(2)John Radcliffe Hospital, Gastroenterology, Oxford- Oxfordshire, United Kingdom;(3)Institute of Translational of Medicine Immunology & Immunotherapy, Gastroenterology, Birmingham, United Kingdom;(4)University of Calgary, Gastroenterology, Calgary- AB, Canada;(5)Asian Institute of Gastroenterology, Gastroenterology, Hyderabad- Telangana, India;(6)University of Buenos Aires, Gastroenterology, Buenos Aires, Argentina;(7)Ivado Labs, Artificial Intelligence, Montreal- QC, Canada;(8)University of British Columbia, Business Administration, Vancouver- BC, Canada;(9)University of Oxford, Medical Sciences Division, Oxford- Oxfordshire, United Kingdom


Computer vision & deep learning(DL)to assess & help with tissue characterization of disease activity in Ulcerative Colitis(UC)through Mayo Endoscopic Subscore(MES)show good results in central reading for clinical trials.UCEIS(Ulcerative Colitis Endoscopic Index of Severity)being a granular index,may be more reflective of disease activity & more primed for artificial intelligence(AI). We set out to create UC detection & scoring,in a single tool & graphic user interface(GUI),improving accuracy & precision of MES & UCEIS scores & reducing the time elapsed between video collection,quality assurance & final scoring.We apply DL models to detect & filter scorable frames,assess quality of endoscopic recordings & predict MES & UCEIS scores in videos of patients with UC


We leveraged>375,000frames from endoscopy cases using Olympus scopes(190&180Series).Experienced endoscopists & 9 labellers tagged~22,000(6%)images showing normal, disease state(MES orUCEIS subscores)& non-scorable frames.We separate total frames in 3 categories:training(60%),testing(20%)&validation(20%).Using a Convolutional Neural Network(CNN)Inception V3,including a biopsy & post-biopsy detector,an out-of-the-body framework &  blue light algorithm.Similar architecture for detection with multiple separate units & corresponding dense layers taking CNN to provide continuous scores for 5 separate outputs:MES,aggregate UCEIS & individual components Vascular Pattern,Bleeding & Ulcers.


Multiple metrics evaluate detection models.Overall performance has an accuracy of~88% & a similar precision & recall for all classes.

MAE(distance from ground truth)& mean bias(over/under-prediction tendency)are used to assess the performance of the scoring model.Our model performs well as predicted distributions are relatively close to the labelled,ground truth data & MAE & Bias for all frames are relatively low considering the magnitude of the scoring scale.

To leverage all our models,we developed a practical tool that should be used to improve efficiency & accuracy of reading & scoring process for UC at different stages of the clinical journey.


We propose a DL approach based on labelled images to automate a workflow for improving & accelerating UC disease detection & scoring using MES & UCEIS scores. Our deep learning model shows relevant feature identification for scoring disease activity in UC patients, well aligned with both scoring guidelines,performance of experts & demonstrates strong promise for generalization.Going forward, we aim to continue developing our detection & scoring tool. With our detailed workflow supported by deep learning models, we have a driving function to create a  precise & potentially superhuman level AI to score disease activity