Automated deep learning-based arousal detection agrees with ten expert scorers in unseen data

Abstract

Study Objectives: Automatic arousal scoring is essential for the cost-effective analysis of polysomnograms. Many previous automatic methods have been tested in homogeneous, single-center datasets, and all against one scorer, on event-by-event basis, or in 30-s segments. We developed a deep learning-based arousal scoring model and tested it at 1-s resolution in an independent dataset scored by ten expert scorers from seven different sleep centers. Materials and Methods: A fully convolutional neural network model was developed using 1847 openly available polysomnograms from the Multi-Ethnic Study of Atherosclerosis (MESA) and tested in 205 hold-out polysomnograms from the MESA cohort, and in an independent Sleep Revolution (SR) cohort of 50 polysomnograms, each scored by ten scorers. Model performance was evaluated on an event-by-event basis and in 1-s segments. In the SR dataset, the performance was tested against each scorer individually and against their majority agreement. Results: In the MESA test set, the model achieved an event-by-event F1-score of 0.77, a 1-s segment κ-value of 0.67, and an arousal index (ArI) intraclass correlation coefficient (ICC) of 0.83. In the multi-scorer SR dataset, the model reached a median event-by-event F1-score, a 1-s κ-value, and an ArI ICC of 0.61, 0.50, and 0.66, respectively, exceeding the corresponding median values of the scorers among themselves (0.57, 0.46, and 0.59, respectively). Conclusions: The model achieved higher median arousal scoring agreement with ten manual scorers than the scorers did among themselves in an independent dataset. This suggests that the model generalizes well and can reliably be used in different sleep recordings.