Research

Multimodal Assessment of Neonatal Pain Using Computer Vision

Abstract

Infants receiving care in the Neonatal Intensive Care Unit (NICU) experience several painful procedures during their hospitalization. Assessing neonatal pain is difficult because the current standard for assessment is subjective, inconsistent, and discontinuous. The intermittent and inconsistent assessment can induce poor treatment and, therefore, cause serious longterm outcomes. The main aim of this project is to develop a robust and comprehensive automatic system that generates a standardized pain assessment comparable to those obtained by conventional nurse-derived pain scores. The continuous monitoring of pain, using affordable, non-invasive, and easily integrable devices, provides immediate pain detection and intervention, and therefore, contribute to improved long term outcomes; i.e., reduce the outcomes of under- and over-treatment. It can also decrease caregivers鈥� bias and assessment burden.

While further research is needed, the preliminary results of our research showed that the automatic assessment of neonatal pain is a viable and more efficient alternative to the manual assessment.

Engineering Team:

Dr. Yu Sun, Professor, Computer Science and Engineering, USF
Dr. Dmitry Goldgof, Professor, Computer Science and Engineering, USF
Jacqueline Hausmann, PhD Student, Computer Science and Engineering, USF
Md Imran Hossain, PhD Student, Computer Science and Engineering, USF
Jiayi Wang, PhD Student, Computer Science and Engineering, USF

Medical Team:

Dr. Thao Ho, Assistant Professor, College of Medicine Pediatrics, USF Health, USF
Dr. Stephanie Prescott, Assistant Professor, College of Nursing, USF Health, USF
Dr. Denise Maguire, Associate Professor, College of Nursing, USF Health, USF
Dr. Yangxin Huang, Professor, College of Public Health, USF Health, USF
Marcia Kneusel, RN, Clinical Research Nurse, College of Medicine Pediatrics, USF Health, USF

Collaborators:

Dr. Md Sirajus Salekin, Applied Scientist, Amazon
Dr. Ghada Zamzmi, Research Fellow, NLM, NIH
Dr. Peter Mouton, Director and Chief Scientific Officer, SRC Biosciences
Dr. Mark Last, Professor and Director of Data Science Research Center, Ben-Gurion University of the Negev, Israel
Dr. Kanwaljeet S. Anand, Professor of Pediatrics, Stanford University, CA, United States

Previous Team Members:

Dr. Terri Ashmeade, Professor, College of Medicine Pediatrics, USF Health, USF
Dr. Rangachar Kasturi, Professor, CSE, USF
Chih-Yun Pai, Master's Student, CSE, USF

Patents

  • US-11631280-B2, "System and Method for Multimodal Spatiotemporal Pain Assessment", April 2023.
  • US-20230309915-A1, "System and Method for Attentional Multimodal Pain Estimation", October 2023.
  • US-11202604-B2, "Comprehensive and Context-sensitive Neonatal Pain Assessment System and Methods Using Multiple Modalities", December 2021.
  • US-20210052215-A1, "System and Method for Multimodal Spatiotemporal Pain Assessment", Feburary 2021.
  • US-20210030354-A1, "Neonatal Pain Identification From Neonatal Facial Expressions", Feburary 2021.
  • US-10827973-B1, "Machine-based Infants Pain Assessment Tool", November 2020.
  • US-20190320974-A1, "Comprehensive and Context-sensitive Neonatal Pain Assessment System and Methods Using Multiple Modalities", October 2020.

Published Papers

Md Imran Hossain, Ghada Zamzmi, Peter R Mouton, Yu Sun, and Dmitry Goldgof
IEEE Engineering in Medicine and Biology Society (EMBC24), 2024



Jacqueline Hausmann, Md Sirajus Salekin, Ghada Zamzmi, Peter R Mouton, Stephanie M.Prescott, Yu Sun,and Dmitry Goldgof
IEEE Access, 2024



Md Imran Hossain, Ghada Zamzmi, Peter R Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry Goldgof
ACM Computing Surveys, 2023



Amelia Llerena, Krystal Tran, Danyal Choudhary, Jacqueline Hausmann, Dmitry Goldgof, Yu Sun, and Stephanie M.Prescott
Frontier in Pediatrics, 2023



Md Imran Hossain, Ghada Zamzmi, Peter R Mouton, Yu Sun, and Dmitry Goldgof
2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS)



Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Peter R. Mouton, Kanwaljeet J. S. Anand, Terri Ashmeade, Stephanie Prescott, Yangxin Huang, and Yu Sun
Medical Image Computing and Computer Assisted Intervention (MICCAI), 2022



Jacqueline Hausmann, Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, and Yu Sun
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshops, 2021



Sidharth Srivatsav Sribhashyam, Md Sirajus Salekin, Dmitry Goldgof, Ghada Zamzmi, Mark Last, and Yu Sun
IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2021



Md Sirajus Salekin, Peter R Mouton, Ghada Zamzmi, Raj Patel, Dmitry Goldgof, Marcia Kneusel, Sammie L Elkins, Eileen Murray, Mary E Coughlin, Denise Maguire, Thao Ho, and Yu Sun
Paediatric and Neonatal Pain, 2021



Md Sirajus Salekin, Ghada Zamzmi, Jacqueline Hausmann, Dmitry Goldgof, Rangachar Kasturi, Marcia Kneusel, Terri Ashmeade, Thao Ho, Yu Sun
Data in Brief, 2021



Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
Computers in Biology and Medicine, 2021



Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
IEEE International Conference on Automatic Face and Gesture Recognition (FG), 2020



Md Sirajus Salekin, Ghada Zamzmi, Rahul Paul, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
IEEE Healthcare Innovations and Point of Care Technologies Conference (HI-POCT), 2019



Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2019



Ghada Zamzmi, Rahul Paul, Md Sirajus Salekin, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
IEEE Transactions on Biometrics, Behavior, and Identity Science, 2019



Ghada Zamzmi, Rahul Paul, Dmitry Goldgof, Rangachar Kasturi, Terri Ashmeade, Yu Sun
International Joint Conference on Neural Networks (IJCNN), 2019



Ghada Zamzmi, Chih-Yun Pai, Dmitry Goldgof, Rangachar Kasturi, Terri Ashmeade, Yu Sun
IEEE Transactions on Affective Computing, 2019



Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Yu Sun
UbiComp, 2018



Ruicong Zhi, Ghada Zamzmi, Dmitry Goldgof, Terri Ashmeade, Yu Sun
Journal of Clinical Medicine, 2018



Ruicong Zhi, Ghada Zamzmi, Dmitry Goldgof, Terri Ashmeade, Tingting Li, Yu Sun
IEICE Transactions on Information and Systems, 2018



Ghada Zamzmi, Rangachar Kasturi, Dmitry Goldgof, Ruicong Zhi, Terri Ashmeade, Yu Sun
IEEE Reviews in Biomedical Engineering, 2018


Ghada Zamzmi, Chih-Yun Pai, Dmitry Goldgof, Rangachar Kasturi, Yu Sun, Terri Ashmeade
Scandinavian Conference on Image Analysis (SCIA), 2017


Ghada Zamzmi, Chih-Yun Pai, Dmitry Goldgof, Rangachar Kasturi, Terri Ashmeade, Yu Sun
International Conference on Pattern Recognition (ICPR), 2016



Ghada Zamzami, Gabriel Ruiz, Dmitry Goldgof, Rangachar Kasturi, Yu Sun, Terri Ashmeade
IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2015

Media Coverage

Talk / Presentation

Aug 06, 2019 - Dr. Dmitry Goldgof, A joint seminar of the Data Science Research Center and the Department of Biomedical Engineering at Ben-Gurion University of Negev, Israel on "Healthcare in the Age of AI and Deep Learning: Automatic Assessment of Neonatal Pain"
Jun 18, 2019 - Dr. Dmitry Goldgof, Weekly Idea/Research Collabaration Meeting at Johns Hopkins All Children's Hospital, St. Petersburg, FL, USA, on "Automatic Neonatal Pain Detection"
Mar 26, 2019 - Dr. Dmitry Goldgof, High Profile Talk at 最新天美传媒, Tampa, FL, USA, on "Healthcare in the Age of AI and Deep Learning: Automatic Assessment of Neonatal Pain"

Short Description

This pilot dataset (USF-MNPAD-I) is a part of a neonatal pain assessment project, which aims to develop new algorithms and techniques for the automatic monitoring of neonatal pain. We are currently involved in an ongoing effort to collect a large multimodal pain dataset from newborns and premature neonates. USF-MNPAD-I dataset contains recorded subjects in the Neonatal Intensive Care Unit (NICU) at Tampa General Hospital. The neonates were recorded during baseline and while undergoing procedural painful procedures (e.g., heel lancing and immunization). In addition to the recorded raw videos, this dataset contains vital signs data as well as ground truth labels documented by two trained nurses.

Release of Dataset

To advance the state-of-the-art in neonatal pain assessment, portion of the dataset will be made available to researchers on a case-by-case basis. All requests for USF-MNPAD-I must be submitted in writing and sent to the Principal Investigators of this project. The requestor must sign this document and agree to observe the restrictions listed below. In addition to other possible remedies, failure to observe these restrictions may result in access being denied for any other portion of the database. Current installments of the database may be made available to researchers via a protected internet site or on CD. There will be no charge for imagery made available and downloaded via the Internet.
* Please note that initially shareable subjects from the USF-MNPAD-I will be shared. A larger dataset, will be made available, as per request, for research use.

Citations

Any manuscript developed completely or partially using USF-MNPAD-I dataset must include the following citations:


Md Sirajus Salekin, Ghada Zamzmi, Jacqueline Hausmann, Dmitry Goldgof, Rangachar Kasturi, Marcia Kneusel, Terri Ashmeade, Thao Ho, Yu Sun
Data in Brief, 2021


Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
Computers in Biology and Medicine, 2021

Contact

To get the datasets, you need to contact the following person with a full description of why you need it.


Jacqueline Hausmann
Department of CSE, 最新天美传媒, United States
hausmannj@usf.edu

Our Sponsors

Sample

 

Sample

 

Sample