Med Care AI: A Multimodal Artificial Intelligence-based Healthcare Information and Medical Scan Analysis System

Abstract

Digital health care platforms continue to cause an increase in global access to medical information, but they haven been fully able to facilitate access through continued fragmentation and lack of accessibility, as the majority of these systems use text-based descriptions of disease alone, with no capability of intelligent interpretation or interactivity and do not support multimodal forms of data (e.g., medical imaging). In this paper, we present a multimodal artificial intelligence-based platform for health care information retrieval and analysis called MedCare AI, which consists of three components: A curated knowledge base of 610 different diseases across 22 different categories from the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC); An image analysis module that uses artificial intelligence (AI) to analyze scans for six different types of imaging technology, specifically: X-ray; computed tomography (CT); magnetic resonance imaging (MRI); ultrasound; positron emission tomography (PET); and electrocardiogram (ECG) imaging. Our analysis module utilizes a fine-tuned version of the ResNet-50 convolutional neural network (CNN) built on a defined preprocessing pipeline; A health care assistant that uses natural language processing to drive conversational interaction and uses a bi-directional long-short-term memory (BiLSTM)-based named entity recognition system. MedCare AI's performance outcomes were assessed on a dataset comprised of 500 queries, 300 conversations and 200 scans, achieving an average of 92.6%, 91.4%, 90.8%, and 91.1% on accuracy, precision, recall and F1 score respectively, when compared to current chat-based applications, demonstrating up to a 6.2% improvement over standalone chat-based applications. Robustness evaluation results identified less than 2.1% degradation of F1 score as a result of three differing levels of noise; demonstrating that the platform's capabilities as a multimodal integrative system meet the current gaps identified within existing literature, by providing access to disease knowledge retrieval, scan-based diagnostic and conversational interaction from a single easy to use Internet-based platform.