A Modular Multimodal Framework for Automated Misinformation Investigation: Integrating OCR, NLP, Vision-Language Captioning, and Speech Analysis for Evidence-Based Trust Scoring

Abstract

The rapid spread of manipulated images, doctored videos, and misleading text across digital platforms has created a pressing need for automated tools that can assist human fact-checkers rather than replace them. This paper presents the design and implementation of a modular, multimodal misinformation investigation platform that combines optical character recognition (OCR), natural language processing (NLP), vision-language image captioning, and automatic speech recognition (ASR) within a single evidence-fusion pipeline. The system extracts on-screen text, spoken audio, visual content descriptions, and linguistic sentiment/clickbait indicators from a submitted image or video, and combines these independent signals into a composite trust score and human-readable explanation through a rule-based fusion engine. The platform is implemented as a FastAPI backend with a MySQL relational store, JSON Web Token (JWT) based authentication, and a React single-page-application frontend that supports investigation history, analytics, and report export. We describe the architecture, the responsibilities of each processing module, the design of the evidence-fusion scoring function, and the security and data-persistence layers. We further discuss the current limitations of the rule-based fusion approach, including its sensitivity to keyword-based clickbait detection and its reliance on unimodal confidence rather than learned cross-modal weighting, and we outline directions for extending the system toward a trained fusion model and stronger authentication practices. Keywords— misinformation detection, multimodal fusion, optical character recognition, natural language processing, vision-language models, speech recognition, trust scoring, FastAPI, digital forensics.