Bharat Sum: OCR-Enabled Multilingual News Summarization and Bias Analysis Framework
Abstract
India is home to an incredible number of languages which leads to the production of substantial amounts of news articles provided in their regional languages including Telugu, Tamil, Hindi, Bengali, Kannada, and Malayalam. Unfortunately, current systems for processing this data do so independently, rather than as part of a comprehensive framework utilizing all the necessary components in one queue of processing pipelines; OCR extraction, Translation, Summarization, and Bias Detection must all be completed one at a time, and do not allow for seamless data passing between functions. In this paper, we will present Bharat Sum, a multilingual news summarization and bias detection system that incorporates OCR capabilities into five different types of processing stages; OCR Text Extraction using Tesseract, Automatic Language Detection using LangDetect, Topic Segmented Abstractive Summarisation using mT5, Translation to English using mBART, and Sentiment based Bias Classification using DistilBERT - all accessible through a single scalable architecture running on commodity hardware and implemented via Streamlit. Our testing involved 150 news articles covering each of the five languages named above. The results achieved were as follows; OCR extraction accuracy of 89.7%, Language Detection Accuracy of 94.2%, Summarisation Quality (ROUGE-L F1) of 92.1% Translation consistency of 91.4% and Sentiment Classification Accuracy of 88.6%. The average end-to-end processing time was between 10 and 16 seconds. Our analysis of Bharat Sum has revealed that it significantly outperforms previous single function systems by providing an Integrated, Real-Time Multilingual Processing capability which currently does not exist in this context. Bharat Sum has the potential to address significant gaps in the research literature regarding the analysis of Integrated Multilingual Media, and will likely serve as an economically viable solution for organisations conducting Digital Journalism, Media Monitoring, or Accessing Multilingual Information.