Memory Assist: An AI-Assisted Support System for Alzheimer’s Patients

Abstract

This paper presents an AI-assisted desktop support system designed for Alzheimer’s patients that runs continuously on standard consumer hardware. The system uses a webcam and microphone to passively detect human presence and conversational activity by fusing deep face detection (SCRFD with HOG pedestrian fallback), acoustic energy (RMS), and lip-sync dynamics (Mouth Aspect Ratio). By introducing this novel event-gated multimodal recording architecture, the system ensures patient privacy-by-design and achieves an 84% reduction in local storage overhead and cloud API execution costs compared to continuous recording. When a conversation is detected, audio is automatically recorded, transcribed, and sent to Gemini 2.5 Flash, which generates a structured summary along with extracted events, tasks, medication mentions, and an importance score for each memory. When a recognized caregiver or visitor appears on camera again, the system announces their name, relationship, and a recall of the last conversation with that person via text-to-speech, providing immediate contextual memory support to the patient. Beyond conversation memory, the system extracts upcoming appointments and tasks into a dedicated event store, fires time-based voice reminders, and supports natural-language queries such as "When is my next appointment?" answered directly from the stored memory database. A multi-caregiver web dashboard built with Flask allows authorized caregivers to browse memories, manage known persons, configure reminders, and review extracted events. The system is built on ArcFace face recognition pre-trained on WebFace600K and WIDERFace, and evaluated on real recorded conversations and enrolled caregiver identities.