A Vision-Based Human-Computer Interaction System Using Hand Gestures

Abstract

Human–Computer Interaction (HCI) traditionally depends on physical input devices such as a mouse, keyboard, touchpad, and touchscreen. Although these devices provide efficient interaction, they require direct physical contact and may not always be convenient in situations where touchless interaction is preferred. This paper presents a Virtual Mouse Using Hand Recognition System, a computer vision-based system that enables users to control the mouse cursor and perform basic mouse operations using hand gestures. The proposed system uses a standard webcam to capture real-time video frames and employs OpenCV for image processing and MediaPipe Hand Tracking for detecting and tracking hand landmarks. The detected landmark coordinates are analyzed to identify finger positions and recognize predefined gestures. PyAutoGUI is used to translate the recognized gestures into operating-system-level mouse actions such as cursor movement, left click, right click, and scrolling. A coordinate mapping mechanism converts the hand position from the webcam frame to the computer screen coordinates, while a smoothing technique is incorporated to reduce unwanted cursor movement. The proposed system eliminates the need for a physical mouse and provides a natural and contactless method of computer interaction. The system can be useful in accessibility applications, interactive presentations, smart environments, educational demonstrations, and hygienic touch-free computing environments. Experimental observations indicate that the combination of real-time hand landmark detection and gesture-based interaction can provide responsive and intuitive mouse control under suitable lighting and camera conditions.