Research on Scene Perception and Task Semantic Understanding for Indoor Service Robots: Based on Literature Review and Case Analysis

Semantic scene graphs for robot-based maintenance and inspection of civil infrastructure

Task-Oriented Navigation with Semantic Perception for Monocular Camera-Based Mobile Robots

For mobile robots to execute commands given by humans in environments lacking localization infrastructure, they must be capable of making semantic inferences about the environment. To this end, we present a task-based navigation method for a mobile robot using single monocular camera. The robot’s perception mechanism consists of a SSD-InceptionV2 for target object detection and MiDaS-Small for depth map estimation. By exploiting the geometric advantage of the motion plane of ground robots, metric distances derived from relative depth outputs are used in the obstacle avoidance algorithm, and a Kalman filter is applied to ensure continuity in target object tracking. Tests conducted on a Jetson TX2 platform verified that the system exhibits stable performance at approximately 15 FPS and successfully completes its task. The results demonstrate that mobile robot navigation systems can be developed in infrastructure-free environments using solely camera-based perception.

A BIM-enabled, Agent-based Discrete-event Simulation Platform for Robotic Studies: A Method based on Graph Theory

Indoor robots are increasingly employed for facility management tasks such as cleaning and inspection. These applications primarily rely on navigation and can be effectively supported by predefined routes or perception-driven Simultaneous Localization and Mapping (SLAM) techniques. However, more complex tasks, such as locating and repairing leaking pipes, require not only navigation but also access to building information, including the location, geometry, material, and operational attributes of components. Existing navigation approaches provide only limited environmental understanding and cannot readily supply such information. In contrast, Building Information Modeling (BIM) contains rich geometric, semantic, and operational information that remains largely underutilized in robotic applications. This study proposes a BIM-enabled, agent-based simulation platform for knowledge-driven indoor robot navigation and operation planning. Within the framework, indoor environments are discretized into grid cells that are mapped to graph nodes and classified as target, obstacle, or regular nodes according to their spatial relationships with building elements. Traversal costs are assigned to edges connecting neighboring nodes, enabling graph-theoretic algorithms to compute efficient and collision-free navigation paths while avoiding obstacles. Simulation results demonstrate that the proposed graph representation enables efficient and collision-free navigation. A key limitation associated with coarse discretization, namely overlap between target-occupied and obstacle-occupied cells, is identified and mitigated through grid refinement, improving spatial accuracy and path feasibility. The proposed platform supports virtual evaluation of robotic operations prior to deployment and provides a foundation for BIM-informed robotic systems in facility management.

LLM-Based Semantic Navigation on a Low-Cost ROS Mobile Robot: A Hybrid Edge–Cloud Architecture

Autonomous mobile robots require robust perception and high-level reasoning to operate in complex indoor environments. While the Robot Operating System (ROS) provides a modular framework for mapping and navigation, classical pipelines lack semantic understanding and natural-language interaction. This paper presents a semantic-aware autonomous navigation framework implemented on a ROS 1 (Melodic) mobile robot equipped with a two-dimensional light detection and ranging (LiDAR) sensor and an RGB-D camera. The system integrates LiDAR-based simultaneous localization and mapping (SLAM), the ROS navigation stack (move_base), and a lightweight You Only Look Once (YOLO) object detector for real-time on-board perception, and it anchors detections into the metric map to build a semantic map. A large language model (LLM) interprets natural-language instructions and converts them into structured navigation goals. Perception and control run entirely on-board the Jetson Nano, whereas the LLM is invoked episodically as a cloud service, yielding a hybrid embedded/cloud architecture. In indoor trials over a semantic map of two object classes, the system grounded all ten multilingual commands to the correct objects at a reasoning cost of about one second, and safely rejected a command referring to an unmapped object. We present this as an in-depth single-platform case study: owing to the 4 GB memory budget, the on-board detector and the full navigation stack are time-multiplexed rather than run continuously in parallel; nonetheless, a single degraded end-to-end trial confirmed that perception, online semantic anchoring, language grounding, and navigation compose within one continuous session. The framework offers a low-cost, extensible basis for language-guided robots in smart environments.

Open-Vocabulary Semantic Segmentation with Uncertainty Alignment for Robotic Scene Understanding of Indoor Building Spaces

Society is steadily embracing the idea of coinhabitant service robots in the built environment for assistance with household tasks and chores. In particular, the growing number of older adults, individuals with disabilities, and people requiring daily assistance has further created an urgent need for intelligent coinhabitant robots capable of supporting mobility, independence, and safety within physical living spaces. The growing complexity of modern built environments and the increasing prevalence of coinhabitant robots have underscored the need for advanced computational methods to support analysis, monitoring, and management. To provide users with meaningful assistance and clear scene understanding in such settings, robots must be able to perceive, interpret, and reason about their surroundings in a way that aligns with human spatial understanding. Place segmentation, which delineates spatial regions such as rooms or functional areas, together with semantic recognition, which assigns meaningful labels to these regions, forms the basis for accurate scene understanding in building-scale environments. Existing deep learning approaches have largely relied on closed-vocabulary detection systems, which are limited to a predefined set of categories of rooms and therefore struggle to interpret diverse or naturalistic descriptions of spaces. Furthermore, most current methods neglect the uncertainty inherent in scene recognition—for example, when rooms have overlapping functions (e.g., kitchen–dining areas) or when people use linguistically similar descriptions to refer to distinct spaces—thereby reducing their effectiveness in ambiguous or complex environments. To address these challenges, we propose an open-vocabulary semantic segmentation and classification pipeline that leverages vision–language models (VLMs) and large language models. Our framework follows a “segment–detect–select” paradigm for open-vocabulary scene understanding, consisting of three modules: (1) segmentation of distinct rooms and regions in the scene; (2) VLM-based similarity scoring between descriptive queries and segmented regions; and (3) adaptive conformal prediction to select among candidate regions while explicitly accounting for uncertainty. Evaluation results on the large-scale Matterport3D dataset, which comprises diverse photorealistic indoor environments, show that the proposed framework outperforms state-of-the-art open-vocabulary methods in scene segmentation [by 30% in AP50 and 5% in mean intersection over union (mIOU)], classification (by 200% in F1 score and 263% in mean average precision), and selection (by 50% in room mIoU). These advances are significant for built environments, as improved scene understanding directly supports safer, more adaptive, and more intuitive coinhabitant assistive agents for humans in complex indoor spaces.

Human–AI Co-Navigation for Indoor Object Search under Uncertainty

Assistive technologies for people with visual impairments increasingly use artificial intelligence to support object-finding and navigation in indoor environments. Yet fully autonomous perception remains unreliable in such settings, as indoor spaces are visually complex, only partially observable from the user’s current viewpoint, and subject to continuous change. Our work takes the position that effective assistive navigation is inherently collaborative; the system performs continuous perceptual processing, while the user provides occasional natural-language guidance when the search becomes uncertain or inefficient. To this end, we propose a human–AI collaboration framework that utilizes a Vision-Language Model (VLM) as the perceptual and semantic backbone of a navigation agent. A human user, modeled by a simulated intervention controller, provides sparse and structured guidance, which is integrated with the VLM to update its semantic search hypotheses toward the likely location of the target object. Evaluation is conducted in the Habitat simulator on photorealistic scenes from the Habitat-Matterport3D dataset. Experiments analyze how human guidance affects task success and navigation efficiency, showing that guidance is most effective when it corrects the VLM's misaligned semantic search hypotheses, providing insights into the role of minimal human input in VLM-based assistive navigation systems.