Multimodal Perception Methods for Embodied Robots: Sensors, Fusion Architectures, and Human–Robot Interaction Tasks

Abstract

Embodied robots must perceive geometry, motion, contact, human behavior, and task context under latency and safety constraints that no single sensor can satisfy. Multimodal perception addresses this limitation by combining complementary observations, but adding modalities also introduces calibration, synchronization, representation, missing-data, and confidence-management problems. This paper presents a method-centered review organized around the chain from sensing to representation and alignment, fusion architecture, interaction task, and failure handling. We examine frame and event vision, depth and ranging, audio, human-state cues, force/torque sensing, and tactile sensing; then compare data-level, feature-level, decision-level, and temporal fusion. The review connects these choices to scene and object understanding, affordance and physical-property estimation, human activity recognition, and intent or task-stage prediction. A central finding is that fusion depth should follow the strength and stability of cross-modal coupling. Early fusion can preserve fine correspondence when sensors are well calibrated; intermediate fusion supports learned cross-modal interactions; late fusion offers modularity and graceful degradation; and temporal state estimation is essential when observations arrive asynchronously. Robust systems should represent sensor validity and uncertainty explicitly, train with realistic modality dropout and corruption, and separate semantic inference from safety-critical constraints. Foundation models improve open-vocabulary grounding and task communication, but do not eliminate timing, observability, or verification requirements. The paper concludes with a condition–method–benefit–cost guide and identifies benchmark practices needed for deployment-relevant evaluation: synchronized raw data, missing-modality protocols, calibration perturbations, end-to-end latency, uncertainty calibration, and task-level consequences of perceptual error.