Design of a Bluetooth Earbud Heartrate and Emotion Detection System based on Multimodal Deep Learning

Abstract

As the arrival of digital age, the social context becomes more competitive. People are living, working and studying under increasingly tremendous pressure. In today’s society’s evaluation system with the immense academic stress, anxiety, mental breakout, depression and other mental issues emerge among teenagers.  The trend of the aging of population leads the elderlies to face health related challenges like loneliness, decline of cognitive competence, sleep disorders, etc. Therefore, the monitor of mental and physical health has become one of the most important issues the society need to concern. In recent years, smart watches, wristband and other wearable devices have already made some progress in the aspect of monitoring physiological signals. Still, these devices have the limitations such as low wearing comfort, low long-term compliance, insufficient privacy and security. Thus, current solutions hardly achieve a continuous, private, and highly accurate monitor on the users’ mental and physical state of health. Targeting on the issues above, this research proposes a Bluetooth Earbud Heartrate and Emotion (BEHE) detection system based on multimodal deep learning. Utilizing the advantageous characteristic of portability, high privacy and convenience of earbud, we integrate photoplethysmography (PPG) sensor, physiological acoustics and inertial measurement unit (IMU) sensors to achieve the real-time heartrate and emotion detection, and provide further individualized intervention, like emotion-based music suggestion, voice assistance and feedback for emotional support or adjustment. This research provides new technical opportunity for mental health assistance.