A Comparative Study of LSTM-based Multimodal Human Activities Recognition

Abstract

The aim of this paper is to investigate the impact of incorporating gravity and attitude data into gyroscopes- and accelerometers-based datasets on the performance of multimodal human activities recognition. Most human activities recognition studies have been based on two sensor data, gyroscopes and accelerometers. However, with the increasing number of smart devices that can now host multiple sensors, there are many potential possibilities that can enhance human activity recognition performance. In order to understand the impact of other modalities on human activity recognition, this paper trains and evaluates Long Short-Term Memory (LSTM) models for different modalities through comparative experiments, with the condition of equal weights for all modalities, and records the evaluation process in detail. The results were evaluated in two ways: confusion matrix and global interpretation of SHapley Additive exPlanations (SHAP) values. The experimental results show that the model using all modalities outperforms the other models in terms of recall and precision, even though the model performance does not increase linearly with the number of modalities. On the other hand, after a global interpretation of the anomalous result "walking", it was found that gravity causes the model to misclassify "walking" as "jogging" more easily, but gravity and attitude data together can enhance the contribution of gyroscope and acceleration data in the model. From the results, using all 4 modalities gives the best performance of the model.