Multimodal hand gesture recognition using single IMU and acoustic measurements at wrist
Autoři:
Nabeel Siddiqui aff001; Rosa H. M. Chan aff001
Působiště autorů:
Department of Electrical Engineering, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong, China
aff001
Vyšlo v časopise:
PLoS ONE 15(1)
Kategorie:
Research Article
doi:
https://doi.org/10.1371/journal.pone.0227039
Souhrn
To facilitate hand gesture recognition, we investigated the use of acoustic signals with an accelerometer and gyroscope at the human wrist. As a proof-of-concept, the prototype consisted of 10 microphone units in contact with the skin placed around the wrist along with an inertial measurement unit (IMU). The gesture recognition performance was evaluated through the identification of 13 gestures used in daily life. The optimal area for acoustic sensor placement at the wrist was examined using the minimum redundancy and maximum relevance feature selection algorithm. We recruited 10 subjects to perform over 10 trials for each set of hand gestures. The accuracy was 75% for a general model with the top 25 features selected, and the intra-subject average classification accuracy was over 80% with the same features using one microphone unit at the mid-anterior wrist and an IMU. These results indicate that acoustic signatures from the human wrist can aid IMU sensing for hand gesture recognition, and the selection of a few common features for all subjects could help with building a general model. The proposed multimodal framework helps address the single IMU sensing bottleneck for hand gestures during arm movement and/or locomotion.
Klíčová slova:
Accelerometers – Acoustic signals – Acoustics – Body limbs – Nonverbal communication – Wrist – Microphones – Hands
Zdroje
1. Jiang X, Merhi L-K, Menon C. Force Exertion Affects Grasp Classification Using Force Myography. IEEE Trans Human-Mach Syst. 2018 Apr;48(2):219–26.
2. Savur C, Sahin F. Real-Time American Sign Language Recognition System Using Surface EMG Signal. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). Miami, FL, USA: IEEE; 2015. p. 497–502.
3. Galka J, Masior M, Zaborski M, Barczewska K. Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition. IEEE Sensors J. 2016 Aug;16(16):6310–6.
4. Jeong-Mook Lim, Dong-Woo Lee, Bae-Sun Kim, Il-Yeon Cho, Jae-Cheol Ryou. Recognizing hand gestures using wrist shapes. In: 2010 Digest of Technical Papers International Conference on Consumer Electronics (ICCE). Las Vegas, NV: IEEE; 2010. p. 197–8.
5. Xu C, Pathak PH, Mohapatra P. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch. In: Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications—HotMobile ‘15. Santa Fe, New Mexico, USA: ACM Press; 2015. p. 9–14.
6. CoolSo. Available from: http://www.coolso.com.tw/ (accessed on 10 August 2019)
7. Zhu Y, Jiang S, Shull PB. Wrist-worn hand gesture recognition based on barometric pressure sensing. In: 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN). Las Vegas, NV, USA: IEEE; 2018. p. 181–4.
8. Shull PB, Jiang S, Zhu Y, Zhu X. Hand Gesture Recognition and Finger Angle Estimation via Wrist-Worn Modified Barometric Pressure Sensing. IEEE Trans Neural Syst Rehabil Eng. 2019 Apr;27(4):724–32. doi: 10.1109/TNSRE.2019.2905658 30892217
9. Jiang S, Lv B, Guo W, Zhang C, Wang H, Sheng X, et al. Feasibility of Wrist-Worn, Real-Time Hand, and Surface Gesture Recognition via sEMG and IMU Sensing. IEEE Trans Ind Inf. 2018 Aug;14(8):3376–85.
10. Watakabe M, Mita K, Akataki K, Itoh Y. Mechanical behaviour of condenser microphone in mechanomyography. Med Biol Eng Comput. 2001 Mar;39(2):195–201. doi: 10.1007/bf02344804 11361247
11. Siddiqui N, Chan RHM. A wearable hand gesture recognition device based on acoustic measurements at wrist. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) [Internet]. Seogwipo: IEEE; 2017. p. 4443–6.
12. Zhang C, Starner T, Inan O, Abowd GD, Xue Q, Waghmare A, et al. FingerPing: Recognizing Fine-grained Hand Poses using Active Acoustic On-body Sensing. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems—CHI ‘18. Montreal QC, Canada: ACM Press; 2018. p. 1–10.
13. Beck TW, Housh TJ, Cramer JT, Weir JP, Johnson GO, Coburn JW, et al. Mechanomyographic amplitude and frequency responses during dynamic muscle actions: a comprehensive review. Biomed Eng Online. 2005 Dec 19;4:67. doi: 10.1186/1475-925X-4-67 16364182
14. Orizio C, Liberati D, Locatelli C, De Grandis D, Veicsteinas A. Surface mechanomyogram reflects muscle fibres twitches summation. J Biomech. 1996 Apr;29(4):475–81. doi: 10.1016/0021-9290(95)00063-1 8964777
15. Guo W, Sheng X, Liu H, Zhu X. Mechanomyography Assisted Myoeletric Sensing for Upper-Extremity Prostheses: A Hybrid Approach. IEEE Sensors J. 2017 May 15;17(10):3100–8.
16. Harrison AP. A more precise, repeatable and diagnostic alternative to surface electromyography—an appraisal of the clinical utility of acoustic myography. Clin Physiol Funct Imaging. 2018 Mar;38(2):312–25. doi: 10.1111/cpf.12417 28251802
17. Orizio C. Soundmyogram and EMG cross-spectrum during exhausting isometric contractions in humans. J Electromyogr Kinesiol. 1992;2(3):141–9. doi: 10.1016/1050-6411(92)90011-7 20719607
18. Hermens HJ, Freriks B, Disselhorst-Klug C, Rau G. Development of recommendations for SEMG sensors and sensor placement procedures. J Electromyogr Kinesiol. 2000 Oct;10(5):361–74. doi: 10.1016/s1050-6411(00)00027-4 11018445
19. Kim T-K, Shimomura Y, Iwanaga K, Katsuura T. Influence of force tremor on mechanomyographic signals recorded with an accelerometer and a condenser microphone during measurement of agonist and antagonist muscles in voluntary submaximal isometric contractions. J Physiol Anthropol. 2008 Jan;27(1):33–42. doi: 10.2114/jpa2.27.33 18239348
20. Repnik E, Puh U, Goljar N, Munih M, Mihelj M. Using Inertial Measurement Units and Electromyography to Quantify Movement during Action Research Arm Test Execution. Sensors. 2018 Aug 22;18(9):2767
21. Orizio C, Perini R, Veicsteinas A. Muscular sound and force relationship during isometric contraction in man. Eur J Appl Physiol Occup Physiol. 1989;58(5):528–33. doi: 10.1007/bf02330708 2759079
22. Fulcher BD, Jones NS. hctsa: A Computational Framework for Automated Time-Series Phenotyping Using Massive Feature Extraction. Cell Syst. 2017 22;5(5):527–531.e3. doi: 10.1016/j.cels.2017.10.001 29102608
23. Liu L, Wang S, Hu B, Qiong Q, Wen J, Rosenblum DS. Learning structures of interval-based Bayesian networks in probabilistic generative model for human complex activity recognition. Pattern Recognition. 2018 Sep;81:545–61.
24. Estevez PA, Tesmer M, Perez CA, Zurada JM. Normalized Mutual Information Feature Selection. IEEE Trans Neural Netw. 2009 Feb;20(2):189–201. doi: 10.1109/TNN.2008.2005601 19150792
25. Liu X, Vega K, Maes P, Paradiso JA. Wearability Factors for Skin Interfaces. In: Proceedings of the 7th Augmented Human International Conference 2016 on—AH ‘16. Geneva, Switzerland: ACM Press; 2016 p. 1–8.
26. Liu Y, Norton JJS, Qazi R, Zou Z, Ammann KR, Liu H, et al. Epidermal mechano-acoustic sensing electronics for cardiovascular diagnostics and human-machine interfaces. Sci Adv. 2016 Nov;2(11):e1601185. doi: 10.1126/sciadv.1601185 28138529
Článek vyšel v časopise
PLOS One
2020 Číslo 1
- Proč se v Česku šíří několik „vymýcených“ infekcí najednou? Ptáme se odborníků
- ADHD jako evoluční výhoda? Výzkum se zaměřil na lovce a sběrače
- Není migréna jako migréna: Co pociťují ženy a co muži?
- Metamizol jako analgetikum první volby: kdy, pro koho, jak a proč?
- Není statin jako statin aneb praktický přehled rozdílů jednotlivých molekul
Nejčtenější v tomto čísle
- Severity of misophonia symptoms is associated with worse cognitive control when exposed to misophonia trigger sounds
- Chemical analysis of snus products from the United States and northern Europe
- Calcium dobesilate reduces VEGF signaling by interfering with heparan sulfate binding site and protects from vascular complications in diabetic mice
- Effect of Lactobacillus acidophilus D2/CSL (CECT 4529) supplementation in drinking water on chicken crop and caeca microbiome
Zvyšte si kvalifikaci online z pohodlí domova
Všechny kurzy