Mapping a Continuous Vowel Space to Hand Gestures

Authors

  • Yadong Liu The University of British Columbia
  • Pramit Saha University of British Columbia
  • Arian Shamei University of British Columbia
  • Bryan Gick University of Britih Columbia
  • Sidney Fels University of British Columbia

Abstract

Converting hand gestures to speech sounds has been proved to be successful in Glove Talk II [Fels and Hinton, IEEE Transactions on Neural Networks, 8(5), (1997), 977]. This work mapped hand gestures to English speech sounds through an adaptive interface including gloves, space trackers and a foot-pedal. A set of hand gestures were designed and each gesture corresponded to one English segment, and apparently, users had difficulties in producing diphthongs with a natural transition. The present study aims to develop a more intuitive and compact, single-handed user interface which converts hand movements directly to a continuous formant space to generate English vowels through a formant based speech synthesizer. We have collected kinematic glove data of two participants using Cyberglove corresponding to wrist movements (up-down) and finger abduction (sideways) for 8 different English vowels as well as diphthongs. We employed a variety of deep neural networks, with varying hyperparameters, mapping the finger and wrist movements to the continuous vowel quadrilateral formant space (F1 and F2) and analysed the performance of these networks. Results demonstrated that our system achieved successful continuous mapping of one hand movements to the formant space, thereby generating English vowels accurately from a variety of hand gestures, and also showed the prospect of producing vowels of other languages.

Author Biography

Yadong Liu, The University of British Columbia

I am a Ph.D student at UBC linguistics

Additional Files

Published

2020-03-17

How to Cite

1.
Liu Y, Saha P, Shamei A, Gick B, Fels S. Mapping a Continuous Vowel Space to Hand Gestures. Canadian Acoustics [Internet]. 2020 Mar. 17 [cited 2024 Nov. 8];48(1). Available from: https://jcaa.caa-aca.ca/index.php/jcaa/article/view/3373

Issue

Vol. 48 No. 1 (2020)

Section

Proceedings of the Acoustics Week in Canada

License

Author Licensing Addendum

This Licensing Addendum ("Addendum") is entered into between the undersigned Author(s) and Canadian Acoustics journal published by the Canadian Acoustical Association (hereinafter referred to as the "Publisher"). The Author(s) and the Publisher agree as follows:

  1. Retained Rights: The Author(s) retain(s) the following rights:

    • The right to reproduce, distribute, and publicly display the Work on the Author's personal website or the website of the Author's institution.
    • The right to use the Work in the Author's teaching activities and presentations.
    • The right to include the Work in a compilation for the Author's personal use, not for sale.

  2. Grant of License: The Author(s) grant(s) to the Publisher a worldwide exclusive license to publish, reproduce, distribute, and display the Work in Canadian Acoustics and any other formats and media deemed appropriate by the Publisher.

  3. Attribution: The Publisher agrees to include proper attribution to the Author(s) in all publications and reproductions of the Work.

  4. No Conflict: This Addendum is intended to be in harmony with, and not in conflict with, the terms and conditions of the original agreement entered into between the Author(s) and the Publisher.

  5. Copyright Clause: Copyright on articles is held by the Author(s). The corresponding Author has the right to grant on behalf of all Authors and does grant on behalf of all Authors, a worldwide exclusive license to the Publisher and its licensees in perpetuity, in all forms, formats, and media (whether known now or created in the future), including but not limited to the rights to publish, reproduce, distribute, display, store, translate, create adaptations, reprints, include within collections, and create summaries, extracts, and/or abstracts of the Contribution.