Unlocking hand magic: can AI understand your hand gestures?

New ways to control machines often lead to significant breakthroughs, much like how keyboards and touchscreens have revolutionized our lifestyle. A recent and promising idea is to explore using hand movements to control devices. However, the main issue with current technologies is that they often require cameras or numerous sensors, which can be inconvenient to wear. To overcome these limitations and allow easier and less cumbersome control of machines, a recent research conducted at the Smart Systems Lab proposes using machine learning algorithms to recognize hand movements by means of the electrical signals from muscles.

To understand how it is possible to use electrical signals from the body to identify hand motions, we must remember that the human body is made up of cells. These cells produce tiny electrical signals to communicate with each other. For example, such electrical signals allow the brain to control our muscles, tell our hearts to beat, and instruct our lungs to breathe. Therefore, by measuring these electrical signals, it should be possible to understand which motions are being performed. In the research carried out at the Smart Systems lab, the experimental setup consists of an EMG (electromyography) sensor recording four channels of electrical signals from the hand and forearm muscles, as well as an IMU (inertial measurement unit) recording acceleration and rotation of the hand.

In this study, we recorded hand and wrist movements from different individuals using special electrical and motion sensors on their forearms. We then used machine learning to analyze the obtained signals. By providing the type of motion to the AI, it could identify patterns and establish links between the signals and the specific movements performed by the participants. Once the AI training was complete, we developed various models that were able to accurately identify and predict hand motions.

The overall process can be summarized as follows: Firstly, a hand motion is performed, and sensors record signals. Features are then extracted from these signals. Machine learning processes are subsequently applied to these features to recognize patterns and make accurate predictions about the type of hand motion performed. This process can be easily visualized, as shown in the figure below:

In fact, the research found that using specific features from the measured signals makes it possible to accurately identify the type of hand motion. Even when only signals from the wrist were used, the AI still managed to provide predictions with high accuracy (up to 82%). It should be noted that similar hand movements involving similar muscle motions tend to be harder for the AI to differentiate. This indicates that by refining the approach, machine learning could potentially predict hand motions that it has not been trained on, which would be a major breakthrough. The conclusions of this research pave the way for exciting new methods to control machines and robots using just our hands.

This article briefly summarizes the Master’s thesis “Exploring the Potential of Machine Learning in Hand Motion Recognition Through Forearm EMG and IMU Data,” submitted in May 2024 by Lukas Zehner, supervised by Prof. Wallace Moreira Bessa and co-supervised by Salma Salimi. The research and experiments were conducted at the Smart Systems Lab of the Department of Mechanical and Materials Engineering, Faculty of Technology, University of Turku, in Turku, Finland.

Here is a link to the master’s thesis: [UTU PUB]

About the author:

Lukas Zehner is a doctoral researcher at the University of Turku in the Department of Mechanical and Materials Engineering. Originally from Belgium, he spent most of his life in Frankfurt, Germany. He obtained his Bachelor of Science in Mechatronics at the University of Technology of Darmstadt in 2021. After this, he moved to Finland where he got his master’s degree in Mechanical Engineering at the University of Turku in 2024. In his spare time, he is a dedicated marathon runner, passionate to travel and to discover the world. He is also an active part of student organisations, actively supporting the experience of international students in Turku.