SensIR: Detecting Hand Gestures with a Wearable Bracelet using Infrared Tomography

SensIR: Detecting Hand Gestures with a Wearable Bracelet using Infrared Tomography

– I’m Jess McIntosh, I’m from
the University of Bristol, and I’m here today to
present my work SensIR, which is detecting hand gestures
with a wearable bracelet using infrared transmission
and reflection. And this work is done in
collaboration with my colleague, Asier Marzo and my
supervisor, Mike Fraser. So let me begin with a brief introduction and set the motivation for this work. So we can use hand gestures
explicitly to conveniently control wearable devices,
as seen in previous work, and during this session with Pyro. So we can use this to control devices in our surroundings too. But we can also detect our
actions that we do implicitly, and this allows us to
monitor daily activities, or keys for contextual
awareness, or bad habits. But for either use case, it’s clear that there are mobile applications. And therefore, the sensing
apparatus must also be portable. So smart watches are
already seen as a socially acceptable placement
for a wearable, and the watch strap conveniently
allows space for our sensors. And so for these reasons,
a lot of work has gone into placing the sensors here
to detect the hand pose. So there’s a variety of
different sensing methods that’s being used to achieve this, and there is associated properties that determine the method’s
strengths and weaknesses. But in this work, we don’t
compare the sensing methods. Instead we simply try
to improve the accuracy of the existing infrared sensing method, without sacrificing size,
efficiency, or practicality. So there’s been lots of
previous work on wrist-worn gesture detectors using
infrared, but the basic principle of operation is the same for all of these. And indeed for other techniques too. So this principle relies
on the deformation of the shape of the wrist, as
gestures are performed, due to the moving anatomical parts within. So let’s look at the previous
work in a bit more detail. So here’s a cross-section of
a wrist, and this might be a typical bracelet with the
infrared sensors embedded. And there will be multiple pairs of emitters and receivers of infrared light. So these each sense the
distance to the skin, and thereby sensing the deformation
of the shape of the wrist. It is important to note
here that each pair works individually in this configuration. So I will refer to this from now on as a one-to-one emitter to
receiver configuration. But we know that infrared
light of particular wavelengths can actually pass through flesh diffusely. So inspired by this, we made
small changes to the previous configuration, where instead
now, every infrared sensor measures the light level
when any one LED is emitting. So now in addition to
what we had previously, the adjacent sensors also pick
up reflections from the skin, and we also get some amount
of light being transmitted through, which infers
distance, and possibly captures some of the morphological changes that happen within, such as bends moving. So the device can
multiplex through each LED to collect a matrix of brightness values. So we have a ray for each emitter, and each column representing
the receiving elements. So then we feed the raw
data values from this matrix into a neural network, in
order to classify gestures. So here’s what our prototype looks like. There are 14 segments in total. And each segment has an
LED and a photodiode, with a peak sensitivity of 860 nanometers. And this is all connected
to a signal generation and acquisition circuit, which
then relays the information back to a PC for classification. So to test this configuration,
we conducted a user study, which the participants found
incredibly interesting. So we chose a variety of gestures, including pinchers,
multi-finger, and wrist. And each of their gestures
were performed 10 times. So after collecting the
data from the study, we wanted to compare two things. So we first need to compare
the predictive accuracies of the system, when using the
traditional one-to-one system against our new one-to-many system. And secondly to see how the
number and arrangement of the different segments might
affect the predictive accuracy. So here are our results. So it’s clear that using
all the features from the one-to-many system is better than using only single reflective measurements, and significantly so for the first three. It is worth noting that
in fact the smart watch configuration did not suffer
from a major decrease in performance, nor did actually
halving the elements here. But actually four segments did
suffer quite a lot, but that could be used for gesture
onset detection for instance. So in addition to this study, we did some additional preliminary
studies to investigate the effects of cross session misalignment, non-sedentary postures, and skin coupling. So sensor misalignment
occurs when the device is taken off and back on between sessions. So there is usually some
misalignment between the sensors, and this can lead to a
classification error, due to a shift in the features. So detecting this misalignment
is the first step towards calibrating the device
to account for the shift. So we tried using a neural
network regressor on the data from the neutral hand
pose, and we found that we could in fact detect
the shift reasonably well. So the second study that we tried to do was a non-sedentary study. So during the main study,
the users were sitting down in a fixed position and instructed
to keep their arm still. But people are likely to
be moving around and moving their arms in different
positions when in real use. So we tested the system
with three different arm elevations and three
different arm rotations. And we found that the accuracy
still remains quite good in both cases, but as expected
the rotational changes to the forearm makes
classification more difficult. And finally, there may be
some circumstances when the device is maybe not tight
around the wrist in some places, and the sensors aren’t
quite touching the skin. So we conducted some
simple tests, purposefully introducing gaps and
even putting some latex between the band and the skin. And we found that the accuracy
was largely unaffected still. So to summarize, we’ve
demonstrated that a one-to-many system is superior to the
previous one-to-one systems. And this only requires
a minor modification to actually achieve this. And finally, some
preliminary work suggests that calibration could work quite well, and the technique is fairly
robust to natural movement. Thank you very much for listening. I’ll take any questions you have. (audience clapping) – [Audience Member] Hi, my
name is Kon Park from KAIST. I think for your presentation, I have a question about the
lights through the wrist. You said two kinds of lights pass. First one is reflected
by skin, and second one is a light that goes through the wrist. My question is that
how much does the light through the skin affects
to the final visual? Is it really important to measure those lights that goes through the skin? – So you’re asking me about the dynamic range of it or something? – [Audience Member] No, I mean
is it important to measure, I mean for example, if the
light is emitting this part, and you are also measuring the
sensor from this part right? Is it important to measure those far… – Yeah it does depend on the distance. In some cases, some users
wrists were quite large, and therefore the light doesn’t transmit all the way through. So often times it’s just
sometimes the ones around the edges, and not quite opposite,
but sort of four o’clock, five o’clock position can be detected. – [Audience Member] So
you’re saying that major part of the, I mean is it important to measure the adjacent sensors, right? – Sorry? – [Audience Member] I mean you’re saying it is important to measure
values in adjacent sensors? – Yeah. – [Audience Member] Okay, thank you. – Any other questions? I have one for you. So to what extent is this
robust to environmental interference, like from other light? Like if I’m on a bright, sunny beach, and I have a lot of ambient
infrared and things? But it’s all kinda inside the wrist. So is it actually pretty
safe against all that? – So there is some light
leakage, but I think you can use techniques taken from other fields, such as the infrared brain imaging stuff. And they use techniques
where they can mitigate this kind of light leakage
by just subtracting the background noise from your readings. – Okay, great.

Be the first to comment

Leave a Reply

Your email address will not be published.