Keywords

1 Introduction

The hands are used to perform a wide range of daily activities. In this regard, upper limb amputations constitute traumatic experiences that significantly impair the quality of life of amputees [1, 2]. Prosthetic devices have been developed for restoring functions that were lost due to limb loss with advanced mechatronic design and control algorithms. The more advanced types of prosthetic hands make use of electromyographic signals (EMG) to decode commands from activation of remaining muscles [3, 4]. However, such devices are still lacking a fundamental aspect of biological limbs which is the sense of touch.

The sense of touch is directly involved in the control of the body, from posture to the manipulation of objects held with the hands [5]. During manipulation, the adjustment of grip force escalates with object weight and the friction of the surface in an optimal manner where the applied force is slightly higher than the required to prevent objects from slipping [6]. However, the lack of tactile feedback poses a challenge for amputees to properly control their prosthetic limbs. In that sense, objects might be damaged or slip without awareness due to inappropriate grip force [7].

Several tactile sensors have been proposed in the literature with the purpose of improving the sensing capabilities of prosthetic hands. Such sensors are made of different types of materials such as piezoresistive, piezocapacitive, optical and acoustic [8, 9]. Most of these sensors make use of a rigid substrate which is not ideal given that the robotic fingers are in constant motion and interaction with objects [10]. Therefore, an increasing effort has been devoted to the development of flexible tactile sensors given the advances in the field of flexible electronics [11].

In this paper, we present the design of a novel neuromorphic soft optical tactile sensor. The sensor combines a very thin, flexible photodiode array and light-emitting diodes (LEDs) encapsulated within an elastomer that behaves as a soft artificial skin. Forces applied over the skin cause deformations that are characterized by the different patterns of light intensity captured by the photodiodes. The output of the tactile sensor is converted into spikes that are akin to the behavior of Merkel Cells (SA-I), one of the four main types of mechanoreceptors located in the skin that encodes static forces applied over the skin.

2 Methods

The proposed tactile sensor is composed by (I): the sensing part which is composed by the soft skin and the optical elements; (II) the readout circuit that amplifies and filter external noise; (III) a data acquisition system that is microcontrolled and handles the analog-to-digital conversion process and transfers data to a PC via USB and (IV) a computational neuron model to convert the tactile signals into spikes following a neuromorphic approach.

  • A. Tactile Sensor Design

The optical tactile sensor was built using two layers: a 3x2 photodiode array located at the base of the soft artificial skin for a total of 6 sensing elements or taxels [12]; an elastomer (Ecoflex 00-30) composed the top layer with two infra-red (IR) LEDs (750 nm wavelength) embedded inside the skin, directed towards the photodiode array. The sensor was manufactured in a square shape; thus, we adapted the design of the sensor so that the components would fit this shape accordingly (Fig. 1).

Fig. 1
figure 1

A conceptual design of the proposed tactile sensor. The sensor is composed by two layers: The top layer is formed by a soft elastomer material with two IR LEDs embedded into the skin. The bottom layer is composed by the 3x2 photodiode array. Forces applied over the soft skin change the direction and intensity of light captured by the photodiodes. a IR LEDs are embedded into the soft skin. b Soft elastomer material that composed the soft skin. c The 3x2 photodiode array

Full size image

External forces applied on top of the soft skin caused a deformation that promoted changes into the pattern of light detected by the photodiode array.

  • B. Signal Conditioning

The light captured by each photodiode is converted into electrical current and passed to a signal conditioning block. Each signal conditioner is composed by a transimpedance amplifier (OPA381, Texas Instruments, USA) that converts the current into voltage signals. The output of each amplifier is passed to a 3rd order Butterworth filter with a cut-off frequency of 100 Hz. This value of the cut-off frequency was chosen considering that, during the manipulation of objects, there are events that occur very rapidly, such as object slippage. Therefore, in order to detect this type of event, it is necessary to have a higher cut-off frequency.

  • C. Data Acquisition

Data acquisition was controlled by a Cortex-M3 ARM microcontroller (STM32F103C8T6, STMicroelectronics, Switzerland). Each tactile signal was converted by an individual analog-to-digital (ADC) channel. Data was sampled at 1 kHz and sent to a Desktop PC via USB communication (Fig. 2). We have opted to use a 1kHZ sampling frequency due to the need for Izhikevich neuron model to work with a higher rate, in order to avoid numerical instability. A custom-made Python 3 script was developed to read the data from the sensor in real-time.

Fig. 2
figure 2

The output current from each photodiode is passed to a readout circuit. The signal conditioners are composed by a transimpedance amplifier and 3rd order low-pass Butterworth filter. Each signal is then passed to individual ADC channels of the microcontroller. Digitized data is combined into packages and sent via USB to a PC application in real-time

Full size image

  • D. Neuromorphic Model of SA-I Afferents

The Izhikevich neuron model was used to convert the tactile signals into spikes [12]. This model is inspired by the biophysically detailed Hodgkin-Huxley model of the membrane potential of single neurons and the computational principles of integrate-and-fire neuron models. The Izhikevich model is computationally efficient but preserves enough biological complexity to mimic different types of neurons and their firing properties [13].

The differential equations that describe the Izhikevich neuron model can be found in Eqs. 1 and 2 [13].

$$ \frac{dv}{{dt}} = 0.04v^{2} + 5v + 140 - u + I $$
(1)
$$ \frac{du}{{dt}} = a\left( {bv - u} \right) $$
(2)

where v describes the membrane potential of the neuron and \({\text{u}}\) is the membrane recovery variable. The membrane recovery accounts for the activation of Potassium (K) currents and inactivation of Sodium (Na) currents, providing a negative feedback to v [12]. The input current to the neuron is given by \({\text{I}}\). When the membrane potential increases and a threshold is crossed \({\text{v}} \ge 30\) mV, a spike is fired, and the variables properly reset according to Eq. 3.

$$ if\,v \ge 30\,mV,then \left\{ { \begin{array}{*{20}c} {v \leftarrow c} \\ {u \leftarrow u + d} \\ \end{array} } \right. $$
(3)

The parameters a and b describe the time scale and sensitivity to noise. Lower values of a will lead to slower recovery of the membrane potential. The after-spike potential is given by c and d also determines the dynamics of the recovery variable. In this paper we have set \(a = 0.02\), \(b = 0.2\), \(c = - 65\) and \(d = 2\) to mimic the behavior of SA-I afferents [13].

3 Results

The schematics of the readout circuit and the design of the Printed Circuit Board (PCB) have been completed and undergoing fabrication (Fig. 3). In order to test the behavior of a single signal conditioner, a test board was assembled employing the transimpedance amplifier and the microcontroller (Fig. 4).

Fig. 3
figure 3

3D view of the PCB board designed to read the proposed neuromorphic tactile sensor

Full size image
Fig. 4
figure 4

Preliminary circuit built to read individual taxels

Full size image

A proof of concept experiment was conducted to verify the behavior of the Izhikevich model used generate SA-I spikes. In this experiment, we performed a mechanical compression test where the soft skin was indented with different depths (0.5, 1.0, 2.0, 3.0 and 4.0 mm) during one second each in a controlled manner. For each depth, the output voltage of the taxels was measured to verify the relationship between voltage and depth (Fig. 5). Afterwards, the output signal was applied to an Izhikevich neuron and the generated spikes were stored into a data matrix and converted into a rastergram (Fig. 6). The rastergram illustrates the relationship between indentation depth and the firing rate of the SA-I model. Larger deformations of the skin were coded by generating more spikes in the same time interval, i.e. with higher firing rate.

Fig. 5
figure 5

Output voltage of a single taxel for different indentation depths. The obtained results demonstrated that voltage increased with indentation depth with a linear relationship. Changes in 1 mm were detected and could be encoded in the form of spikes with the computational Izhikevich neuron model

Full size image
Fig. 6
figure 6

Rastergram demonstrating the spikes generated by the computational SA-I model over different indentation depths for a single taxel. As deformation of the soft skin increased, more spikes were generated by the model following the behavior of biological SA-I afferents

Full size image

4 Discussion

In this paper, we presented the design of a novel neuromorphic optical tactile sensor for prosthetic hands. The sensor is composed by a soft elastomer and flexible components that mimic the human skin. The output signals were converted into spikes following a neuromorphic approach by applying the Izhikevich neuron model.

The PCB board is under fabrication and complete characterization of the tactile sensor will be conducted. However, we have assembled a prototype version and conducted a proof of concept experiment to study the tactile signals produced by a single taxel. In this experiment, an indenter was pressed against the soft skin with controlled depths. The deformation of the skin produced distinct voltage levels that were used as input to the SA-I neuron model. The model coded higher deformations with higher firing rate as observed in their biological counterparts, demonstrating that it was possible to make use of the Izhikevich model to emulate their behavior [14].

The neuromorphic approach we employed was inspired by previous works on the field that presented applications of neuromorphic touch for texture classification [15, 16] and edge orientation discrimination [17]. In our future experiments we aim at demonstrating the usefulness of our sensor for applications such as the ones presented in those papers but also during manipulation of objects.

Tactile sensors can be especially useful in prosthetic hands promoting low-level control signals that are independent of the user intention [7, 18]. In that sense, it would be possible to alleviate the cognitive load of the amputees, while also promoting more robust control and higher acceptance of such devices. Therefore, it is of special interest to develop sensors that can be easily incorporated into the fingers of commercial prosthetic hands without interfering in their design. This is the goal of the work presented in this paper which is focused on a soft and flexible tactile sensor.

5 Conclusions

In this paper we presented the design of a novel neuromorphic optical tactile sensor that is soft and flexible. This sensor encoded deformation of the artificial skin caused by external forces in the form of spikes akin to their biological counterparts. This approach can be useful for improving the sensing capabilities of prosthetic hands, providing more robust control during manipulation of objects.