Introduction

Flexible and wearable human–machine interface (HMI) system plays an important role in intelligent identification, interactive controls for robots, and augmented/virtual reality [1]. To achieve effective, intuitive, and seamless manipulation of high-performance wearable HMI systems, significant efforts have been made recently, including the development of stretchable electronics [1, 2], self-powered motion sensors [3, 4], and advanced fiber based electronic devices [5, 6] via capacitive [7, 8], resistive [9, 10], piezoelectric [11, 12], triboelectric [13, 14] and optical [15, 16] effects. Among them, conductive textile-based HMIs have sprung up from proof-of-concept devices [17] to commercialized products (e.g., google garments) due to their desirable lightweight, flexible, and conformable characteristics. Note that the current strategies for manufacturing smart textiles are weaving functional fibers with sensing properties into textiles [18,19,20] or depositing electric sensing materials on the fabric surface [21,22,23,24]. In these cases, the design of advanced functional fibers with deformability, sewability, and washability represent great challenges in electrical safety and immunity to electromagnetic interference.

An alternative to electronic HMIs is detection of pressure or strain by fiber-optic sensors, particularly, optical micro/nanofiber (MNF) based sensors. Optical MNF sensors offer attractive advantages compared with their electronic or conventional fiber-optic counterparts, including electromagnetic interference-free detection of high-frequency stimuli, multichannel sensing with negligible crosstalk, and small size [25,26,27,28,29]. Owing to their atomically-precise geometric uniformities, the tensile strength of MNF can be greater than 5 Gpa [30], the bending radii can be less than 1 µm [31]. The excellent mechanical properties of MNF are attractive for assembling high performance tactile sensors. Recently, Li et al. developed a photonic sensor based on a hybrid plasmonic microfiber knot resonator, which can detect wrist pulse, respiration, and finger pulse in high magnetic fields [25]. Zhu et al. utilized a self-assembled wavy microfiber to realize a stretchable and ultrathin optical sensor for the blood pressure monitoring [29]. Recently, we demonstrated a skin-like wearable optical sensor embedded with perpendicularly intersected 2 × 2 MNF arrays. When pressure is applied on the net-joint areas, logic readouts can be obtained [26]. However, the sensor is insensitive to slippage along the MNF, a common action in HMIs, owing to the smooth polydimethylsiloxane (PDMS) surface and uniform MNF diameter. As a practical HMI, the sensing area should be large enough for human–machine interaction, thus, it is necessary to embed multi MNFs with uniform diameter into a large-area PDMS film. Moreover, the PDMS film needs a flexible substrate for robust and feasible manipulation.

Herein, we developed an optical MNF array enabled smart textile for HMI applications by integrating a large-area PDMS film embedded with long-waist MNFs with a piece of textiles. Benefiting from the warp and weft structure, the textile can transduce slippage into vibration, thus, slippage, static pressure, and dynamic pressure were effectively detected with high sensitivity, fast response, and wide pressure detection range. As a proof-of-concept smart textile, when attached to a sweater, control of the movement of a robotic hand was demonstrated. Furthermore, supervised machine learning is utilized to classify the touch manners via analyzing the applied pressure of each MNF. When the smart textile was attached onto a doll, named “Xiaozhi”, the virtual “Xiaozhi” can express different emotional expressions based on the touch manners felt by the realized “Xiaozhi”. We believe the optical smart textiles (OST) pave the way to potential applications in wearable human–machine interaction.

Experimental Section

Fabrication of the MNF

To fabricate the optical MNF with long uniform waist, a modified flame brushing method [32] was used to draw a standard optical fiber (SMF-28, Corning). During the stretching process, a 520 nm laser was coupled into the fiber to excite high-order modes. By monitoring the cutoff of high-order modes and controlling the drawing velocity and heating conditions [33], the diameter and length of the tapered waist were finely controlled at 800 nm and 6 cm respectively.

The diameter uniformity of the micro/nano fiber was investigated by a scanning electron microscope and the maximum diameter fluctuation is about 6 nm [32].

Fabrication of OST

To fabricate OST, the four-step process was developed. Step1: The PDMS basal layer with 0.5 mm thickness was prepared on a supporting glass slide (size of 10 cm × 10 cm) by depositing and curing the degassed PDMS (polymer base: curing agent = 10:1; Young's modulus of 750 kPa; Dow's Sylgard 184) at 80 ℃ for 30 min. Step2: As-fabricated MNFs were placed parallel to each other (1 cm apart) on the PDMS substrate under an optical microscope. The MNFs can be firmly fixed onto flat PDMS surface presumably due to van der Waals interactions. Step3: The MNFs on the PDMS film were enclosed by PDMS interlayer layer, followed by curing at 60 °C for 10 min to form a coating with a thickness of 0.1 mm, then place the textiles on the PDMS interlayer layer, curing at 80 °C for 30 min to bond them. Step4: To employ OST for sensing, it was peeled off from one side the glass slide.

Measurement

To monitor the transmission spectra of OST at different pressure, the broadband light generated from a tungsten-halogen lamp (SLS201L, Thorlabs) was coupled to the optical fiber of OST and the transmitted light was directed into a spectrometer (Maya2000 Pro, Ocean Optics). For pressure and slip sensing experiment, OST was mounted on a motorized force tester (ESM303, Mark-10) equipped with a digital force gauge (Series 5, Mark-10), and then static or dynamic normal force was applied to the sensor using a flat-head probe (D = 2 mm) connected to the force gauge as shown in Fig. 2a. A microscope (Eclipse 90i, Nikon, Japan) equipped with a CCD camera (DS-Fi1, Nikon, Japan) was used to capture the optical micrographs, and a scanning electronic microscope (Ultra 55, Zeiss) was used to characterize the MNF. For OST characterization device more portable, we employed an LED-CCD system for multichannel sensing. The light from an LED (central wavelength of 750 nm) was simultaneously coupled into input fibers of OST, and the grayscale image of the outputs as shown in Fig. S4 was monitored by a CCD camera (BFLY-U3-03S2M, Point Grey, Canada).

Results and Discussion

Design of Optical Micro/Nano Fibers Enabled Smart Textiles

To achieve a smart textile for human–machine interaction, this human–machine interface (Fig. 1a) mimics the biological response process of the human being. First, the MNF tactile sensor receives the external stimuli from the textile layer. Then, the stimuli will be converted to the optical signal via the bending effect of the MNFs. Third, the signal is transferred to the peripheral, such as a computer, after signal processing. By reading the signal, the peripheral with the configuration of a logical algorithm will recognize the positions, magnitude, and modes of the stimuli in real time. Basically, the optical MNF tactile sensor was designed with four layers, namely, textile layer, insulative PDMS intermediate layer, MNF sensory layer, and insulative PDMS interlayer basal layer (inset of Fig. 1a). The MNF sensory layer consist of five MNFs (diameter: 800 nm, waist length: 6 cm, distance between adjacent MNFs: 1 cm), comparable to its biological counterpart (nerve fibers). The textile layer was a piece of denim (thickness 100 µm) with the warp and weft structure, which can transduce finger slippage into periodic vibration of the MNF. As a result, the MNF sensor can detect slippage, static pressure, and dynamic pressure with high sensitivity, fast response, and wide pressure detection range. The PDMS interlayer layer plays a role to bond MNF sensory layer and the textile layer together. As shown in Fig. 1b and c, the as-fabricated MNFs show high flexibility (bending radius: ~ 10 µm) and uniform long taper waist (up to 10 cm), which is necessary for preparing large area HMIs. Enabled by the transition from guided modes into radiation modes of the waveguiding MNFs upon external stimuli, the MNF array offers high sensitivity in detection of static and dynamic force, which is critical for HMI devices. Note that the as-fabricated MNFs, with large fractional evanescent fields exposed to open air are highly susceptible to environmental disturbance (e.g., direct physical pressure or dust adsorption) or fracture, which may lead to unpredictable variations of sensing signals. Herein, we employed PDMS, a highly flexible and biocompatible polymer with refractive index (n = 1.40) slightly lower than that of silica (n = 1.46), to embed the MNF and isolate the evanescent fields. Typically, the evanescent fields of the MNF extends several microns, thus, the PDMS film (100 µm in thickness) is thick enough to contain the evanescent fields. In addition, the package of PDMS offers additional flexibility for stretching or twisting (Fig. 1d–f). Meanwhile, the crosstalk among the MNFs can be neglected because the penetration distance of the evanescent field, defined as the length where the evanescent field intensity decays to 10% of the highest intensity outside the MNF [34], is much smaller than the interval between two adjacent MNFs (1 cm). Figure 1g shows a bending optical MNF enabled smart textile guiding 633-nm-wavelength laser for visualization, indicating an excellent flexibility.

Fig. 1
figure 1

Schematic and characterizations of OST. a Schematic of the entire HMI system, inset shows the structure of OST. b SEM images of the as-fabricated MNF at ~ 10 µm bending radius. c Optical microscopy image of the as-fabricated MNF wrapped around three fiber tapers. d-f Photograph images of OST stretched, twisted by hand. g Photograph image of OST guiding 633-nm-wavelength laser

Full size image

Characterization of OST

To measure the responses of OST to both static and dynamic mechanical pressures, we used a home-made system containing a computer controller, a translation stage, an optical signal receiver and a force-feedback loading probe (Fig. 2a). Such a system can provide an external pressure of up to 80 kPa with both optical and electrical signals simultaneously recorded. Figure 2b shows typical broadband response to static pressures which was stepwisely increased from 0 to 4 kPa. This range is comparable to the typical pressure expected for touch screen or touch threshold of a finger (∼ 1 kPa) [35]. With increasing pressure, the output intensity of the MNF decreases obviously. Benefitting from the broadband wavelength-dependent response, the sensor offers an opportunity for tuning the sensitivity and detection range in the same sensor using different wavelengths (Fig. S1a). As shown in Fig. S1b, the fractional power of the evanescent fields outside the MNF become higher with the increasing of wavelength, resulting in a higher sensitivity in the longer wavelength range. Thus, the output intensity in the longer wavelength range decreases more rapidly as the applied pressure increases, leading to a blue shift of the peak wavelength in Fig. 2b. The sensitivity of the optical MNF tactile sensor is defined as S = (∆I/Io)/ ∆P, where ∆I is the relative change in output intensity, Io is output intensity of the sensor under no load and ∆P is the change in applied pressure. For example, when the sensor was operated at 750-nm-wavelength, an approximately linear relationship between ∆I/Io and applied pressure P in the range of 0–1.5 kPa, leading to a sensitivity value S = 65.5 kPa−1. To further improve the detection limit, we can reduce the thickness of the PDMS, employ MNFs with a smaller diameter and use a light source with a longer central wavelength.

Fig. 2
figure 2

Pressure detection performances of OST. a Setup for the pressure detection experiment. b Transmission spectra of OST measured at different pressures (0–4 kPa). c Sensitivity of OST at three thicknesses of PDMS intermediate layer (100, 200, 300 μm, respectively). d Response of OST to finger actions of tapping, non-contact movement and sliding successively. e Response of OST to force-feedback loading probe sliding movement at different velocities (1.17, 2.13, 3.35 mm/s). f Durability test of OST by applying a pressure at 2 Hz of 1.1 kPa

Full size image

In addition to the wavelength, tuning the thickness of the PDMS film can be used for different pressure sensitivities, whereby thicker PDMS films can be used toward high pressure regimes, while thinner PDMS films are applied for low pressure regimes. As shown in Fig. 2c, with the decrease of the thickness of PDMS interlayer from 300 to 100 µm, the sensitivity increases from 14.9 to 65.5 kPa−1 at 750-nm-wavelength.

To investigate the sensor’s response to finger actions, three types of stimuli, i.e. tapping, non-contact movement and sliding were conducted. It is clear that “tapping” leads to obvious decrease of the transmittance. Moreover, the response time at 90% of range was represented by t90 (25 ms, Fig. S2), which is crucial for real time human machine interaction, whereas “sliding” results, in a series of dips, corresponding to the warp and weft alignments of the textile. As expected, “non-contact movement” does not result in any change in transmittance. The result confirms that the human body, a grounded conducting medium, has no impacts on the sensor. To examine the dynamic pressure sensing capability of our sensors, the output intensity signals were compared with the dynamic pressure inputs at a frequency range of 1–3 Hz (Fig. 2e), corresponding to sliding speeds of 11.17, 2.13, and 3.35 mm/s, respectively. We further demonstrate high durability of our MNF pressure sensor under a pressure of 1.1 kPa at a frequency of 2 Hz (Fig. 2f and Fig. S3), considering that the typical pressure expected for touch screen or touch threshold of a finger (∼1 kPa) [35]. Note that the high signal-to-noise ratios were well maintained and the intensity amplitude exhibited negligible changes after more than 30,000 loading–unloading cycles. Owing to the abilities of simultaneously sensing dynamic and static pressures without electromagnetic interference, together with broad force range and excellent slip detection sensitivity of OST, the optical smart textiles can satisfy the needs for human machine interaction.

Logic Control of a Robotic Hand

To demonstrate the feasibility of the optical MNF enabled smart textile for human machine interaction, a prototype smart textile composed of a 5-MNF-array was used to control a five-finger STM32 based robotic hand (Fig. 3a). In this case, the movement of each finger was controlled by an individual MNF. In order to simplify the optical signal acquisition, we developed a compact LED-CCD device (see supporting information, Fig. S4) as light source and detector, respectively. As shown in Fig. 3b, all the MNFs share one LED (0.5 mW) and one CCD camera, and the intensity of the input light for each MNF is about 1 µW, offering opportunities for low-power operation. Moreover, the interval between the adjacent MNFs was set to 1 cm, and the photograph of each fiber can be well defined, resulting in a negligible cross talk among the MNFs. As expected, the optical MNF enabled tactile sensor was so sensitive that a slight finger touch can result in an obvious decrease in the output intensity. For example, when a finger touches MNF-3, the corresponding endface image of MNF-3 becomes dark with a strong contrast to the others. Based on the change of the output signals (gray value of the endface images), we processed the signals and controlled the robotic hand using the Robot Operating System (ROS). As shown in Fig. 3c–e, the robotic hand can make different hand gestures, which include bending single finger or more complex gesture such as OK, yeah, make a fist and so on (movie S1). The corresponding relationship between robotic hand’s gestures and received binary data are all shown in Table S1. As a wearable human machine interface, the optical MNF enable smart textile can be attached to cloth for working or detached from cloth during washing, offering an alternative manner to weaved-fibers based sensors.

Fig. 3
figure 3

Logic control of a robotic hand. a Schematic of the human–machine logic control process by OST. b Schematic diagram of the human–machine logic control system. c-e Control robotic hand to make ROCK, OK, and FISTED gestures

Full size image

Machine Learning Enabled Emotional Human Machine Interface

In addition to the response of the static press stimuli, OST can detect dynamic finger slide, the time domain of one touch activity may contain more hidden information, including contact sequence, texture induced vibration, touching speed, etc., offering a promising manner for complex human machine interaction. By taking advantages of machine learning, we achieved high classification accuracy using support vector machine (SVM) classifier, which has been proven as effective tool for automatically extracting the features from the time domain data [36, 37]. To demonstrate the capability of OST for time domain-based interaction, it was mounted and connected onto the back of head of a doll name “Xiaozhi”, which is the mascot of Zhejiang Lab (Fig. 4a). Each MNF stands for a channel for tactile data collection. Herein, we defined 8 kinds of emotional expressions images (including smile, laugh, sadness, cry, worry, anger, hello, goodbye) corresponding to 8 kinds of time-domain signals felt by the smart textile (shown as movie S2). Upon applying a weak touch on OST, the virtual “Xiaozhi” on the computer can show different emotional expressions corresponding to the touch manners detected by the smart textile. For example, when one gently taps the “Xiaozhi”, each MNF can feel the touch almost at the same time, as a result, the output intensity of each MNF gives an obviously decrease (Fig. 4b). When the one’s finger slides across the MNFs, the five MNFs will give responses sequentially. By analyzing the time-domain signals, the top-down or bottom-up sliding can be discriminated (Fig. 4c, d). While the finger sliding is long the MNFs, one or two MNF will give a periodic response corresponding to the warp/weft structure of the textile (Fig. 4e). As a result, the virtual “Xiaozhi” showed “hello” “sad”, “smile”, “annoy” emotional expressions, accordingly.

Fig. 4
figure 4

Emotional HMI via machine learning classifier. a Schematic diagram and the process of emotional virtual interaction. b–e Emotional expressions corresponding to different time-domain signals. f–h Confusion matrix of OST with 5, 4, 3 MNFs, corresponding to accuracy of 98.1%, 95.4%, 85.4% respectively

Full size image

In this work, each label was assigned 200 sets of data as a database, and the impact of MNF number on the recognition accuracy were tested. On the basic of traditional two-class of SVM classifiers, multi-class SVM classifiers were adopted by one-to-one principle, and Leave-One-Out-Cross-Validation (LOOCV) was introduced for the accuracy test of the multi-class SVM classifier. Under the premise of using multi-class SVM model, the recognition accuracy of the FSOP composed of five MNFs for different stimuli was as high as 98.1% (Fig. 4f). Our accuracy of 98.1% is much better than some reported works, e.g., 95.2% accuracy with 220 samples [38]; 94.5% accuracy with 1400 samples [39]; 96.8% accuracy with 700 samples [40], and we believe this accuracy can meet the basic requirements for HMI. When the MNF number was decreased from five to three, the recognition accuracy was down to 85.4% (Fig. 4h). To further improve the accuracy, one can increases the amount of data in the database or use more MNFs.

Conclusions

In summary, a highly sensitive and reliable tactile sensor was developed via a low-cost, facile, and scalable method. The sensor achieves the measurement of static/dynamic force with short response time (25 ms), high durability (> 30,000 cycles), and adjustable detection range. Benefiting from the structure of the textile and high sensitivity of MNF sensor, OST can feel slight sliding and pressing of finger. By sewing OST and daily clothes together, the logic control of the robotic hand could be successfully implemented. More interestingly, under the premise of using multi-class SVM model, emotions corresponding to real gestures in virtual scenes can be accurately expressed (accuracy > 98%). To extend the sensing area of the optical smart textile, we can use a longer MNF or weave multiple MNFs inside a single OST for spatially resolved 2-dimensional tactile sensing and human–machine interaction. We expected that this approach would open a new window in designing smart textiles devices for applications in HMIs and IoT such as industrial automation and virtual reality interaction technique.