Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Tactile sensing has been a component of robotics for roughly as long as vision. However, in comparison to vision, for which great strides have been made in terms of hardware and software and which is now widely used in industrial and mobile robot applications, tactile sensing always seems to be a few years away from widespread utility. Therefore, before reviewing the technologies and approaches available it is worthwhile to consider some basic questions:

  • How important is tactile sensing?

  • What is it useful for?

  • Why does it remain comparatively undeveloped?

In Nature, tactile sensing is an essential survival tool. Even the simplest creatures are endowed with large numbers of mechanoreceptors for exploring and responding to various stimuli. In humans, tactile sensing is indispensable for three distinct kinds of activities: manipulation, exploration, and response. The importance of tactile sensing for manipulation is most evident for fine motor tasks. When we are chilled, tasks like buttoning a shirt can become an exercise in frustration. The problem is primarily a lack of sensing; our muscles, snug in our coat sleeves, are only slightly affected but our cutaneous mechanoreceptors are anesthetized and we become clumsy. For exploration, we continually assimilate tactile information about materials and surface properties (e.g., hardness, thermal conductivity, friction, roughness) to help us identify objects. We may have difficulty distinguishing real leather from synthetic leather by sight, but not by touch. Finally, the importance of tactile response, whether to a gentle touch or an impact, is seen in the damage that patients with peripheral neuropathy (e.g., as a complication of diabetes) accidentally do to themselves.

As Fig. 19.1 indicates, the same functional categories apply to robots. However, in comparison to animals, with thousands of mechanoreceptors per square centimeter of skin, even the most sophisticated robots are impoverished. One reason for the slow development of tactile sensing technology as compared to vision is that there is no tactile analog to the charge-coupled device (CCD) or complementary metal–oxide–semiconductor (CMOS) optical array. Instead, tactile sensors elicit information through physical interaction. They must be incorporated into skin surfaces with compliance, to conform locally to surfaces, and with adequate friction to handle objects securely. The sensors and skin must also be robust enough to survive repeated impacts and abrasions. And unlike the image plane in a camera, tactile sensors must be distributed over the robot appendages, with particularly high concentrations in areas such as the fingertips. The wiring of tactile sensors is consequently another formidable challenge.

Fig. 19.1
figure 1_20

Uses of tactile sensing in robotics

Full size image

Nonetheless, considerable progress in tactile sensor design and deployment has been made over the last 20 years. In the following sections we review the main functional classes of tactile sensors and discuss their relative strengths and limitations. Looking ahead, new fabrication techniques offer the possibility of artificial skin materials with integrated sensors and local processing for interpreting sensor signals and communicating over a common bus to reduce wiring.

There is an extensive literature describing touch sensing research. Recent general reviews include [19.1] and [19.2], and these cite several older reviews from the 1980s and 1990s.

Sensor Types

This section outlines five main types of sensors: proprioceptive, kinematic, force, dynamic tactile, and array tactile sensors. A basic review of the first three of these is provided along with contact sensors that provide thermal or material composition data. However, greater emphasis is placed on tactile sensors that provide mechanoreception. Table 19.1 provides an overview of these tactile sensors. When considering tactile sensors, it is useful to begin by considering the fundamental physical quantities that can only be sensed through contact with the environment. The most important quantities measured with touch sensors are shape and force. Each of these may be measured either as an average quantity for some part of the robot or as a spatially resolved, distributed quantity across a contact area. In this chapter we follow the convention of studies of the human sense of touch and use the term touch sensing to refer to the combination of these two modes. Devices that measure an average or resultant quantity are sometimes referred to as internal or intrinsic sensors. The basis for these sensors is force sensing, which precedes the discussion of tactile array sensors.

Table 19.1 Tactile sensor modalities and common transduction types
Full size table

Proprioceptive and Proximity Sensing

Proprioceptive sensing refers to sensors that provide information about the net force or motion of an appendage, analogous to receptors that provide information in humans about tendon tensions or joint movements. Generally speaking the primary source for spatial proprioceptive information on a robot is provided by joint angle and force-torque sensors. Since joint angle sensors such as potentiometers, encoders, and resolvers are well-established technologies, they do not warrant discussion here. Instead, a brief review of proximity sensing via whiskers and antennae as well as noncontact proximity sensing is provided. Force-torque sensors are discussed in greater detail in Sect. 19.1.4.

Whisker and Antenna Sensors

Whisker or antenna sensors are in essence a hybrid of proprioceptive and tactile information. This form of sensing was first explored in the early 1990s, for example, Russell [19.25] developed a whisker sensor with a base angle sensor and tip contact sensor that was attached to a robot arm to explore its environment. Another example by Kaneko et al. [19.26] is one of the earliest examples of active antenna sensing. Kaneko et al. affixed a rigid spring steel antenna to a one-degree-of-freedom (1-DOF) rotating axis used to sweep the antenna from side to side similar to the method an insect would employ. The sweeping motion, in combination with a joint angle sensor and torque sensor, was used to assess encountered contacts. Clements and Rahn [19.27] took a similar approach to Kaneko, but added an extra degree of freedom to the sweeping pattern of their whisker. Clements and Rahn used a motor-driven gimbal to drive their spring steel whisker in two DOFs to explore objects. Cowan et al. [19.28] used a multisegmented piezoresistive antenna to aid a bio-inspired insect hexapod robot in a wall-following control task.

For many animals, whiskers or antennae provide an extremely accurate combination of contact sensing and proprioceptive information, for example, cockroaches can steer themselves along curved walls at 20 body-lengths per second using only the position and rate information that they obtain from their antennae. Other insects, such as arthropods use numerous small hair sensors on their exoskeleton to localize contacts.

Proximity

While proximity sensing does not strictly fall under the category of tactile sensing, a number of researchers have employed various proximity sensors for the application of collision detection between a robot arm and the environment and thus we briefly review these technologies here. Three primary sensor technologies which include capaciflective, infrared (IR) optical, and ultrasonic sensors have been used in this application. Vranish et al. developed an early capaciflective sensor for collision avoidance between the environment and a grounded robot arm [19.29]. Examples of distributed IR emitter–detector pairs utilized within artificial skin for the purposes of proximity sensing have been presented by Lumelskyʼs research group [19.30,31]. A more recent design using optical fibers is reported in [19.32]. Other researchers have developed robot skins that include both distributed ultrasonic and IR optical sensors for the purposes of collision avoidance [19.33]. Wegerif and Rosinski provide a comparison of the performance of all three of these proximity sensing technologies [19.34]. For a more detailed review of this variety of sensors, see Chap. 21 on sonar sensing and Chap. 22 on range sensors.

Other Contact Sensors

There are a variety of other contact-based sensors that are capable of discerning object properties such as electromagnetic characteristics, density (via ultrasound), or chemical composition (cf. animals, senses of taste and smell). While this is beyond the scope of the current chapter, Chap. 60 on biologically inspired robots briefly discusses biologically inspired chemical sensors related to smell and taste. For completeness, thermal sensors and material composition sensors are also briefly discussed below.

Thermal Sensors

Thermal sensing can be used to determine the material composition of an object as well as to measure surface temperatures. Since most objects in the environment are at about the same (room) temperature, a temperature sensor that contains a heat source can detect the rate at which heat is absorbed by an object. This provides information about the heat capacity of the object and the thermal conductivity of the material from which it is made, making it easy, for example, to distinguish metals from plastics.

Buttazzo et al. [19.35] note that the piezoelectric polymer used in their tactile sensing system is also strongly pyroelectric, and use a superficial layer as a thermal sensor. Other sensors use thermistors as transducers, with Siegel et al. [19.36] reporting a 4 × 4 array and Russell [19.37] a 2 × 10 array. Some systems purposely provide an internal temperature reference and use the temperature differential from the environment as a means of finding contacts [19.38,39]. However, objects with a temperature the same as the reference will not be detected. Most of these sensors have a relatively thick outer skin covering the heat-sensitive elements, thus protecting delicate components and providing a conformal surface at the expense of slower response time.

A more recent example of thermal sensing can be found in the work of Engel et al., who present a flexible tactile sensor design that includes integrated gold film heaters and a resistance temperature device (RTDs) on a polymer micromachined substrate [19.40]. While there is a high level of integration presented by Engel et al., these sensing elements still remain fragile, hence tradeoffs concerning construction, performance, and the protection of sensing elements in these systems remains an ongoing challenge.

Material Composition Sensors

There has been a little work on sensors for material composition. In analogy with the human senses of taste and smell, liquid- and vapor-phase chemical sensors could potentially determine the chemical composition of a surface ([19.41,42]). Another sensing modality which provides information about material properties is electromagnetic field sensing, using devices such as eddy-current or Hall-effect probes to measure ferromagnetism or conductivity [19.43,44].

Kinematic Sensors

Although they are not generally thought of as tactile sensors, sensors that detect the position of a limb can provide the robot with geometric information for manipulation and exploration, particularly when the limb also includes sensors that register contact events. Examples of such sensors include the ubiquitous joint angle encoders found in virtually all robots as well as potentiometers, resolvers, and other joint angle measuring devices. For limbs that do not undergo large rotations one can also embed flexible structures such as elements composed of piezoresistive ink, e.g., as used by Cowan et al. [19.28], as discussed in Sect. 19.1.1. Examples of combining information about joint angles with contact status sensors for manipulation include Kanekoʼs work [19.45] on the posture changeability of fingers.

Force and Load Sensing

Actuator Effort Sensors

For some actuators such as electric servo motors, a measure of the actuator effort can be obtained directly by measuring the motor current (typically using a sensing resistor in series with the motor and measuring the voltage drop across the sense resistor). However, because motors are typically connected to robot limbs via gearboxes with output/input efficiencies of 60% or less, it is usually much more accurate to measure the torque at the output of the gearbox. Solutions to this problem include shaft torque load cells (typically using strain gages) and mechanical structures at the robot joints whose deflections can be measured using electromagnetic or optical sensors. For cable- or tendon-driven arms and hands it is useful to measure the cable tension – both for purposes of compensating for friction in the drive-train and as a way of measuring the loads upon the appendage [19.46]. When fingers or arms make contact with objects in the environment, cable tension sensing becomes an alternative to endpoint load sensing for measuring components of the contact forces. Of course, only those components that produce significant torques can be measured with any accuracy. Section 15.3.2 (Robot Hands: Sensors) contains more details concerning cable tension measurement.

Force Sensors

When actuator effort sensors are not sufficient to measure the forces exerted by or on a robot appendage, discrete force sensors are typically utilized. These sensors are found most often at the base joint or wrist of a robot, but could be distributed throughout the links of a robot.

In principle, any type of multiaxis load cell could be used for manipulator force-torque sensing. However, the need for small, lightweight units with good static response eliminates many commercial sensors. The design of force sensors for mounting above the gripper at the wrist has received the most attention [19.48,49], but fingertip sensors for dextrous hands have also been devised. Often these sensors are based on strain gauges mounted on a metal flexure [19.50,51], which can be fairly stiff and robust. Sinden and Boie [19.52] propose a planar six-axis force-torque sensor based on capacitive measurements with an elastomer dielectric. Design considerations for force sensors include stiffness, hysteresis, calibration, amplification, robustness, and mounting. Dario et al. present an integrated fingertip for robotic hands: an integrated force sensing resistor (FSR) pressure array, piezoceramic bimorph dynamic sensor, and force-torque sensor [19.21]. More recently Edin et al. [19.47] have developed a miniature multiaxis fingertip force sensor (Fig. 19.2). For applications where immunity to electromagnetic noise is desirable, Park [19.53] presents a design for a robot fingertip with embedded fiber optic Bragg gratings, used as optical strain gages. Bicchi [19.54] and Uchiyama et al. [19.55] consider the optimal design of multiaxis force sensors in general.

Fig. 19.2
figure 2_20

Miniature fingertip force-torque sensor for a prosthetic hand (after [19.47] with permission)

Full size image

It is interesting to note that more than just force information can be gained by the use of fingertip load sensors. Information from the force sensors can be combined with knowledge of fingertip geometry to estimate contact location, as implied in Fig. 19.3. This method of contact sensing is referred to as intrinsic tactile sensing, and was first presented by Bicchi et al. [19.56]. A comparison between intrinsic and extrinsic contact sensing (i.e., using distributed contact sensors) is presented by Son et al. [19.11]. This is also discussed in further detail in Sect. 19.2.1

Fig. 19.3
figure 3_20

Robot hand with fingertip force and tactile sensing. Information from the force sensors can be combined with knowledge of fingertip geometry to estimate contact location, referred to as intrinsic tactile sensing

Full size image

Dynamic Tactile Sensors

Early special-purpose slip sensors based on displacement detected the motion of a moving element such as a roller or needle in the gripper surface (e.g., Ueda et al. [19.57]). A more recent approach uses a thermal sensor and a heat source: when the grasped object begins to slip, the previously warmed surface under the sensor moves away, causing a drop in surface temperature beneath the sensor. A noncontact optical approach uses correlation to reveal motion of the object surface [19.58]. A number of researchers have suggested using conventional arrays for slip detection, but the array resolution must be good and the scanning rate high to detect the motion of object features soon enough to prevent dropping the grasped object.

In a systematic investigation of the feasibility of using vibration to detect slip, Rebman and Kallhammer [19.59] used single elements from an array sensor to detect normal vibrations at the contact surface. Dario and DeRossi [19.60] and Cutkosky and Howe [19.61] note that piezoelectric polymer transducers located near the contact surface are very sensitive to vibrations and may be used for slip detection. Howe and Cutkosky [19.24] show that using a small accelerometer to sense minute vibrations of a compliant sensor skin is an effective means of detecting slip at its earliest stages. For hard objects held in metal grippers, acoustic emissions may reveal incipient slip [19.62,63]. Morrell [19.64] and Tremblay [19.65] investigated the use of slip sensors in grasp force control.

Buttazzo et al. [19.35] have built a texture-sensing fingernail as part of their anthropomorphic tactile sensing system. A piezoelectric element at the base of the rigid plastic nail produces a large signal as it is dragged over a textured surface. The stress rate sensor [19.22,61,66], the skin acceleration sensor [19.22,24], and the induced vibration sensor [19.67] described above in the context of shape or slip sensing also respond to the small vibrations produced by sliding over fine surface textures. More recent adaptations of these sensors include piezoceramic bimorph dynamic sensors, with integrated FSR pressure array, and force-torque sensor [19.21]. Yamada et al. [19.68], who have developed a piezoelectric artificial skin, are able to distinguish between rolling and slip. Waldron et al. [19.23] describe a tactile sensor that integrates skin acceleration with a piezoelectric array for the application of teledermitology. Ellis has also investigated the use of a very high-resolution tactile array data for discriminating surface textures [19.69]. Omata and Terunuma present a sensor that measures changes in compliance by detecting changes in the resonant frequency of an active piezoelectric element [19.70].

Array Sensors

Array sensors can be subdivided into two primary categories: those that measure pressure and those that measure the deflection of the sensor skin. Tactile pressure arrays are by far more common. Pressure arrays tend to be relatively stiff and utilize a variety of transduction methods along with solid mechanics to calculate contact pressure distribution. Skin deformations/deflections in pressure arrays are on the order of 1–2 mm. On the other hand, skin deflection sensors are constructed in a manner that permits gross sensor skin deformation during contacts, which can be advantageous for grasp stability (see Chaps. 27 and 28).

Hundreds of designs for tactile array sensors have appeared in the literature in the last 15 years, and a few of them are suitable for use with dextrous hands. In terms of transducers, the fundamental requirement is to recover either the shape or pressure distribution across the contact unambiguously from beneath a compliant elastomer layer. This can be done by directly sensing shape [19.71,72] or by sensing multiple components of subsurface strain [19.73,74]. Solid mechanics models (Sect. 19.2.2) can then be used to determine the desired quantities. While the approach used in the models developed to date is quite successful, they must be extended to three dimensions and to incorporate multiple components of strain. Since these models are complex, real-time execution must be addressed.

Contact Location Sensors

There have been a number of tactile sensors that solely provide contact location. Discrete switches placed on the outer chassis of mobile robots were quite common before the proliferation of ultrasonic sensors. Other researchers have presented work using discrete switch arrays. Some such sensors utilize a membrane switch design, found in keyboards. A visually transparent example of such as design is presented by Arai et al. for use in combination with a touch screen display [19.75]. Alternately, Griffin [19.76] shows a membran-switch-inspired contact switch array designed by W. Provancher that consists of flexible circuits separated by thin dielectric strips. Contacts are registered when pressure deflects one flexible circuit into the other, as shown in Fig. 19.4a. More recent work presented by Edin et al. [19.47] shows a nice extension on this idea, with a two-dimensional switch array embedded in a prosthetic hand; see Fig. 19.4b. Some optical tactile sensors [19.77] have also been used primarily as contact location sensors. However, because they can also measure the magnitude of the contact force they are discussed under tactile arrays.

Fig. 19.4
figure 4_20

Contact switch arrays fabricated from flexible printed circuits. (a) A simple 16 × 1 switch array used on the fingertip of a dexterous robot hand (drawn by W. Provancher). (b) Contact switch array embedded in the skin of a prosthetic hand (after [19.47] with permission)

Full size image

Pressure Sensing Arrays

Capacitive Pressure Sensing Arrays

Tactile pressure arrays are one of the oldest and most common tactile sensor types. Some of the earliest work in this area was done by Fearing [19.78] and Fearing and Binford [19.9]. These researchers produced capacitive tactile pressure arrays within a robot fingertip, suitable for dexterous manipulation. The sensor array consisted of overlapping row and column electrodes separated by an elastomeric dielectric, forming an array of capacitors. A change in capacitance results from compressing the dielectric between row–column plates at a particular intersection. The equation for capacitance, based on the physical parameters, is expressed as C ≈ (ϵA)/d, where ϵ is the permittivity of the dielectric between the plates of the capacitor, A is the area of the plates, and d is the spacing between them. Compressing the dielectric between the capacitive plates thus reduces the plate spacing d, providing a linear response with displacement.

Through appropriate switching circuitry, a particular region of a sensor array can be isolated at a particular row–column intersection. Pressure is calculated by means of solid mechanics, as outlined by Fearing [19.79]. Examples of similar capacitive tactile arrays can be found in [19.10,11] and commercially at [19.12]. More recently, Shinodaʼs group has investigated conductive fabric to form pairs of conforming, stacked capacitors more appropriate for covering larger areas than a robot hand [19.13]. The fabric electrodes in the stacked capacitive sensor elements are alternately separated by stiff or soft urethane foam, permitting simultaneous estimation of contact pressure and area within a single sensor element.

Piezoresistive Pressure Sensing Arrays

A variety of researchers have produced tactile sensor arrays that are piezoresistive in nature. These sensors generally either utilize a conductive rubber that is bulk molded or a piezoresistive ink that is generally patterned via screen printing or stamping. Each of these employ a conductive additive (typically carbon black) to create its conductive/piezoresistive behavior. However, because of the fragility and hysteresis that these sensor morphologies exhibit, some researchers have also developed fabric-based piezoresistive sensors.

Russell [19.80] presented one of the first molded conductive rubber tactile sensor arrays composed of conductive rubber column and row electrodes with piezoresistive junctions. However, this sensor exhibited significant drift and hysteresis, which became the topic of later research to try to minimize these effects through proper selection of molding material [19.3]. These issues were never completely solved due to the hysteretic nature of elastomers, but this sensing approach remains attractive due to its ease of manufacturing. Hence it has continued to find applications in the appendages of humanoid robots where extreme accuracy is not required [19.81].

A number of researchers and companies have developed tactile sensors that utilize conductive (piezoresistive) ink, generally referred to as force sensitive resistors (FSRs). This is by far the most common, simplest, and most readily available means to incorporate tactile sensing via off-the-shelf discrete sensors (see Tekscan Flexiforce FSRs). However, to make highly integrated, dense sensor arrays, custom fabrication is necessary. Examples of such sensors are presented by Papakostas et al. [19.4] and Dario et al. [19.21]. To take this approach one step further, Someya [19.5] has produced robotic skin that employs patterned organic semiconductors for local amplification of the piezoresistive sensor array, printed on flexible polyimide film. However, despite being fabricated on a flexible substrate, these sensor arrays are still susceptible to bending fatigue.

Piezoresistive fabrics have been developed to address fatigue and fragility issues found in tactile arrays. Examples of these sensors are presented by De Rossi et al. [19.7], Tognetti et al. [19.8], and Shimojo et al. [19.6]. These sensors tend to be larger (i.e., have lower spacial resolution) and are utilized in applications such as the arms or legs of humanoid robots. Because this technology has the potential to replace ordinary cloth, it is a promising technology for applications in wearable computing or even smart clothing.

One final design that does not fall under the above fabrication categories is a sensor designed by Kageyama et al. that utilizes a piezoresistive conductive gel pressure array along with a multilevel contact switch array via variable contact resistance within sensor layers that they developed for use in humanoid robots [19.82].

MEMS Pressure Sensing Arrays

Micro-electromechanical (MEMS) technology is quite attractive for producing highly integrated packaging for tactile sensing and associated interconnects and electronics. Early devices were produced in silicon through standard silicon micromachining techniques, including the silicon micromachined CMOS-compatible tactile array capable of measuring shear and normal force developed by Kane et al. [19.14]. These sensors functioned well on the laboratory desktop, but were too fragile to survive impact or harsh environments. More recently, researchers have applied MEMS techniques to produce sensor arrays appropriate for embedding within an elastomeric skin with flexible substrates. This has the advantage of improved durability because of the protective skin. Engel et al. have produced a tactile sensor with combined temperature and cantilever elements grown on a polyimide backing [19.40]. Polymer micromachining of the polyimide was performed to give the polyimide substrate preferential compliance. An interesting variation on this design is presented by Noda and Shimoyama [19.83]. These researchers also produced cantilever shear stress elements on their array sensor, however, they do this in a planar fabrication process to fabricate cantilevers with NiCr piezoresistors that are subsequently released from the fabrication substrate and made to stand up off the substrate like hairs by applying a magnetic field and subsequently fixing the cantilevered beams by covering them with Parylene-C (used for sensing shear) [19.83]. Returning to the idea of using silicon load cells for tactile sensing, Valdastri et al. [19.15] have developed a miniature MEMS silicon-based load cell that resembles a joystick and that is appropriate for embedding within an elastomeric skin (see Fig. 19.5). These sensors could be distributed beneath the skin surface to detect complex stress states in the elastomeric skin.

Fig. 19.5
figure 5_20

(a) SEM micrographs of a MEMS three-axis tactile force sensor. (b) MEMS force sensor wire-bonded to a flex-circuit and embedded within a silicone rubber skin [19.15]

Full size image

Skin Deflection Sensing

Brocket [19.84] was one of the first to propose the idea of using a deformable membrane robot fingertips. As noted by Shimoga and Goldenberg [19.85], there are several advantages to using deformable fingertips over more rigid robot fingertips, which include: (1) improved grasp stability, (2) reduced shock, and (3) reduced fatigue for embedded sensor elements. Early work on deformable fingertips includes Russellʼs work to create a compliant silicone rubber robot finger [19.80]. His sensor finger used a rigid backing and polyurethane foam to provide a restoring force for the sensor skin. An array of conductive rubber strain gauge elements and their associated interconnects were cut to the correct shape and bonded to the backside of the silicone rubber skin. The conductive rubber was simply silicone rubber, chosen with minimal mechanical hysteresis, mixed with graphite. The electrical interconnects of a given row tapped into the conductive rubber strain gauge at regular intervals along its length, thus subdividing it into several individual strain measurements, forming a potential divider. Russell shows results for both an 8 × 1 line array and an 8 × 5 array.

Nowlin used Bayesian algorithms to improve data interpretation of a deformable tactile sensor that used magnetic field sensing [19.72]. A 4 × 4 array of magnets were supported above paired Hall-effect sensors off a rigid base by individual fluid-filled balloons. Hall-effect sensors measure the strength of the local magnetic field and should therefore increase with closer proximity to the magnets; however, this is complicated by the proximity of neighboring magnets in the sensor array. Hence, noisy data from the Hall-effect sensors were combined using Bayesian algorithms to predict membrane deformation for a deformable fingertip [19.72].

Later Russell and Parkinson [19.19] developed an impedance tomographic tactile sensor capable of measuring skin deformation over an 8 × 5 array. This sensor was constructed with neoprene rubber and filled with distilled water. Row and column electrodes were made from copper and conductive rubber for the rigid substrate and neoprene skin, respectively. Like the capacitive tactile sensors described above, this sensor utilized multiplexing electronics to reduce the number of electrical interconnects. Square-waveform driver electronics are used to estimate the resistance of a column of water formed between row and column elements, providing a signal that is proportional to the current skin height.

After suggesting the idea of using deformable tactile sensors, Ferrier and Brocket [19.17] implemented a tactile sensor which used optical tracking in combination with models of the sensors skin deformation to predict sensor fingertip skin deformations. The fingertip sensor consists of a tiny CCD camera focused on a 7 × 7 array of dots that are marked on the inside of the gel-filled silicone fingertip membrane. An algorithm is used to construct a 13 × 13 grid over the array of dots. This algorithm uses a combination of the position that is sensed by the CCD camera, which provides the location along a line radially outward from the focal point, and a mechanical model used to solve for the radial distance from the camera focal point based on energy minimization.

Provancher and Cutkosky present another design that uses piezoresistive strain gauges to measure fingertip membrane curvature directly [19.20]. The sensor is constructed on a polyimide substrate with commercial strain gauge strips (used to investigate strain gradients) bonded back to back to isolate bending strains, which are proportional to curvature. This curvature information is important for motion planning in dexterous manipulation. The authors present a mathematical model which uses a set of basis functions and least-squares minimization to calculate a curvature-space curve fit that is insensitive to sensor noise and capable of reconstruction of the deformed membrane shape. Results from a 11 × 1 line array sensor prototype are presented in Sect. 19.2.2.

Other Tactile Array Sensors

An early example of an optical tactile sensor was presented by Maekawa et al. [19.77], who used a single optical position sensing device (PSD) or a CCD camera array to detect the position of scattered light off of a hemispherical optical waveguide fingertip with a silicone rubber cover. Light is scattered at the location of contact and, based on a simple model, multiple points of contact can be estimated. With a textured skin, the magnitude of the force can also be estimated, as the contact area grows in proportion to the pressure. However, an issue with fingertips that use a compliant skin covering a hard substrate is that adhesion between the two materials results in hysteresis. In addition, when the fingertip is dragged over a surface, the friction can produce a shift in the estimated contact position.

Another interesting tactile sensor uses vision to track an array of spherical markers embedded in a transparent elastomer to infer the stress state of the skin material due to applied forces [19.16]. This sensor is currently being commercialized under the tradename GelForce.

A number of sensors have been developed that monitor changes in acoustic energy reflected off of a skin surface or due to the distortion of an acoustic subsurface cavity. Shinoda et al. [19.86] present a sensor that looks at the change in reflected acoustic energy from an acoustic resonator chamber near the surface of the skin and its application for instantaneous friction measurement [19.87]. Ando et al. [19.88] present a more sophisticated ultrasound sensor that achieves 6-DOF displacement sensing via paired plate elements that utilize four ultrasound transducers per plate.

Several researchers have developed sensors with multiple sensing modalities. Examples of sensors that combine tactile and thermal sensing are presented by Siegel et al. [19.89] and Castelli [19.90]. Shimizu et al. [19.91] present an sensor for measuring force and the hardness of an object. This sensor actively uses pneumatic pressure to drive sensing elements into an objectʼs surface.

Another interesting sensor includes a combination of tactile sensing and ultrasound imaging [19.92], in which Methil et al. have developed a haptic system for performing breast examinations that utilizes an optical waveguide tactile sensor.

Tactile Information Processing

Tactile Information Flow: Means and Ends of Tactile Sensing

In discussing the processing of tactile information we return first to the three main uses depicted in Fig. 19.1. For manipulation, we require, foremost, information about contact locations and forces so that we can grasp objects securely and impart desired forces and motions to them. For exploration, we are concerned with obtaining and integrating information about the object, including the local geometry, hardness, friction, texture, thermal conductivity, etc. For response, we are concerned especially with the detection of events, such as contacts produced by an external agent, and in assessing their types and magnitudes. The uses of information are often coupled; for example, we manipulate objects in order to explore them and we use the information obtained through object exploration to improve our ability to control forces and motions in manipulation. Recognizing contact events is also important for manipulation and exploration, as it is for response.

Figure 19.6 summarizes the general flow of information from each type of sensor reviewed in the previous section through primary sensed quantities to information provided for manipulation, exploration and response. A useful thought exercise is to consider exactly what information we use to perform a task such as turning a pen end over end between the fingers. We can easily perform this task with our eyes closed. What information are we using? We need to track the position and orientation of the pen and to monitor the forces that we impose on it to maintain stable manipulation. In other words, we need to know the configuration of our grasp, the locations and movements of contacts over the surfaces of our fingers, the magnitudes of grasp forces, the contact conditions with respect to friction limits, etc. The same requirements apply for robots and are provided by the information flow in Fig. 19.6.

Fig. 19.6
figure 6_20

Force and tactile sensor information flow and signal processing

Full size image

At the upper left corner of the figure, joint angles, combined with the forward kinematic model of the manipulator and knowledge of external link geometries, establish the positions and orientations of coordinate frames embedded in the fingertips. This information is needed to integrate local information about object shape, surface normal orientation, etc. so that the overall geometry and pose of the object can be determined.

Actuator effort sensors provide information about the resultant forces, using the Jacobian transpose: , where f is an n × 1 vector of external forces and moments taken with respect to a coordinate frame embedded in the appendage. is the Jacobian transpose, mapping external forces and moments to joint torques and τ is an m × 1 vector of joint torques for a serial kinematic chain with m degrees of freedom. We require that the k-th column of have elements that are relatively large compared to the overall condition number of J in order to provide an accurate measurement of the k-th element of f. Eberman and Salisbury [19.93] show that it is possible to measure contact force and location using only joint torque measurements if the manipulator has clean dynamics.

Alternatively, we can use a multiaxis force/torque sensor in the fingers, as indicated in Fig. 19.3, or robot wrist to obtain contact forces. This approach has the advantage of providing dynamic force signals with a higher signal-to-noise ratio because they are not masked by the inertias of the robot arm or fingers and their transmissions. If the geometry of the fingertip is known, one can use intrinsic tactile sensing [19.50,94] to compute the contact location as well as the contact force by examining ratios of resultant forces and torques at the sensor.

When the contacts are small compared to the fingertips, so that a point-contact approximation applies, and the fingertips are convex shapes, the contact location is easily computed. Figure 19.7 shows a contact force, f, contacting the fingertip surface at a location r. A force/torque sensor such as that in Fig. 19.2 measures the moment, τ = r × f with respect to the origin. If we consider the lever arm, h, perpendicular to the line of action of f, then h/h = f/f × τ/τ, where h = τ/f is the magnitude of h. We can then write that r = h − α f, where α is a constant obtained by solving for the intersection of the line of action and the fingertip surface. For a convex fingertip, there will be two such points, of which only one corresponds to a positive (inward) contact force.

Fig. 19.7
figure 7_20

Intrinsic tactile sensing: a contact produces a unique line of action and moment about the origin of a coordinate system in the fingertip. The contact location can be obtained by solving for the intersection of the line of action and the fingertip surface

Full size image

Going a couple of steps further, from the contact location one can deduce the local contact normal and contact kinematic type from a small number of force measurements. Bicchi presents algorithms for extending these methods to soft fingers [19.94]. Brock and Chiu [19.51] describe the use of force sensors for the perception of object shape using this approach, and for measuring the mass and center of mass of a grasped object.

For precision tasks involving small objects or small forces and motions, cutaneous sensors provide the most sensitive measurements. In general, as task requirements get smaller, the sensor must be located closer to the contact so that the compliance and inertia of the intervening parts of the manipulator do not interfere with the measurement. Dario [19.95] suggests that fingertip force sensors are useful for forces of 0.1–10.0 N while array sensors can measure distributed forces of 0.01–1.0 N. Son et al. [19.11] find that intrinsic tactile sensing and array sensors can both provide accurate (within 1 mm) estimates of contact location; however the intrinsic tactile sensing method is inherently sensitive to the accuracy of the force/torque sensor calibration and can produce transient errors due to unmodeled dynamic forces.

Proceeding down the left side of Fig. 19.6 we come to the large category of cutaneous array sensors. The interpretation of information from array sensors depends initially on the transducer type. For arrays of binary contact or proximity sensors interpretation amounts mainly to establishing the location and general shape of the contact area. Techniques common to binary vision can be used to obtain subpixel resolution and to identify contact features. This information, in combination with measurements of the grasp forces from actuator effort or force/torque sensors, is sufficient for basic manipulation tasks [19.47].

Solid Mechanics and Deconvolution

A basic problem associated with tactile array sensors is to reconstruct what is happening at the surface of the skin from a finite set of measurements obtained beneath the surface. Typically we are interested in determining pressure and perhaps shear stress distributions associated with contacts on the skin. In other cases, as when the fingertips consist of a gel or soft foam covered by a thin membrane so that the pressure is nearly constant, the local geometry of the contact is of interest.

In the following example we consider the case of an array of elements located at a depth d below the surface of an elastomeric skin. A contact has resulted in a pressure distribution over the region of interest. We establish a coordinate system with z pointing in the inward normal direction and, for simplicity, we examine a one-dimensional loading case, p(y), in which the pressure distribution is unchanging in the x-direction. We further assume that the extent of the skin in the x-direction is large compared to the skin thickness so that strains in the x-direction are inhibited, leading to a plane-strain elasticity problem and we assume that the skin is a homogeneous, isotropic material and that the strains are small enough that linear elasticity theory can be applied. Of course, none of these assumptions is entirely valid in practical cases; however, the results do agree qualitatively with the measurements obtained with actual robot fingers and tactile arrays. A thorough discussion of the general approach and of the accuracy of the linear elastic models can be found in [19.22,79,96,97,98].

Figure 19.8 illustrates the case of two line loads, or knife edges, pressed against the surface of the skin (akin to a planar version of the two-point discrimination test for human tactile acuity). The solution for a single line load, or impulse response, was derived by Boussinesq in 1885. For the case of plane strain the principal stresses in the (y, z)-plane from a normal unit impulse can be expressed in Cartesian coordinates as [19.99]:

(19.1)
(19.2)
(19.3)

where ν is Poissonʼs ratio for the material (typically 0.5 for elastomeric materials).

Fig. 19.8
figure 8_20

Plane-strain stress response for two (unit-magnitude) line loads. Note the blurring that occurs with greater depth

Full size image

For two such line loads located at distances δ 1 and δ 2 from the origin, the solution can be obtained by superposition:

(19.4)
(19.5)

For more general pressure distributions the stresses can be found by convolution of the pressure distribution p(y) and the impulse response G i (y, z):

(19.6)

Also plotted in Fig. 19.8 are curves corresponding to the vertical stress components, σ z , at two different depths, d 1 = 2λ and d 2 = 3λ, where λ is the sensor spacing. As we go deeper beneath the skin, the stresses become smoothed or blurred, and the ability to distinguish between closely spaced impulses diminishes. However, the blurring of concentrated pressure distributions can also provide an advantage when we have a limited number of sensors because the stresses and strains spread over a larger area and are more likely to affect at least one sensor. The elastic skin also provides a kind of automatic edge enhancement because stresses are high at the transitions between loaded and unloaded regions of the skin.

In most cases, for example, in the case of capacitive or magnetic sensors, the sensing elements will measure strains or local deformations of the skin material in the vertical direction. In a few cases, such as pieces of piezoelectric film embedded in an elastomeric skin [19.22], the sensors are sufficiently stiff compared to the surrounding material that they can be considered to measure stresses directly.

For the case of elastic plane strain, the strains are related to the stresses by [19.100]:

(19.7)
(19.8)

where E is the Youngʼs modulus and ν is the Poissonʼs ratio, which we assume is 0.5 for an elastomeric skin.

Figure 19.9 shows the typical measurements that might be obtained from a row of sensing elements from the two line loads applied in Fig. 19.8. Each bar corresponds to the strain, ϵ zi measured by a corresponding element and computed using (19.8), with stresses obtained from (19.4), (19.5), and (19.3).

Fig. 19.9
figure 9_20

Measured strain with assumed 5% noise

Full size image

The problem at this point is to produce a best estimate of the surface pressure distribution, p(y), from this finite set of subsurface strain measurements. The problem is a classic example of estimating a signal from a sparse set of remote measurements. One approach to this process is based on deconvolution techniques [19.22,79,97]. The measured signal from the sensors ϵ z is convolved with the inverse of the impulse strain response H(y) to find an estimate of the surface pressure that produced the signals. The inversion tends to amplify high-frequency noise and the inverse filter bandwidth must be limited according to the spatial density of the sensors and their depth below surface.

Another approach [19.16,96] is to assume that the surface pressure distribution can be approximated by a finite set of impulses p = [p 1, p 2…p n ]t. The sensor readings form a vector, ϵ = [ϵ 1, ϵ 2…ϵ m ]t, where m > n for the bandwidth limitations discussed above. The strain response can then be written as a matrix equation

(19.9)

Each element of H is computed using (19.8) with σ z and σ y computed using equations similar to (19.4) and (19.5), and with σ x from (19.3). The estimated discrete pressure distribution is then found by taking the pseudoinverse of H:

(19.10)

Using the strain measurements from Fig. 19.9, at a depth d = 2λ , the estimated pressure distribution using the pseudoinverse method is seen in Fig. 19.10. In this example, because the assumed set of seven impulses matches fortuitously with the actual loading, the reconstruction is fairly accurate despite the assumed 5% noise.

Fig. 19.10
figure 10_20

Estimated surface pressure distribution using pseudoinverse method for 11 sensors and seven assumed impulses

Full size image

An alternative approach to constructing soft robot finger tips is to enclose a compliant medium such as foam rubber or fluid in a thin elastic membrane [19.18,80,84,101,102,103,104]. Some of the tactile array sensors developed for these fingers are able to measure directly the shape of the membrane, so that a physical model is not needed for interpretation of the signal [19.18]. Another sensing scheme uses an array of magnetic sensors at the center of the finger to measure the changes in the magnetic field generated by deformations of the magnet-loaded membrane [19.104]. A statistical algorithm that can robustly determine the membrane shape from the sensor signals has been developed [19.18]. However, a mechanical model is still required to find the pressure distribution across the contact from the shape information provided by all of these sensors.

Curvature and Shape Information

Another alternative to measuring subsurface strains or deflections is to measure directly the local curvature at each element of an array of sensors [19.105,106]. The curvature information can be applied directly toward identifying contact type and centroid location or it can be integrated to obtain the local shape of the contact as with sensors just described that measure the profile of a membrane. To reduce the effects of noise it is useful to assume a simple model for the membrane and use basis functions to fit the curvature data before integrating [19.20].

Returning to Fig. 19.6, once the local contact shape or geometry has been established the next steps typically include feature identification (e.g., identifying corners or ridges on an object) and determining the overall shape and pose of the object in the hand.

Often the object shape is at least partially known a priori, in which case a variety of surface or data fitting methods can be used, for example, Fearing [19.107] developed a method for calculation of the radius of curvature and orientation of generalized cylinders from tactile array data and [19.108] developed a neural network that performs a similar calculation. Other schemes use contact locations, surface normals, and contact forces to determine information about object shape and orientation with respect to the hand [19.109,110,111,112].

Allen [19.113] uses several different primitive representations for object shape attributes based on the particular exploratory procedure used to sense the object. Object volume and approximate shape are perceived with enclosure grasping, and the resulting shape is modeled using superquadric surfaces. Similarly, measurement of the lateral extent of object faces leads to a face–edge–vertex model and contour following to a generalized cylinder representation.

The question of what constitutes an appropriate set of features is not well understood, although it clearly depends upon the intended application. Ellis [19.69] considers appropriate feature sets and methodologies for acquiring the needed data. Lederman and Browse [19.114] suggest that surface roughness, surface curvature, and oriented edges are used in human haptic perception.

Object and Surface Identification

The most common application of touch information has been in object recognition and classification. In object recognition the goal is to identify one object from a set of known objects using information derived from touch. In classification the goal is to categorize objects according to preselected sensed properties. These systems are usually based on geometric information derived from tactile array or force sensors. Recently the use of other types of touch information (e.g., compliance, texture, thermal properties) in exploration and identification has received some attention [19.115,116,117].

A number of systems have used statistical pattern recognition, which can improve noise immunity since only statistics derived from the sensed data are used. Unfortunately, this also means that only a few object types can be discriminated. Systems have been based on the statistics of tactile array sensor data [19.118,119], and on the statistics of finger joint angles while grasping the object [19.120,121,122].

Many different features derived from tactile array data have been used for model-based recognition and classification. Systems have used geometric features such as holes, edges, and corners [19.105,114,123] and object surfaces [19.124]. Other feature sets include geometric moments [19.125,126], linear transforms [19.127], and sequences of surface tangents [19.128]. Gaston and Lozano-Perez [19.110] use local surface normals and contact locations as features, which could be derived from array, force, or joint sensor information. Siegel [19.129] presents a method for finding the pose of a known object grasped in a robot hand from measurements of the finger joint angles and torques.

Active Sensing Strategies

Because touch provides only local information, movement is an integral part of touch sensing for recognition and exploration. Several workers in object recognition applications have developed strategies for scheduling sensor movements so that each additional observation decreases the number of objects which are consistent with prior observations. This is sometimes described as a hypothesize and test approach. Early examples include Gurfinkel et al. [19.105] and Hillis [19.123]. Schneiter [19.111], Grimson and Lozano-Perez [19.109], and Ellis [19.130] and Cameron [19.131] have developed algorithms for scheduling sensor moves to rapidly recognize polygonal objects using touch measurements of contact location and local surface normal. In Schneiterʼs scheme, each sensing move should cross the boundary of the intersection of all objects consistent with previous sensor observations. Yap and Cole [19.112] show that a probing strategy to determine the shape of a convex planar polygon with V vertices requires at least 3V − 1 measurements. Roberts [19.132] proposes a movement strategy for recognition that includes tracing the robot finger along object surfaces and edges, rather than moving the finger through free space between readings.

In non-model-based approaches, Klatzky et al. [19.117] have suggested that robotics systems can employ the same exploratory procedures used by humans in haptic exploration. These procedures prescribe the finger motions needed for tasks such as tracing object contours, measuring compliance, and determining the lateral extent of object surfaces. Stansfield [19.116,133] and Allen [19.113] have implemented some of these exploratory procedures using multifingered robot hands with tactile array sensors. Dario et al. [19.134] have created similar tactile subroutines, which they have used in a number interesting applications, including the detection of hardened lumps (e.g., tumors) beneath a compliant surface. Kaneko and Tanie [19.45] describe a method for using small finger motions to find the location of finger–object contacts without distributed tactile sensors.

Edge tracking and surface following have also received considerable attention. Muthukrishnan et al. [19.135] look at edge finding algorithms for matching segments between successive tactile array impressions. Berger and Khosla [19.136] have demonstrated curved edge tracking in real time using tactile array information. Pribadi et al. [19.137] propose a control strategy for tracking an unknown object surface using tactile information, and Bay [19.138] designs a surface shape estimator for exploration by a multifingered hand which uses contact location and surface normal information from a force sensor. Zhang and Chen [19.139] present an approach to tactile servoing in which they model the contact between a compliant tactile sensor and an object to obtain a tactile Jacobian, which is used to produce incremental motion commands for a manipulator. They demonstrate the approach for rolling contact and edge tracking tasks.

In practice, surface following and manipulation are often combined. Okamura and Cutkosky [19.140] present an approach in which a rounded fingertip equipped with a tactile sensor is used to trace object surfaces and locate features, which are defined as regions having local curvature that is high in comparison to that of the fingertip.

Dynamic Sensing and Event Detection

For dynamic tactile sensors used to detect such events as gentle contacts or slippage between the fingertips and an object, the main challenge is to detect the event in question reliably, without false positives. The dynamic tactile sensors that produce large signals in response to contact events are also prone to producing large signals in response to vibrations from the robot drive train and to rapid accelerations of the robot hand. Solutions for more robustly detecting contact events include comparing the signals from dynamic tactile sensors at and away from the contact regions and statistical pattern recognition methods to identify the signature of true contact events [19.65,68,93].

Integration of Thermal and Other Sensors

Sensors such as thermal contact sensors are rarely used in isolation; their signals are generally integrated with those from tactile arrays and other sensors to produce additional information for identifying objects. For example, Dario et al. [19.141] demonstrate an approach in which a thermal contact sensor is used in combination with a tactile pressure sensing array and a dynamic tactile sensor used to characterize surface roughness, for discriminating among different objects. Different exploratory procedures, inspired by those used by humans [19.142] are called upon to resolve ambiguities.

Integration Challenges

A critical problem that we have not yet addressed is the difficulty of connecting to a large and diverse array of tactile sensors. In 1987 Jacobsen et al. [19.143] cited the routing of wires as perhaps the most difficult problem in dexterous hand design and, to a large extent, this remains true today. However, some solutions to this problem have been presented in recent years by either using wireless sensors or by use of clever bussing for power and signal connections. Shinoda and Oasa [19.144] embed tiny wireless sensing elements in an elastic skin that uses an inductive base coil to provide power and signal transmission. Each sensing element is a tuned resonator with a distinct resonant frequency whose resonant frequency is stress sensitive. Yamada et al. [19.68] use wireless sensor chips that use light transmitted through a transparent elastomer both for power and to communicate six stress components to a power-receiver chip positioned safely beneath the skinʼs surface. Hakozaki and Shinoda [19.145] embed tactile sensor chips between two layers of conductive rubber that serve to provide means for power as well as a bussed serial communication, thus eliminating discrete wiring. The shrinking dimensions of microprocessors also make it possible to mount devices for multiplexing, signal conditioning, etc. in the immediate proximity of the sensors, thereby reducing the amount of raw information that must be relayed back to the robot.

Conclusions and Future Developments

In comparison to computer vision, tactile sensing always seems to be a few years away from widespread adoption. As explained in the introduction to this chapter, the reasons include physical problems (placement and robustness of sensors, wiring challenges) and the diversity of sensor types for detecting forces, pressures, local geometries, vibrations, etc. As we have seen, the transduction and interpretation methods are typically different for each of these tactile quantities. However, there are some basic issues that apply to tactile sensing in general; for example, sensors are generally located within or beneath a compliant skin, which affects the quantities that they sense in comparison to pressures, stresses, thermal gradients or displacements applied to the skin surface.

When choosing tactile sensors for a robot arm or hand, it is effective to begin with a consideration of which tactile quantities are most desired and for what purpose; for example, the main concern is to obtain accurate measurements of loads or contact forces at sufficient data rates for force servoing, then intrinsic tactile sensing may make the most sense. If manipulating objects with soft contacts and with sliding or rolling, curved array sensors for measuring pressure distributions, or perhaps local skin deflections, may be desirable. If exploring objects to learn about their texture and material composition, dynamic tactile sensors and thermal sensors may be effective.

In an ideal world, one would incorporate all these tactile sensors in a robotic end-effector without regard to cost, signal processing or wiring complexity. Fortunately, the cost and size of transducers suitable for tactile sensing are steadily dropping and the ability to perform localized processing is improving with surface-mounted devices on flexible circuits. In the near future it will be increasingly possible to fabricate dense arrays of transducers in situ on contoured surfaces, using material deposition and laser machining techniques. In this way, robots may finally start to approach the tactile sensitivity and responsiveness of the simplest of animals.