1 Introduction

The ability to sense forces at the microscale is extremely important for the development of new technologies and to expand the capabilities of micro systems. However, sensors at this scale are usually expensive and hard to implement. There are multiple methods [1] that have been used to reach micro-Newton level resolution, such as capacitive MEMS sensors [2,3,4], piezoelectric sensors [5,6,7,8], piezoresistive sensors [9,10,11], strain gauges [12, 13], Atomic Force Microscope (AFM) and optical measurements [14,15,16,17,18,19,20], among others. Some possible applications of such sensors include the field of mechanobiology [21], theranostics [22], biomanipulation [23], and the automated assembly of micro parts. Additionally, force information can be used in parallel with haptic feedback devices [24] to allow for more precise manipulation.

In recent years, the field of micromanipulation has been gaining a lot of interest due to the ability of systems to accurately manipulate parts and perform automated assembly. With the increase in complexity of MEMS devices, there exists a need to assemble multiple microscale components together since microfabrication of a complex stand-alone micro device is usually expensive, extremely challenging, and even sometimes not possible. Adding a force-sensing capability to such micromanipulation systems will increase their range of applications as well as make current manipulation tasks even more precise. Another issue regarding micromanipulation is the uncertainties coming from the substrate where the manipulation occurs. At the microscale, surface forces play a large role and are extremely difficult to model and predict. In an effort to reduce the manipulation uncertainties that arise from surface forces, machine learning models can be used, as in [25]. With the addition of force feedback to the traditional micromanipulation setup, more accurate models can be developed. Additionally, it is also very desirable to create substrates that are able to minimize these uncertainties when pushing parts in the workspace. Micromanipulation of parts on these substrates with force feedback can help in their design.

A vision-based force sensor, which tracks the deflection of a stiffness calibrated compliant structure and uses the basic principles of Hooke’s law to calculate force from displacement, can overcome many of the drawbacks associated with the use of other micro-force sensing techniques. These include AFM and optical sensors, which have great resolution but are expensive and hard to integrate into standard test-beds; capacitive sensors, which can be susceptible to noise and require complicated circuitry; piezoresistive sensors, which are simple and easy to use but are sensitive to temperature and hard to miniaturize; piezoelectric sensors, which are susceptible to charge leakages and have poor spatial resolution; among other sensors. This compliant structure can be made out silicone elastometers with low Young’s modulus and high failure strain, such as polydimethylsiloxane (PDMS), which allows for a high force sensing resolution and reliability, as well as easy integration into standard test-beds. These systems have been used in a multitude of applications, both in one [26, 27] and two dimensions [28,29,30].

In this paper, a proof of concept of a vision-based micro-force sensing probe capable of accurately measuring 3D micro-Newton level forces that can be easily integrated in standard micromanipulation test-beds is presented (Fig. 1). This sensor builds off of the work in [29] but adds an extra dimension of measurement, a decrease in sensor size, and an increase in sensitivity and resolution in all dimensions, and real-time tracking. Section 2 describes the overall design of the sensor and Section 3 described the fabrication. Then, Section 4 addresses the sensor calibration and validation studies. Section 5 describes the selective color tracking algorithm capable of 3D real-time micro-force sensing. Lastly, Section 6 presents two case studies that showcase possible applications of the sensor and how it behaves during actual experiments.

Fig. 1
figure 1

Schematic of the 3D vision-based micro-force sensing probe. The microfabricated micro-scale vision-based force sensor (μ VBFS) is attached to a manipulation probe. 3D deformation of the stiffness calibrated structure is captured by the dual camera vision system

Full size image

2 Design overview

For a vision-based force sensor, the sensed force in a particular direction, Fi, is determined by multiplying the stiffness in that direction, ki, by the measured deflection, δi. This way, the lowest force detected corresponds to the smallest deflection detectable by the vision system. In order to make an accurate measurement, we assume that a minimum deflection of at least a pixel is needed to identify a corresponding deflection in microns. Therefore, there is a trade-off between the size of the field of view (FOV) and the minimum detectable force. For a larger FOV, each pixel corresponds to a larger physical distance, thus decreasing the minimum measurable deflection and therefore, the force.

The vision-based micro-force sensing probe is made up of three main parts, as shown in Fig. 2a: 1. the manipulation probe, 2. the attachment fixture, and 3. the micro-scale vision-based micro-force sensor (μ VBFS). The probe attachment fixture is used to mount the sensor to a standard micromanipulation probe. This part has been 3D printed using a Connex3 350 Polyjet printer. The threaded part of the design can be simply screwed into a standard micromanipulation probe, which makes the sensor easily integratable into test-beds for many applications. On the other end of the fixture, there is a small slot used to attach the body of the μ VBFS. This allows the tip of the μ VBFS to interact with the objects being manipulated. Figure 2a shows the final assembly of the device.

Fig. 2
figure 2

3D vision-based micro-force sensing probe: a final micro-sensor assembly, b overall design of μ VBFS with corresponding dimensions. Here, t = 25, d = 45, W = 719, x1 = 50, and x2 = 122, all in units of μ m

Full size image

In order to be able to measure μ N-level forces, the stiffness of the compliant structure must be small enough so these forces will cause deflections within the detection range of the vision system. Due to these constraints, the compliant structure was made out of PDMS (Sylgard 184 silicone elastometer kit) with appropriate geometry (Fig. 2b) to observe its deflection following the design requirements and guidelines described in [28].

3 Sensor fabrication

The fabrication process of the μ VBFS was designed to allow for the creation of compliant structures with varying stiffness properties depending on the desired application. It is based on subsequent photolithography steps followed by a deep reactive-ion etch (DRIE), which creates both the mold where a PDMS structure will cure and the outer rigid frame of the sensor (Fig. 3). First, positive photoresist AZ1518 is spin coated at 1000 rpm for 30 seconds then it is soft-baked at 100C for 3 minutes. The wafer is then exposed for 30 seconds using a Suss MA6 Mask Aligner (SUSS MicroTec), which creates the outline of the spring structure. The non-polymerized photoresist is then removed using MF CD-26 developer (Microchem Inc., USA) for 30 seconds, followed by a post-bake at 100C for 3 minutes, and the DRIE process, which is used to create the PDMS mold with the final desired thickness of the μ VBFS. In order to fabricate the compliant structure, PDMS is mixed at a base/curing agent ratio of 10:1 or 16.7:1, depending on the desired stiffness, and degassed for 30 minutes to remove air bubbles. The PDMS is then spin-coated onto the wafer at 2000 rpm for 30 seconds, and the excess polymer is removed using a silicone spatula. After the PDMS is cured for 24 hours at room temperature, the wafer is cleaned with acetone, leaving behind the compliant structures embedded into the silicon wafer. The μ VBFS frame is defined in a similar way. A photolithography process as described for the PDMS structure is performed but this time with a different mask. Then, the DRIE process will etch the silicon around the sensor frame. After another acetone cleaning step, the μ VBFS is ready to be released. A back-side window in patterned on the back of the wafer and the sensors are individually collected after a backside etching process, also using DRIE.

Fig. 3
figure 3

Overall fabrication procedure of the 3D vision-based micro-force sensing probe. (i) Fabrication steps for making the μ VBFS, where red represents photoresist, grey represents silicon, and blue represents the PDMS: (1) Photolithography and etching to create the mold for the compliant structure; (2) the deposition of the PDMS structure; (3) the photolithography and etching to outline the rigid outer frame; and (4) the backside etching to release the sensor from the Si wafer; (ii) 3D printed probe attachment fixture; (iii) Final assembly of the μ VBFS to the fixture

Full size image

Once the μ VBFS fabrication is completed, each sensor’s stiffness is individually calibrated in the x, y, and z directions (see Section 4.1). The sensor is then glued to the tip of the 3D printed probe attachment fixture. The fixture is then screwed into a micromanipulation probe. Since the fabrication process allows for the fabrication of sensors with different stiffness values, multiple force sensor/probe attachement pairs are created and they can easily be changed based on desired application and force ranges.

4 Vision-based micro force sensor calibration and validation

There are two key components needed to develop an accurate vision-based micro-force sensor. The first is an accurate stiffness calibration of the PDMS compliant structure in all three dimensions. The second is the image processing algorithm used to measure the deflections of the sensor with relation to its body, also in three dimensions. Each of these critical steps are described in the subsections below.

4.1 Stiffness calibration

In order to be able to measure forces in three dimensions using a vision-based method, it is necessary that the stiffness in the x, y, and z axis to be obtained (kx, ky, kz, respectively). This was done by securing the μ VBFS to a glass slide and using a XYZ micromanipulator (MP-225, Sutter Instruments) to push a commercial force sensor (FT-S100, FemtoTools) against it. Figure 4a–c, show the FT-S100 sensor pushing against the tip of the compliant structure with visible deflection. Multiple force measurements were obtained this way for different deflection amounts. This was repeated for each direction and the results plotted, as shown in Fig. 4d. The stiffness is obtained by fitting a line to the data points and calculating its slope. As shown, for this specific sensor, the stiffness is: kx = 0.1206 N/m, ky = 0.2422 N/m, and kz = 0.0539 N/m. For each of the curves, more than thirty data points were recorded.

Fig. 4
figure 4

Results from the 3D calibration of the μ VBFS. The slope of the curves correspond to the stiffness in the respective direction. The images shown in (a), (b), and (c) represent the deflections in the y, x, and z directions, respectively

Full size image

Based on the results of this calibration, the resolution of the sensor in each direction, as well as range was calculated, as shown in Table 1. In this case, the minimum resolution refers to the minimum force measurable based on the camera pixel resolution (3.37 μ m/pixel for the top camera and 3.54 μ m/pixel for the side camera) and a displacement tracking accuracy of 1 pixel. Note that the camera pixel resolution depends on the size of the field of view for the 1600 x 1200 pixel image. For calibrations, the image is zoomed in to have a more accurate measurements. The sensing range is the maximum force that can be measured before the compliant structure contacts the silicon frame. As seen, the stiffness can be tailored depending on the desired application, with the sensing range can go up to 186 μ N in some cases and the micro-force sensing resolutions going down to the sub-μ N level in others. Note that even sensors with the same PDMS mixing ratio can have different stiffness values. This is due to two main reasons: the PDMS might not spread perfectly across the entire wafer surface, so sensors from different parts of the wafer can have a slightly different stiffness, and the amount of dye in the compliant structure affects its overall stiffness, as shown in [31]. Consequently, every sensor is individually calibrated to ensure accurate force measurement.

Table 1 Table summarizing the results of different μ VBFS
Full size table

4.2 Off-line 3D vision tracking

The goal of the vision tracking algorithm is to accurately measure the displacement of the compliant structure as the probe applies forces to other objects. The experimental setup, as shown in Fig. 5, contains two cameras (side and top views), the XYZ micromanipulator (MP-225, Sutter Instruments), and a custom made test-bed that keeps the working substrates secured during manipulation. Both cameras used are 1.3MP CMOS cameras (PointGrey e2v EV76C560) with adjustable magnifications ranging from 0.75x to 4.5x.

Fig. 5
figure 5

Experimental setup used to record the videos and perform the manipulation tasks

Full size image

In order to validate the proposed technique, an off-line algorithm that uses multiple-object tracking techniques was employed to measure the deflections of the compliant structure. It keeps track of the positions of the tip of the end-effector and the μ VBFS body in two camera views. At first, two regions of interest (ROI’s) are defined and the algorithm records their respective center positions over time (Fig. 6a). This creates a vector between the body and the end-effector tip for each camera view, which changes magnitude (length) and angle during the manipulation process. By tracking these changes and applying a transformation matrix, the deflection in the x, y, and z directions are computed. All of the tracking was performed offline using the Discriminative Correlation Filter Tracker with Channel and Spatial Reliability method [32] and the algorithm was able to run at 15 Hz. Since the stiffness of the compliant structure is known, these deflections are then used to compute the force applied in 3 dimensions, as intended. The custom developed application was written in Python using the OpenCV package (version 3.4.1).

Fig. 6
figure 6

Results of the sensor validation experiments showing the total displacement of the end-effector tip when compared to the body. a The locations of the ROIs from which the algorithm computes the deflections in the y and z directions. The resulting displacement plots for a simple pushing task in the (b) y-direction, c x-direction, and d z-direction

Full size image

4.3 Sensor validation

In order to check the validity and accuracy of this tracking method, a fixed block was pushed multiple times using the probe at intervals of approximately 20 μ m each, with a total displacement of exactly 80 μ m. This was done using the 3-axis micromanipulator, which has a 1 μ m step size. Then, using the 3D vision tracking algorithm described in Section 4.2, the measured deflection in all three axes was recorded. By comparing the measured values with the actual deflection (displacement), the accuracy of the vision-based micro-force sensor tracking algorithm can be validated.

Figure 6 shows the actual ROI tracking and the measured deflections in all three axes. As noted, the deflection increases in increments of 20 μ m and remains stable at 80 μ m at the end of the pushing validation. Clearly, both the x and y directions obtained displacement readings close to the nominal values, with an average final error of approximately 0.58% in the y-direction and 3.63% in the x-direction. For the validation in the z-direction, an extra 20 μ m step was performed, so the total travel was 100 μ m, with an average final error of approximately 0.61%.

5 Real-time micro-force sensing

In order to achieve the full potential of the μ VBFS, it is imperative that the tracking algorithm is able to run in real-time. This way, the user is able to know the force being applied at all times, allowing for manual adjustments as needed instead of analyzing the forces applied after the experiment has been completed. Another major advantage of real-time force feedback is that controllers can incorporate the force readings and allow for precise real-time control and force-guided micromanipulation. As shown in [31], simply by showing the force applied in real-time, the user is able to make adjustments and keep the force below a set threshold during the entire process. Furthermore, a fail-safe can be implemented which would not allow the micromanipulator to move if the force is to go above the threshold set by the user.

Simply using the algorithm described in Section 4.2 will not suffice here, since it is not able to run in real-time. Thus, a new algorithm was developed for real-time micro-force sensing applications. First, the code was written in C++ instead of Python simply because of the much faster speed the code runs in C++. Using the OpenCV trackers as before did not show promising results. Instead, a color-based tracking method was developed. During the setup portion of the algorithm, the user selects a region of interest (ROI) that encompasses the tip and body of the μ VBFS, as shown in Fig. 7a, and then a color thresholding filter is applied to identify two major bodies (the end-effector and body of the sensor). The color filter is applied to the ROI, which makes the algorithm more efficient since it does not need to search for color features in the entire image, but simply inside the ROI. As before, the algorithm then computes the relative distance between the two bodies and compares it to the undeflected distance. In this case, the probe attachment was marked in a different color so that colored region can be used as the body tracker which is compared to the position of the end-effector of the μ VBFS.

Fig. 7
figure 7

a User selected region of interest (ROI) required at the start of the algorith. Notice the bodies of interest are inside of the area and a color marker is used to track the relative position of the sensor body. b Side camera view of the sensor used to measure the μ VBFS angle with respect to the workspace (glass slide in this case)

Full size image

To further speed up the process, the real-time micro-force sensing algorithm only uses one camera view (top view) instead of two cameras. This is possible since the angle of the μ VBFS can be computed ahead of time with the help of the side camera and this angle is held constant. Figure 7b shows the view of the horizontal camera used to compute the sensor’s angle (𝜃) relative to the workspace. The deflection in the x-direction, as seen by the top camera, corresponds to the true x-deflection of the sensor since the angle does not affect its measurement. On the other hand, the y-deflection, as seen by the top camera, is actually a vector made out of the true y and z-deflections of the μ VBFS. Thus these values can be extracted using the known sensor angle relative to the workspace (horizontal plane). One of the issues of this method is that it will introduce more uncertainty to the system relative to the precision of the angle measurement. Therefore, the side camera was used to record the workspace and measure the angle of the sensor more precisely, or at least as precisely as if two cameras were used during the entire tracking algorithm. Using only one camera instead of two effectively increases the speed of the tracking algorithm by a factor 2, since only one image needs to be processed at a time instead of two.

Using the one camera method, a validation was performed in which the sensor was displaced by a known amount and compared to the values measured by the tracking algorithm. This validation method measured the error in the x and y displacement as 3.61% and 1.21%, respectively. Here, we assume that the y and z displacement are coupled, so the relative error in these dimensions is similar. As a result, the average error was similar to the one previously recorded with the two-cameras setup (2.41% using one camera versus 2.11% using two cameras).

Due to this significant improvement in speed and relatively equal accuracy, the one camera method was implemented. With it, microobjects were manipulated in the workspace in real-time with the algorithm running at around 20Hz. In reality, the speed depends a little on the size of the initial ROI selected, so the overall speed ranges from 18Hz up to 28Hz, a significant improvement. These speeds were achieved by running the code using the CPU. If GPU-aided programming is used in the future, the algorithm can reach even higher computing speeds.

6 Case studies

Now that the μ VBFS has been validated, it can be used in various micromanipulation applications. In this section, two case studies are shown: (A) pushing of microscale blocks around the workspace with manipulation force extraction and (B) surface force measurement when manipulating micro-parts on different substrates. Both of these studies show possible applications of the μ VBFS and the range of experiments that can be performed with them.

6.1 Micromanipulation with force sensing

In a general sense, the probe developed here can be used for any micromanipulation application in which the pushing forces fall within the sensor’s range. In order to put into perspective the capabilities of the 3D vision-based micro-force sensing probe, a simple manipulation task was performed with a silicon micro-block, as shown in Fig. 8. The goal is to move the micro object into a specific slot, similar to what would be done in a general microassembly procedure (peg-in-the-hole problem). The manipulation of part was performed manually. As a general procedure for this experiment, the part was pushed using the XYZ micromanipulator until it reached its goal location. While the manipulation was being performed, two cameras (top and side) recorded the entire workspace. The videos were then analyzed using the 3D vision tracking algorithm described in Section 4.2. Since the compliant structure used for the experiment had been previously calibrated, the stiffness value in each direction are known, and the algorithm can compute the forces applied in each direction during the entire manipulation process. Figure 8 also includes these extracted forces for this particular manipulation task.

Fig. 8
figure 8

Screenshots at different time stamps for the object manipulation from the initial (1) to final position (5) from both the top and side view with the measured forces at each moment

Full size image

A similar micromanipulation task was also performed using the real-time force sensing algorithm, which is even more useful, especially for biological applications in which a maximum force exists to prevent damage to the sample of interest. Using this method, the user possesses much more control over the manipulation and it allows for quick response and interference when needed. Figure 9 shows the manipulation of an SU-8 disk around the workspace with real-time micro-force feedback. In this case, the initial ROI selected was relatively large, thus the frame rates are on the lower end (≈ 19 Hz). There are also relatively small frame rate fluctuations throughout the manipulation process, but they do not affect the general performance of the algorithm. This case study shows the potential for using the μ VBFS for 3D closed-loop force controlled manipulation and assembly tasks in the future. Additionally, the integration of haptic devices to control the manipulation will make the entire process much more intuitive and easy to perform. Of course, the haptic integration can only happen if the algorithm is able to run in real-time, as demonstrated, but slightly higher speeds are desirable.

Fig. 9
figure 9

Screenshots at different time stamps for the manipulation of an SU-8 disk micro-part with real-time micro-force feedback. It shows the manipulation from initial position (1) to final position (5). Note: all the force values are in μ N; FPS = frames per second

Full size image

6.2 Measuring surface forces for different substrates

Pushing objects in the microscale is a rather uncertain task, since surface forces play a very large role in the results and are extremely hard to model. Any imperfection in the substrate lattice can cause different friction forces along the same surface. Additionally, even dust or small particles can have a great effect in the manipulation process. Therefore, this case study aims to measure the necessary pushing forces required to manipulate the same object for different manipulation substrates to determine which can produce the most reliable manipulation surface. This is done by pushing a silicon micro-block across different surfaces while recording the necessary force to make the object move. Three substrates were tested: a glass slide, a glass slide cleaned using Isopropanol Alcohol (IPA), and a glass slide with a thin hydrophobic gold layer on its surface.

The hydrophobic coating was obtained by sputtering a thin 60 nm gold layer onto the glass slide and then placing it in a 0.5 mM solution of 1-Dodecanethiol (Sigma-Aldrich) in Ethanol Absolute (94-96%, Alfa Aesar) for 13 hours [33]. Figure 10 shows the magnitude of the resulting 3D force from pushing the same object across the three different substrates. In general, the average force needed to push the object is approximately the same for all substrates. However, the glass slide and the glass slide with IPA displayed several unpredictable force peaks. These are due to the fact that the part encountered a region of the substrate in which the friction force was elevated, thus requiring a lot more force to move the object.

Fig. 10
figure 10

Pushing force in the y-direction for different substrates using the same Silicon micro-block

Full size image

Comparing the glass slide with the IPA cleaned glass slide, both present two large peaks of approximately the same magnitude (approximately 100 μ N), while the regular glass slide has two extra 70 μ N peaks. Clearly, the hydrophobic gold surface is superior in reducing the uncertainties in the surface forces since it does not display any large force peaks during the pushing task. Therefore, the μ VBFS is able to identify and quantify this coating as a robust surface for planar micromanipulation tasks.

7 Conclusions

In this paper, we have presented the proof-of-concept of a 3D vision-based micro-force sensor to be used in a standard micromanipulation test-beds. The μ VBFS was specifically designed to provide sub-μ N level force resolution in three dimensions. The 3D vision tracking algorithm is able to accurately measure the deflections of the PDMS compliant structure and compute the pushing forces, with speeds up to 28Hz. This development enables the μ VBFS to be used in many more applications, making it a more complete tool. This design was then used in two case studies to showcase some of its possible applications. It was able to manipulate a silicon micro-block while measuring forces in three dimensions. It also measured the surface forces encountered by the micro-part on different substrates. In general, the μ VBFS design is a useful tool for force-controlled micromanipulation and assembly tasks that require 3D force sensing with sub-μ N resolution. Future work to make the real-time force sensing algorithm faster, integration with haptic devices, and the use of higher resolution cameras are planned.