Introduction

Needle insertion is commonly used in various clinical procedures including biopsies, treatment injection and anesthesia. The success of these procedures often depends on accurate needle placement to minimize complications and to avoid damage to neighboring tissue. For this purpose, ultrasound (US) is used widely to guide needle insertions because it is safe, real time and low cost [1,2,3,4]. The operator will adjust the needle and transducer by observing both needle and target in the live image to accurately navigate and place the needle to the target location, hence increasing the safety and success rate of the procedure.

However, despite the wide range and long history of image-guided needle insertions, an ongoing issue is poor needle visibility [5,6,7,8]. Needle visualization is especially challenging for deep insertions and for insertions with steep angles relative to the US beam.

Aligning the needle and transducer, and advancing the needle without being able to properly visualize it, can result in targeting errors. Needle detection in out-of-plane cases is even more challenging since only the cross section of the needle with the US beam can be visualized. 3D imaging has also been used to enhance needle visualization, since the additional dimension may give extra information about the needle location by providing the 3D view [9, 10].

Fig. 1
figure 1

Gold standard: 20G echogenic needle extended from the 17G cannula (Reprinted with permission from Elsevier published paper: Beigi et al. [38])

Full size image

A wide variety of attempts have been made to enhance sonographic needle visualization and localization. To get a better understanding of this large body of literature, a scoping review discussing the pros and cons of various techniques is needed in the field. While methods enhancing needle visualization may not improve needle localization directly, they can help subsequent localization techniques achieve a better accuracy. Therefore, in this review, we have included both types of research improving needle visualization or localization.

The large body of literature on this topic can be divided into five main categories: (1) signal and image processing-based techniques to augment the needle, (2) modifications to the needle and insertion, (3) changes to US image formation, (4) motion analysis and (5) machine learning. These techniques were described and evaluated in the literature using different study designs and outcome measures. A summary table describing the literature based on their corresponding category is discussed in Table 1. The aim of this table is to summarize references based on the following important criteria: (1) validation method (in-vivo, ex-vivo or phantom), (2) whether the technique enhances needle visualization or localization (3) tip or shaft accuracy evaluation (for those enhancing localization) and (4) whether a failure rate is reported. To give more context, these are listed based on the specific category each reference belongs to.

Table 1 A sample of the literature summarizing the variety of approaches enhancing needle visibility and localization
Full size table

Validation and gold standards

Validation against a known true location of the needle is also a challenge since, if the needle is visible enough for the operator to manually select it, then there is no need for enhancement. On the other hand, the true location, or gold standard, is unknown in cases where the needle does not have clearly visible features in the image for manual annotation. Such gold standards should also be evaluated preferably against independent methods, for accuracy and possible introduction of other errors.

Optical, electromagnetic (EM) tracking and robot-assisted systems can be used for localization but have their own limitations and localization errors, as will be discussed in “Modified needles” section. EM tracking, for example, with miniaturized sensors placed at the needle tip, has been used in the literature to verify a gold standard. This approach is inherently more accurate compared to techniques requiring the sensor placement at the needle base rather than the tip due to possible needle bending. The accuracy of the electromagnetic sensors is varied for different types of trackers and depends on the surrounding environment. For example, the maximum root-mean-square (RMS) accuracy for Medtronic StealthStation Treon-EM and the NDI Aurora trackers were found to be up to 1.4 mm and 5.1 mm, respectively [22]. Maximum errors for electromagnetic sensors in general though increases to 5 mm to 119 mm when they are attached to 3D transducers [50].

In the literature, the gold standard is usually obtained from the manual annotation of the needle in B-mode images. This of course requires some assumptions about the initial visibility of the needle. Whilst some of the researchers, e.g., [40], assume partial visibility of the needle shaft and tip, most of the papers rely on generally visible needles. An echogenic needle threaded within the cannula of the needle tube detected, and slightly extended beyond the tip is a relatively new paradigm for creating a gold standard, see Fig. 1. Initial phantom tests showed an average error of 0.4 mm in the detected tip against the true tip position, identified as a mean of the estimates of three human observers, although the accuracy depends on the tissue stiffness [38]. The advantage of this gold standard over possible other ones (CT registration to three-dimensional US (3DUS), optical tracking and transparent phantoms) is that it indicates both tip and trajectory in the same US coordinates as the needle measurements, so error due to coordinate transformation to common reference system is reduced. In addition, although fiducials may be controlled in a phantom experimental setup, it is much more challenging to experiment in vivo.

Due to the variations in gold standard verification in the literature, and the inherent error in the gold standards themselves, direct comparison of accuracy measurements of the tip/trajectory among different approaches should be taken in the context of the equipment and methods used. Accuracy as well as the gold standard are reported in the following sections. It is also appropriate to compare the combination of validation approach, gold standard robustness, applicability for clinical practice and failure rate, which are also described when available.

In vivo validation is challenging due to the difficulty of obtaining a gold standard and that testing a particular technique may not be practically achievable in an in vivo setting. A great portion of the state-of-the-art (\(67\%\)) that we reviewed here, therefore, validated their techniques ex vivo or on phantoms, which only leaves \(33\%\) of the them being validated in vivo.

Signal and image post-processing

Since most needles are thin and made of steel, they ideally show up as a fine bright line in the US B-mode image. Therefore, signal and image processing methods such as line detection [11,12,13], image filtering [40, 41] and projection-based approaches [11, 34, 35] have been used to enhance visualization and localization of the needle in the US image. Most of these types of approaches rely on the visibility of the needle in the un-processed US image as a long line-like structure, and line detection techniques (e.g., Hough, Radon and Random Sample Consensus (RANSAC)) are usually used to detect the needle. Visualization and localization often use edge detection and/or filtering.

Edge detection

Methods in this category rely on the original B-mode US image to contain the needle as a long line-like visible structure. The needle is detected by applying edge detection followed by line detection techniques on the B-mode image.

Draper et al. [14] presented a needle detection method based on the variance image specifically for visible needles in the US images. Out of 33 images of a tissue-mimicking phantom, the validation showed an error of 4\(^\circ \) in trajectory angle, 1 mm in trajectory intercept and 1 mm in tip identification in most cases, and max error of greater than 5 mm, measured against an observer identified gold standard.

While most of the literature focuses on straight needles, curved needles have also been investigated. Okazawa et al. [12] proposed a needle segmentation method based on Hough transform to identify the needle curvature. The method uses ray casting, Sobel edge detection, Hough transform and a coordinate transformation. A coordinate transformation was used to identify curved needles within the region of interest (ROI), by extracting slope, intercept and curvature of the needle, see Fig. 2. Validation was performed on a tissue-mimicking phantom, showing mean error of \(0.2--0.8\) mm in needle tip localization measured against the manually identified needle for 10 US images.

Fig. 2
figure 2

Edge detection: illustration of an US image, with the approximate needle axis parallel to the long axis of the box, with perpendicular rays shown as dashed lines. The projection of point scores along the approximate needle axis is computed to detect the strongest linear feature parallel to the axis (Reprinted with permission from Elsevier published paper: Okazawa et al. [12])

Full size image

Image filtering

Methods in this category rely on the original B-mode US image to contain a (partially) visible needle, and usually the needle is assumed to be straight as well. The needle is localized by applying filtering followed by line detection techniques on the B-mode image.

Hacihaliloglu et al. [40] presented a needle detection method that first extracts the image local phase using Log-Gabor wavelets, and then uses RANSAC to locate a needle with mid-steep insertion angle. The method works as long as the initial portion of the shaft (insertion site) and the needle tip is visible in the B-mode image. Radon transform of the phase-based descriptor (constructed from Log-Gabor filtered image) was used to estimate the initial needle trajectory. Phase-based descriptor was performed on an extended ROI surrounding the estimated trajectory to detect the needle (Fig. 3). An average needle tip error of 0.43 mm with maximum error of 1.19 mm was measured for bovine tissue ex vivo against a manually segmented needle.

Ding and Fenster [34] used image projections to segment the needle in 3DUS images in an iterative framework. The approach assumes that the needle appears as a high intensity cluster of voxels. The method is validated on a turkey breast phantom, which showed an average error of 0.70 mm in needle position and \(1.2^\circ \) in needle trajectory.

Wu et al. [35] used a phase grouping approach based on the orientation of the gradient field and intensity variations followed by least-squares fit and a 3D randomized HT to segment line coordinates in the volume. A local optimizer is finally performed by minimizing a distance cost function to refine the estimated needle axis. Needle endpoint was identified using intensity tracking along the needle axis. The method was validated in vivo, showing an error of 1.43 mm in needle tip localization.

Fig. 3
figure 3

a Estimated initial trajectory overlaid on the B-mode image. b Extended ROI trajectory overlaid on the B-mode image. c Local phase features within the ROI. d Final trajectory detection by extracting inliers using MLESAC (Maximum Likelihood Estimation SAmple Consensus) technique. (Reprinted with permission from Springer GmbH published paper: Hacihaliloglu et al. [40])

Full size image

Modified needles

Additions and modifications to the needle could either be electronic enhancements, or physical enhancements to the needle [51]. Sensors and actuators have been adopted widely to help with needle enhancement [15, 16, 18,19,20,21]. Mechanical needle guides [42, 43] as well as robot-assisted needles [42, 48] have also been used. Specific needles were also designed either to enhance reflective US echo (“echogenic” needles) [36, 37] or to ease the guidance especially in robot-assisted procedures (“steerable needles”) [11, 46, 47]. Note that while needle steering and robot assistance do not directly improve needle visualization, they may help with controlling the needle and enhancing its localization.

Sensors, trackers and transducers

An electromagnetic field, generated by an external field generator, could be used to induce a small current in the sensors attached to the needle and transducer. The position of the needle can be estimated from the sensor position with respect to the magnetic field. Hakime et al. [19] used EM tracking on a 18G stylus to detect a biopsy needle. An error of 2 mm was reported for needle position tested in vivo. 3D transducers can deviate error more than 2D transducers (e.g., up to 12 mm); therefore, investigations should be done in advance to minimize interference [50].

Chiang et al. [52] added an “eye” to the needle by placing an US transducer crystal (40 MHz) into the hollow chamber of an epidural needle. The back-scattered A-mode US signal from the transducer visualizes various locations in the tissue. This technique was tested for epidurals, and analysis of the A-mode signal could visualize ligamentum flavum and dura mater surrounding the epidural space in at least \(83\%\) of the cases, accuracy on needle localization however was not reported.

Perrella et al. [16] designed a miniature receive-only sensor at the tip of the stylet that was displayed as a flashing point. An electrical signal is transmitted when the US beam hits the sensor. Experimentation was done in vivo: in 16 out of 20 in vivo insertions (13 clear shaft visibility), the needle was localized “precisely,” while no quantitative analysis on accuracy was performed.

Chan et al.[17] designed a tracking system to localize the needle with respect to the US transducer. The tracking device contains a pair of cameras to track the needle position. The accuracy of the method was evaluated based on the shortest 3D distance between the actual intersection point of the inspected needle and the detected trajectory, resulting in minimum average error of 3.1 mm.

Robot assistance

Robots offer the potential for more robust needle and transducer handling and may ease the procedure by aiding the operator directly. Programmable steerable needles could also be used in conjunction with robot assistance to help with guiding the needle. Flexible needle steering under 3D ultrasound guidance was performed using feedback force and visual servoing [49]. A flexible needle is modeled as a polynomial curve defined by a set of points forming the needle at a given time. The method was evaluated on a tissue-mimicking phantom, mainly considering the feasibility and robustness of technique.

Boctor et al. designed a calibrated robotic system to manipulate the needle insertion and an imaging system for reconstruction [48]. To reconstruct a volume from a series of 2D images, EM tracking (with RMS Accuracy 2.54 mm) was used to track the US transducer and also the needle. The 3D robotic needle insertion was compared against freehand 2D needle insertion ex vivo, and the designed robotic system was shown to be superior with more than 2 mm improvement in targeting accuracy. Specifically, the tip error of 1.56 mm and 5.5 mm was reported for robotic needle insertion ex vivo and in vivo, respectively.

Steerable needles

Steerable needles that adjust the needle trajectory based on steering commands have also started to be used for better insertion accuracy [47].

Ayvali and Desai [11] developed a needle tracking and localization method based on optical flow. The method took advantage of the beveled cannula tip for the out-of-plane case using circular Hough transform, see Fig. 4. Validations were done ex vivo confirming the method’s ability to detect the cannula curvature, but no quantitative analysis with respect to a gold standard on the detection accuracy was reported.

Fig. 4
figure 4

A schematic of discretely actuated steerable cannula

Full size image

Echogenic needles

Echogenic coatings or dimples created on specialized needles to improve sonographic needle visibility are another solution. The machined “Cornerstone” reflectors on the needle surface usually covering the distal portion of the cannula enhance the reflective echo from the needle end and thus improve visibility. Although such echogenic needles have demonstrated enhanced needle visibility [36], they are typically more expensive and therefore less likely to be adopted widely. Most needles for clinical use currently are still made from smooth stainless steel and come with steel or plastic styluses (solid removable cores inside the needles).

Modified transmission and image formation

Another approach to enhance needle visibility is by changing the imaging tools, or modifying the US transmit and receive beamforming sequences. Moreover, the additional dimension in 3DUS compared to 2DUS provides three useful planes of imaging to visualize the needle.

Spatial compounding

Cheung and Rohling [23] proposed the idea of beam steering technique for standard needles that adaptively steers the US beam at an angle perpendicular to the needle, which enhances the needle shaft visibility because of the strong reflections created. Needle insertion angle was estimated using the Hough transform, while slightly jiggling the needle if needed. The brightened needle in the steered image was then fused with the original image to form an enhanced image with a better visualized needle, see Fig. 5. Variations and extensions of this idea have appeared on commercial US machines.

Fig. 5
figure 5

Beam steering enhancing needle visibility: a original and b fused beam-steered image of a needle. (Reprinted with permission from Elsevier published paper: Cheung and Rohling [23])

Full size image

Hatt et al. introduced a machine learning-based approach to segment the needle in beam-steered B-mode images, by first classifying the pixels as needle or background, and then using a Radon transform to localizes the needle [24]. However, their method requires the needle orientation to be known a priori.

Zhuang et al. [44] used spatial compounding on tensor-based filtered images to enhance needle visibility for larger insertion angles. Limitations on maximum steering angle in these methods, in turn, cause limitations on steep insertions and the use of curvilinear transducers.

Curvilinear transducers and modified beamforming

Commercial US systems already have beam-steering solutions to enhance needle visibility with linear array transducers. However, the issue of needle visibility is even more challenging for curvilinear transducers, with less research compared to linear array transducers. Beam steering for curvilinear transducers remains a challenge due to the strong side lobe generated from large steering angles which may degrade the image quality due to the resulting shadows [44].

Daoud et al. [45] introduced a needle localization approach in curvilinear US for the needle trajectory. The method was validated on an agar phantom and ex-vivo tissue samples and resulted in maximum error of 1 mm and 3\(^\circ \) in needle trajectory estimation in the radial and azimuthal coordinates, respectively.

Motion analysis

Motion analysis was also explored to help with needle visibility and localization; although the focus of Doppler-based techniques was on visible needles for validation, motion-based techniques could especially benefit cases where the needle is invisible in the static image [27, 38, 53, 54].

Doppler

Harmat et al. presented a technique that augments a vibrating needle using power Doppler by moving the stylus inside the cannula [25]. They used power Doppler imaging to detect the tissue oscillations caused by rotating a slightly bent stylus inside a stationary cannula (Fig. 6). The method’s performance is mostly independent of insertion angle, depth and stylus curvature, however strongly dependent on tissue stiffness and needle visibility in the US image. The localization error tested ex vivo was reported as 2 mm.

Fig. 6
figure 6

a Schematic views of the bent stylus inside a straight cannula, and needle tip detection using power Doppler at different insertion depths: b 2 cm and c 10 cm. (Reprinted with permission from Elsevier published paper: Harmat et al. [25])

Full size image

Armstrong et al. [26] used color Doppler imaging to detect high-frequency (\(>1000\) Hz) low magnitude (max: 15 \(\mu \)m) vibrations in the needle produced by ColorMark (EchoCath, Inc, Princeton, NJ). Color Doppler gain settings have to be adjusted properly to reduce artifacts. Validations on phantoms and patients undergoing pericardiocentesis showed excellent needle tip localization.

Micro-motion

Color and power Doppler can detect velocities only within a certain range, as it is not sensitive enough to velocities that are too small, and it may suffer from artifacts for velocities that are too large. The minimum detectable frequency by Doppler is inversely proportional to the transducer’s frequency, and higher Doppler shift occurs when the direction of motion is parallel to the beam direction (shallow insertion angles). High velocity mainly occurs due to high-frequency or high-amplitude vibrations. To provide the former, additional apparatus is required such as specific actuators, and the latter would increase the risks to tissue around the insertion. Therefore, instead of using an external tool to induce vibration, manual motion on the needle could be used as an alternative to create intensity variations. The small velocity and displacements, also called micro-motion, could be detected only by software-based techniques.

Stylus motion could therefore be done similar to a well-known ad hoc technique of jiggling the needle and looking for changes in the US display. Maximum intensity projection, Hough transform and polynomial fit were used to localize a needle from a sequence of 3D images captured of a needle with manual stylus oscillations [38].

Vibrations caused by the natural tremor of the operator’s hand were also used to localize a needle by detecting regions moving at tremor frequency range. Optical flow [11, 30, 39], phase-based analysis [29, 30] and learning-based techniques could be used to enhance the needle localization. Tip localization results of 1–2 mm was reported, validated ex vivo and in vivo on porcine models. The needle localization using these techniques does not rely on initial needle visibility in the US image; however, the transducer motion should be minimized for a successful localization based on motion pattern.

Machine learning

Machine learning has gained interest in needle visualization enhancement and localization very recently [24, 28,29,30]. Machine learning-based approaches could be applied to all four categories of needle enhancement. As examples of signal and image post-processing enhancement, Hatt et al. [24] used a segmentation method using AdaBoost for statistical boosting of Log-Gabor features, to segment the needle on beam-steered B-mode images. Localization evaluation was reported based on the difference in the detected needle trajectory and the gold standard position relative to the center of the image, with up to 0.5 mm error.

Pourtaherian et al. [28] used convolutional neural networks to detect a needle in 3DUS volume. Convolutional neural networks were used to classify a voxel based on the voxels values at three orthogonal local cross sections. Validation was performed on 20 ex-vivo samples for 17G and 22G needles, resulting in up to 0.7 mm error in the localization.

Arif et al. [31] used a convolutional neural network to detect needle candidates, followed by a final tracking step to determine the needle position. Performance of the method was evaluated in phantoms and in vivo, with mean errors of 1 mm and 2\(^\circ \) in the tip and trajectory orientation, respectively.

Lee et al. [32] used LinkNet as the baseline network for needle segmentation followed by a Squeeze-and-Excitation module to obtain spatial information. Model was trained on 794 images and tested on 202 images. They were captured from kidney biopsy clips from 8 patients, with RMS orientation error of 13.3\(^\circ \).

Mwikirize et al. [33] used a digital subtraction scheme for enhancement of low-level intensity changes caused by needle tip movement as well as a deep learning scheme for tip detection. The model was trained and evaluated ex vivo on bovine, porcine and chicken phantoms. The framework was evaluated on 700 images from 20 sequences and achieved a tip localization error of 0.72 ± 0.04 mm.

As an example of motion-based enhancement, Beigi et al. [29] proposed a machine learning framework based on a probabilistic support vector machine (SVM) to classify pixels according to micro motion phase features (also in “Micro-motion” section). This method was extended using an online learning framework [55] and spatiotemporal feature selection [30], enabling needle tracking with mitigated effect from the tremor motion on the transducer. Evaluations were performed on in vivo porcine subjects with reported average error of 1.28\(^\circ \) and 0.82 mm in trajectory orientation and the tip.

Discussion

In this paper, we conducted a literature survey for techniques aiding needle enhancement in US-guided procedures. Specular reflection of the needle surface reduces needle echogenicity and thus degrades its visibility. Higher needle insertion angles result in further degradation in needle visibility. Due to speckle, signal fallout, shadowing and reverberation artifacts, standard signal and image processing methods often fail in localizing the needle. Most of the state-of-the-art techniques, however, rely on original visibility of the needle in the US image. In reality, the needle might not show up as a high intensity line of pixels and the challenge of identifying the needle when it is invisible or nearly visible still remains.

Correct localization and visualization of the needle is crucial in ultrasound-guided procedures. Some of the studies reported only the needle trajectory or only the tip, while the identification of the entire needle is left to the operator. Most of the approaches that report tip accuracy use the “drop of signal intensity” as an indicator of the tip location; they therefore rely on clear visibility of the needle shaft. Needle shaft visibility, however, is often challenging especially in applications where sensors are added to the tip of the needle or in cases where steep insertion angle is preferred. Based on our analysis shown in Table I, the literature that focused on sensors, actuators or trackers lacked the shaft localization in \(50\%\) of the cases, and only \(25\%\) of the state-of-the-art in this category reported the accuracy for both the tip and the shaft. The same pattern is observed for robot-assisted category, where \(55\%\) of the cases lacked the shaft localization and only \(30\%\) of the cases reported the accuracy for both the tip and the shaft.

Studies mainly reported the performance using localization accuracy metrics such as the needle shaft or its tip, and rarely report a “failure” rate. Based on Table I, only a small proportion of the literature (\(22\%\)) included the percentage of failed trials. Robustness is a significant factor in a clinical setting, and the percentage of cases for which the method fails would be an important metric to report in addition to other visualization or localization metrics.

Linear-array transducers were mainly the choice in the literature, whereas curvilinear transducers may bring more challenges. 3DUS remains a minor portion of the literature, perhaps because 2DUS is prominent as the faster growing point of care market. Beam steering is a promising approach in terms of clinical utility, however, steering to large angles remains a challenge, especially for curvilinear array transducers. Combining various techniques could be further researched to enhance needle localization accuracy in US-guided interventions. For example, combined ultrasound and fluoroscopy was used to enhance dosimetry for prostate brachytherapy by enhancing needle positioning [56]. Motion-based techniques and possibly robot assistance are seen as key future research directions.

Current clinical demand is inclined toward standard needles and apparatus, which makes customized apparatus harder to be adopted clinically. Machine learning-based techniques have recently come to play a role in needle detection in US. It is therefore predicted to see a trend for learning-based needle detection approaches in parallel with advances of machine learning.

Note that the inconsistency in validation approaches makes it challenging to interpret and compare different studies. From the large body of literature that we studied here, only \(33\%\) of them validated their results in vivo.

In addition, in evaluation of a new needle guidance system, a large-scale human study is preferred. We suggest in vivo animal experimentation as the closest alternative in these cases. Application-specific validation could also be useful as well, as studying the factors affecting the visualization and localization and the potential effects of the particular application could lead to key insights.