Keywords

Introduction

The systematic evaluation of accuracy and precision in orthopaedic surgery has been an open research line since the pioneering work by Simon et al. [1] at Carnegie Mellon Robotics Institute. Almost 20 years have passed and there is still no shared definition or common understanding of accuracy in computer-assisted orthopaedic surgery. This is reflected by the words of Abraham [2]: “the definition of accuracy in current navigation reports is inconsistent and can at times be misleading”. Literature presents many different definitions of accuracy and precision, and several different technical ways to acquire the data used to estimate those parameters. This situation begs at least for a brief but general review of the existing approaches to this problem and the efforts to generate a common consent.

This chapter is divided into two parts: the first part discusses the different definitions commonly found in the literature and their associated measuring methods; the second part examines the ongoing standardization efforts.

Accuracy Is Said in Many Ways

Localization Accuracy in Image-Guided Navigation

Image-guided surgical navigation systems are designed to help the surgeon in the task of correlating what it is seen in the preoperative medical images and the real anatomy of the patient. The principle behind these systems is that there exists a rigid transformation between the preoperative images and the anatomy of the patient. The process to find this transformation is called “registration” and it consists of selecting at least three corresponding pairs of fiducial points in the preoperative images and in the patient anatomy. In the best case, those fiducials are well known anatomical landmarks, but many times, especially in minimally invasive approaches on complex anatomy, those landmarks are very hard to find. Moreover, there are even more fundamental caveats in the registration process, as those described in the work by Fitzpatrick et al. [3]. In that work the authors mention that point-based registration error can be divided into three different errors:

  1. 1.

    Fiducial localization error (FLE): error in locating the fiducial points.

  2. 2.

    Fiducial registration error (FRE): statistic about the distances between corresponding fiducial points after registration (usually reported as registration accuracy by image-guided systems).

  3. 3.

    Target registration error (TRE): distance after registration between points of interest other than the fiducial points.

The main result of Fitzpatrick’s work is the derivation of the statistical distribution for TRE, but the most significant contribution from the application viewpoint is the insight about what is a good registration: a low FRE value is good, but it does not guarantee high accuracy unless other things have been taken into account, like using more than three fiducial points, placing those points far apart and surrounding the target of interest. Another important conclusion is that when “FRE falls below a certain threshold, it gives no further information regarding accuracy”.

A recent study by Stoll et al. [4] in the orthopaedic oncology domain adds a refinement step to the point-based registration. This refinement step generally depends on proprietary surface digitizing devices (surface probing, laser scanning) and computer algorithms that usually perform a small adjustment to the previous fiducial registration step. In their work, Stoll et al. found that even after the surface refinement algorithm provided by a commercial navigation system, the differences between target points and their corresponding points in the navigated images are in a 95 % CI 6.11–16.96 mm.

Evaluation of Osteotomies

Multiple forms of evaluating osteotomies have been published, depending both on the intended area of application and on the technology applied to acquire the data used to perform the evaluation. Barrera et al. [5] propose a method to assess the ‘quality’ of bone preparation for knee arthroplasty bone insertion. One of the steps in this assessment process is the estimation of each single planar cut. The method represents the accuracy using five indices for each cut. There is one translational error index (in mm) and three rotational error indices; these last three, combined, result in an overall rotational index (in degrees). The experiments are executed on synthetic knees and the ‘achieved’ surface is digitized after the cut. The article proposes several methods for capturing the surface but it also warns about the different accuracy and precision parameters of those methods.

In a classic article, Cartiaux et al. [6] propose a method based in the ISO 1101:2004 standard for geometrical tolerancing to evaluate differences between a cutting plane and a target plane. Their work shows that it is possible to express the most significant translational and rotational errors using only the location parameter (L) defined in the mentioned ISO standard. This parameter is the maximum euclidean distance from the executed cutting surface to the target plane in a perpendicular trajectory to the last one. For experimental data gathering a test bed with a block simulating bone tissue is used and errors are estimated with a coordinate measuring machine set in the same frame of reference. The method is also used in [7] for evaluating different bone cutting technologies. In this work the error (L) was 0.92 ± 0.37 mm with a robot-assisted process compared with 1.26 ± 0.88 mm with the freehand process (p < 0.0001) and 1.87 ± 2.09 mm with the navigated freehand process (p < 0.0001).

So et al. [8] introduce a new registration procedure using fluoro-CT matching and evaluate its accuracy using the postoperative gross measurement of surgical margin. The accuracy in this case is related to the planned margin, and it is not an absolute value.

Dobbe et al. [9] propose a method to measure and estimate the normal of an executed plane. This normal is used to compute the dihedral angle with the target plane, that is decomposed in sagittal and coronal plane angles. Then a distance error between the target and executed plane is computed taking the Euler distance between the centroids of the cross sections defined by target and executed planes. This method is validated using a cadaveric limb, with pre and postoperative computed tomography (CT) scans positioned in a common frame of reference using a registration algorithm. The methodological accuracy and precision is also evaluated, showing that the method introduces an error that it is well below 0.5 mm in mean.

Stiehl et al. suggest [10] using tools borrowed from the field of statistical process control in the domain of accuracy and precision evaluation in computer-assisted orthopaedic surgery. Milano et al. [11] follow that path introducing a definition of accuracy and precision that could be used with the industry proved process performance index as a clinical score and using CT scans to digitize the surgical specimen resected from the patient. The introduction of an index helps to avoid the problem of measuring the accuracy against a fixed frame of reference. This work also evaluates the methodological error. This error is below 1 mm in all the test cases. A first application of this methodology in the evaluation of surgical accuracy in 61 osteotomies performed on 28 patients is found in the article by Ritacco et al. [12]; this work shows that the accuracy parameter is 2.52 ± 2.32 mm.

In a recent article, Sternheim et al. [13] use a custom navigation system with synthetic and cadaveric pelvic bones to generate a large number of cuts. The cuts are evaluated by CT scanning the bones after the osteotomies and measuring the entry and exit cut distances and deriving the pitch and roll angle differences. In navigated cuts, using synthetic bones, the entry error is 1.6 ± 1.1 mm and the exit error is 2.3 ± 1.1 mm, while in non-navigated cuts the entry error is 2.8 ± 4.9 mm and the exit error is 3.5 ± 4.6 mm.

Patient-Specific Instrumentation

Patient-specific instrumentation (PSI) technology is an alternative to intraoperative navigation. The accuracy of PSI technology adapted for bone tumor surgery has been studied by Cartiaux et al. in [14]. In that article experiments are conducted using synthetic right hemipelvic bone models that are fixed to a test bed, setting a global reference frame for measurement. The test bed is digitized using a CT-scanner and a simulated tumor is introduced in the bone. A mixed seniority group of 24 orthopaedic surgeons performs the bone cuts with a pneumatic oscillating saw. The experiments show that the location accuracy of the cut planes varies significantly in terms of mean and 95 % confidence interval (CI) among the four target planes. The average location accuracy in the anterior and posterior ilium is 1.0 mm (CI 0.8–1.3 mm) and 1.2 mm (CI 0.9–1.6 mm) respectively and it is significantly different from the average in the pubis and ischium, 2.0 mm (CI 1.5–2.7 mm) and 3.7 mm (CI 2.8–4.9 mm) respectively. The surgical margins achieved in the pubis, with an average of 11.8 mm (CI 11.3–12.3 mm), were significantly higher than those achieved in the ischium and anterior and posterior ilium, with an average of 9.2 mm (CI 8.6–9.7 mm), 10.0 mm (CI 9.5–10.6 mm) and 9.7 mm (CI 9.2–10.3 mm) respectively.

Standardization Efforts

Proposal for New ISO Standardization Activities

So far, there is no standardization work within ISO and IEC directly related to accuracy measurement in CAOS.

In 2004, the International Society for Computer Assisted Orthopaedic Surgery (CAOS-International), in conjunction with the American Society for Testing of Materials (ASTM), undertook the creation of a new ASTM standard for assessing and comparing the performances of CAOS systems [10]. This standard was published in 2010 as ASTM F2554-10 [15] and used the definition of accuracy and precision parameters provided by ASTM standard E177-08 [16]. The standard F2554-10 is used to define the technical specifications (accuracy and precision) of navigation systems and positioning robots for CAOS [17, 18]. In consequence, it cannot be used directly for measuring the accuracy of bone-preparation tasks, but the standard claims that the logical continuation will be to work on additional standards that will address task-specific procedures and surgical applications (joint arthroplasty, osteotomy, tumour biopsy and/or resection, laparoscopy, pedicle screw insertion, brain surgery, and so forth).

Since October 2013, subcommittee ASTM F04.38 has launched a new work item entitled “WK41641 New test method for mechanical influence on computer assisted surgical system accuracy”, aiming to measure the effects of the operating room environment on the accuracy of computer aided surgical systems in relation to the equipment utilized for bone preparation. This work item recently resulted in a new standard published in 2014 as ASTM F3107-14 “Standard Test Method for Measuring Accuracy after Mechanical Disturbances on Reference Frames of Computer Assisted Surgery Systems” [19]. Even if this new standard is clinically relevant for CAOS surgeries, the resulting standard will not be able to be used directly to measure the accuracy of the bone-preparation tasks.

The Standard ISO 5725–1:1994 “Accuracy (trueness and precision) of measurement methods and results – Part 1: General principles and definitions” outlines the general principles to be understood when assessing accuracy (trueness and precision) of measurement methods and results [20]. This standard is significant for the standardization activities proposed here because it forms a relevant basis for defining the terminology of accuracy and accuracy measurement in CAOS.

The Standard ISO1101:2012 “Geometrical product specifications (GPS), Geometrical tolerancing, Tolerances of form, orientation, location and run-out” has been used since the 80s in mechanical engineering to define the geometrical tolerances of mechanical parts, regardless of the fabrication process [21]. This standard is significant for the standardization activities proposed here because it has already been in use for bone tumour surgery since 2009 [6, 7, 14, 22–24], considering bone as a material with specific mechanical properties, to define geometrical tolerances and to assess the accuracy of planar bone-cutting, regardless of the assisting technologies used to execute the bone cuts.

By focusing on systematic and global methodologies and approaches for accuracy measurement of bone-preparation tasks in CAOS, the standardization activities that we propose now for ISO are the logical continuation of the previous and current standardization works made by ASTM and CAOS-International concerning the intrinsic performances of surgical assistance systems.

We aim to produce a new consensus-based international standard on accuracy measurement in computerassisted orthopaedic surgery (CAOS), including the terms and definitions concerning accuracy and accuracy measurement in CAOS, and the methods for measuring accuracy of bone-preparation tasks inCAOS (bone-cutting, bone-drilling and bone-assembly). In addition, we aim to produce an informative technical document to provide the users with practical guidance to the clinical use of the new standard within the workflow of orthopaedic interventions such as joint arthroplasty, spine instrumentation, corrective osteotomy, fracture reduction, bone tumour resection, and so forth.

Proposed Programme of Work

First, we will draw up the terminological basics, including the terms and the definitions concerning accuracy and accuracy measurement in CAOS. These basics will provide the necessary elements for a common language for all the activities of accuracy measurement in CAOS, and especially for consistently understanding and applying the accuracy measurement methods that will be developed for bone-cutting, bone-drilling and bone-assembly. Practically, the basics will include: the elements to define the geometrical specifications of desired bone cutting, drilling and assembly; the elements to define the geometrical accuracy of performed bone cutting, drilling and assembly; the metrology elements to quantify the geometrical accuracy of performed bone cutting, drilling and assembly; and finally the statistical elements to compare the geometrical accuracy of performed bone cutting, drilling and assembly with respect to the desired geometrical specifications.

Second, we will draw up methods for measuring accuracy of bone-cutting, bone-drilling and bone-assembly. Each accuracy measurement method will take the form of a systematic step-by-step approach starting with defining the desired geometrical specifications and then enabling to measure and evaluate the accuracy of performed bone cutting, drilling and assembly. These accuracy measurement methods will be regardless of the CAOS assisting technologies and the systems that could be used during the surgery to assist for the execution of bone cutting, drilling and assembly, such as surgical navigation systems, surgical robots, patient-specific instruments, and so forth.

In consequence, the standardization activities proposed here will result in a new standard document consisting of four parts: the first part for the terminology and the second, third and fourth parts for the three systematic methods for measuring accuracy of bone-cutting, bone-drilling and bone-assembly respectively. This resulting standard will be technical and probably not be able to be used directly in clinical routine. So we propose a second phase in our work program as the following.

In conjunction with the new International Standard, we will draw up a Technical Report as an informative guidance document. The purpose of this Technical Report is to provide the users with practical guidance to the use of the new International Standard for designing and implementing new quality evaluation protocols within the surgical workflow of orthopaedic interventions involving bone-cutting, bone-drilling and/or bone-assembly. Practically, involved orthopaedic interventions are the following:

  • Bone tumor surgery: to measure the accuracy of the bone cutting and assembly that are necessary to resect the bone tumor in safe margin and reconstruct the bone defect with a massive bone graft or a prosthesis.

  • Spine surgery: to measure the accuracy of the bone drilling necessary to insert screws safely within the pedicles of the vertebrae.

  • Knee surgery: to measure the accuracy of the bone cutting and drilling that are necessary to prepare the placement of the femoral and tibial prosthesis components.

  • Hip surgery: to measure the accuracy of the bone drilling that are necessary to insert internal screws or nails to reduce and stabilize a femoral neck fracture.

  • Corrective surgery: to measure the accuracy of the bone cutting and assembly that are necessary to reposition bone fragments and correct a malunited bone fracture.

The standardization activities proposed here will also consider the variety of technologies and systems that could be used during the intervention, not to execute desired bone cutting, drilling and assembly, but to measure the accuracy of performed bone cutting, drilling and assembly. Such technologies and systems can be intraoperative CT or fluoroscopic images to perform image registration and bone segmentation, navigation systems and robots to perform real-time tracking and tool localization, and so forth. Commonly accepted recommendation is to minimize the measuring errors by using measurement procedures and systems with an accuracy of an order of magnitude much greater than the errors expected during the execution of the bone-preparation tasks. The minimization of measuring errors is complex because it accounts for system calibration process, construction of reference frames, transformation and registration process, and so forth.

Protocols for measuring positional accuracy of surgical tracking systems will not be covered because they are already covered by ASTM standard F2554-10, however they will be of significant importance for the standardization works proposed here.

Expected Contributions

The relevant affected stakeholders can be listed as the following:

  • Orthopaedic surgeons and hospitals with orthopaedic surgery departments

  • University laboratories active in CAOS research

  • Industrials active in the field of medical and surgical devices in orthopaedics (implants, instruments, computer-assisted systems, etc.)

  • Regulatory Agencies

The benefits for stakeholders can be listed as the following:

  • Surgeons: practical step-by-step guidance to peroperatively measure the accuracy of bone cutting, drilling and assembly in CAOS with respect to a preoperative desired planning.

  • Researchers: standardization can contribute to the validation and integration of new CAOS technologies that are still in prototyping in the research laboratories.

  • Industrials: these activities can contribute to the assessment of new technologies that are ready for the marketplace.

Overall, standardization can improve the work of, and the communication between, surgeons, researchers and the regulatory agencies. This can push forward a common language to define and quantify the quality of the bone-preparation tasks before we can correlate the improved accuracy with clinical outcomes.

The contributions of the proposed standard can then be listed as the following:

  • Societal benefit: to better know the added value of CAOS technologies and secure acceptance of the notion of quality evaluation of bone-preparation tasks executed with the aid of CAOS technologies;

  • Scientific benefit: to propose a common language for clinicians and researchers to assess the accuracy of CAOS interventions, before we can correlate improved accuracy with functional outcomes during future international long-term follow-up studies;

  • Technological benefit: to facilitate and improve the clinical integration of CAOS technologies and their use in clinical routine;

  • Economic benefit: to increase the use of assistance technologies for orthopaedic surgery.

Toward a First International Workshop

As a result, a work group composed of engineers, surgeons and industry representatives at the international scale has to be formed. We are planning to submit the proposal in 2015 to the ISO Central Secretariat (Geneva) for the development of the new consensus-based standard on accuracy measurement in CAOS. We also believe this is the appropriate time for the CAOS community to initiate a discussion on accuracy standardization. With this in mind, and as a first step, we would like to propose a first international workshop to communicate on this area and solicit the participation and gauge the level of interest of the CAOS community towards the proposed standardization activities.

Conclusion

The objective evaluation of accuracy and precision in computer-assisted orthopaedic surgery is crucial, on the one hand, to assess the performance of different tools and processes applied nowadays during everyday practice. On the other hand, a proper evaluation of future developments in orthopaedic surgery depends to a great extent on the possibility of objectively measuring the performance of new tools and methods during surgical procedures. This review chapter briefly describes the different approaches and achievements of the research line working toward those aims. There is still much work to be done, since computer-assisted surgery is being adopted by and adapted for new orthopaedic applications. The international standardization effort is a well-focused but recent project that, in our opinion, will start producing concrete results in 3–5 years. These are exciting times to work in this field, when exact definitions and formalization of previous intuitive knowledge open the gate for new developments.