Micro-information-level AR instruction: a new visual representation supporting manual classification of similar assembly parts

Abstract

In AR operation guidance training, for assembly parts with similar geometric shapes, there are still two problems in the visual representation of AR instructions: (1) AR instructions cannot accurately represent the micro-geometric differences between similar parts. (2) The AR instruction design specification that reflects the micro-geometric differences of similar parts has not been formulated. Based on such a problem, our team has carried out the following research work: First, the geometric features of parts are defined at the micro-geometric level and micro-information level, thereby explaining the relationships and differences between similar part features. From the above two levels. Secondly, a mathematical model of the geometric features of the parts is established, and the control parameters in the model are given to characterize the feature differences between similar parts. To verify the accuracy of the control parameters, we designed three AR instructions based on the control parameters and verified them through five hypotheses. Our team then analyzed the data from a case study and focused our discussion on test results that did not meet expectations. We have a more in-depth discussion by comparing the differences and analyzing the results. Finally, three implications of AR instructions in representing feature differences between similar parts are given, and future research directions for such work are indicated. It is provided guidance for future research.

1 Introduction

This paper introduces a micro information-level AR instruction that supports the manual classification process of similar assembled parts. This study extends the manual micro-guidance research carried out by our research team [23, 24, 26], focusing on how to design a visual representation of AR instructions to help operators quickly and accurately start from a batch of parts with similar geometries. Research is carried out to improve the screening accuracy and subsequent operation efficiency.

With the improved rendering performance of AR devices such as HoloLens 2,¹ Magic Leap 2² and Google Glass Enterprise Edition,³ AR instructions can provide more realistic visual guidance for operators to perform high-precision training tasks in many industrial scenarios, such as mechanical equipment assembly/disassembly [21, 24], training [10, ] and maintenance [2, 18]. Recently, some important researches focus on integrating assembly intent cues into AR instruction design to enhance the visual representation of target geometry, such as Seeing is believing [4], Mini-Me [16], 3DGAM [27], SHARIdeas [25] etc. In addition, Wang et al. [28] proposed a new AR instruction manipulation platform to share assembly intent cues in a user-oriented remote collaboration environment, and designed a complete workflow for running information-level AR instructions.

However, in AR assembly training, especially in the scenario of finding target parts among a batch of geometrically similar parts, there are few studies on combining 3D CAD models with user visual cognitive rules. Currently, Model-Based Definition (MBD) is a method of creating a 3D CAD model that includes all the geometric data needed to define an assembly part and AR instructions that indicate how to manipulate the assembly part to achieve a specific assembly intent [19]. However, the AR instructions defined by MBD cannot directly and vividly describe the subtle differences between two similar parts, such as differences in roughness, waviness, surface manufacturing defects, etc. This creates confusion for the operator’s screening process. Therefore, there is an opportunity to explore whether the combination of 3D CAD models with user visual cognition rules can improve AR-based visual representation of assembly part micro-information.

Several studies have investigated the visual representation of AR instructions at the micro-geometric level and used high-quality 3D CAD models to describe geometric differences between similar parts [3]. This approach is called microgeometry-level AR instruction design (MGAI). However, these studies can enhance the visual representation of instructions by integrating the user’s visual cognitive rules. For industrial training and maintenance procedures, C.Y. Siew et al. [18] introduced a new adaptive human-machine interface module in the Augmented Reality Maintenance System (ARMS). Their research shows that 3D CAD models with high rendering quality can further improve the quality of assembly training with the help of user cognitive cues such as visual manipulation guidance, tactile manipulation guidance, etc. Wang et al. [26] proposed a visual representation of assembly instructions for AR micromanipulation, and explored the impact of traditional AR instructions and AR instructions with user visual cognitive rules on physical tasks. They found that instructions combined with visual cognitive rules have a positive impact on human-machine collaborative assembly, and information-level AR instructions are the key to characterize assembly intent data in assembly tasks. On the basis of these studies, how to combine the 3D CAD model of the assembled parts with the user’s visual cognition rules to realize the visual representation of the geometric differences of similar parts, there are still some problems to be studied.

In this study, our team explore how micro-information-level AR instructions (MIAI) that integrate 3D CAD models and user visual cognitive rules affect the representation of micro-geometric information in a set of similar geometric objects. Our research hopes to make the following contributions:

• 1. Implementing a visual representation of micro geometric information of assembly object, which supports AR instruction and represents the assembly intention data behind a assembly object to operators.

• 2. Our case study is the first pilot user study to evaluate AR instruction design rules at the level of micro-information awareness.

• 3. Providing instruction design implications for using MIAI in AR micro-assembly interface.

The structure of this paper is shown in Fig. 1.

Fig. 1

The Mind map of micro information-level AR instruction

2 Related works

2.1 Illustration

In this section, the analysis of data obtained through specific operations will focus on the specific application of AR instructions in physical tasks with high precision operation requirements. These studies attempt to rationally design AR teaching to improve the visual expression ability of teaching content for geometric differences of similar parts. This section will introduce the changes in the operation accuracy brought about by the AR command in detail by point of view and step by step.

2.2 Visual representation of AR instruction with high visual complexity

Many researchers have studied the application of MBD-based AR teaching in reducing the cognitive interference of the visual complexity of teaching content to the human brain. Liu et al. [13] designed a MBD-based CPMT smart window to support part machining. A smart window is essentially an advanced Human Machine Interface (HMI). It provides the operator with the manufacturing geometric data involved in the part machining process through AR commands, effectively representing the geometric information of the machined part, and further improving the operator’s visual cognition level of the machine tool machining process. Gattullo et al. [6] proposed a new approach to transform “traditional” instructional content with high visual complexity. The approach is based on using ASD to simplify the optimization of technical English usage of text, converting instructions into 2D graphic symbols defined by MBD, and structuring content by combining Darwinian Information Type Architecture (DITA) and Information Mapping (IM). Vanneste et al. [20] compared the effects of verbal, paper, and AR instructions on physical tasks with high operational precision requirements. This study shows that AR teaching reflects the geometric characteristics of target objects better than oral and written teaching. AR commands were superior to verbal commands in terms of complexities of help-seeking behavior and perception. In addition, they observed that operators using AR instructions with high visual complexity experienced little internal stress on the first assembly attempt, and there was no evidence that such instructions were detrimental to the user in terms of productivity, physical effort, or ability frustration influence. Raji et al. [15] tried to improve the visual performance of AR commands for target objects from the perspective of improving the quality of image transmission. The team weighed wavelet transform performance against future wireless application system requirements, and proposed guidelines and methods for wavelet applications in 5G waveform design. They found that OFDM based on wavelet transform can provide better image data for AR commands, thus providing clearer and more accurate video information for target users. In addition, Nurelmadina et al. [14] hold the same view. They found that the effects displayed on cognitive radios in LPWAN could significantly transform commercial IoT, especially in improving the visual representation environment that supports AR instruction.

In recent years, some researchers have noticed that the visual complexity of AR teaching may be closely related to the user-centered teaching design and the human brain’s visual cognition of instruction content. Lai et al. [11] designed a user-centered visual representation system for AR teaching. The system consists of multi-modal augmented reality (AR) instructions and performs manufacturing feature detection with the support of a deep learning network. Based on the data extension of the CAD model, a comprehensive tool dataset was developed and successfully deployed to detect real tools. Compared with traditional paper instructions, AR instructions can communicate more fully to the operator with less visual resources, thereby reducing the working time and operation errors of assembly tasks by 33.2% and 32.4%, respectively. Wang et al. [28] proposed a user-centric assembly feature definition and visual representation design for AR instructions. They explained the geometric-level and information-level differences of assembly features in manufacturing semantics, and evaluated the impact of AR instructions based on information-level manufacturing semantics on users’ visual cognition level, assembly efficiency, visual experience, mental load, etc. They analyzed that information-level AR instructions consume less assembly resources than geometric-level AR instructions, and can achieve better ergonomics with less information transmission. Laviola et al. [12] explored the possibility of using a new “minimal AR” authoring method to optimize visual resources for communicating manufacturing work orders in an augmented reality (AR) interface. The introduction of too many visual resources will lead to higher visual complexity of the corresponding teaching content, but it will not significantly improve the work performance of operators, and even lead to problems such as prolonged completion time and increased mental burden. So that in some specific cases it will increase the operation error.

The above studies show that incorporating MBD and user-centered semantic definitions into AR instruction design can effectively reduce the visual complexity of instruction representation. However, how AR instructions reflect geometric differences between similar parts (such as differences in surface roughness) at the semantic level of microinformation has not been well explored. Therefore, further research and analysis are still needed to summarize the data to improve the corresponding deficiencies of MBD.

2.3 Visual representation of AR instruction with poor visual field condition

There has been a lot of research on using AR to guide work activities in poor field of vision. Wang et al. [22] proposed a machine vision-based augmented reality blind spot assembly method. The visual representation of AR instructions is realized by projection, and the assembly is accurately guided by the principle of local error amplification. They found that adding highlighted visual elements and visual orientation markers to AR instructions improved their assembly guidance in poor visibility conditions. Shin et al. [17] compared the expression effects of AR instructions with 6 different narrative experience factors on the working intention of manipulation tasks under adverse visual conditions from the perspective of user visual cognition. They found that while visual clusters centered on the visual representation of AR teaching were suitable for higher presence in a sufficiently large space, the same layout resulted in lower narrative engagement for AR teaching. Based on their findings, the team proposes guidelines to enhance the effects of spatial adaptation by considering how users perceive and navigate the augmented spaces produced by different physical environments. Chu et al. [3] proposed and developed a new visual representation of AR instructions to facilitate manual assembly, including visible hidden areas in the work environment, and avoid manual manipulation. The focus of this study is to verify the validity of various auxiliary information in AR commands, including the assembly interface, the operator’s hand movements, and the operator’s parts. Their initial design of the hand model provides the operator with visual cues to quickly and roughly locate the assembly interface in a large, invisible area, and then pinpoint it to a smaller area guided by tactile sensing. They found that the effect of incorporating fixed components was not significant, as positional deviations between real and virtual objects may degrade human hand-eye coordination. Recently, Wang et al. [26] proposed a high-precision cognitive AR instruction (M-AR) for manual operation guidance, which aims to convert high-precision assembly relationships into a series of visual information to satisfy users’ perception of high-precision cognition of precision task intent. They found that M-AR with heuristic visual cognitive elements could help the user’s brain correlate the working process of assembling activities under poor visual field conditions, thereby providing the user with a better visual experience. In order to solve the hand-eye coordination (HEC) problem. Feng et al. [4] proposed a visual representation method of AR instructions based on HEC mode. This method defines a comprehensive and accurate model of hand-eye coordination (HEC). They found that the visual representation of the manipulation direction is the first visual information to be considered for blind spot assembly. The AR instructions that incorporate these visual elements can effectively reduce the visual cognitive difficulty of the operator in the case of poor vision. The above studies show that the research on the visual representation of blind-spot operations is of great significance for improving the safe operation of AR instructions in physical tasks with high-precision operation requirements. However, AR visual representations of special properties involved in physical tasks, such as geometric differences between similar parts in industrial assembly training, have not been intensively studied. Therefore, based on the above experimental analysis, our team still needs to conduct in-depth research on the AR visual expression effect of special attributes.

2.4 Summary

From the studies discussed above, three conclusions can be drawn. First, microgeometry-level AR instructions are ill-defined. Previous research has mainly focused on the visual representation of microgeometric AR instructions (MGAI), using AR instructions containing simple graphic symbols (defined by MBD) to describe special properties (such as surface roughness) [5]. However, MGAI cannot directly represent the geometric differences between similar parts, which makes it difficult for operators to visually distinguish the differences between a group of similar assembly objects. Second, the information-level AR instructions [24, 25] cannot well describe the geometric differences between similar components. A problem that may be overlooked by this type of research is that information-level AR instructions cannot define the working intention behind specific attributes, and the related AR instruction design paradigm has not been clearly explained, which is the shortcoming of this type of research at present. In addition, micro-information in the fields of assembly training, education, and medical care is usually defined by the specific technical requirements of the field, while the visual representation of micro-geometric information in AR instructions has no technical requirements to follow, which makes it difficult to be accepted by peers. is the evaluation result of the representative information-level AR instruction. These problems forced our team to seek to design a new AR instruction to solve the representation problem of special properties.

3 Our approaches

3.1 Definition of MGAI and MIAI

On the basis of the previous research of the research group, the AR instruction representation of micro-geometric information is divided into micro-geometry-level AR instructions (MGAI) and micro-information-level AR instructions (MIAI). Among them, micro-geometric information refers to some special properties of geometric objects, which are difficult for operators to distinguish visually, such as roughness, waviness, surface defects of parts, etc. Micro-geometric visual design (MGV) is a method of expressing geometric differences between similar parts using special symbols defined by MBD. The AR instructions constructed by the MGV are called microgeometry-level AR instructions (MGAI). This instruction follows the visual representation paradigm of the previous AR instruction [7,8,9]. The digital information defined by MBD is directly transplanted into the industrial virtual reality space for application, and will not significantly improve the adverse effects of AR commands on the operator in terms of visual complexity and field of view conditions. Micro-information-level visual design (MIV) is a method of expressing geometric differences between similar parts using special graphics defined by the laws of visual cognition. The laws of visual cognition refer to a series of operational skills extracted from the operator’s work training performance. Defined in terms of operational standards, special graphics defined by the Law of Visual Cognition interpret standards as specific visual elements that represent operational logic. AR instructions composed of these visual elements are called micro-information-level AR instructions (MIAI).

3.2 Prototype system

In this section, we introduce the design and implementation details of our prototype system, including the instruction data processing framework, the mathematical interpretation of the instruction data processing procedure, and the training tasks. Through the above introduction and analysis, our understanding of improving the accuracy of operation and simplifying the operation process will be significantly improved, as follows.

3.2.1 Micro data processing framework

Our team believes that micro data processing can be divided into two stages: data processing at the micro geometric level and data processing at the micro information level. Micro information-level data processing is the reprocessing of micro geometric level data processing results. The resulting MIAI represents the inheritance and upgrading of MGAI. On the one hand, it uses some graphical symbols defined by MBD. On the other hand, the design idea of visual cognitive law is integrated into MGV to further improve the representation effect of AR instruction on geometric differences between similar parts. As shown in Fig. 2, the relationship between two stages is illustrated.

Fig. 2

Flow chart of micro information-level AR instruction

Micro AR instruction, such as MBDAI, MGAI AND MIAI, are the link of information interaction between workers and an agent-based AR instruction system. That is, micro AR instruction is not only the output data of Agent+AR instruction system, but also the input data of user’s cognitive system. In the agent-based AR system, measurement data refers to the manufacturing data collected by the sensor from the manufacturing location, such as geometric size, shape, location, surface quality, etc. The agent is the core of measurement data calculation and data processing decision-making, it processes the measurement data into microscopic information, such as shape error, position error, roughness, waviness, etc. It is often difficult for the user’s eyes to visually see this information from the surface of the component. The user needs to use the evaluation parameters (micro information) output by the agent to perceive the current state of the component.

In the first case, some studies [12] directly normalize micro-information to generate MBD-based AR instructions (MBDAI). MBD is a method of integrating dimensional and tolerance symbols into the 3D CAD model of a part and providing a comprehensive description of the product with the help of manufacturing documentation on the surface of the target part. This is a way to fully express product-defining information using an integrated 3D CAD model. In the corresponding physical model, graphics and texts describe manufacturing information in detail, such as product dimensions, tolerances, and manufacturing process requirements. The AR instruction only treats micro-information as general graphic information, and does not consider how to improve the user’s comprehension. Users still need to rely on abstraction, data mapping, and information cognition to translate such AR commands into understandable content (manual rules). In the second case, some researchers [1, 13, 29] also perform general intelligent processing of micro-information, which is equivalent to the abstract processing of visual information by the brain. Universal Intelligent Processing (UIP) can transform the abstract concepts reflected by microscopic information into microscopic geometric information, and reasonably link the user’s cognition of target geometry with the microscopic information. The visual rendering module combines micro-geometry-level information with digital instruction resources to generate micro-geometry AR instructions (MGAI). In the third case, our team proposes a new micro-information processing method that utilizes advanced intelligent data processing modules to further process the micro-geometric information into micro-cognitive information. Advanced intelligent data processing is a data relationship mapping calculation similar to the user’s brain. It can transform microscopic geometric information into microscopic cognitive information with visual cognitive cues. This transformation enables visual cognitive laws to be well integrated into the visual representation of AR instructions to meet the individual visual cognitive needs of the target object. The visual rendering module combines micro-cognitive information with digital resources, generates micro-information-level AR instructions (MIAI), and informatizes, visualizes, summarizes and displays data through analysis and statistics. Three different situation analysis studies have compared the different operational demonstrations of AR commands in different levels of regions and the remaining problems.

3.2.2 Mathematical interpretation of micro data processing workflow

In user’s cognitive system, no matter what the AR instruction is, it is necessary to integrate the information contained in the AR instruction with the physical resources in the real scene through mental rotation, so as to establish a three-dimensional understanding of the tasks performed in the real scene. Abstract processing module processes AR instruction based on MBD and transforms it from standard information to micro geometric information. The data relation mapping module processes AR instruction at micro-geometry level, and transforms this micro geometric information into micro cognitive information. Users use information cognitive module to transform micro cognitive information into MIAI, quickly discover the manufacturing relationships contained in instruction content, and use these visual cues to form manual rules to meet their cognitive needs. This module often transforms very complex tasks into several simple rules, which enables users to complete operation tasks better under the premise of ensuring work efficiency.

In this section, the data processing conversion relationship between MBDAI, MGAI and MIAI under micro data processing framework will be explained by a mathematical model. A physical task consists of several work units, which can be described by formula (1).

$$ {Y}_A=\left\{{Y}_1,{Y}_2,\dots, {Y}_i,\dots, {Y}_{n-1},{Y}_n\right\} $$

(1)

In Formula (1), Y_A represents a physical task and Y_i represents a work unit.

$$ {Y}_i=\left\{F\left({X}_1\right),F\left({X}_2\right),\dots, F\left({X}_k\right),\dots, F\left({X}_n\right)\right\} $$

(2)

In formula (2), F(X_k) represents a kind of micro AR instruction describing work unit, and F represents the data processing relationship between micro information and micro AR instruction. X_k is a micro information set composed of multiple micro information, which can be expressed by formula (3).

$$ {X}_k=\left\{{x}_k(1),{x}_k(2),\dots, {x}_k\left(\delta \right),\dots, {x}_k\left(n-1\right),{x}_k(n)\right\} $$

(3)

In formula (3), x_k(δ) represents the microscopic information of article δ.

$$ F\left({X}_k\right)=\left\{f\left({x}_k(1)\right)\cdot {x}_k(1),f\left({x}_k(2)\right)\cdot {x}_k(2),\dots, f\left({x}_k(i)\right)\cdot {x}_k(i),\dots, f\left({x}_k(n)\right)\cdot {x}_k(n)\right\} $$

(4)

In formula (4), f (x_k(i)) represents a computing mechanism for processing micro information. In fact, different computing mechanisms will use different data processing methods, such as universal intelligent data processing, advanced intelligent data processing and so on. That is, all coefficients ( f(x_k(1)),f (x_k(2)),…,f (x_k(i)),…, f (x_k(n))) will be equal to a specific micro information expression formula.

(1) AR instruction based on MBD.

MBDAI does not use the computer system of two intelligent processing modules to process micro information, but directly inputs micro information into user’s cognitive system to let a user’s brain complete the corresponding data processing work. Therefore, all coefficients are 1.

$$ F\left({X}_k\right)=\left\{{x}_k(1),{x}_k(2),\dots, {x}_k(i),\dots, {x}_k(n)\right\} $$

(5)

By substituting formula (5) and (2) into formula (1), our team can get that:

$$ {Y}_A=\left\{{x}_1(1)\dots {x}_1(n),{x}_2(1)\dots {x}_2(n),\dots, {x}_n(1)\dots {x}_n(n)\right\} $$

(6)

This formula represents a data processing result of MBDAI.

(2) micro geometric level AR instruction.

MGAI uses a micro information conversion mechanism of universal intelligent data processing module (UIP) to calculate micro information, and inputs a processed information result into user’s cognitive system in the form of micro geometric information. f₁(x_k(i)) denotes a data processing method in UIP module.

$$ f\left({x}_k(i)\right)={f}_1\left({x}_k(i)\right) $$

(7)

By substituting formula (7) into formula (4), our team can get that:

$$ F\left({X}_k\right)=\left\{{f}_1\left({x}_k(1)\right)\cdot {x}_1(1),\dots, {f}_1\left({x}_k(n)\right)\cdot {x}_1(n)\right\} $$

(8)

$$ {Y}_A=\left\{{f}_1\left({x}_1(1)\right)\cdot {x}_1(1),\dots, {f}_1\left({x}_1(1)\right)\cdot {x}_1(n),\dots, {f}_1\left({x}_n(n)\right)\cdot {x}_n(1),\dots, {f}_1\left({x}_n(n)\right)\cdot {x}_n(n)\right\} $$

(9)

Formula (9) represents a data processing result of MGAI.

(3) micro information level AR instruction.

On the basis of UIP, MIAI uses a data conversion mechanism of advanced intelligent data processing module(AIP) to calculate micro information. This mechanism further transforms micro geometric information into micro cognitive information, so that the input instruction content of user cognitive system can bring better psychological experience to user cognition. f₂(x_k(i)) denotes a data processing method in AIP module.

$$ f\left({x}_k(i)\right)={f}_1\left({x}_k(i)\right)\cdot {f}_2\left({x}_k(i)\right) $$

(10)

By substituting formula (10) into formula (4), our team can get that:

$$ F\left({X}_k\right)={f}_2\left({x}_k(1)\right)\cdot \left\{{f}_1\left({x}_k(1)\right)\cdot {x}_k(1),\dots, {f}_1\left({x}_k(n)\right)\cdot {x}_k(n)\right\} $$

(11)

Formula (11) shows that MIAI has one more coefficient than MGAI, which represents the data processing process of MIAI.

$$ {Y}_A={f}_2\left({x}_1(1)\right)\cdot \left\{{f}_1\left({x}_1(1)\right)\cdot {x}_1(1),\dots, {f}_1\left({x}_1(1)\right)\cdot {x}_1(n),\dots, {f}_1\left({x}_n(n)\right)\cdot {x}_n(1),\dots, {f}_1\left({x}_n(n)\right)\cdot {x}_n(n)\right\} $$

(12)

Formula (12) represents a data processing result of MIAI.

3.3 Summary

This section gives the definitions of Microgeometry Level AR Instructions (MGAI) and Micro Information Level AR Instructions (MIAI). MGAI is a method for indirectly expressing special properties through microscopic geometric information. To improve the ability of AR instructions to express special properties, we further extend the microscopic geometric information involved in MGAI to microscopic cognitive information. Through information reprocessing, specific micro-cognition and micro-expression are presented to users. This is an innovative way of expressing. Micro-cognitive information is the core of MIAI, and its purpose is to meet the cognitive needs of users. MIAI is better than MGAI because it conforms to the user’s cognitive habits. In Fig. 2, there is a clear boundary between MGAI and MIAI, which is a visual cognitive law derived from user cognition. In addition, the micro-cognitive information is evolved from the micro-geometric information described by Eq. (12). Next, we will introduce related content to demonstrate two levels of visual information and discuss the specific steps of interface guidance. Its purpose is to more clearly show how the micro-cognitive information can be expressed more concisely and applied to the actual tedious operation and assembly.

4 Special attributes & visual representation

4.1 Illustration

In this section, our team will demonstrate the existence of MBDAI, MGAI and MIAI by examining the surface roughness of the parts. The surface roughness will be displayed to the user through AR commands in three different situations. In our physics tasks, the relevant items mainly involve surface roughness, and the surface data reflects the specific operating parameters, which can provide the user with working instructions. Specific data analysis and application of formulas will be presented to the reader.

4.2 Related items for surface roughness

Take into account the surface of the machined part seen around it. These surfaces come in a variety of finishes with varying characteristics, from shiny smooth to rough and matte. These external differences are due to the different surface roughness of each part. Surface roughness is a tiny geometric feature that consists of closely spaced peaks and valleys on the surface of a part. If you look further at the machined parts, you will clearly notice that their surfaces are complex shapes consisting of a series of crests and troughs of varying heights, depths and spacing. The surface micro-roughness of the part has a great influence on the surface roughness. Secondly, the surface roughness of the parts is also a technical indicator for evaluating the surface quality of the parts. It can significantly affect part compatibility, working accuracy, wear resistance, corrosion resistance, sealing and appearance. On the premise of ensuring the performance of the machine, in order to obtain the corresponding surface roughness of the parts, an appropriate processing method should be selected and the relevant parameters of the surface roughness should be controlled. In fact, more than one parameter is used to evaluate the surface roughness of a part. Different parameter references will be reflected in specific geometric application characteristics during different operations.

Arithmetical mean roughness (Ra): arithmetical mean of the sums of all profile values. Also referred to as the arithmetic mean surface roughness. In Fig. 3, Ra means the value obtained by formula(13) and expressed in micrometer (μm) when sampling only the reference length from the roughness curve in the direction of the mean line, taking X-axis in the direction of mean line and Y-axis in the direction of longitudinal magnification of this sampled part and the roughness curve is expressed by y = f (x):

$$ Ra=\frac{1}{Q}{\int}_0^Qf(x) dx $$

(13)

Fig. 3

Graphic representation of arithmetical mean roughness

Average peak-to-valley profile roughness (Rz): The average peak-to-valley roughness based on one peak and one valley per sampling length. In Fig. 4, Rz shall be that only when the reference length is sampled from the roughness curve in the direction of the mean line, the distance between the top profile peak line and the bottom profile valley line on this sampled portion is measured in the longitudinal magnification direction of roughness curve and the obtained value is expressed in micrometer (μm). When finding Rz, a portion without an exceptionally high peak or low valley, which may be regarded as a flaw, is selected as the sampling length (see formula (14)).

$$ Rz= Rp+ Rv $$

(14)

Fig. 4

Graphic representation of average peak-to-valley roughness

Ten-point mean roughness (RzJIS): In Fig. 5, RzJIS shall be that only when the reference length is sampled from the roughness curve in the direction of its mean line, the sum of the average value of absolute values of the heights of five highest profile peaks (Y_p) and the depths of five deepest profile valleys (Y_v) measured in the vertical magnification direction from the mean line of this sampled portion and this sum is expressed in micrometer (μm).

$$ \boldsymbol{RzJIS}=\frac{\left({Y}_{p1}+{Y}_{p2}+{Y}_{p3}+{Y}_{p4}+{Y}_{p5}\right)+\left({Y}_{v1}+{Y}_{v2}+{Y}_{v3}+{Y}_{v4}+{Y}_{v5}\right)}{5} $$

(15)

Fig. 5

Graphic representation of ten-point mean roughness

The above three parameters can be used to evaluate the surface roughness of parts. To simplify our operational tasks, our team only uses AR instructions to describe Ra. According to the processing accuracy, the data of Ra can be divided into 14 rough levels: 0.012, 0.025, 0.050, 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.3, 12.5, 25, 50 and 100. In fact, these data correspond to the parameters in formula (3): x_k (1), x_k (2), …, x_k (n). The information represented by 14 levels is micro-information. They are the digital parameters reflecting the surface roughness of parts. In the next section, we will explain how universal intelligent processing module and advanced intelligent processing module handle micro-information, so that information processing results become AR instructions at different levels.

4.3 AR instruction design under different data processing levels

Our physics task is to show the user the surface roughness of the target part via AR commands. In order to facilitate data processing, the agent of the AR command system automatically divides Ra into 14 levels. Information indicating these levels is contained in AR instructions, and AR instructions at different data processing levels represent different data processing results.

In the following content, we will introduce the general intelligent processing mechanism and advanced intelligent processing mechanism of micro-information processing in detail. Before that, our team needs to introduce the main content of this task. Nine square pipe plugs of approximate size were selected as test pieces. Each has a different surface roughness, which is determined by their respective machining processes. There are five areas of the plug that need to be checked (i.e. Area A, Area B, Area C, Area D, Area E). Each area has a parameter. When the five evaluation parameters all meet the technical requirements of this subject, the detection result of the target part is qualified, otherwise it is unqualified. The user needs to judge whether a part is qualified or not according to the display content given by the AR command. Determine whether it meets the assembly requirements we need through selective compliance standards.

• (1) MBDAI

MBDAI is a kind of instruction that describes the geometric characteristics of parts by using graphics and text. MBD uses simple graphics that conform to international standards. Most graphics are not only concise, but also intuitive in describing geometric features. However, MBD does not improve the ability of AR instructions to describe microscopic geometric features to an ideal level. The reason is that standardization module is too simple for micro-information processing. Users need to use abstract processing, data relational mapping, information cognition and other functions to complete the understanding of micro-information. That is, users usually need to do a lot of data calculation through the brain to understand what MBDAI presents. In MBDAI, surface roughness is displayed directly to users in text form. Simple graphics show the processing method needed to obtain the surface roughness level. As shown in Fig. 6, the evaluation parameters of the five areas to be measured are designed according to the MBD rule. In our opinion, such AR instructions contain too little content for users to understand.

Fig. 6

Visual Interface based on MBDAI

• (2) MGAI

MGAI is a output result processed by UIP module. UIP module converts the geometric state data in micro information (i.e. surface roughness) into micro geometric information. In fact, micro geometric information can promote users to form specific cognitive logic in their mind, which is obtained through the abstract processing of the brain. For example, 14 levels are interpreted as a series of color information (i.e. red, orange, yellow, green, blue, indigo and purple), which makes AR instruction more vivid and accurate representation of the geometric state of target. Using this information as AR instruction, a user’s brain does not need to spend extra energy to distinguish the special attributes in micro information, which helps to improve the efficiency of user’s understanding of target object. Secondly, the cognitive logic data representing the special attributes of physical objects in MGAI can well reflect the logical relationship between geometric attributes. For example, there is a 14 level logic representing the size from large to small, which can be interpreted as a specific arrangement of color sequences (i.e. red, orange, yellow, green, blue, indigo and purple). The visual cognitive rules implied by these colors will help users better understand the actual state of the surface roughness of physical objects. Each color corresponds to two different roughness levels. Our team uses a special symbol “×2” to reasonably distinguish them(see Fig. 7). For example, when an region is shown in red, the Ra of this region is 0.012. When special symbol appears in the region, the Ra of the region is about twice that of the original data, i.e. 0.025. Furthermore, regular color arrangement will accelerate the efficiency of users’ understanding of micro information. When a user sees red in region A and blue in region B, he can conclude that the rough level of region B is lower than that of region A through micro geometric information in brain, because blue always ranks behind red in the color sequence. These three advantages make a user’s common-sense knowledge and micro-information reasonably linked to achieve the purpose of using micro geometric information to properly express micro information.

Fig. 7

Visual interface based on MGAI

• (3) MIAI

Micro-information-level AR instruction (MIAI) is the calculation result of advanced intelligent data processing module. In this module, micro geometric information (e.g., ordered color information) is further processed into micro-cognitive information to meet users’ cognitive needs. On the one hand, the module converts the surface roughness reflected by micro geometric information into 3D geometric figure. It interprets the data represented by seven colors into a series of color blocks through data relational mapping. The stack height of a set of color blocks represents the data value of Ra in the current region. By observing the accumulation of color blocks, users can intuitively feel the roughness of the region. On the other hand, the module transforms the logic relationship implied in micro geometric information into the sequence relationship between colors. For example, a red block has a Ra of 0.012, while two red blocks have a Ra of 0.025. When Ra is 0.05, it is represented by two red blocks and one orange block. When Ra is 0.1, two red and two orange blocks are used. In addition, MIAI also shows the relationship between the surface roughness level and technical requirements by controlling the position of the surface (i.e., control surface). The control surface represents the current technical requirements to be met. As shown in Fig. 8, when the control surface is in this position, it means that the user needs to view blocks with data larger than Ra 1.6. That is, the stacking height of the color block in a certain area exceeds the control surface, indicating that this region does not meet the technical requirements and the part is unqualified. Our team think that MIAI has replaced the data relationship between user computing roughness, and users only need to perceive the corresponding operation rules from it to complete the task quickly.

Fig. 8

Visual interface based on MIAI

It is worth emphasizing that the first two AR instructions require human brain to further process the guidance content into visual information that is easy for a user to understand, while the third AR instruction allows a user to intuitively perceive the current surface geometric state of target part without too much brain computing power input.

4.4 Summary

In this section, we first explain the concept of surface roughness, which is crucial to our detection task. It is the theoretical basis for describing measurement data (see Fig. 2). In our task, the measurement data is to evaluate the surface roughness of the part. These data are processed by agents to form 14-level parameters, that is, micro-information. The expression of micro-information will be an important parameter criterion.

Different AR instructions process microinformation at different cognitive levels. In MBDAI, the standardization module standardizes these parameters according to the design standards of MBD, and forms AR commands composed of simple pictures and texts. In MGAI, the general intelligent data processing module processes these parameters into seven color information with specific rules. The unspoken rules of this information strictly follow the cognitive rules of the user. In fact, the two AR commands mentioned above both indirectly describe the microscopic geometric features of parts through visual information, which belong to MGV. They cannot directly reflect the user’s cognitive needs for micro-geometric features, and MIAI solves this problem well. In MIAI, the advanced intelligent data processing module shows the user the logical relationship implied by the seven color information. On the one hand, the module interprets the size of 14 levels as the height of the color blocks, and intuitively expresses the surface roughness of the parts through the stacking height of the color blocks. On the other hand, this module inherits the implicit rules of the seven colors, and expresses the multiple relationships between different color blocks through sorting, enabling users to quickly understand the mathematical relationship between different colors. While continuously simplifying the user’s operation process, it is an important goal of our research to accurately improve the user’s operation accuracy.

5 Case study

In order to further prove that MIAI has obvious advantages over MGAI and MBDAI in enhancing the visual representation of AR instruction content, improving users’ information understanding ability and reducing the computational load of users’ brain, our team carried out a user study to identify the geometric differences between similar parts. In this section, we study and put forward five hypotheses. Our team described the test settings, measurement structures and work settings used to validate hypotheses, outlined the experimental process of validating guesses, and reported the test results. For these three interfaces, we have carried out precise experiments and further explained MBDAI, MGAI and MIAI.

5.1 Test setup and hypotheses

Test settings are designed to support this inspection task in a controlled environment. Our team placed a test-bed in a room (see Fig. 9). There are nine square plugs on the desktop. Each plug is about 100 mm long, 100 mm wide and 25 mm high. The user will complete the specified operation according to AR instruction provided by the projector.

Fig. 9

Test setup for micro operation task

In this task, users need to select square pipe plugs from all plugs to meet the technical requirements of surface roughness. In fact, the size and material of the nine plugs are almost the same. In order to optimize AR instruction based on MBD, MGAI and MIAI are proposed. Three visual information designed by three instructions are randomly assigned to three tasks. In order to validate the overall hypothesis, it is necessary to compare the performance of three interfaces in case study. Our team designed an experiment to compare the time users needed to complete a specific task with their subjective feedback. Based on this, the experimental hypothesis is as follows:

• Hypothesis 1:MGAI is faster than MBDAI.

• Hypothesis 2:MIAI is faster than MBDAI.

• Hypothesis 3:MIAI is faster than MGAI.

• Hypothesis 4:In terms of mental experience, users prefer MIAI compared to the other two interfaces.

• Hypothesis 5:In terms of cognitive efficiency, users prefer MIAI compared to the other two interfaces.

5.2 User study structure

Our team conducted a within-subjects study with 25 participants, were students from Northwestern Polytechnical University, mostly with an engineering background. Initially, each participant was asked to read and sign the informed consent form. Next, each participant will do a short pre-experimental questionnaire to ask about their demographic information, including their age, gender, educational background, previous AR experience and part inspection experience. These participants will be informed of the main objectives of the study, and the experimenter will explain to the participants the tips provided by each interface. Subsequently, he or she is told that the projector would display three interfaces on the surface of the part to be tested. They were also told that each AR instruction corresponded to the current surface roughness of the plug. Their goal is to complete the whole process as efficiently as possible and spend as much time as possible to feel that the information provided by AR instruction is fully understood.

Our team uses timers to record how long each participant takes to complete each task. When a task starts, the participant is asked to press the timer at hand, at which point the industrial camera is turned on. The camera will record the video until the user presses the timer again. Participants were also asked to complete a postquestionnaire in which they were asked how confident they were about the tasks before and after they completed the study. The confidence level was recorded by a 7-point Likert scale. Our team evaluated three interfaces by testing the performance of each participant in the experiment. Figure 2 summarizes the interfaces used under three conditions. These interfaces are:

MBDAI: This interface contains visual information such as surface roughness data, simple graphics, regions and so on (see Fig. 10).

Fig. 10

MBDAI-based visual information

MGAI: This interface contains color information, special symbols and region information. Special symbols represent the multiple relationship between surface roughness data (see Fig. 11).

Fig. 11

MGAI-based visual information

MIAI: This interface contains visual information, such as color blocks, control surface and regions. Color blocks represent the logical relationship between surface roughness data (see Fig. 12). The upper left corner of this figure shows a 3D view of the current part.

Fig. 12

MIAI-based visual information

The order of conditions that participants attempt is arranged in the balanced Latin square design to balance the carry-over effect between conditions. Before the experiment started, our team asked each participant to complete a square plug test. The square plug is the same as the square plug used in the actual test. Through this, participants will learn how to perform the tasks required by our team. After the exercises are completed, participants perform tasks using the provided interfaces. After each interface, each participant was asked to complete a questionnaire with a Likert scale rating item (see Table 1) at a level of 1 (completely disagree) to 7 (completely consistent). The timer records the completion time of the task. After all interfaces, each participant was asked to rank the three interfaces on various aspects of their experiences and interviewed.

Table 1 Likert scale rating questions (evaluation item in bold)

Our team invited 25 participants to participate in the experiment, but used only 22 participants’ data. One participant failed to complete the experiment, and two participant was dismissed for conflicting answers to the questions in the questionnaire. The average age of the participants was 22.5 years, 82.61% for males and 17.39% for females.

5.3 Hardware and software setup

The prototype system developed by our team combines the following elements: (1) server, (2) client, (3) agent based platform. Figure 13 shows our prototype system.

Fig. 13

The prototype system for part inspection

Server

It is the core of resources such as visual data and AR instructions. Client can access server and obtain the corresponding resource data. Our team uses the Intel NUC7i7BNH microcomputer as a hardware platform for assembly resources. It uses Intel Ceroi7 7567 U 3.5GHz, 6G RAM, Intel GMA HD 650 graphics card, 32G DDR4 2133 MHz, and Windows 10 Professional 64-bit operating system.

Client

It connects the projector and industrial camera to the PC and uses the projected content to guide the user’s operational behavior. Our team chose the Dell Alienware 17 (ALW17C-D2758) laptop as a client. It uses an Intel Corei7 7700HQ 2.8GHz, 8G RAM, NVIDIA GeForce GTX 1070 graphics card, 16G DDR4 2667 MHz, and Windows 10 Professional 64-bit operating system. Our team chose an industrial camera with 8 to 50 mm long zoom lens. The resolution is 5 million, the maximum frame rate is 60 fps, the type of video interface is USB3.0, and the horizontal angle of view is 6.3 to 37.5 degrees. Our team chose a projector model VPL-DX271. The projection screen size is 40 to 300 in., the luminance (lumen) is 3600, the standard resolution is 1600 × 1200 dpi, and the display technology is 3LCD.

Platform

It is used to implement resource transfer between the server and the client. Under this platform, server sends visual resources and AR instructions to client through WIFI, and client integrates the received resources to guide user’s operations. The user’s operational behavior is recorded by an industrial camera and fed back to server, which will perform statistical analysis on the data.

5.4 Result

• 1. Work efficiency: task completion time

There are significant differences in performance time between these interfaces. Figure 14 shows the average performance time under MBDAI, MGAI and MIAI. Among these three interfaces, MIAI takes the shortest time and MBDAI takes the longest time. MGAI takes longer than MIAI, but shorter than MGAI.

Fig. 14

Task completion time for three interfaces

• (1) MGAI & MBDAI

According to the average data statistics, the completion time of MGAI(M = 81.64, SD = 14.90, SE = 3.18) is 18.65% shorter than that of MBDAI (M = 100.36, SD = 12.59, SE = 2.68). According to the variance and standard deviation data, the data volatility of MGAI is smaller than that of MBDAI, which indicates that users can better adapt to work tasks after using MGAI. In order to compare the differences between the two methods, a paired t-test was carried out for the same subject by two methods, and then the results of the two methods were compared to draw a conclusion whether the data results were accepted. Paired t-test (α = 0.05) showed that the average time difference between two interfaces was statistically significant (t (x) = 11.756, p < .001). Therefore, the above data have obvious correlation, and H1 should be accepted.

• (2) MIAI & MBDAI

According to the average data statistics, the completion time of MIAI (M = 62.36, SD = 13.01, SE = 2.77) is 37.86% shorter than that of MBDAI (M = 100.36, SD = 12.59, SE = 2.68). According to the variance and standard deviation data, the data volatility of MIAI is smaller than that of MBDAI, which indicates that users can better adapt to work tasks after using MIAI. In order to compare the differences between the two methods, a paired t-test was carried out for the same subject by two methods, and then the results of the two methods were compared to draw a conclusion whether the data results were accepted. Paired t-test (α = 0.05) showed that the average time difference between two interfaces was statistically significant ((t (x) = 20.680, p < .001). Therefore, the above data have obvious correlation, and H2 should be accepted.

• (3) MIAI & MGAI

According to the average data statistics, the completion time of MIAI (M = 62.36, SD = 13.01, SE = 2.77) is 23.62% shorter than that of MGAI(M = 81.64, SD = 14.90, SE = 3.18). According to the variance and standard deviation data, the data volatility of MIAI is smaller than that of MGAI, which indicates that users can better adapt to work tasks after using MIAI. In order to compare the differences between the two methods, a paired t-test was carried out for the same subject by two methods, and then the results of the two methods were compared to draw a conclusion whether the data results were accepted. Paired t-test (α = 0.05) showed that the average time difference between two interfaces was statistically significant (t(x) = 9.982, p < .001). Therefore, the above data have obvious correlation, and H3 should be accepted.

To sum up, paired t-test analyzed the authenticity of the three groups of data, and proved that H1, H2 and H3 had a statistically significant difference. Among them, MIAI has absolute advantage in work efficiency, while MGAI as MBDAI does not give satisfactory results.

• 2. Subjective feedback: Likert scale rating questionnaire

After completing the tasks under each interface, participants were asked to answer a questionnaire containing eight questions (see Table 1). Cronbach’s alpha shows that the internal consistency between Likert terms is not too bad (α = 0.728), excluding each item that has a significant impact on reliability (α from 0.710 to 0.760). Figure 15 shows the results of aggregating all Likert items into a single proportional index (from 0 to 100) of the overall experience. All participants answered questions and our team reported the results of each group separately.

Fig. 15

Likert scale rating on guiding experience

Friedman test is a statistical test used to detect whether there are significant differences between multiple (related) samples. It is a nonparametric test method. For factor analysis of the Likert scale results, our team used Friedman tests (α = .05) to see if these interfaces were ranked significantly different. The results showed that the participants were significantly different in all items (Q1: χ²(2) = 35.102, p < .001; Q2: χ²(2) = 31.088, p < .001; Q3: χ²(2) = 33.273, p = .002; Q4: χ²(2) = 36.475, p < .001; Q5: χ²(2) =31.500, p < .001; Q6: χ²(2) =38.700, p < .001; Q7: χ²(2) =37.181, p < .001; Q8: χ²(2) =26.638, p < .001). Its analysis is divided into two steps. The first step is to analyze the significance value. If the corresponding p value is less than 0.05, then there is a difference; if the corresponding p value is greater than 0.05, it means that there is no difference at all. The specific data response of each factor was further analyzed. The results showed that there were significant differences among participants in all factors. This suggests that these three interfaces affect the ratings of participants in all factors, they affect the participants’ perception quality for AR instruction (Q1: enjoyment, Q2: focus, Q3: feeling confident, Q4: feeling natural and intuitive, Q5: feasibility, Q6: feeling efficient, Q7: availability, Q8: understandability).

In addition, our team analyzed the scoring results of each item. Figure 16 summarizes the corresponding results. In most cases, MBDAI is the lowest rating, while MIAI is the highest in most cases. MGAI were mainly between MGAI and MIAI. This shows that in terms of user experience, MBDAI has the worst experience, MGAI is slightly better, and MIAI is the best. It is worth noting that MGAI has more advantages than MBDAI in Q3 and Q8, which indicates that MGAI is only a non-standard visual representation of MBDAI.

Fig. 16

Results of Likert scale rating (1: strongly disagree ~7: strongly agree, “×”: mean, significant main effects are marked as a superscript to the questions)

For pairwise comparisons between interfaces that our team wants to study, our team uses rank tests with Wilcoxon symbols (see Table 2) and Bonferroni corrections (α = 0.167). In principle, Bonferroni correction is to divide the small probability significance level a by pairwise comparison, so as to reduce the critical value of statistical test significance level. In this case, p value must not be greater than α(i.e., 0.167), otherwise it is abnormal.

Table 2 Results of Wilcoxon Signed Rank test on rating questions (significant result in bold)

• (1) working experience

In Q6 (.004 < .167), the ranking of MGAI is significantly higher than MBDAI, which shows that compared with MBDAI, MGAI has a significant impact on manual efficiency of participants. This is consistent with the test result of performance time. Therefore, hypothesis 1 is accepted. In addition, the ranking of MIAI is significantly higher than MBDAI, which shows that compared with MBDAI, MIAI has a significant impact on manual efficiency of participants. This is consistent with the test result of performance time. Therefore, hypothesis 2 is accepted. Similarly, the ranking of MIAI is significantly higher than MGAI, which shows that compared with MGAI, MIAI has a significant impact on manual efficiency of participants. This is consistent with the test result of performance time. Therefore, hypothesis 3 is accepted.

• (2) mental experience

In the evaluation of enjoyment (Q1), focus (Q2), and confident (Q3), the feedback results are abnormal. In the first pair, MGAI ranked no higher than MBDAI in Q1-Q3, which indicated that there was no significant difference between them on user’s mental experience. This means that compared with MBDAI, UGAI does not bring substantial help in the psychomotor aspect of this condition. In the second pair, MIAI is higher than MBDAI only in Q1 and Q3, which indicates that there is no statistically significant difference in focus between two interfaces. This shows that the application scope of MIAI has certain limitations, that is, MIAI can not effectively improve the mental concentration of users. In the thrid pair, MIAI is higher than MGAI in Q1 Q2 and Q3. This shows that the application scope of MIAI has certain limitations, that is, MIAI can not effectively improve the mental concentration of users. Therefore, hypothesis 4 is ultimately denied.

• (3) cognitive efficiency

The results of Q7 and Q8 also did not meet our expectations. The ranking of MGAI in usability is not higher than MBDAI, which indicates that MGAI has no positive impact on users’ psychological representation. In addition, the usability ranking of MIAI is higher than that of Miai and mbdai, which indicates that MGAI has a positive impact on users’ psychological representation and accelerates the efficiency of users’ psychological image construction. The evaluation result of Q4 and Q5 is basically in line with our expectation. In these 3 pairs, MIAI ranked significantly higher than MGAI and MBDAI in terms of N&I and feasibility, indicating that MIAI has a positive impact on users’ spatial cognition. Therefore, hypothesis 5 is accepted.

6 Discussion

To explore the ergonomics of MIAI in micromanipulation, we developed three interfaces and evaluated them through a specific task. In the user study, we put forward three evaluation aspects according to the research focus, such as work efficiency (H1, H2, H3), spiritual experience (H4) and cognitive ergonomics (H5).

• (1) Influence of MBDAI, MGAI and MIAI on work efficiency

According to the task completion time (see Fig. 14), MIAI is nearly 30% more efficient than MBDAI and MGAI. Our team discovered a surprising fact: participants believe that concise guidance information in MBDAI is not always conducive to improving user efficiency. In fact, in addition to the MBDAI reported in this paper, we have also designed other MBDAIs, such as color point cloud, pigment map, etc., whose work efficiency is even 10% - 20% lower than the reported MBDAI. This may be due to the following reasons: users need to spend a lot of time thinking logically to understand the contents of these instructions. MGAI interprets the unpublished logical relations in MBDAI as ordered color information, and users can clearly see the operation intention reflected in the task. Therefore, information rich MGAI can improve user efficiency. However, MGAI has not raised the level of ergonomics to a higher level.

In fact, MIAI provides users with richer visual cues and further shortens operation time. This shows that the informative MIAI is very useful for describing microscopic information. Therefore, our team believes that the only way to improve user efficiency is to reasonably express the logical relationships implied by micro-information. In fact, no matter how much information the MGAI has, such instructions cannot further improve the user’s operating efficiency (see Fig. 14, Q6). MIAI displays user MGAI operation information that has not been calculated (e.g., the position of the control surface). This process improves the user’s ability to understand the task, thereby improving the user’s operating efficien-cy. On the technical level, all-round improvements have been made, and improvements have been made on the basis of cognition.

• (2) Influence of MBDAI, MGAI and MIAI on spiritual experience

Questionnaire survey data show that MGAI and MBDAI have a significant impact on user experience-related issues, including Q1 (enjoyment), Q2 (attention), and Q3 (confidence). This proves that MGAI with rich visual cues can improve mental experience than MBDAI with less. According to Figs. 15 and 16 and Table 2, our team attributed the improvement of the visual experience to two reasons:

• MGAI improved the user’s mental focus. Compared with MBDAI, MGAI uses highlighted color information to continuously attract the user’s attention, so the user can always focus on task process without being distracted.

• MGAI reduces the psychological fatigue of users. Compared with the simple graphics contained in MBDAI, the color information of MGAI can stimulate users’ interest more. In fact, high attention can indeed reduce user fatigue, thereby reducing the incidence of operating errors, which is also proven by user interviews.

In Fig. 16, we also noticed that MGAI’s scores on Q3 (confidence) are lower than MBDAI, which indicates that users are familiar with numbers, which makes users’ evaluation of color information lower. In fact, participants admitted that after a period of training, they believe that color information will bring them a better operating experience than digital information.

Besides, questionnaire data shows that MIAI and MBDAI have a significant impact on issues related to mental experience, including Q1 (enjoyment), Q2 (focus), Q3 (confidence). The questionnaire data also showed that MIAI and MGAI had a significant impact on user experience-related issues. These results show that MIAI has a better mental experience than MGAI. In order to find out the reason, our team believes that the above results are caused by two reasons.

• First, MIAI uses 3D color blocks to represent the relationship between the surface roughness levels of parts, thereby enhancing the user’s intuitive experience of abstract information.

• Secondly, users usually consider rough surfaces to be uneven surfaces. It interprets abstract information as an easy-to-understand accumulation of color blocks, so that users can intuitively feel the unevenness of the part surface. A comprehensive and multi-angle explanation is carried out on the three-dimensional level of geometry.

• (3) Influence of MBDAI, MGAI and MIAI on users’ cognitive ergonomics

Questionnaire data showed that MIAI and MBDAI had significant influence on the response of cognitive efficacy related issues, including Q4(N&I), Q5 (feasibility), Q7(availability), and Q8(comprehensibility). Similarly, there are significant differences between MIAI and MGAI in above items. In fact, MGAI’s description of geometric features is conducive to users’ cognition, but the promotion effect is not high. From Fig. 16, it can be seen that MIAI improves users’ perception ability by describing geometric features, but the difference is that user-centered MIAI tries to restore users’ inner thoughts reasonably, so that users can better grasp the core content of tasks and understand the progress of work. In other words, MIAI uses orderly stacked color blocks to show the rules of roughness level to users. In the process of task execution, users can quickly perceive the logical relationship between roughness and remind users of the current surface roughness level by viewing the superimposed color sequence. Users can make full use of this information to deepen the understanding of task content.

7 Implication and limitation

In this section, according to the hypothesis and the results of the interview, two implications are given. In addition, our team acknowledges that our test experiment has many shortcomings compared with the actual test, and may have a better display scheme under MIAI.

• (1) The design of MIAI must follow visual cognitive rules contained in micro operation task.

In fact, whether AR instruction is designed with MGAI (including MBDAI) or MIAI, visual cognitive rules are potential. This paper only selects a test to prove that: 1) MIAI is better than MGAI to meet the cognitive needs of users; 2) MIAI is better than MGAI to promote the expression of manual intention. Practice has proved that these views are completely correct in any test task and have wide applicability.

• (2) MIAI focuses on improving users’ cognitive level, and does not deliberately emphasize users’ visual experience.

Our research proves that AR instruction should be used to satisfy users’ understanding of micro operation task, rather than to enhance users’ visual information experience. Therefore, for the purpose of research, our team is not focused on describing geometric features, but on expressing the operation logic of tasks. This is the essential difference between MIAI and MGAI. This does not mean that MGAI is useless. Obviously, the construction of MGAI needs geometric characteristics of physical objects. However, MIAI only keeps some parts that are conducive to user perception, and adds more content to meet the needs of user understanding.

Although the results of user research are interesting, there are still many limitations.

• Because the projection content is only 2D image, users can not see the three-dimensional effect of MIAI, which makes it difficult for users to fully understand the advantages of MIAI.

• Compared with the actual task, the content of this task is relatively limited. However, it does have some key elements, such as part inspection. This may limit the applicability of the results.

• Besides, the case study is not very detailed. We only analyze the user’s task time and subjective feedback, but not evaluate the user’s behavior. We believe that assessing user behavior can help us understand the specific impact of MIAI.

8 Conclusion and future work

MIAI is a first formal user study to evaluate AR instructions at micro information-level. Through case studies, it is proved that MIAI has higher user performance than MGAI and MBDAI. These two meanings are not only applicable to parts inspection, but also have a very important meaning for the inspection operation details. Case study are used to reflect the distribution of data and feedback from participants. In terms of work efficiency (Q6), the experimental results support H1, H2 and H3. In terms of mental experience (Q1-Q3), our team found a significant impact between MIAI and MBDAI, MIAI and MGAI. The results of the ranking questionnaire reflect the same facts. In terms of cognitive efficiency (Q4, Q5, Q7, Q8), our team also found a significant impact between MIAI and MBDAI, MIAI and MGAI, which indicates that MIAI has a greater impact on user cognition than the other two. This shows that MIAI uses visual information to express the logical relationship between the detection process and the user, highlighting the practical advantages. In the future, we will try to conduct more research on MIAI to change user behavior. It can be predicted that the research of user behavior in MIAI with haptic feedback is of great significance. For the future, more detailed research has very important guiding significance. There is also a clearer classification discussion direction for demand analysis.