Introduction

Due to the existence of complex joints in engineering rock mass, the mechanical properties of engineering rock mass are different from that of intact rock mass under various engineering actions. It can be obtained from the previous researches (Hoek and Brown 1997), the bearing capacity and stability of rock mass are seriously affected by the mechanical characteristics of joints. As displayed in Fig. 1, the slope is divided into two or more blocks by the connected long joint plane, which increases the risk of slope instability (Brideau et al. 2009). At the same time, some engineering actions may change the inclination of rock slope. Especially, when the joint is in the same direction as the slope inclination, the rock mass fracture or landslide is more likely to occur (Eisbacher 1971).

Fig. 1
figure 1

The Aishihik River landslide

Full size image

It is common that there are many penetrating cracks with different types in the joint planes, and the undulating angles of joint planes are of much importance for studying the mechanical property and controlling the slope instability. Thus, except for the mechanical characteristics of rock mass, conducting related researches on the influence of slope inclination and built-in joints on the rock mass instability is necessary.

The fracture or instability of rock slope is mostly attributed to the overall loading exceeding to the limited equilibrium state under compress-shear or tensile-shear action. Rock mass under different loading types exhibits various failure modes, which have been studied in tensile, compressive, shear, and rheological experiments (Wu et al. 2020a). In addition, in the effect of internal structure on mechanical property, which can be seen compared to intact rock mass, the rock mass with different joint planes has a lower bearing capacity and the failure characteristics embrace much forms (Hencher et al. 2011). In recent years, much effort has been devoted to explore the shear mechanical property of rock mass with joint planes. Patton (1966) summarized the previous researches related to shear tests of rock masses with joints and found that the roughness fluctuation of the joint plane has much effect on the shear strength of joints. By performing tilt and shear tests of joints, Barton (1973), Barton and Choubey (1977), Barton and Bandiss (1982) defined the roughness fluctuation, and give the popular coefficient of roughness (JRC). It is demonstrated that the undulating angle and roughness within joints have a great influence on shear mechanical behaviors. Meanwhile, Hoek and Bray (1981) and Bandis et al. (1983) also indicated the shear mechanical behaviors of rock mass are much affected by undulating angles and roughness under high ground stress. Tong et al. (2014) studied the shear strength of sandstone under the effect of anisotropy by setting the shear plane with different angles. Yao et al. (2021) investigated the AE characteristics of coal petrography with different saturation in variable angle shear experiments, and the transformation process from tensile failure to tension-shear failure of coal petrography was described. By performing comparisons among direct shear, variable angle shear, and triaxial compressive experiments, the characteristics of shear planes under three kinds of experiments were analyzed, and the reliability of obtaining shear strength parameters was also evaluated by Gong et al. (2020).

Macro-shear mechanical behaviors of joint rock mass considering to the effect of through serrated and wavy undulating body were investigated by carrying out direct shear tests and numerical simulation based on PFC; it can be obtained that the damage evolution process of rock mass discontinuities includes five stages (Liu et al. 2021a, b). Yin et al. (2020) studied the direct shear characteristics and failure forms of hard brittle marl with different discontinuous structural plane and gave the variations among undulating angle, peak strength, and failure modes. A series of numerical simulations were conducted to construct a mathematical model of soft structure plane with discontinuity height (Tang and Lin 2017). The stress–strain relationships of direct shear tests for structure planes were also discussed in details. Lee et al. (2014) proposed an experimental model of shear behaviors of joint rock, and the effect of loading and property of rock material on direct shear mechanical behaviors were demonstrated. Chern et al. (2012) performed cyclic shear experiments on gypsum samples with regular triangular joints to explore the degradation and shear mechanical behaviors, and the shear theoretical model was constructed under cyclic shearing. A cyclic shear test system was established, and the mechanical property of joint rock was also observed (Lee et al. 2001).

Considering the direct shear mechanical behaviors of joint rock, many direct shear tests were conducted to study shear failure characteristics, and based on those mechanical analysis, a shear constitutive model was presented (Jafari et al. 2003; Plesha 1987). Huang et al. (2002) proposed three mechanical mechanisms that affect the deformations of joint rock, i.e., sliding, separation of interstitial contact surface, and shear fracture. The shear mechanical behaviors of joint rock with two kinds of bevel angles were also analyzed by performing direct shear tests (Yang and Chiang 2000). Two kinds of roughness indexes were constructed to describe the influence of planes with different geometric features on shear strength (Ban et al. 2020, 2021). Wu et al. (2020b) analyzed the deformation mechanisms of triaxial creep of rock, and a fractional model was proposed to depict compress-shear characteristics. A mechanical model was given to describe shear mechanical behaviors of concrete, and a series of shear tests under different shear angles (Jongvivatsakul et al. 2016).

Based on abovementioned, we can see for the study of shear mechanical behaviors of rock mass with various joint planes; most of current researches focus on the direct shear tests. However, during the prevention of landslide, except for the effect of undulating angle of joint plane, the slope inclination of engineering rock mass is of great significance for controlling slope instability, which is not considered in the direct shear tests. Thus, in order to more truly reflect the deformation characteristics of joint rock mass within the slope, we will conduct a series of variable angle shear tests on rock material including joint planes with different undulating angles to reveal the shear deformation mechanism of joint rock mass with inclination. The outline of this paper is expressed as follows. Section “Introduction” introduces the preparations of experimental samples, apparatus, and program. The analysis of mechanical response and shear strength of tested samples will be performed in details in sect. “Variable angle shear test.” During the sect. “Test equipment and loading scheme, the failure characteristics including actual fracture, DIC testing results, and acoustic emission testing parameters are illustrated and compared to reflect the evolution of variable angle shear deformation. Based on the discussion of macro-failure modes and mechanic analysis, a novel criterion of instability is proposed to distinguish the types of instability in sect. “Experimental results and discussions.” Finally, several conclusions are drawn.

Variable angle shear test

Specimen preparation

The red sandstone experimental samples were selected from Liuyang of Hunan province, China. Cube samples were drilled from intact red sandstone rock mass without obvious cracks and fracture on surface, that displayed in Fig. 2, whose sizes are 50 mm in length, 50 mm in width, and 50 mm in height. In accordance with the standard requirements of the International Society of Rock Mechanics (Franklin et al. 1979), the parallelism and surface flatness are controlled within ± 0.05 mm and ± 0.02 mm, respectively, whose basic sizes are shown in Table 1. The uniaxial compressive strength of initial red sandstone sample approaches to 85 MPa, and the main minerals are quartz 79%, plagioclase 11%, potash feldspar 4%, and calcite 3% obtained from XRD tests. As displayed in Fig. 3, we prepared 5 groups of cube red sandstone samples including 4 groups of joint red sandstone samples and 1 group of non-joint samples; other cube specimens will be processed into joint samples with undulating angles of 15°, 30°, 45°, and 60°, respectively. It is noted that each experimental sample has same joint serrations, and the undulating angle is shown in Fig. 3.

Fig. 2
figure 2

Devices and samples preparation. a Cube machining. b Polish. c Joint cutting

Full size image
Table 1 Mechanical parameters of experimental samples
Full size table
Fig. 3
figure 3

Prepared sample and display of serrations

Full size image

To obtain the evolution characteristics of surface deformation fields of samples under variable angle shear, before performing shear tests, the speckle points should be randomly plotted on the surface of sample (free surface) by using a 0.3-mm black marker (Zhang et al. 2021). It should be noticed that, before plotting speckle points, the free surface of sample should be sprayed pure white. The area of speckle points accounts for about 50% of the total free face (Aliabadian et al. 2021; Song and Yue 2021).

Test equipment and loading scheme

The used experimental apparatus is uniaxial-triaxial compressive experimental system in State Key Laboratory for Geomechanics and Deep Underground Engineering, and all testing procedures are controlled automatically by PC software. The variable angle shear tests can be conducted by utilizing one special fixture, whose angle can be adjusted from 20 to 70°. The variable angle shear fixture is composed of pressure plate, roller, and variable angle plate. As illustrated in Fig. 4a, the experimental sample is placed at the middle of up and down variable angle plate, and the joint strike is in line with shearing direction formed by variable angle shear fixture. For avoiding the instability of experimental sample induced by too large shear angle, the shear angle α is set as 40°, 45°, 50°, and 55° that shown in Fig. 4b. Before carrying out shear tests, the initial loading is applied to 2 kN to ensure the shear fixture contacting well with experimental sample. The loading style is set as displacement loading with constant loading rate of 0.3 \(\mathrm{mm}/\mathrm{min}\) until failing sample.

Fig. 4
figure 4

Test system and experimental layout of sample. a Components of the system. b Diagram of sample holding

Full size image

The acoustic emission system is introduced to collect fracture signal from the variable angle shear tests, which includes two acoustic emission probes with 100 kHz ~ 1 MHz (Farhidzadeh et al. 2014). Its signal receiving threshold value and frequency of data collection is 40 dB and 5 MHz, respectively. The high-speed framing camera is used to capture the evolution process of speckle deformation field (Xiu et al. 2021), whose filming frequency and image resolution are set as 50 FPS and 320 × 256, respectively. It must be noted that the loading system, acoustic emission system, and high-speed framing camera should be started simultaneously.

Experimental results and discussions

Characteristics of load–displacement curves

Figure 5 presents the variations between load and displacement for the samples including non-joint samples and the joint samples with different undulating angles \(\theta\) (15°, 30°, 45°, and 60°) under various shear angles \(\alpha\) (40°, 45°, 50°, and 55°). We can see when \(\alpha =40^\circ\), the peak load under each undulating angle has a relatively small fluctuation, and the effect of undulating angle is weak. When \(\alpha \ge 45^\circ\), the peak load is seriously affected by the variations of undulating angles, and the load–displacement curves after peak load exhibit various kinds of trends. For facilitating distinguishing deformation characteristics under variable angle shear, the joint samples under shear angle with 55° are selected as analyzed examples, and due to the fracture behavior of post-peak of samples with undulating angle 45° similar as that of samples with undulating angle 60°, in this section, we only discuss the deformation characteristics of the joint sample with undulating angle 60°. Based on the loading deformation characteristics shown in Fig. 6, the total deformation curve is divided into five stages, i.e., initial crack compaction stage, linear deformation stage, pre-peak crack development stage, post-peak load drop stage, and plastic slip stage, of which comprehending discussions are given as follows.

Fig. 5
figure 5

Load–displacement curves under different undulating angles. a α = 40°. b α = 45°. c α = 50°. d α = 55°

Full size image
Fig. 6
figure 6

Total process loading-displacement curve of shear angle with 55°. a Non-joint. b θ = 15°. c θ = 30°. d θ = 60°

Full size image

It is illustrated in Fig. 6 that, when the initial load is applied on the fixture, the initial pore or crack within sample is compressed, and the slope of load–displacement curve rises gradually, which is named as initial crack compaction stage (O-A). When sample enters linear deformation, we can see the slope of load–displacement curve becomes constant, and there is no obvious crack on the sample surface, which can be called as linear deformation stage (A-B). The pre-peak crack development stage (B-C) presents that there is a small area of flake shedding from the free surface, and the clear breaking sound occurs in the loading deformation stage. We also can see the obvious cracks appear on the part area of free surface of non-joint sample, and there are many fractures appearing at the part serrations. Hence, the deformation before peak load is thought of as crack development stage. After the peak load, the deformation can be segmented into load drop (C-D) and plastic slip stage (D-E). During the load drop stage (C-D), it can be found that the load–displacement curve will suddenly drop in a short time, and the crack will penetrate the both sides of sample along with shear plane, then the sample will arrive at failure fast. It is obvious that the joint serrations for samples with undulating angle (30° and 60°) have been cut off, the shear plane slips slightly, and the sample has been separated. Last but not least, an interesting deformation stage appears in Fig. 6c, d that is called as plastic slip stage (D-E), and although the joint serrations have been cut off, the sample still keeps stable. Under the effect of frictional force and bite force resulted from the clipped serrations, the deformation continues to develop and bear load until arriving at the final load drop.

Based on the above analysis on segments and discussions of deformation stages, it is demonstrated the peak load and mechanical property of post-peak of samples are much influenced by shear angle α and undulating angle θ, which can be thought when the shear angle is relatively low; higher peak load is, richer energy stored at pre-peak is, and the most of stored energy will be transformed as dynamic energy after fracture severe damage, and load drop at post-peak usually appears. Nevertheless, when the shear angle increases, lower peak load is smaller energy stored than at pre-peak, and just a part of stored energy is converted to dynamic energy, and the interesting deformation will be found, i.e., plastic slip stage. The sample after the first load drop still has certain bearing capacity. Meanwhile, comparing to the joint sample, the non-joint sample can store more energy before peak load, and the failure after peak load will be severe, and the load will also rise down fast.

Subsequently, the variations of peak load of samples under the effect of shear angles α and undulating angles θ will be discussed. As illustrated in Fig. 7, considering the joint samples, the peak load decreases with an increase in shear angle under constant undulating angle. However, when the shear angle is constant, the peak load arrives at maximum value with undulating angle of 30°, which can be accounted for when \(\theta <30^\circ\); with a rise in load, the bite and friction force within joint serrations and planes are not enough to resist shearing action and the sample will enter failure fast at the low load level. On the contrary, when \(\theta >30^\circ\), we can see the peak load rises down with increasing of undulating angle, which can be explained that the applied load exceeding to peak load will result in clipped serrations and instability, and the high serrations with large undulating angles will be easier broken due to effect of shearing moment under variable angle shear. Then, for the non-joint samples, it can be seen that the peak load decreases with an increase in shear angles as illustrated in Fig. 8.

Fig. 7
figure 7

Relationships among peak load, \(\alpha\), and \(\theta\) (joint samples)

Full size image
Fig. 8
figure 8

Relationship between peak load and \(\alpha\) (non-joint samples)

Full size image

Analysis of failure characteristics

It is known that the failure of rock material under variable angle shear is composed of shear fracture and tensile fracture (Liu et al. 2021a, b). The shear failure is mainly caused by shear fracture, which can be verified by our experimental results.

Acoustic emission monitoring (AE) is widely used in laboratory tests to identify micro-fracture modes (shear or tensile) occurring in rock samples. Acoustic emission signals are mainly generated from crack development in solid materials, and the signal waves emitted are received by the probe bonded to the sample surface. There are differences in wavelength, amplitude, and frequency of signals emitted under different fracture modes. Therefore, the micro-fracture characteristics of solid materials can be indirectly reflected by monitoring AE signal characteristics. At present, the commonly used acoustic emission monitoring parameters include ringing count, absolute energy, and AF-RA value. By reviewing previous researches (Ohno and Ohtsu 2010; Farhidzadeh et al. 2014; Aggelis et al. 2010; Soulioti et al. 2009), utilizing acoustic emission parameters (AF and RA) to identify the failure mode of sample is reasonable. It should be noted that the AF is defined as the ratio of ringing counts to duration, and the RA is the ratio of rising time and peak amplitude. As mentioned in current researches, the shear fracture and tensile fracture are corresponding to low AF-high RA and high AF-low RA, respectively. The AF/RA = 90 will also be introduced as the critical value of distinguishing failure mode of rock material in this study (Gan et al. 2020). In this section, the acoustic emission parameters (AF and RA) of joint samples under shear angle with 55° are selected as analyzed examples in Fig. 9.

Fig. 9
figure 9

Variations of AF-RA for the joint samples with \(\alpha =55^\circ\). a S-55-N. b S-55–15. c S-55–30. d S-55–45. e S-55–60

Full size image

As shown in Fig. 9, based on the critical line with slope of 90, the total area is divided into two parts, i.e., NT and NS, which represent the total amount of data distributed in the upper and below region. The proportions of two regions also have been given in Fig. 9. The locations of the acoustic emission parameters are mainly below the critical line (k = 90), which can be regarded as the failure modes of above samples mainly following shear fracture. By analyzing the evolution laws in Fig. 9a-e, we can see when undulating angle exceeds to 45°, the signals from AE tests increase obviously, which can be explained the increasing of undulating angle induces to the serrations are easy to be cut off, and the damage of sample also becomes serious. Meanwhile, the proportion of NS decreases and then increases, which indicates the increasing of undulating angle has much influence on the rising of proportion of shear failure.

For better displaying the variation of shear fracture under various shear angle \(\alpha\) and undulating angle \(\theta\), the shear fracture ratio (SFR) is defined as the ratio of the acoustic emission data below critical line and total acoustic emission data. Figure 10a presents the SFR of non-joint samples arrives at minimum value when \(\alpha =45^\circ\). It is illustrated in Fig. 10b that we can see that either \(\theta\) or \(\alpha\) is constant, the SFR will first decrease, and then increase. When the coupling effect of \(\theta\) and \(\alpha\) is considered, the SFR is least with \(\alpha =45^\circ\) and \(\theta =30^\circ\), i.e., 66.98%, and the SFR arrives at maximum value with \(\alpha =55^\circ\) and \(\theta =60^\circ\), i.e., 95.62%. Thus, although the samples are mainly shear fracture, there is a various relationship between the ratio of shear and tensile fracture. Regardless of the existence of joint planes in the samples, the proportion of shear fracture is always the smallest when the shear angle is 45°, which is attributed to the influence of two factors on the reason of shear fracture. For the small shear angle, the great normal stress leads to increasing of compress-shear fracture. For the great shear angle, the decreasing of normal stress will result in the increasing of SFR caused by faulting of shear face, that is more obvious in joint samples.

Fig. 10
figure 10

Evolution of SFR under various \(\alpha\) and \(\theta\). a Non-joint samples. b Joint samples

Full size image

From the comparisons between evolution of shear fracture ratio and macro-failure characteristic, some corresponding relationships were found to verify the validation of analysis of shear fracture ration. First, the different fracture samples under the coupling action of various shear angles and undulating angles were collected and compared; based on the macro-failure features of broken samples, the failure characteristics are divided into shear fracture (S) and tensile fracture (T) (Bandis et al. 1981; Pereira and Freitas 1993). During the shear fracture, we can see the direction of crack propagation is consistent with that of shear, and the shear failure surface is accompanied by scratches, and the fracture surface is relatively smooth. Instead, the failure or block peeling of free surface is attributed to tensile fracture, and the fracture surface resulted by tensile fracture has more undulation and roughness, and there are none above scratches in the fracture surface (Yin et al. 2020).

The failure mode of free surface, actual fracture characteristics of cross-section, and variations of principal strain fields obtained from DIC tests are illustrated in Fig. 11. It can be seen that, with an increase in α, the fracture of free surface of sample becomes weak gradually, which indicates the damage resulted by tensile fracture is decreasing. Then, with increasing of α, the shear fracture plane of non-joint sample transforms from fluctuation to smooth, and the serration damage of joint samples becomes more serious. The increasing of shear fracture plane symbols the rising of composition of shear fracture. When \(\theta \le 30^\circ\), the damage of serrations mainly occurred at the top of serration, and the most of serrations were cut off. When \(\theta >30^\circ\), the position where serrations were cut off will move toward the bottom of serrations. Thus, the above corresponding relationships between the variations of shear fracture ratios in Fig. 10 calculated by acoustic emission parameters and macro-failure characteristics in Fig. 11 are verified.

Fig. 11
figure 11figure 11

Failure characteristics of the samples. a Non-joint samples. b θ = 15°. c θ = 30°. d θ = 45°. e θ = 60°

Full size image

Meanwhile, we also can obtain the variations of principal strain fields obtained from DIC tests. In DIC test, by spraying white paint on the surface of the specimen and then applying black speckles, the high-speed cameras are used to track and capture the changes of the corresponding speckle positions in adjacent pictures during loading. The displacement changes of each point in the field of vision are calculated, and the speckle deformation cloud map is drawn finally. Affected by the heterogeneity of the sample and different load forms, the displacement of each point on the sample surface is different. Therefore, the speckle deformation cloud map drawn mostly presents different color distribution states. According to the difference of degree of strain concentration, the total strain field will be segmented as localizing core zone (yellow-green zone), localizing transition zone (blue band zone), and external strain mean zone (purple zone). When \(\alpha =40^\circ\), the strain localization core zone is disordered, and the extension of strain localization core zone is not affected by joint direction, and up to 45°, the strain localizing core zones of joint samples are in joint planes. Then, until it rises to 55°, the distributions of strain localization core zones exhibit a tendency that evolves from disorder to order, and the remarkable strain concentration appears in the range of serration width.

A comprehensive analysis of the morphology of the fracture surface and the strain cloud characteristics of the surface shows that the change of shear angle plays a primary role in the influence of fracture characteristics. The increasing of shear angle is more likely to promote the concentration and orderly extension along the shear direction for fracture areas. In addition, the area affected by fracture of surface also decreases, which indicates that, if the slope with large angle collapses in the actual engineering, the surface of fracture rock mass will be smoother and the lateral disaster degree will be weaker. When the angle is small, the fracture surface will be more complex, and the lateral disaster will be more serious.

Criterion factor of instability type

To better judge the instability type of joint samples under variable angle shear, by performing mechanics analysis on joint plane, we can obtain the normal stress and shear stress of joint sample under variable angle shear,

$$\sigma =\frac{P}{A}\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)$$
(1)
$$\tau =\frac{P}{A}\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)$$
(2)

where \(\sigma\) is normal stress, \(\tau\) is shear stress, \(\alpha\) is angle of shear fixture, \(P\) is force applied on fixture, \(A\) is area of fixture along with loading direction, \({f}_{1}\) is friction coefficient of roller (\({f}_{1}=1/nd\)), \(n\) is number of roller, and \(d\) is diameter of roller.

Then, concentrating on one serration, as shown in Fig. 12(b, c), the normal stress \({\sigma }_{1}\) and \({\sigma }_{2}\) that perpendicular to the joint plane (OA) and the shear stress \({\tau }_{1}\) and \({\tau }_{2}\) that parallel to the joint plane (OA) can be expressed,

$${\sigma }_{1}=\sigma \mathrm{cos}\theta$$
(3)
$${\tau }_{1}=\sigma \mathrm{sin}\theta$$
(4)
$${\sigma }_{2}=\tau \mathrm{sin}\theta$$
(5)
$${\tau }_{2}=\tau \mathrm{cos}\theta$$
(6)

where \({\tau }_{1}\) is defined as the stress that prevents slippage of joint plane (OA), and \({\tau }_{2}\) is defined as the stress that driving slippage of joint plane (OA).

Fig. 12
figure 12

Schematic diagram of stress decomposition

Full size image

The joint plane (OA) is selected as analyzed plane, and we define the \({\sigma }_{p}\) and \({\tau }_{q}\) on the joint plane (OA), which is shown as follows:

$${\sigma }_{\mathrm{p}}={\sigma }_{1}+{\sigma }_{2}=\frac{P}{A}\left[\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)\mathrm{cos}\theta +\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)\mathrm{sin}\theta \right]$$
(7)
$${\tau }_{\mathrm{q}}={\tau }_{2}-{\tau }_{1}=\frac{P}{A}\left[\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)\mathrm{cos}\theta -\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)\mathrm{sin}\theta \right]$$
(8)

By considering the effect of friction of joint plane (OA), we can obtain the driving stress \(G\) of slippage of sample,

$$G={\tau }_{\mathrm{q}}-{f}_{2}{\sigma }_{\mathrm{p}}=\frac{P}{A}\left[\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)\mathrm{cos}\theta -\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)\mathrm{sin}\theta -{f}_{2}\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)\mathrm{cos}\theta -{f}_{2}\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)\mathrm{sin}\theta \right]$$
(9)

where \({f}_{2}\) is the friction coefficient of joint plane (OA).

Let \({\eta }_{1}=\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)\mathrm{cos}\theta\), \({\eta }_{2}=\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)\mathrm{sin}\theta\), \({\eta }_{3}={f}_{2}\left(\mathrm{cos}\alpha +{f}_{1}\mathrm{sin}\alpha \right)\mathrm{cos}\theta\), and \({\eta }_{4}={f}_{2}\left(\mathrm{sin}\alpha -{f}_{1}\mathrm{cos}\alpha \right)\mathrm{sin}\theta\), and we propose a novel criterion factor for instability, i.e., \({\eta }_{d}\),

$${\eta }_{\mathrm{d}}=\left({\eta }_{1}-{\eta }_{2}\right)-\left({\eta }_{3}+{\eta }_{4}\right)$$
(10)

The driving stress \(G\) can be deduced,

$$G={\eta }_{\mathrm{d}}\frac{P}{A}$$
(11)

Based on above assumptions and derivations, the instability of sample under variable angle shear can be divided into two sections. When \({\eta }_{d}>0\), the instability type of sample is defined as slipping instability. When \({\eta }_{d}\le 0\), the instability type of sample is regarded as fracture instability. In this study, based on the tests of friction for intact samples, we can obtain the friction coefficient of joint plane, i.e., \({f}_{2}=0.87\). As displayed in Table 2, the instability type of each sample is obtained.

Table 2 Identification of instability type
Full size table

From Table 2, we can see the obtained instability type of joint samples under variable angle shear follows fracture instability. However, during the final stage of shear deformation, there are parts of slipping instability in the serrations under shearing action, which can be accounted for under the initial loading; with the development of loading, the normal stress of joint plane rises up gradually until the serrations are cut off. Then, after a short time, a slight slippage will appear due to friction-induced by broken serrations. Hence, we can ensure that, although the stability type of sample is identified as fracture instability, the total instability process is a transformation from initial fracture to final slight slipping instability. Within the total process, under the increasing of loading, the pore of joint plane is compressed, and the friction of joint plane is consumed to resist shearing deformation until serrations are cut off. It is noted that the criterion factor of sample with undulating angle of 15° is relatively small, which is thought that, although the serrations are cut off and its damage is not obvious, the shearing deformation still experiences fracture instability, and then, the slipping instability follows it.

In order to further illustrate different instability types under variable criterion factor, as shown in Fig. 13, it can be observed that, when the shear angle increases and undulating angle decreases gradually, the slipping instability will happen before applying loading, which can be found that the yellow area evolving into red area is corresponding to criterion factor exceeding to 0 gradually. Hence, the unstable regions are represented within the red boundary in Fig. 13; when \({\eta }_{d}\) under the combination of two kinds of angles varies in this region, slipping instability will happen. When criterion factor is less than 0 (green boundary), the instability type under variable angle shear action is thought of as fracture instability, which also matches with our experimental results. It should be clarified that color projection in the XY plane characterizes the variations of \({\eta }_{d}\); with transformation from red to purple, \({\eta }_{d}\) gradually decreases, and it is more difficult to induce slipping instability due to the changes of joint planes under the action of external actions.

Fig. 13
figure 13

Effect of \(\alpha\) and \(\theta\) on \({\eta }_{d}\)

Full size image

But during the above mentioned, we found that the slipping instability will follow the fracture instability within the criterion factor less than 0, which can be interpreted that, after fracture instability, the initial serrations have been cut off, and some new and disorder undulating angle will appear; the coupling effect of new undulating angle and shear angle will result in part of slipping instability. Based on the above discussions, the reasonability of the proposed criterion factor has obtained verification, and it will provide further reference in study of instability type induced by variable angle shear action.

Conclusions

In order to investigate the coupling effect of shear angle and undulating angle on shear mechanical behaviors of red sandstone, a series of variable angle shear tests were carried out on red sandstone samples including different serrations joints, and the acoustic emission tests and DIC method were also introduced to reveal failure mechanisms, modes, and characteristics of samples. The main conclusions are shown as follows:

  1. (1)

    The load–displacement curves under various shear angles and undulating angles can be divided into five stages, i.e., initial crack compaction stage, linear deformation stage, pre-peak crack development stage, post-peak load drop stage, and plastic slip stage. The variations of shear strength of samples under different shear angles and undulating angles were discussed in details.

  2. (2)

    Based on the acoustic emission testing results, the evolutions of shear fracture ratio under different shear and undulating angle were given to distinguish the fracture mode of this experiment follows shear fracture. By comparing macro-failure characteristics, fracture modes, and strain fields obtained from DIC tests, we can see with an increase in shear angle that the fracture mode will be transformed to shear fracture, and the composition of shear fracture will play a main role in total fracture deformation.

  3. (3)

    By performing analysis of instability types and mechanics mechanisms, we proposed a novel criterion factor of instability \({\eta }_{d}\) to identify the variable angle shear instability. When \({\eta }_{d}>0\), the instability type is thought of as slipping instability, and the serrations will be damaged slightly. When \({\eta }_{d}\le 0\), the instability type is fracture instability, and the serrations are broken seriously and the bearing capacity will drop down.