1 Introduction

Along with the rapid development of 5G network transmission technology, digital images have become the major carrier of information in the network. However, while Internet technology has facilitated communication, it has also brought security risks that cannot be ignored. Therefore, how to ensure the secure transmission of image information in the network has become a hot research direction of scholars.

Encrypting images through encryption algorithms is one of the effective methods to safeguard the image security. However, because of the features of digital images, for instance, large amount of information, high correlation and data redundancy, classical encryption algorithms DES, AES, IDEA, etc., fail to satisfy the requirements of digital image encryption commendably [1]. For this reason, the exploration of new encryption methods to encrypt image data is absolutely necessary. So far, scholars have designed many different types of image encryption methods, which are mostly on the basis of chaos [2, 3], DNA encoding [4, 5], compressed sensing [6,7,8] and other theories [9,10,11,12]. In the above schemes, chaotic systems have strong initial value sensitivity, ergodicity and unpredictability, which make chaotic systems naturally related to encryption in cryptography [13]. Therefore, encryption algorithms combined with chaotic systems have become an important research topic.

Friedrich [14] utilized the chaotic system to image encryption firstly and proposed the general structure of image encryption, “permutation and diffusion.” With the deepening of research, a variety of image encryption methods based on chaos have been proposed. To reduce the computational complexity of the algorithm calculations and elevate the effectiveness of the permutation process, Wang et al. [15] constructed an encryption method for colorful images through applying piecewise linear chaotic map to perform random permutation on heterogeneous planes. Li et al. [16] improved the tent map to make its initial conditions and parameters very sensitive to subtle changes, enhancing the encryption safety. Wang et al. [17] devised a novel encryption scheme by combining DNA coding rules and the improved logistic map, while the fact that this scheme increases the key space, its efficiency of encryption is reduced. Zheng et al. [18] invented a modified 2D logistic sine map and proposed a unique encryption algorithm that combined with dynamic DNA sequence, which increases the complexity and security of this encryption scheme. Based on 3D chaotic maps, Qian et al. [19] designed an efficient color image encryption scheme, but this encryption scheme has certain requirements on the magnitude of original images.

The encryption schemes mentioned above are all proposed based on low-dimensional chaotic systems. Although they can achieve the purpose of image encryption, with the advancement of chaos research, disadvantages of low-dimensional chaotic systems have been gradually exposed [20]. Because the state space of the low-dimensional chaotic system is small, its system behavior is simpler and easier to analyze than the high-dimensional chaotic system, which leads to the poor security of image encryption algorithm based on low-dimensional chaos. Therefore, the research hotspot now gradually shifts to the study of new image encryption methods based on high-dimensional chaotic systems. Currently, there have been some encryption algorithms proposed based on high-dimensional dissipative chaotic systems [21,22,23]. However, dissipative chaotic systems are also not very secure when applied to encryption. Since dissipative systems have attractors, its trajectory cannot reach most of the space near the attractors; the ergodicity is poor. And the risk of the attacker cracking the encryption scheme by reconstructing the attractors is greatly increased.

In contrast to dissipative chaotic systems, for conservative chaotic systems, there are no attractors and no risk of being attacked by reconstructed attractors, and its trajectory can almost traverse the entire phase space. Especially for the Hamiltonian conservative chaotic system, in addition to phase space conservation, its Hamiltonian energy is always conserved, so that Hamiltonian conservative chaotic systems have stronger ergodicity than dissipative systems. In addition, hyperchaotic systems have at least two LEs greater than 0, which exhibit higher randomness compared to other systems. Therefore, we design a novel 5D Hamiltonian conservative hyperchaotic system by constructing the Euler rotation equation, and the energy analysis of the system proves that chaos can be generated. With this as a framework, we construct a novel image encryption scheme, which can realize the secure encryption of any gray and color images. Moreover, we also propose a new bit-plane segmentation method and a bit-plane diffusion operation to enhance the effect of image diffusion. Simulation results and security analysis show that our algorithm is more secure than other algorithms.

The subsequent parts of this paper are constituted as described in the following. In Sect. 2, the design and construction of a novel 5D Hamiltonian conservative hyperchaotic system is recommended, and its Hamiltonian energy and Casimir energy are analyzed. Section 3 explores the fundamental features of this new system. Section 4 designs a new image encryption method using this novel chaotic system. Section 5 simulates the devised encryption method and compares the safety and reliability analysis results with other encryption methods. Finally, Sect. 6 summarizes this paper.

2 System design and model construction

2.1 Prerequisites

Maschke et al. [24] constructed a generalized Hamiltonian system to study the stability of forced Hamiltonian system with dissipation. It can be described as

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {{\dot{{{\textbf {x}}}}} = [J({{\textbf {x}}}) - R({{\textbf {x}}})]\nabla H({{\textbf {x}}}) + g({{\textbf {x}}})u}\\ {y = {g^T}({{\textbf {x}}})\nabla H({{\textbf {x}}})} \end{array}} \right. \end{aligned}$$
(1)

where \({{\textbf {x}}}\) is the state variable vector, and \({{\textbf {x}}} = [{x_1}, \ldots ,{x_n}]^T\); \(J({{\textbf {x}}})\) is a symplectic matrix and satisfies \(J({{\textbf {x}}}) = -{J^T}({{\textbf {x}}})\), which represents the conservative part of the system; \(R({{\textbf {x}}})\) is a positive semi-definite matrix, which satisfies \(R(\mathrm{{x}}) = {R^T}(\mathrm{{x}})\), represents the energy dissipation part of the system. \(H({{\textbf {x}}})\) represents the Hamiltonian energy function of the system; \(g({{\textbf {x}}})\) and u represent the input gain vector and the input of the generalized force received by the system, respectively; and y is the output of the system.

The variation of the Hamiltonian energy \(H({{\textbf {x}}})\) is related to the internal dissipation and the external energy exchange of the system, which can be expressed as

$$\begin{aligned} \frac{{\textrm{d}H}}{{\textrm{d}t}} = - \nabla {H^T}({{\textbf {x}}})R({{\textbf {x}}})\nabla H({{\textbf {x}}}) + {u^T}y \end{aligned}$$
(2)

When the external input and the energy dissipation term in Eq. (2) are 0, the Hamiltonian energy of the system is conserved. In this case, the system is called the generalized Hamiltonian conservative system, and its differential equation can be described as:

$$\begin{aligned} {\dot{{\textbf {x}}}} = J({{\textbf {x}}})\nabla H({{\textbf {x}}}) \end{aligned}$$
(3)

2.2 Construct the 5D Euler equation

The Euler rotation equation is often used to describe the motion of rotating rigid bodies and incompressible fluids [25], which has great value in the field of classical mechanics. According on the theory of [26,27,28], Euler rotation equation can be used to describe dissipative chaotic systems, its corresponding Hamiltonian vector field conforms to the generalized Hamiltonian system shown in Eq. (1). And the generalized Hamiltonian conservative system can be derived by constructing the Euler equation of rigid body without external force. Then, by adding external constants to break the Casimir energy conservation, it is possible to create a Hamiltonian conservative chaotic system [27,28,29,30]. Furthermore, the dynamic specifics of this chaotic system can be validated through analyzing the Casimir energy and Hamiltonian energy.

According to the above theory, a 5D Euler rotation equation describing the motion of a rigid body without external force is designed, which is denoted as system \(\Sigma _5\), and its Hamiltonian vector field satisfies the generalized Hamiltonian conservative system shown in Eq. (3).

And the Hamiltonian energy function \(H({{\textbf {x}}})\) of system \(\Sigma _5\) can be obtained as

$$\begin{aligned} H({{\textbf {x}}}) = \frac{1}{2}\sum \limits _{i = 1}^5 {{\Pi _i}{x_i}^2} \end{aligned}$$
(4)

where \(x_i\) expresses the rotational angular momentum, \(i = 1, 2, 3, 4, 5\). \(\Pi _i = I_i^{-1}\), and \(I_i\) is the principal moment of inertia. In addition, the gradient function of \(H({{\textbf {x}}})\) is \(\nabla H({{\textbf {x}}}) = {[{\Pi _1}{x_1},{\Pi _2}{x_2},{\Pi _3}{x_3},{\Pi _4}{x_4},{\Pi _5}{x_5}]^{T}}\).

Moreover, \(J({{\textbf {x}}})\) is a symplectic matrix, which is defined as

$$\begin{aligned} J({{\textbf {x}}}) = \left[ {\begin{array}{*{20}{c}} 0&{}{{x_4}}&{}0&{}{ - {x_2}}&{}0\\ { - {x_4}}&{}0&{}{ - {x_5}}&{}{{x_1}}&{}{{x_3}}\\ 0&{}{{x_5}}&{}0&{}0&{}{ - {x_2}}\\ {{x_2}}&{}{ - {x_1}}&{}0&{}0&{}0\\ 0&{}{ - {x_3}}&{}{{x_2}}&{}0&{}0 \end{array}} \right] \end{aligned}$$
(5)

The variation of Hamiltonian energy for system \(\Sigma _5\) can be denoted as the differential of the Hamiltonian energy with time. Since \(J({{\textbf {x}}})\) of system \(\Sigma _5\) is a symplectic matrix, \(J({{\textbf {x}}}) = -{J^T}({{\textbf {x}}})\), and \(\dot{H}({{\textbf {x}}})\) can be calculated from Eq. (6).

$$\begin{aligned} \dot{H}({{\textbf {x}}}) = \nabla H({{\textbf {x}}})^{T}J({{\textbf {x}}})\nabla H({{\textbf {x}}}) = 0 \end{aligned}$$
(6)

Within the rigid body dynamic realm, Casimir energy function C plays a significant role, and is a powerful means of describing dynamic systems and analyzing their stability conditions. According to reference [27,28,29,30], it is found that Casimir energy has an important relationship with whether the system can produce chaos. Chaos cannot occur in a system when both its Hamiltonian energy and Casimir energy are conservative. Analyzing the Casimir energy of system \(\Sigma _5\), the function of Casimir energy changing with time is known as the Casimir power \(\dot{C}\), which is defined as

$$\begin{aligned} \dot{C} = \{C, H\} = \nabla C({{\textbf {x}}})^{T} \cdot {\dot{{\textbf {x}}}} \end{aligned}$$
(7)

And the Casimir energy gradient function is \(\nabla C({{\textbf {x}}}) = {[{x_1},{x_2},{x_3},{x_4},{x_5}]^{T}}\); thus, the Casimir power can be obtained as

$$\begin{aligned} \dot{C} = \nabla C({{\textbf {x}}})^{T}J({{\textbf {x}}})\nabla H({{\textbf {x}}}) = 0 \end{aligned}$$
(8)

According to Eqs. (6) and (8), both the Hamiltonian energy and Casimir energy remain conservative for system \(\Sigma _5\), so the system \(\Sigma _5\) cannot generate chaotic behavior. However, it can be regarded as the basic framework for constructing Hamiltonian conservative chaotic system.

2.3 Modeling the 5D Hamiltonian conservative hyperchaotic system

From reference [30], it can be concluded that breaking the conservation of Casimir energy is a crucial element in the formation of chaotic behavior within the system. In order to construct a 5D Hamiltonian conservative chaotic system, we need to break the Casimir energy conservation within system \(\Sigma _5\). By adding an external nonzero parameter c to \(J({{\textbf {x}}})\) can obtain System \(\Sigma _5^H\). And \({J_C}({{\textbf {x}}})\) can be obtained as

$$\begin{aligned} {J_C}({{\textbf {x}}}) = \left[ {\begin{array}{*{20}{c}} 0&{}{{x_4}}&{}0&{}{ - {x_2}}&{}0\\ { - {x_4}}&{}0&{}{ - {x_5}}&{}{{x_1}}&{}{{x_3}}\\ 0&{}{{x_5}}&{}0&{}0&{}{ - {x_2}}\\ {{x_2}}&{}{ - {x_1}}&{}0&{}0&{}{ - c}\\ 0&{}{ - {x_3}}&{}{{x_2}}&{}c&{}0 \end{array}} \right] \end{aligned}$$
(9)

System \(\Sigma _5^H\) is denoted as

$$\begin{aligned} {\dot{{\textbf {x}}}} = J_C({{\textbf {x}}})\nabla H({{\textbf {x}}}) \end{aligned}$$
(10)

For system \(\Sigma _5^H\), its mathematical model is expressed by

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {{{\dot{x}}_1} = ({\Pi _2} - {\Pi _4}){x_2}{x_4}}\\ {{{\dot{x}}_2} = ({\Pi _4} - {\Pi _1}){x_1}{x_4} + ({\Pi _5} - {\Pi _3}){x_3}{x_5}}\\ {{{\dot{x}}_3} = ({\Pi _2} - {\Pi _5}){x_2}{x_5}}\\ {{{\dot{x}}_4} = ({\Pi _1} - {\Pi _2}){x_1}{x_2} - c{\Pi _5}{x_5}}\\ {{{\dot{x}}_5} = ({\Pi _3} - {\Pi _2}){x_2}{x_3} + c{\Pi _4}{x_4}} \end{array}} \right. \end{aligned}$$
(11)

By analyzing the Hamiltonian energy of new system \(\Sigma _5^H\), the result can be obtained that \(\dot{H}({{\textbf {x}}}) = \nabla H({{\textbf {x}}})^{T}{J_C}({{\textbf {x}}})\nabla H({{\textbf {x}}}) = 0\); hence, its Hamiltonian energy is conservative. However, for system \(\Sigma _5^H\), its Casmir power can be calculated as

$$\begin{aligned} \begin{aligned} \dot{C}&= \nabla {C^T}({{\textbf {x}}}){J_C}({{\textbf {x}}})\nabla H({{\textbf {x}}}) \\&= c({\Pi _4} - {\Pi _5}){x_4}{x_5} \end{aligned} \end{aligned}$$
(12)

As can be seen from Eq. (12), when \(c({\Pi _4} - {\Pi _5}){x_4}{x_5} \ne 0\), that is, \(c \ne 0\) and \({\Pi _4} \ne {\Pi _5}\), the Casimir energy is not conservative, and system \(\Sigma _5^H\) can produce chaotic behavior.

Furthermore, the above results can also be confirmed through simulation experiments. Set the system control parameters as \(\left( {{\Pi _1},{\Pi _2},{\Pi _3},{\Pi _4},{\Pi _5}} \right) = (9,7,5,4,8)\), the initial values \(({{x_{10}},{x_{20}},{x_{30}},{x_{40}},{x_{50}}}) = (1,1,1,1,1.5)\). And choose MATLAB ode45 as the solver, the time step is \(T = 0.01\). When the parameter c is set to 0, it is equivalent to system \(\Sigma _5\), and when \(c = 1\), it represents to system \(\Sigma _5^H\). The energy analysis results of system \(\Sigma _5\) and system \(\Sigma _5^H\) are shown in Fig. 1a. And the black line indicates the Hamiltonian energy of the two systems, indicating that under this parameter, the Hamiltonian energy is always 21.54, which is conserved. Red represents the Casimir power trajectory of system \(\Sigma _5\), which is always zero, indicating the conservation of Casimir energy. Blue represents the Casimir power trajectory of system \(\Sigma _5^H\). It can be seen that the trajectory presents irregular oscillation, indicating that its Casimir energy is not conservative. And the phase trajectories of both systems are depicted in Fig. 1b, where the red represents system \(\Sigma _5\) and the blue represents system \(\Sigma _5^H\). As Fig. 1b shows, system \(\Sigma _5\) shows periodic motion, while the trajectory of system \(\Sigma _5^H\) is unstable aperiodic orbits.

The above analysis results demonstrate that the system \(\Sigma _5^H\) with Hamiltonian energy conservation can generate chaos.

Fig. 1
figure 1

System \(\Sigma _5\) and System \(\Sigma _5^H\). a Energy analysis diagram; b the \({x_1} - {x_2} - {x_3}\) space phase diagram

3 System analysis

Unless otherwise noted, the default parameters and initial values for this section are (\({{\Pi _1},{\Pi _2},{\Pi _3},}\) \({{\Pi _4},{\Pi _5},c}) = (9,7,5,4,8,1)\) and \(({{x_{10}},{x_{20}},{x_{30}},{x_{40}},}\) \({{x_{50}}}) = (1,1,1,1,1.5)\), respectively.

3.1 Dynamics analysis

3.1.1 Conservative analysis

For the vector field F, its phase space volume change rate can be expressed by divergence \(\nabla \cdot F\) [31]. According to the magnitude of the divergence \(\nabla \cdot F\), whether the chaotic system is conservative or dissipative can be distinguished. In the case of \(\nabla \cdot F\) being negative, the system is dissipative, and the trajectories gradually gather near the attractor. When \(\nabla \cdot F\) equals zero, the system is conservative and has a constant phase volume. For system \(\Sigma _5^H\), its divergence \(\nabla \cdot F\) can be obtained as

$$\begin{aligned} \begin{aligned} \nabla \cdot F&= \frac{{\partial {F_{{x_1}}}}}{{\partial {x_1}}} + \frac{{\partial {F_{{x_2}}}}}{{\partial {x_2}}} + \frac{{\partial {F_{{x_3}}}}}{{\partial {x_3}}} + \frac{{\partial {F_{{x_4}}}}}{{\partial {x_4}}} + \frac{{\partial {F_{{x_5}}}}}{{\partial {x_5}}} \\&= 0 \end{aligned} \end{aligned}$$
(13)

It can be confirmed from the above equation that system \(\Sigma _5^H\) is conservative.

3.1.2 LEs and bifurcation diagram analysis

Sensitivity to control parameters and initial values is one of the most basic characteristics of chaotic systems. This is usually analyzed by Lyapunov exponents diagram and bifurcation diagram. And in this section, Fig. 2 illustrates the LEs diagram and bifurcation diagram varying with initial value \(x_{20}\).

Fig. 2
figure 2

System \(\Sigma _5^H\). a LEs with initial value \(x_{20}\); b bifurcation diagram with initial value \(x_{20}\)

From Fig. 2a, it can be found that the sum of LEs is always 0, and there are always two positive LEs greater than 0 within a wide initial value range, indicating that system \(\Sigma _5^H\) has high stability and complexity. Moreover, Fig. 2b manifests that as the initial values increase, the phase space traversal range expands. Furthermore, the unpredictability of sequences produced by system \(\Sigma _5^H\) is also stronger.

Specifically, when \({x_{20}} = 1\), the LEs of system \(\Sigma _5^H\) are calculated to be (0.6030, 0.0037,  \(0.0001, - 0.0050, - 0.6018)\), the sum of all LEs is 0, and there are two positive LEs \(L{E_1} = 0.6030\) and \(L{E_2} = 0.0037\), which satisfy the characteristics of conservative hyperchaotic systems. And the Lyapunov dimension of system \(\Sigma _5^H\) can also be given by the Kaplan–Yorke form:

$$\begin{aligned} \begin{aligned} D&= 4 + \frac{{0.6030 + 0.0037 + 0.0001 - 0.0050}}{{0.6018}}\\&= 5 \end{aligned} \end{aligned}$$
(14)

As Eq. (14) shows, the Lyapunov dimension of system \(\Sigma _5^H\) is an integer dimension, which conforms to the characteristics of conservative systems.

3.1.3 Phase diagram and Poincaré section diagram analysis

Phase diagram is a relatively intuitive method to observe the movement trajectory of chaotic system. Figure 3 displays the partial phase diagrams of system \(\Sigma _5^H\). The motion trajectory of system \(\Sigma _5^H\) fills almost the entire phase space, which proves that system \(\Sigma _5^H\) has strong ergodicity and high complexity.

Fig. 3
figure 3

Results of phase diagram. a Plane \({x_1} - {x_2}\); b Plane \({x_1} - {x_4}\); c Plane \({x_2} - {x_5}\); d Plane \({x_3} - {x_5}\)

Poincaré section diagram is a common method to analyze the motion state of the system. Figure 4 shows the Poincaré section diagrams of a partial plane for system \(\Sigma _5^H\). The Poincaré section of system \(\Sigma _5^H\) covers almost the entire trajectory and has no stable line or closed loop, which proves the chaos of the system.

Fig. 4
figure 4

Results of Poincare section diagram. a Plane \({x_1} - {x_2}\); b Plane \({x_1} - {x_4}\); c Plane \({x_2} - {x_5}\); d Plane \({x_3} - {x_5}\)

3.1.4 Initial value sensitivity analysis

The system parameters sensitivity and initial values sensitivity to chaotic systems are two crucial properties that ensures the safety of encryption applications. To observe the effect of various initial values on system \(\Sigma _5^H\), we selected different initial values and obtained the corresponding chaotic sequences within the time \(t \in [0,100]\), and the sampling time is 0.5. The results are shown in Fig. 5.

Fig. 5
figure 5

Results of initial value sensitivity analysis

As depicted in Fig. 5, minor differences in the initial values lead to significant variations in the resulting chaotic sequences, highlighting the extreme sensitivity to initial conditions of system \(\Sigma _5^H\).

3.2 Pseudo-randomness analysis

The safety of chaotic encryption applications is directly associated with the randomness of chaotic sequences. In an effort to confirm the randomness of system \(\Sigma _5^H\), we apply the United States National Institute of Standards and Technology (NIST) SP80022 standard [32] to appraise the test data, which is the chaotic sequence produced by system \(\Sigma _5^H\). NIST offers 15 statistical tests to appraise the stochasticity of random sequences or pseudo-random number generators. Each statistical test is given a P value, and by judging the P values, it is determined whether the sequence is random. For a pseudo-random sequence, its NIST test results should satisfy three conditions:

  1. (1)

    Select the default significance level \(\alpha = 0.01\), and for each test item, the P value cannot be less than \(\alpha \);

  2. (2)

    Select the test sequence data length as l million bits, and divide the data into s groups equally, and the confidence interval can be calculated by \({\hat{p}} \pm 3\sqrt{{\hat{p}}(1 - {\hat{p}})/s}\), where \({\hat{p}} = 1 - \alpha \). Set \(l = 100\), \(s = 100\), the proportion of P values that are greater than or equal to \(\alpha \) should be within the confidence interval [0.9602, 1] [33];

  3. (3)

    P values distribution obeys uniformity.

Set the parameters as \(({{\Pi _1},{\Pi _2},{\Pi _3},{\Pi _4},{\Pi _5},c}) = (9,7,5,4,8,1)\), and the initial values as \((x_{10},x_{20},x_{30},x_{40},x_{50}) = (1,1,1,1,1.5)\). Select the time interval \(t \in [0,200]\) and the sample time \(T = 10^{-5}\) to generate chaotic sequences \(x_1,x_2,x_3,x_4,x_5\). Then, the chaotic sequence X with a length of 100 million bits can be obtained, and \(X = [{x_1},{x_2},{x_3},{x_4},{x_5}]\).

Further, as shown in Eq. (15), convert the chaotic sequence X into test data composed of 0 and 1, and divide it into 100 groups for NIST testing.

$$\begin{aligned} X = \bmod (floor(X \times {10^{15}}),2) \end{aligned}$$
(15)

where mod() is the modulus operation, and floor() represents rounding down operation.

Table 1 explicates the outcomes of the test; for items with multiple sub-tests, the average value is selected as result.

Table 1 NIST test results of system \(\Sigma {_5^H}\)

In accordance with Table 1, the test sequence passes all test items, and all pass rates are greater than or equal to 0.9602, which satisfies conditions (1) and (2). In addition, the uniformity of P values distribution can be verified by a histogram. By dividing the range from 0 to 1 into 10 equidistant sub-ranges, the quantities of P values are illustrated in each sub-range of the histogram. Histogram analysis is performed with the non-overlapping test item, and the result is drawn in Fig. 6. The P values are evenly distributed, satisfying condition (3).

Fig. 6
figure 6

The non-overlapping template P value histogram

As the above results indicate, the test sequence produced by system \(\Sigma {_5^H}\) meets all the test standards, and the chaotic test sequence has good pseudo-randomness. Therefore, system \(\Sigma {_5^H}\) has the capability to be a secure pseudo-random number generator.

4 Design of the image encryption algorithm

A novel diffusion method based on the bit-plane segmentation is designed in this part. And on this basis, an image encryption scheme based on the 5D Hamiltonian conservative hyperchaotic system is proposed. Figure 7 shows the encryption process. The scheme can encrypt both gray and color images of any size, which consists of two parts, namely, key production and plaintext image encryption.

4.1 Initial key production

The initial key is constructed of two segments, namely the SHA-256 sequence HK of the original image, where HK is a hexadecimal sequence with a length of 64, and the initial values of the chaotic system \((x_0,y_0,z_0,u_0,v_0)\). In addition, the encryption key K can be generated by the following process.

Step 1 Convert the hash code HK into a string;

$$\begin{aligned} HK = num2str(HK) \end{aligned}$$
(16)

where num2str() is the number-to-string function.

Step 2 Divide HK into six parts and calculate to obtain \({hk}_0,{hk}_1,{hk}_2,{hk}_3,{hk}_4,{hk}_5\);

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {h{k_0} = sum(HK)}\\ {h{k_1} = sum(HK(1:12))/h{k_0}}\\ {h{k_2} = sum(HK(13:24))/h{k_0}}\\ {h{k_3} = sum(HK(25:36))/h{k_0}}\\ {h{k_4} = sum(HK(37:48))/h{k_0}}\\ {h{k_5} = sum(HK(49:60))/h{k_0}} \end{array}} \right. \end{aligned}$$
(17)

where sum() represents the summation operation.

Step 3 Add the processed hash values to the initial values to obtain the encryption key K.

$$\begin{aligned} K = ({x_0} + {{hk}_1},{y_0} + {{hk}_2},{z_0} + {{hk}_3},{u_0} + {{hk}_4},{v_0} + {{hk}_5}) \end{aligned}$$
(18)
Fig. 7
figure 7

Encryption algorithm flowchart

4.2 Encryption algorithm description

To enhance the effect of high-dimensional chaotic systems on encryption algorithms and improve the diffusion effect, a new bit-plane segmentation operation is designed. The encryption algorithm includes one scrambling and two times diffusion operations, which improve the encryption strength.

4.2.1 Bit-plane segmentation

Figure 8 shows the bit-plane segmentation operation process. The original image is converted into an 8-bit binary array; next, the image array is divided into four groups according to the pattern of two-bit binary numbers. Further processing can produce four sub-images with pixels ranging from 0 to 3. Subsequently, the pseudo-random sequences generated by chaotic system \(\Sigma {_5^H}\) are used for bit-level diffusion, which make the diffusion more sufficient and improve the security of the encryption algorithm.

Fig. 8
figure 8

The bit-plane segmentation process

4.2.2 Encryption process

Encrypt the original image P with size \(M \times N \times A\), and A represents the quantity of channels of the image, RGB color images correspond to \(A = 3\), gray images correspond to \(A = 1\). And the following is the detailed encryption process.

Step 1 Take the encryption key K into the 5D conservative hyperchaotic system as initial values to produce chaotic sequences X0,  Y0, Z0, U0 and V0 with the length of \(M \times N \times A + len\), where len is the discarded length to prevent the implication of transient data, and it is set to 1000. Then, X1, X2, Y1, Z1, U1 and V1 with a size of \(M \times N \times A\) can be calculated by processing as follows:

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {[\sim ,X1] = sort(X0)}\\ {X2 = \bmod (round(X0 \times {{10}^{15}}),256)}\\ {Y1 = \bmod (round(Y0 \times {{10}^{15}}),4)}\\ {Z1 = \bmod (round(Z0 \times {{10}^{15}}),4)}\\ {U1 = \bmod (round(U0 \times {{10}^{15}}),4)}\\ {V1 = \bmod (round(V0 \times {{10}^{15}}),4)} \end{array}} \right. \end{aligned}$$
(19)

where sort() is the function for sorting, and round() denotes the rounding function.

Step 2 Transform image P into an array with one dimension. Then, perform a permutation operation on P and the index sequence X1 to acquire P1;

$$\begin{aligned} P= & {} reshape(P,M \times N \times A,1) \end{aligned}$$
(20)
$$\begin{aligned} P1(k)= & {} P(X1(k)) \end{aligned}$$
(21)

where reshape() is the function to reshape the array, \(k = 1,2, \ldots ,M \times N \times A\).

Step 3 Implement the bit-plane segmentation operation on P1. Firstly, convert P1 into 8-bit binary matrix S, and S is divided into four subsequences s1, s2, s3, s4 by bit-plane segmentation.

$$\begin{aligned} S(:,:)= & {} dec2bin(P1,8) \end{aligned}$$
(22)
$$\begin{aligned}{} & {} \left\{ {\begin{array}{*{20}{l}} {s1 = bin2dec(s(:,1:2))}\\ {s2 = bin2dec(s(:,3:4))}\\ {s3 = bin2dec(s(:,5:6))}\\ {s4 = bin2dec(s(:,7:8))} \end{array}} \right. \end{aligned}$$
(23)

where dec2bin() is the decimal to binary conversion function and bin2dec() is the binary-to-decimal function.

Step 4 Perform XOR processing on subsequences s1, s2, s3, s4 and pseudo-random sequences \(Y1,Z1,U1,V1\), respectively. And the processed results are transformed into binary form to obtain sequence d1, d2, d3, d4;

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {d1 = dec2bin(bitxor(s1,Y1),2)}\\ {d2 = dec2bin(bitxor(s2,Z1),2)}\\ {d3 = dec2bin(bitxor(s3,U1),2)}\\ {d4 = dec2bin(bitxor(s4,V1),2)} \end{array}} \right. \end{aligned}$$
(24)

where bitxor() is a bitwise XOR function.

Step 5 Merge the sequences d1, d2, d3 and d4, to get the 8-bit binary sequence D. And by converting D into decimal, the preliminary diffusion result P2 is obtained;

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {D = \left[ {\begin{array}{*{20}{c}} {d1}&{}{d2}&{}{d3}&{}{d4} \end{array}} \right] }\\ {P2 = dec2bin(D)} \end{array}} \right. \end{aligned}$$
(25)

Step 6 Perform XOR processing on P2 and sequence X2 to complete secondary diffusion to obtain P3, and finally restore P3 to the \(M \times N \times A\) array to get the encrypted result E.

$$\begin{aligned} \left\{ {\begin{array}{*{20}{l}} {P3 = bitxor(P2,X2)}\\ {E = reshape(P3,M,N,A)} \end{array}} \right. \end{aligned}$$
(26)

4.3 Decryption algorithm description

The decryption algorithm is the inverse operation of the encryption algorithm, and the decryption steps of ciphertext image C with the size of \(M \times N \times A\) are briefly introduced as follows:

Step 1 Calculate the decryption key based on the hash sequence and initial values in the initial key. The decryption key generation process is shown in Sect. 4.1.

Step 2 Take the decryption key into the 5D conservative hyperchaotic system as initial values to produce chaotic sequences, and further processing to get the index sequence and pseudo-random sequences. The detailed process is shown in step 1 of Sect. 4.2.

Step3 Decrypt the ciphertext image C according to the reverse process of the encryption process from step 2 to step 6 in Sect. 4.2, and get the plaintext image P.

5 Simulation results and security analysis

For the purpose of validating the security of our proposed image encrypting method, we choose grayscale images sized \(256 \times 256\), such as Cameraman and Lena, and several images in the image dataset USC-SIPI with varying sizes, including color images and gray images, for testing the constructed image encryption algorithm. And the security efficacy of the encryption method is assessed by the factors such as information entropy, correlation, and anti-attack. We use MATLAB R2020a to carry out simulation experiments under Windows 10 system. The computer hardware is set to Intel Core i5-1135G7 CPU (2.40 GHz), 16.0 GB RAM memory. The computer precision is set to 15, and the initial values are \({x_0}=1, {y_0}=1, {z_0}=1.5, {u_0}=1, {v_0}=1\).

5.1 Simulation results

The color image House (4.1.05) and the gray image alarm Clock (5.1.12) with a size of \(256 \times 256\) and the gray image Bridge (5.2.10) with a size of \(512 \times 512\) are encrypted with the proposed encryption method. Figure 9 exhibits the results of simulation.

Fig. 9
figure 9

Results of simulation experiments. ac Source images; df ciphertext; gi decrypted results

5.2 Safety performance analysis

5.2.1 Key sensitivity analysis

The reliability of encryption algorithms is largely affected by the sensitivity of the key, any minor alteration to the key must generate entirely distinct encryption and decryption results. To confirm the sensitivity of our encryption method to the key, we perform encryption and decryption to the plaintext Lena with the correct key, then change only one initial value of the key from \(x_0\) to \(x_0 + 10^{-15}\), and the rest is unchanged, the modified key is applied for decryption. Figure 10 shows the outcome of decryption.

As Fig. 10 illustrates, changing the key results in the inability to decrypt the plaintext from the ciphertext, which explicates that our proposed encryption scheme possesses strong key sensitive.

Fig. 10
figure 10

Results of key sensitivity analysis. a Ciphertext; b decrypted outcome by the initial key; c decrypted outcome by the changed key

5.2.2 Analysis of key space

Encryption methods that possess a larger key space are more resistant to violent attacks, and Alvarez et al. [34] suggested that minimum size of the key space must be no smaller than \(2^{100}\) for secure encryption algorithms. For our proposed encryption method, the key consists the SHA-256 sequence and initial values \(({x_0,y_0,z_0,u_0,v_0})\). When calculation precision of the operating environment is \(10^{-15}\), the key space can be calculated as \({2^{256}} \times {10^{15}} \times {10^{15}} \times {10^{15}} \times {10^{15}} \times {10^{15}} \approx {2^{505}}\). And the result of comparing the key space in our encryption scheme with that in other schemes is expressed in Table 2; bold represents the best results.

Table 2 Key space comparison

As determined in Table 2, our method owns a wider key space; it has a greater capacity to withstand violent attacks.

5.2.3 Analysis of histogram

The pixel distribution of images can be accurately displayed on the histogram. And the more even distribution of the histogram indicates that the safety of the encryption scheme is greater. For the grayscale image Lena, Fig. 11 reveals its plaintext histogram and corresponding ciphertext histogram.

Fig. 11
figure 11

Results of histogram analysis. a Plaintext; b ciphertext

It is evident that the pixel values of the encrypted image are uniformly distributed, which makes it hard for attackers to extract valuable information from the histogram of the encrypted image. To further validate the evenness of pixel distribution in ciphertext images, we perform the Chi-square test to appraise encrypted images. And the Chi-square test result \({\chi ^2}\) can be derived as

$$\begin{aligned} {\chi ^2} = \sum \limits _{k = 0}^{255} {\frac{{{{({p_k} - p)}^2}}}{p}} \end{aligned}$$
(27)

where k is the pixel value, \(p_k\) is the quantity of pixel k, and p denotes the expected quantity of pixel k, \(p = (M \times N \times A)/256\).

Set the confidence level is 95%, i.e., the significance level \(\alpha = 0.05\), the \(\chi ^2\) value satisfying the test criteria should be less than 293.24783. Moreover, the results of Chi-square test for source images and their ciphertexts are manifested in Table 3. All encryption results of test images are far below 293, indicating that the ciphertext pixels are uniformly distributed, and it proves that our encryption method exhibits good resistance to statistical attacks.

Table 3 Results of Chi-square test

5.2.4 Information entropy analysis

As a conventional index to judge the effectiveness of image encryption methods, information entropy can reflect the disorder and randomness of image information. Information entropy can be obtained as

$$\begin{aligned} E(x) = - \sum \limits _{i = 0}^{{2^N} - 1} {P({x_i})\textrm{log}_2{P({x_i})}} \end{aligned}$$
(28)

where \(x_i\) is the pixel value, N represents the bits quantity of \(x_i\), and \(P(x_i)\) denotes the probability of pixel \(x_i\). And for the 256 grayscale ciphertext image, which N equals 8, its information entropy theory value is 8.

We have compared the information entropy of the ciphertexts between our scheme and other algorithms, and the results are presented in Table 4; bold indicates the optimal value. It demonstrates that our encryption algorithm produces ciphertext with information entropy that are closer to 8, and it signifies the designed method possesses a greater level of security.

5.2.5 Correlation analysis

Adjacent pixels in meaningful images display a noticeable correlation, and encryption algorithms should minimize the adjacent pixels correlation in order to provide better protection for the security of encrypted images. The evaluation of correlation analysis ordinarily involves the analyses of horizontal correlation, vertical correlation and diagonal correlation. And correlation coefficient can be derived by Eq. (32).

$$\begin{aligned}{} & {} E(p) = \frac{1}{Q}\sum \limits _{i = 1}^Q {{p_i}} \end{aligned}$$
(29)
$$\begin{aligned}{} & {} V(p) = \frac{1}{Q}\sum \limits _{i = 1}^Q {{{({x_p} - E(p))}^2}} \end{aligned}$$
(30)
$$\begin{aligned}{} & {} \textrm{cov}(p,q) = \frac{1}{Q}\sum \limits _{i = 1}^Q {(({p_i} - E(p))({q_i} - E(q)))} \end{aligned}$$
(31)
$$\begin{aligned}{} & {} {{\textrm{corr}}_{xy}} = \frac{{cov(p,q)}}{{\sqrt{V(p)} \sqrt{V(q)} }} \end{aligned}$$
(32)

where (pq) is a set of adjacent pixels, V(p) and E(p) represent the variance and expectation of pixel p, respectively, and Q denotes the quantity of adjacent pixel pairs selected.

We select 10,000 sets of adjacent pixels randomly, and correlation coefficients in three various aspects are calculated. Figure 12 shows the analysis results of source images and their corresponding ciphertexts. And the source images have a significant correlation. However, the ciphertexts have a low correlation, close to 0. Moreover, Figs. 13 and 14 specifically display the correlation analysis results of grayscale and color images separately.

Table 4 Informatica entropy of ciphertext images
Fig. 12
figure 12

Analysis results of correlation coefficients

Fig. 13
figure 13

Analysis of correlation for Lena. ac Plaintext; df ciphertext

Fig. 14
figure 14

Analysis of correlation for House. ac Red, green and blue channels of the source image; df Red, Green and Blue channels of the ciphertext. (Color figure online)

In addition, Table 5 illustrates the correlation analysis comparison results between our encryption scheme and other schemes, where bold represents the optimal value. And the correlation coefficients of ciphertext are significantly smaller than original images, which signifies that our encryption method exhibits a strong capability to prevent statistical attacks.

5.2.6 Differential attack resistance analysis

Resistance to differential attack is also a crucial metric for evaluating the effectiveness of encryption schemes. Differential attack, also known as plaintext sensitivity analysis, refers to the attacker changing specific pixels of the source image, analyzing discrepancies in the ciphertext, then using obtained data to crack ciphertext. Therefore, a secure image encryption scheme must be sensitive to the plaintext information, even if there is a tiny difference in the plaintext, the encryption result should be significantly different from that before, so as to resist differential attack. And the number of pixels change rate (NPCR) and unified averaged changed intensity (UACI) are performed to represent the capability of encryption schemes to oppose differential attack, which can be obtained by Eqs. (33) and (34).

$$\begin{aligned} \mathrm{{NPCR}}= & {} \sum \limits _{m = 1}^M {\sum \limits _{n = 1}^N {\frac{{D(m,n)}}{{M \times N}} \times 100\% } } \end{aligned}$$
(33)
$$\begin{aligned} \mathrm{{UACI}}= & {} \sum \limits _{m = 1}^M {\sum \limits _{n = 1}^N {\frac{{\left| {{E_1}(m,n) - {E_2}(m,n)} \right| }}{{M \times N \times 255}}} } \times 100\% \nonumber \\ \end{aligned}$$
(34)

where \(E_1\) and \(E_2\) are two encrypted images sized \(M \times N\), and their source images are only a difference of one pixel. D(ij) is the quantity of pixel values that differ between ciphertexts \(E_1\) and \(E_2\).

$$\begin{aligned} D(i,j) = \left\{ {\begin{array}{*{20}{l}} {1 \quad \textrm{for} \quad {E_1}(i,j) \ne {E_2}(i,j)}\\ {0 \quad \textrm{otherwise}} \end{array}} \right. \end{aligned}$$
(35)
Table 5 Correlation analysis results under different encryption schemes

Set the default significance level \(\alpha = 0.05\), and Table 6 reveals the NPCR and UACI theoretical values of multiple image sizes [46].

Table 6 Theoretical values of NPCR (%) and UACI (%) with various image sizes
Table 7 Results of resistance to differential attack
Fig. 15
figure 15

Analysis of robustness against cropping attack. a 1/8 trimming; b 1/4 trimming; c 1/2 trimming; df decrypted images of ac

Table 7 shows the differential attack analysis results of several test images encrypted with our proposed encryption method and other encryption schemes, respectively. The conclusion can be drawn that the resistance capability of the designed encryption algorithm against differential attacks is strong.

5.2.7 Analysis of robustness

The secure image encryption schemes are considered to have strong robustness, especially for data loss and noise that may occur during network transmission of images. The effect of varieties in ciphertext images on the decryption results should be minimized. For the purpose of assessing the robustness of the designed encryption solution, we analyze the capacity of encryption algorithms to defend against cropping and noise attacks, and the peak signal-to-noise ratio (PSNR) is performed to express the robustness strength, which can be obtained by Eq. (37).

$$\begin{aligned} \mathrm{{MSE}}= & {} \sum \limits _{m = 1}^M {\sum \limits _{n = 1}^N {\frac{{{{\left\| {P(m,n) - D(m,n)} \right\| }^2}}}{{M \times N}}} } \end{aligned}$$
(36)
$$\begin{aligned} \mathrm{{PSNR}}= & {} 10 \times \textrm{log}_{10}\left( \frac{{255 \times 255}}{\mathrm{{MSE}}}\right) \end{aligned}$$
(37)

where P represents the source image sized \(M \times N\) and D represents the decrypted result, and MSE is the mean square error of images P and D.

Figure 15 shows the ciphertext images and decryption results of Lena (\(256 \times 256\)) under different clipping intensities. And the PSNR results under different clipping attack degrees is shown in Table 8, where bold represents the optimal value. All the results demonstrate that our encryption method possesses good resistance to cropping attacks.

Table 8 PSNR under cropping attack

Moreover, for the purpose of appraising the capability of this designed encryption method to withstand noise attacks, we apply various intensities of noise to the ciphertext Lena. And Fig. 16 manifests the decryption images of encrypted images with noise. Furthermore, Table 9 explicates the PSNR of deciphered images under different noise intensities and compares it with some existing encryption schemes. From the results obtained through the analyses, our image encryption algorithm also exhibits strong resistance to noise attacks.

Fig. 16
figure 16

Analysis of robustness against salt and pepper noise attacks. ac Decrypted images under different noise level

Table 9 PSNR under noise attack

The analyses presented above illustrate that our encryption method is capable of effectively defending against cropping attacks and noise attacks and possesses strong robustness.

6 Conclusion

In this work, we construct a novel 5D Hamiltonian conservative hyperchaotic system based on Euler rigid body equation and energy analysis. Compared with other chaotic systems, the 5D conservative hyperchaotic system has more complicated dynamic behavior and chaotic sequences generated by it have stronger pseudo-randomness. Depending on this novel hyperchaotic system and the new designed bit-plane segmentation method, we construct a secure image encryption method. Simulation experiments and performance analysis results illustrate that our encryption scheme possesses strong key sensitivity, wide key space, high plaintext sensitivity and robustness. Moreover, the ciphertext image has low correlation and uniform histogram distribution. All analyses indicate that our designed image encryption algorithm effectively ensures security and meets the needs of practical applications. In future, we intend to deeply study the secure transmission of encryption algorithm keys, so as to further protect the security of image information.