1 Introduction

In recent years, the optical image encryption has received more and more attentions. Refrainer and Javidi proposed the double-random-phase encoding (DRPE) method, involving two random phases mask keys in the input and Fourier domains in 1995 [1,2,3]. The optical implementation of this encryption scheme makes it widely applicable in the image encryption due to parallel processing and high speed. Unfortunately, the DRPE-based technique was found vulnerable to resist several kinds of attacks [4]. Subsequently, a novel multiple images encryption methods in the fractional Fourier transform (FRFT) [5] were introduced, which increases key numbers. However, the key space of fractional order is not big enough. Thus, fractional wavelet transform (FWT) [6, 7] was proposed to replace the FRFT in DRPE, which achieves double encryption in the fractional domain and wavelet domain. The fractional order and scale factors of FWT can be selected as the additional keys to expand the key space and improve the security of the cryptosystem.

In order to transmit data in the channel effectively, it is far more important to reduce data of cryptosystem. Compressed sensing (CS) [8] is a fast emerging field in information security, which unifies compression and encryption in a simple linear measurement step. Therefore, some CS-based image compression–encryption algorithms (CEA) were proposed to improve the efficiency and security of the cryptosystem [9, 10]. The digital image encryption methods based on CS and DRPE technique were proposed to keep information secret more effectively [11]. To enhance security further, the image encryption scheme combined CS with Arnold transform was proposed [12]. However, in most CS-based CEA, regardless of whether the measurement matrix is a key, this whole matrix needs to be transmitted. In some image encryption combined CS with DRPE [13, 14] or its derivative algorithms such as FRFT [15] and fractional random transform [16], low-dimensional chaos was adopted to reduce their transmission bandwidth. Although one-dimensional (1D) chaotic map was employed to resist some common attacks in CEA [17], 1D chaotic algorithm has its limitations. The cycle of 1D chaotic sequence may degenerate because of the computers’ finite precision, and its key space is relatively small, so the cipher-text has been widely analyzed or even been deciphered [18]. Thus, some CS-based cryptosystems using high-dimensional (HD) chaos like Lorenz system, hyper-chaos, and chaotic standard map were proposed [19,20,21], which can achieve the large parametric space and high security. Although the encryption efficiency and security have been improved in the previous approaches, there are still some space to improve in the robustness and security because of the growing demands for informational security.

In this paper, a robust image CEA based on two-step phase-shifting digital holography and FWT is proposed. Firstly, the cryptosystem has a good flexibility of optical realization and software implementation, which greatly reduces the complexity of achieving the encryption system. Besides, three pseudo-random sequences, which are generated by the 3D chaos, are used to be the measurement matrix of CS and two random-phase masks (RPM) in the FWT. It not only enhances the algorithms nonlinearity, but also enlarges the key numbers and space greatly. Lastly, in order to enhance the robustness of resisting noises and occlusion attacks, digital holograms are obtained by two-step-only quadrature phase-shifting interference (PSI). Numerical experiments have verified the security and validity of the proposed algorithm.

2 Fundamental knowledge

Here some basic knowledge related to our proposed scheme is introduced.

2.1 Fractional wavelet transform

Mendlovic and Zalevsky [22] defined the fractional wavelet transform (FWT), and the two-dimensional (2D) FWT [23] of 2D signal f (x, y) can be expressed as

$$W(a,\bar {b})=\iint {\iint {{U_{{p_1}}}(x,x^{\prime}){U_{{p_2}}}(y,y^{\prime}){H^{^{ * }}}(x^{\prime},y^{\prime})f(x,y){\text{d}}x{\text{d}}y{\text{d}}x^{\prime}{\text{d}}y^{\prime}}},$$
(1)

where p 1 and p 2 are the fractional orders. The f (x, y) is the input function. The U p1 (x, x ʹ) and U p2 (y, y ʹ) are the kernel functions, the “*” means the complex conjugate, and H *(x ʹ, y ʹ) is the wavelet function of mother wavelet function. And FWT in the fractional domains it can be expressed as

$$\begin{aligned} W({a_{mn}},\bar {b}) & ={\text{FW}}{{\text{T}}_{{\alpha _{1x}},{\alpha _{1y}};{\alpha _{2x}},{\alpha _{2y}}}}\{ f(x,y)\} \\ & ={({a_m}{a_n})^{{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt}\!\lower0.7ex\hbox{$2$}}}}\iint {{H^*}({a_m}u,{a_n}v)} \times \exp (j2\pi u{b_{x^{\prime}}},j2\pi v{b_{y^{\prime}}}) \\ & \quad \times \;{\text{FrF}}{{\text{T}}_{{\alpha _{1x}},{\alpha _{1y}}}}\{ {\text{FrF}}{{\text{T}}_{{\alpha _{2x}},{\alpha _{2y}}}}[f(x,y)](x^{\prime},y^{\prime})\} (u,v){\text{d}}u{\text{d}}v, \\ \end{aligned}$$
(2)

where a mn  = (a m , a n ) is the discrete scaling vector, b = (b x ʹ, b y ʹ) is the shift vector, and a 1x , a 1y, a 2x , a 2y is fractional order. It is an asymmetric system with fractional order on the x and y directions. The FrFT is expressed as the fractional Fourier transform.

However, its back-reconstructing formula in the fractional wavelet domains is

$$\begin{aligned} f(x,y) & ={\text{IFW}}{{\text{T}}_{^{{( - {\alpha _{1x}}, - {\alpha _{1y}}; - {\alpha _{2x}}, - {\alpha _{2y}})}}}}\{ W({a_{mn}},\bar {b})\} \\ & =\frac{1}{C}\iint {{\text{FFT}}\Bigg\{ {\sum\limits_{m} {\sum\limits_{n} {\iint {\frac{1}{{{a_m}{a_n}}}}} } W(a,\bar {b}) \times H({a_m}u,{a_n}v)\exp ( - j2\pi \mu {b_{x^{\prime}}}, - 2j\pi \upsilon {b_{y^{\prime}}})} } \\ & \quad \times {\text{d}}b{x^\prime }{\text{d}}b{y^\prime }\Bigg \} \times ({x^\prime },{y^\prime }){U_{ - {\alpha _{1x}}, - {\alpha _{1y}}; - {\alpha _{2x}}, - {\alpha _{2y}}}}(x,y;{x^\prime },{y^\prime }){\text{d}}{x^\prime }{\text{d}}{y^\prime }, \\ \end{aligned}$$
(3)

where IFWT is the inverse fractional wavelet transform, FFT{} denotes the Fourier transform, and C is a constant.

3 The process of encryption and decryption

Duarte et al. proposed a single-pixel imaging via compressed sampling and achieved optical implementation of CS in recent years [24]. Thus, the proposed encryption system is flexibility, which can be either optical realization or software implementation. The possible optical realization of image encryption scheme is shown in Fig. 1. The multi-reference matched filter (MRMF) is wavelet matched filter, and RPM1 and RPM2 are two random-phase masks (RPM), which are generated by three-dimensional chaotic systems. First of all, the laser is divided into two beams of light by the beam splitter (BS): a beam of light is used as a reference light, and the other is passed through an object as the object wave. The reference wave is controlled to generate different phase reference waves by a PZT by an optical phase shifter. And the corresponding combined object wave is reflected off by a digital micro-mirror device (DMD) which consists of an array of tiny mirrors. Each mirror in DMD independently oriented either towards the optical path of the encryption by FWT or away from it, and the optical implementation devices of FWT consist of three lenses, a pair of RPM, and a MRMF closing to the RPM2. Thus, after encoded by the encrypted optical path of the fractional wavelet, the reflected light is interfered with the reference light through BS and then collected through the L1 and focused onto a single-pixel detector (SPD). Finally, the random samples can be recorded. And two holograms are obtained and saved in PC. The optical image decryption is the inverse applications of the optical encryption. In this process, the wavelet matched filter, two pieces of RPM, and fractional order can be used as a key, so the key space is enlarged greatly. Thus, the proposed encryption system can make the intruder attacks futile and reduce the bandwidth needed for data transmission.

Fig. 1
figure 1

The optical implementation of the proposed encryption method: L1, L2, L3, L4 lens, BS beam splitter, O original image, DMD digital micro-mirror device, RPM1, RPM2 random-phase masks, MRMF multi-reference matched filter, PZT piezo-electric transducer

Our method is based on CS and phase-shifting digital holography in FWT domain. The process of encryption and decryption are symmetrical as shown in Fig. 2. And main operations of encryption processes are introduced as follows.

Fig. 2
figure 2

The process of encryption and decryption

Firstly, we construct three chaotic matrixes by 3D chaotic systems in PC. As chaotic system has good pseudo-randomness, in the proposed scheme, 3D chaotic system, namely Chen’s chaotic system [25], is employed in key generation, which is described by

$$\begin{aligned} x^{\prime}= & {a_1}(y - x), \\ y^{\prime}= & ({c_1} - {a_1})x - xz+{c_1}y, \\ z^{\prime}= & xy - {b_1}z, \\ \end{aligned}$$
(4)

where a 1, b 1, and c 1 are parameters. When a 1 = 35, b 1 = 3, and c 1 ∈ [20, 28.4], the system is chaotic. Thus, it can make our proposed cryptosystem nonlinear. The initial value x 0, y 0, z 0 and the controls parameter a 1, b 1, c 1 are acted as the secret keys.

We take the decimal part of all chaotic time series to limit its elements to the interval [0, 1] using the following operation:

$$\left\{ \begin{gathered} x^{\prime\prime}={\text{abs}}(x^{\prime}) - {\text{floor}}\left[ {{\text{abs}}(x^{\prime})} \right] \hfill \\ y^{\prime\prime}={\text{abs}}(y^{\prime}) - {\text{floor}}\left[ {{\text{abs}}(y^{\prime})} \right] \hfill \\ z^{\prime\prime}={\text{abs}}(z^{\prime}) - {\text{floor}}\left[ {{\text{abs}}(z^{\prime})} \right] \hfill \\ \end{gathered} \right.,$$
(5)

where the operation abs is used to obtain the absolute value, floor is used to obtain an integer outcome, x″, y″, and z″ is three random sequences, the sequence z″ is to generate the measurement matrix Φ of CS, and others sequences x″ and y″ are generated by two complex RPM in FWT.

Then the original image is compacted by DMD, and it mainly has two steps in the process:

Step 1::

An original image α needs to be transformed to its sparse representation by wavelet basis ψ, and thus the N × N sparse image X is formed by

$$X={\psi ^T}\alpha .$$
(6)
Step 2::

Use the chaotic matrix Φ to measure the sparse image; the measurement matrix is M × N (M < < N), and the compressed image Y of M × N matrix is calculated by

$$Y=\Phi X.$$
(7)

To recover the sparse signal X from the measurements Φ although M < < N correctly, Φ should satisfy the restricted isometry property (RIP) [26]. Here the definition of RIP in k order is as follows:

$$(1 - {\delta _k})\left\| X \right\|_{2}^{2} \leqslant \left\| {{\Phi _X}} \right\|_{2}^{2} \leqslant (1+{\delta _k})\left\| X \right\|_{2}^{2},$$
(8)

where δ k ∈ [0, 1] is an isometry constant.

Next all reflected (or transmissive) light from DMD is through optical encoding device to achieve the FWT based on DPRE; here φ 1 and φ 2 are loaded to the RPM, which are generated by chaotic system. In proposed method, the DPRE in FWT domains can be implemented in Fig. 1, and a multi-reference matched filter (MRMF) is placed in the Fourier plane of the wavelet transform. After a one-level DWT with selected wavelet family, the information is decomposed into four parts.

The complex-amplitude distribution F after FWT can be represented by

$$\begin{aligned} F= & {\text{FW}}{{\text{T}}_{{\alpha _{1x}},{\alpha _{1y}};{\alpha _{2x}},{\alpha _{2y}}}}\{ Y\} ={\text{MRMF}}(u,v)\{ {\text{FrT}}{{\text{T}}_{{\alpha _{2x}},{\alpha _{2y}}}}[{\text{FrF}}{{\text{T}}_{{\alpha _{1x}},{\alpha _{1y}}}}(Y)\exp (i{\varphi _1})]\exp (i{\varphi _2}){\text{d}}u{\text{d}}v\} \\ = & {({a_m}{a_n})^{{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt}\!\lower0.7ex\hbox{$2$}}}}\iint {{H^*}({a_m}u,{a_n}v)} \times \exp (j2\pi {b_{x'}},j2\pi v{b_{y'}}) \times {\text{FrF}}{{\text{T}}_{{\alpha _{1x}},{\alpha _{1y}}}}\{ {\text{FrF}}{{\text{T}}_{{\alpha _{2x}},{\alpha _{2y}}}}[Y]\} (u,v){\text{d}}u{\text{d}}v, \\ \end{aligned}$$
(9)

where H *(x ʹ, y ʹ) is the wavelet function of mother wavelet function. a mn = (a m , a n ) is the discrete scaling vector, b = (b , b ) is the shift vector, and a 1x , a 1y , a 2x , a 2y are fractional orders; they is an asymmetric system with fractional order on the x and y directions. FrFT represents the fractional Fourier transform, and MRMF (u, v) can be calculated by

$$MRMF(u,v)=\sum\limits_{m} {\sum\limits_{n} {\left\{ {{H^*}[{a_{mn}}(u - n{u_0},v - n{v_0})]} \right\},} }$$
(10)

where the parameters a m , a n are a series of scale factor, and H * is a certain mother wavelet function.

Finally, two-step-only quadrature PSI holography is used to record the encoding image in Fig. 1. Two phases are 0 and π/2 degree, and were introduced as two reference lights by PZT; the complex amplitudes of two holograms are I 0 and I π/2, respectively, which can be expressed as

$${I_0}={\left| {F+R} \right|^2}={F^2}+{\left| R \right|^2}+{R^ * }F+R{F^ * },$$
(11)
$${I_{\pi /2}}={\left| {F+\operatorname{R} \exp (j\pi /2)} \right|^2}={\left| F \right|^2}+{\left| R \right|^2} - j{R^ * }F+jR{F^ * },$$
(12)

where F and R denote the amplitude of the object wave and reference wave, respectively. |F|2 + |R|2 is the zero-order light wave.

Decryption is the inverse encryption process in Fig. 2. I h is generated with no zero-order term and conjugate term by

$${I_h}=({I_0} - {\left| R \right|^2} - {\left| F \right|^2})+j({I_{\pi /2}} - {\left| R \right|^2} - {\left| F \right|^2})=2{R^ * }F.$$
(13)

The amplitude of the object wave can be calculated as follows:

$$F^{\prime}={I_h}/2{R^*}.$$
(14)

Because there is no record of the intensity of object wave, |F|2 + |R|2 should be calculated from two orthogonal phase-shift holograms. When the value of R reaches a certain value 2R ≥ max (|F|) + min (|F|), which can be determined according to Eq. (11) and Eq. (12), |F|2 + |R|2 can be calculated by

$${\left| F \right|^2}+{\left| R \right|^2}={{(2{R^2}+{I_0}+{I_{\pi /2}})} \mathord{\left/ {\vphantom {{(2{R^2}+{I_0}+{I_{\pi /2}})} 2}} \right. \kern-0pt} 2} - {{\left[ {\sqrt {{{(2{R^2}+{I_0}+{I_{\pi /2}})}^2} - 2(I_{0}^{2}+I_{{_{{\pi /2}}}}^{2}+4{R^4})} } \right]} \mathord{\left/ {\vphantom {{\left[ {\sqrt {{{(2{R^2}+{I_0}+{I_{\pi /2}})}^2} - 2(I_{0}^{2}+I_{{_{{\pi /2}}}}^{2}+4{R^4})} } \right]} 2}} \right. \kern-0pt} 2}.$$
(15)

The light field of encryption image is obtained in Eq. (14). The inverse FWT algorithm in Eq. (16) is applied in a 4f optical inverse processor, which requires the complex conjugate Fourier phase (φ 1 * and φ 2 *) to restore the encryption image:

$$\begin{aligned} Y & ={\text{IFW}}{{\text{T}}_{^{{( - {\alpha _{2x}}, - {\alpha _{2y}}; - {\alpha _{1x}}, - {\alpha _{1y}})}}}}\{ F^{\prime}\} ={F_{^{{ - {\alpha _{1x}}, - {\alpha _{1y}}}}}}\left\{ {{F_{ - {\alpha _{2x}}, - {\alpha _{2y}}}}[{\text{MRMF}}(F^{\prime})\exp (i{\varphi _2}^{*})]\exp (i{\varphi _1}^{*})} \right\}{\text{d}}x{\text{d}}y \\ & =\frac{1}{C}\iint {{\text{FFT}}\left\{ {\sum\limits_{m} {\sum\limits_{n} {\iint {\frac{1}{{{a_m}{a_n}}}}} } F^{\prime}(a,\bar {b}) \times H({a_m}u,{a_n}v)\exp ( - j2\pi \mu {b_{x^{\prime}}}, - 2j\pi \upsilon {b_{y^{\prime}}}) \times {\text{d}}bx^{\prime}{\text{d}}by^{\prime}} \right\}} \\ & \quad \times (x^{\prime},y^{\prime}){U_{ - {\alpha _{1x}}, - {\alpha _{1y}}; - {\alpha _{2x}}, - {\alpha _{2y}}}}(x,y;x^{\prime},y^{\prime}){\text{d}}x^{\prime}{\text{d}}y^{\prime}, \\ \end{aligned}$$
(16)

where IFWT is the inverse fractional wavelet transform, FFT{} denotes the Fourier transform, and C is a constant.

Thus, the measurement image Φ is obtained by 3D chaotic systems. To improve the quality of decrypted image, the BP reconstruction algorithm is adopted to recover the wavelet coefficient matrix. The original image can be reconstructed through the inverse wavelet transform. The reconstructed image α ʹ is denoted as follows:

$$\alpha ^{\prime}=C{S^{ - 1}}(Y^{\prime}).$$
(17)

Here, CS−1 is reconstructed process of CS, namely basis pursuit (BP) algorithm [26].

4 Simulation experiments and analysis

To check the performance of our proposed scheme, several numerical simulations have been performed. The gray image “Lena” is shown in Fig. 3a. It has 256 × 256 pixels size and acts as the plain image in the experiment. In the proposed encryption system, the important parameters are (1) M = 192, N = 256, and a sampling rate of 75%, and the formula is expressed by

$${S_{\text{f}}}=\frac{1}{r}=\frac{{M \times N}}{{N \times N}},$$
(18)

where S f is the sampling rate, and it is the reciprocal of the comparison ratio r. The size of original image is N × N, and the size of compressed image is M × N; (2) x 0 = 0.258, y 0 = 0.368, z 0 = 0.568, and a 1 = 35, b 1 = 3, c 1 = 28 from the 3D chaos; (3) fractional orders a 1 = (a 1x , a 1y ) = (1.5, 1.2), a 2 = (a 2x , a 2y ) = (1.4, 1.6), and scale factor s = 1, b = “haar” mother wavelet from fractional wavelet transform. One of two encrypted holograms and decrypted image are displayed in Fig. 3b, c, respectively.

Fig. 3
figure 3

Image encryption and decryption: a the original image “Lena”; b one of two encrypted holograms (I 0); c the decrypted image

How pixels in an image are distributed by plotting the number of pixels at grayscale is illustrated with an image histogram. Specifically, it should hide the redundancy of plain-text and not leak any information about the plain-text or the relationship between plain-text and cipher-text.

The histograms of “Lena” and “House” are shown in Fig. 4e, f. Their ciphered images produced by the proposed scheme, and their histograms of ciphered images are shown in Fig. 4g, h, respectively. It is clear that the histograms of cipher images are significantly different from those of the plain images. Hence, it does not provide any clue to employ statistical attack.

Fig. 4
figure 4

The original image: a “Lena,” b “House”; the encrypted hologram: c “Lena,” d “House”; histogram of e “Lena,” f “House”; histogram of g encrypted hologram of “Lena,” h encrypted hologram of “House”

The correlation coefficient of adjacent pixels is an important statistical feature of images. The correlation between two adjacent pixels in an original image is usually close to 1 in Fig. 5a, b. And an efficient image cryptosystem should produce the encrypted image with low correlation sufficiently. The correlation coefficients for adjacent pixels of their corresponding encrypted images are given in Fig. 5c, d, from which it is seen that the proposed scheme generally provides a satisfactory correlation performance.

Fig. 5
figure 5

The autocorrelation coefficient distribution of adjacent pixels in original image of a “Lena,” b “House”; encrypted hologram of c “Lena,” d “House”

In our proposed scheme, there are 12 keys in total, namely six initial values x 0, y 0, z 0 and a 1, b 1, c 1 from the 3D chaos, four fractional orders a 1x , a 1y , a 2x , a 2y , the wavelet base type b, and the wavelet decomposition progression s in fractional wavelet domain.

We give the mean square error (MSE) curves in relation to increments Δ i and d i , which are added to x 0, y 0, z 0, a 1, b 1, c 1 and a 1x , a 1y , a 2x , a 2y , s, respectively. The MSE changes versus each deviation are drawn in Fig. 6a–k, and it is clear that the proposed system is sensitive to keys, which means a tiny change of keys can result in a great visual distortion in recovered images. Usually, the MSE and peak signal-to-noise ratio (PSNR) is used to evaluate the quality of recovered images. Their formula is described as follows:

$${\text{MSE}}={\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 {MN}}}\right.\kern-0pt}\!\lower0.7ex\hbox{${MN}$}}\sum\limits_{{i=1}}^{M} {\sum\limits_{{j=1}}^{N} {{{\left( {Y\left( {i,j} \right) - X\left( {i,j} \right)} \right)}^2}} } ,$$
(19)
$${\text{PSNR=}}10\log \left( {{\raise0.7ex\hbox{${{{255}^2}}$} \!\mathord{\left/ {\vphantom {{{{255}^2}} {{\text{MSE}}}}}\right.\kern-0pt}\!\lower0.7ex\hbox{${{\text{MSE}}}$}}} \right),$$
(20)

where Y (i, j) and X (i, j) denote pixel values of recovered image and original image at location (i, j), respectively, and M, N denote the height and width of these images, respectively.

Fig. 6
figure 6

MSE curves for a x 0 + Δ1, b y 0 + Δ2, c z 0 + Δ3, d a 1 + Δ4, e b 1 + Δ5, f c 1 + Δ6, g a 1x  + d 1, h a 1y  + d 2, i a 2x  + d 3, j a 2y  + d 4, k s + d 5

To provide high security of the encryption system, the key space should be large enough to make any brute-force attack ineffective. In our approach, the number of keys and their spaces are both expanded greatly. The total key space includes two processes of confusion and diffusion.

Since x 0, y 0, z 0, and a 1, b 1, c 1 are from the 3D chaos, their key spaces could be computed by the help of MSE curve, when Δ1, Δ2, Δ 3 , Δ4, Δ5, or Δ6 in initial values x 0 + Δ1, y 0 + Δ2, z 0 + Δ3 and a 1 + Δ4, b 1 + Δ5, c 1 + Δ6 drops down to 1 × 10−16. The effective ranges of Δ i (i = 1, 2, 3) are all real number, and according to the IEEE floating-point standard [27], their computational precision of 64-bit double-precision number is about more than 264, while the scope of Δ i (i = 4, 5, 6) is only a few set of values. So we can calculate that the key space from x 0, y 0, z 0 and a 1, b 1, c 1 is about 1016×3 × 1014×3 × 264 × 3 = 5.534 × 10109.

As for the fractional orders a 1x , a 1y , a 2x , and a 2y , their key space could be analyzed as follows. In Fig. 6g–i, we know the maximum value of Δd i (i = 1, 2, 3, 4, 5, 6) (the distances between the two cross-points on the red lines) is about 0.02 when the MSE is equal to 5 × 103. Since the effective range of d i is [–2], the number of possible intervals of d i is 4/0.02 = 200. So the key space from four fractional orders is 2004 = 4 × 108. Besides, the wavelet base type b and the wavelet decomposition progression s for fractional wavelet transform, the key space is more than 264. Thus, the key space S of the entire encryption system is

$$S=\prod\limits_{{i=1}}^{{12}} {{K_i}}>4 \times 5.534 \times {10^{^{{109}}}} \times {10^{^{8}}} \times {2^{^{{64}}}}=4.08 \times {10^{^{{137}}}},$$
(21)

which indicates heavy work for the opponents to decrypt the image by exhausting the keys.

To evaluate the performance of the encryption system, two other methods are used to compare with our proposed method. Thus, the encryption method [15] based on the CS and FRFT by multiple 1D chaotic map is called CS-FRFT. And the encryption method based on the CS and FWT by 3D chaotic map is called CS-FWT, which is a part of the proposed method without adopting holographic technology. In addition, all recovered images are rebuilt by the basis pursuit (BP) algorithm in three encryption methods. In Table 1, it is shown that the key space of our proposed method is greater than that of other methods, and thus it has a stronger ability to resist brute-force attacks for the proposed methods without reducing the quality of decrypted image.

Table 1 The key space comparison of three methods in the 75% sampling rate

The robustness of our proposed encryption system was checked against the attack using the noise and the occlusion. In the image transmission and processing, it is inevitable that the cipher-text will be affected by noise. Therefore, we need to test the ability of resisting noise attack, and different intensity random noises adjusted by parameter k are added into the cipher-text I as follows:

$$I^{\prime}=I\left( {1+kG} \right),$$
(22)

where I ʹ is the noise-affected encrypted amplitude images, k is a coefficient that represents the noise strength, and G is a noise type, which is random noise. Accordingly, their restored images of the proposed methods, CS-FWT and CS-FRFT, are illuminated in Table 2, from which it can be shown that the restored images of the proposed method have a higher image quality under the same intensity of noise. Therefore, compared with two other methods, the proposed method has a better robustness of anti-noise attacks.

Table 2 The anti-noise performance comparison of three methods in the 75% sampling rate

As a digital evaluation to measure the performance of resisting occlusion attack, peak signal-to-noise ratio (PSNR) is mathematically used to evaluate the quality of image. Compression has much effect on the reconstruction of original image. In generally, it cannot effectively resist occlusion attack well. In three encryption methods of the proposed, CS-FWT and CS-FRFT, the basic information of recovered images can all be identified in Table 3, and their PSNR values of the proposed methods are higher than those of two others, indicating that the proposed encryption system has a better robustness against occlusion attack with the aid of the holography.

Table 3 The comparison of performance against cropping of three methods in the 75% sampling rate

An image encryption system can adopt the CS technique to save and process the compressed versions of encrypted images in order to decrease volume in transmission and storage. To evaluate the compression performance of the proposed encryption system, “Lena” was chosen as the original image, and Fig. 7 is the decrypted image PSNR values in different sampling rates through the different reconstruction algorithms of CS. In basis pursuit (BP) [26] reconstruction algorithms of CS, when the sampling rate is about 40%, the PSNR value is 25.04 dB. At this time, we can see subjectively that the decrypted image is basically not distorted, and all the information of the original image is almost recognized, while the information is seriously lost when the PSNR value is 21.54 dB at a sampling rate of 30%. Besides, in orthogonal matching pursuit (OMP) [28] reconstruction algorithms of CS, the change curve can also be clearly seen. And when the PSNR of recovered image exceeds 25 dB (blue green line in Fig. 7), the recovered image has a good quality. Therefore, it can be seen that the quality of recovered image is acceptable when the sampling rate is lower than or equal to 50%. Even when the sampling rate reaches 40%, the PSNR is still not bad.

Fig. 7
figure 7

The decrypted image PSNR values of different sampling rates in the different reconstruction algorithms of CS

5 Conclusions

In this paper, an efficient and secure image cryptosystem, based on CS and PSI in asymmetric fractional wavelet domain by 3D Chen’s chaos map, is proposed. It not only can overcome problems of the small key space and low security in resisting nonlinear attacks, but also decreases a large number of encrypted data greatly. In order to simplify the transmission and storage, the keys generated by 3D chaos map replace the enormous RPM and measurement matrix. With the help of digital holography, the proposed method has a good robustness against occlusion and noise attacks. And the higher security of our cryptosystem is demonstrated by evaluating key space analysis. In addition, for efficiency analysis, the compression performance analysis shows that the PSNR is high enough to recognize the information of original image when the sampling rate is lower than or equal to 50%, and it is acceptable even at 40%. Thus, the proposed encryption system may be a reference for data security transmission and effective storage in encryption.