1 Introduction

Nowadays, image encryption is a prime concern to protect an image from being misused by unauthorized users. Images are used to communicate important information from one end to the other through various types of networks like WhatsApp, Facebook, and YouTube, etc. In addition, transmission of images is often used in diverse areas such as defence services, engineering services, scientific experiments, medical imaging for diagnosis of diseases, advertising, art exhibition, online education, telecommunication, and home automation. Adversaries are eagerly waiting to intercept or steal confidential information from the images that are shared or stored. Thus, we require a new, fast, and secure image encryption method, which offers good security and is immune to common attacks.

Over the last few years, chaos and hyper-chaos based image encryption methods were frequently used [3, 6,7,8,9,10,11,12, 15, 24, 33,34,35, 35, 37, 40, 41, 43,44,45] on account of their high sensitivity to the parameters, and pseudo-randomness, complex dynamical behaviors. They have at least two positive Lyapunov exponents [40, 44]. They provide a great deal of support to the cryptosystems in view of their uncertainty and large size of the keyspace. In [24], a new complex chaotic map was developed over a complex field and used for image encryption. Recently, 5D hyper-chaotic system has also been proposed to encrypt grayscale image. The 5D hyper-chaotic is employed to generate hyper-chaotic sequences in a more complex way [44]. In [45] a new image encryption algorithm based on a linear hyperbolic chaotic system of partial differential equations has been presented. Further, encryption schemes using DNA encoding with elliptic curve cryptography has been proposed in [15]. Image processing with vector quantization is another different method to encode images by decomposing pixels into vectors [8]. In [11], the image size of 512 × 512 × 3 × 8 has been reshaped into 128 × 128 × 128 and 3D image permutation algorithm has been applied to relocate the pixels in image. Pixel transposition between RGB channels using knight’s moving rules and a digital chaotic map is introduced. In contrast, chaos and hyper-chaos based image encryption methods can be broken easily [1, 2, 17, 19,20,21, 27, 28, 30, 32]. Further, the authors in [18], explained that the proposed scheme [7] has security flaws. Likewise, in reference [22], security issues have been studied and confirmed that the method proposed in [35] can be broken effortlessly. Furthermore, Eli Bhiam and Adi Shamir, [4, 5] have discussed differential attacks to various ciphers. Since then it becomes an important attack to analyze during cipher design and it is done through the statistical tests such as The Number of Changing Pixel Rate (NPCR) and The Unified Averaged Changed Intensity (UACI). Wu et al. [38] have addressed the problem related to chaos-based image encryption for differential attacks both numerically and symbolically. For instance, in [13], the image of size 256 × 256; NPCR value is 99.42% and UACI value is 27.78%. These numerical values confirm that the statistical tests are successful, but hypothesis tests for NPCR and UACI were failed for all confidence intervals.

To overcome these security flaws on chaos and hyper-chaos image encryption schemes, we have come up with a solution to accelerate speed and security (resist various types of attacks) by means of a new technique. In this paper, we proposed a new encryption scheme using GHE associated with GVTSG. The proposed encryption scheme offers good security and resists various types of attacks. We have analyzed hypothesis tests for NPCR and UACI and found that they fulfill the demand for a strong image encryption scheme.

2 Organization of the paper

In Section 3, random key generation procedure is explained. In Section 4, we introduce the encoding/decoding scheme. In Section 5, a mechanism to generate the generalized Vigen\(\grave {e}\)re-type table has been elaborated and the complete table is provided. In Section 6, we provide our proposed encryption and decryption procedure. Section 7, contains the details of construction, performance, and security analysis. In Section 8, we provide the comparison Tables 1521, which contain the data that arise out from the comparison of our proposed algorithm with some of the existing competing algorithms. Finally, we conclude the paper in Section 9.

3 Random key generation procedure

The generation of random key sequence is based on a generalized heat equation. The paper provides another notion to use the GHE to generate a Pseudo Random Number Generator (PRNG) and corresponding coefficients present in the heat equation as initial parameters to generate keys. Both, the key generation and encryption are the innovative procedure in the way they differ from conventional key generation and encryption algorithms. Once the keys are generated they are used in Generalized Vigen\(\grave {e}\)re-type Cipher on Symmetric Group Sn (GVCSG). The key generation provides an idea to generate keys from GHE. In this section, we introduce the way to generate the random key sequence and keys for encryption in detail.

3.1 Generalized heat equation and random key sequence

The details of the generalized heat equation can be found in [26]. Equation (1) generates a solution space over x, t and each entry produces sequence of digits involves random entries. The complete process of random key sequence generation is as follows. The generalized heat equation and the corresponding Initial Condition (IC) and Boundary Conditions (BCs) are shown below with (1).

$$ \begin{array}{@{}rcl@{}} PDE: \hspace{0.3cm} \frac{\partial \varphi(x,t) }{\partial t} &=& ({\Delta} x^{\prime})^{2} \varphi(x,t), \hspace{0.3cm} 0<x<L, \\ IC: \hspace{0.3cm} \varphi(x,0) &=& a_{0}+a_{1}x +a_{2}x^{2} + {\dots} + a_{p}x^{p} \hspace{0.3cm} 0<x<L,\\ BC: \hspace{0.3cm} \varphi(0,t) &=& \varphi(L,t) = 0, \hspace{0.3cm} 0<t<T, \end{array} $$
(1)

where \(({\Delta } x^{\prime })^{2} = -\left (\frac {d}{dx} + ix\cot \theta \right )^{2}\). With the help of finite difference method, we can find the solution of the above system given in (1) as follows:

$$ \begin{array}{@{}rcl@{}} \varphi(x,t-{\Delta} t) = \varphi(x,t) \left[ 1+\frac{2{\Delta} t}{{\Delta} x^{2}}+x cot^{2} \theta {\Delta} t-2ixcot \theta\frac{{\Delta} x}{{\Delta} t} \right]\\ + \varphi(x-{\Delta} x,t)\left[ \frac{-{\Delta} t}{{\Delta} x^{2}}+2ixcot \theta\frac{{\Delta} t}{{\Delta} x} \right] + \varphi(x+{\Delta} x,t)\left[ \frac{-{\Delta} t}{{\Delta} x^{2}} \right]. \end{array} $$
(2)

The values of Δx and Δt are secret values that will be sent to the receiver via a secure channel. For simulation purpose, we have taken IC as below

$$ \begin{array}{@{}rcl@{}} \varphi(x,0) = 4x-8x^{2}+4x^{3} \hspace{0.5cm} \forall x \in [0, L]. \end{array} $$

By plugging-in different value of 𝜃 as π/4, π/3, π/2, and 5π/6 in (2), the corresponding absolute values of φ(x, t) are plotted in Fig. 1. Here L and T are lengths of the rod and time period, respectively. We partition the length of the rod into a finite number of intervals each is of length Δx. Similarly, we partition the time T into a finite number of sub-intervals each is of length Δt. Finally, we compute discrete points φ(x, t) by using (2). In order to make the absolute value of φ(x, t) an integer, we multiply absolute value of φ(x, t) by 10k, where k is a sufficiently large positive integer and remove the decimal part. Now, we convert the resulted integer part into binary form. This leads us to have a binary sequence of size Lx × Tt × b, where b represents the number of bits generated by each entry of φ(x, t). Then we convert this binary sequence into one-dimensional array say RK of the same size and select an index j at random in binary sequence and take a block S of size z from j onwards, which is to be XOR-ed as per the code provided to achieve RKS.

Fig. 1
figure 1

Different surface plots for absolute value of φ(x, t) (generalized heat solution) with different 𝜃 values a for 𝜃 = π/4, b for 𝜃 = π/3, c for 𝜃 = π/2, d for 𝜃 = 5π/6

The RK has been obtained mathematically from the solution surface (i.e. ϕ(x, t)) of the GHE (as shown in Fig. 1) by

$$ \phi(x,t) = \left[\begin{array}{lllll} \phi_{1,1} & \phi_{1,2} & \phi_{1,3} & {\dots} & \phi_{1,\frac{L}{{\Delta} x}} \\ \phi_{2,1} & \phi_{2,2} & \phi_{2,3} & {\dots} & \phi_{2,\frac{L}{{\Delta} x}} \\ {\vdots} & {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ \phi_{\frac{T}{{\Delta} t},1} & \phi_{\frac{T}{{\Delta} t},2} & \phi_{\frac{T}{{\Delta} t},3} & {\dots} & \phi_{\frac{T}{{\Delta} t},\frac{L}{{\Delta} x}} \end{array}\right], $$

multiply each element of this matrix by 10k where k is defined as before and round it, then ϕ(x, t) becomes a new matrix represented by Iϕ(x, t) and defined as follows

$$ I\phi(x,t) = \left[\begin{array}{lllll} I\phi_{1,1} & I\phi_{1,2} & I\phi_{1,3} & {\dots} & I\phi_{1,\frac{L}{{\Delta} x}} \\ I\phi_{2,1} & I\phi_{2,2} & I\phi_{2,3} & {\dots} & I\phi_{2,\frac{L}{{\Delta} x}} \\ {\vdots} & {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ I\phi_{\frac{T}{{\Delta} t},1} & I\phi_{\frac{T}{{\Delta} t},2} & I\phi_{\frac{T}{{\Delta} t},3} & {\dots} & I\phi_{\frac{T}{{\Delta} t},\frac{L}{{\Delta} x}} \end{array}\right], $$

convert all entries of this matrix into binary form as represented by BINϕ(x, t) as follows

$$ BIN\phi(x,t) = \left[\begin{array}{lllll} BIN\phi_{1,1} & BIN\phi_{1,2} & BIN\phi_{1,3} & {\dots} & BIN\phi_{1,\frac{L}{{\Delta} x}} \\ BIN\phi_{2,1} & BIN\phi_{2,2} & BIN\phi_{2,3} & {\dots} & BIN\phi_{2,\frac{L}{{\Delta} x}} \\ {\vdots} & {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ BIN\phi_{\frac{T}{{\Delta} t},1} & BIN\phi_{\frac{T}{{\Delta} t},2} & BIN\phi_{\frac{T}{{\Delta} t},3} & {\dots} & BIN\phi_{\frac{T}{{\Delta} t},\frac{L}{{\Delta} x}}, \end{array}\right], $$

convert BINϕ(x, t) into one dimensional row-matrix, which is called RK.

$$ RK {\kern1.7pt}={\kern1.7pt} \left[\begin{array}{lllllll} BIN\phi_{1,1} &\! BIN\phi_{1,2} & BIN\phi_{1,3} & {\dots} BIN\phi_{1,\frac{L}{{\Delta} x}} {\dots} BIN\phi_{\frac{T}{{\Delta} t},1} &\! BIN\phi_{\frac{T}{{\Delta} t},2} &\! BIN\phi_{\frac{T}{{\Delta} t},3} & {\dots} BIN\phi_{\frac{T}{{\Delta} t},\frac{L}{{\Delta} x}} \end{array}\right]. $$

The RK has been elaborated numerically for the solution surface of the GHE. For instance, if one can take some portion on the surface of the generalized heat solution in the form of matrix as below

$$ \phi(x,t) = \left[\begin{array}{lll} 0.724862 & 0.347629 & 0.223964 \\ 0.234621 & 0.489689 & 0.146320 \\ 0.383649 & 0.746321 & 0.923876 \end{array}\right], $$

and multiply by 105 and round it, we have

$$ I\phi(x,t) = \left[\begin{array}{lll} 72486 & 34762 & 22396 \\ 23462 & 48968 & 14632 \\ 38364 & 74632 & 92387 \end{array}\right], $$

and convert all entries into binary form as represented by BINϕ(x, t),

$$ BIN\phi(x,t) = \left[\begin{array}{lll} 10001101100100110 & 1000011111001010 & 101011101111100 \\ 101101110100110 & 1011111101001000 & 11100100101000 \\ 1001010111011100 & 10010001110001000 & 10110100011100011 \end{array}\right], $$

again convert BINϕ(x, t) into one dimensional array to get RK as follows:

$$ RK = \left[\begin{array}{ll} 10001101100100110 \ 1000011111001010 {\dots} 10010001110001000 \ 10110100011100011 \end{array}\right]. $$
figure e

RKS contains the data of binary numbers with size depending upon the number of iterations and the chosen parameters L, Δx and Δt. The randomness of RKS has been proved by —NIST statistical test suite [31] and the results are presented in the Figs. 1517. RKS is generated with the help of GHE having 4,25,00,000 bits. Figures 1517 are generated for 10, 25 and 100 blocks, where each block contains 4,00,000 bits. From these figures, we can find that all testes are passed successfully with good proportion and P-values.

3.2 Keys (K e y i, i = 1,2,3,…7) for encryption generated from RKS

To obtain keys for encryption/decryption, we take a sequence of binary blocks from RKS, where a block is of size t, and t varies from 2 to (Lx × Tt × b). The number of entires in the binary sequences is N × M × 6 + (N × M × 3 × 8)/r, where N and M are, the number of rows and columns of original image, respectively. This binary sequence is converted into decimal sequence. The first N × M × 6 entries from decimal sequence are reshaped into six key matrices of size N × M and we call them as Key1, Key2, Key3, Key5, Key6, and Key7. Then take the next (N × M × 3 × 8)/r entries from the decimal sequence starts from index s = N × M × 6 + 1 and reshape it into one dimensional array. We record the previous positions after sorting. The sequence of previous positions is arranged in a new array called the permutation key, say Key4, which is of size (N × M × 3 × 8)/r.

4 Encoding and decoding scheme

In the encoding process, we are converting the image into a binary bit sequence, then divide this string into blocks where each block contains r number of bits (r can be any divisor of the total number of bits in the sequence). The advantage of this step is that it provides an extra layer of security (due to the fact that the simple shuffling process in the encoded domain will provide good confusion and diffusion both) to the proposed algorithm and also r can be treated as a users’ friendly parameter. For instance, if r = 3, then encoding scheme can be viewed in Fig. 2.

Fig. 2
figure 2

Demonstration of encoding scheme for r = 3

Similarly, the decoding process carried out in a reverse manner.

5 Generalized Vigenére-type cipher on S n group

In this section, we propose a new method to replace pixel values and its position in more random way by using GVTSG (which prevents an adversary from using a brute force attack to find a key). During the process of encryption the generalized Vigen\(\grave {e}\)re-type table has been obtained by using elements of the symmetric group Sn (as in [16]). There are n! elements in Sn, and each subsequent column of the generalized Vigen\(\grave {e}\)re-type table is filled by an element of Sn selected at random without replacement manner. The generalized Vigen\(\grave {e}\)re-type table is designed large enough to make such a search computationally infeasible. The GVTSG is of size n × q, where 1 ≤ qn!. Thus the total number of options to fill the generalized Vigen\(\grave {e}\)re-type table without repeating of elements is (n!) × (n! − 1) × (n! − 2) ×… × (n! − q + 1).

For instance, if we take n = 3, and q = 3 then we get symmetric group S3 as shown in Figure 3 (i.e. symmetric group of equilateral triangle), and the Table 1 is created using the elements of S3.

Fig. 3
figure 3

Symmetric group of equilateral triangle

Table 1 Generalized Vigen\(\grave {e}\)re-type table of size 3 × 3 on S3 group

Similarly, if we take n = 26, q = 26 and use the first 26 elements from S26, (say \(s_{26}^{1}\), \(s_{26}^{2}\), \(s_{26}^{3}\)\(s_{26}^{26}\)) then the columns of the table from 1 to q can be filled with \(s_{26}^{1}\), \(s_{26}^{2}\), \(s_{26}^{3}\) and \(s_{26}^{26}\), respectively. This will give a generalized Vigen\(\grave {e}\)re-type table which is the well known Table in the literature as shown in the Table 2, where symbols are used as alphabets.

Table 2 Generalized Vigen\(\grave {e}\)re-type table of size 26 × 26 on S26 group

Thus, in general, the generalized Vigen\(\grave {e}\)re-type table of size n × q on Sn group is depicted in Table 3, which is used as a final step in the encryption process.

Table 3 Generalized Vigen\(\grave {e}\)re-type table of size n × q on Sn group (n = 256 for images)

5.1 Generalized Vigenére-type encryption on S n group

Let M be the message and K be the key, then encrypted message C can be obtained by the following mechanism.

$$ C = E_{n}(M,K)-1 , \hspace{0.5cm} M < n, $$
(3)

where En is the function takes entry from Mth + 1 row and Kth column of generalized Vigen\(\grave {e}\)re-type table on Sn group.

For example, let us take the generalized Vigen\(\grave {e}\)re-type table on S5 group of size 5 × q where q = 5 as shown in Table 4.

Table 4 Generalized Vigen\(\grave {e}\)re-type table size 5 × 5 on S5 group

Let us say the message be M = 2 and corresponding key be K = 5. Then the encrypted message value will be tracking the value of (M + 1)th = 3rd row and Kth = 5th column which is 1. So the encrypted message be 0.

$$ 0 = E_{5}(2,5)-1 $$
(4)

5.2 Generalized Vigenére-type decryption on S n group

We sort GVTSG column wise and store the previous positions of elements before they sorted. This table will be used for decryption.

Here C can be decrypted with key K by the following mechanism.

$$ M = D_{n}(C,K)-1, \hspace{0.5cm} C < n $$
(5)

where Dn is the function that takes entry from Cth + 1 row and Kth column.

For the above example, we sort Table 4 in column wise and store the previous positions (as shown in Table 5). Then the decrypted message value will track the value of (C + 1)th = 1st row and Kth = 5th column which is 3. So the decrypted message is 2.

$$ 2 = D_{n}(0,5)-1 $$
(6)
Table 5 Generalized Vigen\(\grave {e}\)re-type table for decryption size 5 × 5 on S5 group

Note that the robustness of algorithm depends on the following: In key1 any entry may have a value greater than 255, if we could take t > 5 and this value will be handled by modulo 255. When we look on a conventional Vigen\(\grave {e}\)re cipher. The same table is used for encryption and decryption process. But, encryption and decryption processes are different. The restriction here is that the table can’t be changed (decryption will be difficult and take more time). While, in the proposed algorithm, we use same procedure for encryption/decryption and different tables (as explained in Tables 45). The main advantage is that all possible permutation in each column can be used in Generalized Vigen\(\grave {e}\)re-type table. This is completely an innovative process (during decryption) by handling all permutation without any repetition in each column in the Generalized Vigen\(\grave {e}\)re-type table. The proposed algorithm resists against chosen cipher-image/plain-image attacks completely as the columns of Vigen\(\grave {e}\)re-type table can be filled by any permutation without repetition.

6 Proposed encryption/decryption algorithm

In the following, we discuss the proposed encryption and decryption algorithm.

6.1 Encryption algorithm

This section will elaborate on the proposed encryption procedure in detail and the flowchart is given in Fig. 4. Here, we have divided the encryption process into three simple steps as below:

  1. Step 1:

    Modulo Bit-XOR: At first, we take each component (R, G, and B) of the original RGB image, and then apply bitwise operation by using RKS (as explained in Section 3) obtained from GHE to get XOR-ed components (say, R1, G1, and B1)

    $$ \begin{array}{@{}rcl@{}} R_{1}= (R \oplus Key_{1}) (mod \hspace{0.2cm} 256),\\ G_{1}= (G \oplus Key_{2}) (mod \hspace{0.2cm} 256),\\ B_{1}= (B \oplus Key_{3}) (mod \hspace{0.2cm} 256). \end{array} $$

    This process will be effective only if the keys (Key1, Key2 and Key3) are random. And it shows the effectiveness of proposed algorithm in terms of mapping each pixel value to a new random pixel value with help of RKS.

  2. Step 2:

    Array permutation shuffling in encoded domain: After executing Step 1, we combine each components, R1, G1, and B1 to get a color image I1. Then we apply encoding scheme on image I1 as explained in array permutation shuffling, which is defined as below: As explained in Section 4, we apply the array permutation shuffling on A by using Key4 to obtain a new shuffled array C. Here

    $$ \begin{array}{@{}rcl@{}} C(i) = A(Key_{4}(i)), \end{array} $$

    we decode this shuffled array C into an image I2 by applying the reverse process of the encoding scheme. This step is used to shuffle blocks randomly in the encoded domain that offers confusion and diffusion both. Thus this process will provide another layer of security to the encryption algorithm.

  3. Step 3:

    Generalized Vigenére-type cipher onSngroup operation: Here, we decompose image I2 into three components: R2, G2, and B2 and apply E256 on each components as follows:

    $$ \begin{array}{@{}rcl@{}} R_{3}(i,j) = E_{256}(R_{2}(i,j),Key_{5}(i,j)(mod \hspace{0.2cm} q))-1,\\ G_{3}(i,j) = E_{256}(G_{2}(i,j),Key_{6}(i,j)(mod \hspace{0.2cm} q))-1,\\ B_{3}(i,j) = E_{256}(B_{2}(i,j),Key_{7}(i,j)(mod \hspace{0.2cm} q))-1, \end{array} $$

    En is defined in Sub-Section 5.1. Next, we compose the components R3, G3, and B3 to get an encrypted image I3. The columns of GVTSG are filled with q-permutations which offer very good challenge to the adversary for decrypting correct pixel values. Since the E256 columns are replaced with all possible permutations, the decryption process takes more time. However, we have introduced new method for decryption process (described in Sub-Section 5.2) in an efficient way that the running time is much lower for images containing more pixels.

Fig. 4
figure 4

Flowchart of the encryption algorithm

6.2 Decryption algorithm

  1. Step 1:

    Generalized Vigen\(\grave {e}\)re-type decipher onSngroup operation: Now, we decompose image I3 into three components as R3, G3, and B3 and apply D256 on each components as follows:

    $$ \begin{array}{@{}rcl@{}} R_{2}(i,j) = D_{256}(R_{3}(i,j),Key_{5}(i,j)(mod \hspace{0.2cm} q))-1,\\ G_{2}(i,j) = D_{256}(G_{3}(i,j),Key_{6}(i,j)(mod \hspace{0.2cm} q))-1,\\ B_{2}(i,j) = D_{256}(B_{3}(i,j),Key_{7}(i,j)(mod \hspace{0.2cm} q))-1, \end{array} $$

    Dn is defined in Sub-Section 5.2. Then we compose the components R2, G2, and B2 to get an image I2.

  2. Step 2:

    Array permutation reshuffling in encoded domain: After executing Step 1, We apply encoding scheme on image I2 as explained in array permutation shuffling, which is defined as below:

    Now we apply the array permutation reshuffling on C by using Key4 to obtain reshuffled array A. Here

    $$ \begin{array}{@{}rcl@{}} A(Key_{4}(i))= C(i), \end{array} $$

    decodes this reshuffled array A into an image I1 by applying reverse process of encoding scheme.

  3. Step 3:

    Modulo Bit-XOR: Decompose I1 to get components R1, G1, and B1 , and then apply bitwise operation by using RKS obtained from GHE to get original components (R, G, and B)

    $$ \begin{array}{@{}rcl@{}} R= (R_{1} \oplus Key_{1}) (mod \hspace{0.2cm} 256),\\ G= (G_{1} \oplus Key_{2}) (mod \hspace{0.2cm} 256),\\ B= (B_{1} \oplus Key_{3}) (mod \hspace{0.2cm} 256). \end{array} $$

Finally, compose the components R, G, and B to get an decrypted image I.

The original, encrypted and decrypted images are shown in Fig. 5 for Baboon, Lena, Peppers, and Fruits.

Fig. 5
figure 5

Encryption/decryption results: a original Baboon image, b encrypted Baboon image, c decrypted Baboon image, d original Lena image e encrypted Lena image, f decrypted Lena image g original Peepers image, h encrypted Peepers image, i decrypted Peepers image, j original Fruits image, k encrypted Fruits image, l sdecrypted Fruits image

7 Construction, performance and security analysis

A new key space has been constructed in such a way that it can resists brute force attack infeasible. Further, a secure encryption scheme should resist various types of cryptanalysis such as statistical attacks, brute force attack, cipher-image, and plain-image attacks. We have performed several tests to check the security of the proposed algorithm. Furthermore, we have also shown the efficiency of the proposed algorithm by means of the time complexity.

7.1 Construction of the key space

  • Key space involves the following parameters and construction has been derived as follows:

    • The symbol Δx, and Δt represent the size of intervals in length and time domain, respectively in GHE, such that Δt ≤Δx and used as secret keys. These secret keys are sensitive up to five decimal places. This sensitivity plays a very important role in the encryption process.

    • The parameter 𝜃 in the GHE is also used as a secret key and sensitive up to 5 decimal places.

    • The positive integer k can be taken large enough to maintain high security.

    • The index j varies from 1 to (Lx × Tt × b) − p are taken at random to generate RKS. A slight change in the index j will result in a drastic change in the RKS.

    • The block size z of S in RKS varies from 2 to (Lx × Tt × b) and is very sensitive with respect to its size.

    • The block size r (can be any divisor of the total number of bits in RGB image) in encoding scheme plays an important role during encryption. It can be a user’ friendly parameter to maintain computational costs.

    • The polynomial coefficients of the initial condition are a1, a2, a3,…, ap where each coefficient is sensitive to up to 15 decimal places. This will also provide extreme support to the proposed algorithm in terms of various types of attacks.

  • Generalized Vigen\(\grave {e}\)re-type table of size n × q on Sn group: There are 256! permutations to fill each column in generalized Vigen\(\grave {e}\)re-type table. Generalized Vigen\(\grave {e}\)re-type table of size n × q on Sn group columns can be filled (256!) × (256! − 1) × (256! − 2)... × (256! − q + 1) many ways. Three generalized Vigen\(\grave {e}\)re-type tables are created for R, G and B components. So the key space will be ((256!) × (256! − 1) × (256! − 2)... × (256! − q + 1)) for generating generalized Vigen\(\grave {e}\)re-type table.

So the constructed RKS is of size (256!) × (256! − 1) × (256! − 2)... × (256! − q + 1) × 1015p+ 15 × η, where p is number of coefficients in polynomial, q is number of columns in generalized Vigen\(\grave {e}\)re-type table and η = k × (Lx × Tt × b) − p × ((Lx × Tt × b) − 1) × r. For instance, if p = 10 and q = 100, then the key space is of size > 10151365 which is large enough to resist brute force attack.

7.2 Computational complexity analysis

To evaluate the computational complexity of the proposed algorithm, the total time complexity is O(n × m × 3) for an RGB image size n × m × 3. Since each step in encryption algorithm has the same time complexity for Bit-XOR operation, array permutation shuffling, and GVCSG. The proposed algorithm is designed in such a way that the encryption and decryption processes are taking an average of 0.519 and 0.518 seconds for an image of size 512 × 512 × 3 on IntelR CoreTM i7-6700K CPU@ 4.00 GHZ Processor, 32GB RAM, Windows 10 Pro and MATLAB R2016a. Thus, we can conclude that the running time of the proposed algorithm is very low.

7.3 Histogram analysis

Histogram analysis shows the number of pixels with each intensity values from 0 to 255. A Histogram is very useful for an adversary to analyze graphical view of the data distribution (cipher space). Histograms for distribution intensity of pixels over cipher images of Baboon, Lena, Fruits, and Peppers (Figs. 678 and 9) clearly depicts, each distribution cannot be distinguishable statistically.

Fig. 6
figure 6

Histogram analysis for Baboon image: a original red component, b encrypted red component, c decrypted red component, d original green component, e encrypted green component, f decrypted green component, g original blue component, h encrypted blue component, i decrypted blue component

Fig. 7
figure 7

Histogram analysis for Lena image: a original red component, b encrypted red component, c decrypted red component, d original green component, e encrypted green component, f decrypted green component, g original blue component, h encrypted blue component, i decrypted blue component

Fig. 8
figure 8

Histogram analysis for Peepers image: a original red component, b encrypted red component, c decrypted red component, d original green component, e encrypted green component, f decrypted green component, g original blue component, h encrypted blue component, i decrypted blue component

Fig. 9
figure 9

Histogram analysis for Fruits image: a original red component, b encrypted red component, c decrypted red component, d original green component, e encrypted green component, f decrypted green component, g original blue component, h encrypted blue component, i decrypted blue component

7.4 The mean square error and peak signal-to-noise ratio

The Mean Square Error (MSE) between the original and encrypted/decrypted image is calculated by the following formula

$$ \begin{array}{@{}rcl@{}} MSE = \frac{1}{N\times M}\sum\limits_{i=1}^{N}\sum\limits_{j=1}^{M} [f(i,j)- f_{o}(i,j)]^{2} \end{array} $$

where f and fo are the intensity functions of encrypted/decrypted and original images and (i, j) are the position of pixels. The proposed algorithm offers zero MSE value between original and decrypted image and it is briefly described in Table 6. Similarly, MSE between the original and encrypted images is large enough and it can be seen in Table 7. Thus, given the values presented in Table 6, we can claim that the proposed algorithm has no loss in original data.

Table 6 Mean square error and signal to noise power ratio between original and decrypted images of Baboon, Lena, Fruits, and Peppers respectively
Table 7 Mean square error and signal to noise power ratio between original and encrypted images of Baboon, Lena, Fruits, and Peppers respectively

The term Peak Signal-to-Noise Ratio (PSNR) is the ratio between the maximum value (power) of a signal and the power of distorting noise that affects the signal. PSNR between the original and encrypted/decrypted image is calculated by the following formula

$$ \begin{array}{@{}rcl@{}} PSNR = 20\times log\frac{MA{X_{I}^{2}}}{\sqrt{MSE}}, \end{array} $$

where MAXI is the maximum possible pixel value of the image. PSNR value between the original and encrypted images should be sufficiently small to ensure that the originality between original and encrypted image. The MSE and PSNR between original and decrypted image for Baboon, Lena, Fruits, and Peppers images are 0 and respectively. The values of MSE and PSNR between the original and encrypted images are represented in Table 7.

7.5 Correlation Analysis

Correlation between two adjacent pixels can be tested in three ways: by taking two vertically adjacent pixels, by taking two horizontally adjacent pixels or by taking two diagonally adjacent pixels of the encrypted image as shown in Tables 89 and 10. We have selected randomly 10000 pairs of adjacent pixels to calculate their correlation horizontal and vertical coefficients. And selected randomly 1000 pairs of adjacent pixels to calculate their correlation diagonal coefficient. Here, rxy of each adjacent pair is calculated for the original and encrypted images using the following formulas:

$$ \begin{array}{@{}rcl@{}} r_{xy}=\frac{cov(x,y)}{\sqrt{D_{x}} \sqrt{D_{y}}},\\ cov(x,y)=E\left[(x-E(x))(y-E(y))\right],\\ E(x)=\frac{1}{N} \sum\limits_{i=1}^{N}x_{i},\\ D_{x}=\frac{1}{N}\sum\limits_{i=1}^{N}(x_{i}-E(x))^{2}, \end{array} $$

where x and y are values of two adjacent pixels in each channel, and N denotes the total number of samples taken. This phase essentially shows that after encryption, the correlation among the image pixels is broken whereas decryption will bind the pixels with the original correlation. The correlation plots are presented in Figs. 10 and 11.

Table 8 Horizontal correlation of Baboon, Lena, Fruits and Peppers images
Table 9 Vertical correlation of Baboon, Lena, Fruits and Peppers images
Table 10 Diagonal correlation of Baboon, Lena, Fruits and Peppers images
Fig. 10
figure 10

Correlation analysis for Baboon image: a horizontal correlation of original red component, b horizontal correlation of original green component, c horizontal correlation of original blue component, d horizontal correlation of encrypted red component, e horizontal correlation of encrypted green component, f horizontal correlation of encrypted blue component, g vertical correlation of original red component, h vertical correlation of original green component, i vertical correlation of original blue component, j vertical correlation of encrypted red component, k vertical correlation of encrypted green component, l vertical correlation of encrypted blue component, m diagonal correlation of original red component, n diagonal correlation of original green component, o diagonal correlation of original blue component, p diagonal correlation of encrypted red component, q diagonal correlation of encrypted green component, r diagonal correlation of encrypted blue component

Fig. 11
figure 11

Correlation analysis for Lena image: a horizontal correlation of original red component, b horizontal correlation of original green component, c horizontal correlation of original blue component, d horizontal correlation of encrypted red component, e horizontal correlation of encrypted green component, f horizontal correlation of encrypted blue component, g vertical correlation of original red component, h vertical correlation of original green component, i vertical correlation of original blue component, j vertical correlation of encrypted red component, k vertical correlation of encrypted green component, l vertical correlation of encrypted blue component, m diagonal correlation of original red component, n diagonal correlation of original green component, o diagonal correlation of original blue component, p diagonal correlation of encrypted red component, q diagonal correlation of encrypted green component, r diagonal correlation of encrypted blue component

7.6 Key change analysis

Assume that the intruder is trying to recover the image by having all the original keys except one. However, we can claim that an intruder cannot recover the original image by knowing all the exact keys except one and output can be seen in Fig. 12.

Fig. 12
figure 12

Key space analysis: a incorrectly decrypted image with marginal difference of 0.000000000000001 in the coefficient of initial condition of GHE, b incorrectly decrypted image with slight variation in order of generalized Vigen\(\grave {e}\)re-type table order from (1,2,3) to (3,1,2) corresponding to R, G and B components respectively, c incorrectly decrypted image with shift of index s by s + 1 in array permutation key, d incorrectly decrypted image with small variation in bit-XOR key order (Key1, Key2, Key3) to (Key3, Key1, Key2) corresponding to R, G and B components respectively, e incorrectly decrypted image with exchange of generalized Vigen\(\grave {e}\)re-type key order (Key5, Key6, Key7) to (Key7, Key5, Key6) corresponding to R, G and B components respectively, f incorrectly decrypted image with marginal increment of 0.00001 in secret key Δx, g incorrectly decrypted image with marginal increment of 0.00001 in secret key Δt, h incorrectly decrypted image with marginal increment of 0.00001 in secret key 𝜃, i incorrectly decrypted image with marginal increment of 1 in secret key k, j incorrectly decrypted image with marginal increment of 1 secret key in j, k incorrectly decrypted image with marginal increment of 1 in secret key z, l incorrectly decrypted image with marginal increment of 1 in secret key r, m correctly decrypted image with correct keys

7.7 Differential attacks

Variations in the original image, are made by the attackers who also make use of the algorithm which is proposed to encrypt the original image before and after changing pixels, and through estimating similarity of two encrypted images, they discover the relationship between the original and the encrypted images. However, if any minor change in the original image causes a large change in the encrypted image then the differential attack becomes useless. In this paper, we have put forward a new method to deal with differential attacks.

7.7.1 NPCR analysis

The Number of Pixels Change Rate (NPCR) refers to the change rate of the number of pixels of the encrypted image while one pixel of plain-image is changed. And it is calculated as follows:

$$ \begin{array}{@{}rcl@{}} NPCR:N(C^{1},C^{2}) =\frac{{\sum}_{i=1}^{N} {\sum}_{j=1}^{M}D(i,j)}{N\times M} \times 100<percent>, \end{array} $$

where

$$ \begin{array}{@{}rcl@{}} D(i,j)= \left\{\begin{array}{ll} 1,& \text{if } C^{1}(i,j) = C^{2}(i,j)\\ 0, & \text{otherwise} \end{array}\right. \end{array} $$

where C1 and C2 are encrypted images before and after one-pixel change in plain-image, and a bipolar array D(i, j) defined as above.

7.7.2 UACI analysis

The Unified Average Change Intensity (UACI) measures the average intensity of differences between two encrypted images. It is mathematically defined as:

$$ \begin{array}{@{}rcl@{}} UACI:U(C^{1},C^{2})=\frac{{\sum}_{i=1}^{N} {\sum}_{j=1}^{M}\frac{\left|C^{1}(i,j)-C^{2}(i,j)\right|}{255}}{N\times M}\times 100\%, \end{array} $$

where C1 and C2 are encrypted images before and after one-pixel change in plain-image. Table 11 shows the values of NPCR (> 99%) and UACI (33%) for each color component between two encrypted images. Experimental results show the estimated expectations and variance of NPCR and UACI are very close to the theoretical values, which justify the validity of theoretical values [38]. Hence, the proposed scheme is immune against differential attacks.

Table 11 NPCR and UACI values of the proposed algorithm

7.7.3 Statistical test for NPCR

From [38], we can demonstrate that the proposed algorithm is good by using statistical test for NPCR. Suppose we have two encrypted-images C1 and C2 of size 512 × 512 × 3 each, then hypotheses (H0 and H1) with significance level α for N(C1, C2) are:

$$ \begin{array}{@{}rcl@{}} H_{0}: N(C^{1},C^{2}) = \mu_{N} \end{array} $$
$$ \begin{array}{@{}rcl@{}} H_{1}: N(C^{1},C^{2}) < \mu_{N}. \end{array} $$

Reject H0, if \(N(C^{1},C^{2}) < N_{\alpha }^{*}\), otherwise accept H0, where

$$ \begin{array}{@{}rcl@{}} N_{\alpha}^{*} = \mu_{N}- \phi^{-1}(\alpha) \sigma_{N} = \frac{\left( F - \phi^{-1}(\alpha)\sqrt{\frac{F}{MN}}\right)}{F+1}, \end{array} $$
$$ \begin{array}{@{}rcl@{}} \mu_{N} = \frac{F}{F+1}, \end{array} $$
$$ \begin{array}{@{}rcl@{}} {\sigma^{2}_{N}} = \frac{F}{(F+1)^{2}MN}, \end{array} $$

where F largest pixel value in the original image.

We can observe from Table 12, N(C1, C2) values for Baboon, Lena, and Peppers exceeds \(N_{\alpha }^{*}\) values for α = 0.05, 0.01, and 0.001. So, we can accept the null hypothesis (H0). Hence, the NPCR values confirm that the proposed algorithm is good.

Table 12 Statistical test for NPCR

7.7.4 Statistical test for UACI

Likewise, again from [38], we can demonstrate the proposed algorithm is good by using statistical test for UACI. Assuming that, we have two encrypted-images C1 and C2 of size 512 × 512 × 3 each, then hypotheses (H0 and H1) with significance level α for U(C1, C2) are:

$$ \begin{array}{@{}rcl@{}} H_{0}: U(C^{1},C^{2}) = \mu_{U}, \end{array} $$
$$ \begin{array}{@{}rcl@{}} H_{1}: N(C^{1},C^{2}) < \mu_{U}, \end{array} $$

Reject H0, if \(U(C^{1},C^{2}) \notin (U_{\alpha }^{*+}, U_{\alpha }^{*-})\), otherwise accept H0, where

$$ \begin{array}{@{}rcl@{}} U_{\alpha}^{*+} = \mu_{U} + \phi^{-1}(\alpha/2) \sigma_{U}, \end{array} $$
$$ \begin{array}{@{}rcl@{}} U_{\alpha}^{*-} = \mu_{U} - \phi^{-1}(\alpha/2) \sigma_{U}, \end{array} $$
$$ \begin{array}{@{}rcl@{}} \mu_{U} = \frac{F+2}{3F+3}, \end{array} $$
$$ \begin{array}{@{}rcl@{}} {\sigma^{2}_{U}} = \frac{(F+2)(F^{2}+2F+3)}{18(F+1)^{2}MNF}, \end{array} $$

From the Table 13, U(C1, C2) values for Baboon, Lena, and Peppers belongs to the interval \((U_{\alpha }^{*+}, U_{\alpha }^{*-})\) for α = 0.05, 0.01, and 0.001. So, we can accept the null hypothesis (H0). Hence, the UACI values confirm that the proposed algorithm is good.

Table 13 Statistical test for UACI

7.8 Cropped attack analysis

The encrypted text has been cropped and analyzed over data loss in real-time communication by the proposed algorithm. We performed two different kinds of attack over an encrypted image and resulting images have been analyzed over the loss of data. The intention is to check whether there is visibility that exists or not in the decrypted image and results of this analysis can be seen in Fig. 13.

Fig. 13
figure 13

Cropped attack experiment: a encrypted image from different locations in each layer (R, G and B) is cropped , b decrypted result, c encrypted image from same locations in each layer (R, G and B) is cropped, d decrypted result

7.9 Communication channel noise analysis

The communication channel is influenced by various noises. For example, in the wireless medium, the losses in signals are multi-path propagation environment, attenuation, interference, and near-far problem. These factors make a loss in transferred message bits in the communication channel. Images transmitted through medium should not lose much data once after received at the other end. An algorithm should provide confidence about images received in other ends; not having more data loss. We checked with Salt & Pepper noise test with images and results are depicted in Fig. 14.

Fig. 14
figure 14

Communication channel noise analysis: a encrypted image with Salt and Pepper noise attacked, d decrypted result

7.10 Information entropy

Information entropy is used to analyze the randomness of information contained in the image. The information entropy is defined by the symbol H(m) as below.

$$ \begin{array}{@{}rcl@{}} H(m) = \sum\limits_{i=0}^{2^{N}-1}p(m_{i}) \log_{2}\left( \frac{1}{p(m_{i})}\right) \end{array} $$

where P(mi) denotes the probability of symbol mi. For a purely random source emitting 2N symbols, the entropy is H(m) = N. In gray-scale image pixels are in the range [0, 255], the maximum information entropy is 8. Thus for a secure encryption algorithm, the entropy value should be closed to the value 8. The proposed algorithm provides entropy value greater than 7.9912 for all encrypted images as depicted in Table 14, which is very closed to 8. We have compared the information entropy of encrypted Lena image with other existing algorithms in Table 21. Thus data presented Tables 14 and 21 confirm that the proposed algorithm is robust against entropy attack.

Table 14 The result of information entropy

8 Comparison of proposed algorithm with others

The proposed algorithm has been compared with existing competing methods [6, 11, 14, 15, 23, 25, 39, 46]. Keyspace of the proposed algorithm has been compared with the key space of other competing methods and results are shown in Table 15. The proposed key space is good enough to resist a brute force attack. Values of NPCR, and UACI analyses have been compared and shown in Tables 16 and 17, respectively. The proposed algorithm has better NPCR and UACI values than existing algorithms. Further, a fine comparison on correlation has been done in three different directions as horizontal correlation shown in Table 18, vertical correlation shown in Table 19, and diagonal correlation shown in Table 20. Values presented in these Tables 1719 demonstrate that each pixel value after encryption has very low correlation. Finally, information of entropy on the encrypted image with other different encryption algorithms has been compared and we found that the proposed algorithm has better entropy than others that can be seen in Table 21.

Table 15 Compare the key space of the proposed algorithm with existing algorithms
Table 16 Comparison on NPCR analysis of the proposed method with authors [6, 11, 46]
Table 17 Comparison on UACI analysis of the proposed method with authors [6, 11, 46]
Table 18 Comparison of horizontal correlation of encrypted image of Lena image with other authors [11, 15, 46]
Table 19 Comparison of vertical correlation of encrypted image of Lena image with other authors [11, 15, 46]
Table 20 Comparison of diagonal correlation of encrypted image of Lena image with other authors [11, 15, 46]
Table 21 The information entropy of the encrypted Lena image by using different encryption algorithms

9 Conclusion

A new, fast and secure RGB image encryption using generalized heat equation associated with generalized Vigen\(\grave {e}\)re-type table over symmetric group Sn is proposed. With the help of the generalized heat equation and generalized Vigen\(\grave {e}\)re-type table, we have generated a random key sequence. The randomness of this sequence has been verified successfully by using NIST statistical test suite and results are shown in Figs. 1516 and 17. Using the generated random sequence, we have constructed a new random key space which will provide a lot of support to the proposed algorithm in terms of attacks. Comparison with the existing competing methods have been done and the results are reported in Tables 1521. The data presented in Tables 14 and 21 confirm that the proposed algorithm renders entropy value closed to 8. Further, keyspaces, key sensitivity, histogram analysis, correlation plots, correlation coefficients, MSE, PSNR, NPCR, UACI, and statistical tests for NPCR and UACI have been performed and the results of these analyses suggest that the proposed algorithm can resist exhaustive attacks effectively. Thus the proposed algorithm offers better security and resistance for various types of attacks without compromising the running time. In the future, we plan to explore the usage of RKS in secure communication of UAV’s (Unmanned Aerial Vehicles).

Fig. 15
figure 15

Statistical result over 10 blocks

Fig. 16
figure 16

Statistical result over 25 blocks

Fig. 17
figure 17

Statistical result over 100 blocks