1 Introduction

The rapid advancement of science and technology has fostered an environment where individuals readily exchange personal data across digital platforms. However, this increased reliance on digital infrastructure has concurrently exacerbated concerns regarding unauthorized access to sensitive information, posing a significant challenge to information security [1, 2]. Data protection strategies often employ cryptographic methods like encryption and steganography to ensure information security and confidentiality [3, 4]. While encryption transforms data into an unreadable format, steganography conceals the very existence of sensitive information within innocuous cover media, offering complementary approaches to safeguarding data against unauthorized access and potential breaches [4].

In the field of cryptography, various classical and modern algorithms have been explored and can be divided into as either symmetric or asymmetric algorithms [1, 2]. Symmetric algorithms, also known as private-key cryptography, employ a single shared key for both encryption and decryption processes [1]. Popular examples include the Advanced Encryption Standard (AES), Data Encryption Standard (DES), and Blowfish [1]. Asymmetric algorithms, or public-key cryptography, utilize a pair of mathematically related keys: a public key for encryption and a private key for decryption [1]. Popular examples include the RSA, Elliptic Curve, and Diffie-Hellman [1].

Although traditional cryptographic algorithms such as DES, AES, and RSA have proven effective for text encryption, their direct application to image data presents significant challenges [3]. Inherent properties of images, including large data size, high redundancy, and strong inter-pixel correlations, make these algorithms inefficient and potentially vulnerable when applied to image encryption [3]. Several algorithms have been proposed over the past decade for use in image encryption to meet security needs and protect the privacy of image information. One of the main techniques is a chaotic image encryption [3].

Chaotic encryption exploits the inherent complexity and nonlinear dynamics of chaotic systems to improve data security, with the goal of generating highly unpredictable ciphertexts that are resistant to traditional cryptanalysis methods [1,2,3]. By incorporating chaotic maps into encryption algorithms, researchers exploit properties such as sensitivity to initial conditions and deterministic randomness to establish complex relationships between plaintext, ciphertext, and encryption keys [1,2,3]. Thus, many chaos-based image encryption techniques [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46] have been presented in recent decades. these techniques use the chaos to introduce randomness, disrupt statistical models, obscure any perceptible connection between the original image, the encrypted image and the secret key and thus respect the principles of confusion and diffusion (recommendations of Shannon) [1].

Confusion is a fundamental principle in image encryption that aims to obscure the relationship between the plaintext image, the ciphertext image, and the encryption key, making it difficult for attackers to decipher the original content [1,2,3]. This is typically achieved through operations such as pixel permutation, substitution, or a combination of both, often utilizing chaotic systems or other sources of randomness [1,2,3]. Permutation focusing on disrupting the pixel positions within an image to obscure its visual content and break correlations between adjacent pixels. By rearranging the pixel locations according to a specific rule or key, permutation introduces confusion into the encryption process, making it difficult for attackers to analyze the image statistically or visually [11,12,13,14,15]. Substitution involves changing pixel values based on a predefined rule or key, thereby hiding the original content of the image and introducing confusion into the encryption process. Unlike permutation, which rearranges pixel positions, substitution directly modifies pixel values, often using substitution tables (S-Boxes) or mathematical functions derived from chaotic systems. Efficient substitution techniques aim to achieve high nonlinearity and an avalanche effect, where a minor change in the plaintext image results in a substantial change in the cipher image [16,17,18,19,20,21,22,23,24]. There are various substitution methods, including simple XOR operations [5], the S-Boxes inspired by the Vigenère cipher [25,26,27,28], the Hill's cipher improvements [29, 30], the Feistel improvements [31, 32], the genetic algorithms [33,34,35,36,37,38,39, 42, 46].

Diffusion is a critical process in image encryption that aims to propagate the effect of a single plaintext change throughout the entire ciphertext, thereby obscuring statistical relationships and enhancing resistance against cryptanalysis. Unlike permutation and substitution, which focus on individual pixel positions or values, diffusion operates on a larger scale, spreading the impact of modifications across multiple pixels. This is typically achieved through iterative operations involving XOR, modular arithmetic, or other mathematical functions, often incorporating chaotic sequences for added randomness. Effective diffusion techniques strive for a high avalanche effect, ensuring that even a small change in the plaintext image results in a significantly different ciphertext, making it difficult for attackers to decipher the original content [3].

In this article, we introduce an effective method for image encryption. It is based on the Hill cipher, the Vigenère cipher and three chaotic maps. Our technique takes place in four steps: First, five chaotic vectors and two control vectors are created from three chaotic maps (chaotic map, skew tent map and Henon map). In the second step, the original image is vectorized using chaotic vectors. In the third step, we proposed two improvements, one on the classical method of Hill and the other on the classical method of Vigenère. The fourth step is to encrypt the images using a combination of improved Hill and Vigenère methods and a diffusion mechanism to chain the system.

The main innovations and contributions of this article are as follows: (1) The proposal of a new chaotic system composed of three chaotic maps. The system allows seven pseudo-random vectors to be generated. (2) the use of concatenation attached to confusion with chaotic vectors to vectorize the original image. (3) The construction of a new S-box inspired by the Vigenère matrix. (4) The construction of a new dynamic matrix of size 3 × 3 inspired by the static Hill matrix. (5) The proposal of a new method to combines improved Hill's method and improved Vigenère’s method using a control vector. (6) The use of a chaining mechanism connecting each pixel to the previous encrypted pixel to increase the avalanche effect. (7) The design of a new image encryption algorithm based on the combination of improved S-Box and Hill matrix and using a chaining mechanism.

This article is structured as follows: the second part describes some previous works. The third part presents the theoretical basis used in our method. The steps of our approach as well as the results obtained and their interpretations are the subject of the fourth and fifth parts. The conclusion and perspectives of this work are presented in the last part.

2 Related work

Several approaches for image encryption have been developed in recent decades, including:

The paper [6] describes a color image encryption algorithm that determines the necessary parameters for pixel scrambling using a two-dimensional chaotic map (2D-LSCM). The first step involves transforming the pixels of each channel by switching the rows and columns of different channels in accordance with the random sequences produced by the 2D-LSCM. Next, the four most significant bits of each channel's pixels are concatenated and shifted in a circular manner, both horizontally and vertically. The second stage consists of diffusing the scrambled pixels of each channel using the diffusion sequences controlled by the control sequences. The proposed method achieves high security and efficiency for color image encryption. Its main strengths include plaintext sensitivity, large keyspace, and fast encryption speed suitable for real-time applications. However, it only scrambles the most significant bits of each pixel, potentially leaving the least significant bits vulnerable. Additionally, assessment of resistance to specific advanced attacks is limited.

Noura Khalil et al.,[7] develop an efficient image encryption scheme for both color and grayscale images, employing a hybrid approach with chaotic maps. A 2D sine–cosine cross-chaotic map is utilized in the confusion phase to scramble pixel positions and break correlations. A 1D combined Logistic-Tent map generates a chaotic sequence for diffusion, modifying pixel values through bitwise XOR operations. This combination leverages the strengths of both 1D and 2D chaotic maps, achieving a balance between security and efficiency. The technique's reliance on the combined Logistic-Tent map and 2D sine–cosine map without thorough analysis of their long-term unpredictability raises potential security concerns.

In the article [8], the authors introduce a novel image encryption algorithm combining chaotic systems with Latin squares. The algorithm utilizes a permutation-substitution network based on the two-dimensional logistic map and Latin square operations for confusion and diffusion. Key strengths include a large key space, resistance against various attacks (brute-force, differential, chosen/known plaintext, and statistical), and robustness to noise during decryption. The scheme also incorporates LSB noise embedding and cyclic shift operations for added security. The proposed technique relies on the 2D logistic map, which is known to have weaknesses when discretized and may be susceptible to initial value estimation.

The authors of the article [9] develop a novel 2D hyperchaotic map called 2D-CLSS and utilizes it for image encryption. The 2D-CLSS demonstrates promising chaotic properties and a wide hyperchaotic range, making it suitable for cryptographic applications. The proposed image encryption scheme employs a simultaneous permutation and diffusion strategy, modifying both pixel positions and values based on chaotic sequences generated by the 2D-CLSS. The scheme exhibits strong security performance against various attacks, including statistical, differential, noise, and data loss attacks.

A color image encryption technique based on permutation, substitution, Boolean operations, and chaotic maps was proposed by Tahir Sajjad Ali, et al. [12]. The scheme employs three steps involving permutation, substitution, and diffusion. Permutation is achieved using a permutation table generated by a piecewise linear chaotic map, while substitution utilizes a chaotic S-box for pixel replacement. Finally, diffusion is implemented using the XOR operation with a logistic map-based random sequence. Security analysis demonstrates that the proposed algorithm effectively resists various attacks, including brute-force, statistical, and differential attacks.

Rasul Enayatifar, et al. [14] provides a novel image encryption method employing a synchronous permutation-diffusion technique for gray-level images. The method utilizes a 3D chaotic logistic map and DNA sequences to achieve both permutation and diffusion simultaneously, resulting in a faster encryption process. By combining chaotic systems with DNA operations, the scheme offers enhanced security against common attacks. Extensive experimental results and security analyses demonstrate the effectiveness and robustness of the proposed method in protecting image data during transmission.

Majid Mollaeefar et al. [15] proposes a novel color image encryption scheme based on two newly designed chaotic maps: BCosinus-Arcsinus (CA) and BSinus-Power Logistic (SPL), offering improved chaotic behavior. The scheme includes a chaotic-diagonal permutation method that is highly dependent on the plain image, resulting in efficient pixel shuffling and low correlation between adjacent pixels. The diffusion phase utilizes a coupled map based on the SPL map and a random selection technique to modify pixel values in each color channel independently. The encryption process boasts a large key space, high key sensitivity, and resistance against various attacks.

The article [21] proposes a fast and secure image encryption algorithm utilizing novel 1D chaotic systems and a new plain image substitution technique (PIST). The PIST enhances sensitivity to the original image pixels by performing bit-wise XOR operations between adjacent pixels. The algorithm employs S-boxes generated from the chaotic systems to achieve confusion and diffusion through a scrambling-masking process. Security tests and evaluations demonstrate the efficiency and the reliability of the proposed cryptosystem.

Younes Qobbi, et al. [24] present an image encryption algorithm using two chaotic maps (map and tent), dynamic permutation and a large substitution table. The permutation is generated by three parameters calculated from the pixel values of three-color channels (Red, Green and Blue) of the plain image. The second phase consists of performing a substitution of the pixels of each channel using a large substitution table generated by three chaotic permutations. In order to increase the avalanche effect of the proposed system, an enhanced ECBC chaining mode is used. Experimental results and security analyzes demonstrate the robustness of the proposed algorithm against statistical, brute-force, and differential attacks.

Ritesh Bansal, et al. [28] present a new image encryption scheme based on chaotic maps and the Vigenère scheme. This scheme has a turn consisting of two stages: diffusion and confusion. The first stage consists of three steps: forward scattering, the matching process using the Vigenère scheme, and backward scattering. In a later part, position swapping using a chaotic map is used to swap pixel positions.

M. Essaid, et al. [29] propose a new image encryption algorithm based on a secure variant of Hill Cipher (HC) and three improved one-dimensional (1D) chaotic maps. The proposed scheme fully satisfies the two basic concepts of security, namely confusion and diffusion. The confusion is ensured by the product of a vector consisting of the key-pixel pair and a 2 × 2 Hill matrix on the one hand, and the addition of another pseudo-random translation vector on the other hand. And diffusion is ensured by a strong avalanche effect which connects each encrypted pixel to its neighbour.

The article [32] describes a new cryptosystem which integrates the most powerful genetic operators in the field of color image encryption. This technique begins with the application of an advanced Feistel scheme and ends with the implementation of deeply modified genetic operators. After vectorization of the original image, an application of the advanced Feistel scheme on blocks of random size will be launched. The output vector is transcribed into restricted ASCII code to carry out a genetic crossing suitable for the encryption of color images. The resulting output vector is transcribed into restricted ASCII code to significantly improve the effects of genetic crossing. Simulations on various images demonstrate the effectiveness and security of the proposed cryptosystem against known attacks.

The image encryption method proposed by Mahdieh Ghazvini, et al. [35] is based on genetic algorithm and chaos. The encryption process includes three main stages: the confusion phase, the diffusion phase and the improvement phase using a genetic algorithm. At first, Chen's chaotic map is used in the confusion stage to generate a scrambled image by shuffling the pixels of the simple image, and in the diffusion stage, the Logistic-Sine map changes the gray level values of these pixels. It produces some of the encrypted images which were considered as the initial population for the genetic algorithm. Then, using the genetic algorithm, the encrypted images are optimized to the maximum. Finally, the best encrypted image is the final encrypted image.

Jieyu Zheng, et al. [36] expose a new image encryption scheme combining dynamic DNA sequence encryption with an improved 2D logistic sine map (2D-LSMM) for enhanced security. The 2D-LSMM exhibits better ergodicity and randomness compared to traditional chaotic maps, while dynamic DNA encoding rules, determined by chaotic sequences, add complexity to the encryption process. A logistic map generates a chaotic image for processing DNA operations with the plaintext image. Experimental results and security analysis demonstrate the effectiveness of the scheme in resisting various attacks, including statistical and differential attacks.

Bhaskar Mondal et al. [39] presents a secure image encryption scheme utilizing a novel hybrid pseudo-random number generator (HPRNG) and genetic operations. The HPRNG combines a linear feedback shift register (LFSR) with chaotic maps to generate a highly random bit sequence. The encryption process involves XOR operations and genetic operations like mutation and multipoint crossover, applied to image blocks for confusion and diffusion. The scheme boasts a large key space and produces cipher images with low correlation to the original images and high entropy values. The proposed method is efficient and demonstrates comparable or superior security performance compared to existing schemes.

3 Theoretical foundations

3.1 Chaotic maps

Chaotic maps are mathematical functions that exhibit complex, unpredictable behaviour even though they are deterministic. These maps are widely studied in various fields including physics, biology, and cryptography due to their ability to generate randomness and simulate chaotic systems. Examples of the most popular chaotic maps include the logistic map, the Henon map, and the Tent map.

3.1.1 The logistic map

The logistics map is a one-dimensional chaotic map. The following equation provides its definition:[7]

$$\begin{array}{c}{u}_{n+1}={\mu }_{0}{u}_{n}\left(1-{u}_{n}\right)\end{array}$$
(1)

where \({u}_{0}\) is the initial condition and \({\mu }_{0}\) is the control parameter.

The Fig. 1 presents the bifurcation plot of the logistic map with \({u}_{0}=0.79878796\) and \({\mu }_{0}\in \left[2 4\right]\).

Fig. 1
figure 1

Bifurcation plot of the logistic map [7]

Full size image

As shown in Fig. 1 the logistics map behaves chaotically when \({\mu }_{0}\in \left[\text{3,75} 4\right]\), so in our experiment we chose \({u}_{0}=0.79878796\) and μ0 = 3.755.

3.1.2 The tent map

The Tent map is a one-dimensional chaotic map described by the following recurrence relation [7]:

$${v}_{n+1}=\left\{\begin{array}{c}{\mu }_{1}{v}_{n} if {v}_{n}<0.5\\ {\mu }_{1}\left(1-{v}_{n}\right) Otherwise\end{array}\right.$$
(2)

where \({v}_{0}\) is the initial condition and \({\mu }_{1}\) is the control parameter.

The Fig. 2 presents the bifurcation plot of the tent map with \({v}_{0}=0.89878796\) and \({\mu }_{1}\in \left[0 2\right]\).

Fig. 2
figure 2

Bifurcation plot of the tent map [7]

Full size image

As shown in Fig. 2 the logistics map behaves chaotically when \({\mu }_{1}\in \left[1 2\right]\), so in our experiment we chose v0 = 0.89878796 and μ1 = 1.39.

3.1.3 The Henon map

The Henon map is a two-dimensional discrete-time dynamical system known for its simple formulation and complex chaotic behaviour. It is defined by the following equation: [33]

$${x}_{n+1}=1-{\mu }_{2}{x}_{n}^{2}+{y}_{n}{y}_{n+1}={\mu }_{3}{x}_{n}$$
(3)

The Henon map is governed by two control parameters \({\mu }_{2}\) and \({\mu }_{3}\).

The Henon map may also be deconstructed into a one-dimensional map, defined as follows:

$${w}_{n+2}=1-{\mu }_{2}{w}_{n+1}^{2}+{\mu }_{3}{w}_{n}$$
(4)

where w0 and w1 is the initial conditions and \({\mu }_{2} and {\mu }_{3}\) is the control parameters.

The Fig. 3 presents the bifurcation plot of the Henon map with w0 = 0.47856, w1 = 0.78796, \({\mu }_{2}\in \left[1 1.5\right]\) and μ3 = 0.3

Fig. 3
figure 3

Bifurcation plot of the Henon map [33]

Full size image

As shown in Fig. 3 the Henon map behaves chaotically when \({\mu }_{2}\in \left[1.3 1.45\right]\), so in our experiment we chose w0 = 0.47856, w1 = 0.78796, μ3 = 1.4 and μ3 = 0.3.

3.2 The HILL cipher

The Hill cipher [41] is a text encryption system, based on the segmentation of original text to blocks \({\varvec{X}}=\left\{{{\varvec{x}}}_{1}, {{\varvec{x}}}_{2},\dots ,{{\varvec{x}}}_{{\varvec{p}}}\right\}\) of size \({\varvec{p}}\), and on the construction of an invertible matrix \({\varvec{A}}\) of size \({\varvec{p}}\times {\varvec{p}}\) whose elements are in \({\varvec{Z}}/26{\varvec{Z}}\). Encrypted blocks \({\varvec{Y}}=\left\{{{\varvec{y}}}_{1}, {{\varvec{y}}}_{2},\dots ,{{\varvec{y}}}_{{\varvec{p}}}\right\}\) are obtained by the following encryption formula:

$$\text{Y}=\text{A}\times \text{X }(\text{mod}26)$$
(5)

Decryption is determined by the equation below:

$$\text{X}={\text{A}}^{-1}\times \text{Y}$$
(6)

\({\mathbf{A}}^{-1}\) is the inverse matrix of \(\mathbf{A}\)

3.3 The Vigenère cipher

The Vigenère cipher [43] is a symmetric cipher using polyalphabetic substitution to encrypt and decrypt the secret message. To encrypt a text, we choose an encryption key, to each letter of the Plain-Text we match a letter of the key (the key being repeated as many times as necessary). The correspondence uses a substitution table of size 26 × 26 (Matrice de Vigenère). The letter of the Cipher Text will be taken from the column corresponding to the letter of the Plain-Text, and from the row corresponding to the letter of the key.

Let m, be a positive integer and K = (k1, k2, …, km) be a key where each \(Ki\in {Z}_{26}\). If we set C the Ciphertext, P as the Plain-Text and K as the key, we can express this mechanism by the following formula:

Encryption:

$${C}_{i} = ({P}_{i} + {K}_{i mod m})mod 26$$
(7)

Decryption:

$${P}_{i} = ({C}_{i} - {K}_{i mod m})mod 26$$
(8)

4 Proposed method

Our method for image encryption contains four steps. In the first, we used the logistic map, the tent map and the Henon map to generate three chaotic sequences which are used to construct five pseudo-random vectors and two binary control vectors. In the second, the original image is vectorized utilizing the chaotic vectors. In the third stage, we introduce improvement to the Hill cipher and Vigenère cipher. Finally, the images are encrypted employing a combination of the improved Hill and Vigenère methods, along with a diffusion mechanism.

The various steps of our image encryption method are shown in Fig. 4 below.

Fig. 4
figure 4

Diagram of image encryption by our method

Full size image

4.1 Chaotic vector development

To build our algorithm, we use the logistic map, the tent map and the Henon map to generate three chaotic sequences\({u}_{n}\), \({v}_{n}\) and \({w}_{n}\). These three sequences are used to construct five pseudo-random vectors (V1, V2, V3, V4 and V5) and two control binary vectors (CV1 and CV2). The generation of these vectors is determined by the following algorithm 1:

Algorithm 1
figure f

Control and chaotic vector creation

Full size image

4.2 Preparation of the original image

4.2.1 Vectorization of the original image

After the three color channels were extracted (RGB) and converted to vectors (\(Vr),\left(Vg\right)\text{ and }(Vb)\) of size \((1,nm)\), we apply a concatenation attached to a confusion with the chaotic vectors to generate the vector \({\varvec{X}}\left({{\varvec{x}}}_{0},{{\varvec{x}}}_{1},........,{{\varvec{x}}}_{3{\varvec{n}}{\varvec{m}}-1}\right)\) of size \((1, 3nm).\) The algorithm 2 explains this operation.

Algorithm 2
figure g

Original image vectorization

Full size image

4.2.2 Block subdivision \(\left({{\varvec{U}}}_{{\varvec{i}}}\right)\)

The image vector (X) will be converted into a matrix U of size (nm, 3). The matrix U is given by algorithm 3:

Algorithm 3
figure h

Block subdivision

Full size image

4.3 Combined Hill-Vigenère method

4.3.1 Improved Vigenère method

This step consists of establishing a substitution matrix (SB), whose first row is generated from a permutation (P) obtained using the reference method [42], and the other rows are obtained by shifts from the first row. The algorithm 4 explains this operation.

Algorithm 4
figure i

Substitution matrix development

Full size image

The application of the improved Vigenère method on a Ui block is given by the following formula:

$$VG\left({U}_{i}\right)= SB\left(V1\left(i\right),{U}_{i}\oplus \text{V}3(\text{i})\right)$$
(9)

4.3.2 The Improved Hill method

This step proposes a classic Hill cipher improvement using an affine transformation composed of an invertible matrix of size (3,3) and a translation vector.

The matrices (Mi) are invertible dynamic matrices of size (3, 3) constructed by the following formula:

$${M}_{i}=\left(\begin{array}{ccc}1& V1(i)& V3(i)\\ V2(i)& 1+V1\left(i\right)*V2(i)& V4(i)\\ 0& 0& 2V5\left(i\right)+1\end{array}\right)$$
(10)

The translation vectors are invertible dynamic vectors of size (1,3) constructed by the following formula:

$${T}_{i}=\left(\begin{array}{c}V5(i)\\ V4(i)\\ V3(i)\end{array}\right)$$
(11)

4.3.3 Combined Hill-Vigenère method

Our proposed method combines improved Hill's method and improved Vigenère’s method using the following formula:

$$\begin{array}{c}{W}_{i}=\left\{\begin{array}{c}\begin{array}{c}{M}_{i}\left(VG\left({U}_{i}\right)\right)\oplus {T}_{i} if CV1\left(i\right)=0\\ VG\left({M}_{i}\left({U}_{i}\right)\oplus {T}_{i} \right) else\end{array}\end{array}\right.\end{array}$$
(12)

4.4 The encryption steps

The encryption process begins by calculating an initialization vector IV, then applying the combined Hill-Vigenère method on the (IV XOR U1) to obtain the vector W1. Then, to apply the combined Hill-Vigenère method on the (W1 XOR U2) to obtain the vector W2. The same steps will be repeated on the other vectors Ui to obtain the vectors Wi that form the encrypted image (see Fig. 4). The algorithm 5 illustrates this step.

Algorithm 5
figure j

Encryption algorithm

Full size image

4.5 The decryption step

The proposed approach is a symmetric technique with broadcast implementation. Therefore, in the decryption step, we apply the following decryption functions:

$$\begin{array}{c}{U}_{i}=\left\{\begin{array}{c}\begin{array}{c}{M}_{i}^{-1}\left({VG}^{-1}\left({W}_{i}\right)\right)\oplus {T}_{i} if CV1\left(i\right)=0\\ {VG}^{-1}\left({M}_{i}^{-1}\left({W}_{i}\right)\oplus {T}_{i}\right) else\end{array}\end{array}\right.\end{array}$$
(13)

  • –M−1 denotes the inverse matrix of the matrix (M). Its expression is given by:

    $${M}_{i}^{-1}=\left(\begin{array}{ccc}\alpha & \beta & \gamma \\ \delta & 1& \theta \\ 0& 0& \rho \end{array}\right)$$

    With:\(\alpha =1+V1\left(i\right)*V2\left(i\right)\),\(\beta =-V1(i)\), \(\gamma =\frac{\left(2V5\left(i\right)+1\right)V3\left(i\right)-V4(i)}{2V5\left(i\right)+1}\),\(\delta =-BL(i)\), \(\theta =\frac{V4\left(i\right)-V2\left(i\right)*V3(i)}{2V5\left(i\right)+1}\) and \(\rho =\frac{1}{2V5\left(i\right)+1}\)

  • –The inverse Vigenère function VG−1 is given by the following formula:

    $${VG}^{-1}\left({W}_{i}\right)= {SB}^{-1}\left(V1\left(i\right),{W}_{i}\oplus \text{V}3(\text{i})\right)$$
    (14)

  • –The inverse substitution matrix SB−1 of size 255 × 255 is given by the following formula:

    $${SB}^{-1}\left(i,SB(i,j)\right)=j$$
    (15)

5 Experimental results

To evaluate our encryption approach presented in this article, various tests are used, namely: visual evaluation, keyspace analysis, Keyspace Sensitivity, statistical attack analysis (entropy, histogram, correlation) and differential attack analysis (UACI, NPCR, avalanche effect). The tests are run on a set of standard images of various sizes, extracted from the USC-SIPI database [47]. For the performance analysis, we will present the simulation of ten images of different sizes (Female (256 × 256), Couple (256 × 256), House (256 × 256), baboon (512 × 512), Lena (512 × 512), Peppers (512 × 512), Splash (512 × 512), Airplane (512 × 512) Stockton (1024 × 1024) and Washington (2250 × 2250)) chosen from the images used to test our method. For comparison with recent publications, we will choose the three most commonly used images in the literature (Baboon, Lena and Peppers).

5.1 Visual evaluation

This test allows us to visualize if the encrypted images can provide information about the original image and if the decrypted images are identical to the original images. Table 1 presents the results obtained for ten images chosen from the test images.

Table 1 Simulation results
Full size table

We notice no resemblance between the encrypted image and the original image. Therefore, an attacker cannot extract any information about the original image from the encrypted image. We also see that the decrypted images are the same as the encrypted images.

5.2 Brute Force attack

5.2.1 Keyspace analysis

Keyspace analysis tests the resistance of an encryption system against brute force attacks. To do this, the cipher must have a keyspace greater than 2100. Our system has a total key space composed of four initial conditions u0, v0, w0, w1 and four control parameters μ0, μ1, μ2, μ3 (reals of 64 bits), which is 28*64 = 2512. Thus, it is large enough, which makes brute force attacks impractical. Table 2 shows total key space comparison results with the current encryption algorithms. It is obvious that our key space has high precision, and it should defend the brute force attack.

Table 2 Keyspace comparison
Full size table

5.2.2 Keyspace sensitivity

Our encryption key has high sensitivity, which means that a slight modification of a single parameter will automatically lead to a significant difference compared to the original image. Figure 5 illustrates this property, ensuring that in the absence of the correct encryption key, the original image cannot be restored. Without possession of the real secret encryption key, the attacker is unable to reconstruct the original image.

Fig. 5
figure 5

Keyspace Sensitivity

Full size image

5.3 Statistical attack analysis

5.3.1 Histogram analysis

A histogram is a graphical representation of the distribution of pixels in an image. To avoid a statistical attack, the encrypted images must have a uniform histogram. Table 3 shows the histograms of the original images and their encrypted images generated by the proposed system.

Table 3 Histograms of the original images and encrypted images
Full size table

The results in Table 3 illustrate that the histograms of the encrypted images are quite uniform and different from those of the original images. This shows that the encrypted image does not provide any information to use a statistical attack.

5.3.2 Correlation analysis

The correlation determines the independence of adjacent pixels. It is defined as follows:

$${C}_{x y}=\frac{\sum_{i=1}^{N}\left({x}_{i}-E(x)\right)\left({y}_{i}-E(y)\right)}{\sqrt{\sum_{i=1}^{N}{\left({x}_{i}-E(x)\right)}^{2}\times \sum_{i=1}^{N}{\left({y}_{i}-E(y)\right)}^{2}}}$$
(16)

where E(x) is the average of the pixel values of the image.

Table 4 illustrates the values of the vertical, horizontal and diagonal correlation of the clear and encrypted images by our system.

Table 4 Correlation coefficients
Full size table

An encrypted image should show no correlation between adjacent pixels. Our method examined several measures of image correlation, all of which are very close to zero. This can protect our method against statistical attacks.

5.3.3 Entropy analysis

Entropy is a measure of randomness used to describe simple image texture. The entropy value is close to 8 for encrypted images. Mathematically, it is defined as:

$$H\left(X\right)= -\sum_{i=0}^{n-1}Pr({X}_{i})\times {log}_{2}\left(\frac{1}{Pr({X}_{i})}\right)$$
(17)

where X denotes the test image, Xi is the value of the pixel and Pr(Xi) represents the probability of Xi.

The entropy values of the five images encrypted by our system are shown in Table 5 below.

Table 5 Entropy analysis
Full size table

The data in Table 5 indicate that all the entropy values of encrypted images are close to 8. Therefore, the proposed algorithm can withstand information entropy analysis attacks due to the high randomness of the l encrypted image.

5.4 Differential attack analysis

Differential attack analysis is a technique used to determine how an encrypted image changes when the original image's pixels or key value are slightly altered. Differential attacks are managed by NPCR (number of pixels changed Rate), UACI (Unified Averaged Changed Intensity) constants and AE (avalanche effect).

  • –The NPCR is given as follows:

    $$NPCR=\frac{\sum_{1}^{w}\sum_{1}^{h}{D}_{i j}}{w*h}\times 100 (\%)$$
    (18)

    where H and W are the height and width of the image.

    D(i, j) is given as follows:

    $${D}_{i j}=\left\{\begin{array}{c}1 si {C1}_{i j}\ne {C2}_{i j}\\ 0 sinon\end{array}\right.$$

  • –The UACI is given as follows:

    $$UACI=\frac{1}{w\times h}\frac{\sum_{i j}\left|{C1}_{i j}-{C2}_{i j}\right|}{255}\times 100 (\%)$$
    (19)

    With C1 the encrypted image of the original image and C2 the encrypted image of the modified original image.

  • –The avalanche effect corresponds to the number of bits that have been modified if a single bit of the original image is modified. The mathematical expression of this avalanche effect is given by Eq. 20.

    $$\left(\frac{\sum_{i}bit change}{\sum_{i}bit total}\right)*100$$
    (20)

To calculate the NPCR, UACI and AE values, a pixel is randomly chosen and its value is modified. Then the original image and the modified image are encrypted according to the proposed method. The NPCR, UACI and AE values obtained are presented in Table 6 below.

Table 6 NPCR, UACI and avalanche effect values
Full size table

NPCR, UACI and avalanche effect values are higher than expected values (99.60% for NPCR, 33.40% for UACI and 50% for AE). These results demonstrate how sensitive our system is to small changes in an image; even if there is only a one-bit variation between two images, the decrypted image will be entirely different. This demonstrates that the suggested approach has good performance in defending against differential attacks.

5.5 Peak signal-to-noise ratio (PSNR) analysis

Peak Signal-to-Noise Ratio (PSNR) is a commonly employed metric to quantify the fidelity of a reconstructed or processed image compared to its original version. In the context of image encryption, the plaintext image is considered the reference signal, while the encrypted image represents the degraded or noisy version. PSNR is typically expressed in decibels (dB) due to the logarithmic nature of the scale, allowing for a compact representation of a wide range of signal-to-noise ratios. It is mathematically given as:

$$PSNR=20 {log}_{10}\left(\frac{255}{\sqrt{MSE}}\right)$$
(21)

where MSE is given as follows:

$$MSE= \frac{1}{M\times N}\sum {\left(IO\left(i,j\right)-IC\left(i,j\right)\right)}^{2}$$
(22)

M × N is the image size, IO(i,j) is a pixel of the original image and IC(i,j) is a pixel of the encrypted image.

The PSNR of a good encryption algorithm should be as small as possible to ensure the high security of encryption algorithm. Table 7 lists the PSNR of ten images. It can be seen that the PSNR value of the encryption algorithm proposed in this article is small, indicating that the plain image and encrypted image are very different, so the encryption algorithm has higher security.

Table 7 PSNR values
Full size table

5.6 Encryption time and complexity

The encryption time of a crypto system is a parameter that can affect the effectiveness of an encryption algorithm. The paper simulates the proposed algorithm on an Intel Core I5 CPU 1.7 GHz and 8 GB of RAM. The encryption times of the seven simulation images are shown in the following Table 8. Using the proposed method, image encryption times change in the interval [0.05 2].

Table 8 Encryption time
Full size table

The execution time depends on the hardware used, the algorithm and the size of the image. On the other hand, the complexity depends solely on the algorithm. So, to show the effectiveness of our algorithm in terms of speed, we will calculate its complexity. Suppose the image size is 3 NM. The time consumed by the proposed algorithm can be divided into five parts. The first part concerns the generation of chaotic vectors (algorithm 1). The complexity of the generation algorithm is O(3NM). The second part is the vectorization operation (algorithm 2) which has a complexity of order O(3NM). The time complexities of Algorithms 3 and 4 are O(3NM), respectively. The last part is the encryption process (algorithm 5). The implementation of the broadcast operation is based on the XOR operation between pixels, and the corresponding complexity is O(3NM). In general, the time complexity of the proposed algorithm is O(3NN).

5.7 Comparison

Table 9 shows the comparison of the proposed method with other recently published methods based on values of entropy, NPCR, UACI, and correlation coefficient and using three images.

Table 9 Performance comparisons with the existing approaches
Full size table

It is shown in Table 9 that our technique gives satisfactory results compared to other data in the literature. The entropy of the three images gives values extremely close to the ideal value of 8 and higher than the values obtained by most other methods. For the values of the correlation coefficients according to the three directions, we realize that all the methods have values of the correlation coefficient very close to zero. For the NPCR and UACI, the values of the proposed scheme are very close to the values of NPCR (99.6%) and UACI (33.4%) expected, and they are the highest compared to the other techniques. Accordingly, from Table 8, we can conclude that the suggested method outperforms many recently published studies in terms of correlation entropy values of the NPCR and UACI.

6 Conclusion

In this article, we introduced a new method for image encryption. The method uses three chaotic maps to extract five pseudo-random vectors and two control vectors, which are used to vectorize the original image. Then it proposes two improvements, one on the classical method of Hill and the other on the classical method of Vigenère. Then it uses the combination of these two enhancements with a chaining mechanism to encrypt the images. The proposed method is found to be sensitively dependent on the single image, as shown by the key sensitivity analysis. The size of the key space is large enough to resist brute force attacks. The Histograms of encrypted images show a uniform distribution of pixels. The correlation coefficients of the encrypted images are very close to zero. An information entropy greater than 7.999 is an indication of randomness. The average NPCR (> 99.6), UACI (< 33.4) and AE (> 40) values show that the proposed method can effectively resist differential attacks. From the security analysis, it seems that the perfect original image cannot be recovered by applying known cryptographic attacks. It is therefore secure and applicable in real-time image encryption transmission applications. In future work, we hope to generate very large S-boxes and large Hill matrices and minimize the encryption time in order to be able to encrypt images in real time.