Enhancing the robustness of image watermarking against cropping attacks with dual watermarks

Abstract

In this study, a QR-based digital watermarking scheme that can use color images is proposed. The main purpose of this method is to enhance robustness against cropping attacks. To achieve this aim, each bit of the robust watermark has four copies, which are hidden in different image blocks. The embedding rule is designed based on the sinusoidal function, and the wavelength of the sinusoidal function controls the trade-off between imperceptibility and robustness. The four copies of the watermark bit may be inconsistent if the watermarked image undergoes cropping attacks. Therefore, after the four copies of the watermark bit are extracted, the actual value of the watermark bit is judged based on the result of the tampering detection. Experimental results indicate that improving robustness with the help of tampering detection results effectively assists in watermark extraction. In addition, the method is superior to other methods in terms of invisibility, robustness, and embedding payload.

1 Introduction

The transmission of digital images through the Internet has become more convenient and frequent. However, digital messages are susceptible to malicious modification or illegal copying. Malicious modifications may cause misinterpretation if a message receiver cannot verify an image’s integrity. A false medical image may cause an erroneous diagnosis, or a falsified personal photo may damage a person’s reputation [1]. Illegal copying may cause copyright disputes among individuals. Therefore, the integrity verification and copyright protection of digital images are crucial research concerns.

Digital watermarking is a common method for protecting copyright or verifying the integrity of digital images [2]. The process of digital watermarking comprises two stages, namely watermark embedding and watermark extraction. For copyright protection, a watermark, which is a logo or binary stream, is embedded into a host image according to an embedding rule in the first stage. In the second stage, the watermark is extracted and used to examining whether copyright piracy has taken place. Because a person may attempt to alter the watermarked image to erase the watermark, the watermark must be robust against attacks. If the watermarked image undergoes modifications, the extracted watermark may be different from the original watermark but still recognizable. To detect tampering, the watermark representing an image feature is embedded into the host image in the first stage. In the second stage, the watermark is extracted and compared with the image feature to verify integrity and determine the tampered area. Because malicious alterations may be small to avoid being easily noticed, the watermark must be sensitive to any alteration. Regardless of the use of a robust or fragile watermarking scheme, the original host image or original watermark should not be used when extracting the watermark; otherwise, the practicability of the method is reduced. A watermarking scheme without the host image to extract the watermark is called a blind scheme.

The watermark operates in a spatial [3,4,5] or frequency domain [6,7,8]. In a spatial domain scheme, pixel values of the host image are modified to embed the watermark, whereas in a frequency domain scheme, frequency coefficients of the host image are modified to embed the watermark. The most common methods used to transform the host image from the spatial domain to the frequency domain are discrete Fourier transform (DFT) and discrete wavelet transform (DWT). Recently, some studies have investigated another domain that is derived from matrix factorization by using methods such as singular value decomposition (SVD), QR decomposition, and LU decomposition. A matrix factorization method is used to decompose the host image into matrices, and the watermark is embedded into one of these matrices. Watermarking schemes for copyright protection generally embed the watermark into the frequency or matrix domain because these domains are more robust against attacks than the spatial domain. By contrast, the watermarking scheme for tampering detection generally embeds the watermark into the spatial domain, particularly in the least significant bit planes.

Robustness can be enhanced by increasing the energy of the watermark, such as embedding the watermark in low-frequency bands or high-order bits of the image. Such domains generally remain invariant if the watermarked image undergoes image processing, such as contrast adjusting, brightening, darkening, and JPEG compression. However, the trade-off between the robustness and imperceptibility must be considered. Moreover, if the watermarked image undergoes cropping attacks, some parts of the watermark may be removed. Therefore, the extracted watermark may be unrecognizable or insufficient for validating copyright. Robustness against cropping attacks is crucial because most malicious attacks are cropping attacks. A cropping attack removes a part of the image and replaces it with white or black color. However, an attacker may replace the cropped part with a meaningful object to disguise malicious behavior. The replacement of the cropped part with a black-and-white or meaningful object generates the same underlying manipulation; that is, some pixels of the image are changed. Cropping attacks can be prevented using two methods, namely the uniform distribution of the watermark into the image [9] or embedding multiple copies of the watermark [10]. Creating multiple backups is a favorable strategy. When a watermarked image is cropped, some copies may be removed, and some may survive. Therefore, the probability of successfully extracting the watermark becomes higher. However, this strategy has one notable drawback. When the watermarked image is cropped, each copy of the same watermark bit may be inconsistent. Because the original watermark image is not available, determining which copy is the authentic one is difficult. In light of increasing the probability of prediction, the watermark bit can be determined according to a majority copy or by a high-priority copy.

As we have mentioned above, cropping attacks are usually malicious; therefore, robustness against cropping attacks is more important than that against common image processing. In this study, a digital image watermarking scheme with dual watermarks was proposed to enhance robustness against cropping attacks. Robust and fragile watermarks were simultaneously embedded into the image. The robust watermark operates in an R matrix of the image decomposed using QR factorization. Each watermark bit is embedded four times, and the four copies are effectively distributed in the host image. To avoid the overflow problem caused by watermark embedding, the allowed ranges of the elements of the R matrix were analyzed. Therefore, it is ensured that the pixel value of the image is within the legal range when the watermarked R matrix is multiplied by the original Q matrix to restore the image. When the watermark bit is being extracted, determining appropriate copies for extraction is essential if all copies are not the same. The prominent feature of the proposed scheme involves using the fragile watermark to determine unaltered carriers of the copies and the extracted watermark bit according to the copies obtained from these carriers. Through this approach, the appropriate rate of the recovered watermark bits and robustness against cropping attacks can be increased. For security reasons, the keys used in this study were encrypted using asymmetric encryption before transmitting them to a receiver.

The remainder of this paper is organized as follows: In Section 2, some methods for robust watermarking using matrix decomposition were reviewed, along with studies related to dual digital watermarking schemes. In Section 3, the proposed scheme is described in detail. Section 4 presents experimental results and analysis. Finally, conclusions are provided in Section 5.

2 Related studies

Until now, three matrix decomposition methods have been used in watermarking schemes: QR, SVD, and LU decomposition. These matrix decomposition methods factorize a matrix into several matrices, and the original matrix can be returned by multiplying these factorized matrices. These three matrix decompositions, without explaining the mathematical theories behind them, can be described as follows: LU decomposition factorizes a u × u matrix A into a u × u lower triangular matrix L and a u × u upper triangular matrix U, with A = L × U. QR decomposition factorizes a u × v matrix A into a u × u orthogonal matrix Q and a u × v upper triangular matrix R, with A = Q × R. SVD decomposition factorizes a u × v matrix A into a u × u orthogonal matrix U, a u × v diagonal matrix Σ, and a v × v orthogonal matrix V, with A = U × Σ × V^T.

Wang et al. [11] proposed an LU-based watermarking scheme. In this scheme, both the host and watermark images are in full color. The host image is transformed into the frequency domain through 1-level DWT. Subsequently, low–high (LH) and high–low (HL) bands are divided into 4 × 4 blocks, and each block is factorized through LU decomposition. The watermark bits are embedded into the element in the upper right corner of each matrix U. The experimental results indicate that robustness against cropping attacks reduced with an increased crop area. Moreover, Su et al. [12] proposed a blind color image watermarking scheme based on LU decomposition. The host image is divided into 4 × 4 blocks, and each block is factorized into matrices L and U using LU decomposition. The watermark bit is embedded blockwise by adjusting the relationship between the sizes of two elements of matrix L. The embedding capacity of these two methods is 0.3, which is not particularly satisfactory. Liu et al. [13] transformed the host image into the frequency domain through 1-level DWT, but the working band was low–low (LL). The LL band is factorized through SVD decomposition, and the watermark is embedded into matrix Σ. The embedding capacity of the scheme is considerably satisfactory, but the original image is required to assist in extracting the watermark.

QR decomposition is frequently used in image processing [14] and digital watermarking schemes [9, 15]. QR decomposition is more accurate than LU decomposition for least square problems. Furthermore, LU factorization can only be applied to square matrices when QR decomposition can be applied to rectangular and square matrices. QR decomposition has lower computational complexity and a less severe false positive detection problem than SVD [16, 17]. A QR-decomposed matrix has a favorable value for robust watermarking. If columns of matrix A are correlated with each other, the absolute values of the elements in the first row of the R matrix are probably higher than those of the other rows. Because the adjacent pixels of the image are generally highly correlated, the first row elements of the R matrix exhibit favorable properties related to robustness and hiding capacity [15, 17]. Naderahmadian and Hosseini-Khayat [18] proposed a rapid watermarking scheme based on QR decomposition in the wavelet domain. The host image is a grayscale image and the watermark image is binary. However, the robustness of this scheme against cropping attacks is not satisfactory. Su et al. [15] embedded the watermark bit into the element in the upper right corner of each R matrix because they believed that only modifying this element has the lowest impact on the original image. Su et al. [9] and Su et al. [16] observed that the values of the second row first column element and third row first column element of the Q matrix are highly correlated. As a result, they designed a watermark embedding rule based on the relationship between these two values. However, the modification of the elements in the Q matrix affects the visual quality of the host image. Therefore, Su et al. [16] performed a compensatory operation on the elements in the first row of the R matrix. Although the design principle of the watermark hiding rules of Su et al. is the same as that of Su et al. [9, 15, 16], compensation for the R matrix is not given special mention in these papers. The host image and watermark of the aforementioned methods of Su et al. are all color images. A common problem is that the hiding capacity is not strong, and robustness against cropping attacks is not appropriate. When the cropping ratio of the image is 25%, the method of Su et al. [9] can ensure 90% similarity between the extracted watermark and original watermark. However, when the cropping ratio reaches 50%, none of these methods achieves a similarity of more than 90%.

In terms of the merit of QR decomposition for robust watermarking schemes, a QR-based robust watermarking scheme was proposed in this study, and the working domain was the R matrix. Changing the values of the decomposition matrix elements must be cautious because the restored pixel values may be out of range. Therefore, most QR-based digital watermarking schemes cannot completely utilize the elements of the R matrix. Consequently, such methods may have a low data-hiding capacity. Therefore, this study proposed a method with a modifiable range of element values of the R matrix. All elements of the first row of the R matrix were used to embed the watermark. Considering the necessity of robustness against cropping attacks, this study proposed two strategies corresponding to cropping attacks. The first strategy is to dispersedly embed multiple copies of the robust watermark into the image. The second strategy is to simultaneously embed the fragile watermark and use it to enhance the extracted robust watermark. Limited studies have proposed schemes to embed both robust and fragile watermarks. However, these studies have not proposed the concept of using the fragile watermark to strengthen robustness, and these methods have not been strengthened specifically for cropping attacks. Schlauweg et al. [19] proposed a semifragile watermarking scheme using a quantization index modulation technique. They later proposed a digital watermarking scheme with dual watermarks. The robust watermark was embedded according to the geometric structure of objects to attain favorable robustness against lossy compression [20]. The fragile watermark (i.e., the image feature) was embedded through their previous work [19]. Furthermore, Shivani et al. [21] proposed a dual watermarking scheme where the robust watermark was embedded in DWT coefficients, and the image features were embedded in the first least significant bitplane. In their experimental results, Shivani et al. validated robustness against the image processing of histogram equalization, Gaussian, and JPEG compression. However, Schlauweg et al. [20] and Shivani et al. [21] did not focus on robustness against cropping attacks.

3 Proposed scheme

In the proposed scheme, the robust and fragile watermarks are embedded into the host image. Therefore, the proposed scheme has two functions: tampering detection and copyright protection. During the embedding process, the robust watermark was repeatedly embedded to enhance its robustness. Because the watermarked image may be tampered with, all copies may be different from each other. The utilization of the fragile watermark to assist in determining an appropriate copy is the most distinguishing feature of the proposed method. This feature can further improve robustness during the extraction process. The proposed scheme was first employed for a grayscale host image and binary watermark image. The extension of the proposed method to color host and watermark images was later studied.

3.1 Embedding process

Figure 1 presents the embedding process. In Fig. 1, the host image H is a grayscale image of size M × N (M rows and N columns), and the robust watermark image RW is a binary image of size m × n, where m = ⌊M/4⌋ and n = ⌊N/4⌋. The host image H is factorized through QR decomposition, and the robust watermark RW is embedded in the R matrix of H. The R′ matrix with RW is composed with the Q matrix to return to pixel values of the watermarked image H′. The feature FW of H′ is then extracted and embedded in the least significant bitplane of H′ to generate the final watermarked image H″.

Fig. 1

Illustration of the embedding process

3.1.1 Robust watermark embedding

The following describes the hiding rules of a single watermark bit and the distribution of all watermark bits in the entire image.

Initially, H is divided into 4 × 4 nonoverlapping blocks, each of which is factorized into the Q and R matrices through QR decomposition. The watermark is hidden in the first four elements of each R matrix. “Hidden” refers to changing the element value such that the state of the element value can represent the watermark bit. In this study, a sine function was used to present the state of the element values. For any element value x₀, watermark bit 1 and bit 0 were represented using f(x₀) ≥ 0 and f(x₀) < 0, respectively, and function f is defined as follows:

$$ f(x)=\mathit{\sin}\left(x.K.\frac{\pi }{180}\right) $$

(1)

where K ≥ 0 is a real number. The wavelength λ of the function f is (360/K), and thus, parameter K is a wavelength controller.

When hiding the watermark bit, the state of the element value may be inconsistent with it. When this happens, the element value must be adjusted such that its state conforms to the watermark bit to be hidden. Let w, x₀, and x′ be the watermark bit, element used to embed w, and adjusted value of x₀, respectively. If w = 0 and x₀ is in the negative half cycle, or if w = 1 and x₀ is in the positive half cycle, then the value of x′ must be the midpoint of the same half cycle. If w = 1 and x₀ is in the negative half cycle, or if w = 0 and x₀ is in the positive half cycle, then the value of x′ has two candidates: the midpoint x_l of the left adjacent half cycle and the midpoint x_r of the right adjacent half cycle. Let d_l and d_r be the relative distance between x₀ and x_l and between x₀ and x_r, respectively. If d_r ≤ d_l, x′ is set to x_r; otherwise, x′ is set to x_l. In summary, the rules for element value adjustment are as follows:

• Case 1:w = 1 and f(x₀) ≥ 0

$$ {x}^{\prime }={x}_0-r+\frac{\uplambda}{4} $$

(2)

• Case 2:w = 0 and f(x₀) < 0

$$ {x}^{\prime }={x}_0-r+\frac{3\uplambda}{4} $$

(3)

• Case 3:w = 1 and f(x₀) < 0

$$ {x}^{\prime }={x}_0-r+\frac{\uplambda}{4}+\uplambda .\left\lfloor \frac{4r}{3\uplambda}\right\rfloor $$

(4)

• Case 4:w = 0 and f(x₀) ≥ 0

$$ {x}^{\prime }={x}_0-r-\frac{\uplambda}{4}-\uplambda .\left\lfloor \frac{4r}{\uplambda}\right\rfloor $$

(5)

The symbol r in the aforementioned rules is the remainder of x divided by λ and is defined as follows:

$$ r=x-\uplambda .\left\lfloor \frac{x}{\uplambda}\right\rfloor $$

(6)

The distribution of watermark bits in the image was then studied. As mentioned previously, image H is divided into 4 × 4 nonoverlapping blocks. Each image block A can be viewed as a 4 × 4 matrix and is factorized into the Q and R matrix through QR decomposition. For security reasons, all image blocks are reordered according to a pseudo-random number generator before the watermark is embedded. Let R_i and w_i represent the R matrix of the rearranged ith image block and the ith watermark bit, where i = 0..(q − 1) and q = ⌊M/4⌋ × ⌊N/4⌋. To improve robustness, each watermark bit is duplicated four times. The jth copy of w_i is embedded into the element R_{(i + j)%q}[0, j], where j = 0..3. To elucidate the watermark distribution rule of the proposed scheme, Fig. 2 is presented as an example. Let five watermark bits (w₀ to w₄) be embedded into five R matrices (R₀ to R₄). Figure 2 indicates that four copies of w₀ and w₁ are embedded into R₀[0, 0], R₁[0, 1], R₂[0, 2], and R₃[0, 3] and into R₁[0, 0], R₂[0, 1], R₃[0, 2], and R₄[0, 3], respectively. The distribution of the four copies of the other three watermark bits, w₂, w₃, and w₄, are evident in Fig. 2.

Fig. 2

Distribution of the robust watermark

After modifying the value of the R matrix, it is multiplied by the Q matrix and is returned to the image block. As mentioned previously, the returned value may exceed the legal range of grayscale image pixel values, which is 0–255. Therefore, it is necessary to confirm that the adjusted value x′ does not cause the restored pixel value to exceed the allowable range. If x′ is not between the upper and lower bounds, the value must be adjusted to the closest boundary before returning to the pixel value. For any element value x₀ ∈ R[0, j], the adjusted value x′ should be within the lower and upper bounds as follows:

$$ lower\_ bound=\underset{i\in \left\{0,1,2,3\right\}}{\max}\left( LB\left[i\right]\right) $$

(7)

and

$$ upper\_ bound=\underset{i\in \left\{0,1,2,3\right\}}{\max}\left( UB\left[i\right]\right) $$

(8)

where

$$ LB\left[i\right]=\left\{\begin{array}{cc}-\infty & if\ Q\left[i,0\right]=0,\\ {}\mathit{\min}\left(\left(R\left[0,j\right]-\frac{A\left[i,j\right]}{Q\left[i,0\right]}\right),\left(R\left[0,j\right]+\Big(255-\frac{A\left[i,j\right]}{Q\left[i,0\right]}\right)\right)& otherwise.\end{array}\right. $$

(9)

and

$$ UB\left[i\right]=\left\{\begin{array}{cc}\infty & if\ Q\left[i,0\right]=0,\\ {}\mathit{\max}\left(\left(R\left[0,j\right]-\frac{A\left[i,j\right]}{Q\left[i,0\right]}\right),\left(R\left[0,j\right]+\Big(255-\frac{A\left[i,j\right]}{Q\left[i,0\right]}\right)\right)& otherwise.\end{array}\right. $$

(10)

3.1.2 Fragile watermark embedding

Similar to the robust watermark embedding stage, image H′ is first divided into 4 × 4 nonoverlapping blocks. Feature extraction and embedding are performed blockwise. In this section, the hidden approach of fragile watermarking for a single 4 × 4 block is explained.

Let (x, y) and p_i denote the coordinate of the block in H′ and the pixel of the image block, respectively, where i = 0..15. A 256-bit message digest MD of the image block was first generated as follows:

$$ MD=\mathrm{SHA}256\left(\left.x\right\Vert \left.y\right\Vert \left. SK\right\Vert {p}_0\gg 1\left\Vert {p}_1\gg 1\left\Vert \dots \left\Vert {p}_{15}\gg 1\right.\right.\right.\right), $$

(11)

where SHA256 is a type of Secure Hash Algorithm 2, “>>” is a bitwise right shift operator, “||” is a concatenation operator, and SK is a secret key. MD is then split into 16-bit strings S₀, S₁, …, S₁₅, and each string is defined as follows:

$$ {S}_i={\left({b}_{i\times 16},{b}_{i\times 16+1},{b}_{i\times 16+2},\dots, {b}_{i\times 16+15}\right)}_2, $$

(12)

where b_i ∈ {0,1} and i =0..15. The feature of the block, (i.e., the fragile watermark FW) is generated as follows:

$$ FW={S}_0\oplus {S}_1\oplus \dots \oplus {S}_{15}={\left({w}_0,{w}_1,{w}_2,\dots, {w}_{15}\right)}_2, $$

(13)

where ‘⊕’ is a bitwise XOR operator and w_i ∈ {0,1}. Finally, each pixel p_i of the image block is modified to p_i′ according to the following formula:

$$ {p}_i^{\prime }=\left(\left({p}_i\gg 1\right)\ll 1\right)+{w}_i, $$

(14)

where i = 0..15 and “<<” is a bitwise left shift operator. When all the blocks are processed according to the aforementioned method, the final watermarked image H″ can be obtained.

The entire watermark embedding algorithm is as follows:

3.1.3 Extension for color images

If the host image and robust watermark are both color images, we propose a method of converting image formats so that the method proposed in this study can process color images. Figure 3 illustrates the conversion process. If the size of the color host image is M × N (M rows and N columns), the size of the color watermark is m × n, where m = ⌊⌊M/4⌋/3⌋ and n = ⌊⌊N/4⌋/3⌋. Both the color host image and watermark image are decomposed into three single-tone images in a red–green–blue (RGB) model. The pixel of the decomposed image is an 8-bit value, which represents the intensity of the tone. Therefore, these decomposed images can be treated as grayscale images. The three grayscale watermark images are binarized. Fuller explanation of binarization will be presented in the next paragraph. Let us return to our main subject. After the color host image is decomposed and the watermark is binarized, the scenario is simplified into embedding a binary watermark into a grayscale host image. Therefore, these three binary watermarks can be embedded into the three monotone images of the host image by using the method presented in Section 3.1.1. The three monotone watermarked host images are separately watermarked with an authentication message according to the method presented in Section 3.1.2. Finally, the three monotone watermarked images are combined based on the RGB model to restore the color watermarked image.

Fig. 3

The conversion process of watermark embedding with color images

We now return to the method of binarization which we postponed in the above paragraph. For the grayscale watermark, each pixel is converted into an 8-bit binary value, and each binary value is then transformed into a 3 × 3 binary block. For example, assume that the pixel value of the top left corner of the grayscale watermark is 74, whose binary value is (01001010)₂. The content of the corresponding 3 × 3 binary block is then represented as follows:

$$ \left[\begin{array}{ccc}0& 1& 0\\ {}0& 1& 0\\ {}1& 0& 1\end{array}\right] $$

When each pixel is sequentially transformed into a 3 × 3 binary block, a binary image can be obtained. This binary image is not a normal image. The position of the bottom right corner of the 3 × 3 block is not used. The value of this position is fixed to 1. For future research, this location will be used to hide other information.

3.2 Extracting process

Figure 4 illustrates the process of watermark extraction. The fragile watermark is first extracted from the test image T to detect tampering area. The output of tampering detection is error map E, which is a Boolean matrix that records the falsified blocks. After the test image is decomposed through QR factorization, the robust watermark can be extracted from the R matrix with the error map E as an input. The robust watermark can help us verify the copyright of the image. The method of tampering detection and copyright verification are thoroughly explained as below.

Fig. 4

Illustration of the extraction process

3.2.1 Tampering detection

Let T denote the test image of M × N pixels and E denote the error map, which is a Boolean matrix of the size of ⌊M/4⌋ × ⌊N/4⌋. The test image T is first divided into 4 × 4 nonoverlapping blocks. For each block, the feature is generated in the same way as shown in section 3.1.2 and then compared with the fragile watermark extracted from the least significant bitplane of the block. If the comparison result indicates that they are inconsistent, this block is considered to be tampered with, and the Boolean value ‘true’ is recorded in error map E. Otherwise, record the Boolean value ‘false’ in error map E. The detailed steps are listed as follows, and the output is error map E, which can display the falsified area.

3.2.2 Copyright verification

In the previous section, each watermark bit has four copies hidden in the first row of the R matrix; thus, the image is first factorized through QR decomposition to retrieve these copies. For any element value x₀ of the first row of R, the embedded bit w is decoded as follows:

$$ w=\left\{\begin{array}{cc}1,& f\left({x}_0\right)\ge 0\\ {}0,& otherwise.\end{array}\right. $$

(15)

According to the robust watermark embedding process mentioned in Section 3.1.1, multiple copies of the watermark bit are obtained from different blocks. If each copy is the same, the value of the watermark bit can be determined. However, the image may be altered, which results in inconsistencies in each copy of the watermark. As a result, determining whether the embedded watermark bit is 0 or 1 is difficult. A feasible solution is to adopt a majority decision, that is to say that the watermark bit value depends on the most frequently occurring value of the four copies. For example, if the value of most copies is 1, the value of the watermark bit is 1, and vice versa. However, adopting a majority decision has several limitations. If the number of copies is an even number, the majority decision may not be determined in some cases. In addition, the majority is not necessarily correct if most of the blocks where the copy is located are destroyed. In this study, the result of tampering detection (i.e., error map E) is used to assist in determining the watermark bit. Before deciding the value of the watermark bit, the result of tampering detection is used to judge whether the blocks are altered. If all copies are consistent and are hosted in untampered blocks, then these copies can accurately determine the value of the watermark bit. If these copies are inconsistent, the value of the watermark bit is determined using the copies extracted from the blocks that have not been altered. If none of the four copies are from the untampered blocks, then the watermark bits are determined using these four copies according to the majority voting principle. It is reasonable to assume that copies extracted from untampered blocks are consistent. However, if the falsification of blocks is misjudged, that is, those blocks have been tampered with, but the opposite was concluded in the process, then these copies may be inconsistent. When such situations occur, this study also adopts majority voting principle to determine the value of the watermark bit. The aforementioned misjudgment is termed a false negative. The false negative may not be serious because the probability of its occurrence is low, and the probability is analyzed in a subsequent section. The robust watermark extraction algorithm is as follows:

3.2.3 Extension for color images

When the test image is a color image, it is decomposed into three monotone images based on the RGB model. Each of the three images uses the method proposed in Section 3.2.1 to generate their respective error maps. Then, the method in Section 3.2.2 is used, with respective error maps, to extract the robust watermark from the three monotone images. The three extracted watermarks of size (3 m × 3n) are binary and meaningless images. Each binary watermark is then restored to a grayscale image by reversing the binarization mentioned in Section 3.1.3. Finally, the three grayscale watermark images based on the RGB model are combined to obtain the color watermark image.

4 Experimental results and analysis

This study primarily focuses on color images. The size of the watermark image depends on that of the host image. According to Section 3.1.3, if the height and width of the host image are M and N, respectively, the height and width of the watermark image are ⌊⌊M/4⌋/3⌋ and ⌊⌊N/4⌋/3⌋, respectively. In our experiments, the host image is a color image with 512 × 512 pixels. Therefore, the size of the watermark is 42 × 42 pixels. The wavelength controller K is 31.

The most crucial requirements for robust watermarking schemes are imperceptibility and robustness. Imperceptibility means that the watermarked image must be visually indistinguishable from the original host image, and robustness denotes that the extracted watermark should exhibit some similarity with the original watermark. Generally, robust watermarking schemes calculate the peak signal to noise ratio (PSNR) of the watermarked image against the host image to measure imperceptibility. The higher the PSNR is, the better the imperceptibility is [22]. Robustness is generally measured through normalized correlation (NC) of the extracted watermark against the original one. The NC value ranges from 0 to 1, where 1 and 0 indicate that the two images are exactly identical and completely different, respectively [23]. As regards the tampering detection performance, the following performance evaluation measurements are defined. Let FN, FP, TN, and TP be the number of false negative pixels, false positive pixels, true negative pixels, and true positive pixels, respectively. The tampering ratio (TR), false negative rate (FNR), and false positive rate (FPR) used to measure tampering detection accuracy are respectively defined as follows [23, 24]:

$$ TR=\frac{FN+ TP}{M\times N}. $$

(16)

$$ FNR=\frac{FN}{FN+ TP}. $$

(17)

$$ FPR=\frac{FP}{FP+ TN}. $$

(18)

4.1 Experimental results

In this study, 64 color images from CVG-UGR image database were used as the host images [25]. The average PSNR value of 64 images after embedding watermarks is 42.9153. Various attacks with different cropping ratios were conducted on the 64 images, and the average experimental data are listed in Table 1. In Table 1, CR denotes the cropping ratio of the watermarked image. The PSNR indicates the difference between the cropped watermarked image and original watermarked image. The NC represents the similarity between the original watermark and the watermark extracted from the cropped image. The FNR and FPR represent the results of the tampering detection of the watermarked image. Columns 2 to 5 represent the average of the PSNR, NC, FNR, and FPR for 64 images corresponding to different cropping ratios. Figure 5 shows a line chart with NC values corresponding to the cropping ratio. Table 1 and Fig. 5 indicate that when the cropping ratio reaches 70%, the extracted watermark is still 90% similar to the original watermark. When the cropping ratio is up to 80%, the similarity between the extracted watermark and original watermark can still reach 80% or higher. These data show that the method proposed in this study exhibits satisfactory robustness against cropping attacks. To save space, only the results of the attacks on Bamboo image are presented here. Figure 6a and b are the host image and watermark image, respectively. The result of the robust and fragile watermark embedding processes is shown in Fig. 6c, and the PSNR value is 42.9153. When the PSNR value is higher than 30, the human eye cannot distinguish differences from the original image. Figure 6d is the watermark extracted from Fig. 6c, and the NC value is 1.0, which indicates that the extracted watermark is exactly the same as the original watermark. Figure 7 shows the simulated cropping attacks with various cropping ratios. Figure 8 shows the watermarks extracted from the respective attacked images of Fig. 7 and their NC values. When the cropping ratio is as high as 70%, the extracted watermark still exhibits a 90% similarity to the original watermark. In general, people who steal images may modify the image to avoid copyright disputes. The modification is a type of collage attack, which is to add other objects to the image. Figure 9a is the result of a simulated collage attack on Fig. 6a, and the object added on Fig. 6a is an image made by us. Figure 9b is the authentication result based on the error map, and Fig. 9c is the extracted robust watermark. The NC value shows that the proposed method exhibits satisfactory robustness against collage attacks.

Table 1 Experimental result for 64 color test images under cropping attacks

Fig. 5

Line chart of CR versus NC

Fig. 6

The experimental results. a The host image b The watermark image c The watermarked image (PSNR = 42.9153) d The extracted watermark (NC = 1.0)

Fig. 7

Cropping attacks to the watermarked image with different cropping ratios

Fig. 8

Extracted watermark images under various cropping attacks

Fig. 9

Collage attack. a Collage attack (TR = 0.25, PSNR = 15.0195) b Authentication result (FNR = 0.0, FPR = 0.0340) c Extracted watermark (NC = 0.9994)

For fairness, the comparisons in the Figs. 10 and 11 are based on the experimental results of Lena and Airplane as the host images. Figure 10 shows the NC values obtained using the proposed method and other methods at cropping ratios of 25% and 50%. For the cropping ratio of 25%, our proposed method and those proposed by Su et al. [9, 16] have an NC value of more than 90%. When the cropping ratio is expanded to 50%, only the NC value obtained using this method can reach more than 90%, and the NC value of Su et al. [9] is less than 10% of the NC value of the proposed method. Regardless of whether the cropping ratio is 25% or 50%, the NC value of the proposed method is close to 1, which indicates that the extracted watermark is highly similar to the original watermark. Figure 11 compares the PSNR value of the proposed method and that of other methods and indicates that the invisibility of the proposed method is superior to that of other methods. It seems reasonable to conclude that the proposed method does not lead to unsatisfactory imperceptibility because of improvements in robustness.

Fig. 10

Comparison of the robustness against cropping attacks

Fig. 11

Comparison of the imperceptibility of methods

4.2 Analysis and discussions

4.2.1 Trade-off of imperceptibility and robustness

The trade-off between imperceptibility and robustness of a robust watermarking scheme is generally controlled by a parameter, and in this study, the wavelength controller K plays this role. Researchers usually pre-test different parameter values to select appropriate one. This study also performed pretests. The experiments were performed with K values from 1 to 150, and TRs from 0% to 95% to analyze the effect of K values on PSNR and NC values. Because of the limited length of the article, only some of the experimental data are presented in Table 2, and complete experimental data is drawn as a line chart shown as Fig. 12. Figure 12a shows the effect of different K values on imperceptibility. A larger K value can increase the PSNR value of the watermarked image, which indicates improved imperceptibility. However, as Fig. 12b indicates, larger K values may cause the extracted watermark not to be complete. The incompleteness results from that the embedded fragile watermark may destroy the previously embedded robust watermark. Observing the NC value of a 0% cropping ratio column of Table 2, we can see that the watermark can be completely extracted without destruction if the K value is less than 33. Figures 12c–f are NC values corresponding to different K values under various cropping ratios. When the K value is higher than 40, the NC value has a relatively obvious downward trend. It follows from what has been observed that the K value should not exceed 33 to maintain the integrity of the watermark and should not less than 25 to ensure the PSNR value to be greater than 40. View in this light, this study set the value of K at 31.

Table 2 Pretest results for parameter K

Fig. 12

Pretest results for parameter K

4.2.2 Embedding payload

In theory, the embedding payload of the proposed scheme can be derived as follows:

$$ \frac{\frac{\raisebox{1ex}{$M$}\!\left/ \!\raisebox{-1ex}{$4$}\right.}{3}\times \frac{\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$4$}\right.}{3}\times 24}{M\times N\times 3}=\frac{1}{18}\cong 0.05556. $$

(19)

Figure 13 shows the comparison of the embedding payload using different methods. This figure indicates that the embedding payload of the proposed method is 1.78 times higher than that of the other methods.

Fig. 13

Comparison of embedding payload of different methods

4.2.3 Utility of tampering detection

Fragile watermarks are generally used for tampering detection. In this study, fragile watermarking has another role, which is to enhance the ability of the robust watermark against cropping attacks. To show whether the fragile watermark actually functions, Fig. 14 shows robustness against cropping attacks when robust watermarks are extracted with and without the assistance of fragile watermark. Extracting the robust watermark without the fragile watermark means that the value of the bit is determined only according to the mode of all the copies of the watermark bit and without the assistance of the error map. For the attack of various cropping ratios, the NC value of the robust watermark without the fragile watermark and that with the fragile watermark is drawn as a line graph (Fig. 14). This figure indicates that when an assistant of the fragile watermark is available, the robustness of the extracted watermark improves. In particular, when the cropping ratio is large, the fragile watermark can be more effective. It is one of the crucial contributions of the proposed method in enhancing robustness against cropping attacks with the results of tampering detection. The results presented in Fig. 14 confirm the merit of this study.

Fig. 14

Effect of fragile watermarking on improving robustness

According to Section 3.2, if the blocks where the four copies of the watermark bit are located are evaluated to not have been tampered with, these copies should be consistent. If that evaluation is a false negative, then these copies may be inconsistent. However, the probability of a false negative is extremely low, which can also be validated from the data in Table 1. Under different cropping ratios, the average FNR values of 64 test images approaches 0. In fact, the theoretical value of the FNR of the proposed method is 2⁻¹⁶.

4.2.4 Time complexity

The embedding procedure consists of the “EmbedRobustWatermark” and “EmbedFragileWatermark” algorithms. In the “EmbedRobustWatermark” algorithm, m × n blocks of the host image are shuffled using the Fisher-Yates shuffle algorithm in time T_shuffle = O(mn). Then, the host image is factorized by QR decomposition in a block-wise manner, and the time complexity is T_qr = O(mns³) since the time complexity for facotrizing a s × s matrix is O(s³). Next, the three-layer for-loops for altering the R matrices takes T_alter = O(mn) since the inner for-loop can be done in O(1) time. Finally, the inverted QR decomposition also takes T_qr time. Therefore, the time complexity for the “EmbedRobustWatermark” algorithm is given by.

$$ {T}_{\mathrm{rwembed}}={T}_{\mathrm{shuffle}}+{T}_{\mathrm{qr}}+{T}_{\mathrm{alter}}+{T}_{\mathrm{qr}}=\mathrm{O}(mn)+\mathrm{O}\left({mns}^3\right)+\mathrm{O}(mn)+\mathrm{O}\left({mns}^3\right)=O\left({mns}^3\right). $$

Note that the matrix is with fixed 4-by-4 dimensions, so effective T_rwembed will be O(mn). Regarding the “EmbedFragileWatermark” algorithm, the time complexity is T_fwembed = O(mn) because performing SHA-256 and hiding the authentication message for a 4 × 4 block can be done in O(1) time. In conclusion, the time complexity for the embedding procedure is given by.

$$ {T}_{\mathrm{embed}}={T}_{\mathrm{rwembed}}+{T}_{\mathrm{fwembed}}=O(mn)+O(mn)=O(mn). $$

Because the watermark extraction is the reversion operation of the watermark embedding, thus the time complexity of watermark extraction is also O(mn).

4.2.5 Robustness against common image processing

When the watermarked image is altered by malicious attacks, some copies of the watermark bit may survive because malicious attacks usually do not change the entire image. However, if the image is modified by some common image processing, all pixel values will be changed, and thus, the elements of the R matrix are all modified. Also, the four copies of the watermark may have been changed. One possible way to cope with this situation is to add an extra step to embed each watermark bit again. At this extra step, each watermark bit is not carried by a single element but carried by the relationship between two elements of the R matrix. Note that the extra step is taken after all copies of the watermark are embedded into the R matrices and before the pixel values are recovered. Let a and b denote the R_i[0,0] and R_i[0,1] elements of the R matrix, respectively. For the watermark bit w_i, a is set to a′ as follows:

$$ a^{\prime }=b+\left[\left({w}_i-0.5\right)\times 2\right]\times \left[1-\left(f(a)\oplus f(b)\right)\times 0.5\right]\times \lambda, $$

(20)

where ‘⊕’ is a bitwise XOR operator and f(·) is the sine function as Eq. 1. It is still necessary to make adjustment if the modified value a′ is out of bounds according to Eqs. 7 and 8. At the stage of watermark extraction, the result of tampering detection (i.e., error map E) is used to assist in determining the watermark bit. The watermark bit value is determined by the copies embedded in untampered blocks as described in section 3.2.2. However, when none of the four copies are from the untampered blocks, the watermark bit w_i is determined in the following manner.

$$ {w}_i=\left\{\begin{array}{cc}1,& {R}_i\left[0,0\right]\ge {R}_i\left[0,1\right]\\ {}0,& otherwise.\end{array}\right. $$

(21)

To examine the performance of the extra step mentioned above, we also use the 64 color images from CVG-UGR image database as the host images and Fig. 6b as the watermark. We simulated different brightness adjustments to the watermarked image, and Fig. 15a is the line chart with average NC values of the extracted watermark corresponding to various brightness level. This diagram shows that the NC values against different brightness adjustments are all above 0.95, which indicates that the quality of the extracted watermark is satisfactory. Figure 15b is the line chart with average NC values of the extracted watermark corresponding to various cropping ratios. This diagram helps to prove that the proposed extra embedding step does not reduce the ability to resist cropping attacks. The average NC value can still reach 90% or higher when the cropping ratio is up to 75%. However, the imperceptibility is not as good as the previous experiment in section 4.1. The average PSNR value of 64 watermarked images is 32.3211. Therefore, how to balance the robustness against image processing and imperceptibility of the watermarked image is our future research topic.

Fig. 15

Robustness against brightness adjustment and cropping attacks

4.2.6 Other measurement of imperceptibility

Most studies use PSNR to measure imperceptibility of the watermarking scheme, but some use structural similarity (SSIM) index [9, 15, 16]. The SSIM index is designed based on human visual system, and hence measures the perceptual difference between two images [26]. The perceptual quality is believed to be good if the value of SSIM is above 0.9 [27]. Figure 16 shows the comparisons of the imperceptibility based on SSIM. This figure indicates that our method achieves satisfactory perceptual quality, and the SSIM of the proposed method is almost the best of these methods.

Fig. 16

Comparison of the imperceptibility based on SSIM

4.2.7 Other discussions

This study used a sine function to design the embedding rule. The advantage of using the sine function is that it can process real numbers, and the value of the R matrix includes real values. In addition, the sinusoidal function has two distinct states, namely positive and negative cycles, which correspond to 0 and 1 of the watermark bit. In addition, because we embedded both the robust and fragile watermarks, the additional benefit of this study is that the proposed method can have two functions: tampering detection and copyright protection. Moreover, the proposed method can extract the robust watermark without using the original host image and original watermark; thus, the method exhibits improved practicability.

This study used a secret key to scramble the image block before embedding the robust watermark. Therefore, even if the attacker knows the algorithm of this study but does not have this secret key, they cannot restore the order of the blocks and acquire the watermark. Therefore, it is possible to prevent an attacker from modifying the embedded watermark content to claim the copyright of the image. In addition, this study includes a secret key when generating the image authentication message. Therefore, the attacker cannot deduce the authentication message according to the algorithm of the study and cannot deliberately change the content of the verification message to invalidate the tampering detection. In summary, this method satisfies the Kirchhoff’s principle: all algorithms must be public, and only the key is secret. Kirchhoff’s criterion is frequently used to evaluate the usefulness of cryptographic methods [9, 15]. If the secret key must be passed over a public network to a specific person, asymmetric cryptography can be used to protect the security of the secret key during transmission. The secret key can be encrypted with the public key of the sender before transmitting it. After receiving the specific key, the receiver can use his/her private key to decrypt the content of the secret key.

5 Conclusions

This study proposed a QR-based digital watermarking method for improving robustness against cropping attacks. Our approach has the following merits: (1) The watermark embedding rule is designed based on a sine function that can handle real values of the R matrix; (2) each watermark bit has multiple copies spread across different blocks, so that the survival probability of the watermark bit may increase; (3) robustness against cropping attacks is enhanced by using fragile watermark; (4) the modifiable ranges of elements of the matrix R are analyzed; therefore, the hidden space of the R matrix is effectively used to improve the hiding capacity further; (5) the method can process the color host image and watermark, and the watermark can be extracted without the original host image and original watermark; and (6) the proposed method has a dual function: copyright verification and tampering detection. The experimental results of the proposed method are superior to those of the other methods in terms of imperceptibility, robustness against cropping attacks, and embedding payload. The main purpose of this study is the enhance the robustness against cropping attacks; therefore, we do not pay much attention to the robustness against common image processing. In this study, parameter K balanced imperceptibility and robustness. The same K value was used for the four elements of the R matrix. In fact, setting different K values for different hidden locations may increase imperceptibility and robustness. Therefore, in future research, the evolutionary algorithm will be used to determine the optimal K value for different hidden positions.