How to establish a security key is an important critical issue in cryptography, early cryptologists usually assumed that there was a secure channel, the key could be exchanged through in this channel. However, this assumption becomes unmanageable when the number of participants increases greatly, or when secure channel is not available. To get rid of this awkward problem, in 1976, Diffie and Hellman introduced a kind of protocol [1], key agreement (KA) protocol. In a KA protocol, each party can contribute fairly her/his part to establish a shared secret key over an insecure public channel. Although the protocol [1] was later proven vulnerable to Man -in-Middle attack as it does not provide authentication of the communicating parties, the design philosophy of the protocol had fundamentally changed the way cryptosystems might work. Since then, many researches have been focused on designing new KA protocols or formalizing the security model for KA protocols [2–10].

However, the securities of the proposed protocols above are often designed based on trapdoor functions. Cryptologists often suppose that it is easy to compute trapdoor function f(x) given the variable x but impossible to deduce x from f(x) without the trapdoor. However, so far, no one has proved the existence of any one-way function with a trapdoor. In other words, all the KA protocols above rely for their security on unproven mathematical assumptions, e.g. solving the discrete logarithm problem or factoring large numbers are difficult. In addition, the discovery in 1994 by Shor for fast factorization of integers [11] with a polynomial algorithm in a quantum computer and the discovery in 1996 by Grover for searching an unsorted database [12] with N entries in O(N1/2) time and using O(log N) storage space in the linear quantum model cast more doubts on the existence of the trapdoor functions. The security of these KA protocols face serious challenges, how to design KA protocols which do not rely for their securities on unproven mathematical assumptions becomes urgent. Fortunately, in 2004, a new type of KA, quantum KA (QKA), was proposed by Zhou et al. [13], the protocol uses quantum teleportation and maximally entangled states to generate secret key over public channels, the security of the protocol is guaranteed by the principle of quantum mechanics. However, the study [14] showed that Zhou et al.’s protocol was not secure. In 2010, Chong et al. presented a QKA protocol [15] based on BB84 protocol [16]. In 2013, a multi-party QKA protocol, an extension of two-party QKA protocol [17], was proposed by Shi et al. However, Liu et al. pointed out that Shi et al.’s protocol was not secure, then they proposed a new multi-party QKA protocol using single particles [18]. Sun et al. gave an improvement to Liu et al.’s protocol by introducing two additional unitary operations [19], they claimed that the efficiency of the protocol could be raised for (N-1) times, where, N is the number of the participants in the protocol. Furthermore, they claimed that the improved protocol can overcome the privacy leakage problem existed in Liu et al.’s protocol. However, the study [20] showed that any participant’s sub-key still could be deduced by other two participants next to him/her in Sun et al.’s improved protocol. Yin et al. proposed a three-party QKA protocol based on two-qubit entangled states [21], they claimed that each party in the protocol has an equal contribution to the establishment of the final shared secret key and the protocol can resist against both the outsider attack and the insider attack. In 2014, Shukla et al. proposed two QKA protocols based on Bell state and Bell measurement [22]. Huang et al. presented a QKA protocol with the block transmission of EPR pairs [23]. To solve the problem that QKA protocols cannot be immune to decoherence, Huang et al. proposed another QKA protocol [24] based on BB84 protocol. Recently, Xu et al. presented another three-party QKA protocol based on GHZ (Greenberger -Horne-Zeilinger) states [25].

Through analyzing the above introduction, we can see that QKA is being a new research hotspot in cryptography, more and more protocols were proposed. However, the cryptanalysis of QKA protocol has not drawn enough attention. As that described by Gao et al. in [26], cryptanalysis plays an important role in the development of cryptography, it estimates a protocol’s security level, finds potential loopholes, and tries to overcome security issues. In the study of quantum cryptography, quite a few effective attack strategies have been proposed, such as entanglement-swapping attacks [27], channel-loss attacks [28], denial-of-service attacks [29], Trojan horse attacks [30], participant attacks [31] and so on. Understanding those attacks will be helpful for us to design new schemes with high security. In these kinds of attacks, we should pay more attention to the participant attacks. In contrast to an outside attacker, an inside participant, especially in a multi-party quantum cryptography protocol, usually has more power to attack the protocol for her/his participant identity. Later studies showed that quite a number of quantum cryptographic protocols could not resist participant attacks [32–37].

In this paper, we will show that Yin et al.’s protocol [21] is not secure against participant attack. Through launching a special kind of attack, two dishonest participants can totally offset the third participant’s role in the generation of the final key, they can determine the final shared key alone. To avoid the flaw, a further improved protocol is proposed. To maintain the integrity of the paper, let us give a brief review of the Yin protocol which is composed of the following 7 steps.

  • Step 1 Participant Alice (Bob, Charlie) generates a random string K A (K B , K C ), where, K J = {j 1,...j n }, J ∈ {A, B, C}, j ∈ {a, b, c}, a i , b i , c i ∈ {0, 1}, i = 1, ... n. In the meanwhile, Alice (Bob, Charlie) prepares n EPR pairs |ψ +〉 A1A2(|ψ +〉 B1B2,|ψ +〉 C1C2) and then takes the first and the second particle from each pair to form sequences \(S_{A_{1}} \left ( {S_{B_{1}} ,S_{C_{1}} } \right )\) and \(S_{A_{2}} \left ( {S_{B_{2}}, S_{C_{2}}}\right )\) respectively, where, \(\left |\psi ^{+}\right \rangle =\frac {1}{\sqrt {2}}\left (\left |01\right \rangle +\left |10\right \rangle \right )\). To check the security of the transmission, each participant prepares enough decoy photons which are randomly chosen from Z-basis {|0〉, |1〉} or X-basis {|+〉, |−〉} and then randomly inserts them into \(S_{A_{2}} \left ({S_{B_{2}} ,S_{C_{2}}}\right )\). Alice (Bob, Charlie) sends the mixed sequence \(S_{A_{2}} \left ({S_{B_{2}} ,S_{C_{2}} } \right )\) to Bob (Charlie, Alice).

  • Step 2 After having been confirmed that Bob (Charlie, Alice) has received the sequence \(S_{A_{2}} \left ( {S_{B_{2}} ,S_{C_{2}} } \right )\), Alice (Bob, Charlie) announces the position and the basis of each decoy particle. Bob (Charlie, Alice) and Alice (Bob, Charlie) check the security of the channel. If the error rate exceeds the threshold, they restart the protocol.

  • Step 3 Bob (Charlie, Alice) picks out the decoy particles from the sequence \(S_{A_{2}} \left ( {S_{B_{2}} ,S_{C_{2}} } \right )\) and then performs unitary operation \(U_{b_{i}} \left ( {U_{c_{i}} ,U_{a_{i}} } \right )\left ( {i\text {= 1,2,...,n}} \right )\) on the rest particles to form sequence \(S_{A_{2}}^{1} \left ( {S_{B_{2}}^{1} ,S_{C_{2}}^{1}} \right )\), where, U 0 = I = |0〉〈0|+|1〉〈1|, U 1 = σ x = |0〉〈1|+|1〉〈0|. In the meanwhile, Bob (Charlie, Alice) prepares enough decoy photons and then randomly inserts them into \(S_{A_{2}}^{1}\left ( {S_{B_{2}}^{1} ,S_{C_{2}}^{1}} \right )\). Bob (Charlie, Alice) sends the mixed sequence \(S_{A_{2}}^{1} \left ({S_{B_{2}}^{1} ,S_{C_{2}}^{1}} \right )\) to Charlie (Alice, Bob).

  • Step 4 After having been confirmed that Charlie (Alice, Bob) has received the sequence \(S_{A_{2}}^{1} \left ( {S_{B_{2}}^{1} ,S_{C_{2}}^{1}} \right )\), Bob and Charlie (Charlie and Alice, Alice and Bob) perform the second eavesdropping check. If the error rate exceeds the threshold, two parties abort this protocol.

  • Step 5 Charlie (Alice, Bob) first picks out decoy photons from the sequence \(S_{A_{2}}^{1} \left ( {S_{B_{2}} ^{1} ,S_{C_{2}}^{1}} \right )\) and then performs unitary operation \(U_{2c_{i}} \left ( {U_{2a_{i}} ,U_{2b_{i}} } \right )\left ( {i\text {= 1,2,...,n}} \right )\) on the rest particles to form sequence \(S_{A_{2}}^{2} \left ( {S_{B_{2}} ^{2} ,S_{C_{2}}^{2}} \right )\), where, U 0 = I = |0〉 〈0| + |1〉 〈1|, U 2 = σ z = |0〉 〈0| − |1〉 〈1|. In the meanwhile, Charlie (Alice, Bob) prepares enough decoy photons and then randomly inserts them into \(S_{A_{2}}^{2} \left ( {S_{B_{2}}^{2} ,S_{C_{2}}^{2}} \right )\). Charlie (Alice, Bob) sends the sequence to Alice (Bob, Charlie).

  • Step 6 After having been confirmed that Alice (Bob, Charlie) has received the sequence \(S_{A_{2}}^{2} \left ( {S_{B_{2}}^{2} ,S_{C_{2}}^{2}} \right )\), two parties perform the third security check. If they confirm that the transmission is insecure, this process is aborted.

  • Step 7 Alice (Bob, Charlie) picks out the decoy particles from the sequence \(S_{A_{2}}^{2} \left ( {S_{B_{2}} ^{2} ,S_{C_{2}}^{2}} \right )\) and then performs Bell-state measurement on the corresponding particles in sequences \(S_{A_{1}} \left ( {S_{B_{1}} ,S_{C_{1}} } \right )\) and \(S_{A_{2}}^{2} \left ( {S_{B_{2}}^{2}, S_{C_{2} }^{2}}\right )\). According to the measurement results, each party can obtain the other two parties’ secret keys. Thus they can determine the final shared secret key K = K A ⊕ K B ⊕ K C .

As that pointed out by Sun et al. in [19], one principle in the design of QKA protocol is that the protocol should have fairness property, that is, all involved participants are equally to influence the final shared key. In other words, no non-trivial subset of the participants can determine the shared key. Yin et al. claimed that each participant in their proposed protocol has an equal contribution to the establishment of the shared secret key and three participants are entirely peer entities [21]. However, we will show that two dishonest participants in Yin et al.’s protocol can conclude to determine the shared key alone, in the next, we will show how two dishonest participants can do this. Without loss of generality, we suppose that Alice and Bob are two dishonest participants. In step 3, Alice and Bob can disguised as legitimate participants to perform unitary operation on the corresponding particles. In step 5, it is obviously that Alice and Bob can deduce Charlie’s unitary operations though measuring the particle pairs in which one particle has been performed unitary operation by Charlie. Then Bob can choose different unitary operations to perform on the particles and then send these particles to the honest participant Charlie. The role of the honest participant in the generation of the final key can be totally offset by this different unitary operations, the final key will be only determined by these two dishonest participants, so the protocol is not secure against the participant attack. In the next, we give the details of our proposed attack. In step 3, Bob (Charlie, Alice) first picks out the decoy photons and then performs unitary operation \(U_{b_{i}} \left ({U_{c_{i}}, U_{a_{i}}}\right )\) on the rest particles in the sequence \(S_{A_{2}} \left ( {S_{B_{2}} ,S_{C_{2}}}\right )\) to form sequence \(S_{A_{2}}^{1} \left ( {S_{B_{2}}^{1}, S_{C_{2}}^{1}} \right )\). Bob (Charlie, Alice) prepares enough decoy photons and then randomly inserts them into the sequence \(S_{A_{2}}^{1} \left ( {S_{B_{2}}^{1}, S_{C_{2}}^{1}} \right )\). Bob (Charlie, Alice) sends the mixed sequence \(S_{A_{2}}^{1} \left ( {S_{B_{2}}^{1} ,S_{C_{2}}^{1}} \right )\) to Charlie (Alice, Bob). After checking the security of the quantum transmitting, Bob can deduce Charlie’s unitary operation with the help of Alice through performing Bell-state measuring on the corresponding particles in sequences \(S_{B_{1}} \) and \(S_{B_{2}}^{1} \). For example, if Bob and Alice get result |ψ +〉, they can deduce that Charlie’s unitary operation is U 0, which means that the corresponding bit of Charlie’s sub key is 0. If Bob and Alice get result |ϕ +〉, they can deduce that Charlie’s unitary operation is U 1, which means that the corresponding bit of Charlie’s sub key is 1. After the Step 4 having been completed, Bob holds sequences \(S_{B_{1}} \) and \(S_{C_{2}}^{1}\), Alice holds sequences \(S_{A_{1}}\) and \(S_{B_{2}}^{1}\). If all the parties are honest, in the next, Bob will perform unitary operation \(U_{2b_{i}}\) on the corresponding particle in the sequence \(S_{C_{2}}^{1} \)to form sequence \(S_{C_{2}}^{2}\) and then send the sequence to Charlie, Alice will perform unitary operation \(U_{2a_{i}}\) on the corresponding particle in the sequence \(S_{B_{2}}^{1}\) to form sequence \(S_{B_{2}}^{2}\) and then send the sequence to Bob, Charlie performs \(U_{2c_{i}}\) on the corresponding particle in \(S_{A_{2}}^{1} \) to form sequence \(S_{A_{2}}^{2}\) and then send the sequence to Alice. Charlie (Alice, Bob) can deduce other two parties’ sub keys through performing Bell-state measurement on the corresponding particles in sequences \(S_{C_{1}} \left ( {S_{A_{1}}, S_{B_{1}}}\right )\) and \(S_{C_{2}}^{2} \left ( {S_{A_{2}}^{2} ,S_{B_{2}}^{2}} \right )\). We give the following Table 1 to show the relations among Alice’s (Bob’s, Charlie’s) first unitary operations, Bob’s (Charlie’s, Alice’s) second unitary operations and Charlie’s (Alice’s, Bob’s) measurement results.

Table 1 Relations among Alice’s (Bob’s, Charlie’s) first unitary operations, Bob’s (Charlie’s, Alice’s) second unitary operations and Charlie’s (Alice’s, Bob’s) measurement results. Alice's (Bob's, Charlie’s) second unitary operations are listed in the first row
Full size table

However, if Bob chooses a different unitary operation \(I\otimes U_{2c_{i}} \left ( {I\otimes U_{2b_{i}}}\right )\) to perform on the corresponding particle in sequence \(S_{C_{2}}^{1}\) to form sequence \(S_{C_{2}}^{3}\), through analyzing Yin et al.’s protocol, we can deduce that the corresponding particle pair in sequences \(S_{C_{1}}\) and \(S_{C_{2}}^{3}\) is in state \(I\otimes U_{2c_{i}} \left ( {I\otimes U_{2b_{i}}}\right )I\otimes U_{a_{i}} \left |\psi ^{+}\right \rangle _{C1C2}\). We give the following Table 2 to show the relations among Alice’s sub keys, Bob’s sub keys, Charlie’s sub keys and Charlie’s measurement results after Bob has performed \(I\otimes U_{2c_{i}} \left ({I\otimes U_{2b_{i}}}\right )\) on the corresponding particle in sequence \(S_{C_{2}}^{1}\). Bob prepares enough decoy photons and then inserts them into sequence \(S_{C_{2}}^{3}\) randomly. Bob sends \(S_{C_{2}}^{3}\) instead of \(S_{C_{2}}^{2}\) to Charlie. After having confirmed the security of the quantum transmission, Charlie performs Bell-state measurement on the photon pairs in his two sequences to deduce the final shared key. However, the final key K = K A ⊕ K B ⊕ K C ⊕ K C = K A ⊕ K B will be only determined by Alice and Bob, Charlie can’t detect this kind of attack.

Table 2 Relations among Alice’s sub keys, Bob’s sub keys, Charlie’s sub keys and Charlie’s measurement results after Bob has performed \(I\otimes U_{2c_{i}} \left ( {I\otimes U_{2b_{i}}}\right )\) on the corresponding particles in \(S_{C_{2}}^{1}\). Charlie’s sub keysare listed in the first column, Alice’s sub keys and Bob’s sub keys are listed in the first row
Full size table

As seen in Table 2, through performing a different unitary operation on the corresponding particles in sequence \(S_{C_{2}}^{1}\) according to his own sub key and Charlie’s sub key, Bob can totally offset the role of Charlie in the generation of the final key, the final key is only determined by Bob and Alice, the protocol is not secure against the insider attack. We take a 4 bits key generation process as an example to show this attack, without loss of generality, we suppose that Alice and Bob want to generate a predetermined key K = 1111, they can randomly generate a 4 bits string R = 0101 as Alice’s sub key K A = 0101, K B = K ⊕ K A = 1111 ⊕ 0101=1010. After having received the sequence \(S_{C_{2}}\) from Charlie, Alice performs unitary operation \(U_{a_{i}} \left ( {i=1,2,3,4} \right )\) on the ith particle in sequence \(S_{C_{2}}\), where, a 1 = 0, a 2 = 1, a 3 = 0, a 4 = 1. Without loss of generality, we suppose that Alice and Bob deduce that Charlie’s sub key is 1101, after having received sequence \(S_{C_{2}}^{1}\) from Alice, Bob performs unitary operation \(I\otimes U_{2c_{i}}\) \(\left ({I\otimes U_{2b_{i}}}\right )\) \(\left ({i=1,2,3,4} \right )\) on the ith particle in sequence \(S_{C_{2}}^{1}\) and then sends the sequence back to Charlie, where, b 1 = 1, b 2 = 0, b 3 = 1, b 4 = 0, c 1 = 1, c 2 = 1, c 3 = 0, c 4 = 1. Charlie performs Bell-state measurement on the corresponding photon pairs, he will get results |ψ +〉 C 1 C 2 , |ϕ −〉 C 1 C 2 , |ψ −〉 C 1 C 2 and |ϕ −〉 C 1 C 2 , which means that the corresponding bits in Alice’s and Bob’s sub keys are 00, 11, 01, and 11 respectively. We know that Charlie’s sub key is 1101, in this case, Charlie can deduce the final shared key through computing the equation K = (0 ⊕ 0 ⊕ 1, 1 ⊕ 1 ⊕ 1, 0 ⊕ 1 ⊕ 0, 1 ⊕ 1 ⊕ 1) = 1111, which in fact has been determined before the execution of the protocol.

The above analysis shows that two dishonest participants in Yin et al.’s protocol [21] can conclude to determine the shared key alone, the protocol can’t resist against participant attack. Actually, to avoid the above attack, we should make sure that any two dishonest participants can’t choose a suitable unitary operation to offset the role of the third parties’ sub key in the generation of the final secret shared key. Inspired by the reference [22], we give an improved protocol as follows. In step 3, after having performed unitary operation \(U_{j_{i}} \left ( {i\text {= 1,2,...,n}}\right )\) on each particle in the sequence \(S_{J_{2}}\) to form sequence \(S_{J_{2}}^{1}\), each participant applies a permutation operator (π n ) J on sequence \(S_{J_{2}}^{1}\) to form a new sequence \(S_{J_{2}}^{1\ast }\). In the meanwhile, each participant prepares enough decoy particles and then inserts them into sequence \(S_{J_{2}}^{1\ast }\). Each participant sends the mixed sequence \(S_{J_{2}}^{1\ast }\) to the next participant. In the end of the protocol, three participants publicly announce the details of their permutation operators. Alice (Bob, Charlie) can deduce the shared key according to the Bell-state measurement results and the details of Bob’s (Charlie’s, Alice’s) permutation operator.

In summary, we show that the Yin et al.’s QKA protocol [21] is not secure against participant attack. Two dishonest participants can totally offset the third participant’s role in the generation of the final shared key by launching a special kind of attack. In order to avoid this flaw, a further improved protocol is proposed.