1 Introduction

IoT forms a system through interconnecting various machines, devices, and software services. IoT can increase the quality of human life and play a significant role in today’s modern life since through energy-efficient automation. It should be noted that since IoT systems are resource-constrained and have an ad-hoc nature can be influenced by cyber attackers [1]. This necessitates designing reliable and secure IoT, and the challenges should be overcome so that other systems and human lives are not damaged or destroyed. Some attacks like a gray hole attack have illegal penetration into the system. When an attack affects an IoT, it is difficult to remove the threat and bring the system back online. It should be noted that the conventional approaches for securing information like intrusion detection or encryption are not adequate when these risks are dealt. The sensor and actuator measurements compatibility factor with IoT control mechanism and the physical process are not taken into account in these plans. These cases are essential for the protection scheme. Due to the high volume of sensing and network data generated by IoT systems and devices, it is very effective to have cryptographic authentication in constant monitoring and analysis for the IoT systems’ security. For instance, as observed in Fig. 1, a gray hole attacker can strive to imitate the wireless router connecting the IoT devices with the rest of the network. Then, it can reroute the traffic by the legal wireless router.

Fig. 1
figure 1

IoT gray hole attacks based on application scenarios

In this paper, a gray hole attack discovery and prevention method via informing other nodes in the internet of things are proposed. The proposed scheme first verifies the trust level for the nodes in the network and then discovers and eliminates the malicious gray hole nodes using control packets. The proposed method is based on the AODV routing protocol and is presented under the MTISS-IoT name which stands for the reduction of gray hole attacks using check node information.

The paper presented here is organized as the following. Section 2 converses the security attacks for IoT networks. The related work is discussed in Sect. 3. Section 4 brings the proposed MTISS-IoT strategy. In Sect. 5, the simulation results are discussed to demonstrate the efficiency of the proposed MTISS-IoT. Finally, conclusions and future works of this research are discussed in Sect. 6.

2 Security attacks

IoT systems are vulnerable to function degradation and security risks. They might be passive or active threats as they have the reliance on wireless channels for communication. A security threat targeting IoT is provided in Figs. 2 and 3. In this paper, the following vulnerability is of interest.

Fig. 2
figure 2

Simple gray hole attack

Fig. 3
figure 3

Cooperative gray hole attack

2.1 Gray hole attack

In a gray hole attack, there is a malicious node among the selected path nodes that drops the messages and does not forward them to the adjacent nodes. In this regard, two possibilities can be mentioned. In the first possibility, the gray hole node fakes itself as a routing node by observing the routing protocol accurately. In another possibility, the attackers do not observe the routing protocol and so that it violates the protocol specifications by exploiting its vulnerabilities. The gray hole node fakes itself as a routing node in different ways based on the used protocol. The gray hole attacks can be divided into two groups based on the number of malicious nodes—simple and cooperative gray hole [2].

2.2 Simple gray hole attack

In this kind of gray hole attack, a malicious node foists itself as a medium node that belongs to the shortest route to the destination. Regardless of the routing table, the gray hole node is always accessible to reply route requests to receive data packets and drop them instead of forwarding them. In the flooding-based protocols, before sending a reply by healthy nodes, the gray hole node sends a reply to the requesting node. In this way, the selected route will contain a malicious node which drops the packets or sends them to incorrect nodes. The process of a gray hole attack is shown in Fig. 2. As seen in this figure, the data packets should be transferred from a source \( {\text{Th}}_{\text{S}} \) to a destination \( {\text{Th}}_{\text{D}} \). For this purpose, a proper route—from the origin to the destination—should be detected. So, if \( {\text{Th}}_{\text{M}} \) is malicious, it fakes itself as a node present in the shortest path to the destination. Then, it will respond to the request by sending a reply to the \( {\text{Th}}_{\text{S}} \) sooner than other nodes, in this way, the \( {\text{Th}}_{\text{S}} \) will send the data packets to \( {\text{Th}}_{\text{M}} \) and discard the replies received from other nodes [3].

2.3 Cooperative gray hole attack

Cooperative gray hole attack is the other way of implementing a gray hole attack where there is cooperation among different malicious nodes in the attack for violating the security system and routing protocol. As indicated in Fig. 3, when malicious \( {\text{Th}}_{{{\text{M}}1}} \;{\text{and}}\; {\text{Th}}_{{{\text{M}}2}} \) cooperate, \( {\text{Th}}_{{{\text{M}}1}} \) refers to \( {\text{Th}}_{{{\text{M}}2}} \) as its subsequent hop. Given the scenario described in [4], the source \( {\text{Thing}}_{\text{S}} \) transfers a further request packet (\( {\text{FReq}}_{\text{packet}} \)) to \( {\text{Th}}_{{{\text{M}}2}} \) by another route other than via \( {\text{Th}}_{{{\text{M}}1}} \) (for example, \( {\text{Th}}_{\text{S}} - {\text{Th}}_{\text{C}} - {\text{Th}}_{\text{E}} - {\text{Th}}_{{{\text{M}}2}} \)). The \( {\text{Thing}}_{\text{S}} \) requests \( {\text{Th}}_{{{\text{M}}2}} \) if it is the subsequent hop of \( {\text{Th}}_{{{\text{M}}1}} \) and if it has a reliable path to the destination \( {\text{Th}}_{\text{D}} \). As there is a cooperation between the \( {\text{Th}}_{{{\text{M}}2}} \) and \( {\text{Th}}_{{{\text{M}}1}} \), its further reply (\( {\text{FReq}}_{\text{packet}} \)) is positive. Thus, it is assumed by the source \( {\text{Th}}_{\text{S}} \) that the path \( {\text{Th}}_{\text{S}} - {\text{Th}}_{{{\text{M}}1}} - {\text{Th}}_{{{\text{M}}2}} \) has security, and it begins transferring the data packets over it. \( {\text{Th}}_{{{\text{M}}1}} \) will drop the packets following being intercepted.

3 Related works

Various security measurements have been developed and used in different works for addressing denial-of-sleep attacks and protecting IoTs against gray hole attacks. It is not a recent issue, and there are extensive studies on it. Different approaches have been suggested by different studies in order to address these attacks.

In this paper, a new real-time hybrid intrusion detecting framework is provided including specification-oriented and anomaly-based intrusion detecting modules to detect two well-known routing attacks in IoT known as selective-forwarding and sinkhole attacks. Hence, the host nodes’ behavior is analyzed by the specification-oriented intrusion detecting elements situated in the router nodes, and their local outcomes are sent to the root node via normal data packets. Furthermore, the unverified optimal-path forest algorithm is used by an anomaly-oriented intrusion detection agent situated in the root node to project clustering models via incoming data packets. This agent that is on the basis of the MapReduce architecture is able to act in a distributed platform to project clustering models and therefore detect parallelly the anomalies as a global detection method. By the suggested technique, decisions are made regarding suspicious performance via a voting mechanism. Remarkably, the suggested technique is also prolonged for detecting wormhole attack. Deploying the hybrid suggested model is also assessed in a smart-city scenario via a present platform. The scale of the free network and the capability in identifying malevolent nodes are two key properties of the suggested framework that are assessed via various tests in this work [5].

Here, the authors provide an algorithm in terms of the exponential smoothing concept for detecting the nodes’ topological isolation owing to black hole attack. Exponential smoothing is a method to smooth time series data utilizing the exponential window function, and it is utilized for short, medium, and long-term prediction. We use the exponential smoothing in our suggested algorithm to approximate the next onset time of packets at the sink node from each of other nodes in the LLN. This estimation is used to design the algorithm for identifying the malevolent nodes initiating real black hole attack [6].

In [7], SCOTRES, a trust-oriented system, is proposed for secure routing in ad hoc networks to advance the network entities’ intelligence using five innovative metrics. The resource consumption of each node is considered by the energy metric to impose similar quantity of collaboration and to increase the network’s lifetime. The topology metric knows the positions of the nodes and improves the load balancing. The tolerance in periodic malfunctioning is provided by channel-health metric owing to bad channel circumstances, and the network is protected versus jamming attacks. The collaboration of each subject for a particular network operation is evaluated by reputation metric to detect the specific attacks; however, the total compliance is estimated by trust metric, protecting against combinatorial attacks. The system’s security features are validated by the theoretic analysis.

Black hole and selective forwarding routing attacks are addressed in [8], which are the basis security attacks on the data routing in IoT networks. Today, most IoT tools, from medical instruments to connected vehicles, and even smart buildings are able to communicate with each other wirelessly. In this work, a trust-oriented routing protocol is provided for lossy and low-power networks stating black hole and selective forwarding attacks. We indicate that our suggested protocol is secured from black hole and selective forwarding attacks, and it does not enforce undue overheads on network traffic.

In [9], a comprehensive study of RPL and its known attacks are represented and the mitigation approaches are suggested to counter these attacks. We performed a complete review of the RPL standard, containing a currently suggested modification. Moreover, all recently published attacks on RPL and their mitigation approaches were investigated within the literature. According to this assessment, and as we know, this is a first-of-its-kind categorizing outline for the mitigation approaches in terms of the methods utilized for the mitigation. Moreover, we systematically deliberated RP-oriented intrusion detection systems (IDSs) and their categorizations, while remarking the most lately suggested IDSs.

This work suggested constructing a trust-oriented framework for RPL to counter black hole attacks. It can be run at two levels of an intra-DODAG and an inter-DODAG. Incremented dropping the packets, depleting the resources, and high packet overhead are the impacts of black hole attacks in an IoT network. It eventually leads to destabilizing the network owing to incremented packet delay, rank modifying, and disturbance in the topology. Regarding the rank modifying, the ranks are computed again, therefore, activating a local repair later initiating a repair thoroughly by the root. Such regular repairs might end up influencing the network efficiency [10].

In [11], SIEWE (Strainer-based Intrusion Detection of Black hole in 6LoWPAN for the internet of things), an intrusion detection mechanism, is proposed to recognize black hole attack on routing protocol RPL in IoT. To arrange the black hole attack, a malevolent node first should broadcast a comparatively great routing metric to the nearby nodes so that it seems to be the best candidate chosen as a parent. The above fact is used by SIEWE to filter out the nodes broadcasting a comparatively great routing metric and appending their node IDs to a suspect list. Furthermore, detecting and verifying are performed by only the nodes with at least one entrance in their suspect list set. The above nodes analyze the nodes’ behavior in their respective suspect lists while sending their observations to BR node. Therefore, SIEWE includes only those nodes in the vicinity of suspected nodes rather than involving every resource inhibited node in the network for detecting and verifying procedure. Hence, overall consumed energy is saved by SIEWE within the network and the number of observation packets moved to BR node is limited.

SPLIT, a secure and scalable RPL routing protocol, is proposed in [12] for IoT networks. A lightweight remote attestation method is effectively utilized by SPLIT to guarantee the network nodes’ software integrity. SPLIT piggybacks attestation is processed on the RPL’s control messages to prevent further overhead resultant by attestation messages. Consequently, SPLIT benefits the RPL protocol’s scalability propertied and low energy consumption that are essential for the resource-constrained large-scale networks like IoT. The simulation outcomes for various IoT setups indicate the SPLIT effectiveness in comparison with the state of the art by existing various kinds of attacks regarding metrics like energy consumption and packet delivery ratio.

One of the methods offered to detect and dispel a DoS attack in contrast to the IoT middleware—which is also known as NPS—is the REATO method. A real test bed is used to authenticate the premeditated solution for NPS architecture. This solution is composed of an NPS sample mounted on a Raspberry Pi which receives open data feeds in real time using an adaptable source set. To find a solution to detect DoS attacks in the IoT, it should be noted that we should consider all the potential circumstances—attacks to the data sources and IoT platform [13].

A deep learning established machine learning method has been presented in [14] for the IoT to detect the routing attacks. The Cooja IoT emulator has been employed to generate high-fidelity attack data within IoT networks having 10–1000 nodes. They have recommended a highly scalable, profound-learning-based attack detection approach to uncover the IoT routing attacks which are decreased rank, hello-flood, and version number modification attacks through extraordinary accurateness and meticulousness. Applying the deep learning for cyber security in the IoT necessitates the accessibility of considerable IoT attack data.

Qin et al. [15] proposed an IMLADS to manage the security of the IoT in a well-organized manner. This method differs from the traditional systems so that it employs the movable agents—instead of stationary one—to complete the data collection and analysis stages. In this way, the movable agent running platform controls the mobility despite the fact that that is irrelevant to its setup system. The data can be transferred to the other nodes based on a preset monitoring task. By taking advantage of this technology, the number of agents running in the system can be increased significantly while enhancing the steadiness and scalability of the system. Therefore, various approaches have been designed by the authors of the mentioned study for node and system-level security monitoring. In the first approach, they employed a lightweight data collection and analysis method. It should be noted that only small local computing resources can be dominated by this method. In the second approach, they proposed a parameter calculation technique. However, this approach has perpetual computational complexities.

The previous works to design IDS for the IoT are listed in Table 1 (“−” indicates the indefinite characteristics).

Table 1 Summary of the approaches for IoT literature

4 The proposed MTISS-IoT approach

In the following section, we design a gray hole-security threats-immune schema by employing the cryptographic authentication. The MTISS-IoT consists of four phases, such as the verifying thing trust in the IoT is discussed in Sect. 4.1. Testing the routes is discussed in Sect. 4.2, gray hole thing discovery in MTISS-IoT is discussed in Sect. 4.3, and the malicious thing elimination process in MTISS-IoT is discussed in Sect. 4.4.

4.1 The assumptions applied in the proposed MTISS-IoT

The proposed method works under the following assumptions:

  • All of the things are identical in their physical characteristics.

  • If thing X is in the transmission range of thing Y, then thing Y is in the transmission range of thing X too.

  • All of the things have been verified and can take part in the transmissions.

  • Both the source thing and the destination thing are IDS things that are used to detect the malicious thing along the route.

4.2 Phase 1: Verifying thing trust in the IoT

In the proposed MTISS-IoT method, each node monitors the information regarding its immediate neighbors, the nodes within a single step distance, and also other nodes in the network using RREQ packets. In the proposed MTISS-IoT method, the route discovery process is the same as the base AODV method. If the source node needs to discover a route to reach the destination, it broadcasts the RREQ packet. According to our proposed method, once an intermediate node receives the RREQ packet for a specific destination, it carries out the following actions:

For each intermediate node on the RREQ packet route, once receiving this message from a node, it increases the ID of that node and its RREQ_C credit field by one (the initial RREQ_C credit value is equal to zero) and inserts it into its monitoring table, and then checks whether the RREQ packet is duplicate (checks to see if its own ID is listed in the source route). If this is the case, the node drops that RREQ packet. Otherwise, if the node has a route to the destination, then it submits its request into the RREQ packet and sends an RREP packet back to the source node. However, if this is not the case and the node does not have a route to the destination, then the node adds its ID to the source route and broadcasts the RREQ packet in the network. After sending the RREQ packet to its neighboring nodes, the node transfer field (RREQ_T), which has an initial value of zero, is increased by one. The RREQ_C credit field for a specific node is increased by its adjacent nodes every time it forwards an RREQ packet. Each time a node transmits a packet to one of its neighboring nodes, the RREQ_T field for that neighbor is increased by one. This way, each node can collect more information about the behavior of other nodes during the route discovery process. Each node sets its trust to other nodes to either high or low according to the value of the RREQ_C credit field. The monitoring table is also run periodically to represent the current topology and the behavior of the network nodes. Tables 2, 3, and 4 demonstrate an example of a monitoring table for Fig. 4.

Table 2 An example of a monitoring table for Thing 4
Table 3 An example of a monitoring table for Thing 7
Table 4 An example of a monitoring table for Thing Do
Fig. 4
figure 4

An example of thing layout in the proximity of a suspicious thing in an IoT

As presented in these tables, the Do node has received two different RREQ packets from nodes 4 and 7 but has not sent any packets to node 2. In the meanwhile, it has only sent one packet to node 5 and eliminated some packets selectively, while other nodes have sent the packet to their neighbors normally after receiving it. For instance, node 4 has sent the packet to node Do after receiving it from node 8. The trust value X of node X for node Y is calculated using Eq. (1):

$$ T(X,Y) \, = {\text{ The}}\;{\text{calculated}}\;{\text{RREQ}}\_{\text{C}}\;{\text{credit}}\;{\text{by}}\;{\text{thing}}\;{\text{A}}\;{\text{for}}\;{\text{thing}}\;{\text{B}} . $$
(1)

According to the AODV routing protocol, a node can eliminate a received RREQ packet only in two cases: first, when it receives an RREQ packet that has been received before and the node has already broadcast it; second, when the node receives an RREQ which contains a new route to the requested destination. In this case, the node deletes the RREQ packet and sends an RREP to the source node. The trust threshold value for a neighbor is represented by \( T_{C} \). Selecting the optimal threshold value is highly valuable for ensuring the good performance of the network. We analyze this value by testing the RREQ packets and with regard to the behavior of the nodes in the network. Node \( X \) is trustworthy for node \( Y \) if the \( {\text{RREQ}}\_{\text{C}} \) and \( {\text{RREQ}}\_{\text{T}} \) columns for node X in the monitoring table of node \( Y \) are identical. In other words, it transmits every unique RREQ it receives to its immediate neighbor.

For our analysis, we considered a threshold \( T_{c} \) which is the difference between the RREQ_C and \( {\text{RREQ}}\_{\text{T}} \) values for a specific node and its neighbors. Since the gray hole nodes eliminate some forwarded RREQ packets, we set the threshold value equal to zero for the nodes that are considered normal. This is because the nodes are supposed to transmit every unique RREQ packet they receive. In fact, the threshold value is calculated by subtracting the number of RREQ packets transmitted from a specific node from the number of RREQ packets transmitted to that node (\( {\text{RREQ}}\_{\text{T}} \)). Equation (2) demonstrates the \( T_{C} \) calculation.

$$ T_{C} = \left( {\sum {{\text{RREQ}}\_{\text{T}} - \sum {{\text{RREQ}}\_{\text{C}}} } } \right). $$
(2)

Just as the monitoring table is updated periodically, the behavior of each node is also observed by other nodes at specific time intervals. Each node needs to transmit the last RREQ packet it has received at different time intervals. Therefore, we say that the trust value for a node is high if the trust field of that node is equal to the threshold value, in other words, it sends all of the packets. On the other hand, we say that the trust value for a node is low if the trust field for that node is higher than the threshold value, i.e., it does not send some of the packets. Equations (3) and (4) demonstrate this calculation.

$$ {\text{The}}\;{\text{trust }}\;{\text{value}}\;{\text{ of}}\;{\text{ thing}}\; \, X\;{\text{ for }}\;{\text{thing}}\; \, Y\;{\text{ is}}\;{\text{ high}}\;{\text{ if:}}\;T(X,Y) = T_{C} $$
(3)
$$ {\text{The }}\;{\text{trust }}\;{\text{value}}\;{\text{ of }}\;{\text{thing}}\; \, X \, \;{\text{for}}\;{\text{ thing}}\; \, Y\;{\text{ is }}\;{\text{high}}\;{\text{ if:}}\;T(X,Y) > T_{C} . $$
(4)

In Fig. 4, the neighboring nodes of node DoFootnote 1 (the suspicious node) are nodes 2 and 4. We assume that nodes 2 and 4 are normal nodes and node Do is suspected to be a gray hole node. Every node except Do will take part in the RREQ packet transmission process normally. However, node Do will delete some of the transmitted RREQ packets, and therefore, its trust value will be higher than zero (\( T_{C} \)). The trust value for the intermediate nodes and the neighboring nodes of the suspicious node are presented in Table 5. These values were calculated after sending the monitoring table and with the trust threshold equal to zero. The behavior of the nodes after receiving an RREQ packet is shown in Fig. 5.

Table 5 Trust table
Fig. 5
figure 5

The behavior of the nodes after receiving an RREQ packet

4.3 Phase 2: Testing the routes

Once an intermediate node receives a RREP packet, it copies the trust value assigned to the next node on the route to the destination. Afterward, it sends that packet to the previous node and this process is repeated until the RREP packet reaches the source node. According to the algorithm presented in Fig. 6, all of the current nodes on the route from the source to the destination, which send the response packet, increase the trust value for their next node in the RREP packet. When the source node receives the first RREP packet, it checks the corresponding RREQ packets to see if the response is from the destination or from intermediate nodes. If the response is from the destination, then it considers the route for packet transmission. However, if the received response is from an intermediate node claiming to have a new route to the destination, then the source node selects the RREP packet which has the highest trust value assigned to the nodes along its corresponding route. Our method, on the other hand, adds one more checkpoint for discovering malicious nodes in addition to discovering highly trustworthy routes.

Fig. 6
figure 6

Flowchart of the MTISS-IoT

Once the source node receives the RREP packet, it sends a test packet comprised of multiple blocks to the destination along the routes with the highest trust values. The destination is bound to send an acknowledgment packet for that route which includes the number of the received blocks. Each intermediate node in the route needs to count the number of blocks in the test packet it hands over to the next node in a Number of Packets Forward manner. When the destination node receives the data packets from the source node, it starts counting and stores the number of packets in the received data in a block. Obviously, if a route is infected by a gray hole node, then the test packet will either not reach the destination at all or it will not reach the destination in its entirety. The destination IDS node sends the number of blocks in the test packet to the source node using the acknowledgment packet. Once the source IDS node receives the acknowledgment packet, it checks to see if the number of blocks is the same as the number of blocks in the test packet. If fewer blocks were received by the destination than the ones sent by the source, then the route is not safe. For this purpose, a parameter named \( P_{BH} (r) \) is defined. If the number of blocks in the acknowledgment packet for a route is fewer than the number of the blocks in the test packet or the acknowledgment packet is not received at all, then the \( P_{BH} (r) \) value will be increased for route r. However, if the test packet reaches the destination completely, then the received acknowledgment packet will contain the right number of blocks. This means that the route does not have a gray hole attack. In this case, the \( P_{BH} (r) \) value will be decreased. In the proposed method, the test packet transmission process is carried out with a predefined number of blocks.

The initial value of\( P_{BH} (r) \): If the route is verified according to the malicious node detection process, the initial value of \( P_{BH} (r) \) will be set to zero. However, if the route is not verified, i.e., all of the data blocks in the test packet did not reach their destination, then \( P_{BH} (r) \) will be set to 100. Afterward, based on whether or not it receives the acknowledgment packet, the source node updates the table for different routes according to Table 6. If the acknowledgment packet with the right number of blocks is received from the destination, \( P_{BH} (r) \) will be decreased by 50. However, if the acknowledgment is not received from the destination or it contains the wrong number, then \( P_{BH} (r) \) will be increased by 20. This process is repeated 2 times, and \( P_{BH} (r) \) is updated for all of the routes. Then, if the \( P_{BH} (r) \) value is higher than 50 for a route, that route will be identified as infected. The malicious nodes need to be discovered for the routes labeled as infected. This is carried out in the next phase of the proposed method. If the \( P_{BH} (r) \) value is lower than 50, then the route is valid.

Table 6 Route testing

After testing the routes, the source IDS node starts discovering the malicious gray hole nodes along the infected routes.

4.4 Phase 3: Gray hole thing discovery in MTISS-IoT

As mentioned before, in the gray hole attack, the gray hole nodes eliminate some of the packets they receive. A data control packet is proposed using this characteristic to check the intermediate nodes in an infected route. This packet includes three parameters as presented in Table 7.

Table 7 Fields of the control packet

The fields of the proposed data control packet in Table 7 are as follows:

Node ID: This field refers to the ID of the node that created the packet.

ID-NEXT: This field refers to the next node on the route to the destination.

Hash SHA-256: In order to discover the malicious gray hole node on the infected route, the source creates an encrypted message using the SHA-256 hashing function and puts the encrypted message in this field. This message must be identical in all of the data packets along the route. Each node needs to calculate the output once it receives the message and then forwards it to the source IDS node and also sends it forward with its own specifications. A hashing function gets a string of digits and letters and generates a unique output with a fixed length. The proposed control packet is some kind of data packet. Therefore, the malicious nodes usually eliminate this packet, just like other packets, and do not send it to normal nodes. Also, these malicious nodes are unable to send the correct output to the source node.

Hashing functions have some important properties:

  1. 1.

    Their output does not change for a specific input string. If you give the same message to the hashing function over and over again, you will get the same output every time.

  2. 2.

    Finding the reverse transform is very difficult. If the output for a hashing function is available, it is almost impossible to find the corresponding input.

  3. 3.

    Finally, the slightest change in the input will change the output entirely. Therefore, even if a gray hole node does not eliminate the packet, it will be unable to generate the correct output and send it forward.

After discovering the infected routes, the source IDS node initiates the malicious gray hole node discovery process by generating a hash function. Then, it sends the data control packet to the next hop along the route (\( {\text{ID-NEXT}} \)). Upon receiving the data control packet, each node needs to extract the hash function and generate a new data control packet with its own specifications. Then, send the hash function output back to the source node and send the new control packet to the next node. The output of the data packet which is sent along the reverse route is considered as the response for the original data packet. Once this response is received, if the received output is the same as the encrypted message then the source node updates its monitoring table and increases the \( {\text{RREQ}}\_{\text{C}} \) and \( {\text{RREQ}}\_{\text{T}} \) columns corresponding to that node by one. This is because the data packets have been sent correctly. This process is repeated along the route to the destination until one of the following occurs:

  1. 1.

    The data packet reaches the destination: in this case, the destination sends the ACK message with the correct output back to the source node along the reverse route. This means that the route is safe and there are no malicious nodes along this route.

  2. 2.

    The received output is not the same as the encrypted message that was sent: In this case, the next hop node which has sent the output message (ID-Next) will be identified as the malicious node. The source node will identify this malicious node by its ID.

  3. 3.

    The next intermediate node does not send a response for the data packet: this might be because either the node itself is malicious or the node before it is a malicious node, and although it has sent a response back to the source, it has prevented the message from being sent forward. Therefore, the source node needs to verify whether the node itself or the previous node is malicious. So, the source node asks both of the nodes to send their monitoring table and reviews their monitoring tables.

If the column \( {\text{RREQ}}\_{\text{T}} \) is set to 1 in node \( X \) and column \( {\text{RREQ}}\_{\text{C}} \) is set to zero in node \( Y \) (node \( Y \) is the neighbor of node \( X \)), then node \( Y \) is a malicious node. By carrying out this process, all of the malicious nodes on every route will be identified. These nodes must be introduced to the entire network.

4.5 Phase 4: Malicious thing elimination process in MTISS-IoT

In the proposed method, after the malicious nodes are discovered by the source IDS node, the source node broadcasts a message containing the ID of the malicious node in the network in order to remove that node from the routing process in the entire network. Once the malicious node is identified and separated from the network, all of the nodes separate any routing information containing the malicious node from their routes and future RREP packets will not include the malicious node.

The proposed MTISS-IoT method tries to select the best and most reliable routes for data transmission by monitoring intermediate nodes. Furthermore, MTISS-IoT does not stop there and verifies the routes and intermediate nodes by sending test and control packets. The proposed method also discovers the malicious gray hole node and removes it from the routing process by using the monitoring table and sending encrypted control packets. The flowchart for the proposed MTISS-IoT method is presented in Fig. 6.

5 Performance evaluation

In following section, we show and discuss the experimental simulation results of the MTISS-IoT to prevent gray hole attack.

5.1 Performance metrics

In following section, we investigate the performance and effectiveness of the MTISS-IoT by a numerical simulation in NS-2. The results are compared with REATO and IRAD, and IMLADS approaches proposed in [13,14,15], respectively. The notations considered for MTISS-IoT are given in Table 8.

Table 8 Abbreviated notations

FPR The FP is calculated by the total number of things wrongly detected as the malicious things divided by the total number of normal things [16,17,18]. Therefore, the \( {\text{FPR}} \) is defined as illustrated in Eq. (5).

$$ {\text{FPR}} = \left( {\frac{\text{FPR}}{{{\text{FPR}} + {\text{TNR}}}}} \right)*100\quad {\text{where}}\quad {\text{TNR}} = \left( {\frac{\text{TNR}}{{{\text{TNR}} + {\text{FPR}}}}} \right)*100. $$
(5)

FNR The rate of the malicious thing to total normal things that were mistakenly marked as a normal thing [19,20,21]. Equation (6) demonstrates the FNR calculation.

$$ {\text{FNR}} = \left( {\frac{{{\text{TPR}} + {\text{TNR}}}}{\text{All}}} \right)*100\quad {\text{where}}\quad {\text{TPR}} = \left( {\frac{\text{TPR}}{{{\text{TPR}} + {\text{FNR}}}}} \right)*100. $$
(6)

DR It is determined as the ratio of the number of gray hole attack nodes marked to the total number of the existing gray hole attack nodes in the IoT. DR is calculated by Eq. (7). Table 9 lists the parameters used for DR [22,23,24,25,26,27].

Table 9 The parameters specified for DR
$$ {\text{DR}} = \left( {\frac{\text{TPR}}{{{\text{TPR}} + {\text{FNR}}}}} \right)*100\quad {\text{where}}\quad {\text{All}} = {\text{TPR}} + {\text{TNR}} + {\text{FPR}} + {\text{FNR}} . $$
(7)

5.2 Simulation setup and comparing algorithms

Because of the difficulty in debugging and implementing IoTs in real networks, it is necessary to view simulations as a basic design tool. The primary benefit of simulation is that analysis is simplified and protocol is verified; mostly, it is evident in systems in large scales [28,29,30,31,32,33,34,35]. The performance of the suggested method is assessed in this part by the use of NS-3 as the simulation means, and the discussion on the obtained results is presented. It should be noted that it is assumed that all REATO, MTISS-IoT, IMLADS, and IRAD settings and parameters are equal.

5.3 Simulation results and analysis

In this section, we analyze the security performance of MTISS-IoT under the four attack scenarios (described in Table 10). This attack is categorized into DoS attack. There are 500 IoT things uniformly deployed in the network area initially. Some important parameters are listed in Table 10.

Table 10 Setting of simulation parameters

The main simulation settings for four scenarios are summarized in Table 11.

Table 11 The setting of simulation parameters for four scenarios

Tables 12, 13, and 14 compare the performance of MTISS-IoT with that of REATO, IRAD, and IMLADS in terms of FPR, FNR, and DR.

Table 12 FPR vs misbehaving thing ratio
Table 13 FNR vs misbehaving thing ratio
Table 14 DR vs misbehaving thing ratio

Average values of all methods for all metrics under gray hole attack are shown in Table 15.

Table 15 Average values of all methods (24% malicious things)

According to Table 16, the average delay of all approaches is large at the first; however, after a while and few iterations, the decision-making time is reduced. The reason for this reduction is that in this situation, the convergence and delay become static. Totally, based on this table, we can conclude that the delay of MTISS-IoT is the least among considered approaches.

Table 16 Average delay (ms) comparison of MTISS-IoT with other approaches

In the following, we evaluate the performance of the proposed MTISS-IoT method in the situation a gray hole attack occurs in the IoT network. The results are presented in Table 17.

Table 17 Average values for all metrics under gray hole attack

FPR Figure 7 presents the FPR of nodes participating in data sending/receiving operations in MTISS-IoT, REATO, IRAD, and IMLADS methods based on network topology. As observed in Fig. 7a, the FPR produced by the suggested design showed small and mild growth in comparison with the other three designs if the malicious things’ rate is raised from 8 to 24% and the range of normal things is from 50 to 500. The FPR of the suggested MTISS-IoT is below 12% while the number of normal things is 500, and the malicious things’ rate is 8%. Nevertheless, it is set to 47% for IMLADS, 24% for the REATO, and it is 31% for the IRAD. The suggested design is superior because of the quick discovery of malicious things and their elimination by normal things and source thing cooperation. Moreover, it is also because the suggested algorithm discovers gray hole attack and separates them from the IoT network. Thus, the FPR, which happens due to attacks, is reduced. As shown in the figure, MTISS-IoT decreases the FPR by more than 17, 26, and 32% those of REATO, IRAD, and IMLADS models, respectively.

Fig. 7
figure 7

Comparison of the MTISS-IoT proposed scheme, REATO, IRAD, and IMLADS approaches in terms of FPR

FNR: Figure 8 shows the comparison of the MTISS-IoT proposed scheme, REATO, IRAD, and IMLADS models in terms of FNR in gray hole attack, (a) number of things (8% malicious), (b) number of things (16% malicious), (c) number of things (24% malicious), and (d) misbehaving things ratio, respectively. As indicated by the diagrams, the MTISS-IoT design’s FNR has raised a little. However, its value is so higher in the IMLADS, IRAD, and REATO. As observed in Fig. 8a, when the number of normal things is 500, the suggested design’s FNR is below 18%. However, it is 25, 30, and 40% for the other three methods. In Fig. 8b, when the malicious things’ rate is 16%, it is below 22% in the suggested schema while it is 24, 27, and 46% for the other three approaches. In Fig. 8c, d we observe that the adaptation capability of MTISS-IoT is higher than that of other approaches. This superior performance can be attributed to mainly MTISS-IoT detection scheme.

Fig. 8
figure 8

Comparison of the MTISS-IoT proposed scheme, REATO, IRAD, and IMLADS approaches in terms of FNR

DR: Figure 9 shows the comparison of the MTISS-IoT proposed scheme, REATO, IRAD, and IMLADS models in terms of DR, (a) number of things (8% malicious), (b) number of things (16% malicious), and (c) number of things (24% malicious), and (d) misbehaving things ratio, respectively. The diagrams indicate that the detection rate in the three approaches is decreased according to two scenarios, particularly when there are a high number of attacks. This decrease is evident more in the REATO compared to other mechanisms. The suggested schema is able to discover all of these attacks with a detection rate above 87.5%. When the number of normal things is 500 and the malicious things’ rate is 16%, this result can be realized. When there are 24% malicious nodes in the IoT network which are using MTISS-IoT, the DR is about 83.13% as shown in Fig. 9. In the case of REATO, IRAD, and IMLADS, the DR for all number of things is about 71%, 61%, and 56%, respectively.

Fig. 9
figure 9

Comparison of the MTISS-IoT proposed scheme, REATO, IRAD, and IMLADS approaches in terms of detection rate

The comparison results of the MTISS-IoT, in terms of PDR at different percent of gray hole attack, are provided in Fig. 10.

Fig. 10
figure 10

PDR vs different numbers of nodes at different percent of gray hole attack in MTISS-IoT

6 Conclusion and future work

By an increase in the use of IoT as well as to an easy implementation of these networks, these networks are being increased by day-to-day. Therefore, the security was known as a necessary need for providing the protected communications among IoT nodes. In order to overcome the challenges, there is a need to create a secure novel intelligent agent-based strategy achieving both vast protected mode and the performance of the desired networks. In this study, a multi-level trust-based intelligence schema using the cryptographic authentication was proposed to avoid gray hole attacks in IoT. The proposed scheme first verifies the trust level for the nodes in the network and then discovers and eliminates the malicious gray hole nodes using control packets. We investigated the MTISS-IoT scheme performance using NS-3. According to the results of the simulation, the MTISS-IoT was highly powerful against gray hole attack. It was demonstrated that it enjoys a low FPR (below 11.104%) and a high level of security, high detection rate (above 94.50%), and low FNR (below 17.49%) in comparison with present methods. In the future work, the use of firefly optimization is suggested to further reduce consumption energy and malicious attacks on the internet of things. Firefly algorithm is proposed to cluster nodes and authenticate in two levels to prevent from attacks.