1 Introduction

The wireless multi-hop networks and devices are becoming increasingly popular and they provide users access to information and communication anytime and anywhere. Now, Mobile Ad-hoc Networks (MANETs) are continuing to attract attention for their potential use in several fields such as collaborative computing and communications in smaller areas (building organizations, conferences, and so on). Communications in battlefields and disaster recovery areas are other examples of application environments. Similarly, communications using a network of sensors or using floats over water are other potential applications. The increase use of collaborative applications and wireless devices may further add to the needs and usage of Ad-hoc networks and MANETs. It should be noted that mobility and the absence of any fixed infrastructure make MANETs very attractive for mobility and rescue operations and time-critical applications.

MANETs are a collection of wireless mobile terminals that are able to dynamically form a temporary network without any aid from fixed infrastructure or centralized administration. Most of the work in MANETs has been done in simulation, as in general, a simulator can give a quick and inexpensive understanding of protocols and algorithms. However, experimentation in the real world are very important to verify the simulation results and to revise the models implemented in the simulator. A typical example of this approach has revealed many aspects of IEEE 802.11, like the gray-zones effect (Lundgren et al. 2002), which usually are not taken into account in standard simulators, as the well-known ns-2 simulator. So far, we can count a lot of simulation results on the performance of MANETs, e.g. in terms of end-to-end throughput, delay and packet loss. However, in order to assess the simulation results, real-world experiments are needed and a lot of testbeds have been built to date (Kiess and Mauve 2007). The baseline criteria usually used in real-world experiments is guaranteeing the repeatability of tests, i.e. if the system does not change along the experiments. How to define a change in the system is not a trivial problem in MANETs, especially if the nodes are mobile.

In this paper, we concentrate on the performance analysis of a small testbed of five computers acting as nodes of a MANET. We use Optimized Link State Routing (OLSR) and Better Approach To Mobile Ad-hoc Networking (B.A.T.M.A.N.), which are pro-active routing protocols and they are gaining great attention within the scientific community. Furthermore, the olsrd (Tønnesen A, OLSRd: Implementation code of the OLSR. http://www.olsr.org/) and batmand (Neumann et al. BATMANd: implementation code of the B.A.T.M.A.N. https://www.open-mesh.net/batman/) software we have used in our experiments are the most updated software we have encountered.

In our previous work, we proved that while some of the OLSR’s problem can be solved, for instance the routing loop, this protocol still have the self-interference problem. Moreover, there is an intricate inter-dependence between MAC layer and routing layer, which can lead the experimenter to misunderstand the results of the experiments. For example, the horizon is not caused only by IEEE 802.11 Distributed Coordination Function (DCF), but also by the routing protocol.

The structure of the paper is as follows. In Sect. 2, we present the related work. In Sect. 3, we give a short description of OLSR and B.A.T.M.A.N.. In Sect. 4, we present the testbed system description and explanation about the implementation of the testbed interface. In Sect. 5, we present experimental evaluation. Finally, conclusions are given in Sect. 6.

2 Related work

Many researchers performed valuable research in the area of wireless multi-hop network by real experiments or computer simulations (Nordström 2002; Owada et al. 2005). Most of them are focused on throughput improvement, but they do not consider mobility (Draves et al. 2004). In Bicket et al. (2005), the authors implemented multi-hop mesh network called MIT Roofnet. Roofnet consists of about 50 nodes. They consider the impact of node density and connectivity in the network performance. The authors show that the multi-hop link is better than single hop link in terms of throughput and connectivity. In Maltz et al. (2001), the authors analyze the performance of an outdoor ad-hoc network, but their study is limited to reactive protocols such as Ad hoc On Demand Distance Vector (AODV) (Perkins et al. 2003) and Dynamic Source Routing (DSR) (Johnson et al. 2001). In Gray et al. (2004), the authors perform outdoor experiments of non standard pro-active protocols. Other ad-hoc experiments are limited to identify MAC problems, by providing insights on the one-hop MAC dynamics as shown in Anastasi et al. (2005).

The closest work to ours is that in Kawadia and Kumar (2005). However, the authors did not care about the routing protocol. In Clausen et al. (2001), the disadvantage of using hysteresis routing metric is presented through simulation and indoor measurements. Our experiments are concerned with the interaction of transport protocols and routing protocol. Furthermore, we compare the performance of the testbed for different scenarios.

In Johnson and Hancke (2009), the authors presents an experimental comparison of OLSR using the standard hysteresis routing metric and the Expected Transmission Count (ETX) metric in a 7 by 7 grid of closely spaced Wi-Fi nodes to obtain more realistic results. The throughput results are similar to our previous work and are effected by hop distance (De Marco et al. 2007). In Hanashi et al. (2009, 2007), the authors propose a dynamic probabilistic broadcasting scheme for mobile ad hoc networks where nodes move according to different mobility models. Simulation results show that their approach outperforms the FP-AODV and simple AODV in terms of saved rebroadcast under different mobility models. It also able to achieve higher saved rebroadcast and low collision as well as low number of relays than the fixed probabilistic scheme and simple AODV.

3 Routing protocols

3.1 OLSR

The link state routing protocol that is most popular today in the open source world is OLSR from olsr.org. OLSR with Link Quality (LQ) extension and fisheye-algorithm works quite well. The OLSR protocol is a pro-active routing protocol, which builds up a route for data transmission by maintaining a routing table inside every node of the network. The routing table is computed upon the knowledge of topology information, which is exchanged by means of Topology Control (TC) packets. OLSR makes use of HELLO messages to find its one hop neighbors and its two hop neighbors through their responses. The sender can then select its Multi Point Relays (MPR) based on the one hop node which offer the best routes to the two hop nodes. By this way, the amount of control traffic can be reduced. Each node has also an MPR selector set which enumerates nodes that have selected it as an MPR node. OLSR uses TC messages along with MPR forwarding to disseminate neighbor information throughout the network. Host Network Address (HNA) messages are used by OLSR to disseminate network route advertisements in the same way TC messages advertise host routes.

OLSRv2 is currently being developed at IETF. It maintains many of the key features of the original protocol including MPR selection and dissemination. Key differences are the flexibility and modular design using shared components such as packet format packetbb and neighborhood discovery protocol.

In our OLSR code, a simple RFC-compliant heuristic is used (Clausen and Jacquet 2003) to compute the MPR nodes. Every node computes the path towards a destination by means of a simple shortest-path algorithm, with hop-count as target metric. In this way, a shortest path can result to be also not good, from the point of view of the packet error rate. Accordingly, recently olsrd has been equipped with the LQ extension, which is a shortest-path algorithm with the average of the packet error rate as metric. This metric is commonly called as the ETX, which is defined as ETX(i) = 1/(NI(i) × LQI(i)). Given a sampling window W, NI(i) is the packet arrival rate seen by a node on the i-th link during W. Similarly, LQI(i) is the estimation of the packet arrival rate seen by the neighbor node which uses the i-th link. When the link has a low packet error rate, the ETX metric is higher. The LQ extension greatly enhances the packet delivery ratio with respect to the hysteresis-based technique (Douglas et al. 2003).

3.2 B.A.T.M.A.N

In OLSR, there is a serious synchronization problem between the topology messages and the routing information stored inside every node. In other words, a mismatch between what is currently stored in the routing tables and the actual topology of the network may arise. This is due to the propagation time of the topology messages. Routing loops are the main effect of such problem. To solve this problem, B.A.T.M.A.N. has been introduced. In B.A.T.M.A.N., there is no topology message dissemination. Every node executes the following operations:

  1. 1.

    Sending of periodic advertisement messages, called OriGinator Message (OGM). The size of these messages is just 52 bytes, containing: the IP address of the originator, the IP address of the forwarding node, a TTL value and an increasing Sequence Number (SQ).

  2. 2.

    Checking of the best one-hop neighbor for every (known) destination in the network by means of a ranking procedure.

  3. 3.

    Re-broadcasting of OGMs received via best one-hop neighbor.

The B.A.T.M.A.N. uses a timer for sending OGMs and SQ (OGM) for checking the bi-directionality of links. If the SQ of an OGM received from a particular node falls within a certain range, the corresponding link is considered bi-directional. For example, suppose that in a time interval T, the node A sends Tr messages, where r is the rate of OGM messages. The neighbors of A will re-broadcast the OGMs of A and also other node’s OGMs. When A receives some OGMs from a neighbor node, say B, it checks if last received OGM from B has a SQ less or equal to Tr. If it does, then B is considered bi-directional, otherwise it is considered unidirectional. Bi-directional links are used for the ranking procedure. The quantity Tr is called bi-directional sequence number range. The ranking procedure is a substitute of the link quality extension of OLSR. In few words, every node ranks its neighboring nodes by means of a simple counting of total received OGMs from them. The ranking procedure is performed on OriGinator (OG) basis, i.e. for every originator. Initially, for every OG, every node stores a variable called Neighbor Ranking Sequence Frame (NBRF), which is upper bounded by a particular value called ranking sequence number range. We suppose that there is a rank table in every node which stores all the information contained in the OGMs. Whenever a new OGM is being received via a bi-directional link, the receiving node executes the following steps.

  1. 1.

    If the sequence number of the OGM is less than the corresponding NBRF, then drop the packet.

  2. 2.

    Otherwise, update the NBRF = SQ (OGM) in the ranking table.

  3. 3.

    If SQ (OGM) is received for the first time, store OGM in a new row of the rank table.

  4. 4.

    Otherwise, increment by one the OGM count or make ranking for this OGM.

Finally, the ranking procedures select the best one-hop neighbor the neighbor which has the highest rank in the ranking table. Let us note that the same OGM packet is used for: link sensing, neighbor discovery, bi-directional link validation and flooding mechanism. While this feature eliminates routing loops because no global topology information are flooded, the self-interference due to data traffic can cause oscillations in the throughput as we will see in our experiments. Other details on B.A.T.M.A.N. can be found in Haas et al. (2006). In B.A.T.M.A.N., every node re-broadcasts received OGMs only once, and only those OGMs, which have been received via the best-ranked neighbor. This is a kind of selective flooding, which practically reduces the overhead of the flooding. Another analogy can be found in gossip protocols (Haas et al. 2006). In gossip protocol, every node decides to re-broadcast received data with some probability, p. This is equivalent with eliminating some links in the network and then supposing that every node re-broadcast with probability 1. In gossip protocol there is a threshold for p and the density of nodes after which the success ratio Footnote 1 is almost surely 1. In B.A.T.M.A.N., the probability p is changed according to ranking procedure. It is the probability that an OGM is reached via the best rank neighbor. The expression of this probability is left for further analysis. Let us note that the selective flooding eliminates possible misbehavior of the ranking procedure. In fact, cumulative count of the OGM could be greater than the total number of OGM received via the current best neighbor.

4 Testbed design and implementation

4.1 Target environment

We have implemented a MANETs testbed which provides a realistic platform for analysing various aspect of these networks, including the different topology models. For our testbed, we make the following considerations.

  • We consider an indoor environment at our departmental floor.

  • We investigate the effect of mobility and topology changing in the throughput of MANETs testbed.

  • We constructed three experimental models: Model 1 (all nodes are in stationary state); Model 2 (only one relay node is moving); Model 3 (three nodes are in stationary state and two relay nodes are moving).

  • The mobile nodes move toward the destination at a regular speed. When the mobile nodes arrive at the corner, they stop for about 3 s.

  • In order to make the experiments easier, we implemented a testbed interface and web tool (see Fig. 1).

  • Experimental time is 10 s.

Fig. 1
figure 1

Testbed system overview

Full size image

4.2 Testbed description

Our testbed is composed of four laptops and one gateway (GW) Footnote 2 machine as shown in Figs. 2, 3, 4. We constructed three experimental models. The experimental parameters are shown in Table 1. In Fig. 2, all nodes are in a stationary state. We call this model STA. The node #2 position and movement are shown in Fig. 5. In Fig. 3, only one relay node (node id #2) is moving. The mobile node moves toward the destination at a regular speed. When the mobile node arrives at the corner, it stops for about 3 s. The round trip time is 50 s. We call this model MV1. In the third model, two relay nodes (nodes id #2 and #3) are moving (see Fig. 4). We call this model MV2.

Fig. 2
figure 2

STA model. Node #1 is accessible via node #2 and #3. When the destination node is #4, the hop distance is 2, i.e. \(1\rightarrow2\rightarrow4\)

Full size image
Table 1 Classification of nodes for each experimental model
Full size table
Fig. 3
figure 3

MV1 model. Only one relay node #2 is moving. Round trip time is 50 s

Full size image
Fig. 4
figure 4

MV2 model. Between nodes there are some obstacles (metal doors and walls)

Full size image
Fig. 5
figure 5

Snapshot of node #2. All laptops are on a chair

Full size image

The operating system mounted on these machines is Fedora Core 4 or Ubuntu 9.04 Linux with kernel 2.6.x, suitably modified in order to support the wireless cards. The wireless network cards are from Linksys. They are usb-based cards with and external antenna of 2 dBi gain, transmitted power of 16 ± 1 dBm and receive sensitivity of −80 dBm. We verified that the external antenna improves the quality of the first hop link, which is the link connecting the ad-hoc network. The driver can be downloaded from the web site in references (Ralink RT2570 USB enhanced driver. http://homepages.tu-darmstadt.de/~p_larbig/wlan/; Rt2x00 project. http://rt2x00.serialmonkey.com/)Footnote 3.

The source node serves as HTTP, FTP, DNS and Internet router for the nodes in the MANET. These features are provided by the iptables mechanism, readily available under Linux machines. The iptables is a user space command line program used to configure the Linux 2.4.x and 2.6.x IPv4 packet filtering rule-set. By this way, the GW can be accessed ubiquitously from anywhere. Moreover, the GW hosts also all the routines used to coordinate the measurement campaign, as well as graphical tools to check network connectivity. In our testbed, we have two systematic background or interference traffic we could not eliminate: the control traffic and the other wireless Access Points (APs) interspersed within the campus. The control traffic is due to the ssh program, which is used to remotely start and control the measurement software on the source node. The other traffic is a kind of interference, which is typical in an academic scenario.

4.3 Testbed interface

Until now, all the parameters settings and editing were done by using command lines of bash shell (terminal), which resulted in many misprints and the experiments were repeated many times. In order to make the experiments easier, we implemented a testbed interface. For the Graphical User Interface (GUI) we used wxWidgets tool and each operation is implemented by Perl language. wxWidgets is a cross-platform GUI and tools library for GTK, MS Windows and Mac OS X.

We implemented many parameters in the interface such as transmission duration, number of trials, source address, destination address, packet rate, packet size, LQWS, and topology setting function. We can save the data for these parameters in a text file and can manage in a better way the experimental conditions. Moreover, we implemented collection function of experimental data in order to make easier the experimenter’s work.

5 Experimental results

5.1 Experimental settings

The experimental parameters are shown in Table 2. We study the impact of best-effort traffic for two link state routing protocols. We collected data for three metrics: the throughput, round trip time and packet loss. These data are collected using the Distributed Internet Traffic Generator (D-ITG), which is an open-source Internet traffic generator (Botta et al. 2007). D-ITG computes the packet loss as the number of lost packet divided by the effective number of sent packets.

Table 2 Experimental parameters
Full size table

In previous experiments (De Marco et al. 2007; Ikeda et al. 2008, 2009), we realized that an external antenna improves the received radio signal. The Constant Bit Rate (CBR) of the data flows is 122 pps equal to 499.712 Kbps, i.e., the packet payload is 512 bytes. All experiments have been performed in indoor environment within our departmental floor of size roughly 100 m. All nodes are in radio range of each other. We measured the throughput for UDP, which is computed at the receiver.

As MAC protocol, we used IEEE 802.11. The transmission power was set in order to guarantee a coverage radius equal to the maximum allowed geographical distance in the network. Since we were interested mainly in the performance of the routing protocol, we kept unchanged all MAC parameters, such as the carrier sense, the retransmission counter, the contention window and the Request to Send (RTS) / Clear to Send (CTS) threshold. Moreover, the channel central frequency was set to 2.412 GHz (channel 1). In regard to the interference, it is worth noting that, during our tests, almost all the IEEE 802.11 spectrum had been used by other APs disseminated within the campus. In general, the interference from other APs is a non-controllable parameter.

5.2 Experimental measurements

Results of our measurements are shown in Table 3. The observed outcome of a treatment is the tetralogy of the metrics medians, \((\widehat{T},\widehat{RTT},\widehat{J},\widehat{P_L}),\) except the case of the second group, i.e. with the TCP, where obviously the packet loss is always 0. From Table 3, by looking at "Factor" item, we can see that there is a significant difference in the treatments (T) of the first group (from A to F) where we use MV1 model. For instance, in treatments C, we have three factors: UDP data flow, MV1 model and B.A.T.M.A.N. protocol. All results showed the median values. In Table 3, \(1\rightarrow2\) means source node id → destination node id.

Table 3 Each treatments and groups of median results \(({\user2{\widehat{T}}}, {\user2{\widehat{RTT}}}, {\user2{\widehat{J}}}, {\user2{\widehat{P_L}}})\)
Full size table

In order to show the range of variability of the data, we also report the box and whisker plot of the metrics according to the model types, as shown in Figs. 6 and 7. Box and whisker plot is a convenient way to show groups of numerical data by lower quartile (Q1), median (Q2), upper quartile (Q3), and the outliers. In the plot, the bottom and top of the box are always 25th and 75th percentile (Q1 and Q3, respectively), and the band near the middle of the box is always the median (Q2). The end of the whiskers can represent the lowest datum which is still within 1.5 inter-quartile range of the lower quartile, and the highest datum which is still within 1.5 inter-quartile range of the upper quartile.

Fig. 6
figure 6

Results for throughput (MV1)

Full size image
Fig. 7
figure 7

Results for throughput (MV2)

Full size image

In Fig. 6 and Fig. 7, the horizontal axis show the hop distance (1 → 2 means source node id → destination node id) and the vertical axis shows the throughput (Kbps), which is computed at the receiver. In Fig. 6, we show the throughput results for MV1 (one mobile node: treatments A–F). In groups D and E, the TCP overhead is high, so the throughput is decreased compared with UDP (groups A and B). Moreover, the throughput drop about 50% after the third hop. Let us note that this happens also for group E. It seems that the topology could not exploit direct links, e.g. from host 1 to 5. In this case, the hop-count can change because of impairments of the radio links and/or MAC problems, such as the gray-zones problem. The high variability for \(1\rightarrow5\) connection in group F confirms this fact. Due to the fixed sampling window of the link quality sensing mechanism, nodes use routes with low quality. Consequently, a dynamic adaptation of the neighbors sensing messages rate could ameliorate the situation.

In Fig. 7, we show the throughput results for MV2 (two mobile nodes: treatments G– L). In treatments G, H and I, the results of UDP are almost the same. In other words, when we use UDP data flow with different protocols or different LQWS, the throughput results are not affected. Also, the experimental results for models MV1 and MV2 have the same tendency. On the other hand, when we use TCP data flow, the experimental results are affected by LQWS. If we use small LQWS, then we have high throughput, so the \(\widehat{RTT}\) is decreased compared with LQWS100. From experimental results, we found that for OLSR if we use TCP data flow, we got better results when the LQWS value was 10.

In Fig. 8, we report the results of all cases for B.A.T.M.A.N. (1 Flow case, 1:2 Main case and 1:2 Background case) in term of median values. In particular, we use the median of the goodput. If the median is zero, it means that there are a lot of oscillations. We consider Mesh Topology (MT) and Linear Topology (LT). The MT introduces more interference and the median of the goodput can be zero (i.e., there are a lot of oscillations). For 1:2 flow case, things get more complicated. In this case the aggregate bitrate used by the gateway is 2·122 = 244 pkt/s ≈ 1 Mbps. In this case, the first hop is saturated, because it can get 1/3 of the total bandwidth. The additional B.A.T.M.A.N. messages rate is neglectable. For MT case, the goodput has more oscillations than LT. The cause of these remarked oscillations could be the augmented self-interference of data traffic.

Fig. 8
figure 8

Throughput/Goodput fairness results for B.A.T.M.A.N. (STA). The throughput is expressed as the median of the raw data. The background flow is always from the gateway towards the node at 5 hop-distance

Full size image

For TCP, we can see that there is no horizon effect, but only oscillations and outliers. This fact can be explained by observing that TCP inherently self-reduces the send rate when losses arise. Due to its congestion control, TCP can smooth the number of packets per second injected into the network. To demonstrate this fact, we also traced the congestion window of TCP at the gateway by means of tcptrace . The results are shown in Fig. 9. We can see that there is a correlation between the magnitude of oscillations, and the number of outliers with the magnitude of the maximum of congestion window. For example, the higher is the upper bound of the congestion window, the higher is the number of outliers and/or the coefficient of variation of the throughput. In all cases, the median of the throughput is roughly 122 Kbps. However, the share of the bandwidth is not equal between the two flows, as we can see in Fig. 8, for all four treatments. In one case, it seems to be fair share of the bandwidth, and it is the case of A treatment, i.e. UDP in LT. In this case, both flows are “forced” to follow the same path and at the same bitrate, then our claim is that this case is equivalent to the case of one flow but with higher bitrate Footnote 4. Anyway, there is a misleading result for the hop distance 4, where the goodput is higher than that of the other nodes. Maybe, in this case shadowing effects could mask the signal of node 5. In summary, we claim that the outliers are due to OGMs rate, while oscillations are due to self-interference.

Fig. 9
figure 9

Maximum of the TCP congestion window measured at the gateway

Full size image

6 Conclusions

In this paper, we carried out experiments with a small MANET with five nodes. We used OLSR and B.A.T.M.A.N. protocols for experimental evaluation. In our experiments, we considered three models: STA, MV1 and MV2. In STA, all nodes are in stationary state. In MV1, one relay node is moving. In MV2, two relay node are moving. We assessed the performance of our testbed in terms of throughput, RTT, jitter and packet loss.

From our experiments, we found the following results.

  • For OLSR if we use TCP data flow, we got better results when the LQWS value was 10.

  • There are some oscillations in each model. This is because of hop distance and environment.

  • The number of packet loss increases after the third hop.

  • The OLSR protocol has not a good performance when the relay node is moving.

  • For mobile nodes, OLSR protocol has better goodput than B.A.T.M.A.N. protocol.

In this work, we considered only proactive routing protocols. In the future, we would like to consider the effect of reactive protocols and compare the experimental results with simulation results. Moreover, we would like to consider a new link quality metric and extend our testbed.