Keywords

1 Introduction

Large systems require rigorous testing before commissioning. Let us consider the distributed application, in addition to the correct operation of each of its components, and it is necessary to take into account network interactions. In this work, we focus on testing of such communications.

There are several modelling tools   [8] (for example, GNS3  [2], eve-ng  [1]) with which you can deploy a network of virtual machines and test the application on it. However, these testing systems require the user to know in the field of system administration.

Applications can be tested on real users using tools like Pumba  [6] or Gremlin  [3], but such a solution carries the risk of losing users’ interest in the service.

The main goal of our work is to provide an easy way to model a scalable network. An application developer does not require in-depth knowledge of network technology to use the tools described in this paper. We propose to consider a model of a network consisting of Docker image nodes and routers. This approach allows emulation of a distributed system locally without going into the details of network interactions. The model enables reproducing effects observed by developers, such as duplication, loss, delay, or distortion of data packages.

2 Related Work

There are about two dozen emulators of distributed networks  [24]. Most of them are free and available in the public domain (including GitHub). Also, HP, Huawei, Cisco have their network emulators oriented to work with equipment of a specific manufacturer. We considered the most popular network modelling environments. We selected some characteristics for comparison (Table 1): ease of use, scalability, and versatility. The ease of use of the tool is evaluated by the following criteria: configuration format, the availability of visualization, automatic deployment, and launch of distributed applications, the ability to access applications on the Internet.

There is no universal solution for emulating large-scale systems. In some cases, preference is given to modelling process details of the network layer L2; in other cases, the interaction is considered at the level of model abstractions. L3NS  [4] and MADT  [5] solutions allow while preserving the complexity of the services under consideration, to simplify the network abstraction as much as possible, providing the availability of modelling on one or several computing nodes and platforms.

Table 1. Comparative table of existing modeling technologies
Full size table

3 Problem Definition

Our goal is to create a universal, scalable solution with which you can quickly simulate the interaction of network components and monitor or change the state of the model in real-time.

Fig. 1.
figure 1

Technology stack used by MADT and L3NS.

Full size image

To emulate a virtual computer network, the Docker containerization  [18] platform is used. Moreover, when the model is run on a cluster WireGuard  [15] is used to model IP subnets. Also, a set of FRRouting utilities is used, which allows the implementation of dynamic routing protocols, and the Tcconfig shell, which allows to control  [14] the quality of the network dynamically and which is used to improve the realism of the model. The hypergraph separation algorithm KaHyPar is used to organize distributed modelling of virtual  [20] computer networks. The jgraph, Flask, and zmq messaging libraries are used to operate the user interface (Fig. 1).

Fig. 2.
figure 2

MADT/L3NS workflow: 1. Commands are sent directly to docker server; 2. Docker server creates containers using Linux namespaces; 3. Docker connects containers with virtual LANs; 4. Virtual LANs and docker containers form a virtual global network.

Full size image
Fig. 3.
figure 3

Modeling a distributed network using the WireGuard tool.

Full size image

4 Proposed Architecture

The traditional way of modelling computing nodes of a network  [12] is the use of hypervisors that manage virtual machines. The kernels of the operating systems are separated; this provides isolation for applications  [22]. In case of modelling a network of interacting application services, nodes and applications do not need isolation at the kernel level of the OS, isolation of the network stack that the computing node  [11] use is sufficient.

The Linux kernel provides a tool (Namespaces)  [13] that allows isolation of the low-level environments  [17]. A container is an isolated set of namespaces on the top of which an application is running. The Docker configuration platform allows to create such environments corresponds to predefined templates and run applications in them. Besides, using Docker, we can combine network interfaces located in different network namespaces with a virtual bridge into one common IP subnet, the hosts of which are containers. Our solutions allow pre-setting of the network topology and launch parameters of containers. This network is launched by Docker  [25] and it turns out a virtual network model that fully works on one computer (Fig. 2). If the simulated application requires a large number of resources (Fig. 3), it is advisable to divide the simulation into several computers. This solves the problem of efficiently sharing the load across hosts to minimize latency in virtual networks connecting containers  [16] located on different hosts. In each of the containers launched as part of the model, a Unix socket is mounted. This allows sending messages to the monitoring system  [26], even if the network interfaces of the container are blocked or have a long delay, thus ensuring stable monitoring  [9]. Using the web interface, you can monitor the state of the model in real-time and interactively manage network parameters for each node, controlling  [19] delays, losses, duplicate packets.

5 Modeling Algorithm

The work proceeds according to the following algorithm:

  1. 1.

    Using the API in Python, a “Laboratory” is created — a description of the network model, which includes a list of network nodes, their distribution over subnets, and startup parameters.

  2. 2.

    “Laboratory” starts by creating network nodes based on Docker containers and network connections between them.

  3. 3.

    Displaying the emulated network is available in the web interface for conducting experiments and monitoring their progress.

6 Implementations of the Proposed Architecture in L3NS and MADT

We have tested several approaches to the implementation of the proposed architecture. The first of them is the MADT program complex, divided into three parts: a library for describing “Laboratories”, a server for launching them, and a monitoring interface with zmq sockets. The second approach is implemented in the form of the L3NS library containing all the components necessary for modelling the network in the format of one programmable solution. While MADT allows to familiarize yourself quickly with the concept of the proposed architecture and set the experiment visually, L3NS has the functionality to more quickly and flexibly describe the model of node interaction and implementation in the CI ecosystem.

7 Example: Modeling in Applications with Client-Server Architecture

Let us consider how, using the proposed concept. A network is built with a set of routers and dynamic routing. Such a system can be represented as a tree, the nodes of which are routers and clients. Figure 4 shows how the network looks in case of height of the tree, and the number of leaves is set by 3 and 3.

Fig. 4.
figure 4

Network model: server in the middle, blue boxes represent routers, yellow circles depict client nodes. (Color figure online)

Full size image

The construction of such a network is carried out according to the following principle: the graph is expanded using several iterations of adding subnets for child nodes. At the end of each iteration, the array of external nodes is updated. For all nodes, except the external ones, the operation of the dynamic routing protocol is configured. This allows traffic to pass through the tree.

Each child node of the graph is assigned a Docker image that is used to create the client node.

Another step that has to be taken for the correct operation of our virtual network is to define the server address for each client so that they can interact.

When the model starts, it becomes possible to set network problems manually and monitor how they affect the application  [21].

8 Modeling Management in the MADT Visual Interface

In MADT, after saving the configuration of the built network, we can go to the graphical interface and monitor the operation of the application. To configure the network status, you have to click on any node and set the interference parameters using the window that opens (Fig. 5a) and then press the “tcset” button. For example, to simulate 50% package loss for a router, we need to set the corresponding value of the loss field.

Fig. 5.
figure 5

MADT visual interface. (Color figure online)

Full size image

After we set the problem for several nodes, we can observe the result on the graph. The colour of the node depends on the status of the incoming message. Green indicates regular operation, and yellow indicates a malfunction, red indicates critical problems, and purple indicates unexpected exceptions (Fig. 5b).

9 Differences in Modelling Capabilities of the L3NS Library and the MADT Software Package

The L3NS code is independent of MADT. The main functional features are the logic of the distribution of IP-addresses. Addresses are distributed according to the description of the “Laboratory”, due to which the IP addresses of some nodes can be used to describe others (for example, immediately define the server address for clients)  [10]. Also, work with Docker networks has been dramatically simplified in L3NS: network IP addresses, as well as the IP addresses of nodes in them, are set directly, while in MADT a second IP address is given to each interface on the network.

When describing network elements, different strategies for starting and stopping nodes can be used in L3NS. For example, using the local Docker container submodule, the nodes are launched as Docker containers. Meanwhile using cluster submodule, the containers are started on remote machines. L3NS provides the ability to configure remote machines via ssh. In other words, it is possible to access the client on a specific host, and, for example, view a list of Docker images used on it. To run the service system model on several virtual machines, you need to:

  • Configure ssh keys,

  • Install Docker,

  • Install dependencies in python,

  • Install WireGuard.

In L3NS support for parallel launching of containers is implemented using ThreadPoolExecutor, which speeds up the launch of “Laboratories” by order of magnitude, as well as networking via Docker Swarm  [23] compared to MADT. The approach based on the creation of a single solution in the form of a library for a user programmer showed greater flexibility in system scaling tasks than the MADT program package, which separates the functions of describing, running and configuring “Laboratories”.

10 Conclusion

The solutions we developed for testing distributed service systems allow to:

  • Create realistic models of large IP networks;

  • Deploy distributed applications based on Docker images;

  • Manage the quality of work of different parts of the simulated network;

  • Visualize the state of a distributed application working in real-time.

Two approaches were implemented — the MADT software package and the self-contained L3NS library. Using the library, it is convenient to test network interaction by emulating the network both on one node and simulating the interaction of nodes in a distributed model. Its use can significantly accelerate the launch of a simulated network without a web interface. While using the complex of programs MADT, it is easy to carry out interactive search experiments. Thus, MADT and L3NS can be used for:

  • Comparing the work of several distributed applications under the same conditions;

  • Checking application stability regarding unstable network operation;

  • Studying the impact of network structure on the operation of each application;

  • Checking for network plan vulnerabilities in application interaction.