Keywords

1 Introduction

The development of Industrial Internet of Things (IIoT) and Cyber-Physical System (CPS) has enabled the manufacturing industry to transform and upgrade to digital, networked, and intelligent, and move towards the industry 4.0 era [1]. By building the networked industrial infrastructure and intelligent production systems, the industrial production efficiency is greatly improved. In IIoT paradigm, industrial intelligent control system is an important part, which can support intelligent industrial production with the abilities of autonomous decision-making and system autonomy [2].

Currently, industrial production presents new trends such as processes upstream and downstream coupling, complex and changeable objects, network coordination, and multiple sources of information. Due to limited computing and storage resources, traditional PLC and DCS control stations cannot cope with the control requirements of complex production processes with nonlinear, time-varying, and distributed parameters. Moreover, it cannot meet the urgent needs of smart factory applications for intelligent perception, autonomous decision-making and network collaboration functions. Therefore, it has become a new development trend of that enable industrial control systems to have intelligent control capabilities with cloud-side collaboration, and achieve real-time perception, real-time control and intelligent analysis close to the industrial field.

In our previous works, we have proposed the principle and system architecture of software-defined edge computing (SDEC) to realize the unified management, reconstruction, sharing, reuse and collaboration of edge device resources [3]. In this paper, combined with the ideas and technologies of SDEC and CPS, we propose the edge computing-based solution and framework of software-defined industrial intelligent control (SDIIC) for the autonomous and intelligent control of industry. It mainly consists of SDEC platform at the software level and software-defined edge controller at the hardware level. The SDEC platform uses techniques such as semantic modeling and knowledge graphs to virtualize, abstract, and digitize hardware devices on the edge. The control and management functions of devices are separated from hardware. Combining device resource scheduling and orchestration, sharing coordination, rule engine and other service capabilities, the edge hardware resources of industrial control system can realize flexible management and autonomous coordination in the way of software.

The scheme and system architecture of software-defined edge controller is designed by adopting a multi-processor hybrid heterogeneous hardware framework. It adopts lightweight virtualization technology to realize the virtualized mapping, scheduling, orchestration and management of hardware resources in the controller, and realizes optimal scheduling and dynamic reconstruction of computing capability in a manner of software definition. It can implement software-defined virtual controllers with differentiated control and computing capabilities on common standardized hardware resources. Software-defined edge controller can simultaneously support real-time task processing such as logic control, process control, and motion control, as well as non-real-time task processing such as industrial vision, deep learning, and intelligent optimization. The hardware requirements of edge intelligent control are met.

The target of SDIIC is to enable the intelligent control, intelligent computing, autonomous collaboration and system autonomy in the edge side. Based on the SDEC platform and software-defined edge controller, software and hardware integrated industrial intelligent control solutions with software definition as the core can be realized to support the development of IIoT.

The remainder of this paper is organized as follows. Section 2 introduces the evolution direction of IIoT architecture from the vertical perspective and horizontal perspective. Section 3 proposes the system architecture of SDIIC. Section 4 proposes the industrial intelligent control oriented SDEC platform scheme. Section 5 proposes the scheme and system architecture of software-defined edge controller. Section 6 draws a conclusion.

2 The Evolution Direction of IIoT Architecture

2.1 Vertical Perspective

As shown in Fig. 1, we compare the current and future IIoT architecture from the vertical perspective. It is mainly reflected in the following aspects:

Fig. 1.
figure 1

Vertical comparison of current and future IIoT architecture.

Full size image

  • The currently closed and chimney-style application development mode will gradually move towards openness, sharing and collaboration.

  • The industrial assets such as hardware resources, data, information, knowledge, and service capabilities are shared and reused by various industrial applications through edge computing platform and cloud platform.

  • The upper-layer applications are decoupled from the lower-layer hardware, and application developers do not need to care about the deployment details of the lower-layer hardware [4, 5].

  • The underlying devices can be freely combined, flexibly arranged, and deployed on demand based on application requirements like “building blocks” [6].

  • The binding hardware-defined mode, which industrial applications and hardware resources are tightly coupled, will gradually transform into a new flexible and programmable software-defined mode [7, 8].

2.2 Horizontal Perspective

We compare the current and future IIoT architecture from the horizontal perspective in Fig. 2. For the traditional five-layer architecture based on ISA-95 standard, although each system has a clear division of labor and is relatively independent, there are still some problems to be solved. For example, data and information island, complex interfaces, data delays, difficulties in data fusion, system lock-in, and lack of interoperability standards. Moreover, most of the industrial control system infrastructure in this architecture mode is based on dedicated hardware, which limits the flexibility of system.

Fig. 2.
figure 2

Horizontal comparison of current and future IIoT architecture.

Full size image

In the future, the smart entity will be the basic component of new IIoT architecture. The smart entity is a system unit with decision-making ability. It includes not only intelligent industrial equipment, but also the end-to-end system composed of computing power, algorithms and terminal devices. The most significant feature of smart entity is the ability to make independent decisions, which can realize end-to-end closed-loop control applications. Multiple smart entities can also realize interconnection and mutual collaboration to implement complex intelligent industrial applications.

The future IIoT will develop in the direction of the autonomous system based on CPS. The autonomy is reflected in two levels.

  • Smart entity level: the smart entity itself can make independent decisions and implement some simple and low-latency intelligent control applications.

  • Edge computing platform and cloud platform level: through the model of “Platform + APPs”, more complex and intelligent industrial applications can be implemented by combining with these technologies and capabilities such as intelligent analysis, multi-source data integration, resource optimization scheduling, and multiple smart entities collaboration.

In the CPS-based autonomous system, on the one hand, smart entities can achieve interconnection, intercommunication, and interoperability through new industrial network technologies (e.g., OPC UA, TSN, 5G, etc.). On the other hand, through digitally modeling the smart device objects in physical space, a one-to-one corresponding digital twin model is formed in information space. On this basis, the CPS-based control system with real-time interaction capabilities between information space and physical space can be constructed to realize intelligent control of industrial production process [9].

3 The System Architecture of Software-Defined Industrial Intelligent Control

Industrial control has experienced three generations of technological evolution in the development history. The first generation is characterized by mechanical control. The second generation is characterized by electronic control. And the third generation is characterized by computer control to solve large-scale loop control problems. Currently, it is evolving to the fourth-generation control system, that is, the industrial intelligent control system, whose core feature is intelligence.

Compared with traditional industrial control systems, the characteristics of new intelligent control are as follows:

  • It can collect various data from the edge side, and also perform real-time processing, analysis and feedback control in the edge side to reduce the manual labor of workers.

  • The traditional industrial equipment can be upgraded to the smart entity in CPS, so that it has the self-management and autonomous operation capabilities with the characteristics of self-awareness, self-adaptation, autonomous control and self-diagnosis.

  • It has the capabilities of edge data collection, edge data processing, edge AI inference and decision-making, real-time control, intelligent coordination among various devices, and edge security.

  • The computing capability of controller hardware is stronger. And it can be flexibly defined and combined with software for different application scenarios to provide flexible control capabilities.

  • Through the cloud-edge collaboration mechanism, the powerful processing capabilities of cloud can be introduced into control system to support complex and diverse industrial applications [10].

As shown in Fig. 3, we propose a system architecture of SDIIC. On the whole, it is an “end-edge-cloud” collaborative architecture. The cloud computing-based industrial internet platform provides intelligent management, industrial intelligence, and big data services for production and operation, and enables intelligent control at the edge. The controller layer includes both software-defined edge controller and traditional PLC/DCS. And software-defined edge controller can be used as edge computing node of traditional PLC/DCS to provide stronger computing capability support. Edge computing platforms are deployed on edge servers, industrial computers and other hardware with strong computing capability to provide information modelling, basic edge services, edge intelligence, cloud-edge collaboration and other capabilities in edge-side.

Fig. 3.
figure 3

System architecture of SDIIC.

Full size image

For the proposed SDIIC solution, it mainly consists of the following two key technologies in both software and hardware.

  • Software level: software-defined edge computing platform. By digitizing, virtualizing, and abstracting the description and modeling of industrial edge hardware devices, the virtual device twin model is formed. The basic capabilities and services of software-defined edge computing platform are constructed to realize the unified management and control, sharing and intelligent collaboration of edge device resources. An edge-side industrial autonomous system is also formed.

  • Hardware level: software-defined edge controller. The edge controller adopts multi-processor hybrid heterogeneous architecture. Using lightweight virtualization technology, the hardware resources (such as computing, storage, network, and IO) in the edge controller are virtualized, scheduled, orchestrated and managed. So the optimal scheduling and dynamic reconstruction of computing capacities are implemented in a software-defined manner. The software-defined virtual controllers have differentiated control and computing capability, and can support both real-time and non-real-time task processing simultaneously to realize industrial intelligent control at the edge.

Relying on the software-defined edge computing platform and the software-defined edge controller, the capabilities of intelligent control, intelligent computing, autonomous coordination and system autonomy are implemented at edge-side. And the software and hardware integrated intelligent control system solution is realized.

4 Industrial Intelligent Control Oriented Software-Defined Edge Computing Platform Scheme

The SDEC is based on the ideas of software definition and CPS. From the perspective of cyber-physical space mapping, the technologies of software definition and virtualization modeling are extended to edge hardware resources (including: terminal device resources, edge control device resources, edge computing resources, edge storage resources) and edge application services. The goal of SDEC is to build an intelligent edge autonomous system to realize intelligent control and closed-loop applications in the edge side [11].

According to the characteristics of SDEC technologies and IIoT applications, we propose the SDEC platform for industrial intelligent control, as shown in Fig. 4. The device resources in the edge side include sensing devices, execution devices, smart entity devices, controllers, edge computing nodes, and edge storage devices. The SDEC solution describes and models these device resources in a digitized, virtualized, and abstract manner. The virtual device twin models are constructed and used as the cornerstone and base of IIoT applications. In SDEC platform, some basic functions and capabilities are implemented, including edge device resource scheduling and orchestration, virtual controller orchestration, lightweight rule engine, lightweight AI inference engine adapted to edge-side, knowledge base/rule library/component library/basic algorithm library, container management/microservice, etc. It realizes unified management and control, sharing and intelligent collaboration of edge-side device resources, and supports edge intelligent applications such as motion control, logic control, equipment fault diagnosis, and industrial machine vision. This solution can realize system autonomy on the industrial edge side, and enable the edge hardware resources of industrial control system to realize flexible management and autonomous coordination in software.

Fig. 4.
figure 4

Technical architecture of SDEC platform.

Full size image

In IIoT application solution based on SDEC platform, on the one hand, the edge hardware devices are abstracted and modeled into virtual device twin models. There is a one-to-one correspondence between physical device and digital twin model [12]. They can synchronize with each other through real-time dynamic interaction. Physical device can synchronize the status and data to digital twin model in real time for platform and application to make decision. At the same time, the digital twin model can also synchronize the decision results of platform and application to physical device to execute decision instructions. To this end, the intelligent industrial control in edge side and the cyber-physical space integration and interaction are realized. On the other hand, the control, management, and scheduling functions of devices are decoupled and separated from hardware and implemented in software. Through the on-demand configuration and dynamic reorganization of virtualized resources, the sharing and reuse of edge hardware resources can be realized. At the same time, the collaboration mechanism, linkage mechanism and control logic between physical devices become editable and executable in the form of knowledge base and rule library. Combined with the basic capabilities of rule engine and inference engine, intelligent collaboration and system autonomy in edge side can be implemented. Through the unified interface (API) definition and encapsulation of device twin model, knowledge and rule base, edge basic services and capabilities, the upper-level applications can easily call various models, data and basic services without paying too much attention to the deployment details of the underlying hardware devices. This reduces the amount of code development and simplifies application development and deployment.

The SDEC platform can connect into the industrial cloud platform to collaborate with the remote industrial cloud services. Depending on the capabilities of cloud-based big data training, multi-source data fusion analysis, global management and control, cloud-edge collaboration-based intelligent production line control and optimization can be implemented.

5 Software-Defined Edge Controller Scheme

In smart factory, there are some new features and requirements, e.g., multi-type intelligent equipment, multi-machine dynamic cooperative control, networked cooperative manufacturing. It is urgent to research the edge controller with cloud-edge cooperation and intelligent control abilities to realize real-time perception, real-time control and intelligent analysis near the industrial field ends. This has become a new development trend of industrial control system.

Based on the idea of software definition, we propose the technical architecture of software-defined edge controller shown in Fig. 5. Different from traditional Programmable Logic Controller (PLC), the edge controller adopts a hybrid heterogeneous hardware architecture of multi-core CPU + GPU + FPGA. It has both real-time control capabilities and edge computing capabilities. It can not only support logic control, process control, motion control and other real-time task processing, but also support industrial vision, deep learning, intelligent optimization and other non-real-time task processing. The edge controller can meet the requirements of computing resources and load capacities for complex control tasks in the edge side and AI computing tasks in smart factory applications, and realize the target of intelligent control in new IIoT scenarios.

Fig. 5.
figure 5

Technical architecture of software-defined edge controller.

Full size image
  1. 1)

    Edge controller hardware resources

The hardware resources of edge controller include computing resource, storage resource, communication resource and IO resource. There are different types of computing resource. CPU is responsible for supporting high-speed closed loop control. GPU is responsible for supporting machine vision, AI model and optimization model inference. And CPU + FPGA is responsible for supporting high-speed and high-precision motion control. The storage resource consists of DDR, flash, SRAM, etc. The communication resource consists of TSN, Ethernet, fieldbus, etc. The IO resource consists of AIO, DIO, and PIO. All these resources form the hardware foundation of software-defined edge controller.

  1. 2)

    Resource virtualization (Hypervisor)

Hypervisor is responsible for managing the resource allocation and virtualization of each virtual edge controller. It includes the following key technologies.

  • Virtual CPU core scheduler: The instruction sequence of virtual CPU inside each virtual controller is allocated to the actual physical CPU to run according to the scheduling strategy.

  • HyperCalls: It is responsible for providing the external interface of the hypervisor layer, and resource management interface calls for the virtual device driver layer and Root OS.

  • Memory address map manager: Each virtual controller runs in the virtual machine’s physical address space pre-allocated by the hypervisor layer, and only has access rights to this address space. Each virtual controller runs and accesses space is isolated from each other. The memory address map manager is responsible for converting the physical address of virtual controller into the actual machine physical address.

  • Virtual IO handler: It is responsible for processing the IO request of each virtual controller and routing it to the physical IO driver to complete the real IO operation.

  • Configurator: It supports the configuration of virtual resources occupied by each virtual controller. The module establishes the actual mapping relationship between virtual resources (e.g., virtual CPU, virtual device, virtual memory) and physical resources (e.g., physical CPU, physical device, physical memory) occupied by each virtual controller according to the configuration.

The operating system in virtual edge controller includes two types: Guest OS and Root OS.

  • Guest OS: In addition to the conventional functions of traditional OS, it also includes a virtual peripheral driver program, which completes the operation function of the virtual peripheral through the HyperCalls module.

  • Root OS: Hypervisor is responsible for managing each virtual edge controller. The processing of this module must be streamlined, otherwise the execution efficiency will be greatly reduced. Therefore, Hypervisor is responsible for processing the key tasks with a short execution time, and the tasks that take a long time are handled by Root OS. Root OS adds a new virtualization component based on the original local OS function. The component is responsible for the auxiliary execution of each task request from virtual machine.

  1. 3)

    Virtual edge controller

The above virtualization method performs virtual mapping, scheduling, orchestration and management of hardware resources in edge controller, and realizes network-based computing power optimal scheduling and dynamic reconstruction in a software-defined way. The diversified virtual controllers with different control and computing capabilities are constructed for meeting different application requirements.

In the software-defined edge controller, a physical controller can virtualize multiple real-time systems and multiple non-real-time systems. They can support real-time control and edge intelligent computing tasks at the same time, and realize the collaboration between multiple industrial control subsystems. For example, on a production line, an edge controller can control multiple six-axis industrial robots and two-axis positioner systems at the same time. With the assistance of industrial vision, they can perform production tasks collaboratively while completing image-based product defect detection.

The edge controller can interconnect and intercommunicate with the industrial cloud platform, and send the industrial data collected from the edge side to the cloud platform for big data analysis and AI model training. The cloud platform sends the trained model to the edge controller to perform model inference at the edge side. This scheme can realize intelligent control of cloud-edge collaboration.

The software-defined edge controller solution integrates real-time and non-real-time control systems such as industrial control, edge computing, and industrial vision. It opens up the platform capabilities of edge and cloud. It has the characteristics of integration, intelligence, real-time, and flexible expansion. This provides a new solution with high-efficiency, differentiation, low cost, space saving, and easy maintenance for networked intelligent manufacturing.

6 Conclusion

This paper has proposed the solution and framework of SDIIC to realize the goal of industrial intelligent control based on edge computing from two levels of software and hardware. For the level of software, the SDEC platform scheme has been proposed. The flexible management and autonomous collaboration of edge devices are realized by software-defined modeling to support real-time industrial applications. For the level of hardware, the system architecture and scheme of software-defined edge controller have been proposed. By virtualizing the hardware resources in edge controller, the optimal scheduling and dynamic reconfiguration of controller resources are realized. It can provide differentiated service capabilities, and build software-defined virtual controller that can support both real-time and non-real-time task processing to meet the different hardware requirements of industrial intelligent control. The proposed SDIIC scheme aims at building an intelligent industrial control system. It uses software to define the hardware, and strives to provide flexible and efficient edge intelligent control services. This solution can be applied to discrete industry, process industry, smart rail transit, smart subway and other industrial fields, and promote the intelligent development of IIoT.