Keywords

1 Smart Lifting Devices

Lifting devices are tremendously utilized in numerous heavy load transportation industries, and therefore, the smart control of crane systems is becoming an interesting and important research field. Innovative control systems designed for processes of precise positioning of objects and heavy cargo can be equipped with intelligent interaction systems between lifting devices and their human operators.

Third-generation smart control systems combine artificial intelligence and cognitive functions so that they can provide an interface between the operator’s augmented reality vision and the physical lifting device and its environment. The systems for smart control of lifting devices typically consist of diverse components: tracking cameras, augmented reality goggles, laser trackers, speech and natural language processing methods, geometry and topography scanners, photogrammetric cameras, movement scanners, interactive manipulators, measurement tools, sensors for signal acquisition, actuators for performing or triggering actions. The smart control systems address natural interfaces, speech interaction [1], interactive communication, vision systems, augmented reality, interactive manipulators, feedback channels, force feedback, and distributed control.

Smart control of lifting devices incorporate functions of sensing of device working conditions and environment, actuation and control of mechanical and mechatronic systems, in order to analyze and model a task or situation, and make decisions based on the acquired data in a predictive or adaptive manner, thereby performing smart control actions for the lifting device and processes. In general the smartness of the control system can be attributed to intelligent control operation based on augmented reality vision, natural interfaces, sensorial systems, increased safety, and cooperation capabilities. Advanced smart control systems address safety, efficiency and ergonomic challenges like automation, optimization, supervision, flexibility, adaptability, robustness, and the ability to reuse knowledge. They are for that reason increasingly used in a large number of different tasks. Key sectors in this context are construction, transportation, logistics and manufacturing.

2 Smart Control Systems with Innovative Interfaces

The proposed conceptual design of smart control systems assumes that they feature natural-language speech interfaces, intelligent visual-aid systems based on augmented reality, as well as interactive manipulation systems providing force feedback. A sketch of the technical and scientific problem space facing a scalable and universal realization of the smart control system is shown in Fig. 1.

The aim of the presented research is to design an innovative human-machine speech-based interface using natural languages for smart lifting devices. The proposed interface [2,3,4,5] equipped with patterns and antipatterns and artificial intelligence methods allows for safety and performance improvement. This especially applies when the expert assistance is needed regarding distant effects and complex decisions, heuristic circumstances of decisions, and sudden changes of conditions. This knowledge can be used for distributed smart control.

The innovative design of the smart control system of lifting devices involves the use of an intelligent and natural interface applying patterns and antipatterns for automatic processing of commands, tasks and processes in order to extract hidden information through uncovering implicit relations for various dependencies between tasks, conditions, processes and parameters. Another approach of exploration of implicit relations concerns mining in meaning correlations between commands, tasks and strategies, using patterns and antipatterns.

The design and implementation of the control system with innovative interfaces for smart lifting devices is based on simulations and experiments with developed kinematic models of loader cranes. They allowed for the development of intelligent methods for processing data in operating cycles of the control process. An analysis of the configuration system of the crane’s loading system was carried out in order to ensure collision-free motion of parts, as well as kinematics models for different tasks were developed. Moreover a set of tasks for the crane were devised, motion components for movable elements were determined, as well as ranges of motion and allowed trajectories depending on characteristics of executed tasks. In addition the patterns representing correct execution of tasks were devised, in the form of motion sequences of crane’s working parts.

The analyzes were used to enable rigorous development of algorithms and software for modeling the manner of execution of selected operator’s actions of controlling the motion of crane’s working parts. The research also allowed for rigorous development of algorithms and software for modeling the motion of crane’s working elements, which take into account components of distance from the target point.

Fig. 1.
figure 1

Map of the technical and scientific problem space for the smart control systems of lifting devices with application of artificial intelligence and antipatterns.

Full size image

3 Intelligent and Natural Interfaces

Smart lifting devices feature control systems with innovative interfaces using artificial intelligence methods and techniques for processing speech and natural language, as well as supporting decision-making processes. The concept of the interaction systems between lifting devices and their human operators assumes that they are equipped with the following subsystems: augmented reality vision, voice communication, natural language processing, command effect analysis, command safety assessment, command execution, supervision and diagnostics, decision-making and learning, interactive manipulation with force feedback.

Fig. 2.
figure 2

Smart control systems with intelligent and natural interfaces.

Full size image

The smart control system with an intelligent and natural interface is presented in abbreviated form in Fig. 2. The numbers in the cycle represent the successive phases of information processing. The system is equipped with specialized modules using the following real-time data sources: crane tracking and photogrammetric measurement of geometrical parameters of working space, laser scanning of geometry and topography of working space, forces-measuring and sensorial processing. The obtained data is sent to the subsystem of effect analysis and safety assessment. The operator’s augmented reality view is based on the extended field of view from cameras making a virtual camera. The lifting device is controlled using spoken natural language commands and manual control with multiple feedback. Developed new methods for stability evaluation and optimization of trajectories are also included in the smart control.

4 Models of Patterns and Antipatterns of Commands, Tasks and Processes

In the proposed concept, smart control systems feature developed methods of intelligent mining of structured data and information for generation of patterns and antipatterns in the interactive control process. The concept proposes reasoning models for inference mechanism of implicit relations between commands, tasks and processes. The proposed methodology is based on probabilistic neural networks [6] (Fig. 3A) and hybrid neural classifiers (Fig. 3B).

Fig. 3.
figure 3

Inference mechanism of implicit relations between commands, tasks and processes based on probabilistic neural networks (A) and hybrid neural classifiers (B).

Full size image

The intelligent data mining methodology allows to determine associated commands with their connected actions and objects. It also provides associated tasks and their linked operations and targets. The networks discover relationships between actions and linked objects, and on this basis trigger execution of complete commands which cause the same actions and effects or control of machine assemblies e.g. boom systems. Mobile cranes feature boom systems which are composed of non-extendable booms, extendable telescoping booms, as well as sub-booms. An example can be the following command ‘continue to raise the 2nd boom (telescoping boom) and lower the 1st boom’ which causes the boom system to move the object (cargo) away from the crane. Another example could be the following command ‘lower the 2nd boom and rise the 1st boom’ which causes the object to be moved toward the crane. The methodology based on hybrid neural classifier allows for classification of process conditions and their relevant parameter values to condition groups for selected crane configurations. Inputs of the network comprise grouped parameters with values for each consecutive classification stage. The network’s output produces interpretation of the complex process with its associated parameters for a specified conditions.

Fig. 4.
figure 4

Modeling patterns and antipatterns for reasoning based on neural networks.

Full size image

The developed methodology of modeling patterns and antipatterns for reasoning can be used in automation of command recognition, safety analysis and assessment in human-machine interaction. The methods allow for extraction of hidden information about the control process of operating lifting devices (cranes) through uncovering implicit relations for commands and their associated actions on objects (Fig. 4), tasks and their component operations on targets (Fig. 5A), as well as processes and their parameters with conditions (Fig. 5B).

Fig. 5.
figure 5

Reasoning models for information mining based on neural networks to extract hidden information about tasks (A) and processes (B) through uncovering implicit relations for tasks and their component operations on targets, processes and their associated parameters with conditions.

Full size image

In the concept, issued commands, given tasks (strategies) and executed processes are processed using probabilistic neural networks and neural classifiers in order to assign them to appropiate classes. The commands provide information about actions and objects. The data about tasks are based on operations and targets. The processes depend on parameters and working conditions. As the result, the extracted structured data with associated information allows for development of patterns and antipatterns for the crane control and safety analysis by uncovering implicit relations between commands, tasks and processes.

5 Conclusions and Perspectives

In the past few years, smart control systems have attracted a great deal of attention from both academia and industry due to many challenging research problems and a wide range of applications. The paper proposed an innovative concept of smart lifting devices with speech-based interaction using patterns and antipatterns generated with the use of a developed modeling methodology. It features reasoning models for inference mechanism of implicit relations between commands, tasks and processes in the interactive control. The developed methods can be used in automation of command recognition, safety analysis and assessment in human-machine interaction. Patterns and antipatterns generated from relational data mining used in the smart control process pose unique opportunities and challenges for interactive intelligence and its applications.