1 Introduction

With high computing power available, the demand for AI-based applications is growing rapidly. The market size value of AI in agriculture is expected to be 13.89 Billion USD in 2030 [1]. This can already be seen today, as more and more AI applications are developed for the agricultural sector. Often, images captured by unmanned aerial vehicles (UAVs) are used to retrieve high resolution images, which are then used to train an AI model. In [2] for example, Butte et al. used those images to train a model to distinguish between healthy and stressed potato plants. Another use case is weed control. Gao et al. [3] trained an object detection model to distinguish between sugar beets and hedge bindweed. They trained the model with 2723 images in total, from which 2271 images were created using data augmentation. Combining synthetic and collected field images yield to a higher mean average precision than using only field images. The aim of the synthetic images was to overcome labeling work.

According to [4], the quality of labels determines the quality of results. But the labeling process takes most of the time of the development of AI based applications. As an example, labeling the ImageNet dataset would have taken 19 years if done by a single human. Labeling and data preparation takes up to 80% of the total project duration [5]. To receive high quality results, high quality labels are necessary. In addition to that, thousands of labels are necessary for reliable predictions in changing environments. [6]

To improve this pipeline of collecting data, labeling, model creation and deployment, several no-code platforms are available. Some examples are Viso Suite [7], Roboflow [8], Label Box [9], and Supervise.ly [10]. Both platforms assist non-AI experts to create and deploy AI models. Another advantage of no-code platforms is that all steps can be executed by using just one tool. We developed another no-code platform called Zauberzeug Learning Loop, which focuses on data quality and model optimization, while being as simple to use as an email program. Moreover, the aim of the Zauberzeug Learning Loop is to collect and use most meaningful data. This is especially important in the field of agriculture, where data collection is especially expensive and time consuming.

2 Zauberzeug Learning Loop

The Zauberzeug Learning Loop is a web based no-code AI platform that allows to iteratively improve AI models. The main focus is to reduce the amount of data needed to receive a satisfactory AI model using Active Learning [11]. The user is guided through the different processing steps needed to create and use an AI model from plain image data using a streamlined Kanban process. This workflow was originally introduced by Toyota to streamline automotive production processes [12]. It can be visualized as a whiteboard having different columns, where each task is represented as a card which moves from left to right through the process. In our application, we use this principle for data management. We call the lanes inside our application Inbox, Annotate, Review and Complete. Later, those will be explained in detail. Every image uploaded passes through those lanes to control which images are labeled and used for AI model creation at the end. The upload process happens automatically through a RESTful API. The principle also encourages collaborative work by defining the task that has to be executed depending on the lane the images are at. Our system is available at https://learning-loop.ai. In order to utilize the tool, please contact us to set up a user account.

2.1 Active Learning and Transfer Learning

Active Learning, also referred to as Human-in-the-loop machine learning, is a strategy to assist the AI with human intelligence. Active Learning tries to increase the accuracy of an AI model, while reducing labeling effort [11]. In practice, this means to only add ’meaningful’ images for the next training cycle. To support this process, we use AI assisted image sorting.

Moreover, we use Transfer Learning. It describes the process of transferring knowledge from one domain to another. In terms of AI models, a model that already learned to identify bicycles learns faster to identify scooters, than a model that was trained on chessboards [13]. According to [14], Transfer Learning reduces the number of samples necessary to achieve comparable results compared to models trained without Transfer Learning. Another advantage of this approach is that the problem of sparse data availability is overcome. For most AI tasks in agriculture, the availability of public data is limited due to the need of highly use case dependent samples. To gather this data, time-consuming and expensive drone flights or field experiments have to be conducted. Transfer Learning helps to minimize those efforts (Fig. 1).

Fig. 1
figure 1

Active learning with the Zauberzeug Learning Loop

Full size image

2.2 Workflow

The workflow can be divided into several steps. First, a human selects images where the AI is uncertain. The images are pre-sorted based on their value for the AI model training. Next, these images are labeled and added to the next model training. The improved AI model can then be deployed to cloud or edge. We call this process the Learning Loop (Fig. 2).

Fig. 2
figure 2

Dataflow in the Zauberzeug Learning Loop

Full size image

Kanban Stages: In total, there are four different stages an image has to pass. First, new uploaded images are automatically added to the Inbox. Here, a human selects the most valuable images for training. The user is assisted by inferences from previous AI models. Those inferences were created after every training using the best model. Furthermore, those inferences are used to sort the images to present the most "novel" ones at the top. In addition to that, it is possible to weight some categories stronger than others to show those images at top. The next stage is called Annotate. Here, images are labeled for training. The labeling process allows collaborative multi-user editing. After that, images are moved to the Review column. In this stage, a human examines the created labels and approves them by adding them to the Complete section. All images in Complete are used for the next training iteration. Therefore, the dataset grows from iteration to iteration. The tool also incorporates a mobile friendly view within the Complete section to minimize wrong annotated data. In general, we use a tag based system to further organize the images in every stage. A tag ’human’ for example creates a short-cut on our dashboard that shows all images containing that tag (Figs. 3 and 4).

Fig. 3
figure 3

Mobile friendly view to identify wrong human or AI labels

Full size image
Fig. 4
figure 4

Kanban stages of learning loop

Full size image

AI Training and Deployment: For training, it is possible to use one of our preconfigured trainers, but also a custom solution that is connected to the platform through public APIs. The created model is used to make predictions on all already exiting images of the Kanban process. The predictions of the trained model are sent to the Learning Loop. Those help the human to find most novel images for the next training iteration by selecting only those images, where the AI model is uncertain. ’Uncertain’ in the most basic case means that the confidence ranges from 30% to 60%. For the future, we plan to include a more complex AI based decision pipeline.

Additionally, the human is supported by an analysis step, where missing or wrong annotations are highlighted and resolvable by the human. This leads to an improved quality of annotations.

2.3 System Architecture

The system consists of several containerized services which are connected to present the web based cloud platform. It consists of an Angular UI frontend [15], a FastAPI backend [16] and a PostgreSQL database [17]. Through the platform, data can be uploaded, accessed and annotated. The actual Machine Learning is done on external services called Learning loop Nodes. Those could be a Trainer, which is responsible for model creation, but also a node called Detector, which is used for inference calculation. It is connected to the Zauberzeug Learning Loop via SocketIO [18]. For example to send important images, i.e. where the AI model is uncertain, from the Detector Node to the Zauberzeug Learning Loop.

Learning Loop Nodes: Currently, four different types of Nodes are used: Trainer Nodes, Converter Nodes, Detector Nodes and Annotation Helper Nodes. Beside the Nodes that we use in our daily work, it is possible to create and connect custom Nodes to the Zauberzeug Learning Loop through our open-source library [19]. By implementing a few methods new training techniques can easily be integrated to use the Zauberzeug Learning Loop. Trainer Nodes are responsible for AI model creation and can be used for object detection, instance or semantic segmentation. Converter Nodes create different formats of AI models optimized for inference calculation on edge devices. Detector Nodes are connected to the Zauberzeug Learning Loop to perform inferences. Thereby, an AI model can be deployed onto the system where the Detector Node runs. All inferences made are analyzed. If certain thresholds are not met, the data is automatically submitted to the Learning Loop via RESTful API. In addition to that, the Detector Node performs Active Learning by sending images where configured criteria met, i.e. the AI was unsure about a detection, directly to the Zauberzeug Learning Loop, where they are added to the Inbox.

Last, we have a Node type called Annotation Helper. Those nodes assist the user while labeling. In case of segment annotation, an AI could assist the human to receive a fitting polygon with fewer clicks and user interactions than traditional labeling tools. To enhance flexibility, the Annotation Node can define its own SVG content, which is displayed in the browser.

3 Usage in Practice

Our platform seamlessly integrates with comprehensive business scenarios. The first step for a new use case is the collection or uploading of a few images which are then labeled and used for a first training. Based on this AI training, the remaining images are analyzed and sorted. Then, the labeller can only start picking only a few images again which are used for the next training to improve the model instead of labeling thousands of images without knowing their value for the AI. This process is repeated in a loop to iteratively improve the AI model.

We will present two examples within this section. The first use case is our outside-in marker tracking of robots driving inside and near buildings, where Global Positioning System (GPS) and other tracking methods are not working. The second one is the zUCKERrübe project [20], funded by the Federal Ministry of Education and research. Here, a robot is used for weed control of sugar beets. Both robots are equipped with our control unit Zauberzeug Robot Brain [21]. This customizable industrial-grade AI edge computer controls the robot and runs the Detector Node. In our use case, the Detector sends images, where the model is unsure, directly to the Learning Loop. Those images are afterwards sorted based on an algorithm to ensure that the user of the platform gets presented the most meaningful images for the next training to support the Active Learning approach.

3.1 Marker Tracking

Our outside-in marker tracking was initially developed to collect horse droppings in open stables, but has matured as a robust and flexible way to track and navigate indoors and near buildings. For indoor navigation, it is a great alternative to GPS based solutions. To observe the working area, multiple cameras are installed in elevated positions. Those camera images are stitched together to create a map of the environment. Furthermore, these are used to train an AI model that gets deployed to the Detector Node. This approach is used by the Active Cleaner robot. It is used to automatically collect horse droppings and was awarded with a golden Innovation Award at Eurotier 2022 [22]. To track the robot while collecting, it is equipped with a ’Shadow Marker’. This marker consists of four circular shaped holes, uniquely identifying the robot from its surroundings. Beside the marker being tracked by an AI, another AI model identifies the horse droppings. If a certain amount is detected, the optimal path to collect those gets calculated. While navigating, the robot is able to adapt the path to moving obstacles like horses or humans. After collecting, the robot unloads the droppings at a custom defined place before navigating back to its charging station (Fig. 5).

Fig. 5
figure 5

Active cleaner

Full size image

3.2 zUCKERrübe

The zUCKERrübe project aims to reduce manual workload in organic sugar beet farming to improve the cost-benefit relation. While the demand of organically cultivated sugar beets is rising, the manual workload of 80 to 300 man-hours per hectares reduce profitability. Therefore, new cultivation systems are needed. As part of the project, the Uckerbot robot was developed. It concentrates on the weeds within the region around the sugar beet rows and removes them using a driller. Other activities are left to the farmer with conventional machinery.

To be able to distinguish between weed and sugar beets, the robot is equipped with a camera that records the working area to ensure sugar beets and weed can be detected. More cameras capture the surrounding for local navigation. Beside the cameras, a Zauberzeug Robot Brain is build into the robot. On that computer, a Detector Node runs an AI model to distinguish between weed and sugar beets in real time. Thanks to a solar panel, the robot can work 24 h a day. While working, the Detector Node uses the Learning Loop for continuous model improvement.

As it can be seen in Fig. 6, the plants were divided into three different classes: Sugar beet, weed and big weed. The distinction between weed and big weed was made since the weed plants need different treatment. If there were more than four leaves on a weed plant, it was categorized as big weed. All categories were annotated as ’Points’. Those are a special type of annotation, not existent in most annotation tools. It is inspired by key-point detection, as used for motion capturing. As points are simpler to annotate, and we are only interested in a single point location to drill, the midpoint of the points is enough to identify the root of the weed we want to remove. The perimeter can be configured freely.

The Zauberzeug Learning Loop has been used to label the images. After an object detection algorithm was trained with only 3000 images in total, a detection rate of 95% for sugar beets and 90% for weed was reached. The inference speed and detection rate are fast enough to let the robot work with a speed of at least 2 km/h, which is already higher than the allowed driving speed for autonomous machinery.

The results show that the mean weeding efficiency (reduction of weed compared to control group) was significantly higher compared to other mechanical weeding techniques. It could also be shown that the robot tended rows had the highest bio-diversity, even compared to the non-weeded control area. In a next step, this will be improved further by distinguishing different kind of weeds for a more selective treatment.

Fig. 6
figure 6

Labeled image from Learning Loop with detections. Classes were sugar beets (orange), weed (pink) and big weed (purple)

Full size image

4 Conclusion and Future Work

In summary, the Zauberzeug Learning Loop served multiple purposes - it was employed for data storage, as a labeling tool, for AI model development with our Trainer Node, and for object detection using our Detector Node. The whole pipeline from data gathering, labeling, AI model creation and inference calculation to new data uploading was successfully passed without having our project partners to write a single line of code. Moreover, the active learning approach reduced the amount of data needed and ensured that only meaningful data was added to the next training circle. The Kanban inspired workflow guided the user through the different steps needed to create an AI model. The resulting models were automatically deployed to the edge computers build into the robots.

At this stage, the Learning Loop only offers one pre-configured Trainer Node for object detection . In the future, Nodes for object segmentation and semantic segmentation will be added. Another point we would like to further improve is the sorting of images in the Inbox. Moreover, we try to minimize the number of wrong labels in our data. Therefore, we implemented a mobile friendly view where the user can solve conflicts between annotations and detections. In addition to that, the labeling process could be accelerated using more AI based assistance tools. Also, the Zauberzeug Learning Loop still needs to be evaluated statistically and compared to other no-code platforms.