Keywords

1 Introduction

The study of motion is currently a popular research area in image analysis, computer vision, machine learning, and biomechanics. Motion analysis involves the detection, recognition, and tracking of moving objects, including humans [6]. Theoretically, motion capture systems can capture as many details as the human body can show. The gathered information includes facial and finger movements, estimation of body position, and activities performed. The problems with human motion analysis have been discussed in many academic works. Most of the relevant works are summarized in a recent compilation [15]. One of the oldest and most cited works on human motion analysis was written by J.K. Aggarwal and Q. Cai Human Motion Analysis: A Review [1]. It delineates three challenges in interpreting human movement research:

  • analysis of movement including parts of the human body,

  • tracking a moving person,

  • recognition of human activities based on image sequences.

Fig. 1.
figure 1

The German and the Japanese Martial Arts. The figure on the left depicts german longsword strikes, and one is the Krumphau stroke (source: taken from the manual by Joachim, Meyer [13] from 1570). The left picture presents the fastest Naginata (source: taken from https://www.youtube.com/watch?v=yFgBxIxx4P4)

Full size image

The well-known methods of movement analysis in sport [7, 11, 14] focus on the classification of movements that differ significantly from each other. Fencing is an example of a discipline where the detection and evaluation of similar body postures are essential. Authors [22] proposed a new repository called Martial Arts, Dance, and Sports dataset for 3D human pose estimation. To estimate human pose based on depth, they provided stereo depth images from a single viewpoint. They used five actors to collect data, with each actor performing one category of actions. Two martial arts masters performed a pre-arranged series of movements in Tai-chi and Karate. In paper [5], five master sword strokes were recorded by the MoCap system to determine the pattern of correct cutting and show the mistakes made by amateurs. Paper [16] proposed using the motion capture function from Microsoft Kinect, where the user can develop basic training and practice taekwondo using a virtual trainer. In recent years, there are also reviews [20] where researchers conducted a systematic study of scientific publications on martial arts and motion capture. In 2020, we created one of the world’s first reference databases of fencing actions, which included five master long sword strikes with kinetic, kinematic, and video modalities [9]. We were able to assess these movements and suggest potential study directions based on the developed analysis methods and algorithms. This paper proposes to extend those registration technologies for long swords and swordsmanship to similar combat, such as Japanese Kendo sword fencing (Fig. 1).

2 Five German Longsword Mastercuts Analysis

Fighting techniques from the late Middle Ages and Renaissance are the main emphases of the historical alliance of European Martial Arts [3, 4]. We conducted groundbreaking research to automate the identification of particular German sword-cutting techniques [18, 19] to popularize this craft and develop a reference dataset with data captured from expert and amateur performances [8]. The information about fencing needed to complete this study came from references, interviews with subject-matter experts, and recordings. The most representative movements of German Longsword Masterstrokes [11] were registered and studied:

  1. 1.

    Zornhau (Strike of Wrath) is a devastating above-the-shoulder diagonal blow. A strike to the opponent’s left ear from the right shoulder.

  2. 2.

    Schielhau (Squinting Strike) is a false-edged strike. This blow hits the opponent’s shoulder while deflecting his weapon.

  3. 3.

    Zwerchhau (Cross Strike) is a high, horizontal hit to the head that deflects the attack of the opponent.

  4. 4.

    A vertical attack from above to the opponent’s wrists or sword is known as a Krumphau (Crooked Strike).

  5. 5.

    Scheitelhau. A high, vertical attack with a long edge directed at the top of the head or face while simultaneously retreating is known as a “crown strike”.

The master, an expert fencer, performed reference movements that are examples of proper stroke patterns for blows captured by amateur fencing. The full-body plug-in gait configuration (39 markers), initially intended for medical applications, was used to record movements. In the measuring arrangement, 16 EMG electrodes (surface EMG) were employed to capture the activity of specific muscle groups during the cuts. All studies were conducted in the Human Motion Lab, R &D Center, Polish-Japanese Academy of Information Technology (bytom.pja.edu.pl) to create the first public database of German Longsword Mastercut. We recorded the movement of four subjects according to the assumed multimodal measurement protocol. The activity was recorded using 30 motion capture cameras (39 reflective markers on the body and five on the sword), 16 EMG electrodes, two ground reaction forces, and three hardware-synchronized video cameras [10]. The data was semi-automatically (manually) cleaned and saved in C3D and AVI formats. As a result of data recording, we obtained 404 modalities consisting of the positions of 3D markers, virtual markers, centers of gravity, angles, forces, moments, powers, electromyography, video, and generated markers of events. Most of the 3D data includes time series of positions of subsequent markers or Euler angles of detection degrees of freedom of joints. The base sampling rate of the multimodal measurement system was 200 Hz. The dataset of five master longsword strikes reported in [9] had been created due to multimodal measurements. The ultimate poses for each of the five master strokes: Zornhau, Schielhau, Zwerchhau, Krumphau, and Scheitelhau, were chosen after analysis of the movements. Numerous illustrations of these positions were provided, drawn from 750 recordings made by 15 people who range in skill level when it comes to making sword strikes. The same mastercuts performed by various actors vary greatly. Both professional and amateur recordings are available in the established database. The fencer’s finishing positions are visible in 3750 recognized frames from the motion recordings. Each item in the original database has more than 404 modalities to represent it.

Fig. 2.
figure 2

(source: own elaboration)

Final positions of the master sword strokes. In sequence, from left: Strike of Wrath, Squinting Strike, Cross Strike, Crooked Strike, Crown Strike

Full size image

2.1 Conclusions from HEMA

We have collected a reference database of chosen pieces of training using the Master Strikes of the German Longsword, thanks to the established algorithms and collaboration with an expert. This database contains three parts: B.M1 - registered position of 5 markers placed on the sword, B.M2 - registered position of 39 markers placed on the actor’s body, and B.M3 - recorded angles of the actor’s 26 joints. Of course, it is challenging to picture participant training in a recording studio identical to the one used for this project. The reference multimodal measuring methodology would then be replaced with monocular video in the next phase. We might also think about developing a base similar to this for additional fence types, such as saber, foil, and sword.

3 Japanese Fencing Analysis

Knowledge about Japanese fencing we have received from experts and literature sources such as [12, 17]. Fencing knowledge from studies [2, 21] also was essential for our research. Other researchers can apply the proposed and described registration technologies for both long sword and Japanese fencing to similar types of combat, such as Kendo (Japanese sword fencing), Naginata (Japanese stick fighting), and Jyojyutu (long stick fighting). Recognizing the dynamics of action in footwork in fencing seems crucial when considering movements in Kendo. The main focus can be on following fast movement, paying attention to even the most minor details, and learning movement patterns. For each of the five participants, ten captured recordings included T-pose position, cut in place, cut with a step forward, cut back, and cut with a step back. The starting position has the following body positioning: legs at shoulder width, right leg extended half a foot forward, left leg half a foot back. Right hand on top of the hilt, in a loose grip, left hand at the bottom; knuckles of the hand pointing down; elbows straight. The end of the sword is at the level of the sternum. The cut looks as follows: hands above the head, gently bent at the elbows, sword at an angle of 0-45\(^\circ \)C to the ground; move the sword in a single line, not sideways; strike down to the height of one’s sternum with the complete deceleration of the movement. The movement should always end at the same height. However, transitions with the step include the following actions: when raising the leg, raise the sword; the most important thing is to lower the sword at the same time as lowering the leg. When stepping, the cut should end at the exact moment as the step. Comparing the records of medieval sword strokes [13] with the released movements (strikes) of the Kamae of Japanese Kenjitsu [16], one can see a remarkable similarity. A more thorough analysis of these similarities is currently being conducted. The movement Forehead Strike (Fig. 3) is very similar to the movement (strike) Zornhau (Fig. 2 - graphics first on left) (Fig. 4).

In the Cyber-Physical Systems Laboratory, an OptiTrack system with passive markers was used for motion capturing. The system consists of eight Flex 13 cameras with 30–120 FPS at 1.3 MP resolution, OptiHub 2 synchronization devices, and Motive software for marker tracking. The OptiTrack system can track movements with a tolerance of less than 0.5 mm. Synchronization of the cameras is provided by connecting them to two OptiHub 2 devices and then connecting them to a computer. The Motive software uses standard sets of skeleton configurations. Using algorithms and collaboration with an expert, we have collected a reference database of chosen pieces of Japanese mastercuts training composed of 240 recordings of 6 performers (Fig. 5).

Fig. 3.
figure 3

(source: taken from [17])

Forehead strike

Full size image
Fig. 4.
figure 4

(source: taken from [17])

Downwoard strike

Full size image
Fig. 5.
figure 5

(source: own elaboration)

Process of laboratory set-up for mocap recordings. Cyber-Physical Systems Laboratory, Faculty of Information and Communication Technology, Wrocław University of Science and Technology

Full size image

4 Conclusions

Japanese fencing Kendo could use the registration technology proposed here for longsword mastercuts and fencing. Long weapons used in similar forms of combat include the naginata and stick. It appears to be quite helpful to recognize the action dynamics in fencing footwork when analyzing Kendo’s moves. Based on publications and discussions with martial arts specialists, we will modify the fencing expertise from the papers that are foundational to this research. Our primary goals will be tracking quick motion, focusing on the minor details, and memorizing motion patterns. We suggest using augmented reality (AR) glasses to deliver real-time immersive feedback for performing blade work, visual cues, identifying action dynamics in fencing footwork, and practicing fencers. Therefore, fencers can correct their actions during training rather than after they receive feedback.

We propose to develop a simplified measurement protocol based on experiments from multimodal gait analysis reference laboratories and the results of the described experiments (Sects. 2 and 3). We intend to simplify the measurement configuration based on data registered from German Longsword and Japanese strikes. As a result, the target experiment will occur beyond the reference laboratories in real-world conditions. The steps that follow assume:

  1. 1.

    the use of prepared data from laboratories as reference data for kinematics and kinetics,

  2. 2.

    unification of the specification of cutting movements (both European and Japanese) for analysis in a simple measurement environment,

  3. 3.

    preparation of a simplified measurement protocol limited only to monocular camera recordings.

The next step is to perform video pose analyses, use accelerometer data for proper data analysis in a simplified environment, and refer to reference data recorded in motion laboratories. The extended procedures are based on the described steps for running video cameras outside the Laboratory with the possible support of local IMU sensors for real-world data recording. The ultimate goal is to develop correct mastercut recognition models solely from video data.