Keywords

1 Introduction

An argument combines a premise or a set of premises and a conclusion. Historically, Dialectics and Philosophy are the ancient roots of the discipline of argumentation. Arguments have always been considered an important branch of Philosophy and, with the passage of time and advancement in technology, its relevance has grown exponentially in other fields such as Literature, Logic, Law, and also in Mass Communication and Artificial Intelligence. Arguments are the fundamental tools for human beings to argue and reach their objectives. During debates, the conclusion of an argument is the focal point of the discussion. Premises are the vehicle that supports the conclusion’s reasoning and approval. There are premises that reinforce other premises and as such add strength to the conclusion. During a discussion, facts, figures and further evidence as well as logic are provided to support, attack and/or refute the opponent’s arguments. At a time when social media is one of the most important discussion platforms available, the number of users expressing their opinion has grown exponentially. Usually, such opinions are expressed through an array of premises that generate ideas and claims. Considering the relevance of argumentation in everyday life and its ubiquity in the judiciary, this study was made to analyse and evaluate the natural language used in argumentative legal documents. To automatically identify the argument in an unstructured text, a system was developed in three stages or modules. The first stage or module is the Argument Element Identifier, henceforth referred to by its acronym AEI. In this module, the main aim was to identify the argumentative and non-argumentative sentences in a corpus of legal documents. The structuring of arguments is addressed in the second stage or the Argument Builder Module, henceforth referred to as AB. In the third stage, the Argument Structurer Module (henceforth referred to as AS), the system will distinguish the arguments’ components (premise and conclusion). The corpus selected for this study was the Case-Law issued by the European Court of Human Rights (ECHR) annotated by Mochales-Palau and Moens [8]. Details of the corpus are described in [11].

Mochales-Palau and her colleagues [6,7,8,9,10, 13] have published several papers identifying and extracting arguments from both the ECHR Corpus and the Araucaria CorpusFootnote 1. Moens et al. [9] used features such as n-gram, verb nodes, word couples, and punctuation and their average accuracy results was close to 74% in various types of text but dropped slightly to 68% in the legal corpus. Mochales-Palau and Moens [8] added more features such as modal auxiliary, keywords, negative/positive words, text statistics, punctuation keywords, same words in both the previous as well as the following sentence, and first and last words in the next sentence and reported accuracy results of 90%. Mochales-Palau and Moens [10] also defined the argument boundaries i.e. the beginning as well as the end of an argument. Since components of the argument can be found scattered throughout the text, the authors suggest using semantic distance to solve this issue and argue for the use of context-free grammars (CFG) to detect the argument structure and claim to have reached and accuracy of 60%. The technique presented by these authors is applied only to a very limited number of Case-Laws.

Stab et al. [15, 16] analysed argumentative writings from a discourse structure perspective. They used structural, lexical, syntactic and contextual features to determine argumentative discourse structures in persuasive essays. Their experiment succeeded in establishing the f-measure for identifying argument components at 0.726. They focused on word indicators and lexical features that highlight an argumentative sentence. Doddington et al. [4] described four challenges and identified five types and 24 subtypes of relations. The “Role” type of relation, which refers to the part a person plays in an organization, can be subtyped as Manager, General Staff, Member, Owner, Founder, Client, Affiliate-Partner, Citizen-of or Other. The “Part” type of the relation can be subtyped as Subsidiary, Part-of or Other. The “Near” type identifies relative locations. The “Social” type can be subtyped as Parent, Sibling, Spouse, Grandparent, Other-Relative, Other-Personal, Associate, or Other-Professional.

Bunescu and Mooney [2] presented a novel approach to extract the relation between entities by presenting a new kernel for the relation extraction, based on the shortest path between the two relation entities in a dependency graph. They deployed an “Automatic Content Extraction” on a corpus of newspaper articles and were able to show significant improvements over a recent dependency tree kernel. Biran and Rambow [1] also aimed to identify argumentative relations while Cabrio and Villata et al. [3] used a combination of textual entailment framework and bipolar abstract argumentation approach to evaluate argument texts and find the relation between the arguments. Florou et al. [5] used a grammatical approach of future and conditional tenses and moods. They highlight the impact of illustration, justification, and rebuttal wording in the argument. Poudyal and Quaresma [12] have found that the Support Vector Machine is the best machine learning algorithm in identifying name entity relation.

2 Proposed Approach

The system we propose consists of three sequential modules or phases as illustrated by Fig. 1.

Fig. 1.
figure 1

Proposed Architecture of the System

Full size image
  1. 1.

    Argument Element Identifier (AEI): identifies argumentative and non - argumentative sentences in legal texts;

  2. 2.

    Argument Builder (AB): handles arguments’ components’ clustering;

  3. 3.

    Argument Structurer (AS): distinguishes arguments components (premise and conclusion).

During the Argument Element Identifier (AEI) phase, our main task was to find an optimal machine learning algorithm with appropriate features to distinguish an argumentative from a non-argumentative sentence in legal documents. We conducted several experiments with various machine learning algorithms and classified them according to the type of features used. Figure 2 presents an overview of the AEI phase. After identifying the argumentative sentences in a legal text, it is necessary to organize these sentences into argumentative clusters composed by a set of argumentative sentences interconnected or related to each other. Detecting the boundaries of an argument is a very challenging task mainly due to the fact that its components (premise and conclusion) may be connected or related to other arguments. To cluster such sentences, we deployed a fuzzy clustering algorithm (FCA) that provides a membership value ranging from 0 to 1 for each sentence cluster. The membership values are the key assets of the FCA, which allows us to associate each sentence to more than one argument cluster. The performance of the algorithm depends on the type of features selected. In this study, we focus on the following features: ‘N-gram’, ‘Word2vec’, and ‘Sentence Position’. Figure 3 offers an overview of this phase. On the AS phase, argument components (premise and conclusion) are identified as having a premise or a conclusion basis. The sentences identified as having a premise basis are outright premises or consist of a premise clause. The sentences identified as having a conclusion basis are obvious conclusions or point towards one. Many sentences that have a premise basis and are tagged as such may also include a conclusion clause and the same happens to the sentences labeled as displaying a conclusion basis. To accomplish this task, we deployed indicator features. Indicator features play an important role in identifying argument’s premises and conclusions. Words such as “finally,” “therefore,” “concluding,” and “thus” clearly introduce a conclusion and play an important role in the process of identifying argument’s conclusions. It is also highly probable that sentences containing words like “should,” “could,” “almost,” “must be,” “because,” “seems,” and “would like,” are premises. A major limitation in the AS phase is that each sentence may have one or several premises but only one conclusion, and also the system’s accuracy rate will diminish whenever the classifier is not able to identify the sentence’s conclusion, or identifies more than one conclusion in a single argument.

Fig. 2.
figure 2

Overview of the Argument Element Identifier Module (AEI)

Full size image
Fig. 3.
figure 3

Overview of the Argument Builder (AB) Module

Full size image

3 AEI Preliminary Results

The main goal of the AEI phase was to select the algorithm with the most appropriate parameters. We aimed to develop a system that will automatically identify the argumentative sentences on an unstructured textual document. As Fig. 4 illustrates, the AEI system’s architecture follows several steps. Initially, the corpus needs to be refined. Once the features are extracted, the classifier can then be built and its performance evaluated.

The words that form a document must be mapped in accordance to a predetermined token and TF-IDF in order to normalise the length of each unit. In our experiment, this procedure created 11374 features. The TF-IDF [11] function was calculated as:

$$\begin{aligned} tf-idf(w_{i},d)=tf(w_{i},d)ln\frac{N}{df(w_{i})} \end{aligned}$$
(1)

where \( tf(w_{i}d)\) is the frequency word \(w_{i}\) in document d and \( df(w_{i}) \) is the number of documents where \(w_{i}\) appears and N is the number of documents in the corpus. To measure performance we used precision, recall and f-measure [14] methods. We ran several experiments with the machine learning algorithms Support Vector Machine (SVM) and Random Forest (RF) to determine their performance in identifying argumentative sentences in accordance with the features provided. We selected the top-n informative features (using the gain ratio measures) with \(n \in \{100, 200, 500, 1000, 2000, 5000, 11374\}\) and tested the polynomial kernel SVM with various values for the complexity parameter (\(C \in \{0.001, 0.01, 0.1, 1, 10,\) and \(100\}\)). Similar experiments were conducted deploying the Random Forest algorithm using several trees (\(nt \in \{7, 11, 17, 50, 100\}\)).

Fig. 4.
figure 4

Architecture of argument element identifier (AEI) Module

Full size image
Fig. 5.
figure 5

F-Measure of SVM algorithm

Full size image
Fig. 6.
figure 6

F-Measure of RF algorithm

Full size image

Figures 5 and 6 show the graph of f-measure vs. Support Vector Machine (SVM) algorithm and f-measure vs. Random Forest Algorithm respectively. In the SVM chart (Fig. 5), as the number of features increases, the performance of f-measure increases, up to 2000 features. The highest f-measure value of 0.595 was achieved with c = 0.1 and 2000 features in the SVM algorithm experiment. In case of the graph of f-measure obtained from the Random Forest Algorithm chart, (Fig. 6) as the number of features increases, a peak f-measure of 0.52 was reached with 1000 features and 100 trees. Then, the f-measure value decreases up to 2000 and remains constant till 11681 features. We can therefore conclude that the SVM algorithm produced better results than the RF algorithm. Overall, the results achieved are quite promising and support our proposal for the creation of a new argument mining framework.

4 Conclusion and Future Works

We are proposing a new approach to automatically identify arguments in legal documents which is phased in three modules: Argument Element Identifier (AEI), Argument Builder (AB) and Argument Structurer (AS). The preliminary results of the AEI are extremely promising and support to the development of a new argument mining framework. Further research must be done on the use of string kernel as well as other alternative representation models, including linguistic features such as POS tags, Parse trees and Tree Kernel.