1 Introduction

Question Answering (QA) is an application of Natural Language Processing. A system which provides people with a convenient and natural interface for accessing information is Question Answering system. Nowadays, the need to develop accurate systems gains more importance due to available structured knowledge bases and the continuous demand to access information rapidly and efficiently. QA system has applications in various domains like education, health care, and personal assistance. QA system retrieves the precise information from large documents according to the user question [1].

India is a multilingual society with 30 languages which are spoken by more than a million native speakers, and these languages are written in different scripts. However, users who do not use English as the first language are also significantly high in number. In India, most of the people speak in their regional language and hence find it difficult to express their queries in English and a huge amount of information on the Internet is available in English. Thus, this work might help to develop a search system in the Marathi language [2].

Question classification maps a question into a category that indicates which information should be present in the answer. QC is a very important step because the question category helps mainly in two cases. One is to reduce the number of possible answer options and other is to decide the strategy to search for answer depending on the question category.

Example:

Who is Mozart?

Is directly mapped into Human: Description

What is Australia’s national flower?

The headword flower is identified and mapped into the category Entity: Plant.

The paper is organized as follows: Sect. 2 presents the background study; Sect. 3 discusses the data creation; Sect. 4 describes the QC system; Sect. 5 presents the results and experiments for various features used; Sect. 6 discusses error analysis; and Sect. 7 concludes the paper with the future work.

2 Background Research

2.1 “Learning Question Classifiers: The Role of Semantic Information”—Xin Li and Dan Roth

Li et al. [3] present a question classification system that is modeled with the machine learning approach with a multi-class classification task with 50 classes. A hierarchical classifier was used to classify questions in fine-grained classes. Results of [3] conclude that a learning approach is better as the problem can be solved with 90% of accuracy.

2.2 “From Symbolic to Sub-symbolic Information in Question Classification”—Joao Silva, Luisa Coheur, Ana Cristina Mendes, and Andreas Wichert

Silva et al. [4] present the evaluation of a question classifier which is rule based. Direct match to the question or identifying the question headword and then applying WordNet to map the question are the two scenarios that are followed [5]. Direct match uses a set of rules that are manually built for compilation. The second scenario also uses manually built rules, but these rules express a path in the parse tree for question headword. After headword extraction, the headword hypernyms are analyzed until it matches to the possible question category. Results obtained by [4] gave a precision of 99.9% for coarse-grained class and 98.1% for fine-grained class in case of a direct approach. Whereas a precision of 86.4% for coarse-grained class and 78.5% for fine-grained class was obtained for headword extraction approach with WordNet mapping.

2.3 “Answer ka type kya he?” Learning to Classify Questions in Code-Mixed Language—Khyathi C. Raghvavi, Manoj Chinnakotla, and Manish Shrivastava

Raghvavi et al. [6] present a QC system for English–Hindi code-mixed questions which is Support Vector Machine (SVM) based. The language resources like chunker, parser does not exist for code-mixed languages. [6] made use of word-level resources such as language identification, transliteration, and lexical translation to the features in a single common language and then trained the SVM model to predict the question classes. Translating into a resource-rich language such as English would be helpful since it has QC training data. [6] created a code-mixed question dataset for English–Hindi language pair from college students and evaluated the approach to it. [6] achieves a coarse-grained average accuracy of 63% and fine-grained accuracy of 45%

Table 1 summarizes the discussed papers.

Table 1 Literature review
Full size table

3 Data Creation

Predicting the entity type of the answering sentence for a given question is performed in question classification. QC [7] is performed by classifying the question to a category from a set of predefined categories.

For research and development for QC system, data was obtained. The dataset was created through KBC questions fromFootnote 1 and comprises a total of 1000 questions. For each question, the dataset includes the questions and their question class as the class for the classifier. These questions were tagged into their respective classes using the hierarchical question ontology defined by [3] which is shown in Table 2. All question have been annotated with syntactic features. Further, the complexity of the dataset was analyzed. Since the data is of KBC, it had English–Hindi code-mixed words in it, which were taken care separately.

For Marathi questions, the same questions were translated in Marathi manually and under linguist guidance. So, total parallel dataset of 1000 questions in English and Marathi has been generated. The dataset created is expected to contribute to others for their research and also simulate other research in the field of question classification. Example of the dataset is given Fig. 1.

Fig. 1
figure 1

Dataset example

Full size image
Table 2 Taxonomy for question classification
Full size table

4 Question Classification System

This section describes the QC system in detail. For classification, the taxonomy of question classes proposed by [3] is used. This research mainly focuses on the classification of Marathi questions for which two approaches were used. One is translation and second is direct, i.e., using syntactic features on Marathi words as it is [8].

4.1 Translation Approach

The architecture of the system for translation approach is shown in Fig. 2. Given a question in the Marathi language, we initially perform word-to-word translation. The questions are translated into English. Given a translated question, we transform it into a feature vector and pass it through all the SVMs and output the SVM class which outputs the maximum score. We use Part Of Speech (POS), Named Entity Recognition (NER), and Chunk as the features for all the words in the question [9]. The classifier according to training data classifies the test questions.

4.2 Direct Approach

In this approach, the process starts from the second step of the architecture. Given a question in the Marathi language, we directly transform the Marathi question into a feature vector, rather than translating it, and pass it through all the SVMs and output the SVM class which outputs the maximum score [10]. We use POS, NER, and, Chunk as the features for all the words in the question. Later, the classifier according to training data classifies the test questions.

Fig. 2
figure 2

Architecture

Full size image

To compare and study the results, the English questions were classified using the existing question classification system.

5 Results and Experiments

The overall results are presented in Table 3. It shows the results obtained for English and Marathi dataset for coarse and fine class.

Along with this, we would like to mention the challenges faced in translation. For translation, Goslate API, Google translator, and Shata AnuvaadakFootnote 2 were tried.

The results obtained by Goslate API and Google translator are almost same. But Goslate API is available only limited times a day. None of the above gave better results for an entire question at once; hence, word-level translation was experimented and better results were obtained.

Table 3 Results
Full size table

From the example given in Fig. 3, we can see that word-level translation gives a better result. When the results were compared to the results of Shata Anuvaadak, it was observed that few words remained as it is which it could not translate. In addition, the coverage of the translation model was less.

On the other hand, Google translator translates most of the word; the words that were difficult to translate were transliterated.

Fig. 3
figure 3

Translation example

Full size image

6 Error Analysis

During the course of our experiments, a few observations stood out that highlight the errors made by the approach used to extract the features. We observed that when the features were passed on word level, the results were poor.

Example:

How far is Goa from Gujarat?

numeric : distance

How are you?

Entity: other

Here is an example in which the word how is categorized into two, and in the test example, the word might get wrongly classified. So, it is better to consider the whole sentence for better results.

7 Conclusion and Future Work

This paper presents our proposal of question classification for Marathi questions using translation and direct approach.

For this, we have used syntactic features like POS, NER, Chunk. The experimental results show that for a given set of questions in English and Marathi, the results are almost the same, whereas while using Marathi syntactic features directly gives a poor result.

Further, the features that are available and are useful to improve results can be experimented using better techniques.