Keywords

1 Introduction

Explainable Artificial Intelligence (XAI) aims at explaining the algorithmic decisions of AI solutions with non-technical terms in order to make these decisions trusted and easily comprehensible by humans [1]. If these AI solutions are based on learning algorithms and perceived as black boxes due to their complexity, XAI makes them more transparent and interpretable too. This is of great interest for both logical reasoning in rule engines and Machine Learning (ML) methods. The explanation of a reasoning process can be very difficult, especially when a system is based on a set of complex logical axioms whose logical inferences are performed with, for example, tableau algorithms [4]. Indeed, inconsistencies in logical axioms may be not well understood by users if the system limits to just report the violated axioms. Indeed, users are generally skilled to understand neither formal languages nor the behavior of a whole system. This is crucial for some applications, such as a power plant system where a warning message to the user must be clear and concise to avoid catastrophic consequences. On the other hand, ML methods are based on statistical models of the data where some explanatory variables (i.e., the features) of the data are leveraged in order to predict a dependent variable (i.e., a class or a numeric value). Many statistical methods (e.g., the principal component analysis) are able to detect what are the main involved features in a ML task. These involved features can be used to explain to user the reason of a particular decision. These features are usually handcrafted by human experts and consequently present a shared semantics. Modern Deep Neural Network (DNN) are able to learn these features with no need of human effort. However, the semantics of these learnt features is nor explicit or shareable with humans. Therefore, a human-comprehensible explanation about how and why an AI system took a decision is necessary.

A shared and agreed definition on explainability has not been reached in the AI community so far. Here we follow the definition of Adadi and Berrada [1] that argue for a distinction between interpretability and explainability. The former regards the study of a mathematical mapping between the inputs and the outputs of a black-box system. The latter regards a human comprehension of the logic and/or semantics of such a system. Doran et al. [15] refine the notion of explainability stating that an explainable (or comprehensible) system should provide a reason or justification about its output instead of focusing solely on the mathematical mapping. Moreover, they argue that truly explainable systems have to adopt reasoning engines that run on knowledge bases containing an explicit semantics in order to generate a human comprehensible explanation. In addition, the explainability power depends also on the background knowledge of the users.

To this extent, the logical reasoning associated to semantics is fundamental as it represents a bridge between the output machine and human concepts. This differs from other XAI works that try to analyze the activations of the hidden neurons (i.e., the learnt features) with respect to a given output without attaching a shared semantics. However, logical reasoning on the back-box output is not sufficient as it performs a post-hoc explanation of the black-box guided only by the axioms of a knowledge base. Indeed, no explicit link from the black-box learned features and the concepts in the knowledge base is used. The contribution of the paper addresses this issue.

We propose a novel semantic-based XAI (SeXAI) framework that generates explanations for a black-box output. Differently from Doran et al., such explanations are First-Order logic formulas whose predicates are semantic features connected to the classes of the black box output. Logic formulas are then easy to translate in natural language for a better human comprehension. Moreover, the semantic features are aligned with the neurons of the black box thus creating a neural-symbolic model. This allows reasoning between the output and the features and the improvement of both the knowledge base and the black box output. In addition, the semantics in the knowledge base is aligned with the annotation in the dataset. This is fundamental both for the neural-symbolic alignment and for the black box performance. The latter were tested with experiments on image classification showing that a semantic aligned with the training set outperforms a model whose semantics is deduced from the output with only logical reasoning. The rest of the paper follows with Sect. 2 that provides a state-of-the-art of techniques for generating explanations from logical formulas. Section 3 describes the main concepts of the SeXAI framework whereas Sect. 4 shows a first application and results of the framework in an image classification task. Section 5 concludes the paper.

2 Related Work

The research on XAI has been widely explored in the last years [19], but most of the contributions focused only on the analysis of how learning models (a.k.a. black boxes) work. This is a limited view of the topic since there is a school of thought arguing that an effective explainability of learning models cannot be achieved without the use of domain knowledge since data analysis alone is not enough for achieving a full-fledged explainable system [8]. This statement has been further discussed recently by asserting that the key for designing a completely explainable AI system is the integration of Semantic Web technologies [21, 22, 31]. Semantic Web technologies enabling the design of strategies for providing explanations in natural language [2, 28] where explanations are provided through textual rule-like notation. NLG strategies have been designed also for generating natural language text from triples [38] and for translating SPARQL queries into a natural language form understandable by non-experts [17]. Here, we focused on the integration of semantic information as enabler for improving the comprehensiveness of XAI systems. Our aim is to generate natural language explanations as result of the synergies between neural models and logic inferences for supporting end-users in understanding the output provided by the systems.

The explanation of the logical reasoning in an ontology is implemented with two orthogonal approaches: justifications and proofs. The former computes the minimal subset of the ontology axioms that logically entails an axiom. The latter computes also all the inference steps [27].

One of the first user studies dealing with explanations for entailments of OWL ontologies was performed by [26]. The study investigated the effectiveness of different types of explanation for explaining unsatisfiable classes in OWL ontologies. The authors found that the subjects receiving full debugging support performed best (i.e., fastest) on the task, and that users approved of the debugging facilities. Similarly, [30] performed a user study to evaluate an explanation tool, but did not carry out any detailed analysis of the difficulty users had with understanding these explanations. While, [5] presents a user study evaluating a model-exploration based approach to explanation in OWL ontologies. The study revealed that the majority of participants could solve specific tasks with the help of the developed model-exploration tool, however, there was no detailed analysis of which aspects of the ontology the subjects struggled with and how they used the tool. The work [25] presents several algorithms for computing all the justifications of an entailment in a OWL-DL knowledge base. However, nor study or user evaluation is performed to assess the capability of the computed justifications of the logical entailments. The work in [20] focuses on the explanation, through justifications, of the disclosure of personal data to users (patients and staff) of hospitals. This is performed by translating SWRL rules inconsistencies into natural language utterances. Moreover, the SWRL rules translation is performed axiom by axiom, thus generating a quite long sentence. This could require too much time for reading and understanding. Whereas, our method returns only a single utterance summarizing the whole justification.

Formal proofs are the other form of explanation for logical reasoning. In [33] the authors present an approach to provide proof-based explanations for entailments of the CLASSIC system. The system omits intermediate steps and provides further filtering strategies in order to generate short and simple explanations. The work proposed in [7] first introduced a proof-based explanation system for knowledge bases in the Description Logic ALC [4]. The system generates sequent calculus style proofs using an extension of a tableaux reasoning algorithm, which are then enriched to create natural language explanations. However, there exists no user studies to explore the effectiveness of these proofs. In [29] the authors proposed several (tree, graphical, logical and hybrid) visualizations of defeasible logic proofs and present a user study in order to evaluate the impact of the different approaches. These representations are hard to understand for non-expert users. Indeed, the study is based on participants from a postgraduate course (who have attended a Semantic Web course) and from the research staff. In general, proof algorithms for Description Logic are based on Tableau techniques [4] whereas proof algorithms for other logics are studied in the field of Automated Reasoning [35].

This wide range of approaches to explanation of logical entailments is more focused on the development of efficient algorithms than on effective algorithms for common users. Indeed, all the computed explanations are sets of logical axioms understandable only by expert users. The aim of our work is to provide and effective representation to explanation for all users. This representation is based on the verbalization of the explanation in natural language. This verbalization can be performed by using methods that translate axioms of an OWL ontology in Attempto Controlled English [23, 24] or in standard English [3] with the use of templates. This last work also presents some users’ studies on the quality of the generated sentences. However, these works do not handle with the reasoning results (justifications or proofs), indeed, no strategy for selecting and rendering an explanation is studied.

3 The Framework

In the fields of Machine Learning and Pattern Recognition, a feature is a characteristic or a measurable property of an object/phenomenon under observation [6]. Features can be numeric or structured and they are crucial in tasks such as pattern detection, classification or regression as they serve as explanatory variables. Indeed, informative and discriminating features are combined in a simple or complex manner by the main ML algorithms. This also holds in our everyday experience, a dish composed by pasta, bacon, eggs, pepper and aged cheese (features) is recognized as pasta with Carbonara sauce (the class). Diseases are recognized according to the symptoms (features), or some features of a person (e.g., age, high meat consumption, obesity and sedentary life) can be the cause of a certain disease. The price of the houses is computed according to the features of, e.g., location, square meters and years of the real estate. However, with the rise of Deep Neural Networks (DNN), features are learnt by the system from the raw data without the necessity of handcrafting from domain experts. This has improved the performance of such systems with the drawback of loosing comprehensibility from users. Indeed, DNNs embed the data in a vector space in the most discriminating way without any link to a formal semantics. The aim of SeXAI is to link a DNN with a formal semantics in order to provide a comprehensible explanation of the DNN output to everyday users.

Following the definitions of Doran et al. [15], we ground the notion of explainable system into the concept of a comprehensible system, that is a system that computes its output along with symbols that allow users to understand what are the main semantic features in the data that triggered that particular output. Here, we refine the work of Doran et al. by introducing the concept of semantic feature. These are features that can be expressed through predicates of a First-Order Logic (FOL) language and represent the common and shared attributes of an object/phenomenon that allow its recognition. Examples can be ContainsBacon(x) or ContainsEggs(x) indicating the ingredients of a dish in a picture. Semantic features in principle can be further explained by more fine-grained semantic features. For example, the ChoppedBacon(x) feature can be explained by the HasCubicShape(x) and HasPinkColor(x) features. However, in a nutritional domain, these latter features do not add further comprehension to users and can represent an overload of information. Therefore, the knowledge engineering and/or domain expert have to select the right granularity of the semantic features to present to users and therefore ensuring a sort of atomic property of these features. Semantic features are different from the learnt numeric (and not comprehensible) features of a DNN. The aim of a comprehensible system is to find an alignment between the learnt and the semantic features.

The connection between a DNN output and its semantic features is formalized through the definition of comprehension axiom.

Definition 1 (Comprehension axiom)

Given a FOL language with \(\mathcal {P} = \{O\}_1^n\, \cup \, \{A\}_1^m\) the set of its predicate symbols, a comprehension axiom is a formula of the form

$$ \bigwedge \limits _{i=1}^k O_i(x) \leftrightarrow \bigwedge \limits _{i=1}^l A_i(x) $$

with \(\{O\}_1^n\) the set of output symbols of a DNN and \(\{A\}_1^m\) the corresponding semantic features (or attributes).

A comprehension axiom formalizes the main tasks of a DNN:

  • Multiclass Classification: the predicate \(O_i(x)\) represents a class (e.g., pasta with Carbonara sauce or sushi) for x and \(k = 1\) as a softmax is applied in the last layer of the DNN. The semantic features represent, for example, ingredients contained in the recognized dish.

  • Multilabel Classification: \(O_i(x)\) is part of a list of predicates being computed by the DNN (e.g., dinner and party) for x and \(k > 1\) as a sigmoid is applied in the last layer of the DNN. The semantic features represent, for example, objects in the scene, such as, pizza, table, bottles, person and balloons.

  • Regression: \(O_i(x)\) can be part of a list of predicates being computed by the DNN (e.g., the asked price and the real values of house) for x. Here \(k \ge 1\) with a sigmoid applied in the last layer of the DNN. The semantic features are properties of interest for buying a house.

Once a set of comprehension axioms is returned by our comprehensible system, the former can be easily transformed into a graph representation where the nodes are the unary predicates \(O_i\) and \(A_i\) plus other information such as a possible neural network scores for these predicates. The edges are the logic relations between these predicates, such as implications and n-ary predicates with \(n > 1\). A single comprehension axiom can be represented as a star-shape graph with O in the center, \(A_i\) at the end of the branches and the biimplications as edges. A graph representation can be easily rendered as an image or a natural language sentence used in a dialogue with the user. All these kinds of rendering can be performed with automatic tools. Moreover, the predicates \(A_i\) can be linked with other predicates through logical relations. This would make the explanation more structured with more information for the users. In addition, such a structured representation can be easily queried with languages such as SPARQL. Graph representations for explanations are proved to be effective for the users, that is, more comprehensible, in persuasion systems for healthcare [12]. In this case, explanations are attached to a user model in order to return a tailored explanation according to the user obstacles and capacities.

Fig. 1.
figure 1

In the SeXAI framework data are annotated with symbols of a knowledge base. A symbolic system is aligned with a DNN in order to provide an output and a set of semantic features consistent with the comprehension axioms in the knowledge base.

Full size image

We present the SeXAI framework for comprehensible systems in Fig. 1. The knowledge base \(\mathcal {KB}\) contains both the predicate symbols in \(\mathcal {P}\) and the comprehension axioms for annotating the data. This annotation requires an effort that depends on both the classification task (annotating a picture with bounding boxes requires more effort than annotating the scene without bounding boxes) and on the complexity of \(\mathcal {KB}\). Indeed, if the output is linked with many semantic features this would make the annotation challenging. However, this effort can be alleviate with the use of crowdsourcing and of emerging approaches and tools for annotating documents [34]. Among those, ProdigyFootnote 1 is one of the most promising and targeted at tasks typical for Artificial Intelligence, Machine Learning, Natural Language Processing, and Computer Vision. Prodigy is also open to the customisation of annotation tags linked with concepts coming from a domain knowledge base. The comprehension axioms are passed to the symbolic system that is in charge of i) analyzing the output of the DNN and the associated semantic features; ii) reasoning about them according to the comprehension axioms; iii) returning a, possibly refined, output along with the related semantic features. This makes the proposed framework a local interpreter that provides a reason (the semantic features) for a given output. This architecture extends the one in [15], where a reasoner computes the explanation of the output, with a semantic module that enables several tasks that improve the comprehension and the transparency (i.e., the interpretation) of the DNN:

  • Output and semantic features refinement: The DNN is trained to return both the output and the semantic features. Then, with the use of fuzzy reasoning or neural-symbolic systems [9, 11, 13, 14, 32], both outputs can be refined according to the comprehension axioms and to the evidence coming from the scores of the DNN.

  • Feature Alignment: Once a DNN is trained, it is possible to analyze which are the most activated neurons of the last hidden layer [18] for each semantic feature. In this manner, we can align the high-level features of the DNN with the semantic features in \(\mathcal {KB}\). Such an alignment can also be learnt with an additional supporting neural network [36]. In this way SeXAI would become a model-specific interpretable model.

  • Causality: Once the features alignment is performed, the system can turn off the neurons corresponding to a given semantic feature and check the performance degradation with respect to the output. No degradation of the performance means that the particular semantic feature has just a correlation with the output and, therefore, it can be removed from the corresponding comprehension axiom or stated as a simple correlation. On the other hand, a degradation of the performance indicates a causality of the semantic features with respect to the output. The more the performance degrades the higher the causality degree for that feature is. This makes the proposed system a local interpretable model.

  • Knowledge base improvement: Once the importance degree of the semantic features is computed it can be used as a prior weight in the comprehension axioms and thus enriching \(\mathcal {KB}\).

  • Model improvement: Analyzing the semantic features returned by wrong output predictions allows the system to detect the presence of some common semantic features that alter some predictions. Therefore, the model can assign a lower weight to the neurons aligned with that semantic features.

The comprehension axioms could in principle express different information for an output O. For instance, in the pasta with Carbonara sauce recipe, some axioms in \(\mathcal {KB}\) state the standard food categories but other axioms could state a slightly different version for Carbonara with different categories. This case can be addressed by assigning different weights to each comprehension axiom. These weights represent the trust of the system to each recipe and can be a-priori defined or learnt from the data. Neural-symbolic systems [9] are able to deal with these weighted axioms in order to return a reliable output along with the more probable involved semantic features.

The symbolic system in SeXAI extends the framework of Doran et al. [15] by computing the alignment of semantic and DNN features that enables the improvement of both \(\mathcal {KB}\) and of the model. Differently, in [15] the reasoner module is able to only generate the output and the semantic features. In addition, our work makes a step forward a more structured definition of explanation by defining the explanation as a feature vector of semantic features. These can be further linked with other predicates in \(\mathcal {KB}\) enabling a more structured (and richer in semantics) representation for the explanation.

4 SeXAI in Action

Section 3 provided the general description of the SeXAI framework that we proposed for increasing the overall comprehensiveness of AI models. In this Section, we show how the SeXAI framework can be instantiated within a real-world scenario. In particular, we applied the SeXAI framework to image classification with the aim of demonstrating how the integration of semantics into an AI-based classification systems triggers both the generation of explanations and, at the same time, an improvement of the overall effectiveness of the classification model.

As described in Sect. 3, the SeXAI framework is composed by different modules that, depending on the scenario in which the framework is deployed, can be instantiated or not. Let us consider a running example of food recognition from images. In this case, the output is the label of the food in the dish (e.g., pasta with Carbonara sauce) and the semantic features are the detected food categories (aged cheese, cold cuts, pasta and eggs) that have high classification scores. In this case a possible rendering of the system output and explanation in natural language would be: “I recognized a pasta with Carbonara sauce dish as I have a good confidence about its food categories: aged cheese, pasta, cold cuts and eggs”. Information about food categories are particularly useful in scenario where physicians are supported by information systems concerning the diet monitoring of people affected by nutritional diseases (e.g., diabetes, hypertension, obesity, etc.). However, as very first evaluation of this framework, we focused on predicting only the semantic features in order to check whether the semantics posed at dataset annotation level is more effective than the semantics posed after a prediction applying simple logical rules. The aim of our experiments is to check the right place for the semantics. This setting does not lose generality as a multitasking neural network can be trained to jointly predict both the food image classification and its semantic features (the contained food categories). The performance of a multitasking setting are the same on the semantic features here presented (see Table 1) and good ones on the food image classification (more than 70% of mean average precision). The aim of our evaluation is expressed by the following research question:

  • RQ: Does the injection of knowledge at data-annotation level improve the explainability of the SeXAI framework?

This involves to check whether the annotation of the dataset with the comprehension axioms improve the quality of the semantic features and therefore the system explainability. We address the research question with a multi-label classification setting of the semantic features. Therefore, by starting from the SeXAI architecture shown in Fig. 1, we instantiated the modules as follows.

  • The “Data” module contains our dataset of food images we used for training the classification model. A more detailed description of the dataset is provided in Sect. 4.1.

  • The “Knowledge Base” contains, beyond a taxonomy of recipes and food categories, the composition of each recipe in terms of its food categories. Recipes compositions are described through object properties within the knowledge base. More specifically, in our scenario we adopted the HeLiS ontology [16] where we have the food category-based composition of more than 8,000 recipesFootnote 2. Each food image is annotated with both the recipe label (pasta with Carbonara sauce) and the corresponding food categories (pasta, cold cuts, etc.).

  • As “Black-box model”, we implemented a DNN trained with recipe/food images annotated with the list of related food categories. Given a food image x, the recipe/food label represents the O(x) output neuron, while the food categories represent the semantic features A(x) output neurons. As mentioned above, we decided to not include the O(x) output neurons and to classify each image by its semantic features A(x). Hence, each neuron of the DNN output layer indicates if one of the food categories contained in the dataset has been detected within the images or not.

  • Finally, in our scenario the “Symbolic System” links together the “Knowledge Base” and the output of the DNN for generating natural language explanations of the classification results.

The evaluation of explanations quality is still an open topic within the AI research area [21]. Moreover, in our scenario, explanations aim to provide a comprehensive description of the output rather than being a vehicle for improving the model. Hence, the evaluation of their language content is not of interest. Instead, the SeXAI framework evaluation, provided in this work, focuses on the effectiveness of exploiting semantic features for both training and classification purposes. As baseline, we used a post-hoc semantic-based strategy where images used for training the DNN were annotated only with the corresponding recipe label. In our running example, the baseline would automatically retrieve the corresponding food categories from HeLiS once the DNN classified the input image with, e.g., pasta with Carbonara sauce. Here, the list of food categories has been extracted after the classification of each images by exploiting the predicted recipe label. Figure 2 shows the building blocks of the baseline. For readability, hereafter we will refer to the instantiation of the SeXAI framework as “multi-label classifier”, while the baseline will be labeled as “single-label classifier”.

Fig. 2.
figure 2

The architecture of the baseline system we used for comparing the effectiveness of the SeXAI framework concerning the food images classification task.

Full size image

4.1 Quantitative Evaluation

In the considered scenario, a good performance on recognizing food categories is important as the misclassification of images could trigger wrong behaviors of the systems in which the classifier is integrated. For example, if the framework would be integrated into a recommendation system, a misclassification of a food image would lead to the generation of wrong messages or even no message to the target user.

The Food and Food Categories (FFoCat) DatasetFootnote 3. We leverage the food and food category concepts in HeLiS for the multi-label classification. However, current food image datasets are not built with these concepts as labels, so it was necessary to build a new dataset (named FFoCat) with these concepts. We start by sampling some of the most common recipes in Recipe and use them as food labels. The food categories are then automatically retrieved from BasicFood with a SPARQL query. Examples of food labels are Pasta with Carbonara Sauce and Baked Sea Bream. Their associated food categories are Pasta, AgedCheese, VegetalOils, Eggs, ColdCuts and FreshFish, VegetalOils, respectively. We collect 156 labels for foods (Recipe concept) and 51 for food categories (BasicFood concept). We scrape the Web, using Google Images as search engine, to automatically download all the images related to the food labels. Then, we manually clean the dataset by checking if the images are compliant with the related labels. This results in 58,962 images with 47,108 images for the training set and 11,854 images for the test set (80–20 ratio of splitting). Then, we use the comprehension axioms to annotate the images with both the food label (the output O) and the corresponding food categories labels (that are our semantic features \(A_i\)). This has been obtained by leveraging HeLiS properties, we enrich the image annotations with the corresponding food category labels to perform multi-label classification. The dataset is affected by some natural imbalance, indeed the food categories present a long-tail distribution: only few food categories labels have the majority of the examples. On the contrary, many food categories labels have few examples. This makes the food classification challenging.

Experimental Settings and Metrics. For both multi and single-label classification we separately train the Inception-V3 network [37] from scratch on the FFoCat training set to find the best set of weights. The fine tuning using pre-trained ImageNet [10] weights did not perform sufficiently. This is probably due to the fact that the learnt low-level features of the first layers of the network belong to general domains and do not match properly with the specific Mediterranean food domain. Indeed, many food images in ImageNet belong to oriental food. For the multi-label classification, we use a sigmoid as activation function of the last fully-connected layer of the Inception-V3 and binary cross entropy as loss function. This is the standard setting for multi-label classification. Regarding the single-label classification, the activation function of the last fully-connected layer is a softmax and the loss function is a categorical cross entropy. This is the standard setting for multi-class classification. We run 100 epochs of training with batch size of 16 and a learning rate of \(10^{-6}\). At each epoch images are resized to \(299\times 299\) pixels to fit the input format required by Inception-V3 and are augmented by using rotations, width and height shifts, shearing, zooming and horizontal flipping. This results in a training set 100 times bigger than the initial one. We used early stopping (with a patience of 15 iterations) to prevent overfitting. The training has been performed with the Keras framework (TensorFlow as backend) on a PC equipped with a NVIDIA GeForce GTX 1080. All these details can be checked in the link pointing at the source code.

As performance metric we use the mean average precision (MAP) that summarizes the classifier precision-recall curve: \(MAP = \sum _{i=1}^n(R_n - R_{n-1})P_n\), i.e., the weighted mean of precision \(P_n\) achieved at each threshold level n. The weight is the increase of the recall in the previous threshold: \(R_n - R_{n-1}\). The macro AP is the average of the AP over the classes, the micro instead considers each entry of the predictions as a label. We preferred MAP instead of accuracy as the latter for sparse vectors can give misleading results: high results for output vectors with all zeros.

Results. Given an (set of) input image(s) , the computing of the precision-recall curve requires the predicted vector(s) of food category labels and a score associated to each label in . In the multi-label method this score is directly returned by the Inception-V3 network (the final logits). In the single-label and inference method this score needs to be computed. We test two strategies: (i) we perform exact inference of the food categories from HeLiS and assign the value 1 to the scores of each ; (ii) the food categories labels inherit the uncertainty returned the DNN: the score of each \(y_i\) is the logit value \(s_i\) returned by . Results are in Table 1. The direct multi-label has very good performance (both in micro and macro AP) in comparison with the single-label models. The micro-AP is always better than the macro-AP as it is sensible to the mentioned imbalance of the data. This means that errors in the single food classification propagate to the majority of the food categories the given food contains. That is, the inferred food categories will be wrong because the food classification is wrong. On the other hand, errors in the direct multi-label classification will affect only few food categories. We inspected in more detail some of the errors committed by the classifiers in order to have a better understanding of their behaviors. In some cases, the single-label method misclassified an image with Backed Potatoes as Backed Pumpkin thus missing the category of FreshStarchyVegetables. Another image contains a Vegetable Pie but the single-label method infers the wrong category of PizzaBread. In another image, this method mistakes Pasta with Garlic, Oil and Chili Peppers with Pasta with Carbonara Sauce, thus inferring wrong Eggs and ColdCuts. Here the multi-label method classifies all the categories correctly. Therefore, the multi-label method allows a more fine grained classification of the food categories with respect tto the single-label method. The latter has better results if the score returned by the DNN is propagated to the food categories labels withe respect to the exact inference.

Table 1. The multi-label classification of food categories outperforms in average precision (AP) the methods based on single-label classification and logical inference.
Full size table

4.2 Discussion

The experience of designing the SeXAI framework and the analysis of results obtained from a preliminary validation within a real-world use case highlighted two important directions towards the long-term goal of achieving a fully-explainable AI system.

First, the integration of semantic features with black-box models enabled the generation of comprehensive explanations. SeXAI can be considered a neuro-symbolic framework conjugating the effectiveness of black-box models (e.g., DNN) with the transparency of semantic knowledge that, where possible, can support the generation of explanations describing the behavior of AI systems. This aspect opens to a very interesting and innovative research direction centered on the content of the generated explanations. Indeed, the integration of semantic features for generating explanations can be exploited for refining the statistical model itself (as described in Sect. 3). For instance by analyzing correlations between the presence of specific semantic features within explanations and the performance of the black-box model. Future work will focus on strengthening this liaison within the SeXAI framework in order to validate if an inference process could improve the classification capability and, at the same time, to observe how inference results could be exploited for refining the black-box model.

Second, the integration of semantic features can lead to better classification performance. Results presented in Table 1 show that through the integration of semantic features, it is possible to improve the overall effectiveness of the black-box model. This is a very interesting finding since it confirms the importance of a by-design integration of semantic features. Future activities will further investigate this hypothesis within other scenarios with the aim of understanding which are the boundaries and if there exist some constraints in the application of this strategy. For instance, the granularity of semantic features with respect to the entities that have to be classified could play an important role. Hence, a trade-off has to be found in order to maintain the explainable capability of the system and, at the same time, an acceptable effectiveness of the classification model.

5 Conclusions

The aim of Explainable Artificial Intelligence is to provide black-box algorithms with strategies to produce a reason or justification for their outputs. This is fundamental to make these algorithms trusted and easily comprehensible by humans. A formal semantics, provided by knowledge bases, encoded in a logical language allows the connection between the numeric features of a black box and the human concepts. Indeed, a justification in a logical language format can be easily translated in natural language sentences in an automatic way.

In this paper, we presented the first version of SeXAI, a semantic-based explainable framework aiming at exploiting semantic information for making black boxes more comprehensible. SeXAI is a neural-symbolic system that analyses the output of a black box and creates a connection between the learnt features and the semantic concepts of a knowledge base in order to generate an explanation in a logical language. This allows reasoning on the black box and its explanation, the improvement of the knowledge base and of the black box output. The semantics in the knowledge base is aligned with the annotations in the dataset. This improves the performance of SeXAI on a task of multi-label image classification with respect to a system that performs solely logical reasoning on the black box output. Therefore, we satisfied our research question about the right place for the semantics: a semantics at data-annotation level improves the explainability of the system.

As future work, we will perform some experiments on the quality of the alignment between the learnt and the semantic features. In particular, we will evaluate the degree of causality of the semantic features with respect to the output and how the attention of a black box can be moved towards the semantic features in order to improve the model performance.