Keywords

1 Introduction

It is reasonable to expect that agents could be more effective at achieving their goals during social interactions with other agents if they understand the other agents involved. However, understanding other agents requires the capability of modelling and reasoning about other agents’ mental attitudes. These characteristics are intrinsic to Theory of Mind (ToM) [12].

Normally, agents operate under conditions of uncertainty due to the dynamism of the environments in which they are situated [45]. Modelling other agents’ minds also involves uncertainty. ToM involves uncertainty not only due to the dynamism of other agents’ mental attitudes, e.g., agents might change their beliefs constantly, but also because it involves the uncertainty of a message reaching its audience (i.e., the uncertainty of the communication channels working properly), the uncertainty of other (autonomous) agents telling/acting truly, and the uncertainty of an agent reading other agents’ mental attitudes correctly during interactions.

Various studies have investigated the application of ToM in Multi-Agent Systems (MAS). Among them, [8, 9] investigated the advantages of using different levels of ToM in games played by agents, [33] investigated the role of ToM in modelling dishonest attitudes in MAS, and [3, 15, 16, 27, 38] show the advantages of modelling the opponent when considering strategies in argumentation-based dialogues, even though ToM is not represented explicitly. It seems that modelling other agents’ minds is an important topic of research, and existing results show important contributions to MAS.

However, as described in [42], most of the work considering the modelling of other agents’ minds assume such model as a given, which is an understandable assumption due to the complexity of the problem, but unrealistic. Unfortunately, the question of how to represent the uncertainty of beliefs about others’ beliefs when agents acquire and update the model of other agents’ minds, i.e., uncertain ToM, has not been fully investigated in the literature. We believe that agents should also to be able to reason and make decisions using ToM. Therefore, taking inspiration from others who have investigated the use of others agents’ model during reasoning and decision-making, e.g., [3, 8, 9, 15, 16, 27, 38], we propose an approach to model ToM in software agents that reflects the uncertainty present in agent communication. We also took some inspiration from the STAPLE language. STAPLE (Social and Team Agents Programming Language) has its semantics based on joint intention theory [19]. STAPLE has the goal of reaching a fault-tolerant approach to program teamwork, in which the authors argue that a team is more than a collection of individuals working together to achieve a common goal. The agents in a team must have a shared goal as well as a shared mental state [21]. Thus, STAPLE enables agents to specify the models of other agents, as well temporal properties of actions and events, allowing them to reason about group beliefs, team intentions, and team commitments [20].Footnote 1

Our first contribution is the proposal of an approach to model ToM that reflects the uncertainty of information that agents infer about other agents’ minds through communication. To the best of our knowledge, our work is the first to propose a formal model of how agents acquire and update ToM during communication in MAS given the uncertainty of other agents’ model, particularly in the practical context of a BDI based Agent-Oriented Programming Language (AOPL). This approach allows us to implement multi-agent communication that reflects some desired properties from communication and common knowledge theories [6]. For example, how agents increase the certainty of their ToM by communicating more and, consequently, how communicating more reinforces the already existing model of the mental attitudes of those agents. Our second contribution is showing how agents may use our approach to reach (or not) shared beliefs under conditions of uncertainty, and how agents make decisions using ToM and probabilistic reasoning.

2 Background

2.1 Theory of Mind and the Problem of Other Minds

ToM is the ability of humans to ascribe elements such as beliefs, desires, and intentions, and relations between these elements to other human agents. In other words, it is the ability to form mental models of other agents [1]. There are two major theories about ToM. One theory of ToM is the Theory-Theory of Mind (henceforth TT). TT can be described as a theory based approach for assigning states to other agents. While some argue TT is nothing else but folk psychology, others say that it is a more scientific way of mindreading [13]. The other major theory is Simulation Theory of Mind (henceforth ST), which is described to be ‘process-driven rather than theory-driven’ [2]. In other words, ST emphasises the process of putting oneself into another’s shoes. TT argues for a hypothesis testing method of model extraction, whereas ST argues for a simulation based method for model selection.

An important factor that influences the acquisition, formation, or modification of ToM is uncertainty. Inferring the beliefs of others is a notorious epistemological issue named by philosophers The Problem of Other Minds [17]. The problem still stands since the times of Descartes [23]. It would be unreasonable for one to assume that ToM is absolute or that ToM is a universal set of beliefs shared by all agents in a system. Therefore, we believe that a reasonable approach to model how ToM is acquired and updated by artificial agents has to be able to represent the uncertainty with which agents infer beliefs about other agents’ beliefs.

2.2 Agent Communication Languages

Agent communication languages have been developed based on the speech act theory [41]. Speech act theory is concerned with the role of language as actions. Among the agent communication languages which emerged from the speech act theory, FIPA-ACL [11] and KQML [10] are the best known.

In this work, for practical reasons, we choose KQML, which is the standard communication language in the Jason platform [5], the multi-agent platform we choose to implement this work. Knowledge Query and Manipulation Language (KQML) was designed to support interaction among intelligent software agents, describing the message format and message-handling protocol to support run-time agent communication [10, 25].

In order to make KQML broadly applicable, a semantic framework for KQML was proposed in [22]. The semantics for KQML-based messages in the AgentSpeak programming language, as given in [43] and implemented in Jason [5], formalises how the locutions successively operate on the states of agents, making the references to the mental attitudes of BDI agents explicit, thus addressing some of the problems of ACLs pointed out in [44]. Based on that semantics, we put forward the idea that agents are able to infer the likely model of other agents’ minds during the process of communication, i.e., agents are able to acquire and update ToM, as we describe later in this paper.

2.3 Agent Oriented Programming Languages

Among the many AOPL and platforms, such as Jason, Jadex, Jack, AgentFactory, 2APL, GOAL, Golog, and MetateM, as discussed in [4], we chose the Jason platform [5] for our work. Jason extends the AgentSpeak language, an abstract logic-based AOPL introduced by Rao [37], which is one of the best-known languages inspired by the BDI architecture.

Besides specifying agents with well-defined mental attitudes based on the BDI architecture, the Jason platform [5] has some other features that are particularly interesting for our work, for example strong negation, belief annotations, and (customisable) speech-act based communication. Strong negation helps the modelling of uncertainty, allowing the representation of propositions that the agent: (i) believes to be true, e.g., \(\mathtt {about(paper1,uncertain\_tom)}\); (ii) believes to be false, e.g., \(\mathtt {\lnot about(paper2,uncertain\_tom)}\); (iii) is ignorant about, i.e., the agent has no information about whether a paper is about \(\mathtt {uncertain\_tom}\) or not. Also, Jason automatically generates annotations for all the beliefs in the agents’ belief base about the source from where the belief was obtained (which can be from sensing the environment, communication with other agents, or a mental note created by the agent itself). The annotation has the following format: \(\mathtt {about(paper1,tom)[source(reviewer1)]}\), stating that the source of the belief that \(\mathtt {paper1}\) is about the topic \(\mathtt {tom}\) is \(\mathtt {reviewer1}\). The annotations in Jason can be easily extended to include other meta-information, for example, trust and time as used in [26, 30]. Another interesting feature of Jason is the communication between agents, which is done through a predefined (internal) action. There are a number of performatives allowing rich communication between agents in Jason, as explained in detail in [5]. Further, new performatives can be easily defined (or redefined) in order to give special meaning to themFootnote 2, which is an essential characteristic for this work.

3 Running Example

As a running example, we will take the following university scenario with five agents. The first agent, John, plays the role of a professor in the university, and the other agents, named Bob, Alice, Nick, and Ted, play the role of students. John has a relation of adviser with the students. Also, John is responsible for distributing tasks to students, which the students can accept or refuse. John keeps information about the students, in order to assign tasks that the students are more likely to accept.

Our model can be formally defined as \(\langle Ag, \mathcal {T}, \mathcal {A}, \mathcal {S} \rangle \), in which Ag represents the set of agents, \(\mathcal {T}\) the set of tasks of the kind \(\mathcal {T} \subseteq \mathcal {A} \times \mathcal {S}\), describing an action from \(\mathcal {A}\), requiring knowledge about a subset of subjects from \(\mathcal {S}\), that might be executed to achieve the task \(\mathcal {T}\). In our example, we consider the following actions, subjects, and tasks:

  • \(\mathcal {A} = \{\mathtt {write\_paper}, \mathtt {review\_paper}, \mathtt {paper\_seminar}\}\)

  • \(\mathcal {S} = \{\mathtt {mas}, \mathtt {kr}, \mathtt {tom}\}\)

  • \(\mathcal {T}\) = \( \left\{ \begin{array}{l} \mathtt {task(write\_paper,[mas,tom])} \\ \mathtt {task(review\_paper,[kr])} \\ \mathtt {task(paper\_seminar,[tom,mas])} \end{array} \right\} \)

For example, the task to write a paper with the subjects MAS and ToM, \(\mathtt {task(write\_paper,[mas,tom])}\), requires competence on both subjects: \(\mathtt {mas}\) and \(\mathtt {tom}\). Thus, this task has a greater likelihood to be accepted by a student who desires to execute that particular task, or who likes to execute the action \(\mathtt {write\_paper}\) and believes that itself knows the necessary subjects (e.g., \(\mathtt {knows(mas)}\) and \(\mathtt {knows(tom)}\) are necessary to execute this example task). Thus the probability of an agent ag to accept a task \(t_i\) is given by the following equation:

$$\begin{aligned} {{\small P(accepts(ag,task_i)) = {\left\{ \begin{array}{ll} P({Des}_{ag}{(task_i)}) \qquad \qquad \! \quad \text{ if } {Des}_{ag}{(task_i)} \in \varDelta _{ John } \\ P({Bel}_{{ag}}{(likes(a_i))}) \times P({Bel}_{{ag}}{(knows(S'))}) \qquad \!\!\!\!\!\!\! \text{ otherwise } \end{array}\right. } }} \end{aligned}$$

with

$$\begin{aligned} {P({Bel}_{{ag}}{(knows(S'))}) = \prod _{s_i \in S'} P({Bel}_{{ag}}{(knows(s_i))})} \end{aligned}$$

where \(task_i = \mathtt {task(a_i,S')}\), for \(task_i \in \mathcal {T}\), \(a_i \in \mathcal {A}\), and \(S' \subseteq \mathcal {S}\). \(\varDelta _{ John }\) represents John’s knowledge.

Thus, considering our scenario, when John knows that some student ag likely desires to execute a particular task \(task_i\), i.e., \({Des}_{ag}{(task_i)}\), it can use this information to assign the task. Otherwise, John can calculate the likely acceptance for each student ag, based on the probability of each student to like executing that action, \(P({Bel}_{{ag}}{(likes(a_i))})\), and the knowledge the student has about each of the required subjects \(P({Bel}_{{ag}}{(knows(S'))})\). Note that, while modelling the students’ desires is more difficult to obtain in our scenario, the students’ beliefs are easily obtained by John, given that John frequently talks to students about these subjects and tasks.

In reality, agents operate with uncertain information, especially in the cases of thinking about other agents’ minds. The minds of others are considered to be some sort of black boxes that are more or less accessible depending on the given scenario. Reasoning under uncertainty is a classic case where bounded rationality acts as a major constraint on what agents can infer from their beliefs. However, even if agents are constrained by their access to information, it does not mean that the agents cannot reach reasonable conclusions about the minds of other agents [14, 23].

In our scenario, John will reason and make decisions based on information it has about the students’ minds, i.e., information from its ToM. Thus John will reach conclusions based on uncertain information, given that its ToM contains information about students’ minds that has been estimated through the communication John has had with the students. Considering that an approach to reason about uncertain information, uncertain ToM in our case, is using probabilistic reasoning, as described in [14], we have modelled John’s decision-making process based on the probability of each information in John’s ToM to be correct, considering some factors of uncertainty we will describe further in this paper.

4 Modelling ToM from Other Agents’ Actions

In this paper, we are going to consider the modelling of ToM based on communication only, which can be considered a general approach for any application, based on the semantics of each speech act used. On the other hand, the semantics for other actions, e.g., actions agents execute in the environment, might have different meaning according to different application domains.

In order to describe our approach, we use the following notation: \({Bel}_{{ag}}{(\psi )}\) means that an agent ag believes \(\psi \); \({Des}_{ag}{(\psi )}\) means that an agent ag desires \(\psi \); \(\varDelta _{ag}\) represents the ag’s knowledge base. Two distinct agents are represented using \(ag_i\) and \(ag_j\), with \(ag_i, \ ag_j \in Ag\), and \(ag_i \ne ag_j\). We label the updates agents execute in their ToM with \(\gamma \), which can be used to represent the uncertainty of that information. In Sect. 5, we propose an approach for uncertain ToM, which is a particular instance for such \(\gamma \) label.

The speech acts considered in this particular work and their semantics are based on our work in [29]. Messages are represented as \(\langle {\mathtt {sender}},{\mathtt {receiver}},{\mathtt {performative}},{\mathtt {content}} \rangle \), and the meaning of each message is associated with the performative used:

  • \(\langle {ag_i},{ag_j},{\texttt {tell}},{\psi } \rangle \) means a message sent by agent \(ag_i\) to agent \(ag_j\), with the tell performative, and content \(\psi \). When \(ag_i\) sends this message, it carries out the following updateFootnote 3 on its ToM:

    $$\begin{aligned} {\varDelta _{ag_i} = \varDelta _{ag_i} \cup {Bel}_{{ag_j}}{(\psi )}_{[\gamma ]}} \end{aligned}$$
    (1)

    When \(ag_j\) receives this message, it carries out the following update on its ToM:

    $$\begin{aligned} {\varDelta _{ag_j} = \varDelta _{ag_j} \cup {Bel}_{{ag_i}}{(\psi )}_{[\gamma ]}} \end{aligned}$$
    (2)

  • \(\langle {ag_i},{ag_j},{\texttt {ask}},{\psi } \rangle \) means a message sent by agent \(ag_i\) to agent \(ag_j\), with the ask performative, and content \(\psi \). When \(ag_i\) sends this message, it carries out the following update on its ToM:

    $$\begin{aligned} {\varDelta _{ag_i} = \varDelta _{ag_i} \cup {Bel}_{{ag_j}}{({Des}_{ag_i}{(\psi )})}_{[\gamma ]}} \end{aligned}$$
    (3)

    When \(ag_j\) receives this message, it carries out the following update on its ToM:

    $$\begin{aligned} {\varDelta _{ag_j} = \varDelta _{ag_j} \cup {Des}_{ag_i}{(\psi )}_{[\gamma ]}} \end{aligned}$$
    (4)

Before introducing our approach for uncertain ToM, imagine that ToM could be modelled without uncertainty, i.e., that we could ignore \(\gamma \) in our semantic rules. Then, based on these simple semantic rules for agent communication, we are able to show that agents can reach shared beliefs in a relatively straightforward way [29].

Definition 1

(Shared Beliefs Using ToM). An agent \(ag_i\) will reach a state of shared beliefs with another agent \(ag_j\) when, for a belief \(\varphi \), it is able to match its own belief \(\varphi \) with a ToM about \(ag_j\) believing \(\varphi \), i.e., \(\varphi \wedge {Bel}_{{ag_j}}{(\varphi )}\).

Example (Shared Beliefs Without Uncertainty): Following the scenario introduced, imagine that two students, Alice and Bob, need to work together to accomplish a particular task paper_seminar, which requires the subjects mas (Multi-Agent Systems) and tom (Theory of Mind). Also, while Alice only knows the subject of mas, Bob only knows the subject of tom. Considering that both Alice and Bob need to know both topics in order to help each other during the paper seminar, they decide to exchange knowledge about these topics. Thus, they might reach some shared beliefs (knowledge) about both topics. Note that, in this scenario, Alice and Bob assume that both are cooperating and both are rational. Thus, Bob starts the dialogue telling Alice that “Theory of Mind is an approach to model others’ minds”, i.e., \(\langle \)alice, tell, def(tom, “an approach to model others’ mind”)\(\rangle \). At that moment, following the semantics for the tell performative (Eq. (1)), Bob updates its ToM with the following information \({Bel}_{{alice}}{(\texttt {def(tom})}\), “an approach to model others’ minds”). After that, when Alice receives this message, following the semantics for the tell performative (Eq. (2)), Alice updates its belief base with the following information def(tom, “an approach to model others’ mind”), as well as Alice updates its ToM about Bob with \({Bel}_{{bob}}{(\texttt {def(tom})}\), “an approach to model others’ minds”). At this moment, both Alice and Bob reach a state of shared belief about the definition of tom, according to Definition 1.

However, agents operate under conditions of uncertainty in a MAS, and the previous assumptions are hard to obtain; thus, agents will face uncertainty about their ToM, and consequently about their shared beliefs. For example, when an agent sends a message, it faces the uncertainty of the communication channel, i.e., the uncertainty of the message reaching the receiver. Also, when receiving a message, an agent faces the uncertainty of the truth of that statement, e.g., an agent is not able to verify if the other agents are acting maliciously [33, 40], thus it needs to consider the uncertainty of information it receives for those agents based on how much it trusts them [26, 34, 35].

One manner to overcome the uncertainty and reach a more accurate ToM, following the literature on common knowledge [6], is increasing the communication between agents. Thus, an agent is able to increase the certainty on a given agent \(ag_j\) believing \(\varphi \), confirming whether its ToM about agent \(ag_j\) believing \(\varphi \) is correct. That is, the agent is able to infer that \(ag_j\) believes \(\varphi \) by reinforcing this belief through communication. Henceforth we describe our model for uncertain ToM, which is compatible with that behaviour.

5 A Model of Uncertain ToM

In this section we propose an approach to model ToM that reflects the uncertainty present in MAS. In order to show our approach, we are going to consider some parameter values. The first, \(\alpha \), reflects the uncertainty of the communication channel when sending a message. The second, \(\beta \), reflects the uncertainty of the other agents telling the truth, i.e., when an agent \(ag_i\) tells \(\varphi \) to agent \(ag_j\), agent \(ag_j\) is able to model that \(ag_i\) believes on \(\varphi \) with a degree of certainty equal to \(\beta \). For simplicity, we will assume that an agent will model its ToM about the other agents with a degree of certainty equal to the trust it has on the sourceFootnote 4, following the ideas introduced in [34, 35].

Definition 2

The label \(\gamma \) will be instantiated with \(\gamma = (\alpha , t)\) for an agent sending a message, and \(\gamma = (\beta , t)\) for an agent receiving a message, where \(\alpha \) represents the uncertainty of the message reaching the target, \(\beta \) the uncertainty of the sender telling the truth, and t a discrete representation of the time of the MAS in which the message was exchanged.

Thus, following Definition 2, a trace of different updates on the ToM is constructed over time. Note that \(\alpha \) and \(\beta \) reflect the uncertainty of an update at a given time. In order to execute reasoning over the ToM, agents are able to use the trace of these updates to calculate the degree of certainty on their model. Using this trace, we are able to model some desired behaviour from communication theory in agent communication, as we will describe later in this paper.

For example, considering our scenario, when Bob tells Alice that “Theory of Mind is an approach to model others’ minds”, considering also that Bob knows that the efficiency of the communication channel is 0.9, i.e., \(\alpha = 0.9\), Bob will update its ToM, following the semantics for the tell performative (Eq. (1)) and Definition 2, with the information \({Bel}_{{alice}}{(\texttt {def(tom})}\), “an approach to model others’ minds”)\(_{[(0.9,t_i)]}\), with \(t_i\) the discrete time when the communication occurred. When Alice receives this message, considering that the trust Alice has on Bob telling the truth is 0.8, i.e., \(\beta = 0.8\), and following the semantics for the tell performative (Eq. (2)) and Definition 2, Alice updates its ToM with \({Bel}_{{bob}}{(\texttt {def(tom})}\), “an approach to model other minds”)\(_{[(0.8,t_j)]}\), with \(t_j\) the discrete time at which the message was received, with \(t_i < t_j\). Both Alice and Bob model uncertainty of their ToM about each other believing on the definition of ToM.

Considering uncertain ToM, we need to redefine shared beliefs, in order to reflect the uncertainty of agents’ models.

Definition 3

(Shared Beliefs Using Uncertain ToM). An agent \(ag_i\) will reach a state of shared beliefs with another agent \(ag_j\) when, for a belief \(\varphi \), it is able to match its own belief \(\varphi \) with a ToM about \(ag_j\) believing \(\varphi \) with a predetermined degree of certainty \(\chi \), i.e., \(\varphi \wedge P({Bel}_{{ag_j}}{(\varphi )}) \ge \chi \), with \(\chi \) a value describing the certainty necessary to consider \(\varphi \) a shared belief.

Following the literature on common knowledge [6], if two individuals \(ag_i\) and \(ag_j\) can repeatedly communicate, then they can repeatedly reinforce their mental state regarding an information \(\varphi \). For example, telling each other that \(\varphi \) is true, they should increase the certainty of each others’ belief in \(\varphi \). In order to model this desired behaviour in our model, we maintain the trace of all updates an agent executes in its ToM, and using this trace we are able to aggregate different pieces of evidence in order to increase the certainty on ToM. There are many different ways to model this desired behaviour on agent communication, and it could consider the particularities of each application domain. In our scenario, the information communicated by agents, e.g. a concept definition, does not change over time. Thus, for simplicity, we do not weight each information according to the time it was received and the current time of the MAS, we only consider the number of evidences about that information. Thus, we model this desired behaviour using the following equation:

$$\begin{aligned} {P({Bel}_{{Ag}}{(\varphi )}) = {\left\{ \begin{array}{ll} f({Bel}_{{ag}}{(\varphi )}) &{}\text{ if }\, f({Bel}_{{ag}}{(\varphi )}) <= 1\\ 1 &{} \text{ otherwise } \end{array}\right. } } \end{aligned}$$
(5)
$$\begin{aligned} {f({Bel}_{{ag}}{(\varphi )}) = \frac{\sum \limits _{t_i\in \varDelta T} v \ | \ {Bel}_{{ag}}{(\varphi )}_{[(v,t_i)]}}{|\varDelta T|} + (\lambda \times |\varDelta T|)} \end{aligned}$$
(6)

with \(\varDelta T\) the number of occurrences of \({Bel}_{{ag}}{(\varphi )}_{[(v,t_i)]}\) in the agent ToM, and \(\lambda \) the evidence factor, i.e., a parameter that reinforce the certainty on that information according to how often it occurs in the trace. Equation 6 calculates the average of the trace for \({Bel}_{{ag}}{(\varphi )}\) plus the evidence factor. Thus, following Definition 3, \(ag_i\) is able to reach a state of shared belief with another agent \(ag_j\) about a belief \(\varphi \) when it is able to infer \(P({Bel}_{{ag_j}}{(\varphi )}) \ge \chi \) from Eq. 5 with \(\chi = 1\), for example.

Proposition 1

(Reaching Shared Beliefs—ToM with Uncertainty). When \(\lambda \) is a positive value, agents are able to eventually reach a state of shared beliefs, even considering \(\chi \) = 1, provided they communicate the same information repeatedly. Also, the greater the value of \(\lambda \), the faster agents will reach the state of shared beliefs.

Example (Shared Beliefs Under Conditions of Uncertainty): Following our example, imagine that Bob wants to reach a state of shared beliefs with Alice about the definition of ToM under the conditions of uncertainty described above. Thus, after sending the previous message and updating its ToM with \({Bel}_{{alice}}{(\texttt {def(tom})}\), “an approach to model others’ minds”)\(_{[(0.9,t_i)]}\), Bob has two options to increase the certainty on its ToM about Alice believing on that definition: (i) telling Alice that definition more times, or (ii) asking Alice the definition of ToM and waiting for an answer from Alice, in which Alice tells Bob the definition of ToM. Considering \(\lambda = 0.1\), in the first case, when Bob tells Alice about the definition of ToM one more time, following the semantics for the tell performative (Eq. (1)) and Definition 2, Bob adds \({Bel}_{{alice}}{(\texttt {def(tom})}\), “an approach to model others’ minds”)\(_{[(0.9,t_j)]}\) to its ToM, with \(t_i < t_j\). Thus, Eq. 5 returns 1, considering the average \(0.9 + 0.2\) from the evidence factor, which is 0.1 multiplied by the number of evidences (Eq. (6)). Also, following the semantics for the tell performative (Eq. (2)) and Definition 2, Alice updates its ToM with \({Bel}_{{bob}}{(\texttt {def(tom})}\), “an approach to model other minds”)\(_{[(0.8,t_j)]}\), and Eq. (5) returns 1, considering the average \(0.8 + 0.2\) from the evidence factor (Eq. (6)). Thus, they reach a state of shared belief about the definition of ToMFootnote 5, considering \(\chi = 1\) in Definition 3. In the other case, Bob asks to Alice to tell him the definition of ToM, and it waits for the answer. When Alice tells Bob the definition of ToM, Alice and Bob update their ToM with \({Bel}_{{bob}}{(\texttt {def(tom})}\), “an approach to model other minds”)\(_{[(0.9,t_j)]}\), \({Bel}_{{alice}}{(\texttt {def(tom})}\), “an approach to model others’ minds”)\(_{[(0.8,t_i)]}\), respectively. For both, Eq. (5) returns 1, considering the average \(0.85 + 0.2\) from the evidence factor, reaching a state of shared beliefs about the definition of ToM according to Definition 3 with \(\chi = 1\).

6 Decision Making Using Uncertain ToM

Apart from enabling agents to model other agents’ minds and allowing them to improve their models during communicative interactions, it is also essential that agents are able to make decisions using these models. Normally, a decision-making process is associated with the application domain, i.e., it is domain dependent. Therefore, we will present the decision-making process for the task assignment problem introduced in Sect. 3.

In our scenario, during advising sessions, John asks students about different tasks they like to execute, as well as the different subjects the students are reading about (the subjects the students know about). Thus, John acquires ToM about the students, and its ToM becomes more accurate as they have more advising sessions, and consequently they communicate more with each other.

$$\begin{aligned} \begin{array}{l} John_{_{ToM}} = \left\{ \begin{array}{l} {{Bel}_{{alice}}{(likes(paper\_seminar))}_{[0.8]}}\\ {{Bel}_{{alice}}{(likes(write\_paper))}_{[0.7]}}\\ {{Bel}_{{bob}}{(likes(review\_paper))}_{[0.9]}}\\ {{Bel}_{{bob}}{(likes(write\_paper))}_{[0.8]}}\\ {{Bel}_{{nick}}{(likes(review\_paper))}_{[0.6]}}\\ {{Bel}_{{nick}}{(likes(write\_paper))}_{[0.5]}}\\ {{Bel}_{{ted}}{(likes(write\_paper))}_{[0.8]}}\\ {{Bel}_{{ted}}{(likes(paper\_seminar))}_{[0.4]}}\\ {{Bel}_{{ted}}{(likes(review\_paper))}_{[0.6]}}\\ \end{array} \right\} \\ \end{array} \end{aligned}$$

For example, John has asked (in different meetings and times) Bob, Alice, Nick, and Ted which academic tasks they like to execute, e.g., \(\langle bob,\)\( {AskIf}, \mathtt {likes(T)} \rangle \). After receiving this message, according to the semantic rule for the ask performative (Eq. (4)), each student knows that John desires to know which task they like to execute. Based on this knowledge, each student has answered to John the tasks they like to execute, John has received these messages and updated its ToM as shown in \( John_{_{ToM}} \)Footnote 6.

Continuing with the example, during a meeting Alice asks John if there is any scheduled paper seminar about ToM and MAS, i.e., \(\langle \mathtt {john}, {AskIf},\)\( \mathtt {task(paper\_seminar,[tom,mas]} \rangle \). Thus, based on the semantic rule for the ask performative (Eq. (4)), John models that Alice is likely to desire that task, i.e., \({Des}_{alice}{(\mathtt {task(paper\_seminar,[tom,mas])})}_{[0.7]}\), answering positively. Also, imagine that John has asked the students which subject they have knowledge about, resulting in the following additional information to John’s ToM:

$$\begin{aligned} \begin{array}{l} John_{_{ToM}} = \left\{ \begin{array}{ll} {{Bel}_{{alice}}{(knows(tom))}_{[0.8]}} &{}\qquad {{Bel}_{{bob}}{(knows(mas))}_{[0.8]}}\\ {{Bel}_{{alice}}{(knows(mas))}_{[0.9]}} &{}\qquad {{Bel}_{{bob}}{(knows(kr))}_{[0.9]}}\\ {{Bel}_{{nick}}{(knows(kr))}_{[0.8]}} &{}\qquad {{Bel}_{{ted}}{(knows(tom))}_{[0.8]}}\\ {{Bel}_{{nick}}{(knows(mas))}_{[0.7]}} &{}\qquad {{Bel}_{{ted}}{(knows(kr))}_{[0.5]}}\\ {{Bel}_{{nick}}{(knows(tom))}_{[0.8]}} &{}\qquad {{Bel}_{{ted}}{(knows(mas))}_{[0.8]}}\\ \end{array} \right\} \\ \end{array} \end{aligned}$$
Table 1. Likelihood calculation for task assignment
Full size table

Using its ToM, John executes the probabilistic reasoning described in Sect. 3, which computes the likelihood for each student to accept each task as shown in Table 1. Note that the likelihood of Alice accepting the task \(\mathtt {paper\_seminar}\) is based on the information \({Des}_{alice}{(\mathtt {task(paper\_seminar,[tom,mas])})}_{[0.7]}\) in John’s ToM, while the other results are based on the likelihood of the students liking a particular task and knowing the subjects related to that task. Thus, John concludes that it is possible to increase the probability of each task to be accepted by the students by offering the task \(\mathtt {task(paper\_seminar,[tom,mas])}\) to Alice, offering \(\mathtt {task(review\_paper,[kr])}\) to Bob, and offering \(\mathtt {task(write\_paper},\)\( \mathtt {[mas,tom])}\) to Ted.

7 Future Work

Uncertainty does not only arise from noisy communication channels or levels of trust between agents. As future work, we plan to add an environment to our model in order to represent how agents infer ToM from the observation of actions performed by other agents in that environment. The modelling of ToM based on these aspects faces complex issues such as the ones mentioned in [7]: “the slamming of a door communicates the slammer’s anger only when the intended observer of that act realises that the slammer wanted both to slam the door in his face and for the observer to believe that to be his intention”. This means that there is both uncertainty about the slammer’s intentions and uncertainty about the act of slamming the door, which could be caused by an accidental shove or by natural means, hence not represent a communicative act. Therefore, observing such an event occur should not cause the observer to make any inference about the slammer’s mental state. That being said, modelling ToM based on environment observations requires more than only representing both intended and non-intended acts of communication. The slammer might very well not intend to communicate when slamming the door, but that does not stop the observer from reading a message when observing the slamming of the door. These complex issues arise with the inclusion of ToM because agents are able to project beliefs in the minds of other agents they share an environment, or even just a communication channel, with. Therefore, the agents that project beliefs can be subject to what is known as the Mind Projection Fallacy [18]. An agent commits this fallacy using ToM when the agent incorrectly assigns beliefs to another agent’s mindFootnote 7. In our future work, we hope to improve our model in order to be able to represent complex phenomena such as the mind projection fallacy.

8 Conclusions

We have proposed an approach for agents to acquire and update their ToM during communication whilst reflecting on the uncertainty of this process. To the best of our knowledge, our work is the first to explicitly address acquisition and update of uncertain ToM in MAS. In order to show how our approach allows us to model desired properties from communication and common knowledge theories [6], we have proposed a model for uncertain ToM. Using our approach, agents are able to reach accurate ToM and, consequently, accurate shared beliefs by reinforcing their mental attitudes through communication. Furthermore, in this work, we have shown not only how agents acquire and update ToM based on agent communication, but also how agents reason and make decisions using ToM.

The modelling of ToM in MAS under the condition of uncertainty is an important step towards obtaining more realistic and more socially aware artificial agents. We argue that the approach we used to model ToM in MAS is in tune with what we believe to be an upcoming discipline in the field of AI, namely the study of machine behaviour as proposed by [36]. Thus, modelling ToM is relevant to both the AI community and to multi-disciplinary research groups because it offers the possibility to study how agents reach agreements with [24], cooperate with [39], or even behave dishonestly towards [33, 40] other agents using more realistic models of social interactions.