Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This book has provided an overview of several prominent likelihood-free algorithms that are readily available for use when fitting psychological models to data. We began in Chap. 1 by introducing the concepts of approximate least squares, Bayesian inference, and approximate Bayesian computation (ABC). Next, in Chap. 2, we presented several likelihood-free algorithms, along with code for implementing them. Of course there are now many more algorithms available that we could have discussed, but these algorithms are often more complicated and have not yet been used in cognitive science. As such, we limited our focus to algorithms that would be discussed in later chapters. In Chap. 3, we provided a tutorial on fitting the Minerva 2 model to simulated data, and we compared the relative merits of the probability density approximation (PDA; [38]) method, a kernel-based ABC algorithm, and asymptotic expressions for an approximate likelihood function. We then showed how one could extend the kernel-based algorithm hierarchically, and applied it to fit the Minerva 2 model to the data from [4].

Chapters 4 and 5 focused on a small set of recovery and model-fit exercises we have completed in our own research. In Chap. 4, we presented “validations” where each cognitive model had an explicit likelihood function, and so the true posterior could be estimated. We used the true posterior as a metric for evaluating the accuracy of posteriors obtained using various likelihood-free algorithms. These exercises were useful in the applied setting because we have used these examples to provide assurance that we had made appropriate choices when moving to similar models with intractable likelihood functions. Whereas Chap. 4 focused on validating the likelihood-free approach, Chap. 5 provided some interesting examples of fitting models to data whose likelihood functions are currently intractable. Here, we provided summaries of applications in our own work; specifically, we fit the retrieving actively from memory model [44, 83, 92], the dynamic signal detection model [70, 113], the feed-forward inhibition model [114, 115], and the leaky competing accumulator model [100, 114]. While we cannot say whether or not the estimated posteriors in these applications were accurate, we believe them to be based on the simulation results from the “validation” studies.

Our hope is that the techniques discussed in this book may serve as the catalyst in the advancement of mechanistic models of cognition. To us, the primary advantage of likelihood-free techniques is the infinite number of possibilities for new model mechanisms, distributional assumptions, or processing stages. The techniques described here allow freedom from the burden of simplifying assumptions in an effort to acquire mathematical tractability. While it is not clear how often simplifying assumptions are made for the purposes of mathematical tractability, the advantages and disadvantages of the commitment to tractability motivates a stimulating discussion on the role that tractability should play in model development. In developing mathematical models, our goal is to put forth a model that can not only fit data well, but also makes a strong yet accurate commitment to the distribution of data we should see in our experiments [128, 129]. The assessment of a model’s full credentials involves two important considerations: model fit and model complexity [130–132].

In the domain of model development, the word complexity can sometimes refer to either the flexibility of a model and can sometimes refer to the ease of implementation [133]. However, these are two different concepts. The ease of implementation is related to the mathematical tractability, but it is not related to complexity [96, 110]. Within this book, we have described several mathematical models that are easy to implement and fit to data because there are analytic expressions relating the model parameters to the data (e.g., signal detection theory). These expressions make the model very easy to fit to data via maximum likelihood or Bayesian approaches. Unfortunately, tractability does not necessarily map onto fewer parameters, or the degree of model flexibility. As such, tractability is also unrelated to complexity when used as a measure of model performance.

To illustrate, consider as analogy the bind cue decide model of episodic memory (BCDMEM; [85]). The BCDMEM model was proposed as a pure context model of episodic memory, an assumption that was at odds with the dominant models at the time. The model was presented as a simulation model, meaning that the likelihood function relating model parameters to predictions about the hit and false alarm rates was intractable. For years, anytime a researcher wanted to fit BCDMEM to their data, they were forced to rely on simulation methods, such as approximate least squares. Eventually, Myung et al. [96] produced analytic expressions for the model. While these expressions are computationally difficult to evaluate, they can be used to assess the model’s flexibility, complexity, and identifiability [92]. Ultimately, the expressions derived by Myung et al. [96] unlocked one key facilitator in the endeavor of rigorous model evaluation: mathematical tractability.

What can we make of the research conducted in the time between the development of the original model in 2001, and the derivation of analytic expressions in 2007? As the assumptions of BCDMEM were never changed during this time period, the complexity of BCDMEM also never changed. Hence, the ability of BCDMEM to fit data also never changed. In a similar vein, if a researcher published a paper deriving analytic expressions for say, the complex leaky competing accumulator (LCA; [100]) model tomorrow, nothing about the previous fits of the LCA model over the past decade will have changed. Nothing about the model’s complexity will have changed either. Instead, the LCA model would simply be given a compelling pragmatic advantage in choosing among the various models for application purposes because the model would now be (potentially) easier to fit to data (but note the simulation performance differences for the BCDMEM in Chap. 4).

While mathematical tractability is highly advantageous, the analyses in this book highlight the importance of methods for performing inference on simulation-based models. In theory, any computational model can now be fit to data using the likelihood-free approach, allowing researchers to regain access to tried-and-true methods for model evaluation. Our view is that, by using these methods, researchers are free to experiment with as many complex and stochastic model variants as they can imagine, while still assessing model flexibility relative to the data. Of course, tractable models offer compelling advantages, but if compromising assumptions are required to produce tractability, these assumptions may now be rejected on the basis of a theoretical position, prior research, or even curiosity.