Abstract
We overview our recent work on the statistical analysis of simulation models and, especially, economic agent-based models (ABMs). We present a redesign of MultiVeStA, a fully automated and model-agnostic toolkit that can be integrated with existing simulators to inspect simulations and perform counterfactual analysis. Our approach: (i) is easy-to-use by the modeler, (ii) improves reproducibility of results, (iii) optimizes running time given the modeler’s machine, (iv) automatically chooses the number of required simulations and simulation steps to reach user-specified statistical confidence, and (v) automatically performs a variety of statistical tests. In particular, our framework is designed to distinguish the transient dynamics of the model from its steady-state behavior (if any), estimate properties of the model in both “phases”, and provide indications on the ergodic (or non-ergodic) nature of the simulated processes – which, in turns allows one to gauge the reliability of a steady-state analysis. Estimates are equipped with statistical guarantees, allowing for robust comparisons across computational experiments. This allows us to obtain new insights from models from the literature, and to fix some erroneous conclusions on them.
Access provided by Autonomous University of Puebla. Download conference paper PDF
Similar content being viewed by others
Keywords
- Agent-based models
- Statistical model checking
- Ergodicity analysis
- Transient analysis
- Warmup estimation
- T-test and power
1 Extended Abstract
We propose a novel approach to the statistical analysis of economic agent-based models (ABMs). The analysis of ABMs is often constrained by problems of (i) computational time, (ii) correct construction of confidence bands, (iii) detection of model ergodicity, and (iv) identification of transient behaviour. All these issues are pivotal to the validity of a model, both when it is used for thought-experiments, and when it aims at delivering policy insights. Nevertheless, they are often overlooked [17] or solved informally without a commonly shared procedure [10].
We propose fast, easy-to-use, automated, and statistically rigorous procedures to address all these problems. We implement such procedures in MultiVeStA, a model-agnostic statistical analyser which can be easily tool-chained with existing ABMs. Independently from the nature of the ABM at hand, the analyser performs simulations, distributing them in the cores of a machine or a network, computes statistical estimators, and implements the minimum number of simulations necessary to satisfy given conditions on confidence intervals.
The above-mentioned problems are not specific to the ABM context; they affect most simulation-based analysis approaches and were therefore tackled by many scientific communities in the past. In computer science, several automated procedures have been proposed to mitigate these problems. An example is the family of techniques known as statistical model checking (SMC) [1, 18]. Roughly speaking, SMC can be seen as an automated Monte Carlo analysis guided by a property of interest given in an external property specification language. Here we focus on the statistical model checker MultiVeStA [11, 16, 20]. While previous versions of MultiVeStA have been successfully applied in a wide range of domains including, e.g., threat analysis models [4], highly-configurable systems [2, 3, 19], public transportation systems [9, 11, 12], robotic scenarios with planning capabilities [5, 6], and crowd steering scenarios [15], it has never been employed for the analysis of ABMs. Here, we have redesigned and extended MultiVeStA to target analyses of interest for the ABM community (e.g. [13]). For example, we integrated a series of tests that allow for (i) counterfactual analysis, (ii) detection of ergodicity, and (iii) estimation of the transient period.
We demonstrate our approach in [20] using two ABMs from the literature. The first is a macro stock-flow consistent ABM from [8]. We first replicate the results from the original contribution, scaling the runtime analysis from 15 days to 15 h thanks to the automated parallelization of simulations. We also show how the statistical reliability of our approach allows us to perform meaningful counterfactual analysis.
The second ABM is a simple financial market model from [14]. This model has analytical solutions [7] which we use to assess the effectiveness of our approach. Contrarily to computational analyses reported in prior literature [14], which were biased by erroneous under-estimations of the transient period duration and of the process autocorrelation, we match the correct analytical results of the model.
In the near future, we plan to extend the number of tasks performed in an automated and user-friendly manner by our tool, e.g., including the identification of multiple stationary points. We also plan to use our tool for the analysis of other classical and novel ABMs.
The tool, models, and more information are available at: github.com/andrea-vandin/MultiVeStA/wiki.
References
Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comp. Simul. 28(1), 6:1–6:39 (2018)
ter Beek, M.H., Legay, A., Lafuente, A.L., Vandin, A.: A framework for quantitative modeling and analysis of highly (re) configurable systems. IEEE Trans. Softw. Eng. 46(3), 321–345 (2020). http://orcid.org/10.1109/TSE.2018.2853726
ter Beek, M.H., Legay, A., Lluch-Lafuente, A., Vandin, A.: Quantitative analysis of probabilistic models of software product lines with statistical model checking. In: Proceedings 6th Workshop on Formal Methods and Analysis in SPL Engineering, FMSPLE@ETAPS 2015, London, UK, 11 April 2015, pp. 56–70 (2015). http://orcid.org/10.4204/EPTCS.182.5
ter Beek, M.H., Legay, A., Lluch-Lafuente, A., Vandin, A.: Quantitative security risk modeling and analysis with RisQFLAN. Comput. Secur. 109, 102381 (2021). http://orcid.org/10.1016/j.cose.2021.102381
Belzner, L., De Nicola, R., Vandin, A., Wirsing, M.: Reasoning (on) service component ensembles in rewriting logic. In: Iida, S., Meseguer, J., Ogata, K. (eds.) Specification, Algebra, and Software. LNCS, vol. 8373, pp. 188–211. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54624-2_10
Belzner, L., Hennicker, R., Wirsing, M.: OnPlan: a framework for simulation-based online planning. In: Braga, C., Ölveczky, P.C. (eds.) FACS 2015. LNCS, vol. 9539, pp. 1–30. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-28934-2_1
Bottazzi, G., Giachini, D.: Far from the madding crowd: collective wisdom in prediction markets. Quant. Financ. 19(9), 1461–1471 (2019)
Caiani, A., Godin, A., Caverzasi, E., Gallegati, M., Kinsella, S., Stiglitz, J.E.: Agent based-stock flow consistent macroeconomics: towards a benchmark model. J. Econ. Dyn. Control 69, 375–408 (2016)
Ciancia, V., Latella, D., Massink, M., Paškauskas, R., Vandin, A.: A tool-chain for statistical spatio-temporal model checking of bike sharing systems. In: Margaria, T., Steffen, B. (eds.) ISoLA 2016. LNCS, vol. 9952, pp. 657–673. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47166-2_46
Fagiolo, G., Guerini, M., Lamperti, F., Moneta, A., Roventini, A.: Validation of agent-based models in economics and finance. In: Beisbart, C., Saam, N.J. (eds.) Computer Simulation Validation. SFMA, pp. 763–787. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-70766-2_31
Gilmore, S., Reijsbergen, D., Vandin, A.: Transient and steady-state statistical analysis for discrete event simulators. In: Polikarpova, N., Schneider, S. (eds.) IFM 2017. LNCS, vol. 10510, pp. 145–160. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66845-1_10
Gilmore, S., Tribastone, M., Vandin, A.: An analysis pathway for the quantitative evaluation of public transport systems. In: Albert, E., Sekerinski, E. (eds.) IFM 2014. LNCS, vol. 8739, pp. 71–86. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10181-1_5
Grazzini, J.: Analysis of the emergent properties: stationarity and ergodicity. J. Artif. Soc. Soc. Simul. 15(2), 7 (2012)
Kets, W., Pennock, D.M., Sethi, R., Shah, N.: Betting strategies, market selection, and the wisdom of crowds. In: Twenty-Eighth AAAI Conference on Artificial Intelligence (2014)
Pianini, D., Sebastio, S., Vandin, A.: Distributed statistical analysis of complex systems modeled through a chemical metaphor. In: International Conference on High Performance Computing and Simulation, HPCS 2014, Bologna, Italy, 21–25 July 2014, pp. 416–423 (2014). http://orcid.org/10.1109/HPCSim.2014.6903715
Sebastio, S., Vandin, A.: MultiVeStA: statistical model checking for discrete event simulators. In: 7th International Conference on Performance Evaluation Methodologies and Tools, ValueTools 2013, Torino, Italy, December 10–12(2013), pp. 310–315 (2013). http://orcid.org/10.4108/icst.valuetools.2013.254377
Secchi, D., Seri, R.: Controlling for false negatives in agent-based models: a review of power analysis in organizational research. Comput. Math. Organ. Theory. 23(1), 94–121 (2017). http://orcid.org/10.1007/s10588-016-9218-0
Sen, K., Viswanathan, M., Agha, G.: Statistical model checking of black-box probabilistic systems. In: Alur, R., Peled, D.A. (eds.) CAV 2004. LNCS, vol. 3114, pp. 202–215. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27813-9_16
Vandin, A., ter Beek, M.H., Legay, A., Lluch Lafuente, A.: QFLan: a tool for the quantitative analysis of highly reconfigurable systems. In: Havelund, K., Peleska, J., Roscoe, B., de Vink, E. (eds.) FM 2018. LNCS, vol. 10951, pp. 329–337. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95582-7_19
Vandin, A., Giachini, D., Lamperti, F., Chiaromonte, F.: Automated and distributed statistical analysis of economic agent-based models. CoRR abs/2102.05405 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Vandin, A., Giachini, D., Lamperti, F., Chiaromonte, F. (2022). MultiVeStA: Statistical Analysis of Economic Agent-Based Models by Statistical Model Checking. In: Bowles, J., Broccia, G., Pellungrini, R. (eds) From Data to Models and Back. DataMod 2021. Lecture Notes in Computer Science, vol 13268. Springer, Cham. https://doi.org/10.1007/978-3-031-16011-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-16011-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16010-3
Online ISBN: 978-3-031-16011-0
eBook Packages: Computer ScienceComputer Science (R0)