Interpretative structural modeling to social sciences: designing better datasets for mixed method research

Abstract

The multiplication of complex datasets in empirical social sciences calls for methods that can improve the design of complex datasets before the actual gathering of data. Yet mainstream scholars in related fields have rarely explored such methods. In this study, we introduce Interpretive Structural Modeling (ISM) as such a method. As a mixed method, ISM integrates Boolean algebra, matrix theory, and directed graph theory to impose a formal structure to qualitative understanding of a complex system. ISM’s final output is a directed graph that can be visually and easily interpreted. We show that ISM can structure indicators graphically into a multilayered and multi-blocked model, thus uncovering hidden interactions among indicators. By doing so, ISM can reveal hidden and undesired redundancies and incoherencies among indicators within a complex dataset. Most critically, ISM achieves these goals without relying on statistical analysis and hence before the actual gathering of any data. Deploying ISM when designing complex datasets thus facilitates more rigorous conceptualization and understanding of complex social phenomena, steers us away from badly designed complex datasets, and saves precious resource. We use ISM to probe several complex datasets to demonstrate its potentials.

1 Introduction

Empirical social scientists have invested a great amount of human and financial resources in assembling datasets, especially complex ones with multiple components embedded within multiple layers.¹ These datasets seek to capture important and complex social phenomena, such as democracy, quality of governance, state capacity, regime of ethnopolitics, and value system. The quality of these datasets, however, has often been questioned. In fact, scholars have continued to debate the quality of some prominent complex datasets that measures democracy, quality of governance, and state capacity (e.g., Munck 2009; Knutsen 2010; Møller and Skaaning 2010; Thomas 2010; Wilson 2014; Boese 2019; Vaccaro 2021, 2022).

Obviously, problems with conceptualization, measurement, and aggregation can threaten the validity and reliability of these datasets. Among the problems identified in the literature, two frequent criticisms leveled against these complex datasets are conceptual incoherency and redundancy of indicators. A more critical, if never made explicit, problem, however, has been that we can know conceptual redundancy and incoherency within a dataset, only after the dataset has been assembled, released, and analyzed. For example, only after some serious data crunching, was Clarke et al. (1999) able to question the validity and reliability of Inglehart’s (1997) “materialism-postmaterialism” dataset (addressed in detail in Sect. 4). Thus, Clarke et al. (1999, 646) lamented: “biases…may not become apparent for a long time and after a large investment in a flawed measuring instrument has been made.”² (Emphasis added).

In short, social scientists might have wasted much resource in assembling and analyzing not-so-useful data. The root of this problem lies in the flawed designs of these datasets. The recent call for better designed datasets for measuring democracy, governance, and state capacity reflects this belated recognition (e.g., Munck 2009; Møller and Skaaning 2010; Coppedge et al. 2011; Thomas 2010; Seawright and Collier 2014; Boese 2019; Teorell et al. 2019; Vaccaro 2021, 2022; the special issue of Political Research Quarterly 2010). Compared to the large amount of resources devoted to the gathering and analysis of data, however, there have been scant, if any, serious efforts devoted to methods for guiding and aiding the design of dataset. Given the possibility that we may have squandered vast fortunes in gathering raw data and assembling datasets following seriously flawed dataset designs, there is an urgent need for methods that can help improve design of datasets before the actual gathering of data.

This article introduces Interpretive Structural Modeling (hereafter, ISM) to social sciences, for the first time, as a method that can improve designing of datasets before gathering any actual data.³ As a mixed method, ISM integrates Boolean algebra, matrix theory, and directed graph theory to impose a formal structure upon our qualitative reasoning. Moreover, ISM’s final output is a directed graph that can be visually and easily interpreted. To illustrate the potential contributions of ISM for improvement of datasets, we critically examine three datasets, including two from the much-criticized World Value Survey (hereafter, WVS) by Inglehart and his colleagues and one that is better constructed. We show that ISM can structure indicators graphically into a multilayered, multi-blocked, and multi-directional model, thus revealing both overt and hidden interaction among factors. By doing so, ISM graphically reveals hidden and thus undesired redundancy and incoherency, without relying on statistical analysis and hence before the actual gathering of any data.⁴ Most critically, ISM can accomplish this feat even if designers of datasets are unaware of the complex interactions among the indicators within the datasets.

Fundamentally, ISM improves our design of datasets by forcing us to be more explicit and consistent with the logic of dataset design. ISM thus provides us with a unique tool for designing better datasets and analyzing complex systems. Applying ISM to the construction of datasets will not only reduce hidden and undesired redundancy and incoherency within a dataset design but also facilitate better conceptualization and understanding of complex social phenomena, saving us precious resources and troubles afterwards. We therefore urge social scientists to venture beyond their comfort zone (i.e., formal modeling, statistics, simulation) when it comes to methodological toolboxes and give ISM a serious look. Moreover, readers who are unfamiliar with ISM should not be daunted by the matrixes and equations in our text. ISM is actually fairly easy to operationalize. We have also developed a small software program that can operate on any Java-based platform or Python-base platform.⁵

Before we proceed further, several caveats are in order.

First, due to space limitation, our introduction of ISM is necessarily brief. Readers who are interested in the technical details of ISM can read the standard introduction to ISM by Warfield (1974a, 1974b, 1990).⁶ Suffice to say here that ISM is a well-established and versatile method that has been applied to a wide arrange of fields.

Second, ISM cannot replace rigorous conceptualization and theorization that underpin a particular dataset (Munck 2009; Coppedge et al. 2011). Neither can ISM replace quantitative exercises with actual data after the dataset is constructed. What ISM can do is to check the logical coherency and parsimony of a particular scheme of conceptualization and theorization, without relying on real empirical data. Indeed, we strongly urge social scientists to think more rigorously about their design of dataset and then to check the quality of their datasets with actual data, even if the design of their datasets has been facilitated by ISM.

Finally, we have intentionally chosen two datasets that have been extensively used and rigorously scrutinized, the “materialism/post-materialism” (hereafter, M-PM) value within the World Value Survey (WVS) and the regime dataset of Latin America. With the extensive critical literature on the M-PM dataset, we can compare the criticisms advanced by ISM exercises (without actually doing any data processing) and existing criticisms advanced by earlier studies with actual data processing. By doing so, we show that ISM can indeed reveal hidden deficiencies of conceptual incoherency and inconsistency that previously can only be revealed by actual data processing after data gathering. We have intentionally avoided more recent datasets (e.g., the V-Dem dataset, and several datasets that measure state capacity and quality of governance) because they have not been subjected to the same level of critical scrutiny as the M-PM dataset (e.g.,Thomas 2010; Vaccaro 2021, 2022).

The rest of the article unfolds as follows. Section 2 briefly defines the problems to be tackled: undesired redundancy and incoherency within datasets. Section 3 introduces the basic principles and operational procedures of ISM and then highlights what ISM can do for dataset design. Section 4 deploys ISM to probe a sub-dataset from the much criticized WVS to demonstrate ISM’s potentials for constructing better datasets. (Two more ISM exercises with additional datasets are provided in the two appendices as supplementary material.) Sect. 5 discusses the limits and possible extensions of ISM and concludes.

2 Problems in datasets: redundancy and incoherency

Unrecognized and thus undesired redundancy within a dataset is costly because every item within a dataset requires extra time and money for training, coding, data gathering, and data input. Undesired redundancy within a dataset also leads to significant collinearity in quantitative exercises afterwards.

Here, we shall be explicit that we are not against intentionally designed redundancy within a dataset. Due to a practical concern in obtaining data on a particular dimension, especially when conducting surveys, authors of dataset often intentionally ask several seemingly different but similar questions so that respondents will address the dimension even if they answer only one or two of the questions. And when respondents’ answers to these questions corroborate with each other, we have greater confidence in the stability of respondents’ attitude on the particular dimension and thus the quality of the dataset. We thus do not dispute the value of intentionally designed redundancy within datasets.⁷

Sometimes, however, redundancy within a dataset is not by design and thus unintended and unrecognized. Instead, these redundancies are the results of inadequate theoretical or conceptual understandings of a complex system to be measured. For instance, two items within a dataset can overlay with each other significantly via indirect interactions. When this is the case, authors of dataset may not even grasp this hidden redundancy and thus put both items into the dataset. Needless to say, these unintended and undesired redundancies do not help the quality of a dataset.

Unrecognized and undesired incoherencyposes even thornier problems.

First and most critically, incoherency among the indicators within a dataset essentially makes composite indexes constructed from the multiple indicators within the dataset invalid (Coppedge et al. 2011, 250-1). For instance, some items within a dataset may tap into traits that are of such a wide catch that they are simply unsuitable for a dataset that seeks to measure a specific trait. Alternatively, a crucial item may be missing from a category or an item may be incorrectly put under a category it does not belong to (for illustrations, see the dataset on Latin America regime discussed in Appendix B). When this is the case, the validity of the composite index derived from indicators under this category becomes a suspect. Yet, many users of these composite indexes may be entirely unaware this defect (Vaccaro 2022).

Second and more subtly, two items may have an unrecognized positive or negative relationship with each other via indirect interaction. When researchers combine two items with a hidden positive relationship into a composite index, they risk making one item over-represent in the composite index. Conversely, when researchers combine two items with a hidden negative relationship into a composite index, they risk making one item under-represent in the composite index because the two items or traits may cancel each other out.

3 Interpretive structural modeling

In the early 1970s, John N. Warfield (1925–2009) developed “Interpretive structural modeling (ISM)” as a formal modeling tool for understanding complex systems with multiple components, including complex strategic decisions with multiple goals (Warfield 1974a, 1974b, 1990). The key of ISM is to first dissect a complex system into several sub-systems or components by utilizing experts’ practical knowledge of the system and then use mathematical logic and computer tools to structure components and subsystems within the system into a multi-layered (i.e., hierarchical) and multi-blocked/multi-dimensional (i.e., horizontal) structural model.

Today, ISM has become an established and versatile tool for understanding complex systems. ISM has been widely applied to the understanding of complex social and natural systems, ranging from economic geography, urban planning, business (strategic) management, logistics, public policy, energy policy, and resource management (e.g., Wang et al. 2008; Kannan et al. 2009; Kuo et al. 2010; Chandramowli et al. 2011). As far as we know, however, ISM has not been introduced to social sciences. We now proceed to a brief introduction of ISM.

3.1 Principles and procedures of ISM

The key principle of ISM is “Model Exchange Isomorphism (MEI)”. MEI essentially means that a matrix or structured model in one step holds a similar, although not identical, structure as a matrix or structured model in a second step within an ISM exercise.

MEI can proceed along several dimensions: partition by blocs, partition by layers, and partition by strongly connected subsets. Partition by blocs allows researchers to separate subsystems within a system horizontally; partition by layers allows researchers to differentiate layers within a system vertically; and partition by strongly connected subset allows researchers to identify specific pathways through which elements within a system are connected with each other. Together, these procedures can reveal hidden interactions, redundancies, and incoherency among the indicators within a dataset that cannot be easily revealed otherwise.

The core operational tool of ISM is to use Boolegan algebra and matrix theory to delineate a complex system into a multilayered and multi-blocked structural model, via reachability matrix (RM). The standard operational procedures (SOP) of ISM are as follows (see Fig. 1 for a simplified flowchart).

Fig. 1

The standard operation procedures of ISM

First, researchers identify and conceptualize key elements of a complex system. For designing a dataset that measures a complex system (e.g., materialism/post-materialism), this means that researchers must have in-depth knowledge about the complex system and can already identify the key elements within the system.

Second, researchers identify all possible bivariate relationships among elements through practical and logical reasoning and create the initial reachability matrix (IRM, see Fig. 2 below for a concrete example). IRM is produced according to the following rules: 1) Only direct interactions among elements are considered,⁸ and 2) the magnitude or direction of the interaction between one element and another (i.e., weak or strong; positive, negative, or non-linear) is not considered.

Fig. 2

IRM of materialism/postmaterialism indicators

Specific values within the matrix are assigned according to the following rules. Within a matrix, let i denote row elements, and j denote column elements. We assign the following notion (or value) to the relationship between i and j:

“V” at a_ij to denote that element i affects element j, but not vice versa.

“A” at a_ij to denote that element j affects element i, but not vice versa.

“X” at a_ij to denote that element i and element j affect each other. [NOTE: an element always impacts itself, hence, all the “X” on the main diagonal line of an IRM; see Fig. 2.]

“O” at a_ij to denote that element i and element j do not affect each other.

At both stage 1 and stage 2, researchers’ or experts’ knowledge of the system is of critical importance. Unless experts can identify the key elements of a complex system (e.g., democracy) and reason through all the possible bivariate relationships among elements, ISM exercises carried by the computer program will be misinformed and ultimately be of little value. What needs to be pointed out here is that although ISM will eventually structure the elements into layers, blocs, and strongly connected subsets as shown in the final directed graph (to be explained immediately below), experts do not have to use their brainpower to achieve such a goal. In fact, one of ISM’s key strengths is that it can reveal the structure of the elements without experts’ inputs after experts produce the IRM. Moreover, for complex systems, experts’ brainpower may not be enough for such a task.

Third, the IRM is transformed into a Structural Self-interaction Matrix (SSIM), denoted as matrix A (see Fig. 3 below for a concrete example), according to the following rules of assigning values to a_ij: “V” and “X” are assigned the value of 1 whereas “A” and “O” the value of 0. Hence, within SSIM A, 1 denotes the logic that that S_i affects S_j (within one knot) whereas 0 denotes the logic that S_i does not affect S_j (within one knot). Because all relations between two elements are measured as a binary value, the SSIM matrix A is a Boolean matrix (i.e., elements within it can only take the value of zero or one).

Fig. 3

SSIM of materialism/postmaterialism indicators

Fourth, a final reachability matrix (FRM, or M), which captures all possible transitivity among elements, will be derived from the IRM via Boolean matrix multiplications (see Fig. 4 for a concrete example).⁹ The key mathematical step in this process is MEI that is based on Boolean algebra and matrix power multiplications, taking advantage of some unique properties of Boolean matrix (i.e., elements within the matrix can only take the value of zero or one) and the Identity Matrix (i.e., I).¹⁰

Fig. 4

FRM of materialism/postmaterialism indicators

More concretely, via a series of structural self-interaction matrix (SSIM), we arrive at the FRM, $M = (A + I)^{n}$, with n being the number of elements within the matrix (or system). The FRM (i.e., M) now captures all transitivity (or connection) among elements, that is, both the direct and indirect relationships between S_i and S_j, even though some elements are connected with each other only indirectly according to experts’ understanding (contained in the IRM). The logic is straightforward: if S_i can reach S_j, and S_j can reach S_k, then S_i can also reach S_k (that is, if S_i can affect S_j, and S_j can affect S_k, then S_i can also affect S_k). In addition, FRM also captures reflexive reachability (i.e., every individual element affects itself).

When n is large enough, the power operation of $(A + I)^{n}$ will be an impossible task for human brain and a tedious task for even computers. Fortunately, via recursive algorithm, Warshall (1962) and Floyd (1962) had shown mathematically that whenever we reach $(A + I)^{k + 1} = (A + I)^{k}$ [for example, $(A + I)^{4} = (A + I)^{3}$ or $(A + I)^{3} = (A + I)^{2}$], with $k \le n - 1$, we can be certain that we have reached the endpoint of calculating $M = (A + I)^{n}$. Hence, whenever we reach $(A + I)^{k + 1} = (A + I)^{k}$, we can be sure that $(A + I)^{k}$ is the FRM and we do not have to go all the way to $M = (A + I)^{n}$.¹¹

Then, by combing data from SSIM and FRM, we can not only capture all the possible transitivity among elements but also ascertain the nature of transitivity among elements according to the rules dictated in Table 1 below.

Table 1 Rules for differentiating the nature of transitivity between two elements

Fifth, after obtaining the FRM (i.e., M), we then perform the operation of M* = m-m² (where m = M-I) to eliminate redundant connections among the elements within M to arrive at M* as the streamed final reachability matrix (SFRM).

Sixth, results contained within the SFRM are presented in the form of a directed graph to facilitate interpretation (Fig. 5 in the main text; for more illustrations, see Appendixes A & B). The directed graph allows us to visualize the possible complex structure of a social phenomenon and the complex interactions of elements within a complex dataset. By doing so, ISM allows us to detect conceptual and theoretical flaws within the design of a dataset that may not be easily detected by our naked eyes and pure logical reasoning. As far as we can tell, no existing methods can match what ISM can do on this front.

Fig. 5

Directed graph for materialism/postmaterialism values

Finally, after obtaining the final ISM in the form of a directed graph, consistency between the final ISM and the original conceptualization of the system is checked. If there is significant inconsistency between the final ISM and the original conceptualization of the system, it may be necessary to modify the original conceptualization of the system (e.g., to re-delineate the system; or to re-design components within the system) and re-do the whole process, assuming the mathematical processes have been correct all the way. Through this multiple-step process, ISM allows us to check and refine the design of a dataset before the actual gathering of any data.

3.2 How ISM improves dataset design

We believe that when it comes to designing better datasets, ISM offers several key advantages that other conventional mathematical tools such as statistical analysis (e.g., factor analysis) and game theory cannot offer.

First and foremost, ISM can provide us with important glimpses into a complex system before we have actual data or some possible values of the parameters of the system. ISM achieves this feat by transforming our rough ideas of a complex system into a clearly structured model. More concretely, ISM can lay bare the relationships among different components within a system, via a multilayered and multi-bloc structural model. As such, the final directed graph produced by ISM can graphically reveal undesired redundancies and incoherencies within a dataset.

Second, ISM demands rigor to expert opinions in designing datasets: experts have to be very conscious of and explicit about the bivariate relationships among the indicators within the dataset when constructing the Initial Reachability Matrix (IRM). Third, ISM also adds rigor to the logic behind the design of datasets because ISM combines qualitative insights with quantitative tools. ISM codifies researchers’ understanding of a complex system and then derives the final structural model via mathematical operation. When ISM exercises reveal hidden redundancies and incoherencies with a dataset’s design, authors of the dataset will be forced to rethink the logic behind their design. Finally, although ISM combines both qualitative and quantitative methods, ISM is fairly easy to operationalize, because it does not require much advanced mathematics and can now be performed by computers.

So, how can ISM help tackle undesired incoherency and redundancy? To begin with, what ISM seeks to accomplish is related to but fundamentally different from validation, that is, evaluating an existing dataset with statistical tools such as simple correlations across indicators, structural-equation models with latent variables (Bollen 1989), or factor analysis (for a comparative discussion of different approaches of validation, see Seawright and Collier 2014). Most critically, validation with statistical tools is fundamentally about addressing the validity of dataset, whereas ISM is about checking the design of a dataset (i.e., its conceptualization and operationalization).

Moreover, whereas validation with statistical tools must rely on actual data contained within a dataset, ISM does not utilize any data from a dataset at all. Instead, ISM relies on mathematical rigor to check the logic behind the dataset design. In this sense, ISM is a bit more akin to the in-depth expertise-based approach (or case-based according to Seawright and Collier 2014), as exemplified by Bowman et al. (2005) and Mainwarning and Pérez-Liñán (2013) in their measurement of regimes in Latin America, because ISM depends on experts’ opinion.¹² Yet, ISM differs from other expertise-based approaches. ISM operates upon the conceptualization of a dataset before the actual gathering of any data. Moreover, whereas designers of a dataset can only evaluate design with their intuition or logic, ISM provides a rigorous mathematical tool for evaluating their design and can reveal logical incoherency and redundancy that cannot be easily detected. Furthermore, the final directed graph produced by ISM can graphically capture the multiple-layered and multiple-blocked structure among the indicators within a dataset that may not be easily visible to the designers.

ISM explicitly relies on experts’ opinion. Although in quantitative social sciences, expert opinion is often a suspect, we cannot do away with expert opinion, especially when it comes to constructing datasets. Indeed, expert opinion, which often reflects certain theoretical considerations by authors of a dataset (e.g., Bowman et al. 2005), is the very basis of any extant dataset. As such, we see no reason why ISM should not be deployed to examine the expert opinions behind the design of the dataset. In fact, the key purpose of our paper is to convince authors of datasets to apply ISM to their own expert or theoretical opinion before they obtain and code o data so that unrecognized redundancy and incoherency within their conceptualization of a complex system can be revealed. By applying ISM, experts are compelled to think more carefully about the conceptualization and theoretical basis of their design for their datasets. As a result, their design will become less dependent on simple logic or just intuition, thus leading to improvement in the internal coherency of the dataset design.

Specifically, for each of the exercises presented in this paper, we first ask at least two experts to construct their own “initial reachability matrixes” with care and reflections.¹³ We then perform ISM operations with the two different IRMs. Although ISM results obtained from the two different IRMs inevitably diverge, they do overlap with each other significantly to justify the claim that expert opinions coupled with ISM can provide important help for constructing datasets (data not shown). We then ask the experts to critically examine each other’s logic and then produce a single IRM for each dataset with which both of them are comfortable. We then perform ISM operations with the IRM that experts have agreed upon, and results from ISM operations with the agreed IRM (to be presented below) improve upon results from ISM operations with the two earlier IRMs conceived by experts independently. Together with the fact that our ISM results corroborate with existing critical discussions of these datasets, we have reasons to contend that results presented here point to some important improvements for the datasets examined in this paper.

In the next section, we employ ISM to probe the materialism/post-materialism (hereafter, M-PM) dataset from the much criticized WVS by Inglehart and his colleagues to illustrate the procedures and power of ISM. We show that if Inglehart and his colleagues had more rigorously conceptualized about what they wanted to measure and then use ISM to check their designs, they could have saved much precious resource from being squandered. In Appendix A, we also deploy ISM to examine the “achievement motivation”, another sub-dataset from WVS to show that this sub-dataset has more deficiencies. In Appendix B, we use ISM to probe a well-constructed dataset on regime types in Latin America as a hard target to further demonstrate the utilities of ISM: even with a well-constructed dataset, ISM can provide tangible improvements.

4 “Materialism/post-materialism”: an ISM exercise

The World Value Survey (WVS) is an ambitious project led by the late Ronald Inglehart (Inglehart 1988, 1990, 1997; Inglehart et al 2000). Many have questioned the utility of the WVS dataset (e.g., Silver and Dowley 2000; Seligson 2002; Hadenius and Teorell 2005), but few have questioned the conceptual and theoretical design of dataset itself, although it has often been implied (e.g., Jackman and Miller 1996b; Davis 2000; Davis and Davenport 1999; Davis et al 1999; Alemán and Woods 2016).¹⁴

We apply ISM to two (sub-)datasets in WVS, “M-PM” and “achievement motivation”.¹⁵ Our exercises show that there are significant redundancy and inappropriate clustering of elements (or factors) within the two sub-datasets in WVS. We also identify elements that cannot be easily treated and measured as a single element. Furthermore, we reveal serious logical flaws in constructing a composite index for “M-PM” or “achievement motivation” from their individual elements. We achieve these goals, all without utilizing any data collected within the dataset.

Our ISM analysis thus adds more firepower to existing criticism of the two datasets. We show that these two datasets suffer from serious deficiencies without easy remedies and the claim by Inglehart and his colleagues that these two datasets capture two coherent cultural values cannot be easily substantiated (e.g., Inglelart 1988, 1203, 1215).

A caveat is in order. Our exercise does not address the possibility that datasets for both “M-PM” and “achievement motivation” have been wanting is because that there is no such thing called “M-PM” or “achievement motivation”. In other words, Inglehart and his colleagues might have been trying to measure two phantoms. Our discussion neither supports nor disproves such a possibility. What we do highlight is that even if the “M-PM” and “achievement motivation” are real, the measurements designed by Inglehart and his colleagues do not make much sense.

Much criticism has been leveled against the “M-PM” dataset. These criticisms question the validity of the dataset on two related fronts. Most critics focus on whether there is anything called M-PM value. They contend that peoples’ responses to the questions in the survey are not necessarily consistent and coherent and thus do not reflect a stable value system because a) subjects’ replies are subject to the impact of current economic conditions and concerns (e.g., Bean and Papadakis 1994; Clarke et al 1997; Clark et al. 1999); and/or b) that there is no micro foundation for aggregated data reported at the macro level (e.g., Davis et al 1999; Davis and Davenport 1999; Davis 2000). A few others question whether the M-PM value impacts economic growth at all as Inglehart and his colleagues have claimed, even if there is such a thing called M-PM value (e.g., Jackman and Miller 1996a; see also Clarke et al 1997). Here, we focus on the first kind of criticism because it is more fundamental.

A caveat is in order. Although we criticize the design of the M-PM index, we do not necessarily agree with all of the criticisms against it. We concur with Inglehart and his colleagues’ hypothesis that economic development and generational replacement in industrialized societies can cause value shifts, and we agree with some, but not all, of their defense (e.g., Inglehart 1994; Abramson et al. 1997; Inglehart and Abramson 1999). Our aim here, again, is to show that ISM can help us design better datasets by addressing a widely questioned dataset. We show that even with all these existing criticisms, ISM can still reveal underappreciated drawbacks within the design of the M-PM dataset, again without having to rely on actual data.

Two key points regarding the M-PM index must also be noted. First, Inglehart and his colleagues initially used only the first four indicators (S1 to S4). After they discovered that subjects do not always reveal their preferences consistently with the four indicators, however, they added two additional sets of four questions (S5 to S8; and S9 to S12) and thus expanded the list to twelve indicators (Table 2 below). Inglehart and his colleagues thus assume that the three batteries of questions measure the same M-PM value: the three batteries of questions are equivalent. More specifically, S1 plus S3 is equivalent to S5 plus S6, and then S9 plus S10: these indicators tap into materialism. S2 plus S4 is equivalent to S7 plus S8, and then S11 plus S12: these indicators tap into post-materialism. We probe this supposedly improved twelve-item dataset but also discuss the initial four-item dataset, although Inglehart and his colleagues usually drop item S8 from the final composite index without any justification.

Table 2 Materialism/postmaterialism values in political culture

Second, indicators within the M-PM value questionnaire are designed to measure what people think ‘what should be done (within their own country or community)’ regarding potential issues. The original logic behind the M-PM index further assumes that if subjects hold a coherent M-PM value, then when being confronted by the four items in a battery, subjects who first choose a materialism value will be more likely to choose a materialism value as their second choice. Hence, when trying to determine the bivariate relationship between two indicators for constructing IRM, what we ask is whether a “yes” or “no” answer to one question (regarding an issue) entails a “yes” or “no” answer to another question (regarding an issue). For example, when trying to determine the bivariate relationship between “fighting against crime” (S10) and “toward a friendlier and less impersonal society” (S11), what we ask is not whether an individual believes that “fighting against crime” will indeed help a society moves “toward a friendlier and less impersonal society” but rather whether an individual who supports “fight against crime” will be more or less likely to support “move a society toward a friendlier and less impersonal society”. In this way, when constructing the IRM, our experts are actually recreating the mental processes through which subjects respond to questions for the M-PM value if subjects do hold a coherent M-PM value.

Based on the twelve indicators within the M-PM dataset (Table 2), we construct an IRM regarding the twelve indicators (Fig. 2 below). From IRM, we build the SSIM (Fig. 3 below). Again, with Boolean algebra and power operations, we transform the SSIM into the FRM M = (A + I)⁵, because A + I ≠ (A + I)² ≠ (A + I)³ ≠ (A + I)⁴ ≠ (A + I)⁵ = (A + I)⁶ = … = …(A + I)¹². FRM M = (A + I)⁵ now captures all the direct and indirect relationships among the indicators (Fig. 4 below). After obtaining SFRM M* = m-m² (not shown), in which m = M-I and eliminating reflexive reachability, we present M* as a directed graph in Fig. 5.

Figure 5 recaptures some key logic behind the design of the M-PM dataset. First, ISM results show that the three batteries of four-item questionnaire within the dataset do have significant redundancies, by design. We can reasonably argue that S5 (maintaining a high rate of economic growth) and S9 (maintaining a stable economy) are essentially equivalent: both factors tap into individuals’ concern for maintaining economic stability within a state. Similarly, S1 captures individuals’ concern for maintaining political stability within a state, although it is a bit tenuous to argue that S1 (which is political) taps into the same thing captured by S5 and S9 (which is economic), notwithstanding the ISM results. S2 (give people more say in the government) and S7 (give people more say in how things are decided at work and in their community) too are equivalent: They apparently capture people’s concern for their voices on issues at national level (S2) and local level (S7), respectively. Finally, S10 (fight against crime) entails S3 (fight rising prices), a property that also somewhat reflects the logic behind the design of the dataset.

Third, there is some coherency among the indicators. Via S12 (move toward a society where ideas count more than money), S2/S7 does entail S4 (protect freedom of speech). These four indicators thus do seem to form a bloc that captures non-materialism values.

Figure 5, produced by our ISM exercise however, also reveals several areas of incoherency that are inconsistent with the supposedly coherent logic behind the design of the dataset.

First, the twelve indicators form into three rather than two blocs (i.e., materialism and postmaterialism) as the authors of the index have maintained. Most evidently, S6 (make sure that this country has strong defense forces) is by itself all alone. Presumably, S6 taps into individuals’ concern for their country’s and their own physical security from external threats, and it does not easily connect with either materialism or post-materialism.

Second and related to the first, the coherency of the second battery of four-item index (S5, S6, S7, and S8) is deeply suspicious. Within this battery, if we eliminate the interactions between these four items and the other eight items within the dataset as shown in Figure AB6, then clearly S5, S6, S7, and S8 do not have much logic connection with each other: S6 is all alone; S7 cannot be connected with S5 without S12; and S8 has not connection with S6 and S7 without its connection with S10.

Third, S10 (fight against crime) is a major node within the structural model: it connects with many other indicators within the dataset. This is easy to grasp: “fight against crime” reflects a basic need—individuals’ concern for their physical security, and unless this need can be somewhat satisfied, it is hard to see how one can pursue other values, material or non-material. More concretely, many things that people care about depend on S10, from S8 (try to make our cities and countryside more beautiful), to S11 (move toward a friendlier, less impersonal society), to S2/S7 via S12 and then S1/S5/S9. Because S10 connects with so many things, it is difficult to believe that it reflects only material concerns. Having S10 within the M-PM index thus makes the index’s coherency even more suspicious.

Fourth, and perhaps more seriously, the original logic behind the design of the M-PM index entails some unintended consequences for subjects’ responses if subjects do hold a coherent M-PM value. The original logic behind the M-PM index assumes that if subjects hold a coherent M-PM value, then when being confronted by a four-item of a battery, subjects who first choose a (post-)materialism value will be more likely to choose a (post-)materialism value as their second choice. In other words, the specific traits within a particular value system form a coherent whole (via interaction), and subjects, when prompted by the questionnaires, will maintain consistency within their value systems.

Figure 5 does indicate that the twelve elements within the index interact with each other. Unfortunately, the potential interactions among the elements maybe far more complex than Inglehart and his colleagues have foreseen. For instance, within the third battery of questions (i.e., S9, S10, S11, and S12), if subjects do reason, then if they choose S11 (move toward a friendlier, less impersonal society), they are also likely to choose S10 (fight against crime) because S10 is more-or-less indispensable for “a friendlier society”. When this is the case, it should not be a surprise that subjects may be torn apart when picking the second item and many of them may just respond randomly, after they have picked the first item.

Overall, our ISM exercise strengthens some previous criticism against the M-PM dataset. Moreover, our ISM exercise reveals structures (i.e., blocs, layers), interactions, and questionable designs with the M-PM dataset that cannot be revealed by simply intuition, logical deduction, and ex post statistical analysis. For instance, our ISM exercise reveals that S6 (make sure that this country has strong defense forces) does not belong to the dataset. Likewise, our ISM exercise reveals that S10 (fight against crime) is unsuitable for being an indicator for M-PM value either, because it connects with so many things that may reflect either materialism or post-materialism value.

Most critically, although previous critics of the M-PM dataset did question the coherence and hence the validity of the M-PM index, all of them had to rely on statistical analyses with published M-PM data plus some other datasets (e.g., Clarke et al 1997, 1999; Davis and Davenport 1999; Davis et al 1999; Davis 2000). By comparison, we reach roughly the same conclusion without having to rely on any actual data. Moreover, none of the previous analyses reveal the potential vertical interaction among the M-PM indictors.

Moreover, few existing criticisms have emphasized the WVS dataset might have suffered from serious design flaws. In contrast, our ISM exercises have revealed serious conceptualization and design flaws within the WVS dataset without utilizing any data by WVS. When this is the case, ISM should be a very useful tool for social scientists when trying to conceptualize and design datasets, especially for those on more complex social phenomena that require more rigorous theorization and conceptualization and are more costly to produce. After all, a flawed conceptualization and design for large dataset thus means that we have wasted a large amount of financial and human resources because much of the data is redundant, incoherent, and thus useless, if not misleading, for future analysis.

5 Discussion and conclusion

Social scientists often deal with complex social phenomena as complex systems with multiple factors, blocs, layers, and interactions. Yet, most methods that social scientists have employed are based on a limited acknowledgement of systemic complexities (for a similar critique, see Ragin 2000). In this article, we introduce ISM that has been explicitly developed to deal with complex system to the broader social sciences. We first introduce the basic principles of ISM. We then employ ISM to probe three datasets and show that ISM can construct a graphical structure of factors and thus allow us to detect flaws within the design of a dataset without employing any data from the dataset.

The benefits of deploying ISM in constructing complex datasets are multifold.

First and most critically, ISM can contribute to better theorization and conceptualization in designing datasets because it demands a greater role of theorization and conceptualization in dataset construction than do other approaches. At the very first step, authors of datasets have to deploy certain theoretical reasoning to establish the logic of the complex relationship between the various elements within a complex concept. In particular, the judgment about whether and how two elements affect each other is best backed by rigorous theoretical and empirical justification. As a result, empirical data aided by ISM become more theoretically informed. By requiring more careful theoretical deliberation in indices construction and applying objective mathematic derivation procedures, ISM helps us produce better conceptualized indices of complex social phenomena within datasets.

In contrast, many existing studies that question the quality of complex datasets reply on approaches such as correlation, factor analysis, and regression with data within those datasets. Although such studies have their value, they are essentially atheoretical when it comes to the original theorization and design of the dataset, in the sense that they do not contain much a priori theoretical consideration. Moreover, these a posteriori analyses of data cannot do anything about the theoretical and conceptual flaws of a dataset (e.g., missing essential elements, redundancy, and incoherency) because all the data have been compiled already.

Second, with directed graph, ISM produces a better visualization of the complex structure of a complex social phenomenon and the complex interaction of the various elements within a complex dataset that may not be easily obtained by our naked eyes. The directed graph not only identifies blocs of items within a concept, but also shows layers of items and maps out the relationship between items of different blocs and layers. This complex structure is logically and mathematically derived from the initial IRM. Better yet, the directed graph produced by ISM can be easily interpreted with qualitative logic. In contrast, even the best existing conceptualized datasets show only the blocs of items, but not other possible relationships between the items. Also, blocs are solely determined subjectively by researchers, without objective criteria or procedures.

Third and immediately following from the first and second, ISM can help authors of datasets better dissect a complex system and understand the logical relationships among the elements within the system, in an easily interpreted directed graph. It can thus help authors of (expensive) datasets avoid some critical pitfalls in the design of datasets that cannot be easily detected by naked eyes, such as unrecognized redundancy and incoherency. Unrecognized and hence undesired redundancy and incoherency in datasets not only wastes time, financial resources, and human effort, but also tend to produce biased results in later empirical analysis, especially when analysts are not aware of the logic and processes of data production.

Finally, by demanding an explicitly stated process and logic of creating the indices of datasets, ISM makes data production more transparent and thus facilitates more productive discussion on the quality of dataset. The ongoing movement of “data access and replication transparency” (DART) in political science and other social sciences calls for free access to all raw data of published articles in major academic journals. We believe that we should go one step further: we should demand authors of dataset to make public their original conceptualization of the dataset because this is equally important, if not more fundamental than, the detailed coding and recoding of variables. In other words, the original conceptualization of datasets used by researchers should also be subjected to open scrutiny. On this front, although many authors of dataset have now laid out their conceptualization of dataset and variables within the dataset (e.g., Mainwaring, Brinks, and Pérez-Liñán 2008), such a practice is not a mandatory requirement yet. ISM helps in this regard in that it demands authors of datasets to disclose (and follow) explicitly stated logic and principles in conceptualizing, designing, and constructing their datasets, thus fostering better accumulation of knowledge through future replication of research and data.

There are several obvious extensions of the work reported here. We shall mention just two. First of all, this paper addresses only static interactions among the elements. ISM can also deal with social phenomena in which different factors are present and interact with each other differently in different stages of a process. As such, ISM can provide us with a more dynamic picture of complex systems.

Second, ISM can certainly be combined with analysis with real data from datasets. Indeed, if ISM exercises and analyses with real data produce divergent pictures about a system, we should be alerted to some possible hidden factors and pathways among factors within the system, thus refining our understanding of the system. For instance, our ISM analysis provides a possible explanation for Davis and his colleagues’ findings that subjects do not necessarily reply to the “M-PM” questionnaires consistently as suggested by Inglehart and his colleagues. It is highly plausible that the interactions among indicators within the M-PM dataset are far more complex than what Inglehart and his colleagues have projected (Davis and Davenport 1999; Davis et al. 1999).

Like any other methodology, ISM has its limitations, as we have acknowledged at the very beginning. Most importantly, the initial key step in ISM is to build the IRM, and this requires researchers to come to an agreed IRM about the linkages among factors. Evidently this first step requires researchers to have in-depth knowledge about the system and use logic to construct the relationships among the factors. For many complex systems, however, researchers may not command the necessary expertise to get all the logic connections correct. Moreover, different researchers may well disagree on the relationships among elements, partly because there are no fixed rules for determining whether two elements are interconnected, directly and indirectly. When this is the case, it may be difficult to obtain an IRM that is agreed by all the researchers. Repeated discussions by researchers can ameliorate the problem, but not eliminate it. Hence, a key limitation of ISM is its reliance on researchers’ in-depth understanding of a system, usually in the form of qualitative knowledge: No amount of quantitative data and modeling can replace it.

Looking from a different angle, however, this demand of in-depth understanding of a complex system is also one of ISM’s strengths. Precisely because authors of datasets have to grapple with the possible interactions among the indicators, they need to be more explicit and coherent with the logic of their dataset design. Indeed, in light of the results regarding the three datasets we have probed here with ISM, it is quite likely that the dataset on regimes in Latin America is of higher quality than the two datasets within WVS precisely because the former has been designed by in-depth expertise of Latin America. We therefore strongly echo the more recent call for in-depth expertise when constructing complex datasets (e.g., Bowman et al. 2005; Coppedge et al. 2011; Thomas 2010; Teorell et al. 2019). Combining rigorous conceptualization of the complex system to be measured, in-depth country or area expertise, and ISM exercise may lead to better design of datasets.

In the end, we see ISM a methodological tool with considerable potentials in social sciences, although ISM, like any other tool, has its limitations. We hope we have taken a first step in convincing social scientists to give ISM a serious look.