1 Introduction

Scholars devote much attention to understanding how to optimize the interaction of human and technical systems—sociotechnical systems (Trist and Bamforth 1951)—for group and organizational performance (e.g., Ackermann 1996) and to promote innovation and adaptation using technology (e.g., Vaidya and Seetharaman 2011). What seems to receive less attention are the underlying cognitive micro-processes involved in recognizing, understanding, and acting on the integration of technology and social systems (e.g., Ackermann et al. 2018; Kolfschoten and Reinig 2013; Ratzmann et al. 2018). In sociotechnical systems research, technology is construed broadly to include everything from simple tools to complex systems such as evolving artificial intelligence. Social systems are defined as the patterned series of interrelationships existing between individuals, groups, and institutions and forming a coherent whole (Merriam-Webster 2018).

We have vivid examples of people leveraging deep understanding of human and technical integration. Police officers adjust their practice as they carry smartphones to access information and collect evidence in real time (Verhulst and Rutkowski 2018). Incident response teams in complex disaster settings use context-aware multi-party coordination systems with goals of bringing together team members and agencies in dynamic life-threatening environments (Way and Yuan 2014). We have equally vivid examples that more unidimensional perspectives result in less successful outcomes: The introduction of Google Glass seemed to focus on engineering and user outcomes while paying less attention to how Glass, and the wearer, would be perceived in the broader social setting. The term “glasshole,” signals how Glass and its users were initially received (Honan 2013).

Scholars acknowledge the importance of understanding the cognitive micro-processes underlying group outcomes. A recent Group Decision and Negotiation (GDN) special issue notes, “this focus on micro-processes recognises the need to develop an in-depth understanding of what occurs in the context of group decision-making processes, in particular seeking to understand the relationship between the social, behavioural, and the material” (Ackermann et al. 2018, p. 709). A 2013 GDN special issue focused on advancing organizational and team efficiency and productivity, “…but doing so by focusing first and foremost on the individual cognitive activities involved in collaboration activity” (Kolfschoten and Reinig 2013, p. 868). Here we take a focus on the individual cognitive micro-processes, noting that teams are made up of individuals who make sense of their tools, colleagues, and tasks as they come into their group work (e.g., Griffith 2012).

Recently, Schmitz et al. (2016) provide an important step forward in understanding how individuals come to adapt technologies and tasks. Building from DeSanctis and Poole’s (1994) work on group appropriation of technology in adaptive structuration theory, Schmitz et al. examine technical, task, and individual characteristics as inputs to structuration episodes where adaptations to the technology and/or workplace task occur. Others have considered more social practices like negotiated change (Griffith et al. 2002) to trigger such episodes. Sarker and Valacich (2010) tease apart technology adoption in group settings and speak specifically to both individual and group effects.

We build from Brooks (1987) in arguing that “[t]here is no single development, in either technology or in management technique, that by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity” (p. 10). Sociotechnical design must consider ensembles of technologies and team and organizational arrangements to optimize effectiveness (Trist and Bamforth 1951; Winter et al. 2014). Leonardi (2011, p. 151) describes a metaphorical microscope able to view the foundations of this process. We propose that people have different acuity of vision for what they view in that microscope.

Leveraging Sternberg et al.’s work on practical (2000) intelligence, we introduce the construct of systems savvy: an individual’s capacity to see the interdependence of technological and social systems and to construct synergies between them. This analysis is more than an application of practical intelligence to the setting of technology in organizations. It is a refinement of the broader consideration of practical intelligence such that the systems savvy construct can be utilized for assessment, training, and even the design of technology and organizational practice. Thus, we contribute to the important dialogue around the foundations of individual contributions for managing and designing sociotechnical systems and try and respond to requests for research related to the antecedents of sensemaking (e.g., Ratzmann et al. 2018). While others (e.g., Majchrzak et al. 2000; Schmitz et al. 2016; Thomas and Bostrom 2010) have considered how people come to adapt technologies and tasks, here we take a deeper dive into the capacity to understand and design these adaptations.

We define and place systems savvy in context with other characteristics related to technology implementation success, demonstrate differential systems savvy across undergraduate business students and technology consultants and managers, provide invitations for future research related to technology design and implementation, and the role of systems savvy in group design and practice. We contribute to the practical intelligence and tacit knowledge literatures by examining practical intelligence in a new context.

2 Literature Review and Theory Development

2.1 Practical Intelligence and Systems Savvy

Systems savvy is a unique form of practical intelligence necessary to adapt to, shape, and/or select more effective sociotechnical integrations. Practical intelligence is the ability to adapt to, shape, or select better “environments” for real-world success (as compared to academic success) (Sternberg et al. 2000). Sternberg and his colleagues describe practical intelligence as tacit knowledge with three underlying intelligence components: metacognitive, performance, and knowledge-acquisition (2008).

The metacognitive component of intelligence is an ability for reflective, higher order (global) planning, as opposed to impulsive lower order (local) planning (Sternberg 1981). In our systems construct, savvy people do not get locked into thinking about problems from their dominant frame (e.g., an IT professional who only thinks about the technology). In the context of sociotechnical systems, one must encode aspects of the available technology and the social systems, as well as their interactions. Systems savvy people have a well-developed ability to encode elements of technological and social systems in a way that gives them a rich understanding of the problem.

The performance component of intelligence is an ability to infer and act upon relationships inductively. Performance in a sociotechnical setting involves grasping how relationships could be between technology, persons, and social systems; and possible relationships that might be created by people at the social-technology interface. This includes emergence: the ability to recognize the emerging natural patterns as they develop from chaos and capture them into action (Goldstein et al. 2010). Performance components of intelligence are used to execute the stages and strategies of problem-solving. Systems savvy people can develop and act on potential relationships between technological and social systems.

Knowledge acquisition is the ability to learn from experience (Sternberg 1987). In the context of sociotechnical systems, this not just about facts, but system dynamics. Systems savvy people learn from their interactions with technology and social systems in ways that enable them to refine their understanding of sociotechnical synergies.

2.2 Related Concepts

Our systematic search for research on practical intelligence and tacit knowledge as used in work contexts (we excluded school testing and school admissions) found 10 empirical studies between 2008 and 2018 (Baczyńska and Thornton 2017; Baum and Bird 2010; Baum et al. 2011; Bhattacharyya and Soumyaja 2010; Griffith and Sawyer 2006; Joseph et al. 2010; Langer et al. 2014; Mussel 2013; Prasarnphanich et al. 2016; Taylor et al. 2015). The foci of the ten can be classified into four types: adaptation to life situations or change, problem solving, management or leadership, and project management. They involved situations as diverse as immigration, education, professional and managerial roles, information technology, and technical sales. However, none of these studies addressed the ability to see the interdependence of technological and social systems and to construct synergies between them. We offer systems savvy as this unique ability.

In Table 1, we provide a comparison of systems savvy and other individual attributes related to the use of technology in organizations. We assess each of the individual attributes and code them as antecedent, codependent, outcome, or moderator of systems savvy effects. Antecedents include general stable traits such as academic intelligence or personality factors. These can support individual learning from the experiences essential to the development of the tacit knowledge of systems savvy. Codependent attributes appear to share causal inference with systems savvy. Outcomes include those knowledge, skills, and capabilities that result from the systems savvy perspective on how technology and social dimensions interplay. Moderators strengthen the effect of systems savvy on performance outcomes.

Table 1 Comparison of systems savvy and individual attributes related to technology use in organizations
Full size table

2.3 Summary: Systems Savvy

Our review of practical intelligence and tacit knowledge showed the value of these broad concepts in a variety of applied settings. However, we did not find research with an explicit focus on the ability to see the interdependence of technological and social systems and to construct synergies across them, and we argue that this is a conceptual gap that should be filled. The need for explicit consideration of the sociotechnical system is likely stronger now as increasingly powerful, yet subtle, technologies, like artificial intelligence systems, come into play. We present systems savvy as a category of tacit knowledge that may help us understand why some people are more likely to see and engage with the dimensions of a sociotechnical system—rather than trying to work with the social or the technical individually or separately. In the sections below, we present empirical support for the basic construct of systems savvy.

3 Empirical Exploration of the Systems Savvy Construct

We first use a qualitative process (Weekley et al. 2006) to determine if experienced professionals, (identified by peers or superiors as having been consistently successful in integrating technical and social systems), describe the exploration and emergence of the interplay between technical and social systems in organizations as suggested by the theory described above. Our development of a situational judgment task (SJT) provides content and construct validation.

Our first task is to identify the system savvy construct in the experiences of successful experienced professionals. We do this through a series of interviews and focus groups. The judgments of experienced professionals guided the development of a scale to assess system savvy applicable to a wide array of professionals charged with integrating technology, organizational, and team work systems.

We then use a quantitative approach of criterion-related validity to evaluate the effectiveness of field-appropriate measures to differentiate experts from novices, thus supporting the validation and implementation of systems savvy in organizational contexts. By field-appropriate, we mean the tools with content and face validity that would meet organizational time limitations, as well as having construct and criterion validity. We envision that a systems savvy assessment could be used to address needs ranging across training and talent management, change implementation, team demographics and group composition to assess system savvy influence on group processes and technology integration. Criterion-related validity can be tested by showing that the construct assessment can effectively differentiate successful professionals from novices beyond chance. Thus, we hypothesize:

Hypothesis 1

The theoretical keying of responses to tacit systems savvy as assessed via SJT scenarios will differentiate experienced professionals from novices.

Further, we wanted to determine if the system savvy assessment would be more effective at identifying successful professionals than mere self-reports of sociotechnical awareness—the extent to which someone believes that they concurrently consider organizational and technological aspects of change.

Hypothesis 2

Tacit systems savvy will incrementally discriminate experienced professionals from novices beyond that of explicit sociotechnical awareness.

3.1 Developing a Situational Judgment Task to Assess System Savvy

We used a critical incidents technique (Weekley et al. 2006) to identify more and less successful approaches to situations described by Subject Matter Experts (SMEs) related to the integration of technology and organizational systems.

3.1.1 Methods

Sample The first author identified 13 SMEs by presenting the idea of systems savvy to three multi-company audiences (an executive program, a dinner presentation, and a networking group) of technical and non-technical managers in the Silicon Valley. Following the brief description of systems savvy, the audience was asked for help to connect with people who had had success in their organizations by making consistent use of both technology and organizational practices. Introductory emails and follow-on phone calls yielded commitments to participate from 13 SMEs located across the United States and Australasia.

Procedure The first author scheduled telephone interviews with the SMEs to generate scenarios in which they were faced with challenges of integrating technology and people within an organizational context. The schedule confirmation email and the interview itself opened with this stem (drawing from Sternberg and Hedlund 2002):

We are trying to understand the key lessons that managers learn on the job about building and managing broad systems – systems built of technology, organizations, and people. We are trying to identify specific examples of systems savvy or wisdom. Our belief is that this knowledge is often not discussed openly, but nevertheless is used by successful managers as they meet the demands of their jobs. This knowledge may have been learned because of some challenge or problem you faced. It would help if you could tell me a story or relate to me an experience you have had in which you learned an important lesson about technology, organizations, and people. Something you learned that you wouldn’t find in a book.

3.1.2 Eliciting the Construct

The result of each interview was one or more stories that articulated the SME’s observations and insights about integrating technology and organizations. Interestingly, some scenarios started with an organizational problem, challenge, or opportunity which was addressed by applying and integrating into the organization a technological solution, while other scenarios described a technological limitation or capability which led to the adaptation of an organizational system. The descriptions provided by the SME’s of their approach and organizational needs were frequently followed by exploration of technical solutions, and technological limitations or technical capabilities were often responded to by exploring organizational systems. In several of the scenarios the element of exploration and emergence of solutions were evident. Additionally, we could see from how they described their approach to a solution, that when an organizational need was first identified, the exploration began with searching for technical solutions, and when a technological capability was first identified, their exploration began with an analysis of the organizational system in which that technical capability would be situated.

We consider the descriptions provided by our SMEs to be an initial construct validation of the concept of systems savvy. The SMEs were all able to describe how they had used tacit knowledge to develop solutions integrating technological and organizational systems. This snippet of an interview shows learning from past virtual team experiences where the only change was to location (not a success) and then a more multidimensional process to make it work in the next setting:

We decided that one of the big centers of talent was in Beijing. Great graduates, eager people. Challenge is that we wanted to set it up right. I’ve observed when there are engineering campuses that aren’t located down the hall… there’s a desire to treat other sites as vendors – specific tasks, throw it over the wall, call us if there is an emergency - otherwise call us when it’s done. Wanted to do this one right. [Considered] a lot of different factors. Initially leaders sat down as a group – set out clear vision statement as collective…. Needed common tribe that spanned two continents. Systematically, we needed summits. [Treated it] like a trade show every six months: one side or other would fly in and spend a week – like a family reunion….

Their more day to day work took place over teleconferences, but these always started with 15 min built into socialize (or fix the connection). Notes were taken in real time on the screen, so spoken language was less of a problem.

3.1.3 Measure Development

Given systems savvy’s foundation in practical intelligence and tacit knowledge, we selected a situational judgment task (SJT) approach (Sternberg and Hedlund 2002) to measurement development. SJTs present the respondent with plausible scenarios and different alternatives for action, which are then ranked or rated (Chan and Schmitt 2002). We believe it is possible to capture systems savvy at a general (and generalizable) level by keeping the context focused on organizational practices and technology tools, rather than particular job situations understood only by incumbents. This is in line with Sternberg et al. (2000) contention that practical intelligence draws from tacit knowledge about oneself, others, and the situation. Thus, we use the development of a situational judgment task as evidence of the construct and face validity of the systems savvy concept.

It takes several steps to develop an SJT. These include creating the item stem (basic situation scenario), response options (a set of better and worse responses to the situation), and response instructions (guidance on selection of responses), and then evaluating the effectiveness of the responses and scoring the SJT. We followed the guidance of work summarized by Weekley et al. (2006) and used in a similar context in Griffith and Sawyer (2009).

Item Stems and Responses We composed short one paragraph situational scenarios by editing the SMEs’ raw descriptions for brevity and clarity while maintaining the critical story points. We then drafted responses to the scenarios using the SME’s responses and adding other possible responses to match the constructs of our research. Our response generation method was thus a combination of responses provided by the SMEs in the foregoing critical-incident exercise (Weekley et al. 2006) and then the creation of additional responses if all options driven by the theory were not part of the raw set of responses. Each draft scenario contained three unidimensional responses; one each offering a solution based on a technical, organizational, or emergent (allowing for evolution) approach. The other two responses each included multiple dimensions: one with an integration of technology and organization, and one with an integration of technology, organization, and emergence.

To determine response effectiveness, we recruited an additional set of experts from a large Midwestern manufacturing organization with a services group involved in organizational design and collaboration. We conducted two separate 90-min, 25-person, focus groups of professionals from across the organization. Focus group participants first privately considered the desirability of each draft scenario response, then through discussion within the focus group, suggested improvements to the responses. This follows the approach suggested in Weekley et al. (2006, p. 166).

The resulting situational judgment test for systems savvy is composed of six scenarios (one removed and one added as a result of these focus groups) with five behavioral responses for each. (See “Appendix 1” for a sample scenario. The full SJT assessment instrument is available from the first author).

Scoring Key The purpose of the SJT is to assess the extent to which respondents understand the tradeoffs between technology and organization and can identify appropriate ways to integrate across these dimensions. Thus, we chose the rank-order forced-choice response method in order to capture the full range of options and thus maximize variance over the pick best or pick best/pick worst methods (Weekley et al. 2006). The Spearman’s rank-order correlation comparing the respondent’s rank ordering to the best rank ordering (as described below) is the respondent’s score for each scenario.

Based on the theoretical constructs of systems savvy and the focus group discussions, we determined that the best rank ordering of the options would be: (1) scenario responses offering intertwined human and technical dimensions with the acknowledgement of possible emergent outcomes—noted as TOE in “Appendix 1” for technology-organization-emergent), (2) scenario responses offering intertwined human and technical dimensions—noted as TO in “Appendix 1” for technology-organization, (3) scenario responses offering only an emergent dimension—noted as E in “Appendix 1”, (4 or 5) scenario responses offering only an organizational dimension—noted as O in “Appendix 1”, (4 or 5) scenario responses offering only a technology dimension—noted as T in “Appendix 1”. Emergent-only responses were ranked above organization or technology-only responses given their signaling of conceptual flexibility. The ranking of technology-only and organization-only depended on the root source of the scenario. If the scenario was rooted in a technology problem, then technology-only was ranked 5th, recognizing that a technology only response would be ignoring the organizational context and adaptation to the situation. When the scenario was rooted in an organizational problem, the organization-only response was ranked 5th, recognizing that such a solution would ignore the interdependence of organization on technology. (We investigated alternative rankings, including switching the ranking of organization only and technology only, or allowing ties between the technology-only and organization-only rankings. The theoretical rankings provided the greatest fit to systems savvy judgments.)

Our SMEs provided critical incidents that were clearly consistent with the theoretical sociotechnical systems framing. The scenarios reflected the dynamic interplay between technical systems capabilities and organizational system processes. Furthermore, the SMEs provided potential responses to these scenarios that reflected the emergent interplay between technical capabilities and the organizational and social systems in which they were used.

While experience is just one way of developing systems savvy, measures of systems savvy should be predictably correlated with experience with technology systems in organizational settings. Similarly, the systems savvy construct should be able to reliably differentiate between those with substantial technical–organizational system experience versus novices. Thus, we gathered further criterion validity evidence, by constructing a measure of sociotechnical awareness and then testing the relative ability of sociotechnical awareness and our SJT measure of system savvy to differentiate persons successful at sociotechnical problem solving from novices; persons with little or no experience in technical and organizational problem solving. If system savvy is a distinct construct from sociotechnical awareness, then the measure of system savvy should significantly increase our ability to differentiate successful sociotechnical problem solvers from novices beyond that of sociotechnical awareness alone.

3.2 Criterion-Related Validation of Systems Savvy

To test our expectation that the tacit knowledge measure of systems savvy would assess a form of practical intelligence not captured by self-report explicit measures of sociotechnical awareness, we created a survey research design in which we compared our tacit knowledge measure to an explicit awareness measure. We recruited two samples; one of successful experienced professionals and another of academically trained individuals lacking in the experiences postulated to lead to the development of systems savvy tacit knowledge. SJTs are commonly validated by contrasting novice and expert or highly experienced person responses (e.g., Weekley and Ployhart 2006). Here, we tightened the selection to focus on systems savvy designees. These are individuals identified by their peers as successful professionals in sociotechnical settings. This approach avoids sampling individuals who may be technical experts, but who lack a savvy understanding of the interplay between technology and organizations. We compared these designees to a sample of novices who had appropriate business training, but lacked the experiences expected to be necessary to develop systems savvy tacit knowledge. (Comparing disparate groups in this way is one way to demonstrate criterion-related validity, e.g., Collins and Schmidt 1993).

3.2.1 Materials

The materials included a survey consisting of the system savvy SJT described and developed above. The SJT scenarios and their responses were presented in random order. Additionally, we included an eight-item, seven-point Likert-style explicit sociotechnical systems awareness survey (See “Appendix 2”). Inspired by Macdonald and Uncles (2007), the survey items addressed the extent to which the respondent considers organizational and technological changes concurrently. This sociotechnical awareness assessment is intended to measure the extent to which respondents are explicitly aware of the need to consider social or organizational processes and technology together when making decisions about workflow processes or technology adoption.

Because we were concerned that the experience of responding to the SJT would prime respondents to respond in a particular way to the sociotechnical awareness scale, and because we wanted to test the incremental effectiveness of system savvy as measured using the SJT, we presented the sociotechnical awareness assessment first followed by the tacit system savvy SJT.

3.2.2 Participants and Survey Procedures

Designees Each of the 39 respondents in the designee group was sent the survey link by a supervisor or peer (Contact) who identified them as having high levels of success in a technologically complex organizational environment. We instructed the Contacts to share the link with people successful at seeing and mixing together human capabilities, technology tools, and organizational practices. We noted that their ability need not be based on expertise, but rather the understanding that “silver bullet” strategies will not work, and that organizational action will be more successful to the extent that these dimensions are managed in concert with an appreciation for emergence.

This method of identifying respondents is more stringent than used in some prior work where experience or membership in a professional association is used (e.g., Baum et al. 2011). Our pilot interviews suggest that supervisors and peers can effectively identify high versus low systems savvy colleagues; when asked to make designations, did so without hesitation and could describe the behaviors leading to their assessment. Based on Contact feedback, the designees are from Fortune 500 technology and consulting companies, with 43% of from India, and the remaining 57% from North America. All were sent the same link to the anonymous survey so further identification is not possible.

Novices Students in business administration majors at a major eastern U.S. university were recruited through junior and senior level courses within their major to complete an online survey. Students received credit toward the participation grade in their classes. When students signed up to participate, they were asked to indicate their sex, age, and work status. This was done to assure that the novice sample met the criterion that they not have had significant full-time work experience. The novice sample was composed of 58% female, 42% male students (which approximates the 56% female population of the sampled majors), with an average age of 21 which is consistent with the junior-senior undergraduate status. We sent the same materials to the 182 participants who met the novice criterion as was sent to designees.

3.2.3 Results

We compared each of the scoring keys across the designee and novice samples to determine if each scenario differentiated designees from novices based on a t test comparing designee and novice scores. We also tested the resulting scoring key using a binary logit model to determine if the SJT scale increases accurate classification of designees and novices compared to random chance and compared to sociotechnical awareness.

SJT Scoring Analysis We first applied the scoring approach identified in the qualitative analysis to each response to the SJT scenarios. For each respondent, a Spearman rank-order correlation (rho) is computed between the respondent’s rank ordering of the possible responses with the preferred rank ordering generated by the SMEs. Thus, the score for each scenario is the rank order correlation (rho) of the responses with the SME derived rank ordering. We then compared the average scores for the designee and novice samples on each of the six scenarios using t tests to determine the ability of each scenario to differentiate the designee respondents from the novice respondents.

Table 2 shows the average Spearman rank-order correlation coefficient for designees and novices and the difference between Spearman rho given each scoring key for the six scenarios (Virtual Team, Social Media, Computer-Based Collaboration, Bank Workflow Automation, Office Configuration, and Meeting Technology). Five of the six scenarios supported Hypothesis 1, providing significant differentiation between novice and designee respondents. The scenario focused on computer-based collaboration did not significantly differentiate the designees from novices, and thus was dropped from further consideration. The participant’s SJT score then is the average rho across the five scenarios. The bottom panel of Table 2 shows the average of the SJT scores across all designees and novices as well as the t test comparing designee versus novice SJT scores, again confirming Hypothesis 1.

Table 2 Average keyed response score (rho) for six scenarios for designees and novices
Full size table

Hypothesis 2 states that the system savvy SJT measure will incrementally differentiate designees from novices beyond that of self-report sociotechnical awareness alone. To test Hypothesis 2, we first provide an evaluation of the sociotechnical awareness measure. Cronbach’s alpha for the eight-item sociotechnical awareness measure in the 221-person sample is 0.893. Separate reliability analysis by sample indicates that the sociotechnical awareness scale has a slightly higher reliability among novices (0.876) than designees (0.804) although the reliability is within acceptable range for both samples. Confirmatory factor analysis of item covariance matrices using maximum likelihood confirms that the sociotechnical awareness scale is unidimensional with a single factor accounting for 51.77% of the variance. Modification indices indicated that item pairs 1–7 and 2–8 have correlated errors. As noted in the “Appendix”, these are common items with reflected wording, justifying correlated errors. The resulting single factor model resulted in standard measures of fit (Jackson et al. 2009: χ2 = 29.452, df = 18, p = 0.043, RMSEA = 0.054, CFI = 0.987, TLI = 0.980). Multiple-Group CFA of the two subsamples confirms that the single factor structure is common to both samples. Comparing the model constraining the measurement weights to be equal versus an unconstrained measurement weights does not result in significant change to model fit (Δχ2 = 10.087, df = 7, p > 0.10). Thus, the model fits equally well in both samples.

Table 3 presents descriptive statistics and bivariate correlations for sociotechnical awareness, and the system savvy SJT. Both the sociotechnical awareness and the SJT scales do have moderate point biserial correlations with designee/novice status. While significantly different from zero, the correlation of sociotechnical awareness with the system savvy SJT is small, indicating that the two measures are tapping different constructs.

Table 3 Mean standard deviation and correlations of sociotechnical awareness and system savvy SJT with binary novice/designee
Full size table

Because 182 of the 221 respondents are known novices, the baseline expected classification would be to randomly classify all respondents to the novice category. Doing so would result in an overall correct classification rate of 82.4%. However, 17.6% would be misclassification: i.e., 100% of the designees would be misclassified as novices. Thus, there is value in determining if our assessments allow correct identification of those people known to have successful experience; i.e., our designees.

Tables 4, 5, 6 present a two-step binary logistic regression of designees versus novices, first on the sociotechnical awareness, and then the SJT scale. In support of Hypothesis 2, that systems savvy SJT will incrementally discriminate experienced professionals from novices beyond that due to the sociotechnical awareness (we draw on Clevenger et al. 2001 for our evaluation approach), we find that sociotechnical awareness alone results in a Cox and Snell R2 = 0.213 (Table 4) and a correct classification proportion of 86%, with only 33.3% of designees being correctly classified and only 2.7% of novices incorrectly classified (1.0–97.3% correct, Table 5). The addition of the system savvy SJT scale improves the Cox and Snell R2 to 0.353 (an increased R2 = 0.140) and increases correct classification to from 86 to 91.4%. Most notably, there is no change in correct classification of novices while correct classification of designees increased from 33.3 to 64.1% (a 92.5% increase in correct classification of designees).

Table 4 Regression summary of sociotechnical awareness (Step 1) and system savvy SJT (Step 2)
Full size table
Table 5 Classification of novices and designees based on sociotechnical awareness (Step 1) and system savvy SJT (Step 2)
Full size table
Table 6 Regression coefficients of system savvy awareness and system savvy SJT on novice/designee
Full size table

Table 6 shows the slope parameters for both the sociotechnical awareness and system savvy SJT scales. Both slopes are positive and significant indicating that they both contribute uniquely to the correct classification of designees.

3.2.4 Conclusion

Five of the six system savvy SJT scenarios, scored using the theoretically-derived scoring key, significantly differentiated designees from novices. The systems savvy SJT has a low correlation with sociotechnical systems awareness indicating that the system savvy SJT taps a tacit construct separate from sociotechnical awareness. Furthermore, the SJT measure of systems savvy increases the accurate classification of designees from novices over sociotechnical awareness by 92.5%, indicating that it is capturing a meaningful tacit knowledge characteristic of systems savvy.

4 Discussion

We proposed a new theoretical construct: systems savvy—the capacity to see the interdependence of social and technical systems and to construct synergies between them. Results from two studies offer face, construct, and criterion-related validity for the new systems savvy construct. Subject matter experts could describe situations where they had integrated social and technical systems for successful solutions to problems or the creation of new opportunities. This result supports the face and construct validity of systems savvy.

Novice and successful professionals responded in predictably different ways to the systems savvy situational judgment task based on the scenarios. As expected, novices selected responses with preference to either social or technical solutions (unidimensional solutions), without recognition of the interplay or emergence of integrated solutions, while successful professionals consistently coded multidimensional solutions as superior to unidimensional ones. Additionally, within the unidimensional solutions, successful professionals consistently ranked higher unidimensional responses that would broaden the overall dimensions at play within a scenario. That is, they gave preference within the unidimensional responses to technology solutions within scenarios where organizational issues were focal in the stem of the scenario and to organizational solutions where technological issues were focal in the stem of the scenario. Novices, who we expect to have less systems savvy, prefer more simplistic solutions—“silver bullet” (e.g., Brooks 1987) approaches. Successful professionals, who we expect to have more systems savvy, prefer solutions that tap and extend both organizational and technical dimensions. This study, however, does not illuminate the dynamics or depth of systems savvy noted in the construct design and so the possibilities for further theorizing and research are discussed below.

Our sample and context may place limitations on the direct generalizability of this work, though this also suggests opportunities for future expansions. The participants were from North America and India. Some have noted that the rich context presented in SJT could accentuate cultural differences such that participants from different cultures understand the presented situational context differently, and thus, reduce the predictive validity when using the tool across cultures as we have here (e.g., Lievens et al. 2008). In this instance, we think this makes our context a more stringent test as cultural differences across the criterion participants would have likely blurred distinctions. Additionally, given the self-report nature of SJT, the possibility exists that participants faked their savvy (Peeters and Lievens 2005). However, here the demand characteristic would have been to come across as savvy as possible, for all subjects, and we still find differences across the novice and criterion groups. If the tool were being used such that less savvy would have greater value, faking is something that should be considered.

Depending on the goals of future work, the specific scenarios may be limitations as they age in context or are simply not relevant to some set of future participants. However, the theoretical model of organizational, technological, and emergent dimensions offers the opportunity to keep the framework of these scenarios and responses, and change/update the specifics to meet the needs of future studies. We have demonstrated that a conceptually-based scoring protocol provides robust prediction.

4.1 Contributions and Next Steps for Theory and Practice

Broadly, systems savvy is an application and refinement of practical intelligence (Sternberg et al. 2000, 2008) focused on sociotechnical systems. The three primary aspects of intelligence: metacomponents, performance components, and knowledge-acquisition components are translated in the context of systems savvy as encoding across available technological and organizational dimensions, acting in ways that create sociotechnical systems rather than “silver bullet” (unidimensional) approaches, and then learning both for general future use and to handle emergent outcomes in the specific setting. Scholars have addressed practical intelligence across a variety of environments. We believe the evaluation of systems savvy and other broader appraisals of practical intelligence (e.g., organizational change, soft skills), offer greater generalization opportunities than research focused on very narrow contexts (e.g., technical sales engineering, systems analysis). Certainly, future research is needed to replicate the validation of the systems savvy situational judgment test, but the face, construct, and criterion-related validity assessments described here open the practical intelligence research stream to future work focused on the theoretical processes of systems savvy (encoding, acting, and learning). Systems savvy may be an especially fertile area of practical intelligence research given the opportunity to apply technical and structural variations (and the ability to naturally engage with sophisticated measurement tools) in situations where such variations are to be expected, rather than appearing unnatural or experimental.

The study of systems savvy as a form of practical intelligence is also valuable given its inherent focus on action and improvement. We provide an extension for the role of cognition in sociotechnical systems development as we categorize a variety of constructs as antecedents, codependents, moderators, or outcomes to systems savvy. Thus, systems savvy provides an important opportunity to connect practical intelligence as we consider cognition’s role in the design and management of sociotechnical change (e.g., Kolfschoten and Reinig 2013).

Especially high levels of savvy may result in understanding that goes beyond even designed-in features of technology and organizations—in essence, people with high systems savvy capacity might extend the effects of technical and organizational capabilities given their greater understanding of the overall system. These insights could result in ground-breaking innovation and transformation. Research could also focus on whether and how different technology and organization designs support systems savvy development or enactment. For example, does interaction with more malleable technologies result in greater systems savvy over time as people practice with the flexibility? Would the same dynamic result in settings with malleable organizational structures or processes?

Systems savvy also has much to offer team work design. Evidence suggests that while collaborative technologies are used in team settings, the use can be ineffective and simplistic (Beise et al. 1992). We are interested in predicting effectiveness based on the systems savvy represented in the team. Team leaders with greater systems savvy are expected to be more sensitive to triggers for change; team members with systems savvy, and teams with more systems savvy across their membership, are more likely to see and act on needed adaptations. Our results offer one step in response to Ratzmann et al.’s (2018) request for better understanding of team member characteristics and sensemaking.

Systems savvy may play an especially important role in team outcomes when wielded by the team leader. Building from adaptive structuration theory (DeSanctis and Poole 1994), Thomas and Bostrom (2010) identified five triggers for technology adaptation in virtual teams—changes to the team’s technology and use of the technology with the goal of more effective and efficient interventions:

  • External team constraints: scope, time, etc.

  • Internal team constraints: number of team members, diversity, etc.

  • Information/communication technology (ICT) inadequacy: lacking features or operation capabilities, etc.

  • Trust and relationship inadequacy: conflict, communication, etc.

  • ICT knowledge, skills, and abilities inadequacy: team members lacking knowledge to use the ICT, etc.

Thomas and Bostrom note that this triggers approach (and their accompanying trigger diagnostic tool) can be used in training team leaders to be more reflective and help leaders see the situation in a new way. They note:

…leaders who may have a tendency to see technology adaptation context from a task-technology fit perspective will have the opportunity to identify and see the relationships between technology inadequacies and the other triggers. This would hopefully enable them to craft more effective interventions by seeing all the root cases when they intervene. (p. 133)

Our contribution is to offer a way to identify these tendencies in a systematic way. Team composition efforts can consider the systems savvy of the leader and members. Additionally, prior research on training for tacit knowledge (Moskaliuk et al. 2016) supports the idea that systems savvy can be taught. We expect that training focused on reflection and hands-on, contextually-situated presentations (Herrington et al. 2007), will offer some of the benefits that breadth and depth of experience have on the development of tacit knowledge more generally. Use of the situational judgment test offers an opportunity for baseline measurement of selection or training outcomes. We believe that the systems savvy situational judgment test can have additional value in practice as it also gives leaders the ability to assess, support, track, and leverage the systems savvy of their employees.

4.2 Summary

We offer a new construct for understanding the construction of sociotechnical systems. While some prior work (e.g., Cecez-Kecmanovic et al. 2014; Leonardi 2012; Orlikowski and Scott 2008; Zammuto et al. 2007) suggests how technical and organizational processes are intertwined, we have not had the constructs or tools to empirically address human capacities related to how this intertwining takes place. Second, the systems savvy construct is squarely placed within the theoretical frame of practical intelligence, and by examining it in a new context (as shown from our review), we create a tighter link between the intelligence literature and technology design and management. Third, we offer an extended ontology to ideas of sociotechnical systems that includes differential capacities for encoding, acting, and learning from sociotechnical settings. Finally, we provide a field-appropriate metric to enable empirical research to test our propositions and future research questions and give managers the opportunity to track and manage systems savvy in their organizations.

5 Conclusion

People, technology, and organization practice intertwine (Trist and Bamforth 1951; Zammuto et al. 2007), yet we continue to hear of settings where a unidimensional “silver bullet” approach leads to less than satisfactory results (Nelson 2007). From office design (Khazanchi et al. 2018) to mobile phone use (Schmitz et al. 2016) and artificial intelligence support for cancer treatment (Lim and Lee 2017), people engage with technology tools in the complex context of their work. We propose that the capacity to see the interdependence of technological and social systems, and to construct synergies between them, is critical to individual, team, and organizational success. We describe this capacity as systems savvy—a specific form of practical intelligence (e.g., Sternberg et al. 2000)—and validate a situational judgment test that can support future research at the individual, team, and organizational levels of analysis.