Introduction

What competencies should Instructional Design and Technology (IDT) professionals possess and how should these competencies be measured? These questions have been of interest to the researchers in the field of IDT for decades. The research efforts employed various methods and techniques including analysis of job announcements, Delphi technique, surveys and interviews and identified competencies in various domains such as instructional design models, principles, and processes (Kang & Ritzhaupt, 2015; Klein & Kelly, 2018; Moallem, 1995; Ritzhaupt et al., 2018; Sümüer et al., 2006), technology and media development (Liu et al., 2002; Ritzhaupt & Kumar, 2015; Ritzhaupt et al., 2010; Sims & Koszalka, 2008; Sugar et al., 2011, 2012), communication and collaboration (Kang & Ritzhaupt, 2015; Klein & Jun, 2014; Ritzhaupt & Martin, 2014; Ritzhaupt et al., 2010), performance improvement (Fox & Klein, 2003; Giberson, 2010; Klein & Fox, 2004), and project management (Brill et al., 2006; van Rooij, 2010, 2013). While the literature is fairly rich in terms of studies on the development and validation of competencies, the methods or tools that can be used to measure instructional design competencies of professionals have not been researched extensively. One of the tools that can be used to measure instructional design competencies is a valid and reliable self-report instrument. Few early efforts on the development and validation of a self-report instrument focused on professionals’ perceptions of the importance of competencies rather than the degree to which professionals possessed these competencies (Ritzhaupt & Martin, 2014; Ritzhaupt et al., 2018). To fill this gap in the literature, we assigned a five-point Likert scale to the widely acknowledged ibstpi® instructional design competency set and investigated the construct validity of the instrument using the input of a large sample of future IDT professionals in Turkey. Our paper reports on the construct validity and reliability of the Turkish adaptation of the instrument that have been continuously updated and validated by researchers in the field of IDT (Koszalka et al., 2013).

Related literature

Defining the field and the profession

The field of IDT is defined as “the study and ethical practice of facilitating learning and improving performance by creating, using, and managing appropriate technological processes and resources” (AECT Definition & Terminology Committee, 2008; p. 1). This definition indicates that the two major goals of the field are to facilitate learning and improve performance. Furthermore, the field of IDT encompasses the analysis of learning and performance problems and design, development, implementation, evaluation and management of instructional and non-instructional solutions (Reiser, 2018). Professionals in the field of IDT are responsible for conducting the processes that start with identifying learning and performance problems and conclude with evaluating an intervention. Therefore, professionals are expected to possess certain domains of knowledge and exhibit a variety of skills in the cyclical process of facilitating learning and improving performance.

IDT professionals wear a variety of hats as they perform their profession. It is possible to identify a number of job titles used to specify the professionals in the field. Among these titles, the most common ones include instructional designer, instructional technologist, educational technologist, training manager, learning/training project manager, learning designer, curriculum developer, e-learning developer, and performance improvement consultant. While there is a wide range of job titles used, we used IDT professionals as an umbrella term in this study. In the following section, we present a review of the literature on competencies of IDT professionals. As the focus of the current study is on instructional design competencies and previous researchers investigated a wide range of competencies of IDT professionals that include but not limited to the instructional design competencies, our focus will be on instructional design competencies reported in these studies in our literature review.

Competencies for IDT professionals

Competency is defined as “a knowledge, skill, or attitude that enables one to effectively perform the activities of a given occupation or function to the standards expected in employment” (Richey et al., 2001, p. 26). Identifying the competencies that IDT professionals should possess has been a research endeavor for decades. In fact, Gagné (1969) was one of the first researchers who attempted to identify what competencies instructional technologists should have. He asserted that instructional technologists should possess competencies in three domains: values, knowledge, and methodologies. Gagné’s first domain, values, referred to instructional technologists’ adherence to instructional design practices that are based on empirical methods and public communication. The second domain, knowledge, indicated that instructional technologists should know the subject matter of interest, instructional methods, and more importantly, instructional theories. Finally, methodologies included some core competencies such as analyzing and measuring learning outcomes, determining the impact of instruction, conducting statistical analyses, and communicating effectively.

More than half a century later, researchers and organizations continue the effort to identify the knowledge, skills, and attitudes professionals in the field of IDT should possess. A variety of methods are used by these researchers and organizations including analysis of job announcements, review of literature, and expert opinions. One of the early studies conducted by Moallem (1995) identified 150 job announcements and categorized them into three settings as business and industry, government and military, and university/college/school district. Results revealed that instructional design competencies were mostly expected of candidates for jobs in business/industry and government/military. Sümüer et al. (2006) analyzed 101 job announcements that were posted over a 6-month period and categorized them into two groups as academic and corporate. According to the results, instructional design competencies were included in the domains of educational foundations and instructional technology foundations and expected of IDT professionals by both academia and corporate.

While past research shows that instructional design competencies are expected by employers in different settings, there is also evidence that instructional design competencies are among the most frequently mentioned qualifications in job announcements. Kang and Ritzhaupt (2015) identified 400 job announcements and used a framework that consisted of three domains as knowledge, skill, and ability. Results showed that knowledge of instructional design models and principles was the most frequent competency that appeared in 55% of the job announcements in the knowledge domain. The skill domain featured soft skills such as oral and written communication, collaboration and interpersonal communication skills. Finally, while collaborating with different team members was the top competency in the ability domain, instructional design competencies such as developing course materials, evaluating learning products and programs, creating effective instructional products, and applying sound instructional design principles were among the top competencies identified.

Surveys and interviews were also among common methods used by researchers to investigate the competencies of IDT professionals. Klein and Jun (2014) surveyed instructional design professionals to investigate the importance of competencies expected of them. They identified 28 competencies in the literature and then asked professionals to rate the importance of the competencies as well as add new competencies. Results revealed that aligning objectives, interventions, and assessment was the most important competency and followed by preparing measurable goals and objectives and collaborating and partnering with others. In line with these findings, interviews conducted by Ritzhaupt and Kumar (2015) also revealed that instructional designers are expected to have competencies such as instructional design knowledge, theories of learning and instruction, information organization, and assessment. In a more recent study, Klein and Kelly (2018) interviewed 20 instructional design project managers and asked what competencies they expect of instructional designers. Results showed that using analysis techniques for determining content and tasks was the top competency that was mentioned by 18 project managers. Finally, Ritzhaupt et al. (2018) developed a survey to measure the importance of educational technologist competencies in three domains as knowledge, skills, and abilities. The investigation of the 176-item survey’s factor structure revealed a total of 31 factors. Highest rated factors included assessment, evaluation, and teaching techniques and instructional design, development, and online facilitation in the knowledge domain and instructional design, development, and evaluation in the ability domain.

The field of IDT has experienced major shifts of focus throughout the years as new technologies found a place in our personal and professional lives. Due to the new skills expected of IDT professionals, some researchers focused on competencies that required the use of technology. A Delphi study conducted by Sugar et al. (2011) identified web design basics, screencasting, and video production as important multimedia production competencies. Furthermore, communication skills were rated as the most important competency by professionals and the final list of competencies included 15 items in the instructional design and pedagogy domain. The same group of researchers reported that instructional design/ADDIE, collaboration, and e-learning competencies appeared in more than 80% of the job announcements analyzed (Sugar et al., 2012). Moreover, these researchers reported that while corporate settings focused on instructional design and performance improvement competencies, higher education job announcements focused on collaboration and technical skills. Another recent job announcement and survey study that focused on multimedia competencies showed that educational technologists should have knowledge about a variety of software, skills to use various tools, and the ability to conduct instructional design processes (Ritzhaupt et al., 2010). Ritzhaupt and Martin (2014) attempted to develop a scale through analysis of job announcements and survey of IDT professionals to measure the importance of educational technologists’ multimedia competencies. Factor analysis results revealed a 16-factor solution with highest rated factors of theories and methods of instruction in the knowledge domain, soft skills in the skill domain, and working in a team-oriented environment in the ability domain. The factor of conducting an instructional design process included instructional design competencies and was the second highest rated factor in the ability domain.

Competencies for instructional design

IDT professionals are responsible for a variety of tasks at their place of employment and in most cases these responsibilities include instructional design processes. To date, researchers have investigated the roles and responsibilities of instructional designers in various settings (Kumar & Ritzhaupt, 2017; Larson & Lockee, 2004; Moallem, 1995), employer perceptions and characteristics of entry-level and expert instructional designers (Rowland, 1992; Villachica et al., 2010), and the skills and competencies required to conduct learning design (MacLean & Scott, 2007, 2011). In addition to the individual efforts of researchers in identifying the core competencies of instructional design, several organizations attempted to identify what constitutes the instructional design practice. We present a brief review of the competencies developed by these organizations to date below.

International society for performance improvement (ISPI)

ISPI identified a set of standards for practitioners who work in the field of Human Performance Technology (HPT). The Certified Performance Technologist (CPT) Standards include 10 items (ISPI, 2020). Each item consists of a detailed list of competencies and examples as to how competencies could be exhibited in real world. Standards feature competencies for a performance technologist to design, develop, and evaluate instructional and non-instructional solutions for performance improvement.

Association for educational communications and technology (AECT)

AECT developed a set of standards for professional programs in the field of educational technology. These standards delineate the capabilities that a candidate should have in the profession (AECT, 2012). The AECT Standards are categorized into five domains as content knowledge, content pedagogy, learning environments, professional knowledge and skills, and research. In addition to a list of the skills in each domain, the standards also include indicators that can be used to evaluate the degree to which a candidate possesses the knowledge, skill, or attitude of interest.

Association for talent development (ATD)

ATD is an organization that brings together professionals from across the world who work to improve performance. The organization has a competency model which is used to grant Certified Professional in Learning and Performance (CPLP) and Associate Professional in Talent Development (APTD) to IDT professionals (ATD, 2020). The model includes ten talent development areas of expertise such as performance improvement, instructional design, and evaluating learning impact and six areas of foundational competencies that are business skills, interpersonal skills, global mindset, personal skills, industry knowledge, and technology literacy.

International board of standards for training, performance, and instruction (ibstpi)

ibstpi® is a not-for-profit corporation that provides standards, competencies, products and services to professionals working in the field of IDT with the mission to “develop, validate and promote implementation of international standards to advance training, instruction, learning and performance improvement for individuals and organizations” (ibstpi, 2020a). The board currently has five competency sets developed to be used by evaluators, instructors, online learners, training managers, and instructional designers. The instructional designer competency set was first developed in 1986 by a group of instructional design practitioners and scholars (Klein & Richey, 2005) and last updated in 2012 to meet the most recent needs of the field (Koszalka et al., 2013). In the instructional designer competency set, there are 22 competencies and 105 performance statements in five domains: professional foundations, planning and analysis, design and development, evaluation and implementation, and management. Each competency and performance statement are categorized as essential, advanced or managerial based on the level of expertise it requires. The set includes competencies required in the instructional design process such as assessing needs, analyzing learners and context, determining instructional goals, developing assessment, identifying instructional strategy and content, implementing instruction, and managing instructional design processes. The board provides the competencies on its website free of charge (ibstpi, 2020b; also, see Appendix Table 10); however, the use of the performance statements for individual and organizational purposes is regulated. A diagram showing the structure of the ibstpi® instructional designer competency set is provided in Fig. 1.

Fig. 1
figure 1

Framework of the ibstpi® instructional designer competency set

Full size image

Purpose of the current study

The instructional design literature is fairly robust in terms of instructional design competencies; however, the development of a scale was the focus of just a few researchers (Ritzhaupt & Martin, 2014; Ritzhaupt et al., 2018). In the current study, we investigated the construct validity and reliability of the ibstpi® instructional designer competencies to be used as an instrument to measure IDT professionals’ instructional design competencies. In order to accomplish our purpose, we investigated the factor structure of the performance statements in the ibstpi® instructional designer competency set. The research questions that led this study are:

  1. (1)

    How does the conceptualized model of the instrument (i.e. ibstpi® instructional designer competency set with a five-point Likert scale) fit the data collected from future IDT professionals?

    1. (a)

      How does the conceptualized model of the essential performance statements fit the data?

    2. (b)

      How does the conceptualized model of the advanced performance statements fit the data?

    3. (c)

      How does the conceptualized model of the managerial performance statements fit the data?

Method

Participants

Participants consisted of 820 junior and senior students enrolled in a degree program in Computer Education and Instructional Technology (CEIT) in Turkey. Junior and senior students were selected because they had completed a required Instructional Design course, were expected to obtain instructional design competencies, and be familiar with the terminology used by the researchers. After excluding missing data, our final dataset included 717 usable responses from 15 universities in Turkey. The sample included 378 male (52.7%) and 322 female students (44.9%). Seventeen participants (2.4%) did not report gender. Participants’ age ranged between 19 and 36 with a mean of 22.37 and a standard deviation of 2.02. There was a total of 391 junior (54.5%) and 326 senior students (45.5%) in the sample.

Instrument

We used the instructional designer competency set developed by ibstpi® in the study. The competency set includes 22 instructional design competencies (Comp) and 105 performance statements (PS) associated with the competencies in five domains. These domains are (1) professional foundations (PF), (2) planning and analysis (PA), (3) design and development (DD), (4) evaluation and implementation (EI), and (5) management (M). Figure 1 shows the structure of the competencies and performance statements within five domains as well as their categorization as essential, advanced, and managerial. Since the ibstpi® competency set lacks a scale for rating each competency and performance statement, we assigned a five-point Likert scale for participants to rate themselves. Participants responded to the question “how competent are you in the following item?” using the five-point scale which consisted of 1: not at all competent, 2: a little competent, 3: somewhat competent, 4: quite competent, and 5: very competent.

Procedures

The study was conducted in five steps. In Step 1, we contacted the ibstpi® board to obtain permission to use the competency set in a research study. Once permission was granted, we used a back-translation method in Step 2 to translate the competency set into Turkish (Brislin, 1970). In this step, the researchers translated the complete set into Turkish and asked two language experts who were native speakers of Turkish and had full professional proficiency in English to translate the competency set back to English. We compared the two back-translations with the original competency set and revised the Turkish translation accordingly. In the meantime, we identified the terms that had not yet gained sufficient ground in Turkish and were used in the competency set such as instructional and non-instructional interventions, performance improvement, and systems thinking. We identified a total of 15 terms and asked a cohort of eight academics who were experts in the field of IDT to translate these terms into Turkish. We used the input from the cohort to develop the best language for the terms identified in the competency set. When the revisions from the back-translation process were applied and the terms were successfully translated, we finalized the translation process and applied to the Institutional Ethics Committee (IEC) for the research study to be conducted. In Step 3, we obtained the approval of the IEC to carry out the processes to conduct the study. In Step 4, we conducted a pilot study with a group of 29 students from the target population to investigate the comprehensibility of the items. Students were asked to rate the competencies and performance statements on a five-point Likert scale ranging from 1: not at all comprehensible to 5: fully comprehensible. A score of four was decided as the threshold to accept an item as comprehensible, and four items that were marginally below the score of four received minor revisions. Once these minor revisions were completed, we finalized the translation of the competency set and began the data collection process. In Step 5, we collected data via scantron sheets in participating institutions with the permission obtained from each institution. Data collection began in February 2019 and ended in June 2019. Students were informed about the objectives and the voluntary nature of the study prior to the administration of the survey. All participants provided informed consent. Fifteen universities across the country participated in data collection.

Data preparation and analysis

Data preparation and analysis processes were conducted in three steps. In Step 1, we examined the dataset for major missing data and eliminated 103 responses. Furthermore, we investigated the resulting dataset in terms of missing data and conducted Little’s Missing Completely at Random (MCAR) test to find out whether data on some variables were missing independent of the observed or missing values. In this step, we also replaced the missing values using the linear interpolation method. In Step 2, we used IBM SPSS® v.25 for descriptive statistics. Skewness and kurtosis were used to establish univariate normality. We used the cut‐off values of ± 3.0 and ± 10.0 for skewness and kurtosis, respectively (Kline, 2005). Mardia’s normalized multivariate kurtosis value (Mardia, 1970) and the Raykov and Marcoulides (2008) formula were used to establish multivariate normality. In Step 3, we used Mplus 8.1 (Muthén & Muthén, 2017) for Confirmatory Factor Analysis (CFA). As presented in Fig. 1, since the competencies encompass the performance statements hierarchically, we only investigated the factor structure of the 105 performance statements in the study. Moreover, since there is already a strong theoretical framework and support from past research, we initiated our analyses with CFA (Harrington, 2009).

Results

Handling missing data

We examined the dataset for missing data and found that the observed variables missed data with a percentage between 0 and 9.3%. More specifically, 85 variables missed data with a percentage between 0 and 2.0%, 19 variables missed data with a percentage between 2.1 and 4.0%, and one variable missed data with a percentage above 4.0%. We conducted Little’s MCAR test to find out whether the data were missing completely at random. The test result was not significant indicating that the missing data mechanism was MCAR (χ2 = 18,730; df = 18,574; p = .208). After confirming that the missing values are independent of the observed or missing values and the proportion of missing data is considered low (Cohen et al., 2003; Raymond & Roberts, 1987), we replaced the missing values using the linear interpolation method.

Univariate and multivariate normality

We analyzed the dataset for descriptive statistics. There was a total of 105 performance statements in the dataset that were organized under 22 competencies and five domains. The means of the items ranged between 3.022 and 3.877 on a five-point Likert scale and standard deviations ranged between .870 and 1.181. Skewness and kurtosis statistics of the items ranged from − .588 to .030 and − .813 to − .156, respectively. These indices indicated that the dataset showed univariate normality for structural equation modeling purposes (Kline, 2005). Finally, Mardia’s multivariate kurtosis value was calculated as 128.730. We used the Raykov and Marcoulides (2008) formula to determine whether the dataset showed multivariate normality. We used the p(p + 2) formula where p is the number of observed variables and calculated the value as 11,235. Since the Mardia’s multivariate kurtosis value was greatly smaller than the value calculated using the formula, we concluded that the data showed multivariate normality.

Measurement model for essential performance statements

In the next step, we investigated the factor structures of three measurement models as essential, advanced, and managerial in three CFA’s with the maximum likelihood (ML) estimation method. In order to evaluate the model fit, we used the chi-square goodness-of-fit test, Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), and Tucker-Lewis Index (TLI). The first CFA was conducted for the essential performance statement model. There was a total of 44 observed variables under 13 first- and four second-order latent factors (see Fig. 2). Results indicated that the model showed a good fit to the data with a significant chi-square test and the RMSEA, CFI, and TLI values that were within a good fit range (see Table 1).

Fig. 2
figure 2

Measurement model of the essential performance statements

Full size image
Table 1 Chi-square test results and fit indices for the essential performance statement model
Full size table

Convergent validity

An initial investigation of the internal consistency reliability of the constructs revealed that coefficient alpha values ranged between .68 and .88. All factors showed either good or very good internal consistency except Comp2 with a coefficient alpha value that stayed marginally below the recommended .70 threshold (Robinson et al., 1991). Further in the analysis, we investigated whether constructs showed convergent validity by evaluating the individual item reliability, composite reliability, and average variance extracted (AVE). A squared factor loading is recommended to be higher than .25 for individual item reliability to be achieved (Hair et al., 2019). Squared factor loadings for the observed items ranged between .35 and .73, therefore, satisfying the criterion for individual item reliability. Furthermore, we investigated the composite reliability of the first and second order latent constructs. Composite reliability values of the constructs ranged between .68 and .94. Only one construct (i.e. Comp2) had a composite reliability value that was marginally below the .70 threshold (Nunnally & Bernstein, 1994). Finally, we calculated the AVE values of the constructs. Results indicated that the AVE values ranged between .46 and .83. One construct (i.e. Comp1) had an AVE value that was smaller than the .50 threshold (Fornell & Larcker, 1981). Results are presented in Table 2.

Table 2 Results for the measurement model of the essential performance statements
Full size table

Discriminant validity

The next step of the analysis was to assess the discriminant validity of the constructs in the essential performance statement model. Discriminant validity is concerned with differentiating a construct of interest from other constructs in the model. We assessed the discriminant validity of the second order latent constructs by comparing the \(\sqrt {AVE}\) values with the correlation coefficients in the model (Fornell & Larcker, 1981). Results for the essential performance statement model are presented in Table 3. Results indicated that none of the constructs had a perfect discriminant validity as the \(\sqrt {AVE}\) values for PF, PA, DD, and EI were smaller than at least one of the correlation coefficients in the matrix.

Table 3 Discriminant validity for the essential performance statement model
Full size table

Measurement model for advanced performance statements

We conducted the second CFA for the advanced performance statement model. There was a total of 44 observed variables under thirteen first- and five second-order latent factors (see Fig. 3). Results indicated that the model showed a good fit to the data with a significant chi-square test and the RMSEA, CFI, and TLI values that were within a good fit range (see Table 4).

Fig. 3
figure 3

Measurement model of the advanced performance statements

Full size image
Table 4 Chi-square test results and fit indices for the advanced performance statement model
Full size table

Convergent validity

An initial investigation of the internal consistency reliability of the constructs revealed that coefficient alpha values ranged between .71 and .88, therefore, all factors showed either good or very good internal consistency. Furthermore, we investigated the convergent validity for the advanced performance statement model. We used the individual item reliability, composite reliability, and AVE. In the advanced performance statement model, the squared factor loadings for the observed items ranged between .31 and .74, therefore, satisfying the criteria for individual item reliability (Hair et al., 2019). Furthermore, we investigated the composite reliability of the first and second order latent constructs. Composite reliability values of the constructs ranged between .72 and .91. All constructs had a composite reliability value that was above the .70 threshold (Nunnally & Bernstein, 1994). Finally, we calculated the AVE values of the constructs. Results indicated that the AVE values ranged between .49 and .78. One construct (i.e. Comp1) had an AVE value that was marginally below the .50 threshold (Fornell & Larcker, 1981). Results are presented in Table 5.

Table 5 Results for the measurement model of the advanced performance statements
Full size table

Discriminant validity

The next step of the analysis was to assess the discriminant validity of the constructs in the advanced performance statement model. We assessed the discriminant validity of the second order latent constructs by comparing the \(\sqrt {AVE}\) values with the correlation coefficients. Results for the advanced performance statement model are presented in Table 6. Results indicated that none of the constructs had a perfect discriminant validity as the \(\sqrt {AVE}\) values for PF, PA, DD, EI, and M were smaller than at least one of the correlation coefficients in the matrix.

Table 6 Discriminant validity for the advanced performance statement model
Full size table

Measurement model for managerial performance statements

We conducted the third CFA for the managerial performance statement model. There was a total of 17 observed variables under four first- and one second-order latent factors (see Fig. 4). Results indicated that the model showed a good fit to the data with a significant chi-square test and the RMSEA, CFI, and TLI values that were within a good fit range (see Table 7).

Fig. 4
figure 4

Measurement model of the managerial performance statements

Full size image
Table 7 Chi-square test results and fit indices for the managerial performance statement model
Full size table

Convergent validity

An initial investigation of the internal consistency reliability of the constructs revealed that coefficient alpha values ranged between .81 and .92, therefore, all factors showed very good internal consistency. Investigating the convergent validity for the managerial performance statement model entailed evaluating the individual item reliability, composite reliability, and AVE. In the managerial performance statement model, the squared factor loadings for the observed items ranged between .55 and .69, therefore, satisfying the criteria for individual item reliability (Hair et al., 2019). Furthermore, we investigated the composite reliability of the first and second order latent constructs. Composite reliability values of the constructs ranged between .81 and .92. All constructs had a composite reliability value that was above the .70 threshold (Nunnally & Bernstein, 1994). Finally, we calculated the AVE values of constructs. Results indicated that the AVE values ranged between .59 and .80. All constructs had an AVE value that was above the .50 threshold (Fornell & Larcker, 1981). Results are presented in Table 8.

Table 8 Results for the measurement model of the managerial performance statements
Full size table

Discriminant validity

The next step of the analysis was to assess the discriminant validity of the constructs in the managerial performance statement model. We assessed the discriminant validity of the constructs by comparing the \(\sqrt {AVE}\) values with the correlation coefficients. Results for the managerial performance statement model are presented in Table 9. Results indicated that the two constructs do not have a perfect discriminant validity as the \(\sqrt {AVE}\) values for Comp19 and M were smaller than the correlation coefficient in the matrix.

Table 9 Discriminant validity for the managerial performance statement model
Full size table

Discussion

The content validity of the ibstpi® instructional designer competencies and performance statements was established by previous researchers (Koszalka et al., 2013). Our purpose was to investigate the construct validity of the instrument developed using the instructional designer performance statements to measure instructional design competencies of IDT professionals. For this purpose, we translated the competency set into Turkish using a robust translation process, assigned a scale to the competencies and performance statements, and recruited a sample of future IDT professionals. The related literature is scarce in terms of developing and validating a scale to measure instructional design competencies. To date, only several researchers attempted to develop and validate a scale to be used in the field of IDT (Ritzhaupt & Martin, 2014; Ritzhaupt et al., 2018). However, our efforts diverge from these studies because we attempted to measure the competency level of participants rather than the importance of competencies for the profession. Results of the study showed that the instrument developed based on the ibtspi® instructional designer competency set can be used to measure instructional design skills, however, some considerations must be taken in doing so that are discussed in this section.

We adopted three conceptual models provided by ibtspi® to form the models to be tested in the study. These models were formed based on the category of each performance statement namely essential, advanced, and managerial. Our results indicated that each of these models showed a good fit to the data supporting the theoretical foundations of the instructional designer competency set. In the essential performance statement model, there were a total of 44 observed variables and thirteen first- and four second-order latent factors. Four observed variables were loaded on their corresponding second-order factors directly as they were the only essential performance statements in their respective competencies. While model fit is an important method to evaluate the validity of the model, it is also important to evaluate how much variance each factor explains in observed variables. An inspection of the residual variances of the observed variables revealed that 35 of the 44 observed variables had a residual variance that was below .50 which is the ideal situation recommended in the literature (Hair et al., 2019). The remaining nine residual variances ranged between .51 and .65, therefore, stayed below the recommended threshold (Hair et al., 2019). Furthermore, AVE values also showed that all constructs except Comp1 had an AVE value that was above the recommended threshold of .50 (Fornell & Larcker, 1981). Residual variance and AVE values indicated that latent factors explained an important amount of variance in observed variables, therefore, supporting the factor structure. Comp1 as a factor dealt with some of the soft skills whose importance was mentioned in the literature by a number of researchers (Kang & Ritzhaupt, 2015; Klein & Jun, 2014; Ritzhaupt & Kumar, 2015; Ritzhaupt et al., 2010; Sugar et al., 2011). The overarching competency, as stated by ibstpi®, is “Communicate effectively in visual, oral and written form” (ibstpi, 2020b). We argue that since soft skills such as communicating effectively are abstract and difficult to perceive, it affected students’ assessment of themselves when they responded to the items. As a result, the AVE value of the factor stayed marginally below the recommended threshold. In terms of internal consistency reliability, only Comp2 had a coefficient alpha value that was only marginally below the recommended threshold of .70 (Robinson et al., 1991). Other constructs had coefficient alpha values that ranged between .72 and .88 which showed that constructs had satisfactory internal consistency reliability. Additionally, composite reliability values ranged from .68 to .94 and only one construct (i.e. Comp2) had a composite reliability value that was marginally below the .70 threshold (Nunnally & Bernstein, 1994). The overarching Competency 2 is “Apply research and theory to the discipline of instructional design” (ibstpi, 2020b). Both research and theory knowledge and skills were mentioned as important competencies to be possessed by IDT professionals in past research (Kang & Ritzhaupt, 2015; Klein & Jun, 2014; Ritzhaupt & Kumar, 2015; Ritzhaupt et al., 2018). The performance statements included under this competency may have been perceived abstract by undergraduate students which may have led to its relatively low internal consistency and composite reliability. Finally, the evaluation of the discriminant validity in the essential performance statement model revealed that the factors do not have a perfect discriminant validity. This is expected as all essential performance statements are categorized under the wider umbrella of instructional design competencies and they relate to a common higher concept. Hair et al. (2019) argued that while distinctiveness among constructs is important, so is the absence of cross-loadings in the model; and, in the presence of cross-loadings and their poor representation by the measurement model, the CFA fit is also expected to be poor. When we examined the model fit indices, it is evident that the model fit to the data is above satisfactory, supporting our claim that the essential performance statements of the instrument is a valid measure of essential instructional design competencies.

In the advanced performance statement model, there were a total of 44 observed variables and thirteen first- and five second-order latent factors. Five observed variables were loaded on their corresponding second-order factors directly as they were the only advanced performance statements in their respective competencies. An examination of the residual variances of the observed variables revealed that 36 of the 44 observed variables had a residual variance that was at or below .50 and the remaining eight residual variances ranged from .51 to .69 forming a situation that was close to the ideal (Hair et al., 2019). Furthermore, as we investigated the AVE values of the constructs, we found that only one construct’s (i.e. Comp1) AVE value stayed marginally below the recommended threshold and other AVE values ranged from .53 to .78 (Fornell & Larcker, 1981). These values indicated that the latent factors explained a major portion of the variance in observed variables in the advanced performance statement model. Again, we believe that since Comp1 as a factor dealt with soft skills, students’ assessment of their competence level was affected by the abstract nature of the advanced items as well. When we investigated the internal consistency reliability of the factors, we found that coefficient alpha values ranged between .71 and .88, therefore, all factors satisfied the .70 threshold and showed an internal consistency reliability that was extensive or exemplary (Robinson et al., 1991). Additionally, composite reliability values ranged from .72 to .91, therefore, all composite reliability values were above the .70 threshold (Nunnally & Bernstein, 1994). In terms of discriminant validity, results revealed that the constructs in the advanced statements model did not meet the criteria suggested by Hair et al. (2019). However, as discussed above, advanced performance statements also share a common overarching concept that is instructional design competencies. The fit of the advanced performance statement model to data was above satisfactory which supported our assertion that it is possible to use the advanced performance statements in the instrument to measure advanced instructional design competencies.

The last model we investigated was the managerial performance statement model. The model consisted of 17 observed variables and four first- and one second-order latent factors. Residual variances of the observed variables ranged between .32 and .45, therefore, latent factors explained more than half of the variance in observed variables. An investigation of the AVE values revealed that on average the amount of variance extracted from observed variables ranged between 59 and 80%. In terms of internal consistency reliability values of the factors, coefficient alpha values ranged from .81 to .92 and all factors showed an exemplary internal consistency reliability (Robinson et al., 1991). Moreover, composite reliability values also ranged from .81 to .92, therefore, all composite reliability values were above the .70 threshold (Nunnally & Bernstein, 1994). The evaluation of discriminant validity in the managerial performance statement model revealed a similar situation to those of the two previous models. The correlation coefficient between the two factors in the model was higher than the \(\sqrt {AVE}\) values which violated the criteria suggested by Hair et al. (2019). However, since the model fit to the data was very good, we argue that it is possible to use the managerial performance statements in the instrument to measure managerial instructional design competencies.

Implications

To our knowledge, this study is the first attempt to investigate the construct validity of an instrument developed based on the ibstpi® instructional designer competency set to measure instructional design competencies. As the most comprehensive instructional designer competency set, the ibstpi® standards have been widely used by practitioners, educational programs, and organizations. We tested three different models as essential, advanced, and managerial. Each of these categories address a different level of knowledge, skills, and attitudes of their respective domains. We believe our findings have implications for the practice and research of the IDT field. Although the sample of the study was large, participants were recruited from limited demographics consisting of junior and senior students in the Turkey context, therefore, implications should be considered with the unique nature of this study. Furthermore, professionals who would like to make use of the instrument should determine the category of competencies that they would like to address and use the competency set accordingly. One of the main implications is for the educational institutions that offer undergraduate degree programs in the field of IDT. Educational programs can use the instrument with their students to measure students’ essential, advanced, and managerial instructional design competencies and make improvements to the program based on the factors that students have rated themselves poorly. We believe the findings also have implications for the entities that work with novice instructional designers who are in the early years of their career. These entities can administer the instrument to novice instructional designers to measure their instructional design competencies and address instructional design factors that received poor ratings via instructional or non-instructional solutions. Finally, the findings also have implications for the research community. Researchers can use the findings of this study to further investigate the construct validity of the instrument. As the purpose of the study was to investigate the construct validity of the instrument developed based on the ibstpi® instructional designer competency set, we refrained from making any modifications to the model by removing or amending items as we wanted to share our findings as they are with the researchers in the field and initiate a discussion over how to improve the psychometric properties of the instrument proposed in this study. Therefore, researchers can use the findings of this study as a starting point and investigate the psychometric properties of the instrument with participants from different demographics in a variety of contexts. We would like to note that any use of the competency set should be consulted with the ibstpi® board.

Limitations and future directions

There are several limitations to the study. First, we investigated the construct validity of the instrument with participants from limited demographics consisting of junior and senior students in the Turkey context. Therefore, it may be difficult to generalize our findings to other contexts or cultures with professionals coming from a variety of demographics. We encourage researchers to investigate the construct validity of the instrument with participants from different demographics, especially with graduate students and experienced instructional designers actively working in the field of IDT. Second, we used participants’ self-assessment of instructional design competencies and these reports may be biased. While it is not feasible to use other methods of assessment such as observations and portfolios with large samples, we encourage researchers to use these methods as well in future research when and if the sample size permits. Finally, the ibstpi® instructional designer competency set was originally developed in English and translated into Turkish for the purposes of this study. Although we used a robust translation process, unfamiliar terms tend to take some time in order for individuals to fully comprehend their meaning. Researchers who would like to translate the competency set into other languages may address this limitation by including an introductory session on unfamiliar terms for participants before data collection.

Conclusion

We investigated the construct validity of an instrument developed based on the ibstpi® instructional designer competency set and the results showed that the instrument can be used as a measure of instructional design competencies. We encourage researchers, practitioners, educational programs, and employers to use the competency set for individual and organizational purposes as outlined in this study. The ibstpi® instructional designer competency set has the potential to pinpoint the instructional design competency areas that require improvement. Further research in different cultures is recommended to consolidate the construct validity findings as well as keep the competency set up to date.