Agentic Engagement in Massive Open Online Courses Scale

Abstract

The Agentic Engagement in Massive Open Online Courses Scale (Kim & Song, 2023) is an inventory/questionnaire designed to measure agentic engagement within self-paced massive open online courses (MOOCs). The scale development process adhered to Hinkin’s (1995) three-step procedure, initiating with a comprehensive literature review to identify potential items related to agentic engagement in MOOCs. This initial item pool was largely informed by the Agentic Engagement Scale (Reeve, 2013; Reeve & Tseng, 2011). Subsequently, a Delphi survey was conducted to refine these items, resulting in a set of 14 initial items, with 7 specifically derived from the Delphi expert consensus. Experts reviewed both English (Reeve, 2013; Reeve & Tseng, 2011) and Korean (Bong et al., 2012) versions of the items for comparison and consistency. Content validity was rigorously assessed based on Content Validity Index (CVI) values, leading to the removal of ambiguous or double-barreled items and the establishment of a final list of seven items. These seven items underwent a back-translation process into English, reviewed by three medical doctors proficient in both English and Korean. The finalized measure was then administered to learners utilizing the K-MOOC system. Factor analysis subsequently supported the existence of three distinct dimensions among the seven final items. Empirical evidence supported the reliability and validity of the scale.

Keywords

Student Engagement, Agentic Engagement, Self-Paced Massive Open Online Courses, Agentic Support Request, Agentic Learning Strategy, Agentic Learning Construction.

Authors

Kim, Rang; Song, Hae-Deok

The primary purpose of the Agentic Engagement in Massive Open Online Courses Scale is to assess the level of agentic engagement exhibited by learners in self-paced massive open online courses.

Validity

The Agentic Engagement in Massive Open Online Courses Scale demonstrates strong evidence of validity through various analyses:

• Construct Validity: Factor analysis results provided robust support for the structural validity of the measure, indicating that the theoretical constructs are appropriately represented by the scale’s items.

• Convergent Validity: All standardized factor loadings for the scale’s items exceeded the threshold of 0.5, with a mean magnitude of 0.77, surpassing the recommended convergent validity threshold of 0.7 (Hair et al., 2018). Furthermore, all standardized factor loadings were statistically significant, confirming the appropriateness of the model’s convergent validity.

• Discriminant Validity: The square root of the Average Variance Extracted (AVE) for each construct ranged from 0.71 to 0.91, consistently higher than the correlations observed among the constructs. Additionally, the Heterotrait-Monotrait (HTMT) ratio ranged from 0.56 to 0.78, indicating adequate discriminant validity of the scale (Henseler et al., 2015).

Reliability

The Agentic Engagement in Massive Open Online Courses Scale exhibits high reliability, as demonstrated by internal consistency measures:

• Internal Consistency:

  • For the Exploratory Factor Analysis (EFA), the overall scale demonstrated a high reliability of 0.87, which exceeds the standard threshold of 0.7 (Nunnally & Bernstein, 1994). The reliability for the individual dimensions was as follows: agentic support request (0.86), learning strategy (0.87), and learning construction (0.91).

  • For the Confirmatory Factor Analysis (CFA), Cronbach’s alpha values were 0.91 for agentic support request, 0.68 for learning strategy, and 0.76 for learning construction.

  • Additionally, Composite Reliability (CR) values were 0.91 for agentic support request, 0.75 for learning strategy, and 0.76 for learning construction. All CR values surpassed the reliability threshold of 0.60 (Fornell & Larcker, 1981).

Factor Analysis

Both Exploratory and Confirmatory Factor Analyses were conducted to examine the underlying structure of the Agentic Engagement in Massive Open Online Courses Scale:

• Exploratory Factor Analysis (EFA): EFA was performed using Exploratory Structural Equation Modeling (ESEM). Initial results indicated that 1- and 2-factor models did not meet appropriate criteria for model fit. However, a 3-factor model demonstrated an appropriate fit. Parallel analysis further supported the 3-factor model as the most suitable, as starting from a 4-factor model, the eigenvalue for random data (0.99) became higher than that of the actual data (0.50).

• Confirmatory Factor Analysis (CFA): CFA results for the 3-factor structure model confirmed an acceptable model fit, with the following indices: χ²v2 = 30.972, df = 10, TLI = 0.938, CFI = 0.970, SRM = 0.048.

Instrument: Agentic Engagement in Massive Open Online Courses Scale

• Test Type: Original

• Format: The items are rated using a 5-point Likert-type scale.

• Language Available: English

• Population Group: Human (Male and Female).

• Age Group: Adulthood (18 years and older), including Young Adulthood (18-29 years), Thirties (30-39 years), and Middle Age (40-64 years).

• Population Details: The respondents were university students located in the Republic of Korea.

• Test Methodology: The methodology included Test Validity (Construct Validity, Content Validity, Convergent Validity, Discriminant Validity), Test Reliability (Internal Consistency), Factor Analysis (Confirmatory Factor Analysis, Exploratory Factor Analysis), and Structural Equation Modeling.

Authors

• Kim, Rang

  • Author ORCID Identifier: 0000-0002-4657-1170

  • Affiliation: Chung-Ang University Center for Teaching and Learning

• Song, Hae-Deok

  • Author ORCID Identifier: 0000-0003-0879-9999

  • Affiliation: Chung-Ang University College of Education, Department of Education

  • Email addresses: [email protected]

  • Correspondence Address: [email protected]

Permissions & Fee and Test Year

• Permissions: Contact Publisher

• Fee: No

• Test Year: 2023

• Commercial: No

References

Kim, R., & Song, H.-D. (2023). Developing an agentic engagement scale in a self-paced MOOC. Distance Education, 44(1), 120–136. doi.org/10.1080/01587919.2022.2155619

Items of the Agentic Engagement in Massive Open Online Courses Scale

• Number of Items: This is a 7-item measure.

• Factors and Subscales: The scale comprises 3 dimensions:

  • Agentic support request

  • Learning strategy

  • Learning construction

• Test Items Available: Yes

Agentic Engagement in Massive Open Online Courses Scale

Note: ASR: agentic support request; ALS: agentic learning strategy; ALC: agentic learning construction. Items are rated using a 5-point Likert-type scale that ranges from “strongly disagree” to “strongly agree”.