Direct Observation Form (DOF)

Abstract

The Direct Observation Form (DOF; Butterfield et al., 2023) was developed as an index/indicator to accurately assess medical student performance on psychiatric interviews within the psychiatry clerkship. Its content was established through a review of relevant sources, including those from the Association of Directors of Medical Student Education in Psychiatry (ADMSEP; Roman et al., 2016), and further refined through pilot testing. The resulting 4-item DOF was subsequently evaluated with a sample of third-year medical students. Ratings for this evaluation were provided by supervising residents, fellows, attending physicians, or other team members. The study reported on the reliability and validity of these items, showing the DOF to be a valuable tool for medical student evaluation in this specific context.

Keywords

Test Responsiveness, Direct Observation, Medical Student Performance, Psychiatric Interviews, Psychiatry Clerkship, Educational Measures, Medical Residency, Medical Students, Psychiatric Training, Student Characteristics

Authors

Austin Butterfield, Ashley Curry, Joel Yager, Joseph Sakai

---

Purpose

The primary purpose of the Direct Observation Form (DOF) is to provide an accurate and comprehensive assessment of medical student performance specifically within the psychiatry clerkship, with a focus on their skills in conducting psychiatric interviews.

Validity

Test Responsiveness: The Direct Observation Form (DOF) demonstrated significant test responsiveness, as evidenced by a substantial increase in the mean DOF score from time 1 to time 2 (p < 0.001). This indicates that the instrument is sensitive to changes in performance over time.

Predictive Validity: Scores from the DOF submitted at time 2 showed a positive and significant association with end-of-course clinical grades (p < 0.001). This association remained statistically significant even after controlling for several confounding variables, including the time of academic year the course was completed, rater rank, complexity of the case, and difficulty of the interview (time 2 p < 0.001). Furthermore, mean scores from the DOF were associated with the time of year students took the course, with students assessed earlier in the academic year exhibiting lower average scores (p-values = 0.01 at time 1, 0.002 at time 2).

Association with Rater Rank: Scores on time 1 DOFs were positively associated with rater rank (p = 0.005), indicating that residents tended to assign higher scores than faculty members.

Correlation with EPA Assessment: The DOF scores correlated strongly with an Entrustable Professional Activity (EPA) assessment of the interview, demonstrating a significant positive relationship (time 1 r = 0.76, p < 0.001; time 2 r = 0.79, p < 0.001).

Lack of Correlation with Shelf Exam Scores: Notably, the DOF scores did not correlate with shelf exam scores (time 1 r = 0.10, p = 0.24; time 2 r = 0.11, p = 0.21), suggesting that the DOF assesses a distinct set of skills not captured by traditional written examinations.

Reliability

Internal Consistency: The Direct Observation Form (DOF) demonstrated good internal consistency, as indicated by Cronbach’s alpha values ranging from 0.88 to 0.89. This suggests that the four items within the scale consistently measure the same underlying construct.

Factor Analysis

No data is Available.

Instrument

Test Type: Original
Format: The Direct Observation Form consists of four items, each scored on a 5-point scale, with 5 being the highest possible score. The items are: (1) opening portion and closing of the interview, (2) interview techniques, (3) presence/attunement to patient, and (4) alliance building and boundaries. Additionally, the rater has optional checkboxes to indicate if an interview was particularly challenging for the learner’s level of training or if the interview exhibited high complexity. Finally, the rater is required to provide an overall score, formatted as an Entrustable Professional Activity (EPA) assessment question.
Language Available: English
Population Group: Human (Male and Female)
Age Group: Adulthood (18 years and older)
Population Details: The respondents for the evaluation of the DOF were residents and other team members, located in the United States. The instrument is intended for use with third-year medical students.
Test Methodology: Test Validity, Predictive Validity, Test Reliability, Internal Consistency.

Keywords

Test Responsiveness, Direct Observation, Medical Student Performance, Psychiatric Interviews, Psychiatry Clerkship, Educational Measures, Medical Residency, Medical Students, Psychiatric Training, Student Characteristics.

Authors

Author ORCID Identifier:

• Butterfield, Austin: http://orcid.org/0000-0003-0231-7266

• Sakai, Joseph: http://orcid.org/0000-0003-1600-6100

Affiliation:

• Butterfield, Austin: University of Colorado School of Medicine

• Curry, Ashley: University of Colorado School of Medicine

• Yager, Joel: University of Colorado School of Medicine

• Sakai, Joseph: University of Colorado School of Medicine

Email Addresses:

• Butterfield, Austin: [email protected]

Correspondence Address:

• Butterfield, Austin: [email protected]

Permissions & Fee and Test Year

Permissions: Contact Corresponding Author.
Commercial: No
Fee: No
Test Year: 2023

References

Butterfield, A., Curry, A., Yager, J., & Sakai, J. (2023). A direct observation form for evaluation of the psychiatric interview: Pilot testing during the psychiatry clerkship. Academic Psychiatry, 47(4), 368–373. https://doi.org/10.1007/s40596-023-01762-0

Items of the Direct Observation Form (DOF)

The Direct Observation Form (DOF) consists of four core items, each rated on a 1-5 scale (with 5 being the highest score). These items are designed to assess key aspects of a medical student’s performance during a psychiatric interview:

1. Opening Portion and Closing of the Interview: This item evaluates the student’s ability to effectively initiate and conclude the interview, encompassing aspects like establishing rapport, setting the agenda, and summarizing findings.

2. Interview Techniques: This item focuses on the student’s proficiency in employing various interviewing techniques, such as active listening, empathetic responses, appropriate questioning, and managing the flow of the conversation.

3. Presence/Attunement to Patient: This item assesses the student’s ability to be fully present and engaged with the patient, demonstrating empathy, understanding non-verbal cues, and maintaining a respectful and therapeutic demeanor.

4. Alliance Building and Boundaries: This item evaluates the student’s skill in establishing a positive therapeutic alliance with the patient while maintaining appropriate professional boundaries throughout the interview.

In addition to these four scored items, the form includes optional boxes for the rater to indicate if the interview was particularly challenging for the learner’s level of training or if the case had a high level of complexity. Finally, the rater is instructed to provide an overall score for the interview, framed as an Entrustable Professional Activity (EPA) assessment question, which reflects the student’s readiness to perform this activity with decreasing supervision.