Analyzing Results from AdvancED’s Classroom Observation Tool

In 2012, the Effective Learning Environments Observation Tool™ (eleot™) became an integral part of both AdvancED® Accreditation and Diagnostic Reviews. Given the widespread use of eleot, the AdvancED research team has collected and analyzed data from more than 45,000 direct classroom observations, the results of which are summarized below. The analysis constitutes only a small number of potential analyses that could and have been done with the current eleot data. In addition to the knowledge gained from the data, AdvancED conducts regular analyses to ensure that all of the measures are performing as designed and to guide recommendations for future updates of the measures.

Description of the eleot

The eleot is comprised of 30 items organized in seven learning environments based on a review of widely used observation instruments such as those developed by Marzano and Danielson and the Classroom Assessment Scoring System (CLASS). A literature review also was conducted on learner-centric tasks, attitudes and dispositions conducive to optimal learning, including digital learning as set forth by the International Society for Technology in Education (ISTE) Standards. In essence, eleot measures the extent to which there is observable evidence (or no evidence) that students are engaged in certain activities or demonstrate certain knowledge, attitudes and/or dispositions in a classroom during a defined period of time as measured on a four point scale (1 being “not observed;” 4 being “very evident”).The environments examined are:

• High Expectations
• Equitable Learning
• Supportive Learning
• Active Learning
• Progress Monitoring and Feedback
• Well-Managed Learning
• Digital Learning

Trained observers spend at least 20 minutes in all or nearly every classroom in the school and record their observations on a standardized reporting template. Data are then uploaded and stored by AdvancED.

The eleot provides structured and quantifiable data on the extent to which learners are engaged in activities and/or demonstrate knowledge, attitudes and/or dispositions that are conducive to effective learning. The tool provides an aggregate picture for an entire school, but could potentially be used for grade level and/or in content-specific ways (e.g., to examine the overall performance of sixth grade math teachers) as opposed to providing ratings of individual teachers. This aspect of eleot, as well as its focuson students’ experiences instead of the teachers’ performance, differentiates it from other widely used measures of classroom practice.

The eleot provides structured and quantifiable data on the extent to which learners are engaged in activities and/or demonstrate knowledge, attitudes and/or dispositions that are conducive to effective learning.

The eleot has demonstrated strong psychometric qualities. The overall reliability of the measure is .94 using Cronbach’s Alpha, which is considered a very strong level of reliability. In addition, confirmatory factor analysis of the measure revealed the root mean square error of approximation (RMSEA) as .068, which also is very good in social science research. The RMSEA is a measure of how well the theoretical model structure matches the actual structure from the data.

Summary Results

Across 45,272 classrooms observed, the average overall eleot score was 2.79, meaning that, on average, observers saw some evidence of each of the environments measured. Looking individually at each of the seven environments, inidividual ratings were:

• Well-Managed Learning Environment (average of 3.11)
• Supportive Learning Environment (3.05)
• Active Learning Environment (2.95)
• High Expectations Environment (2.81)
• Progress Monitoring and Feedback Environment (2.76)
• Equitable Learning Environment (2.66)
• Digital Learning Environment (1.88)

Digging deeper into the data, AdvancED examined whether there were differences based on the subject being taught. The table below shows the results from that analysis. The table shows a fair level of consistency across subject areas. In all subject areas, the consistently lowest-rated environment was the Digital Learning Environment, indicating that technology integration remains low in a large number of classrooms. At the same time, teachers seem to be fairly consistent in their use of effective strategies across all environments outside of the Digital Learning Environment, with aspects of the Well-Managed Learning Environment being the most observed across all subjects except for Special Education.

Examining scores across the environments based on grade level, a similar pattern emerges, whereby the Well-Managed Learning Environment is consistently the most observed environment. Interestingly, scores across all environments are highest for grades K-5 while lowest for grades 9-12 except for the Digital Learning Environment, where grades 9-12 have the highest average scores. The differences across all environments between grades K-5 and grades 9-12 are statistically significant.

When looking at individual items, the three highest-rated items, all of which are part of the Well-Managed Learning Environment, are as follows:

• Speaks and interacts respectfully with teacher(s) and peers: 3.42
• Follows classroom rules and works well with others: 3.32
• Knows classroom routines, behavioral expectations and consequences: 3.26

The three lowest-rated items were as follows:

• Uses digital tools/technology to conduct research, solve problems and/or create original works for learning: 1.79
• Uses digital tools/technology to communicate and work collaboratively for learning: 1.82
• Has ongoing opportunities to learn about their own and others’ backgrounds/cultures/differences: 1.96

Additional examination of the items shows that one item in particular, “Has ongoing opportunities to learn about their own and others’ backgrounds/cultures/differences” tends to be confusing for respondents. Trainers for eleot report that they have updated the training modules to highlight this item to address confusion about the item. The AdvancED research team will revisit this issue in future data analyses to see if additional training has improved the performance of this item (as well as all of the other eleot items) or whether it will need to be re-written in future iterations of the eleot.

In summary, analyses of the eleot confirm the reliability and validity of the measure’s ability to accurately reflect classroom practices across a school on a given day. The result of extended psychometric review reveals that the performance of eleot is robust across multiple subjects and grades, as well as extremely stable across multiple environments. In the future, the AdvancED research team will examine the relationship of eleot scores to other outcomes of interest including student academic, social/emotional and behavior outcomes as well as teacher professional development outcomes.