Meeting the Promise of Continuous Improvement

The term continuous improvement has been part of the lexicon of school improvement for decades. From state accountability systems and district and school improvement plans to teacher and classroom protocols, continuous improvement practices have been replicated at different levels of scale throughout our educational system. Yet all evidence suggests that this universally recognized practice has failed to fulfill its promise. That is particularly true in high-poverty schools, where the ZIP code remains as strong a predictor of student success as it was a half century ago, before school improvement gained prominence.

This whitepaper examines some of the reasons why, despite the common use of continuous improvement language and practices, school and system efforts often fall short. It then describes the key components of successful continuous improvement implementation in a school setting and introduces the AdvancED^® Continuous Improvement System, including its research-based elements and processes. And, as described in the summary of findings below, it shows how AdvancED’s work conducting external engagement reviews and observations of more than 250,000 classrooms demonstrates strong relationships between effective continuous improvement practices and high performance.

Summary of Findings

AdvancED provides improvement and accreditation services to more than 34,000 schools and school systems across the United States and in 70 countries. As such, it is required to observe and analyze at least 5,000 institutions each year. Conducted by highly trained external engagement review teams, these school-based analyses are summarized in our Index of Education Quality^® (IEQ^®). The IEQ correlates AdvancED’s seven essential School Quality Factors with overall school quality by measuring the impact of teaching and learning, leadership capacity, and the use of resources to support student learning. 

Data compiled from the ratings of schools that underwent these accreditation reviews in the 2015-16 school year— and from ongoing classroom observations in schools in the AdvancED network— provide important insights into how continuous improvement practices can lead to more effective schools. The following findings show how continuous improvement correlates to the AdvancED School Quality Factors in high-performing schools: 

• Clear direction. Engaging all stakeholders in common goals is a hallmark of effective continuous improvement. Among the schools rated by AdvancED in 2015-16, the lowest-performing (as identified by their overall IEQ scores) demonstrated little agreement among faculty and staff members that schools are focused on student success. Conversely, the highest-performing schools had unusually strong agreement— more than 4.5 on a 5-point scale—that student success was a clear priority.
• Resource management. AdvancED found high correlation between school quality and three key areas of resource management and school quality: (1) sufficient instructional time and resources to support goals and priorities, (2) sufficient resources and materials to meet school needs and (3) availability of a variety of information resources to support student learning. High- and low-performing schools saw differences of 35 to 41 percentage points in these measures of resource management in our research.
• Healthy culture. An environment in which all members of the school community—students and adults alike—are actively engaged, feel empowered to effect positive change, and enjoy congenial and supportive relationships is vital for success. Schools that received low “culture” ratings had significantly lower measures of overall school performance on the IEQ than those that fostered a healthy culture (scores of 262 vs. 297 on a 400-point scale).
• Implementation capacity. Monitoring implementation is a vital part of continuous improvement efforts. Data from faculty surveys administered as part of the accreditation process found that schools that appear to struggle in this area had substantially lower overall school quality ratings than those where leaders excelled in monitoring continuous improvement data (scores of 261 vs. 297, respectively).
• Efficacy of engagement. Effective continuous improvement efforts engage stakeholders both inside and outside of the building, including parents and other community members. During the 2014-15 school year, high-performing schools (with an IEQ rating of 300 or greater) more frequently received high scores in parent surveys of opportunities for parental engagement. The majority of highly rated schools also excelled in engaging parents and other stakeholders in activities such as field trips and career days and in reporting student progress to parents.
• Student engagement. Focusing excessively on adult behaviors has often helped undermine traditional continuous improvement efforts. AdvancED data show a positive relationship between vibrant learner engagement (as gauged by classroom observation) and overall school quality as measured by the IEQ. The more opportunities students have to be owners of their learning, collaborate with other students, and participate in activities that require movement, voice, and high-order thought, the higher the school’s overall rating tended to be.
• High expectations. The belief that all learners have the potential to achieve is a key factor in driving success and can have a significant impact on overall school performance. AdvancED certifies schools that meet high standards for STEM education based on indicators that are somewhat different from AdvancED’s regular accreditation standards. This group of schools allows for additional investigation about how continuous improvement leads to high performance. When comparing student perceptions of high expectations among AdvancED’s network of STEM Certified schools and the non-STEM schools in our network, we found the STEM Certified schools ranked significantly higher in setting an environment of high expectations (3.1 vs. 2.7 on a 4-point classroom observation scale). The STEM Certified schools also significantly outmatched their non-STEM counterparts in other areas measured by AdvancED’s classroom observation tool, including indicators of access to technology, student engagement in rigorous coursework and student collaboration. 
• Impact of instruction. On average, schools that exhibited higher levels of student collaboration during instructional time tended to score in the highest quartiles of overall school quality. During the 2015-2016 school year, two-thirds of schools in the top quartile of student collaboration as measured by classroom observations also were in the top quartile of overall IEQ results. 

Room for Improvement

Even the highest-performing schools struggle in certain— and important— areas. Among our 2015-16 IEQ results: 

• One out of five high-quality schools had difficulty consistently establishing high expectations for all students. 
• One-quarter of these schools also struggled to create classroom opportunities for students to take risks in learning. 
• Nearly 30 percent of high-achieving schools had classrooms that, on average, ranked in the bottom half of all classrooms across the network in terms of requiring students to ask and respond to questions requiring higher-order thinking, such as applying, evaluating and synthesizing information.
• More than one-quarter of high-quality schools had difficulty providing students with opportunities to respond to questions about their individual progress or learning.
• Nearly 30 percent of high-achieving schools included classrooms that, on average, struggled with providing students opportunities to review or improve work based on feedback from teachers. The same was true when considering whether students were “provided additional/ alternative instruction and feedback at the appropriate level of challenge for her/his needs.”

This paper outlines the history of continuous improvement as a discipline, the core principles of the underlying theory, challenges schools face in implementing it, and the elements of an effective continuous improvement system. Most notably, it identifies the correlations between key principles and high-performing schools, and where schools fall short. The document includes a detailed description of each AdvancED School Quality Factor and the supporting research, the components of the AdvancED Continuous Improvement System, and recommendations for administrators, teachers, parents, students, governing authorities and other policymakers to make effective continuous improvement a reality.

Introduction

While continuous improvement practices have been replicated at different levels of scale throughout our educational system, for the most part, documentable large-scale improvement has been elusive. Though achievement in the United States on standardized tests and other measures has risen for a number of years, the 2015 results from the National Assessment of Education Progress (NAEP) showed a decline in average math scores for the first time since 1990. And, when compared to other countries through cross national tests like the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), the United States has shown, at best, mediocre results.

In the United States, ZIP codes, family and community resources, English language proficiency, and race and ethnicity all play a role in learners’ chances of having successful educational experiences that prepare them for the future. In their article in the February 2016 issue of The Atlantic (The Concentration of Poverty in American Schools), reporters Janie Boschma and Ron Brownstein draw attention to an in-depth study of testing results in all 12,000 of the nation’s school districts. In that study, Stanford University professor Sean Reardon found that “ … school poverty turns out to be a good proxy for the quality of a school.”

The article notes that “the cumulative effect of these disadvantages has proven overwhelming almost everywhere … And while they have not finished sorting all of the data, the preliminary results underscore how difficult it is for schools alone to overcome the interlocking challenges created by the economic segregation of low-income students.”

Little has changed, then, since 1966, when the then United States Office of Education published Equality of Educational Opportunity (Coleman et al.), concluding that family background, not school, was the determining factor that predicted student achievement. Coleman and other social scientists from the 1960s to the 1970s believed that poverty and parents’ lack of education prevented children from learning, no matter the quality of instruction they received.

We reject this reasoning today, believing instead that high quality instruction can fundamentally affect all learners’ future success in important ways. But we also know that spending inordinate amounts of time developing a voluminous school improvement plan often yields no improvement. These compliance-centered activities are characterized by checklists and assurances as well as long lists of goals, objectives, strategies and activities. Even beyond such plans, most institutions also engage in some iteration of an improvement process for the purpose of increasing student achievement. They modify curricula and apply new instructional strategies to meet criteria established by analysis of limited data. However, they often are not able to identify or address the issues underlying their struggles.

Most educational institutions approach improvement from a compliance perspective or an adult-centric perspective. That is, strategies center on what teachers and leaders must “do.” Unfortunately, according to many scholars, compliance-based efforts generate results at the end of the improvement cycle (typically a school year) that usually are not sustainable the subsequent year; moreover, those results fall short of the potential impact of a continuous improvement approach (Derrick-Mills, 2015). This happens repeatedly in countless schools with well-written improvement plans for two reasons. First, the focus is specific targets, not the needs of the individual learners. Second, educational institutions emphasize the end result, not the process.

Continuous improvement in itself has been around for decades. Missing has been an appreciation of the value of understanding the needs of individual students and committing to improving the educational experience for everyone.

We have worked side by side with thousands of schools during their continuous improvement journeys, observed students in over 250,000 classrooms, and reviewed thousands of improvement plans. We have seen firsthand that schools prioritizing “learners first” and embracing the improvement process as continual can make a lasting difference in the lives of their students.

There is no one-size-fits-all solution for school improvement. Our research offers insights on what successful schools do well— and where all schools, even the highest-performing ones, can improve.

Part I: What Is Continuous Improvement?

According to the Carnegie Foundation for the Advancement of Teaching (2013), quality improvement is the “disciplined use of evidence-based quantitative and qualitative methods to improve the effectiveness, efficiency, equity, timeliness or safety or service delivery processes or outcomes for ‘users’ or customers of the system”. Viewing such improvements as “continuous” acknowledges the importance of constant and dedicated attention and action on the part of an institution’s stakeholders.

Continuous improvement is more than the development of a plan (Elgart 2016). Embedded systemic continuous improvement is based on multiple sources of evidence and focuses on what is happening across the many functions of schools and systems (see chart below). Institutions leverage the voices of all stakeholders, including students, inside and outside the building to achieve positive and sustainable results.

AdvancED has developed a definition rooted in our research: Continuous improvement is an embedded behavior within the culture of a school that constantly focuses on the conditions, processes, and practices that will improve teaching and learning.”

Origins of Continuous Improvement

Continuous improvement is grounded in systems thinking particularly related to social organizations, including educational institutions. The theory dates back to the late 1930s and has been applied to many professions, organizations and industries to solve problems from a holistic perspective (von Bertalanffy, 1968). “The whole is often greater than the sum of its parts,” a well-known principle catchphrase, derives from the theory. Theorists categorize schools as living systems; numerous elements inside and outside of the institution that are always in movement and are connected. Therefore, how educational institutions embrace and make sense of these elements can determine their level of success in reaching goals and sustaining progress. Living systems function in a competent or an incompetent manner— the choice belongs to its leaders and stakeholders. Nurturing the four characteristics below is essential to continuous improvement in a school:

General systems theory spurred research into and development of other theories and practices for the purpose of improving organizations. For example, Total Quality Management, or TQM (Deming, 1982), focused on businesses. A combination of TQM theories and continuous improvement methods provides a practical “if/then” logic that works well for schools. “If” leaders emphasize student experience, stakeholder engagement, and data collection and analysis, “then” the school becomes a dynamic institution that ceaselessly evolves.

The continuous improvement process requires acceptance of relatively slow yet sustainable progress, and frequently evaluating the relationships among stakeholders and the energy level of the institution as a whole. The problem is that schools often feel pressured by annual performance goals and react by unsystematically changing curricula or programs in the attempt to reach short-term targets. But before explaining at greater length, we must examine in detail the elements of an effective continuous improvement system.

Elements of an Effective Continuous Improvement System

Quality Factors 

Through a review of the research literature on effective schools (Edmonds, 1982; Levine and Lezotte, 1990; Witte & Walsh, 1990) and from our experience and continuous physical presence in schools, we have learned the conditions that highly effective institutions monitor, manage and build on to achieve and sustain effectiveness.

From the inside of struggling urban public schools to STEM-focused charters to parochial high schools, our experience provides us wide and deep insights into the inner workings of teaching and learning. Over 34,000 public and non-public institutions constitute the AdvancED Improvement Network; we observe and analyze at least 5,000 every year. These professional review engagements offer a rich and compelling view on how institutions work and what it takes for them to achieve their goals. From these experiences and our review of research literature, we have established an Index of Education Quality (IEQ) , which correlates seven factors with overall school quality, factors essential to the drive for improvement.

The AdvancED School Quality Factors are:

• Clear Direction
• Healthy Culture
• High Expectations
• Impact of Instruction
• Resource Management
• Efficacy of Engagement
• Implementation Capacity

Following is a brief overview of and the research supporting each factor.

Clear Direction— the capacity to define and clearly communicate to stakeholders the direction, mission, and goals that the institution is committed to achieving .

Research base: The Wallace Foundation (2010) identified the importance of clear direction:

A critical aspect of leadership is helping a group to develop shared understandings about the organization and its activities and goals that can undergird a sense of purpose or vision (Hallinger and Heck, 2002). The most fundamental theoretical explanations for the importance of leaders’ direction-setting practices are goal-based theories of human motivation (e.g., Bandura, 1986; Ford, 1992; Locke, Latham and Eraz, 1988). According to such theory, people are motivated by goals which they find personally compelling, as well as challenging but achievable. Having such goals helps people make sense of their work and enables them to find a sense of identity for themselves within their work context.

It turns out that leadership not only matters; it is second only to teaching among school-related factors in its impact on student learning (Leithwood, 2004). Effective schools and systems are able to articulate a clear vision and a compelling mission that are shared by all stakeholders (Kirk & Jones, 2004, and Sammons et al., 1995).

Role in continuous improvement: As seen in Figure 1 below, the lowest-performing schools (with an overall IEQ score of 250 or lower) staff and faculty agreed to a disappointing extent that their schools were focused on student success. Conversely, schools with higher IEQ averages demonstrated significantly higher agreement— more than 4.5 on a 5-point scale— that student success was a clear priority.

Healthy Culture— the shared values, beliefs, written and unwritten rules, assumptions and behavior of stakeholders within the school community that shape the school’s social norms and create opportunities for everyone to be successful.

Research Base: A healthy school culture fosters an environment in which the school community at all levels is actively engaged, feels empowered to effect positive change, enjoys congenial and supportive relationships, and is supported and mentored for success. A number of studies (Brucato, Melton-Shutt, Cunningham) have confirmed the relationship between a healthy school culture and students’ academic achievement. School culture has been defined as a historically transmitted cognitive framework of shared assumptions, values, norms, and actions— stable, long-term beliefs and practices about what organization members think is important.

We often discuss climate and culture together. Hoy et al. (2003) asserted:

A healthy school climate is imbued with positive student, teacher and administrator interrelationships. Teachers like their colleagues, their school, their job and their students, and they are driven by a quest for academic excellence. They believe in themselves and their students and set high but achievable goals. Students work hard and respect others who do well academically. Principal behavior is also positive; that is friendly and supportive. Principals have high expectations for teachers and go out of their way to help teachers. Healthy schools have good relationships with the community.

The role of culture on school quality has been confirmed by the research literature. As reported by Wagner (2006), research conducted by Melton-Shutt (2002) found significant relationships between scores on a school culture survey and state assessment scores in 66 Kentucky elementary schools: “In every case, the higher the score on the survey, the higher the state assessment score, and the lower the survey score, the lower the state assessment score.” A study of 61 schools in Florida provided similar results to Melton-Shutt’s findings (Cunningham, 2003): “The higher the score on the survey, the higher students scored on Florida’s Comprehensive Assessment Test in reading. The lower the survey score, the lower the reading scores.”

Role in continuous improvement: A healthy culture sets the tone for excellence throughout an institution. As shown in Table 1 below, AdvancED found that schools that received low culture quality ratings as part of the external review process in 2015-16 had significantly lower measures of overall school performance on the IEQ than those which fostered a healthy school culture (scores of 262 vs. 297, on a 400-point scale).

High Expectations— an institution’s stated commitment to expectations for all stakeholders, including student learning outcomes, teacher quality, leadership effectiveness, community engagement and parent involvement

Research Base: The Pygmalion, or Rosenthal, effect, whereby higher expectations lead to an increase in performance and achievement, has been closely investigated. Studies by Rosenthal and Jacobson (1992) established that “ … expectations affect teachers’ moment-to-moment interactions with the children they teach in a thousand almost invisible ways. Teachers give the students that they expect to succeed more time to answer questions, more specific feedback, and more approval: They consistently touch, nod and smile at those kids more.”

Another study, by Schilling and Schilling (1999), reported on the positive and potentially negative effects expectations have on performance: “The literature on motivation and school performance in younger school children suggests that expectations shape the learning experience very powerfully … [C]lassic studies in the psychology literature have found that merely stating an expectation results in enhanced performance, that higher expectations result in higher performance, and that persons with high expectations perform at a higher level than those with low expectations, even though their measured abilities are equal.”

Role in continuous improvement: A culture of high expectations sets high standards for teaching and learning. In such schools, leaders monitor learners’ growth and achievement using rigorous measures of success; provide equitable and challenging learning opportunities that require an individual’s best; engage learners in innovative and creative problem-solving using higher-order thinking skills and in reality-based experiential activities; and consistently implement a rigorous, relevant, and interesting curriculum. Adults as well as students are expected to bring their best and are supported in doing so.

Research suggests that the ascending STEM educational model elevates those expectations. When comparing student perceptions of high expectations between AdvancED’s network of STEM Certified schools and the non-STEM schools in our network, we found the AdvancED STEM Certified schools ranked significantly higher in creating an environment of high expectations (3.1 vs. 2.7 on a four-point scale, as measured by the AdvancED’s eProve™ Effective Learning Environments Observation Tool^® (eleot^®). As seen in Figure 2 below, the STEM Certified schools also significantly outmatched their non-STEM counterparts in other areas measured by eleot, including indicators of access to technology, student engagement in rigorous coursework and student collaboration. In each instance, the difference between averages represents at least 10 percent of the total possible range of eleot item scoring (1 to 4 scale).

Impact of Instruction— the capacity of every teacher to purposefully and intentionally create an environment that empowers all learners to be successful and reach expected levels of achievement including readiness to transition to the next level of learning or career pathway.

Research Base: Myriad studies have shown the effect of teacher quality on learner performance and outcomes. Both controversial and groundbreaking, Sanders (1996) investigated the cumulative effect on learners of multiple years with effective, high-performing teachers versus the detrimental effects of having low-performing teachers:

Over a multi-year period, Sanders focused on what happened to students whose teachers produced high achievement versus those whose teachers produced low achievement. He discovered that when children, beginning in 3^rd grade, were placed with three high-performing teachers in a row, they scored on average at the 96^th percentile on Tennessee’s statewide mathematics assessment at the end of 5^th grade. When children with comparable achievement histories, starting in 3^rd grade, were placed with three low-performing teachers in a row, their average score on the same mathematics assessment was at the 44^th percentile, an enormous 52 percentile-point difference for children who presumably had comparable abilities and skills.

Researchers have even been able to quantify the average effects on learning of specific instructional strategies. Marzano et al. (2001) conducted a meta-analysis that identified a number of strategies that, when effectively implemented, resulted in percentile gains of 29-45 points in student achievement. Imagine if all teachers consistently executed high-quality implementation of such instructional strategies. An average student at the 50^th percentile could reach the upper 70^th percentile or even exceed the 90^th percentile.

The explicit instruction in and development of noncognitive factors in learning has also shown an impact on learner performance. According to Nagaoka et al. (2013), researchers at the Consortium on Chicago School Research developed a framework to organize these noncognitive factors into five categories: academic behaviors, academic perseverance, social skills, learning strategies, and academic mindsets. The researchers found that paying attention to these factors is particularly important for improving the achievement of minority and other underserved learners.

Role in continuous improvement: Instruction has a significant impact on these noncongitive skills, positive beliefs, and attitudes— which, in turn, affect learner performance. To that end, AdvancED collects data that focus on how instruction occurs rather than the achievement assessment outcomes. On average, schools that exhibited higher levels of student collaboration during instructional time also tended to score in the highest quartiles of overall school quality. During the 2015- 16 school year, two-thirds of schools in the top quartile of student collaboration (as measured by classroom observations using eleot) also were in the top quartile of overall IEQ results, as seen in Table 2.

Resource Management— the ability of an institution to plan, secure and allocate its resources (human, material and fiscal) to meet the needs of every learner.

Research Base: Miles and Frank (2008) emphasize the purposeful use of resources to support student learning. Based on over two decades of research on the use of resources in thousands of schools, they found that good stewardship of public resources and carefully considered strategies for their allocation were foundational to high-performing schools across a variety of educational settings. Several years later, in outlining the fundamental principles and process of Strategic School Design, Miles and Ferris (2015) found that high-performing schools respond to continuously changing contexts in education by using people, time, technology and money in ways significantly different from the status quo, including three basic principles of strategic resource use:

(1) Excellent teaching for all students: Organize teachers and teams to maximize student learning and continuously nurture talent.

(2) Personalized learning and support: Match grouping, learning time, technology and program to students’ individual needs.

(3) Cost effectiveness through creative solutions: Organize jobs, partnerships and technology to maximize resources that support teaching and learning. High-performing schools and systems develop strategies to organize and allocate their resources.

Much like Miles et al., Levacic (2010) asserts that the primary goal of resource allocation is to maximize student learning within given resource constraints. She draws on current research to support the claim that educational leaders in this era of accountability need to: (1) develop and rely on their own and their staff’s professional judgments, and do so based on evidence, (2) evaluate, to the extent possible, the consequences for student learning of their schools’ decisions about resource allocation, and (3) thereby develop, for their own practice, an evidence base derived from the outcomes of particular uses of resources. Levacic’s central message to education leaders is that school budgeting needs to be fully integrated with teaching and learning.

Role in continuous improvement: All schools face the challenge of limited resources, requiring prioritization instead of adopting one new improvement plan after another. As shown in Table 3 below, AdvancED found high correlation among faculty and staff perceptions of three areas of resource management and overall school quality as measured by the IEQ: (1) instructional time and resources to support goals and priorities (2) sufficient resources and materials to meet school needs and (3) a variety of information resources to support student learning. High- and low-performing schools saw differences of 35 to 41 percentage points in these measures of resource management in our research.

Efficacy of Engagement— the capacity to engage learners and other stakeholders in an effective manner to improve learning outcomes.

Research Base: Stakeholder engagement— among students as well as their parents and others in the learning community— is a critical component of high-performing schools. Multiple studies have shown a strong correlation between learner engagement and achievement. Klem and Connell (2004) found that student engagement robustly predicts student achievement and behavior in school, regardless of socioeconomic status, while students with low levels of engagement are at risk for a variety of long-term adverse consequences, including disruptive behavior in class, absenteeism and dropping out. Finn and Rock (1997) analyzed data for low-income minority students in grades 8 through 12 in the National Educational Longitudinal Study of 1988, finding that students who displayed engagement as measured by coming to class on time, being prepared for and participating in class work, and making the effort to complete assignments were more likely to be academically successful, have passing grades throughout high school, and graduate on time.

Engagement is vital not just for students. In 2002, the Southwest Educational Development Laboratory (SEDL) published A New Wave of Evidence, a report that synthesized research from 51 studies over the previous decade that investigated the relationship between parent involvement and student learning. While many of the studies were correlational or case studies, the meta-analysis yielded positive findings and offered important insights. The study found that students with involved parents, no matter their income or background, were more likely to:

• Earn higher grades and test scores, and enroll in higher-level programs
• Be promoted, pass their classes, and earn credits
• Attend school regularly
• Have better social skills, show improved behavior, and adapt well to school
• Graduate and go on to post-secondary education

SEDL found that “programs and interventions that engage families in supporting their children’s learning at home are linked to higher student achievement.” Other forms of parent involvement, such as volunteering or attending school events, appeared to have less direct effect on student achievement, particularly in high school.

Role in continuous improvement: As stated earier, understanding whether students are engaged in learning— and how they are learning— is critical to improving outcomes. In AdvancED STEM Certified schools, which emphasize a rigorous, student-centered approach to teaching, eleot classroom observation data shows a positive relationship between measures of learning environments that capture learner engagement and overall school quality as measured by the IEQ. The more opportunities students have to be owners of their learning, collaborate with other students, and engage in activities that require movement, voice, and high-order thought, the higher the school’s overall rating tended to be, as shown in Figure 3 below.

It is critical to engage not only students but the entire stakeholder community in purposeful and meaningful efforts and activities that drive participation, and to gather their perceptions, experiences, and opinions for use in changing processes and outcomes. During the 2014-15 school year, high-performing schools (with an IEQ rating of 300 or greater) more frequently received high scores in parent surveys that identified ways in which schools provided parents opportunities to be engaged; this data is explored in greater detail in our discussion of stakeholder engagement throughout the continuous improvement process in Part II.

Implementation Capacity— the ability of an institution to consistently execute actions designed to improve organizational and instructional effectiveness.

Research Base: A substantial body of research investigates implementation, including fidelity to the model, quality of implementation, and implementation monitoring. The elements of implementation fidelity, according to Carroll et al. (2007), are (1) adherence to the model, encompassing content, coverage, frequency, and duration (2) moderators such as intervention complexity, facilitation strategies, quality of delivery, and participant responsiveness, and (3) identification of the essential components of the model. In school and system settings, it is not infrequent that, as new interventions are planned for and put in place, little attention is paid to implementation fidelity.

Mihalic et al. (2004) summarize the danger of poor implementation: “Discovering what works does not solve the problem of program effectiveness. Once models and best practices are identified, practitioners are faced with the challenge of implementing programs properly. A poorly implemented [though well-designed] program can lead to failure as easily as a poorly designed one.”

Role in continuous improvement: Given that schools often plan well but fall short during implementation, progress monitoring is a vital part of continuous improvement efforts. Results from faculty surveys administered as part of the AdvancED accreditation process indicated that schools that appeared to struggle in this area had substantially lower overall school quality ratings than those whose leaders excelled in monitoring (scores of 261 vs. 297 on the IEQ’s 400-point scale), as shown in Table 4 below.

In the section that follows, we step back from the individual factors of school quality and look more broadly at the process of continuous improvement, as well as the challenges schools face in making it a reality.

Part II: The Challenges of Change and the AdvancED Continuous Improvement System

A multitude of improvement models are available to institutions. Educational institutions, like the students they serve, are unique and have different needs. Some schools have long embraced the elements of continuous improvement and experienced sustainable success and continued growth in identified areas. Others are newcomers to the continuous improvement journey— perhaps they are in their first years of operation and more focused on establishing practices and procedures to manage day-to-day operations.

All schools— high-performing, low-performing, and everywhere in between— has one thing in common. All need to embrace change and commit to continuous improvement practices, even when overall school quality indicators point to a successful learning environment.

Even the highest-performing schools struggle in important areas related to student learning. In 2015-16, we found the following among the schools that received the highest overall quality rankings on the AdvancED IEQ; these findings will be explained in more detail later in this report:

• One out of five high-quality schools had difficulty consistently establishing high expectations for all students.
• One-quarter of these schools also struggled to create classroom opportunities for students to take risks in learning.
• Nearly 30 percent of high-achieving schools had classrooms that, on average, ranked in the bottom half of all classrooms across the network in terms of requiring students to ask and respond to questions demanding higher-order thinking, such as applying, evaluating and synthesizing information.
• More than a quarter of high-quality schools had difficulty providing students with opportunities to respond to questions about their individual progress or learning.
• Nearly 30 percent of high-achieving schools included classrooms that, on average, struggled with giving students opportunities to review or improve work based on feedback from the teachers. The same was true when considering whether students were “provided additional/alternative instruction and feedback at the appropriate level of challenge for her/his needs.”

Regardless of where they are in their evolution, all educational institutions are dynamic, living systems in which some form of change is inevitable. What is essential is that the school has an accurate diagnosis and acceptance of its current reality, so that it knows which phase of continuous improvement it needs to enter or focus on. This reality check occurs when an institution collects and analyzes a variety of data from diagnostics, such as inventories, surveys, and observation tools that provide information and quantifiable data to inform the development of the institution’s improvement plan.

The AdvancED Continuous Improvement System

The AdvancED Continuous Improvement System encompasses phases or cycles similar to those in other widely used models, but distinguishes itself through a significant emphasis on learners’ experiences, stakeholders’ voices, and stakeholders’ engagement in each of three interrelated and interconnected phases to guide an educational institution’s improvement journey: Learn and Share, Examine and Plan, and Act and Evaluate.

In keeping with the key principle of systems theory, each element in the continuous improvement system functions independently, but its results affect the other elements so the “whole is greater than the sum of its parts.” No one element can exist without connection to and the support of the entire system. Our system is designed so each element has unique actions and diagnostics that can be used apart from or in conjunction with the other elements. Together, they contribute to the overall continuous improvement journey; the data from their implementation provide schools a rich and comprehensive real-time understanding of its learners, environments and stakeholders.

In the pages that follow, we examine these three phases and share insights from the AdvancED network that illustrate the challenges and opportunities continuous improvement practices hold for all schools.

Learn and Share: The Importance of Authentic Stakeholder Engagement

The first element of the AdvancED Continuous Improvement System and a starting point for many schools, Learn and Share, calls on members of each institution’s educational ecosystem to share perspectives and experiences and learn how the work of the institution affects every stakeholder, including, most importantly, students. Central to success in the Learn and Share element is the inclusion of all stakeholders, as each unique voice makes an important contribution to the institution’s improvement journey.

In a compliance-based model for school improvement, often the stakeholder engagement process focuses on schooling rather than education. For example, members of an institution’s community have been trained to begin by asking “What is the school’s mission and vision?” rather than patiently collecting and analyzing its stakeholders’ perspectives and experiences, especially students’ level of engagement in their learning environments.

A true continuous improvement paradigm, directs an institution to begin by asking, “What is our current reality and how do we know that is the case?”

The first step involves collecting perception and experiential data from stakeholders. We have learned an important lesson: Schools where parents report the highest levels of satisfaction with the sharing of student and school progress are also the schools that tend to rate in the highest quartile of the AdvancED Index of Education Quality. These schools also involve stakeholders in matters related to student learning and the institution’s purpose and direction.

Figure 4 shows that during the 2014-15 school year, schools with an AdvancED IEQ rating of 300 or above more frequently received scores of 4 or above on survey items related to providing parents with opportunities for engagement. In general, this indicates a positive correlation between stakeholder engagement and school quality. We also can conclude that the majority of these highly rated schools excelled in engaging parents and stakeholders in activities such as field trips and career-day-type events and in reporting student progress to parents.

Information on student engagement with learning experiences also is important for schools in discovering their current realities. Figure 5 below shows the positive relationship between eleot classroom observation data that capture different measures of learner engagement and school quality as measured by the IEQ. Based on the same academic year and set of schools visited by external review teams, our data show that the more opportunities students have to be owners of their learning, collaborate with other students, and engage in activities that require movement, voice, and higher thought, the higher the school’s quality rating tended to be.

Collecting data to gauge perceptions and experiences is essential to a school’s efforts to both acquire information and share it with stakeholders. The AdvancED Continuous Improvement System’s surveys, inventories, direct classroom observation using eleot, and other resources are empowering. They help institutions gather results and engage stakeholders to first understand their reality and then to collaborate in dialogue and action to address issues vital to the school and each student’s success.

Examine and Plan: Developing a Framework to Focus Improvement Efforts

The next element of the AdvancED Continuous Improvement System focuses on institutions grasping their circumstances, then choosing outcomes to strive for in executing an improvement plan— rather than jumping directly to plan implementation. Examine and Plan encourages schools to further diagnose themselves in terms of the seven quality factors present in effective schools whitepaper.

The experience of schools in the AdvancED network confirms the importance of ongoing evaluation and diagnostics for schools at all levels of performance. As mentioned in the introduction to this section, data collected from a sample of 819 schools indicated that even high-performing schools struggle in certain areas as shown in Table 6 below.

In similar fashion, nearly 30 percent of high-quality schools had classrooms that, on average, struggled with providing students opportunities to review/improve work based on feedback from teachers, as seen in Table 7 below. The same was true when considering whether students were “provided additional/alternative instruction and feedback at the appropriate level of challenge for her/his needs.”

Surveys, inventories and classroom observations generate data related to the school quality factors. With that in-depth information, the school community can identify and prioritize improvement areas and create a Strategy Map. This comprehensive plan, easily communicated to all stakeholders, is a visual reminder of the school’s chosen direction.

As with the first phase, stakeholder feedback is crucial in developing this plan, but many schools fail to engage beyond the initial feedback stage. Consider these three troubling findings from our work: In a quarter of schools where external engagement teams observed the least improvement planning, surveys indicated that parents nevertheless strongly believed their children’s schools had improvement plans. Conversely, in nearly a third of schools where our teams found the strongest improvement planning process, surveys indicated that parents did not know a plan existed. And in approximately one out of every 20 schools, teachers believed in the existence of an improvement plan for which the external engagement teams found minimal evidence.

The necessity for stakeholder communication begins with the initial examination of all the data a school collects to determine its needs. It extends to identifying the priorities to share with stakeholders and into the final phase.

Act and Evaluate: Executing and Monitoring Progress to Ensure Success

Third in the Continuous Improvement System, Act and Evaluate, is the phase in which data collection, analysis and diagnosis coalesce into proposals for action. From the long-range Strategy Map created in the previous phase, schools extract critical initiatives to address in a one-year span. This happens through (1) the development of an Annual Improvement Plan and (2) the implementation of a more detailed Action Plan.

By dividing long-term goals into short-term stages, the improvement process is more manageable. The long-term goal to increase the graduation rate, for example, would lead to an annual plan comprising strategies such as securing additional training for school counselors, which in turn would lead to a narrow action plan to develop training on a specific topic in line with time and budget constraints. As results emerge and are continuously measured, schools gain a clear picture of progress and success (or a lack thereof). School leaders can regularly evaluate the annual plan and action plan, and revise them as needed to realize the broader goals in the Strategy Map.

However, too many schools struggle with this crucial phase, often developing improvement plans according to the predefined criteria of a template. For example, our external on-site reviews of more than 1,500 schools in one year found that 351 were deficient or needed improvement in two crucial areas (see Figure 6, below), while fewer than one in 15 were exemplary in either practice.

Addressing these common shortcomings at individual schools requires focused effort from school leaders, educators and others involved in the learning community. At the system level, it requires the active participation of an even broader range of stakeholders and policymakers, as we describe in the final section of this whitepaper.

Call to Action

The No Child Left Behind (NCLB) era in American education created a climate and culture that compelled schools to measure success and progress in terms restricted to such metrics as standardized achievement scores on reading and mathematics in the 4^th, 8^th, and 10^th grade. Yet school quality encompasses so much more than such metrics can measure.

It is past time for schools to move from accountability-driven, compliance-based checklists to meaningful processes that guide and drive verifiable improvement. The end of the No Child Left Behind era provides a real opportunity to move beyond narrow approaches to school improvement focused solely on student test results. NCLB’s successor, the Every Student Succeeds Act (ESSA), has offered state education leaders the chance to build new accountability and improvement systems from the ground up. That could give schools more flexibility to improve without eliminating ambitious goals of improving learning outcomes, and thus opportunities, for all students.

Some elements of the law are likely to improve results— chief among them, a redoubled emphasis on multiple measures of student learning and school climate as states develop new accountability systems. The law also emphasizes continuous improvement in designing supports and interventions. However, much work remains to ensure that effective continuous improvement systems become deeply embedded practice in schools across the country.

The continuous improvement journey is neither singular nor solitary. The institution needs its entire community. When everyone is committed to, engaged in, and informed about the process, institutions can tap into the rich and varied resources, experiences and perspectives their stakeholders bring. Ensuring that stakeholders understand and participate in all phases throughout the ever-evolving quest is daunting, but not insurmountable. An abundance of research describes best practices for enlisting enthusiastic participation. A sense of unity, along with periodic moments of triumph, propels an institution to extraordinary long-term accomplishments.

To that end, AdvancED offers these challenges:

1. We call on school, district and state leaders to (a) seek out, hear the voices of, and regularly and energetically involve all stakeholder groups in decision-making, (b) apply a combination of thinking about both the current reality and the future to guide the improvement planning process,(c) focus on faithfully implementing the process, not forcing outcomes to meet compliance-driven targets; and (d) openly share progress, barriers, and successes.
2. We call on educators to (a) be leaders who fully engage with the continuous improvement journey within the context of their position, (b) know and embrace the annual priorities, and (c) diligently collect data and other information to monitor the progress of the process.
3. We call on parents, business leaders, philanthropists, and community members to (a) become informed about the institution’s improvement journey, and know and do what is required of them, (b) participate in meetings to learn about and provide feedback on the progress and obstacles the institution experiences and (c) respond to the institution’s efforts to collect data from surveys, focus and other means.
4. We call on students to (a) become actively engaged in and owners of their learning both inside and outside the school, (b) express their opinions and perspectives about their experiences to the institution’s adults and (c) organize programs, service projects, and student-led conversations to improve the learning environment.
5. Finally, we call on governing authorities, boards and political leaders to (a) provide resources so institutions can build capacity and fully engage in the continuous improvement process, which prioritizes success for all learners; (b) be partners with institutional leadership to absorb data and other information relative to the process and (c) embrace the process, deemphasize the outmoded attachment to compliance-driven targets.