Why Schools "Fail" or What If "Failing Schools" Aren't?

Most discussions of "school reform" focus on the need to close or restructure failing schools or else provide students with choice as a way out of them. Embedded within these discussions are theories or suggestions about why these schools are failing. Often left unexamined is the actual claim that these schools are failing.

Since I have had the opportunity to study many schools in Louisiana throughout my career as a member of the faculty in the College of Education at Louisiana State University, and through the lens of my service as a School Board Member for sixteen years, I will focus on schools in Louisiana. When a school in Louisiana is assigned the letter grade F, it is almost universally accepted in the media and in school reform policy debates that children in this school are receiving a sub-standard education, almost by definition since the school itself is seen as "failing." Yet the second part of the title of this paper, which comes from a chapter in the Late Gerald W. Bracey's 2003 book "On the Death of Childhood and the Destruction of Public Schools," raises an interesting question.

Should we be confident that the letter grade F assigned by the State of Louisiana actually indicates that the quality of teaching in the school is the reason for the failure? If it is not the quality of teaching, then what is it? After all, it is not the building that is responsible for the education of the students within its walls.

Drawing upon research on teaching and learning, and data on student achievement we will examine how failing schools "fail" and ask if they indeed are failing. We will also draw from some of the good work being done in Louisiana to improve the quality of teaching and the quality and effectiveness of our schools.

Given what we have been hearing in the media on a regular basis, it would seem to be a safe hypothesis that poor quality teachers and teaching are the cause of failing schools. The group "Education's Next Horizon" seems to suggest this as a possible cause in a letter introducing their "Education Briefing Book: A Primer for Policymakers on Louisiana PreK-12 Education."

The teacher is the most important factor for school success. "On the job" professional development and improving working conditions of teachers are the keys to school improvement.

We find further support for the role of teaching in failing schools in other places as well. For example, we heard strong comments from Dr. Tabitha Grossman, Senior Policy Analyst at the National Governors Association, when she came and spoke before the Blue Ribbon Commission on Educational Excellence. In her Building a High Quality Workforce: a governor's guide to human capital development she begins on page one with this succinct point:

"Teacher effectiveness is the primary influence on student achievement, followed by principal effectiveness."

She gives as her reference to this, a Wallace Foundation publication, certainly a reputable source in many education and policy circles.  On page three of the guide, Dr. Grossman again emphasizes the important role of the teacher in the academic achievement of students: “Research shows that teacher quality is the primary influence on student achievement.^" For her reference for this statement she uses a major report on teacher quality published by the Organization for Economic Cooperation and Development (OECD).  Again, this gives us additional confidence in her statement.

We can see in these quotes a clear relationship being outlined between "teacher effectiveness" as the "primary influence" on student achievement, and the teacher as the "most important single factor" for student success. Since the letter grade of a school is, for the most part, determined by "student success" as measured by standardized test scores, the reformers seem to be on solid ground in assessing the blame for low scores to inferior teaching. Before we conclude that so many of Louisiana's failing schools are filled with poorly performing teachers, however, we might want to examine other possible factors that might explain the low scores in these schools. 

If teaching is key, we should find evidence that quality teaching can reverse the impact of other factors that might be hindering student achievement in the failing schools. After all, if the earlier quotes represent the best research on student achievement, that research certainly suggests that teacher quality is the dominant factor. To test this, let's see if other factors appear to play a large role in failing schools.

One claim that we often hear from those challenging the reformers is that poverty is the reason for failing schools. If poverty is the cause of failing schools, then we should find that strong patterns exist linking poverty to school performance, even in charter schools and in voucher programs where reformers claim that competition for school success should help to overcome any challenges in educating students from low-income homes.

We will now explore some data to see whether poverty seems to be related to school failure within Louisiana. To accomplish this we used data on the letter grades and school poverty that used to be available from the Louisiana Department of Education's website. An earlier version of this paper had links to all of the data used in the following examples, but unfortunately, State Superintendent of Education John White has had nearly all research data sets removed or modified to ensure that studies such as this cannot now be done. Luckily, this study was completed before the removal of the historical public data. We removed selective enrollment and alternative schools from the list of schools in the state containing letter grades and information on student poverty, since both types of schools obviously start off with students selected by achievement levels. Dr. George Noell, the architect of the state's value-added assessment system, has pointed out that prior test scores, where available, are clearly the most powerful predictors of future achievement, so a school that selects for high-achieving students or is filled with students whose academic achievement has been severely hampered before their arrival in the school would not be giving us legitimate information about the quality of teaching within the school.

If teacher quality is the most important factor in student achievement, then we should find patterns suggesting that high-performing schools such as those with letter grades of A are overcoming the impacts of poverty, while those schools labeled F are not as effective in overcoming the challenges of poverty. We will use the category of students qualifying for free lunch, since that contains students in families with incomes up to 130% of the poverty line, or closest to the federal guidelines for poverty.

The average percentage of students qualifying for free meals in the A schools, is approximately 34%, which is almost the same as the average percentage of students in all schools across the United States who qualify for free meals.  When we look at schools identified as failing schools, we find much higher levels of poverty.  Approximately 89% of the students in schools rated F qualify for free meals. Examining schools labeled B or C or D, we find that they also fit this pattern. We find their respective percentages of low-income students to be 45%, 60%, and 80%. To make it easier to visualize this pattern for the state of Louisiana, we have charted it below.

From this chart it appears that there is a powerful pattern in the relationship between the concentrations of poverty in schools and the assigned letter grades for schools. However, we should note that for four years Louisiana put out a report that highlighted High-Poverty High Performing Schools, which suggested that there are, perhaps, many schools that defy this pattern. After carefully examining the lists, which reported higher numbers of schools each succeeding year, with 56 schools in the 2011 release, we noted that many of the schools actually had a lower percentage of students qualifying for free meals than the state average. In addition, most of the schools were magnet schools or schools where Gifted/Talented programs were masking lower test scores for other groups of students in the schools.  Finally, there were schools like Lake Forest Elementary, in New Orleans, that had extensive application and testing procedures that eliminated low-scoring students from the schools.  We also noted that there were no schools that had been on the list every year. Not one school out of over 1300 schools in the state that had overcome the challenges of poverty every year.

The pattern shown above suggests, in fact, that poverty is playing a very important, and perhaps dominant role in student achievement. Perhaps Louisiana is the only state that has these patterns? However, we note similar patterns on the National Assessment of Educational Progress (NAEP) subject area assessments. For example, the chart below shows 8th grade reading assessments for public and for private schools (including religious-affiliated schools). As can be seen in the chart, family income seems to matter in achievement scores for all schools. The gaps between those students who DO NOT qualify for free or reduced meal prices and those who DO qualify for reduced meal prices are 14 points in the public schools and 18 points in the private schools. The gaps between those who DO NOT qualify for either free or reduced meal prices and those who qualify for FREE meals are 25 points for public schools and 26 points for private schools.

On these NAEP Reading Scores, a 10 to 12 point gain in the scale scores is roughly equal to a year's gain in achievement. (The numbers you see are called "scale scores" and are used to enable researchers to more easily compare achievement across schools and groups and to see differences.) To put this in the simplest terms, students scoring 10 to 12 points higher than other students have achievement levels we would expect from those having an extra year of schooling. This data suggests that students living in families at or near the poverty level are about two years behind their middle class peers. Clearly then, poverty seems to greatly affect student achievement in public and in private schools. We note that although the private schools seem to have an advantage on the scores, they enroll very few special education students, and they get to select their students.

We also explored data in districts across the country that have had charter schools as well as voucher programs and other common school reforms in place for a number of years. We again looked at NAEP scores in order to see what kind of impact these reforms have had on reversing the apparent relationship between poverty and student achievement.

NAEP scores can be useful checks against a natural tendency of states, districts, and schools to focus on teaching to the test, because NAEP assessments are much more difficult to game or teach to than state level tests. An example of this can be found in states where 90 percent or more of students receive passing scores in their state at the basic level, when only 20 or 30 percent of their students are passing NAEP at the Basic Level. We note that the Proficient Level of NAEP is not at all equal to what we might assume is being on grade level, but is instead significantly higher.

Cleveland, Ohio school reformers achieved success at the U.S. Supreme Court in 2002 with a ruling that vouchers could be used in religious schools. Currently, over 5000 students in Cleveland attend schools that accept vouchers and most of these students are attending church-related schools. (Vouchers are named for the payment vouchers used by families to fund their children's schooling at private, parochial or other schools outside of existing public school choice opportunities such as magnet schools. Vouchers should not be confused with scholarships, which are used to reward merit, or scholarship.) In 2003, right after vouchers for religious schools were approved to begin, the average 8th grade NAEP reading assessment scale score in Cleveland was 240 for low-income students. Eight years later the average score for low-income students remained at 240. We should note that Cleveland does not have enough students who do not qualify for either free or reduced meal prices to actually have a published NAEP score for those students.

In New York City, where school reformer Chancellor Joel Klein shook up the school system and radically changed the structure of the nation's largest school system, with charter schools taking over many existing schools, and also in Chicago, where current U.S Secretary of Education Arne Duncan used many of the same reforms, score trends on 8th grade reading were also flat. Average scores for low-income students from each of these two cities in 2011 straddled the national average for low-income students. Students in New York City, after years of reform scored 151, Chicago students scored an average of 149, and the national average for low-income students is 150. Clearly, the evidence suggests that poverty matters, and it matters in a powerful and predictable way. In spite of early claims that teachers in high-profile schools in these cities had reversed the impact of poverty, closer examination nearly always reveals that the impacts of poverty certainly remain.

            To show how concentrations of poverty play out for individual schools, researchers plotted the scores of students at elementary, middle, and high schools in Ascension Parish, Louisiana (In Louisiana, "parishes" are the equivalent of "counties" in other states.). Ascension Parish is useful because the school level data is not affected as much as it is in other districts by charters and magnet schools that distort the data. You can easily imagine how a school that selected mostly high-achieving students who also qualified for free meals would distort patterns of achievement for that school, as well as other schools around them. (Obviously, there are students from low-income families who are also extremely high scoring! It is for this reason, among many others, that low expectations for students are never justified!) For each level, elementary, middle and high, the straight line on each chart represents the statistical fit of the relationship between the percent of students qualifying for free meals and the schools' performance scores. (It is this School Performance Scores, or SPS, that determines the actual letter grade of a school.)

 It is almost frightening that the schools were fitting so closely on the line that represented the correlation or relationship between poverty and school performance scores. There are actually two schools where both the school performance scores and the corresponding percentages of low-income students were so nearly identical that they appear over one another in the chart.

The pattern for middle schools was even stronger than that for elementary schools in Ascension, as illustrated on this next chart.

The powerful relationship between poverty and performance scores in these Middle Schools is almost beyond description. In statistical terms, the correlation between the percentage of students eligible for free meals and the school performance scores for Middle Schools in Ascension was negative .987 which, simply stated, means that as poverty increased, scores went down, in an almost perfectly linear relationship.

For the Ascension Parish High Schools the relationship was as perfect as most of us have ever seen. The incredibly powerful numerical correlation of -.998 once again suggested that the percentage of free lunch eligible students in a school could easily be used to fairly accurately predict the school performance scores of a school.

These charts can be incredibly powerful in helping us visualize the degree to which poverty matters. In school districts where there were schools that had a group of "gifted" or "talented" students, or where there were minimum requirements to get into the school, such as the Baton Rouge Foreign Language Academic Immersion Magnet (Baton Rouge FLAIM), we almost always found these schools to be above the line, meaning they achieved at a higher level than their free lunch percentages would have predicted, for obvious reasons. Conversely, schools that dealt with high concentrations of students who had not been achieving well on standardized tests in the high-stakes world of "reformy" Louisiana, and who often directed their frustration in socially unacceptable ways, were almost always below the line. Schools with high concentrations of students qualifying for special services due to mental, physical, or emotional handicapping conditions almost always lie below the line.

In examining District Performance Scores (DPS) across Louisiana, we notice the same patterns. The DPS scores are, like SPS, mostly influenced by student achievement on standardized tests. Zachary, with the lowest percentage of students qualifying for free meals in the entire state, had the highest DPS. On the other end of the spectrum, the Recovery School District and St. Helena Parish have the two lowest DPS scores, with St. Helena having the highest percentages of students qualifying for free meals.

The strong nature of the relationships we found yet again convinced us that we needed to seriously question the statements we quoted earlier in this paper about the powerful effects of teachers on student academic achievement.  When we went to the original sources of the texts used by Tabitha Grossman, for example, we discovered that her statements were not consistent with the statements in the documents she referenced.

Dr. Grossman wrote: "Teacher effectiveness is the primary influence on student achievement, followed by principal effectiveness." The Wallace Foundation research report she referenced actually said: "Teachers have the most immediate in-school effect on student success." We trust you can note the not so subtle-difference. "In-school" and "immediate" do not necessarily equate to "primary influence." 

What about her other quote: "Research shows that teacher quality is the primary influence on student achievement."  When we went to the OECD report she referenced, this is what we found:

Three broad conclusions emerge from research on student learning. The first and most solidly based finding is that the largest source of variation in student learning is attributable to differences in what students bring to school – their abilities and attitudes, and family and community background. Such factors are difficult for policy makers to influence, at least in the short-run. (Emphasis added)

The second broad conclusion is that of those variables that are potentially open to policy influence, factors to do with teachers and teaching are the most important influences on student learning. In particular, the broad consensus is that “teacher quality” is the single most important school variable influencing student achievement. (Emphasis added.)

The OECD report and the earlier report from the Wallace Foundation suggest that teacher effectiveness is important, but they are clearly not reporting that among all factors impacting student achievement, that teacher impacts are the most important.  In fact, the OECD report is quite explicit on this.

If researchers can use rates of poverty, independent of any knowledge of actual teacher and teaching quality, to predict school performance scores, what does this say about the claims of reformers that it is the teaching in the schools that is deficient?  What it suggests is the need to ask the question in the second part of the title of this paper.  What if failing schools… aren't?

Policy makers in Louisiana, and in other states, have suggested that value-added measures (VAM) be used to explore the actual impact of teachers on student achievement. Supporters often use the following argument to suggest that this complex mathematical procedure is necessary because it would be ridiculous to use the actual test scores of students. "Think about it" they say. "Would anyone want to teach the students who have the most challenges to overcome, and who score the lowest on nearly all standardized tests?" Teachers might only want to teach advanced students, or gifted students.  "Obviously" they say, this would not be "fair." How is it "fair" to not account for these challenges when measuring "school performance?"  How is it "fair" to condemn schools as failing when the accountability system is so flawed that it does not even begin to measure the quality of teaching within the schools?

I believe the data and arguments presented in this paper make clear that many schools are being unfairly stigmatized with the label of failing. More to the point, it suggests, very strongly, that the letter grades themselves seem to have no particular value in predicting the quality of teaching within schools! I recently tweeted that "You can know nothing about the quality of teaching in a school by looking at the school's letter grade." To put it bluntly, the evidence suggests that school performance scores are clearly and powerfully related to the degree of poverty in the schools. I believe that there are likely to be some schools that have excellent teaching faculties, and some schools that do not have such a strong teaching faculty. I also believe the data suggest that neither the School Performance Scores nor the "transparent" Letter Grades are useful in determining which schools have which faculties.

Wonderful schools which are nearly filled to overflowing with students struggling from the negative impacts of poverty do not deserve to be automatically labeled as failing schools by anyone who has not carefully, and honestly, examined the teaching going on within their walls.

Parents, teachers, students and even school reformers perhaps, can fairly make determinations of whether a particular classroom is conducive to learning. It may take multiple observations and interactions with the teacher and the students, but fair and accurate evaluations of the quality of teaching can, and should be made. School reformers often seem to know numbers, but don't always appear to know what those numbers actually represent, or if they represent anything at all. I argue here and now, stridently and with the support of incredibly powerful data, that to label a school as failing based on the scores of students in the school, without taking into account factors that Dr. George Noell, the Wallace Foundation, and the OECD and others have clearly suggested impact student achievement independent of the classroom teachers, is neither ethically nor morally defensible!

Our current accountability system for schools does nothing to account for the effects of poverty on students in our schools, and the challenges it represents to those teaching and learning in these schools. Louisiana's accountability system does not, in any way, account for the incredibly varied abilities students bring with them when they first arrive at our schools. The current system fails to recognize evidence such as Kindergarten inventories that clearly point out the effects poverty has had on children before entering school.  The work of Dr. Noell and others who point out that these scores are powerful predictors of future performance, and would provide evidence of the extremely difficult challenges faced by high-poverty schools, is totally ignored.

Recognizing poverty, and the incredibly powerful effects it has on student achievement in our schools, especially prior to students even entering Kindergarten, would provide increased support for initiatives that have incredibly powerful and proven effects on student performance. Community programs that focus on pediatric care or parenting awareness programs are not as strong as they need to be in a state where children rank last in virtually all early-childhood categories on comparisons across all states. The number of books in the home, and the hours of television watched might become topics of conversation in a community seeking to increase the education quality of the entire community.

The current accountability system labels schools as failing in spite of no evidence that it is the teaching that is failing. The labeling is done on the basis of test scores alone, which have more to do with the incredible poverty faced by the students than with any evidence of poor teaching. When so many high-poverty schools across the state receive the Letter Grade F, it has the effect of eroding public confidence and desperately needed community support for our neediest public schools, yet the grades are assigned by computer, with no one going to the schools to actually see what is happening in the classrooms!  This does a great disservice to teachers, to students, and to parents who more than anything else, simply want to have their children in schools where teachers meet their children where they are, and then teach their children well.

The real choices to be made, then, are perhaps not the "choices" promoted by reformers who often see profits in the condemning of public schools. Instead, a more important and meaningful choice is whether we choose to do something about the failure of our accountability system to fairly and accurately provide parents and communities with useful information about the quality of our public schools.

The important question, for now, is whether we will have the moral and political courage to stop falsely labeling schools as failing, with all of the negative consequences such a label carries, absent real and convincing evidence concerning the quality of teaching within the school. The thousands of dedicated teachers and administrators in our schools deserve better, and the hundreds of thousands of students certainly deserve nothing less!

I used the term "failing schools" in this paper over a dozen times, and chances are you didn't stumble over the term, since we have heard it so many times. I believe that we should banish the use of this term in the media, and in our policy debates, until the reformers can assure the public that they ONLY mean those schools that are actually failing to provide quality instruction for student, and not simply those who face incredible challenges, and in spite of those overwhelming challenges provide excellent opportunities to learn for all children. 

This paper is the first in a series which will begin to offer an alternative vision of the very real choices Louisiana can make as it moves beyond its bicentennial as a state. Will we make the hard choices, that require more than the labeling and blaming of schools for the challenges we face?  I believe we must. I thank you for reading this, and I welcome your comments and your suggestions for ways we can further inform the future of education.

Copyright©2013 by Noel Hammatt. All rights reserved. Permission is hereby granted for the publication of all or portions of this paper by those seeking to share the truth about public schools in Louisiana, and elsewhere, as long as this complete copyright notice is included.