Monday, June 10, 2013

When Do State Education "Rankings" Become Rank?

     When I competed in the NATO Swimming and Diving Competition in Sindelfingen, Germany in the early 1980's I wrote to my future bride about my amazing accomplishments. I was totally honest about my achievements… and totally misleading at the same time:  these characteristics also describe the nature of the constant battle between states over their "rankings" in education and student achievement.

     As a nation we seem to be fascinated with the practice of ranking almost everything! It seems that every local publication produces their rankings of food establishments or coffee houses or hamburgers, for example. Nationally we rank communities on crime, education, "friendliness" and honesty, pollution, "livability" and almost any other measure you can imagine! At the same time, however, it is abundantly clear that such measures often project a measure of precision that is patently absurd.
     For the record, in the 10 Meter Platform Diving event I achieved Third Place, and in the 1 Meter Diving Competition I received Fifth Place. The statements are a bit misleading, however. When asked to compete as part of the US Team in this competition I informed the coach that I really was NOT a good diver, by any measure! He quietly informed me that my ENTRY into the two diving categories would earn points for our team, and that he simply had no other divers. This leads to the misleading part of my accomplishment.  I didn't have to actually do ANYTHING other than dive!  Unless I was disqualified, I would win points for my team because there were only going to be two other contestants in the 10 Meter and 4 other in the 1 Meter classes. I would get points for the team without really doing anything, although diving from the 10 Meter Platform was more challenging that I could have imagined! In fact, I was almost embarrassed to surface after each dive. I placed dead last in both events, but I survived, and the US team won the overall competition.

     What does this story have to do with the very serious educational "competition" between states, and within states as well? In much the same way as I was able to win points for my team by simply taking part in the events, some states are able to win a high position in state "rankings" on the National Assessment of educational Progress (NAEP) by simply…being.  What do I mean by this?  One of the most clearly recognizable patterns we find in testing is that family income seems to account for a lot! States at the top of the rankings on NAEP tend to have more wealthy students, and fewer students qualifying for free meals.  Mississippi and Louisiana share the dubious honor of having the highest percentage of students in their public schools coming from low-income homes.  And both states "bring up the rear" when it comes to the NAEP rankings. Should anyone really be surprised by this relationship?

     Perhaps we should ask whether the respective rankings on NAEP actually represent a measure of the quality of teaching in the states or the quality of the school boards or some other school-related measure. Or perhaps we should ask whether the ranking simply corresponds to other measures of the quality of life such as, well, how about poverty! Louisiana and Mississippi are perennial "bottom feeders" on national measures of the quality of life of children when it comes to early childhood measures before these children ever enter the schoolhouse doors! Why is their no righteous anger when the annual "Kids Count" report is released by the Annie E. Casey Foundation and Louisiana is either in last or second to last place in those factors that we know impact our children's educational attainment and success in later life. As the 2011 report notes on page 24 "Children who are nurtured and well cared for in the first five years have better social-emotional, language, and learning outcomes." Does it take a degree in rocket science to see that how we care for children in their first five years is going to impact academic achievement for our schools?

     We need to keep in mind one thing, and it is very important in understanding another problem with the rankings between states on NAEP. The scores and rankings released annually and debated in the media and in the legislature are based on public schools only. Why is this important? Private and parochial schools are much less likely to admit students with severe physical, emotional, or mental handicapping conditions. They are much less likely to admit large numbers of students coming from the lowest income families. They are much more likely to use academic achievement tests in their admissions policies than they are to use "means testing."  For this reason, as more and more students in a given area are admitted to private and parochial schools, the more likely it is that the public schools in the same area are going to find themselves much lower in achievement-based rankings than other similar areas without private and parochial schools. This finding is true of states as well, of course.

     It is for this reason that the ranking of states in the Kids Count report is even more damning for Louisiana. It reports on factors for all children. Yet those children that arrive in our public schools throughout Louisiana face challenges at a significantly higher rate that the Kids Count report suggests, for the higher percentages of private and parochial schools in Louisiana are skimming off those students who are likely to have more resources than average, not less.

     I admitted the truth about my diving achievements (pun not intended, but valid) to my future wife in a letter just a few days after my initial letter. When are the media and the powerful political "reform" forces in our states going to "fess up" to the truth about the rankings in education? When are they going to recognize and admit the truth?

     The measures such as "Letter Grades" and "School Performance Scores" they are using to "damn" public education in Louisiana or particular school systems, are not legitimate rankings, but are instead a rank, odious and misleading set of measures that are being used to condemn our students, our teachers, and our communities. These same myths of "failure" based on these rankings are leading to a future of failed "status quo" reforms that are based on false premises, and less likely to improve education for all children than almost any measures recommended by groups like the Annie E. Casey Foundation in their 2011 Kids Count report:

A vast body of research shows that high-quality early childhood development programs for           disadvantaged children and their families are one of the most cost-effective investments for reducing the harmful effects of economic hardship. These programs include an array of home visiting and parenting support programs for families with infants and toddlers and comprehensive pre-kindergarten programs for three- and four-year-olds. (p.24)

     Don't expect to find "charter schools" or "value-added" measures in the Annie E. Casey report. They aren't there. What is there in the report is evidence-based findings of researchers drawing from data from around the country that how we measure student achievement, and rank the states, or districts around the country is not really a measure of teaching and learning in schools, but is instead a measure of who shows up at the public schools in each state. 

     Will you stand up and be counted as one who no longer allows the media and the "reformers" to damn your schools, and your state, in the interest of their politics and profit? Get informed! Get involved. Get real! And get ready, for the rank rankings will continue until we call them out for what they are! Dive in!

Saturday, June 8, 2013

Why Schools "Fail" or What If "Failing Schools" Aren't?

Most discussions of "school reform" focus on the need to close or restructure failing schools or else provide students with choice as a way out of them. Embedded within these discussions are theories or suggestions about why these schools are failing. Often left unexamined is the actual claim that these schools are failing.

Since I have had the opportunity to study many schools in Louisiana throughout my career as a member of the faculty in the College of Education at Louisiana State University, and through the lens of my service as a School Board Member for sixteen years, I will focus on schools in Louisiana. When a school in Louisiana is assigned the letter grade F, it is almost universally accepted in the media and in school reform policy debates that children in this school are receiving a sub-standard education, almost by definition since the school itself is seen as "failing." Yet the second part of the title of this paper, which comes from a chapter in the Late Gerald W. Bracey's 2003 book "On the Death of Childhood and the Destruction of Public Schools," raises an interesting question.

Should we be confident that the letter grade F assigned by the State of Louisiana actually indicates that the quality of teaching in the school is the reason for the failure? If it is not the quality of teaching, then what is it? After all, it is not the building that is responsible for the education of the students within its walls.

Drawing upon research on teaching and learning, and data on student achievement we will examine how failing schools "fail" and ask if they indeed are failing. We will also draw from some of the good work being done in Louisiana to improve the quality of teaching and the quality and effectiveness of our schools.

Given what we have been hearing in the media on a regular basis, it would seem to be a safe hypothesis that poor quality teachers and teaching are the cause of failing schools. The group "Education's Next Horizon" seems to suggest this as a possible cause in a letter introducing their "Education Briefing Book: A Primer for Policymakers on Louisiana PreK-12 Education."

The teacher is the most important factor for school success. "On the job" professional development and improving working conditions of teachers are the keys to school improvement.

We find further support for the role of teaching in failing schools in other places as well. For example, we heard strong comments from Dr. Tabitha Grossman, Senior Policy Analyst at the National Governors Association, when she came and spoke before the Blue Ribbon Commission on Educational Excellence. In her Building a High Quality Workforce: a governor's guide to human capital development she begins on page one with this succinct point:

"Teacher effectiveness is the primary influence on student achievement, followed by principal effectiveness."

She gives as her reference to this, a Wallace Foundation publication, certainly a reputable source in many education and policy circles.  On page three of the guide, Dr. Grossman again emphasizes the important role of the teacher in the academic achievement of students: Research shows that teacher quality is the primary influence on student achievement." For her reference for this statement she uses a major report on teacher quality published by the Organization for Economic Cooperation and Development (OECD).  Again, this gives us additional confidence in her statement.

We can see in these quotes a clear relationship being outlined between "teacher effectiveness" as the "primary influence" on student achievement, and the teacher as the "most important single factor" for student success. Since the letter grade of a school is, for the most part, determined by "student success" as measured by standardized test scores, the reformers seem to be on solid ground in assessing the blame for low scores to inferior teaching. Before we conclude that so many of Louisiana's failing schools are filled with poorly performing teachers, however, we might want to examine other possible factors that might explain the low scores in these schools. 

If teaching is key, we should find evidence that quality teaching can reverse the impact of other factors that might be hindering student achievement in the failing schools. After all, if the earlier quotes represent the best research on student achievement, that research certainly suggests that teacher quality is the dominant factor. To test this, let's see if other factors appear to play a large role in failing schools.

One claim that we often hear from those challenging the reformers is that poverty is the reason for failing schools. If poverty is the cause of failing schools, then we should find that strong patterns exist linking poverty to school performance, even in charter schools and in voucher programs where reformers claim that competition for school success should help to overcome any challenges in educating students from low-income homes.

We will now explore some data to see whether poverty seems to be related to school failure within Louisiana. To accomplish this we used data on the letter grades and school poverty that used to be available from the Louisiana Department of Education's website. An earlier version of this paper had links to all of the data used in the following examples, but unfortunately, State Superintendent of Education John White has had nearly all research data sets removed or modified to ensure that studies such as this cannot now be done. Luckily, this study was completed before the removal of the historical public data. We removed selective enrollment and alternative schools from the list of schools in the state containing letter grades and information on student poverty, since both types of schools obviously start off with students selected by achievement levels. Dr. George Noell, the architect of the state's value-added assessment system, has pointed out that prior test scores, where available, are clearly the most powerful predictors of future achievement, so a school that selects for high-achieving students or is filled with students whose academic achievement has been severely hampered before their arrival in the school would not be giving us legitimate information about the quality of teaching within the school.

If teacher quality is the most important factor in student achievement, then we should find patterns suggesting that high-performing schools such as those with letter grades of A are overcoming the impacts of poverty, while those schools labeled F are not as effective in overcoming the challenges of poverty. We will use the category of students qualifying for free lunch, since that contains students in families with incomes up to 130% of the poverty line, or closest to the federal guidelines for poverty.

The average percentage of students qualifying for free meals in the A schools, is approximately 34%, which is almost the same as the average percentage of students in all schools across the United States who qualify for free meals.  When we look at schools identified as failing schools, we find much higher levels of poverty.  Approximately 89% of the students in schools rated F qualify for free meals. Examining schools labeled B or C or D, we find that they also fit this pattern. We find their respective percentages of low-income students to be 45%, 60%, and 80%. To make it easier to visualize this pattern for the state of Louisiana, we have charted it below.

From this chart it appears that there is a powerful pattern in the relationship between the concentrations of poverty in schools and the assigned letter grades for schools. However, we should note that for four years Louisiana put out a report that highlighted High-Poverty High Performing Schools, which suggested that there are, perhaps, many schools that defy this pattern. After carefully examining the lists, which reported higher numbers of schools each succeeding year, with 56 schools in the 2011 release, we noted that many of the schools actually had a lower percentage of students qualifying for free meals than the state average. In addition, most of the schools were magnet schools or schools where Gifted/Talented programs were masking lower test scores for other groups of students in the schools.  Finally, there were schools like Lake Forest Elementary, in New Orleans, that had extensive application and testing procedures that eliminated low-scoring students from the schools.  We also noted that there were no schools that had been on the list every year. Not one school out of over 1300 schools in the state that had overcome the challenges of poverty every year.

The pattern shown above suggests, in fact, that poverty is playing a very important, and perhaps dominant role in student achievement. Perhaps Louisiana is the only state that has these patterns? However, we note similar patterns on the National Assessment of Educational Progress (NAEP) subject area assessments. For example, the chart below shows 8th grade reading assessments for public and for private schools (including religious-affiliated schools). As can be seen in the chart, family income seems to matter in achievement scores for all schools. The gaps between those students who DO NOT qualify for free or reduced meal prices and those who DO qualify for reduced meal prices are 14 points in the public schools and 18 points in the private schools. The gaps between those who DO NOT qualify for either free or reduced meal prices and those who qualify for FREE meals are 25 points for public schools and 26 points for private schools.

On these NAEP Reading Scores, a 10 to 12 point gain in the scale scores is roughly equal to a year's gain in achievement. (The numbers you see are called "scale scores" and are used to enable researchers to more easily compare achievement across schools and groups and to see differences.) To put this in the simplest terms, students scoring 10 to 12 points higher than other students have achievement levels we would expect from those having an extra year of schooling. This data suggests that students living in families at or near the poverty level are about two years behind their middle class peers. Clearly then, poverty seems to greatly affect student achievement in public and in private schools. We note that although the private schools seem to have an advantage on the scores, they enroll very few special education students, and they get to select their students.

We also explored data in districts across the country that have had charter schools as well as voucher programs and other common school reforms in place for a number of years. We again looked at NAEP scores in order to see what kind of impact these reforms have had on reversing the apparent relationship between poverty and student achievement.

NAEP scores can be useful checks against a natural tendency of states, districts, and schools to focus on teaching to the test, because NAEP assessments are much more difficult to game or teach to than state level tests. An example of this can be found in states where 90 percent or more of students receive passing scores in their state at the basic level, when only 20 or 30 percent of their students are passing NAEP at the Basic Level. We note that the Proficient Level of NAEP is not at all equal to what we might assume is being on grade level, but is instead significantly higher.

Cleveland, Ohio school reformers achieved success at the U.S. Supreme Court in 2002 with a ruling that vouchers could be used in religious schools. Currently, over 5000 students in Cleveland attend schools that accept vouchers and most of these students are attending church-related schools. (Vouchers are named for the payment vouchers used by families to fund their children's schooling at private, parochial or other schools outside of existing public school choice opportunities such as magnet schools. Vouchers should not be confused with scholarships, which are used to reward merit, or scholarship.) In 2003, right after vouchers for religious schools were approved to begin, the average 8th grade NAEP reading assessment scale score in Cleveland was 240 for low-income students. Eight years later the average score for low-income students remained at 240. We should note that Cleveland does not have enough students who do not qualify for either free or reduced meal prices to actually have a published NAEP score for those students.

In New York City, where school reformer Chancellor Joel Klein shook up the school system and radically changed the structure of the nation's largest school system, with charter schools taking over many existing schools, and also in Chicago, where current U.S Secretary of Education Arne Duncan used many of the same reforms, score trends on 8th grade reading were also flat. Average scores for low-income students from each of these two cities in 2011 straddled the national average for low-income students. Students in New York City, after years of reform scored 151, Chicago students scored an average of 149, and the national average for low-income students is 150. Clearly, the evidence suggests that poverty matters, and it matters in a powerful and predictable way. In spite of early claims that teachers in high-profile schools in these cities had reversed the impact of poverty, closer examination nearly always reveals that the impacts of poverty certainly remain.

            To show how concentrations of poverty play out for individual schools, researchers plotted the scores of students at elementary, middle, and high schools in Ascension Parish, Louisiana (In Louisiana, "parishes" are the equivalent of "counties" in other states.). Ascension Parish is useful because the school level data is not affected as much as it is in other districts by charters and magnet schools that distort the data. You can easily imagine how a school that selected mostly high-achieving students who also qualified for free meals would distort patterns of achievement for that school, as well as other schools around them. (Obviously, there are students from low-income families who are also extremely high scoring! It is for this reason, among many others, that low expectations for students are never justified!) For each level, elementary, middle and high, the straight line on each chart represents the statistical fit of the relationship between the percent of students qualifying for free meals and the schools' performance scores. (It is this School Performance Scores, or SPS, that determines the actual letter grade of a school.)

 It is almost frightening that the schools were fitting so closely on the line that represented the correlation or relationship between poverty and school performance scores. There are actually two schools where both the school performance scores and the corresponding percentages of low-income students were so nearly identical that they appear over one another in the chart.

The pattern for middle schools was even stronger than that for elementary schools in Ascension, as illustrated on this next chart.

The powerful relationship between poverty and performance scores in these Middle Schools is almost beyond description. In statistical terms, the correlation between the percentage of students eligible for free meals and the school performance scores for Middle Schools in Ascension was negative .987 which, simply stated, means that as poverty increased, scores went down, in an almost perfectly linear relationship.

For the Ascension Parish High Schools the relationship was as perfect as most of us have ever seen. The incredibly powerful numerical correlation of -.998 once again suggested that the percentage of free lunch eligible students in a school could easily be used to fairly accurately predict the school performance scores of a school.

These charts can be incredibly powerful in helping us visualize the degree to which poverty matters. In school districts where there were schools that had a group of "gifted" or "talented" students, or where there were minimum requirements to get into the school, such as the Baton Rouge Foreign Language Academic Immersion Magnet (Baton Rouge FLAIM), we almost always found these schools to be above the line, meaning they achieved at a higher level than their free lunch percentages would have predicted, for obvious reasons. Conversely, schools that dealt with high concentrations of students who had not been achieving well on standardized tests in the high-stakes world of "reformy" Louisiana, and who often directed their frustration in socially unacceptable ways, were almost always below the line. Schools with high concentrations of students qualifying for special services due to mental, physical, or emotional handicapping conditions almost always lie below the line.

In examining District Performance Scores (DPS) across Louisiana, we notice the same patterns. The DPS scores are, like SPS, mostly influenced by student achievement on standardized tests. Zachary, with the lowest percentage of students qualifying for free meals in the entire state, had the highest DPS. On the other end of the spectrum, the Recovery School District and St. Helena Parish have the two lowest DPS scores, with St. Helena having the highest percentages of students qualifying for free meals.

The strong nature of the relationships we found yet again convinced us that we needed to seriously question the statements we quoted earlier in this paper about the powerful effects of teachers on student academic achievement.  When we went to the original sources of the texts used by Tabitha Grossman, for example, we discovered that her statements were not consistent with the statements in the documents she referenced.

Dr. Grossman wrote: "Teacher effectiveness is the primary influence on student achievement, followed by principal effectiveness." The Wallace Foundation research report she referenced actually said: "Teachers have the most immediate in-school effect on student success." We trust you can note the not so subtle-difference. "In-school" and "immediate" do not necessarily equate to "primary influence." 

What about her other quote: "Research shows that teacher quality is the primary influence on student achievement."  When we went to the OECD report she referenced, this is what we found:

Three broad conclusions emerge from research on student learning. The first and most solidly based finding is that the largest source of variation in student learning is attributable to differences in what students bring to school – their abilities and attitudes, and family and community background. Such factors are difficult for policy makers to influence, at least in the short-run. (Emphasis added)

The second broad conclusion is that of those variables that are potentially open to policy influence, factors to do with teachers and teaching are the most important influences on student learning. In particular, the broad consensus is that “teacher quality” is the single most important school variable influencing student achievement. (Emphasis added.)

The OECD report and the earlier report from the Wallace Foundation suggest that teacher effectiveness is important, but they are clearly not reporting that among all factors impacting student achievement, that teacher impacts are the most important.  In fact, the OECD report is quite explicit on this.

If researchers can use rates of poverty, independent of any knowledge of actual teacher and teaching quality, to predict school performance scores, what does this say about the claims of reformers that it is the teaching in the schools that is deficient?  What it suggests is the need to ask the question in the second part of the title of this paper.  What if failing schools… aren't?

Policy makers in Louisiana, and in other states, have suggested that value-added measures (VAM) be used to explore the actual impact of teachers on student achievement. Supporters often use the following argument to suggest that this complex mathematical procedure is necessary because it would be ridiculous to use the actual test scores of students. "Think about it" they say. "Would anyone want to teach the students who have the most challenges to overcome, and who score the lowest on nearly all standardized tests?" Teachers might only want to teach advanced students, or gifted students.  "Obviously" they say, this would not be "fair." How is it "fair" to not account for these challenges when measuring "school performance?"  How is it "fair" to condemn schools as failing when the accountability system is so flawed that it does not even begin to measure the quality of teaching within the schools?

I believe the data and arguments presented in this paper make clear that many schools are being unfairly stigmatized with the label of failing. More to the point, it suggests, very strongly, that the letter grades themselves seem to have no particular value in predicting the quality of teaching within schools! I recently tweeted that "You can know nothing about the quality of teaching in a school by looking at the school's letter grade." To put it bluntly, the evidence suggests that school performance scores are clearly and powerfully related to the degree of poverty in the schools. I believe that there are likely to be some schools that have excellent teaching faculties, and some schools that do not have such a strong teaching faculty. I also believe the data suggest that neither the School Performance Scores nor the "transparent" Letter Grades are useful in determining which schools have which faculties.

Wonderful schools which are nearly filled to overflowing with students struggling from the negative impacts of poverty do not deserve to be automatically labeled as failing schools by anyone who has not carefully, and honestly, examined the teaching going on within their walls.

Parents, teachers, students and even school reformers perhaps, can fairly make determinations of whether a particular classroom is conducive to learning. It may take multiple observations and interactions with the teacher and the students, but fair and accurate evaluations of the quality of teaching can, and should be made. School reformers often seem to know numbers, but don't always appear to know what those numbers actually represent, or if they represent anything at all. I argue here and now, stridently and with the support of incredibly powerful data, that to label a school as failing based on the scores of students in the school, without taking into account factors that Dr. George Noell, the Wallace Foundation, and the OECD and others have clearly suggested impact student achievement independent of the classroom teachers, is neither ethically nor morally defensible!

Our current accountability system for schools does nothing to account for the effects of poverty on students in our schools, and the challenges it represents to those teaching and learning in these schools. Louisiana's accountability system does not, in any way, account for the incredibly varied abilities students bring with them when they first arrive at our schools. The current system fails to recognize evidence such as Kindergarten inventories that clearly point out the effects poverty has had on children before entering school.  The work of Dr. Noell and others who point out that these scores are powerful predictors of future performance, and would provide evidence of the extremely difficult challenges faced by high-poverty schools, is totally ignored.

Recognizing poverty, and the incredibly powerful effects it has on student achievement in our schools, especially prior to students even entering Kindergarten, would provide increased support for initiatives that have incredibly powerful and proven effects on student performance. Community programs that focus on pediatric care or parenting awareness programs are not as strong as they need to be in a state where children rank last in virtually all early-childhood categories on comparisons across all states. The number of books in the home, and the hours of television watched might become topics of conversation in a community seeking to increase the education quality of the entire community.

The current accountability system labels schools as failing in spite of no evidence that it is the teaching that is failing. The labeling is done on the basis of test scores alone, which have more to do with the incredible poverty faced by the students than with any evidence of poor teaching. When so many high-poverty schools across the state receive the Letter Grade F, it has the effect of eroding public confidence and desperately needed community support for our neediest public schools, yet the grades are assigned by computer, with no one going to the schools to actually see what is happening in the classrooms!  This does a great disservice to teachers, to students, and to parents who more than anything else, simply want to have their children in schools where teachers meet their children where they are, and then teach their children well.

The real choices to be made, then, are perhaps not the "choices" promoted by reformers who often see profits in the condemning of public schools. Instead, a more important and meaningful choice is whether we choose to do something about the failure of our accountability system to fairly and accurately provide parents and communities with useful information about the quality of our public schools.

The important question, for now, is whether we will have the moral and political courage to stop falsely labeling schools as failing, with all of the negative consequences such a label carries, absent real and convincing evidence concerning the quality of teaching within the school. The thousands of dedicated teachers and administrators in our schools deserve better, and the hundreds of thousands of students certainly deserve nothing less!

I used the term "failing schools" in this paper over a dozen times, and chances are you didn't stumble over the term, since we have heard it so many times. I believe that we should banish the use of this term in the media, and in our policy debates, until the reformers can assure the public that they ONLY mean those schools that are actually failing to provide quality instruction for student, and not simply those who face incredible challenges, and in spite of those overwhelming challenges provide excellent opportunities to learn for all children. 

This paper is the first in a series which will begin to offer an alternative vision of the very real choices Louisiana can make as it moves beyond its bicentennial as a state. Will we make the hard choices, that require more than the labeling and blaming of schools for the challenges we face?  I believe we must. I thank you for reading this, and I welcome your comments and your suggestions for ways we can further inform the future of education.

Copyright©2013 by Noel Hammatt. All rights reserved. Permission is hereby granted for the publication of all or portions of this paper by those seeking to share the truth about public schools in Louisiana, and elsewhere, as long as this complete copyright notice is included.