Two Education Crises: Which is Fake, Which Real?
Last week there were two important studies released. One tells us that the international test data used to declare our schools broken and uncompetitive is bogus. The other tells us we have a very different crisis we should be concerned about: the percent of students who are engaged and excited about school drops dramatically between elementary and high school. The policies being pursued to fight the first, phony crisis are likely to be making our real problem of declining student engagement worse.
When "no excuses" reformers like Michelle Rhee or Bill Gates want to justify their demands for policy shifts in our schools, the first card they play is the one that says our schools are failing in comparison to those of our international rivals. Michelle Rhee's StudentsFirst ran TV ads last summer that depicted our students as an out of shape athlete bumbling around on a gym mat. Bill Gates on Oprah a few years ago asserted that if we could get rid of all the nation's "bad teachers," our scores would rise to the top of the world rankings.
But researchers have long been skeptical about these rankings, and careful investigation by Martin Carnoy and Richard Rothstein have confirmed that our standings are quite respectable. Our international standings are depressed not by "bad teachers," but by the relatively high number of students we have living in poverty.
They explain:
Disaggregation of PISA test scores by social class group reveals some patterns that many education policymakers will find surprising. Average U.S. test scores are lower than average scores in countries to which the United States is frequently compared, in part because the share of disadvantaged students in the overall national population is greater in the United States than in comparison countries. If the social class distribution of the United States were similar to that of top-scoring countries, the average test score gap between the United States and these top-scoring countries would be cut in half in reading and by one-third in mathematics. Disadvantaged U.S. students perform comparatively better than do disadvantaged students in important comparison countries. The test score gap between advantaged and disadvantaged students in the United States is smaller than the gap in similar post-industrial countries...
The authors include this in their conclusion:
We have shown that U.S. student performance, in real terms and relative to other countries, improves considerably when we estimate average U.S. scores after adjusting for U.S. social class composition and for a lack of care in sampling disadvantaged students in particular. With these adjustments, U.S. scores would rank higher among OECD countries than commonly reported in reading--fourth best instead of 14th--and in mathematics--10th best instead of 25th.
So perhaps we are not international laggards after all.
But we are aware that test scores are not the thing that really matters in the long run. In the long run, what we ought to care about most is how well prepared our students are to be happy, productive adults. And higher test scores could actually mean we are doing worse in that regard. The recent Gallup poll results on student engagement draw our attention to another set of indicators, and by these, we are in real trouble.
But we are aware that test scores are not the thing that really matters in the long run. In the long run, what we ought to care about most is how well prepared our students are to be happy, productive adults. And higher test scores could actually mean we are doing worse in that regard. The recent Gallup poll results on student engagement draw our attention to another set of indicators, and by these, we are in real trouble.
The Gallup Student Poll surveyed nearly 500,000 students in grades five through 12 from more than 1,700 public schools in 37 states in 2012. We found that nearly eight in 10 elementary students who participated in the poll are engaged with school. By middle school that falls to about six in 10 students. And by high school, only four in 10 students qualify as engaged.
Mr. Busteed provides a strong dose of reality to our test-obsessed system:
The drop in student engagement for each year students are in school is our monumental, collective national failure. There are several things that might help to explain why this is happening -- ranging from our overzealous focus on standardized testing and curricula to our lack of experiential and project-based learning pathways for students -- not to mention the lack of pathways for students who will not and do not want to go on to college.
A few months ago I visited a high school in Albuquerque that has figured out how to reverse this dynamic. In targeting students who had already dropped out or were disengaged from school, ACE Leadership High had to confront this crisis head-on. They did so in ways that echo Mr. Busteed's insights, through projects with strong connections to the real world these students see around them.
Let's be clear about why so many students have disconnected from high school. As the pressure to perform well on tests is exerted ever more on everyone in our schools, we have shifted the very reason for our work. When students ask "why are we learning this?" our best answers revolve around the students themselves. We ought to be teaching things that are really useful in their lives, and which satisfy their curiosity about the world. It is our job as teachers to provoke that curiosity, and build on it. It is our job to make connections to the real world visible and compelling. When our answer to the question "why are we learning this?" is "because it is on the test," or "because it is Common Core standard 3.6a," we have lost our way, and our students know it.
It is time to abandon the phony imperatives of test-driven reform, and listen to what our students are telling us.
What do you think? Is our obsession with test scores killing student engagement?
Continue the dialogue with me on Twitter at @AnthonyCody
International Tests Reveal Surprises at Home and Abroad
Policymakers and pundits raise alarms whenever international test results are announced. In December, upon release of new scores from the Trends in International Mathematics and Science Study, or TIMSS, U.S. Secretary of Education Arne Duncan called them "unacceptable," saying they "underscore the urgency of accelerating achievement ... and the need to close large and persistent achievement gaps."
It was no different a little over two years ago, when the Program for International Student Assessment, or PISA, released its latest scores. Secretary Duncan said they showed Americans "napping at the wheel. ... As disturbing as these national trends are for America, enormous achievement gaps among black and Hispanic students portend even more trouble for the United States in the years ahead."
Such conclusions are oversimplified, frequently exaggerated, and misleading. They ignore the complexity of testing and may lead policymakers to pursue inappropriate and even harmful reforms.
—iStockphoto.com/Nikolass Boden
Both TIMSS and PISA publish not only average national scores, but also a rich database from which analysts can disaggregate scores by students' socioeconomic characteristics. Examining these can lead to more nuanced conclusions than those suggested from average national scores alone.
Yet, for some reason, although TIMSS published average national results this past December, it only released its underlying database last week. This puzzling procedure ensures that commentators draw quick but ill-informed interpretations. Analysis of the database takes time, and headlines from the initial release are codified before scholars can complete more careful study.
Since the last PISA release in 2010 (of a test given in 2010), we have been digging deeper into its database and examining older databases for TIMSS and for our domestic National Assessment of Educational Progress. We concentrated on adolescent scores—8th graders on TIMSS and NAEP, 15-year-olds on PISA—in the United States; three top-scoring countries (Canada, Finland, and South Korea); three similar postindustrial countries (France, Germany, and the United Kingdom); seven American states; and three Canadian provinces for which trends are available.
The TIMSS executive director deemed the report of our findings helpful (without endorsing our analysis in detail), but officials of PISA and of the U.S. Department of Education were harshly critical. We have posted their criticism, andour response to it
, online.
,
Some of our conclusions are obvious; some are counterintuitive or startling. Here are a few:
• A larger proportion of students in the United States is disadvantaged than in any comparison country. The number increased rapidly over the last decade, while in comparison countries it did not. Nonetheless, reading and mathematics achievement of lower-social-class U.S. students improved substantially, while achievement of similarly disadvantaged students declined in countries to which the United States is frequently unfavorably compared.
Thus, while the reading achievement on PISA of the lowest-social-class students in the United States grew by more than 0.2 standard deviations from 2000 to 2009, it fell by an even larger amount in Finland. In math, U.S. students from the lowest social class also gained substantially, while scores of comparable Finnish students declined. This is surprising because the proportion of disadvantaged students in Finland also fell, and we might expect this to make the task of devoting resources to them easier.
Certainly, even for the lowest social class of students, Finland's scores remain higher than ours, but examination of trends (performance changes over time), as well as levels (performance at a single point in time), challenges the easy assumption that simply imitating Finnish education is a recipe for U.S. success.
• Unsurprisingly, every country has an achievement gap between its most- and least-disadvantaged students. But unexpected is that this gap is smaller in the United States than in similar postindustrial countries, and often only slightly larger than gaps in top-scoring nations.
In some comparisons, the U.S. achievement gap is relatively small, while the scores of both disadvantaged and advantaged students are relatively low. But in other comparisons, the relatively small gap is attributable to our disadvantaged students' scores being relatively high, while the scores of our advantaged students are relatively low, especially in math.
This brings into question our widespread policy consensus that American educational problems are concentrated among disadvantaged students.
• Creating truly representative national samples is difficult for test-makers, and errors can have big consequences. For example, in 2009, PISA apparently oversampled low-income U.S. students who attended schools with very high proportions of similarly disadvantaged students, artificially lowering the average U.S. score. While 40 percent of the PISA sample in 2009 was drawn from schools in which half or more of the students were eligible for the subsidized-lunch program, only 32 percent of students nationwide attended such schools, according to National Center for Education Statistics data.
A top PISA official disputes our claim. While acknowledging that 40 percent of the PISA sample for the United States was, indeed, composed of students who attended schools where more than half qualified for subsidized lunches, he suggests that free-lunch participation must have jumped because of the recession.
This explanation is theoretically possible, but not plausible, considering that the NCES database shows that for the same year (2009) a considerably smaller proportion of all secondary school students (not a representative sample) attended schools with such high poverty concentrations.
So we stand by this conclusion: If we make two reasonable adjustments to the reported U.S. average, our international ranking substantially improves. The first adjustment reweights the social-class composition of U.S. test-takers to the average composition of similar postindustrial countries. The other reweights the distribution of lunch-eligible students by the actual concentration of such students in schools. Once we further eliminate ranking differences based on tiny and meaningless statistical differences, these adjustments raise the U.S. international ranking on the 2009 PISA from the publicly reported 14th to sixth in reading, and from the publicly reported 25th to 13th in mathematics. While there is still room for improvement, these are more respectable showings.
• We should be cautious about drawing conclusions about international comparisons from any single test. From 2000 to 2006, U.S. math scores on PISA fell substantially, causing great alarm among U.S. policymakers and pundits. But few noticed that during roughly the same period, U.S. math scores on TIMSS were rising, as they did on NAEP. We cannot attribute this to alleged superior alignment of TIMSS to U.S. curricula because in the next period, the pattern reversed: From 2006 to 2009, PISA and NAEP scores both rose, while TIMSS scores were flatter.
• Just as population sampling is complex, so is the choice of topics to test. New TIMSS results show that the United States' and Finland's scores are now about the same overall in mathematics. But underlying data show that the United States does better than Finland in algebra but worse in number properties (e.g., using fractions). Algebra and numbers now each constitute 30 percent of test items (proportions have changed over time, adding another challenge to interpretation). But in the United States, we have made a big policy push to expose all students to algebra in 8th grade. If algebra had a bigger timss weight, our average score would be higher.
When it comes to domestic tests, U.S. analysts are relatively sophisticated. Federal law requires that average scores be disaggregated by race, ethnicity, and family income. But in our rush to condemn U.S. international performance, we ignore this common-sense approach. It is time for analysts to apply the same sophistication to international scores that is now second nature to us domestically.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.