Monday, April 1, 2013

prerequisite questions

Nicholas Beaudrot of Donkelicious has some good points about a typically biased piece on education evaluation from the Times, written by Jenny Anderson, but I also think he's missing the forest for the trees. He says:
The evaluation systems seem to place an extremely high number in the highest-two rating categories, and only identifying a handful of teachers--2-3% seems to be a common number--as ineffective. The reformers seem to have this hazy, not-very-well-thought-out view that perhaps 10, 20, or 50% of teachers (no one is willing to put a number on this) are inadequate to the task of teaching, which would be astonishing for any large enterprise. The 2% figure lines up roughly with the number of LAPD cops who averaged at least one excessive force complaint per year during the late '80s (keep in mind this covers allegations, not proven cases or even disciplinary actions). Over that same timeframe, 0.5% of officers averaged about 1.5 complaints per year, and a handful of cops were truly disastrous. Even if America's public schools were to have double or triple the number of bad apples as the LAPD, that would still mean the overwhelming majority of teachers are at least doing their jobs adequately.
Beaudrot is correct to point out that the constant assertion of ed reformers that we have a teacher quality crisis in this country is not supported by evidence. However, I will persist in saying that there are some prerequisite questions which underlie all of these other questions. The most important, to me, is simple: what portion of the observed variance in perceived educational outcomes among different students is attributable to teacher effects, and what portion attributable to other factors? Also, is that attribution static, or does it change with age and circumstance? And is the portion equal between students, or is it too conditioned by demographic factors? To ask these questions is both to engage in elementary social science and to ask for basic fairness and efficacy from education reformers, yet as the Times piece demonstrates, the question essentially does not exist in our educational debate.

I have been tinkering for a long time with a long-form piece on these questions, where I look at extant empirical research and try to assess the question of how much impact teachers can have on the worst-performing students and their educational outcomes. I hope to publish it elsewhere. First, though, I have to make it through this month and finish my coursework. Until then, you'll have to take the conjectural, asserted version: any summation of the broad swath of educational research suggests that in fact demographic, parental, and environmental factors vastly outweigh teacher inputs in conditioning student outcomes. What's more, the students who perform least well are those who are likely the least subject to instructional effects as expressed in their student outcomes. Selection bias is not a factor in perceived educational outputs; it is the factor. But I'll have to make that case, and you can feel free to disregard my opinion until then.

If nothing else, these questions are empirical. The notion that teacher quality leads straightforwardly to student outcomes is one that would have to be empirically assessed, has not been proven to any minimal satisfaction, and yet is endlessly asserted by school reformers. As is typical, educational reformers' interest in empiricism is directly proportional to the degree to which empiricism confirms their preexisting assumptions. Again and again and again, uselessly emotive statements about how we must rescue every child drown out carefully considered questions about what, actually, could do the most good.

I've said this before: let's have an academic decathlon. You choose a team based on whatever pedagogical criteria you want. You can choose students from public school or private, unionized teachers or not, parochial or secular, from charter or magnet, from Montessori or KIPP or whatever else you want. However, I choose the demographics of the students on your team. For my team, the situation is reversed: you choose the pedagogical factors for my students, but I choose the demographics. You stock your team kids from whatever educational backgrounds you think work, and mine with whatever educational systems you think don't work. Meanwhile, I give you all children from the poverty-stricken, crime-ridden inner city and impoverished rural districts where we see the most failure. I stock mine with upper-class children of privilege. I would bet the house on my team, and I bet if you're being honest, you would too. Yet to accept that is to deny the basic assumption of the education reform movement, which is that student outcomes are a direct result of teacher quality. 

If you think that take is harsh, remember that I am a socialist, and if I had my druthers, there would be no more poverty and no more entrenched disadvantage to set these kids back so far before they even enter school.

Now here's what's really funny. The ed reformers assume that these current evaluation systems are flawed because too many teachers are being rated as effective. 
Grover J. Whitehurst, director of the Brown Center on Education Policy at the Brookings Institution, said variations in teacher quality had been proven to affect student academic growth. If an evaluation system is not finding a wider distribution of effectiveness, “it is flawed,” he said. 
“It would be an unusual profession that at least 5 percent are not deemed ineffective,” he added.
In other words, Whitehurst assumes that there is a natural distribution of quality in any field, where some significant percentage of people are always going to be below a necessary level of ability. That's an interesting case to be made in this context, the context of No Child Left Behind and the typical assumption of education reform, which ludicrously asserts that all children are capable of meeting certain arbitrary quality standards. But perhaps that's the inevitable consequence of a movement in which the person whose voice is heard is the person who shouts the loudest, rather than the person who pays most attention to what is constructive, to what is achievable, and what is true. In that context, it becomes a crime to state the simple reality that in a system of massive entrenched inequality, we will always have educational failure.


Morgan Warstler said...

Each year, Jack Welch would fire at least the the bottom 1%.

We start with a good 10% house cleaning on teachers, and then follow Jack's rule. No matter what, and outside of retirement / attrition, at least 1% who want to work get fired.

Dude, computers can track every student and every teacher, the rest is just finding the netflix prize for ranking teachers.

If a few mistakes are made and a few teachers suffer, who cares.

Students matter 100x more than teachers.

Flip the classroom and let teachers become tutors and we wont have this problem.

Brett said...


I stock mine with upper-class children of privilege. I would bet the house on my team, and I bet if you're being honest, you would too.

Are we talking arbitrarily high levels of funding here? Because I'm not so sure in that case. They may be deeply impoverished, troubled kids, but I can afford for tons of assistance and some really good "in-school" food.

More seriously, I tend to think that teachers' main contributions is bringing students up to some "baseline", after which their capabilities as teachers usually don't matter as much as student subject interest, temperament, and access to activities that help those interests. It's sort of like taking someone who is malnourished and bringing them to proper nutrition - you don't get the same type of jump in capabilities if you then take them to "super-nutrition", whatever that might be.

@Morgan Warstler
Each year, Jack Welch would fire at least the the bottom 1%.

Ha! I see what you did there.

I'd be careful about joking on that, though. There are some reformers out there who might almost take something perhaps not so obviously brutal seriously, even when it's disastrous (look up what happened to Microsoft when they started that kind of percentage-based ranking system).

David said...

I've said this before: let's have an academic decathlon.

A superb thought exercise.

I'm not convinced that the demographically stacked team would win, but it's a great framework in which to test and prod my thoughts.

James said...

Freddie, thanks for this post. You say that ours is "a system of massive entrenched inequality" -- couldn't agree with you more.

Couple of points of my own -- you make a comparison between the teaching force and the police force: "Even if America's public schools were to have double or triple the number of bad apples as the LAPD, that would still mean the overwhelming majority of teachers are at least doing their jobs adequately." That leaves me thinking that if a cop is not using excessive force on a regular basis, then he is "doing his job adequately." Is that really true? Taking the analogy to the teaching force, that suggests that teachers are "doing their jobs adequately" if they are not engaging in acts of sex with or violence against students. Hmmm...

As to your point about Team Poverty vs Team Lap of Luxury: what if we were to exclude the top and bottom 20% and focus on the middle 60% (or something)? How confident are you that your assertion still holds in that scenario?

Mike said...

The charter sector in MA is a pretty direct refutation of this point. Suburban charters- stocked with the prized demographic- badly underperform.

Urban charters- stocked with "inner city kids from crime-ridden neighborhoods" - outpace just about everybody.

This has been demonstrated multiple times with randomized controlled experiments:

1. Tom Kane and friends in the 2009 Boston charter paper.
2. Josh Angrist with multiple explorations of KIPP, most notably the KIPP Lynn paper.
3. Mathematica's treatment of the charter sector in MA.
4. The CREDO treatment of the urban charter sector, showing unusually large gains in Boston.

Mike said...

"What's more, the students who perform least well are those who are likely the least subject to instructional effects as expressed in their student outcomes. "

Angrist's KIPP Lynn paper dismantles this.

"Charter skeptics have sometimes argued that while relatively motivated and able students may benefit from charter school attendance, weaker students lose out. We briefly explored this type of treatment effect heterogeneity by estimating a model that adds an interaction between appli- cants’ baseline (fourth grade) scores and years at KIPP Lynn, normalized so that the main effect of years at KIPP Lynn is evaluated at the mean of the baseline score distribution. Panel B of Table 3 shows that KIPP Lynn raises achievement more for weaker students. Children with baseline scores half a standard deviation below the appli- cant mean appear to get a rough 0.06σ additional achievement boost from each year they spend at KIPP Lynn."

There are better arguments to make than the one you're making.

Freddie said...

And what, Mike, did Credo find nationally? Hmmmm?

Mike said...

"And what, Mike, did Credo find nationally? Hmmmm?"

Not sure what you mean here, and took a long time to figure out how it could possibly be relevant. Maybe the oft-cited but completely irrelevant data on nationwide charter performance?