TribBlog: TEA's Scott "Happy to Scrap" Projection Measure [Updated]

Texas Education Commissioner Robert Scott has been taking heat for ducking questions from reporters and a legislator regarding the Texas Projection Measure, the magic formula that last year suddenly moved thousands of Texas schools into higher state rating categories with little underlying achievement gain by students.

First, he skipped a legislative subcommittee, to which state Rep. Scott Hochberg, D-Houston, invited the commissioner and, Hochberg says, worked around Scott’s schedule. That came up in the latest of a series of columns from the Houston Chronicle’s Rick Casey, who has been pounding Scott and the TEA about the formula — which the agency has said rewards student academic “growth” but, as is now clear, simply inflates failing grades passed on a questionable projection of future performance. Scott’s absence from the meeting also came up in our story of last week, which reported for the first time that those projections may be wrong up to half the time. That means schools are getting credit for students who fail state tests but are projected to pass in subsequent years — but as TEA’s own data testing the formula shows, the projections may often give credit where none is due. Scott had been unavailable to comment on that story, too.

But Scott finally took questions, in an extended interview with the Tribune over the weekend. That followed the news, also reported in our Friday story, that the commissioner had sent a letter to district administrators warning them he might “suspend” or substantially change the projection measure for the 2011 school year.

Scott walked a fine line between defending the formula’s much-maligned statistical validity and saying it wasn’t his idea in the first place and, as he put it, “I’m happy to scrap it” if legislators and other critics have a big problem with it.

“What everybody failed to recognize is that there was a bill passed by the Legislature and that we were also required by federal law to have a growth measure in accountability,” Scott told me. “This was not a TEA initiative.”

Scott backed off the "federal law" assertion when I asked follow-up questions about it. The Texas Projection Measure was approved — but not mandated — by the feds, who offered states an opportunity to conduct pilot programs with “growth measures.”

He’s right about the state legislation (though we did indeed note the state law in our story). But the legislation, Scott conceded, mandated something other than what the TEA ultimately did. The state education code now reads: “The commissioner shall determine a method by which the agency may measure annual growth in student achievement from one school year to the next.” The actual formula doesn’t measure student growth; it guesses at how much growth they might get in the future — with some statistical quirks causing wildly optimistic predictions — and it often guesses wrong.

Asked what brought him to the point of potentially “scrapping” the measure, Scott said: “It’s brand new; we’re working through it. If it doesn’t work out, we’ll take a look at it. … Statistics tell me it’s reliable and accurate, but there’s criticism, so what the heck.”

Then there’s the matter of those statistics. Up until last week, Scott and the TEA claimed a 92 percent accuracy rate for the formula. But that’s based on its projections for all Texas students — not the small subset of failing-turned-passing projections that are used to bump literally hundreds of schools into the state’s “recognized” and “exemplary” categories and take hundreds more out of the failing category. So why use projections for all students to study “accuracy” and never study the accuracy of the projections that actually matter?

“When you look at statistical validation, the larger the sample size the better,” Scott said. “So we looked at all the scores.”

I’m no statistician, but that makes no sense to me. It’s like studying every Major League Baseball game to find out how likely it is the Houston Astros will win, for the sake of “sample size.”

But here’s the real problem, in Scott’s view. The education world is all atwitter about “growth measures” that can determine how much “added value” a school gave a student — even if that student failed. And no one, in Texas or anywhere else, has come up with a foolproof measure for this. Scott has a point here: The legislature told the TEA to find a way to give credit to failing students, based on the progress they made toward passing. And any method of doling out credit for failure is, first, difficult to craft and, second, subject to criticism. Failing, after all, isn’t passing.

When the next round of school ratings comes out, expected at the end of the month, the Texas Projection Measure will remain in place, Scott said. Legally and logistically, Scott said, it’s too late to turn back now. But the projection measure could well disappear for the following year. To get rid of “growth measures” permanently would take legislative action, Scott said, but after the fiasco with the Texas Projection Measure, Scott may well be the one asking for it. 

[UPDATE:]

Scott called the Tribune this afternoon to dispute Hochberg's assertion that he ducked out on the education subcommittee meeting mentioned above, at which Scott's subordinates took tough questions about the Texas Projection Measure in an extended grilling from Hochberg. First, Scott said, he was never informed that the Texas Projection Measure would be discussed at all, so it would be difficult, to say the least, for him to dodge a discussion he didn't know was coming. Scott provided an agenda and a published notice of the meeting, neither of which included scheduled discussion of the projection measure. He called Hochberg's earlier comments about him skipping the meeting "disingenuous" and said he had never confirmed he would appear.

In response, Hochberg had this advice for Scott: "When you're in a hole, stop digging." Hochberg conceded that Scott may or may not have known the discussion of the Texas Projection Measure was coming. But that's not the only item for which his presence was requested, Hochberg said, and Scott did indeed confirm his scheduled appearance through a staffer. Hochberg provided an email exchange between a committee staffer and a TEA staffer, in which the TEA staffer wrote that Scott would appear and likely address all the items on the agenda (the Texas Projection Measure wasn't mentioned in the email, but according to Hochberg, committee agendas are often vague and change without notice.). The bottom line, said Hochberg, is that it's "highly unusual for an agency head who has committed to show" to bail on a legislative meeting and allow his staff to take the heat.