fitting the rubric and maintaining the curve: inside the standardised-test industry

April 8, 2011 at 9:00 am Leave a comment

by Alec Patton

This is, to say the least, an alarming article. Jessica Lussenhop, writing for the Minneapolis City Pages, spoke to the markers and supervisors who determine the test scores that students will get on their essays – and anyone reading this blog knows how high-stakes these tests can be, both for students and for schools. Probably the most shocking claim is that supervisors routinely changed scores (without reading essays) in order to maintain the ‘bell curve’ that their teams were expected to maintain. But the story that really made my guts clench is this one:

Most essays, depending on the criteria established in the state, are scored by two readers. As Puthoff and his fellow scorers whipped through their essays, their supervisor had their own eyes glued to a screen, keeping them apprised of whether Reader #1 agreed with Reader #2. If so, both got a 100 percent agreement score for that essay. If one differed by a point or so, the essay would be counted as “adjacent” agreement.

Puthoff had thus far been an agreement-rate superstar. He was consistently in the high 80s.

Then came the question from hell out of Louisiana: “What are the qualities of a good leader?”

One student wrote, “Martin Luther King Jr. was a good leader.” With artfulness far beyond the student’s age, the essay delved into King’s history with the civil rights movement, pointing out the key moments that had shown his leadership.

There was just one problem: It didn’t fit the rubric. The rubric liked a longer essay, with multiple sentences lauding key qualities of leadership such as “honesty” and “inspires people.” This essay was incredibly concise, but got its point across. Nevertheless, the rubric said it was a 2. Puthoff knew it was a 2.

He hesitated the way he had been specifically trained not to. Then he hit, “3.”

It didn’t take long before a supervisor was in his face. He leaned down with a printout of the King essay.

“This really isn’t a 3-style paper,” the supervisor said.

Puthoff pointed out the smart use of examples and the exceptional prose. The supervisor just shook his head and pointed out how short the paragraphs were.

“You know, it’s more of a 2,” the supervisor repeated. “Not enough elaboration.”

Now, I’m conscious that the people Lussenhop spoke to may have had any number of axes to grind, so I don’t want to automatically take their accounts as unvarnished truth. But the story above strikes me as one that is absolutely unavoidable, as long as we continue to believe that essays can be ‘objectively’ marked. The kid in question probably did all right in school despite getting a lower score than they deserved on a standardised test, and they may have learned the extremely valuable skill of intuiting what you audience is looking for, giving it to them, and knowing just how far you can defy their expectations before you lose them. And I can understand that these tests are not designed to reward the next Zadie Smith but to identify (and therefore support) those whose writing is extremely poor. Nevertheless, I think the standardisation of writing is extremely worrying.

Advertisements

Entry filed under: Education & Children's Services.

The world’s most innovative health care: Patient hotels in Sweden Experiencing The PublicOffice: radically rethinking public services in an Essex church

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


The Innovation Unit website

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 29 other followers

Archives

Twitter Updates

Follow innovation_unit on Twitter

%d bloggers like this: