September 04, 2003

Professor-Saving Technology

Professor-saving devices:

Unfogged Archives: From an NYT story today on computer-automated grading of essays on standardized tests:

For example, a high score almost always contains topically relevant vocabulary, a variety of sentence structures, and the use of cue terms like "in summary," for example, and "because" to organize an argument. By analyzing 50 of these features in a sampling of essays on a particular topic that were scored by human beings, the system can accurately predict how the same human readers would grade additional essays on the same topic.

None of the current algorithms can detect illogical arguments, though. GMAT essays are already read by computers -- and by human readers -- and the computer and the readers agree on scores 97 - 98% of the time.

Having the computers write the essays is the obvious next step (the ultimate in "teaching to the test"). Currently 40% of Ogged's entries are written by an algorithm which assembles sentences according to a frequency analysis and a list of fifteen key nouns, including political, RSS, and Aguilera.

Posted by DeLong at September 4, 2003 10:57 AM | TrackBack

Comments

What, TA's aren't cheap enough anymore?

Posted by: LHP on September 4, 2003 11:09 AM

"Having the computers write the essays is the obvious next step."

Yes!

Heading for the beach....

Posted by: lise on September 4, 2003 11:26 AM

"Having the computers write the essays is the obvious next step." - and I don't think that will make us head for the beach. Beaches will probably be private then, owned by the computers that went to the right schools and got the best grades.

Posted by: Mats on September 4, 2003 12:13 PM

"Distinguished computer scientist Marvin Minsky has stated that computers will eventually be so vastly superior to us that we will be lucky if they keep us around as household pets." - from: http://web.media.mit.edu/~picard/dignity/node2.html

"It will be a long time before we learn enough about common sense reasoning to make machines as smart as people are. Today, we already know quite a lot about making useful, specialized, "expert" systems. We still don't know how to make them able to improve themselves in interesting ways. But when we answer such questions, then we'll have to face one, even stranger, one. When we learn how, then should we build machines that might be somehow "better" than ourselves? We're lucky that we have to leave that choice to future generations. I'm sure they won't want to build the things that well unless they find good easons to." - by Marvin Minsky at: http://www.ai.mit.edu/people/minsky/papers/ComputersCantThink.txt

Posted by: Bob on September 4, 2003 12:57 PM

"None of the current algorithms can detect illogical arguments, though. "

Instapundit breathes a sigh of relief...

Posted by: hackticus on September 4, 2003 02:13 PM

> What, TA's aren't cheap enough anymore?

1) Does the word "budget cuts" ring a bell?

2) About 50% of the TAs I've had did not speak English as a first language. Interestingly, I don't think they did a bad job at all grading (short) essays, but students (our "clients") don't like the idea much. Of course, they might like the idea of computers doing the work even less...

3) A completely shocking fraction of what passes for student writing these days is so bad that I'm sure at least a first round of mechanical correction couldn't hurt much.

4) A completely shocking fraction of student writing is plagiarized from the internet these days, and it would be nice to automate detection of that.

5) Note that (3) and (4) are not mutually exclusive.

More seriously, I expect there is something more deeply wrong with the system when writing assignments only evoke essays that are so statistically predictable (if of high quality) that machines really do as good as (or even a better job than) people.

Posted by: Jonathan King on September 4, 2003 03:26 PM

Undergraduate writing is ATROCIOUS. Frankly, I'm not surprised proffessors don't mind giving up this part of their job. Little chance of running into something stimulating.

Posted by: Paul on September 4, 2003 03:45 PM

"Undergraduate writing is ATROCIOUS."

Yet somehow when a student graduates their writing morphs into the magically delicious. This isn't particularly fair of me, I'm sure there are plenty of college graduates writing the English language as fluently as they speak it. However, I wouldn't too quickly count Rob Wilson of the University of Hawaii among their numbers:

"If such a sublime cyborg would insinuate the future as post-Fordist subject, his palpably masochistic locations as ecstatic agent of the sublime superstate the need to be decoded as the 'now-all-but-unreadable DNA' of a fast deindustrializing Detroit, just as his Robocop-like strategy of carceral negotiation and street control remains the tirelessly American one of inflicting regeneration through the violence upon the racially heteroglossic wilds and others of the inner-city."

Most people can't write. I wouldn't limit the pool to undergrads. It's kind of insane, when you think about it. What if it were a world of people who couldn't form a sentence off the cuff?

Posted by: Mark R. on September 4, 2003 05:14 PM

Hm. Productivity improvements in the academic factory.

I wonder if you can buy a grade estimator plug-in for M$ Word.

See also turnitin.com

Posted by: Ben Hyde on September 5, 2003 05:50 AM

I wasn't aware most Profs graded essays anyway. And on the GMAT the human essay graders are paid piece-work. Which is to say, they just quickly glance at the essay, look for the key words, look for basic paragraph and sentence structure and give it a grade. In other words, they work like the computer does, but aren't as fast.

However many TA's grade so badly that I don't suppose most undergrads will notice a difference. During my time tutoring students, the example that sticks out in my mind is where a client received a worse grade than a friend of his. We took a look at both essays. The client had hit all the key points in an essay one third of the size of his friends while the friend's essay was longer and missed half the key points. Needless to say my client went to the Prof and got the grade improved. All of which told me simply that the TA didn't actually read the essays.

Another of my favourites was a class where the TA gave everyone in the class either a B or a C+. The Prof did nothing upon being informed, saying that she had not hired the TA, was only a sessional lecturer and had no authority over the TA. On the next essay only the people who had complained directly to the TA received grades other than a B or C+.

Based on personal and anecdotal evidence these are not isolated episodes.

That said, I hate the idea of essays being marked by computers.

Posted by: Ian Welsh on September 5, 2003 08:30 AM
Post a comment