I found this article late last week, at Inside Higher Ed… entitled “The Black-Box of Peer Review“, about a new book by Michele Lamont entitled ” How Professors Think: Inside the Curious World of Academic Judgment“. I’m looking forward to reading this book… when I can get my hands on it and have a bit of spare time (insert big laugh here). According to the Higher Ed. article- for the writing of this book Dr. Lamont was able to go inside the ‘anonymous’ peer review system and have a look around. She did this because she was interested in the idea of excellence… according to the Higher Ed. article:
Michèle Lamont decided to explore excellence by studying one of the primary mechanisms used by higher education to — in theory — reward excellence: scholarly peer review.
Excellent indeed. So- how’s the whole thing working? To quote Higher Ed again:
For those who have always wondered why they missed out on that grant or fellowship, the book may or may not provide comfort. Lamont describes processes in which most peer reviewers take their responsibilities seriously, and devote considerable time and attention to getting it right.
She also finds plenty of flaws — professors whose judgment on proposals is clouded by their own personal interests, deal making among panelists to make sure decisions are made in time for panelists to catch their planes, and an uneven and somewhat unpredictable efforts by panelists to reward personal drive and determination over qualities that a grant program says are the actual criteria.
Uh huh…sure- I get this, and I believe it- it’s sorta old news, but it’s nevertheless nice to see it in print so that we know we are not crazy!!
But I’m more interested in the whole idea of excellence, and I’ll tell you why. At a recent meeting I attended, I was eavesdropping on a discussion at lunch at the table behind mine (ok, it was loud so I didn’t actually have to eavesdrop much), about which grants should be funded. There was one contingent at the table that was arguing in a rather heated way that the ‘best’ science should be funded, without regard to the significance of the public health problem and other factors. I guess that made me wonder how two reviewers of a particular grant figure out what’s the ‘best’ science, or which grant is the most ‘excellent’. I don’t think Dr. Lamont’s findings in the Inside Higher Ed. article made me feel any better:
The most common flaw she documents is a pattern of professors applying very personal interests to evaluating the work before them. “People define what is exciting as what speaks to their own personal interest, and their own research,” she said.
Other flaws in the peer-review process that are covered in this article… and appear in the book:
- 1. Little time spent discussing proposals with either broad or no support- most attention directed toward the middle of the pack. Alliance building among panel members – and ‘strategic’ voting to get particular proposals to win approval. Giving higher than deserved scores to particular proposals to keep them alive.
- 2. What the article calls luck of timing… rare to go backwards in a discussion to a more deserving proposal that has already been discussed, so it matters where your proposal is in the order of the review… cause they won’t go back to it.
- 3. Favoritism for work similar to one’s own… or for some personal interest (other than direct personal ties).
- 4. “Morality and Character”… ??? According to Dr. Lamont, panels link their evaluation of a particular proposal to the applicant’s character.- ok, to me this is just favoritism by some other name.
- 5. Broad agreement that diversity is a good thing, but little attempt to put this into actual practice- other than working on the assumption that elite research universities are ‘priviledged in the competition process’. This means that applicants from such institutions get the benefit of the doubt- when, if you are located at a different type of institution you probably won’t benefit from that. (That’s a pretty freakin’ weird definition of diversity- you people on panels)…
Yikes. The ugly underbelly- uncovered. What say you all?