I found this article late last week, at Inside Higher Ed… entitled “The Black-Box of Peer Review“, about a new book by Michele Lamont entitled ” How Professors Think: Inside the Curious World of Academic Judgment“. I’m looking forward to reading this book… when I can get my hands on it and have a bit of spare time (insert big laugh here). According to the Higher Ed. article- for the writing of this book Dr. Lamont was able to go inside the ‘anonymous’ peer review system and have a look around. She did this because she was interested in the idea of excellence… according to the Higher Ed. article:
Michèle Lamont decided to explore excellence by studying one of the primary mechanisms used by higher education to — in theory — reward excellence: scholarly peer review.
Excellent indeed. So- how’s the whole thing working? To quote Higher Ed again:
For those who have always wondered why they missed out on that grant or fellowship, the book may or may not provide comfort. Lamont describes processes in which most peer reviewers take their responsibilities seriously, and devote considerable time and attention to getting it right.
She also finds plenty of flaws — professors whose judgment on proposals is clouded by their own personal interests, deal making among panelists to make sure decisions are made in time for panelists to catch their planes, and an uneven and somewhat unpredictable efforts by panelists to reward personal drive and determination over qualities that a grant program says are the actual criteria.
Uh huh…sure- I get this, and I believe it- it’s sorta old news, but it’s nevertheless nice to see it in print so that we know we are not crazy!!
But I’m more interested in the whole idea of excellence, and I’ll tell you why. At a recent meeting I attended, I was eavesdropping on a discussion at lunch at the table behind mine (ok, it was loud so I didn’t actually have to eavesdrop much), about which grants should be funded. There was one contingent at the table that was arguing in a rather heated way that the ‘best’ science should be funded, without regard to the significance of the public health problem and other factors. I guess that made me wonder how two reviewers of a particular grant figure out what’s the ‘best’ science, or which grant is the most ‘excellent’. I don’t think Dr. Lamont’s findings in the Inside Higher Ed. article made me feel any better:
The most common flaw she documents is a pattern of professors applying very personal interests to evaluating the work before them. “People define what is exciting as what speaks to their own personal interest, and their own research,” she said.
Other flaws in the peer-review process that are covered in this article… and appear in the book:
- 1. Little time spent discussing proposals with either broad or no support- most attention directed toward the middle of the pack. Alliance building among panel members – and ‘strategic’ voting to get particular proposals to win approval. Giving higher than deserved scores to particular proposals to keep them alive.
- 2. What the article calls luck of timing… rare to go backwards in a discussion to a more deserving proposal that has already been discussed, so it matters where your proposal is in the order of the review… cause they won’t go back to it.
- 3. Favoritism for work similar to one’s own… or for some personal interest (other than direct personal ties).
- 4. “Morality and Character”… ??? According to Dr. Lamont, panels link their evaluation of a particular proposal to the applicant’s character.- ok, to me this is just favoritism by some other name.
- 5. Broad agreement that diversity is a good thing, but little attempt to put this into actual practice- other than working on the assumption that elite research universities are ‘priviledged in the competition process’. This means that applicants from such institutions get the benefit of the doubt- when, if you are located at a different type of institution you probably won’t benefit from that. (That’s a pretty freakin’ weird definition of diversity- you people on panels)…
Yikes. The ugly underbelly- uncovered. What say you all?
Blue Lab Coats 
In other news, dog bites man.
C PP- Sorry to bore you. For the uninitiated this might not be old news.
Yeah, I had the same thought as PP.
Newsflash: system is broken.
I didn’t mean that you are boring me. I just meant that it is not news that *any* system of human decision-making is infected with these kinds of irrational biases. It is simply the way human decision-making works, and is not unique to science or any other human endeavor.
Ah…just ignore CPP–he’s old and cranky! 😉 As subjective as it is, I’d still rather members of study section fund proposals they believe are “good” and “interesting” than simply fund investigators they know and like. Believe it or not, there are senior faculty at my institution who openly advocate the latter!
“It is simply the way human decision-making works, and is not unique to science or any other human endeavor.”
Yes, but if we decide that the process results in undesirable outcomes, we can tweak the methods to try to mitigates the cognitive delusions that bias and undermine decision making. That’s why the behavioral parameters of decision making are so important. It’s through understanding these that we hope to improve decision making during grant awards, admissions decisions, disasters (of any sort), in house buying and bubbles, loan making, politics, and economics, too. That’s why it’s worth studying.
In addition, science is one of those things where peopel often think their immune to the parameters of human decision making (because, after all, we’re really vulcans at heart).
“In addition, science is one of those things where people often think they’re immune to the parameters of human decision making (because, after all, we’re really vulcans at heart).
Believe it or not, there are senior faculty at my institution who openly advocate the latter!
Blog it!!!
DM–Love the avatar! Okay, gimme a few days….one new blog post coming up.
Ping!
Diversity of institution types is important. People in departments where the highest degree granted is a B.S. or M.S. can still make important contributions to research, and involving their students in research is a good thing. If you want to open up academic science to as many people as possible, then it should be possible for undergraduates to participate in good research programs with adequate resources, even if they aren’t studying for their B.S. in a department with a first-tier Ph.D. program.
Two things for R1 faculty to remember about undergraduate departments:
1) Your next student might already be writing his/her first paper, on work done in an undergraduate department. Money spent on research in a department like mine is used to train your next student.
2) Your current student may be hired into an undergraduate department, because there will always be more Ph.D. recipients than jobs at R1 schools. Money spent on research in undergraduate departments will enable some of your good students to continue making important contributions to science.
Pingback: Some links science-y and academic « Entertaining Research
Actually, the impression I got is that R15 applications from institutions with a strict undergraduate focus are generally favorably received at the NIH. Not your experience?
Whimple-
I was responding to DrDrA’s statement regarding diversity of institution types: “That’s a pretty freakin’ weird definition of diversity- you people on panels.”
I haven’t tried for an R15 or SC2 yet, but I will this summer. So far I’ve written a few foundation and NSF grants (with success on the foundation grant). I do know that a lot of people at my school have had success with NIH, so I suspect you’re right.