Since I’m having SO.MUCH.FUN. with the response to previous reviews- I thought I’d share a few more gems with you from one of the previous rounds. So here goes-
- #1: Some general weaknesses and concerns center on the application’s focus on over-investigated areas such as phenotype X, Y, and Z….
I’d take this criticism standing up, with a stiff upper lip- I really would… but for one thing. There were NO experiments in the proposal to investigate phenotypes X, Y, and Z. NONE. Now we can all have a good laugh about this and think that this person is off their rocker…. and this is easy to rebut, but maddening that this carelessness passes for peer-review.
- #2: It would have been nice to have seen some thinking that is removed from the obvious XYZ.
Again- I’m not sure what is being referred too- since XYZ systems weren’t included in the proposal. But the other thing that gets me riled up about statements like this is the ‘it would have been nice’ part. WTF??? Note to reviewers- strike the ‘it would have been nice’ statements from your reviewing vocabulary. Say it out loud when you think the focus is in the wrong place- I’ve got no problem with that. But if you are going to do this- make sure you actually read the proposal first. Secondly- if I only had a crystal ball to determine what reviewers think would be ‘nice’ or ‘most interesting’. I’ve got a lot of topics and subtopics as part of this project that I think it would be ‘nice’ and ‘most interesting’ to think about- but to me the bottom line is- on this proposal, is the question being asked important? Are the approaches being proposed to address said question likely to answer the question? Have the pitfalls been adequately considered and alternatives (likely to work) proposed? Outside of that I’m just stuck with these pointless- it would have been nice- statements. Now I’ll probably get a whole bunch of comments saying that I’m hopelessly naive… but that’s OK.
- #3. The preliminary findings are predictable and would be true for most organisms.
This one really takes the cake, though. In order to realize that you’d have to know that in my preliminary data I examined a highly specific subset of factors- so specific in fact, that they belong only to a single species.
So- for all of you who think that this peer-reviewed thing is always done with the care and caution that you would do it with- think again. There is randomness and carelessness in this system, just like in any other.
Blue Lab Coats 
Response to previous reviews:
1) There were NO experiments in the proposal to investigate phenotypes X, Y, and Z. NONE.
2) XYZ systems weren’t included in the proposal.
3) In fact, they belong only to a single species.
I reviewed a proposal last week and gave my first “poor” rating. Epic Fail. Took about 4 hours of WTF? It was a resub, so they just deleted the problems of the previous rejected version! I wish reviewers could see the previous versions to pick out issues of bad reviewing or bad PI writing. Panels can see the previous versions, dammit.
jc- Uh huh. That’s pretty much what I wrote. There was another one where the reviewer singled out something on which no experiments were proposed until the area was mentioned as a potential place to go next in the summary. Gotta love that.
I hate wasting any of my 3 page rebuttal on this crap though.
This lack of care on the part of your reviewer may be symptomatic of a bigger problem: the reviewer is simply disinterested in your proposal. Think about the reaction you are trying to elicit in the reviewer with your resubmission. Is the reviewer going to read your rebuttal and say:
A) “Oh of course, she’s right, I can’t believe I missed that!”
or are they going to say
B) “Just stop sending me this boring crap already!”
If the reviewer catches even an imaginary whiff from you that you think they’re a poor reviewer (even if they factually are a poor reviewer), you’re sunk. Remember that reviewers are volunteering their valuable time and they think they’re doing you a favor by reading your proposal.
You and the reviewers need to be on the same team. If you feel irritated at the reviewers when writing your resubmission, I think that means it’s not yet time for you to write the resubmission.
“This lack of care on the part of your reviewer may be symptomatic of a bigger problem: the reviewer is simply disinterested in your proposal.”
This could be true, and is often the reason for a really poor review (meaning a cursor, meaningless, and un-useful one). But, it’s also one that’s impossible to re-but, since they haven’t told you anything. I wonder if NIH should allow reviewers to actually state clearly that the proposal should be rejected with no opportunity to resubmit, the way that one can with manuscripts.
neurolover, whimple- I’ve got no trouble with a reviewer saying that they don’t find the work interesting, and they don’t think one should bother resubmitting. If this is the case- then come out and say that. I’ll add at this point that this particular proposal got a good score- so I’m tending more toward the side of careless, and less toward the side of hate it, don’t want to see it again. evah.
But I take your point whimple- that it can only be advantageous to see oneself and the reviewers as on the same side- after all, this is how you want the reviewer to see it should they decide to go to bat for your proposal- so making it easy for them to be on your side is a good idea.
“I’ve got no trouble with a reviewer saying that they don’t find the work interesting, and they don’t think one should bother resubmitting.”
Me neither, but they never seem to. Instead, the modus operandi seems to be to damn with faint praise, and since praise has to be ebullient these days, the faint praise can actually be pretty good and yet, meaningless.
The score, on the other hand, does mean something.
Still keeping my fingers crossed for you drdrA. I do think that the stimulus money is supposed to be coming to some potentially in the pipeline grants, in addition already existing grants. So, if you have a scored grant, you should be talking to talking to people about how to make the cut. One quirk I’m hearing is that the money needs to be spent fast, and create jobs (which is not a scientific argument).
With respect to “it would have been nice” statements that are actually on topic, this is tough. Reviewers are not supposed to re-write the grant but if you just say “this is too boring/limited/etc” then the applicant is really stuck on revising because she has no idea what is in the mind of this specific reviewer. So people who like the grant (you got scored, right, they liked….) may be trying to communicate their best thought on what is going to put your next revision over the hurdle.
It may be an idiosyncratic reviewer opinion or it may be an attempt to communicate the direction that plays well in that particular study section…
It is not so much that anyone is naive, drdrA, just that any one of us has a limited set of experiences and is trying to navigate a a murky environment. Ultimately only the PI can reach a conclusion on the best response/course of action but perhaps the peanut gallery can lay out the range of plausible interpretations/motivations when it comes to the summary statement.
To speak to reviewer reluctance to say “this is boring” or “you know, I just don’t think this is going anywhere productive even if successful as planned”:
The problem is that this really personalizes the judgment. It makes the decision a matter of the subjective opinion of one peer rather than a dispassionate analysis that any other similarly-expert peer would make as well. Obviously, the system strives for the latter over the former.
Regardless of what actually happens in practice, writing “I just think this stuff is dull as ditchwater” is an alert. Personally, I would try to find more objective-sounding ways of expressing myself in such a situation. Ways that might be harder to interpret than “You know, I just find your proposed stuff boring”.
I learned recently that SROs rewrite the reviews to make them “more reasonable”. I’m pretty sure that the statment “I find your work really boring.” or “I just think this stuff is dull as ditchwater.” would be taken out by an SRO. Reviewers are strongly discouraged from writing “value judgements” like “I think this is boring.”
There’s a strange interaction between the concept of a review (trying to help the PI make a better proposal – that’s the point of writing a review, right?) and the concept of judging (deciding who gets money). Unfortunately, that interaction sometimes makes for strange language.
PS. Is ditchwater really dull? I suspect that there are some very good proposals studying ditchwater. Maybe public health ones?
BikeMonkey- I like to hear all of your (and others, like Whimple for example) interpretations- because sometimes I feel I’m trying to learn the foreign language skills that it feels are required (or just darned helpful) for interpreting the reviewer’s comments. I feel like I’ve got a pretty good handle on this on paper reviews- but grant reviews feel different to me. So the multiple translations, the PO opinion, and my own sense from reading everything in context help a lot- actually.
qaz- We have that strange interaction between helping and judging quite a lot in academia- I often feel like this when I’m reviewing papers too- and well- jr. faculty mentoring- that’s the razor’s edge between helping and judging … and FYI- I’m all about ditchwater!! Cool microbes probably grow in there.
Bear in mind also that a pretty good score might reflect two reviewers who really liked a proposal and one reviewer who really hated it: (1+1+3)/3=167. It seems plausible that this reviewer really did not like the topic, and you might try to meet him/her at least halfway by offering some aspect that might seem less “obvious” or “over-investigated” (while not going too far out on a limb, thereby creating a new target).
There’s a strange interaction between the concept of a review (trying to help the PI make a better proposal – that’s the point of writing a review, right?) and the concept of judging (deciding who gets money). Unfortunately, that interaction sometimes makes for strange language.
wrong and wrong.
The NIH reviewer is pretty explicitly warned off of both trying to help the PI write a better proposal AND deciding about funding. The “F-word” is not supposed to be spoken in my panel.
The job of the reviewer is supposed to be limited to describing, critiquing and ranking the relative merits/weaknesses of proposals in that specific review round.
How often do reviewers handling multiple proposals attach the wrong reviews to the wrong email replies?
Lab Lemming- I wonder how often that happens too – but in this case based on other comments I don’t think that’s what happened.
Well if you wanna talk about what you do like to see in reviews, swing by the lounge sometime:
http://lablemminglounge.blogspot.com/2009/02/what-to-do-when-you-review.html