So let’s see, how much can I say about this topic. Summary statements (SS) arrived while I was on vacation. Thankfully I couldn’t open them from abroad.
These SS are from a proposal that was scored under the old scoring system as an A1 submission, and scored again using the new system as an A2 submission. AND IT WENT BACKWARDS- by a lot… to the 40 something percentile. That’s an impact/priority score in the 50s using the new system. How can a proposal that I thought was better- and addressed all the critiques from the previous 2 rounds of review… do WORSE??? Apparently the reviewers didn’t think it was better, and that’s the only thing that counts.
Because everyone keeps asking me what one gets back as a review now that there is a new format, I’m gonna tell ya’ll. First, I got an overall impact/priority score and percentile ranking, followed by a short paragraph summarizing the review panel discussion. The description of the project (provided by me) followed, as did the public health significance section. I got three individual critiques, in a standard format. Each individual critique began with individual scores (on a 1 -9 scale, I think- although it doesn’t say) in each of the following categories: significance, investigator, innovation, and approach. In the next section of each critique each of the areas that I listed in the previous sentence as well as ‘overall impact’ were evaluated by their strengths and weaknesses- given as 2-3 bullet points for each. So, for example, one of the critiques had 5 bullet points total for ‘overall impact’ – pretty equally divided between strength and weakness categories.
How the individual 1-9 scores in each category were added, subtracted, calculated, averaged- or what-the-hell-ever was done to them- to arrive at the big number … the ‘impact/priority’ score, was unclear to me. I’ll have to poke around and see if there is an explanation of how this works on some NIH website, or maybe writedit can explain it to me. It seems odd that I didn’t get an average 1-9 score (averaged out from the 3 reviewers maybe?) for each individual category (significance, etc.), or an individual score in each of the areas from the whole panel. Looking at the scores from the individual critiques for each category, the scores seem to be all over the map. Furthermore, the bullet points and the score in a category sometimes don’t match. I mean, do you give someone a 2 for innovation- then say ‘approach really isn’t very different from previous work’??? Cause if 1 is the best score you can get for innovation, or any other category for that matter, a score of 2 for innovation should be pretty fricking innovative. But WHATEVER.
My overall impression of this new arrangement? First, I thought the critiques were quite general… just lacking in specificity. And more blunt. If I were going to turn this proposal over again- that I would find it hard to address these very broad brush strokes. I also think it would be harder to determine when a proposal should really be dumpster bound- because those blunt general criticisms appear more negative than they really are. Perhaps.
Second, I’m uncertain how to define ‘significance’. What I find significant for work in my field, colleagues in my field might not find so significant- so where does the consensus lie? For example- I think it would be hugely significant to definitively describe, for the first time, a set of genes that are important for a particular bacterium to live in a particular niche that is critical for its ability to cause disease. But, colleagues in my field might say- who cares about your pathogen-du-jour- such pathogen-centric results aren’t significant in the context of the wider field of microbiology. Or- we can come back to the whole hypothesis driven vs. descriptive argument again- we already know that many scientists value studies that determine a molecular mechanism over other studies that uncover the potential for a new molecular mechanism (since such studies by definition would be descriptive). Who’s to say which idea of significance should get a score of 1, and which should get a 10?
Third, I spent such enormous effort learning how to read the reviews in the old system, figuring out what was meant by each sort of ‘stock’ critique- there will definitely be a learning curve to deciphering this, in all its generality, in the new scoring system.