Monday, January 12, 2009

NSF CAREER proposal--shredded!

I found out today that my NSF CAREER award proposal was declined. This was completely expected, and as I mentioned in my previous blog I have been looking forward to the constructive feedback that I know I need. Whether consciously or not, though, I now realize I was unrealistically hoping that I would receive mostly positive feedback and a few easy things to improve upon. Thus, my first read-through this afternoon of the detailed reviews was very surprising and wickedly demoralizing. Let me try to think of some other emotions: bewilderment, depression, frustration... OK that's enough. I think the most demoralizing aspect was that a response shared by many reviewers was that the proposal was poorly written and confusing. This must be true, given that the sentiment was shared, but as of now, I have no idea how to overhaul the proposal to correct for this, and I'm dreading having to figure that out.

At this point (5 hours later), I have carefully re-read all of the feedback, and my emotions are much more tempered, and a couple new ones have emerged: hope, gratitude. The gratitude is for the many scientists (5 reviewers, plus program manager, plus panel members) who clearly invested a huge amount of time in reading (and re-reading and deciphering my bad writing) my proposal and composing very detailed and thoughtful feedback. I truly feel badly and frustrated that I can't thank these reviewers personally, due to their anonymity. Really, the only thing I can do is to do my best to follow all of the advice they have given and correct all my mistakes with the resubmission this summer...and I plan on doing this.

Following advice of Jean-Claude Bradley, Cameron Neylon, and others, I have posted most of the potentially interesting and non-confidential parts of the proposal on Scribd. I am new to the site, but I think this is a link to the proposal. WARNING: reading the proposal has been deemed difficult and perhaps a waste of time by at least 5 respected scientists!

I am not going to post the reviewers comments, since I think they had the expectations that I would not share them. I will give a brief summary here.

Overall
  • Two panels reviewed the proposal. Both panels ranked it low priority (approximately 50th percentile or worse). Ouch! One panel was the Molecular Genetics panel and the other was a special interdisciplinary panel between Mathematical and Physical Sciences and Molecular and Cellular Biosciences for interdisciplinary proposals such as this.
  • I have feedback from 5 anonymous reviewers. I don't know how they fit into the two panels, or whether it's a 3rd source of feedback for the program manager. 4 reviewers understood the impact (in my opinion), whereas one reviewer seemed to hate my writing and didn't seem to understand because of that.
  • All panels and individual reviewers had significant concerns about the proposal. There was no way this thing could have been funded without some kind of empty senate seat under my control.
  • Involvement with OpenWetWare and open science was noticed and cited as a big positive for the broader impacts.
Shared Major Negative eedback

  • Dearth of preliminary data / overly ambitious / seriously concerned about feasibility of any aspect of the proposal. This is very reasonable and somewhat expected. The students have been working hard on this project, we already have some significant preliminary data (related to "Shotgun DNA Mapping") and I expect a lot more before July of this year. I had sent in an update about the shotgun DNA mapping results in October, but based on reviewer feedback, I think that the program manager did not send this update to anyone. Another way I will address this issue is to re-organize the proposal (somehow) to make the feasibility more obvious.
  • Poorly written and confusing. This sentiment was pretty much shared by all reviewers and even included in one panel summary. I have re-read the proposal many times since July and never really considered that it was so confusing. I was modeling the format after a couple funded CAREER awards that I had seen (in other fields, though). Well, clearly I need to improve this somehow and I don't have any good ideas. I feel bad that I caused so much trouble and wasted so much time of the thoughtful reviewers. One of the reviewers basically admitted to giving up, which I don't really blame him, given the comments from the others. I feel like I've been told that my baby is ugly and just can't see it. BTW, my kids are fantastically cute.
  • Too many specific aims / third aim not well developed. I can see that this is true. One thing I stupidly didn't realize is that the reviewers basically expect NIH-style things such as "specific aims." I had actually suspected it would be viewed negatively if made it look too "NIH-ey." I will think a lot more about this, but right now what I'm thinking is that I should focus the science solely on Pol II transcription (ditch the DNA damage repair), and then reorganize the aims to be (1) shotgun DNA mapping (2) unzipping in vitro Pol II complexes and (3) mapping pol II and nucleosomes on native chromatin. This would be organized from most feasible to least feasible.
Shared Positive Feedback
  • Most of the reviewers specifically mentioned that involvement with OpenWetWare was a big positive for the broader impacts. This was really good to see, and perhaps the most notable thing to people who happen to read this blog.
  • All reviewers agreed that the impact would be very high (if successful). Further, I think some were even intrigued / excited about the idea. Many of the reviewers understood and seemed to agree with a main point I was trying to sell: that the combination of single-molecule analysis with yeast genetics is a potentially powerful and unique tool for chromatin biology.
I suppose this positive feedback is actually sufficient, if I can successfully correct the very serious flaws. After all, trying to look at things optimistically, I'd be in a much worse situation if the reviewers had thought the proposal were very well written and feasible, but lacking potential impact.

Overall, writing this blog entry has been therapeutic and also constructive. And if you do happen to skim any of the proposal, I welcome harsh criticism--I can stomach it!

SJK Note added Feb. 5, 2010:  I did end up submitting again in 2009.  The quick report is that I rewrote it in NIH style, and improved writing thruoghout.  The reviewers liked the science, but lacked confidence in me as a PI, due to lack of preliminary results.  Fair criticism that I think I can correct this year.  Most importantly: the panel LOVED open science as broader impacts.  Here're some links for my 2009 NSF CAREER proposal:

22 comments:

  1. What I take from this is that you have a really great idea but didn't write a great proposal, which I think is a good position to be in. If the idea is boring or disliked or whatever there's not much you can do about it except go back to the drawing board. But here almost all of the negative criticisms are things you can fix the next time around. My advisor has given talks on how to write grants, and while they are for NIH style grants I'll see if he'll let me post his tips so you can see if any of them are useful. Getting feedback from people not familiar with your research before you submit the proposal should also help with the writing issues. Anyway, enough from the peanut gallery...

    Oh, and I am indeed tickled that they liked the OWW aspects. Open science FTW!

    ReplyDelete
  2. I had to look up FTW, now that I know what it means, I agree! And I also agree that I'd much rather be in this situation than the inverse. The ironic thing is that I've actually taken a year-long monthly grant writing course, attended mock panels, and a bunch of other things. So, I'm mostly worried by my (apparent) deficiency in knowing what's so sucky about the grants. I'll figure it out, though...Thanks, Shirley!

    ReplyDelete
  3. Don't be disappointed, though, I heard from so many people that getting a CAREER from the first try is almost impossible. The most important thing is that you have an excellent idea, so you wouldn't have to write something from the scratch this year (I know at least two PIs who did it for the second try, after the first try returned as a complete disaster).

    Ughh, still waiting for mine. The fastlane says "pending". Which division was yours?

    ReplyDelete
  4. Thanks for the encouragement! Man, I would be so stressed out if I had to start from scratch. So, you're right I am in a good position. I was MCB, "genes and genomes systems cluster." I know someone who was funded in December (not sure which division). Maybe it's good news for you that it's still pending? I hope so!

    ReplyDelete
  5. Mine is DBI. Says to wait until mid-February

    I know there is not a single chance to get it from the first try, so I'm hoping for informative reviews at least. A colleague of mine who submitted his proposal to the same division already got his declined. And I'm still waiting (sooo frustrating!)

    ReplyDelete
  6. Steve,

    I had been reading your blog after submitting my proposal in 2009. I just had my proposal declined, the rating in panel summary statement is "rocommended". The reviewer comments are pretty supportive. All of them wanted to see broad impact. I am planning to resubmit it this year. BTW, did you submit your proposal for 2009 competition?

    ReplyDelete
  7. Yes, I did submit, and I decided to go completely "open science" with it. I was declined again. The panel loved the open science! So, if it's appropriate for your field, I'd recommend including open science in your broader impacts. I will edit the bottom of the blogpost above to put some links in to my 2009 experience.


    Good luck with your 2010 submission!

    --Steve

    ReplyDelete
  8. I am sorry to learn that your proposal was not funded. I guess such setbacks are part of our profession. I was very hopeful about my proposal. I spent almost 5 months just on one career proposal. I still feel that this is the best proposal I could write. Anyways, I am going to resubmit it. Its going to be tough requesting new support letters again. Since NSF doesnt have "memory" of the previous submissions/reviews, it is hard to be hopeful.

    ReplyDelete
  9. Anonymous: Everyone I have spoken with who has experience with the CAREER award (either applying or on review panels) has shared the opinion that there is heavy favoritism towards people in their 3rd and final submission. Furthermore, sometimes the review panels from one year to the next share many members. So, there is memory in those regards.

    In my case, I don't think there was any bias because it was not my third try. I had serious deficiencies. However, in your case, if everything seemed perfect and it was your first try (or second), I would not be discouraged. Just try to improve it a bit next time and for sure submit all three times. I have seen some really good proposals turned down on their 2nd attempt and then funded on 3rd attempt without much changed.

    ReplyDelete
  10. My NSF CAREER proposal to Bio-DBI is still pending. Is it normal that it takes so long?

    ReplyDelete
  11. Wow, you have over 1000 reads for this proposal! Were you planning on sharing the updated proposal?

    You touched on this and I'm sure that you made changes since then, but even with some background knowledge about the project, it seemed a little dense for me. That probably comes with the territory of molecular genetics (those darn abbreviations), but I would think that it would be a little easier to read by narrowing it down a bit. It sounds like this was a good experience for you (and for all of your readers) though. Best of luck in your new submissions!

    ReplyDelete
  12. Hey Diego, Thanks for pointing out that I didn't link to the newer version on the 2008 Scribd page. I added a link on the Scribd page.

    I wrote the new (2009) version in NIH format. For the 2008 version, I had intentionally not done that because some people had advised me that NSF will not fund NIH-fundable projects. I mistakenly took this to mean I shouldn't write it in that format. But that was a very bad idea, because all the people on the review panel write and / or review NIH proposals and so, they're used to the Specific Aims, Background, Preliminary, etc.

    Combined with some simplifications, the new format was a big hit with the panel. They didn't say this explicitly, but the new reviews showed that they "got it," loved the project -- just wanted a lot more preliminary data, which is very fair. (They basically worry that I'm incompetent, showing no published results after 3 years, which, again, is fair...and correctable.) So, we have a couple months to do this and then I think we have a good shot at getting it funded!

    Here's the 2009 version.

    ReplyDelete
  13. Steve, I am here again after a year... i am waiting for my NSF CAREER results... this is the resubmitted version of the last submission... i was told that if the proposal is selected the the program manager will make a call before Christmas, if not then need to be prepared for decline notice in Feb-March...

    ReplyDelete
  14. Hey Anonymous -- Good Luck!!! Don't be 100% sure if you don't hear before Christmas. I know someone who found out very late like months into this year they were funded. Sometimes money comes in.

    As for us, we did not submit last year. The project, since it is not funded, has been very slow and we are still struggling for the necessary preliminary data. I decided that waiting another year made the most sense, and I think we will make it this year. Lots of hard work ahead, but much progress has been made, and some bad broken equipment problems overcome too.

    OK, Good luck, and come back and post when you find out the good news!

    ReplyDelete
  15. hmmmm..... still no status change on my career proposal.... still showing "pending" with status date of submission.... its exactly six months now.... looks like they are going to decline mine this time too...

    ReplyDelete
  16. I just saw that the status date is changed but the status is still "pending". No calls nothing from PD. Seems like mine is shredded this time too.

    ReplyDelete
  17. Steve:

    I just found that my career is declined. It was "highly recommended" by the panel with ratings: Excellent/Very Good, Very Good, Excellent, and Very Good. This was my second submission.

    ReplyDelete
  18. Here is a link to my 2011 NSF CAREER disaster http://stevekochresearch.blogspot.com/2011/08/2011-nsf-career-proposal-ugh-failures.html

    ReplyDelete
  19. Steve, I just got call from my program manager and mentioned that my career proposal is recommended for funding; but emphasized that the conversation is not award notification. I am hoping that this is coming my way! I visited this blog several times in last years whenever I was emotionally down when my proposal was declined two times before...

    ReplyDelete
  20. That is great news! Let us know when you receive the news for sure & will be curious to know what the research is. Congratulations!

    ReplyDelete
  21. Steve, just recieved the official email confirming the award. thanks for starting a blog on this topic...

    ReplyDelete
  22. I wonder if you still have it, but can you please share the rejected proposal again? The Scribd link you mentioned in the post does not seem to work.

    ReplyDelete

 
Creative Commons License
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.