NSF CAREER proposal--shredded!

I found out today that my NSF CAREER award proposal was declined. This was completely expected, and as I mentioned in my previous blog I have been looking forward to the constructive feedback that I know I need. Whether consciously or not, though, I now realize I was unrealistically hoping that I would receive mostly positive feedback and a few easy things to improve upon. Thus, my first read-through this afternoon of the detailed reviews was very surprising and wickedly demoralizing. Let me try to think of some other emotions: bewilderment, depression, frustration... OK that's enough. I think the most demoralizing aspect was that a response shared by many reviewers was that the proposal was poorly written and confusing. This must be true, given that the sentiment was shared, but as of now, I have no idea how to overhaul the proposal to correct for this, and I'm dreading having to figure that out.

At this point (5 hours later), I have carefully re-read all of the feedback, and my emotions are much more tempered, and a couple new ones have emerged: hope, gratitude. The gratitude is for the many scientists (5 reviewers, plus program manager, plus panel members) who clearly invested a huge amount of time in reading (and re-reading and deciphering my bad writing) my proposal and composing very detailed and thoughtful feedback. I truly feel badly and frustrated that I can't thank these reviewers personally, due to their anonymity. Really, the only thing I can do is to do my best to follow all of the advice they have given and correct all my mistakes with the resubmission this summer...and I plan on doing this.

Following advice of Jean-Claude Bradley, Cameron Neylon, and others, I have posted most of the potentially interesting and non-confidential parts of the proposal on Scribd. I am new to the site, but I think this is a link to the proposal. WARNING: reading the proposal has been deemed difficult and perhaps a waste of time by at least 5 respected scientists!

I am not going to post the reviewers comments, since I think they had the expectations that I would not share them. I will give a brief summary here.

Overall

• Two panels reviewed the proposal. Both panels ranked it low priority (approximately 50th percentile or worse). Ouch! One panel was the Molecular Genetics panel and the other was a special interdisciplinary panel between Mathematical and Physical Sciences and Molecular and Cellular Biosciences for interdisciplinary proposals such as this.
  
• I have feedback from 5 anonymous reviewers. I don't know how they fit into the two panels, or whether it's a 3rd source of feedback for the program manager. 4 reviewers understood the impact (in my opinion), whereas one reviewer seemed to hate my writing and didn't seem to understand because of that.
  
• All panels and individual reviewers had significant concerns about the proposal. There was no way this thing could have been funded without some kind of empty senate seat under my control.
• Involvement with OpenWetWare and open science was noticed and cited as a big positive for the broader impacts.
  

Shared Major Negative eedback

• Dearth of preliminary data / overly ambitious / seriously concerned about feasibility of any aspect of the proposal. This is very reasonable and somewhat expected. The students have been working hard on this project, we already have some significant preliminary data (related to "Shotgun DNA Mapping") and I expect a lot more before July of this year. I had sent in an update about the shotgun DNA mapping results in October, but based on reviewer feedback, I think that the program manager did not send this update to anyone. Another way I will address this issue is to re-organize the proposal (somehow) to make the feasibility more obvious.
• Poorly written and confusing. This sentiment was pretty much shared by all reviewers and even included in one panel summary. I have re-read the proposal many times since July and never really considered that it was so confusing. I was modeling the format after a couple funded CAREER awards that I had seen (in other fields, though). Well, clearly I need to improve this somehow and I don't have any good ideas. I feel bad that I caused so much trouble and wasted so much time of the thoughtful reviewers. One of the reviewers basically admitted to giving up, which I don't really blame him, given the comments from the others. I feel like I've been told that my baby is ugly and just can't see it. BTW, my kids are fantastically cute.
• Too many specific aims / third aim not well developed. I can see that this is true. One thing I stupidly didn't realize is that the reviewers basically expect NIH-style things such as "specific aims." I had actually suspected it would be viewed negatively if made it look too "NIH-ey." I will think a lot more about this, but right now what I'm thinking is that I should focus the science solely on Pol II transcription (ditch the DNA damage repair), and then reorganize the aims to be (1) shotgun DNA mapping (2) unzipping in vitro Pol II complexes and (3) mapping pol II and nucleosomes on native chromatin. This would be organized from most feasible to least feasible.
  

Shared Positive Feedback

• Most of the reviewers specifically mentioned that involvement with OpenWetWare was a big positive for the broader impacts. This was really good to see, and perhaps the most notable thing to people who happen to read this blog.
• All reviewers agreed that the impact would be very high (if successful). Further, I think some were even intrigued / excited about the idea. Many of the reviewers understood and seemed to agree with a main point I was trying to sell: that the combination of single-molecule analysis with yeast genetics is a potentially powerful and unique tool for chromatin biology.

I suppose this positive feedback is actually sufficient, if I can successfully correct the very serious flaws. After all, trying to look at things optimistically, I'd be in a much worse situation if the reviewers had thought the proposal were very well written and feasible, but lacking potential impact.

Overall, writing this blog entry has been therapeutic and also constructive. And if you do happen to skim any of the proposal, I welcome harsh criticism--I can stomach it!

SJK Note added Feb. 5, 2010:  I did end up submitting again in 2009.  The quick report is that I rewrote it in NIH style, and improved writing thruoghout.  The reviewers liked the science, but lacked confidence in me as a PI, due to lack of preliminary results.  Fair criticism that I think I can correct this year.  Most importantly: the panel LOVED open science as broader impacts.  Here're some links for my 2009 NSF CAREER proposal:

• FriendFeed thread discussing broader impacts feedback.
• Slide 6 of this presentation shows some of the positive reviews of open science from panel.
• Link to 2009 proposal on Scribd.

Koch Lab NSF CAREER Award Intro--Single-Molecule Analysis of Yeast Chromatin Remodeling

Any day now I should be hearing the results of my NSF CAREER award application. I've talked about this in one of my earlier blogs, where I pointed out that there is very little chance the grant will get funded--this is due to overall odds and the fact that it's my first submission. The latter point reduces the odds in two ways. First, I've been told by numerous sources that there is a bias (whether intended or not) towards 3rd-submission proposals (by rule, the final submission). The other reason it being my first submission reduces my chances is likely by far the most strong -- I have a lot to learn from the feedback I will receive! My resubmission in July 2009 will be much stronger due to this critical feedback as well as the very cool preliminary data my students have been acquiring (and which I hope to start telling you about in a couple weeks).

(As an aside, I feel compelled to insert this note: If you're my NSF program manager and you happen to have the time to to read this random blog, and you are in the middle of dialing my number to tell me that my grant was funded, and are now having second thoughts after having read that paragraph above, PLEASE KEEP DIALING!)

Aside from the fact that I lack some talents helpful for grant writing--and therefore find it painful--I did enjoy aspects of the NSF CAREER award writing. This is because the award is, by design, tailored towards research university professors and has required components on research, education, broader impacts, etc. This made me think deeply about my whole career (research, teaching, service, family) and even now when I go back and read my proposal (6 months later), I am excited about the integrated career plan. One particularly satisfying aspect was that I was able to include (admittedly vague) plans for open science in a very natural way and aligned with the mission of NSF.

So, I am really looking forward to the feedback about this proposal, and I think it will be the most helpful feedback I will have received so far. Up to this point, I have received some helpful critical feedback from the DoD Breast Cancer Research Program (BCRP) Idea Award (although that backfired on me...subject of another post), some internal awards, and that's about it. Many of the other programs--private foundations--provide little or no feedback (usually none) as a matter of policy, which makes the rejection somewhat less rewarding (ha ha). I am still considering the idea of posting my whole proposal (perhaps on Scribd per Cameron Neylon and Jean-Claude Bradley's suggestions). Until I figure that out, I figured I'd post the first page of my proposal, which gives an overview of my career plan 6 months ago. Please give me any negative or positive feedback you have!

CAREER: Single-Molecule Analysis of S. Cerevisiae Chromatin Remodeling

Career Mission

The long-term mission of our laboratory is to enable important discoveries in molecular cell biology by innovating new biophysical methods and culturing interdisciplinary research and education partnerships. The specific research goals of this 5-year proposal are to develop new methods for single-molecule analysis of DNA and chromatin extracted from living yeast cells, and thus open a new research area that combines the powers of single-molecule and genetic approaches. Our long-term mission is also supported by our specific educational and broader impacts goals in this proposal, which will multiply the impact of our research goals. These include participation in Open Science, integration of research and university education, and community and local school outreach. Together, these research and education goals in this proposal will establish a successful career path for this PI as leader of an exciting biophysics research laboratory collaborating with leading chromatin and transcription biologists, successful university educator, and recruiter of underrepresented minorities to research careers.

Research Overview

DNA in eukaryotic cells exists as chromatin, which is repeating units of DNA wrapped around histone proteins. These DNA-histone units are called nucleosomes, and play a fundamental role in both positive and negative regulation of proteins that require access to the DNA code. Cells have a variety of enzymes that can modify the structure of the chromatin by moving, removing, or adding histones, or by modifying specific amino acid residues on the histones. This chromatin remodeling affects the ability of other proteins to access the DNA and has a profound impact on critical processes such as DNA repair and gene transcription by RNA polymerase. Understanding of these dynamic processes is currently hampered by the inability to characterize with high spatial and temporal resolution the changes to chromatin inside living cells. Therefore, we are developing biophysical tools with single-molecule sensitivity to address this need. One of the main goals we are pursuing is to develop a single-molecule method (see Fig. 1) for mapping proteins on chromatin that will far surpass the capabilities of the currently most powerful technique—chromatin Immunoprecipitation (ChIP). We are also pursuing other goals for analysis of genomic DNA and chromatin with optical and magnetic tweezers and nanostructured devices. The unifying themes of our research goals are single-molecule analysis and chromatin biology, and we are seeking to build a career foundation in a new arena of single-molecule biophysics applied to in vivo systems.

Using Dynamic Page Listing (DPL) with MediaWiki to help with research goals and tasks

For about a year now, I have been using Dynamic Page Listing (DPL) for a variety of purposes on our private lab wiki (hosted by OpenWetWare). I find it very useful for keeping track of ideas, grant writing, and assigning goals and tasks, among other things. In this post, I want to talk about using DPL for goals and tasks, which I think is a system that could help out other people who want to carry out open science using MediaWiki with the DPL extension installed.

I met with an ECE undergraduate student today, Caleb Morse, and we decided to try out a small "open research" project on OpenWetWare. We are following advice in responses to my earlier blog (see the friendfeed discussion), and are going to try to do a small project completely openly. Thus, for this project we're starting from scratch and going to try to record everything on the wiki and other various public resources (to be determined: blogs, etc.). Our hub for this project will be this page on our lab wiki, and as we get it going, I'll describe it in a future post. We're going to use DPL Goals to help us in this project, and so today I set them up for Caleb on OWW and I also wrote an OWW help page and an example user to help others who may want to try it out.

What are DPL Goals?

What this system amounts to is really just a method for tagging and displaying tags with simple "AND / OR" logic. The only reason I have to write this article is because there isn't yet (as far as I know) a good way for tagging and good, simple tag filtering in MediaWiki. DPL is really good at doing this, but it's a powerful and complicated extension, so it's a little confusing to set up the system I'm describing here. The system basically works like this:

• Pages are created to describe goals or tasks. Categories are added to these pages to (a) mark them as a goal, (b) assign them to a specific person, and (c) optionally categorize them further, such as "high priority."
• Task List or Goal List pages are created to display any pages across the entire wiki database which have been marked as goals or tasks for a specific user. This listing page updates dynamically whenever another page on the wiki is added.

The following figure shows a screen shot of an example page for listing goals and task pages. Each of the links is to a separate page somewhere on the wiki (it doesn't matter where) that can have a lot of detail about a research goal, or could have no text at all, with the title simple being a task reminder.


There are four lists presented in the previous figure. The wiki code for this page has four blocks of DPL code to generate the lists automatically and dynamically...as tags (categories) are added or removed from pages around the wiki, the list will be updated (when the page is refreshed). Two of these blocks of DPL code are shown in the next figure.


You can see that the code isn't too complicated, but it's not completely straightforward either. I should note that this is just one particular way of displaying pages. DPL is quite powerful, both in it's extraction of pages (and bits of pages) and in the ability to format the results. What I am showing is a very simple usage with just page titles that tends to work well for me.

The DPL code necessary for generating the display page is a little bit complicated, but only needs to be generated once (and you can tweak the display later if you like). In contrast, the wiki code needed to make new pages show up in the list is very simple--This is why I think the method can be helpful to me and potentially others. I show in the next picture the basic use of MediaWiki categories, which is not too complicated.


The use of templates, which I won't show here, can make it even easier. The template method is actually what I use, and it's exceedingly easy to remember and quick to use. Furthermore, if OWW or your site decides to make tagging easier (check boxes, or tagging like on other sites), the method should get even easier. Along those same lines, what I'm describing is still a kludge and a much better method may already exist or may be introduced soon. I think it's easy enough to do and has been so helpful to me that I think it's worth implementing if you don't have a better option.

What next?

As I mentioned above DPL is very versatile, and what I'm describing is a very limited application that has been very helpful to me. I use the same basic paradigm for many purposes on our private and public wiki. Many of the pages on OpenWetWare tend to be long articles, in the style of wikipedia. On the other hand, scientific research tends to generate bits of information here and there--this makes it much more convenient for me to make small entries that have some tags (aka labels, categories) so I can aggregate them on some other page. I'd say I spend 80% of my private wiki time with this method. For example, when grant writing, I'll take notes without worrying where I list the page, and then I'll just add an "NIH R01" tag to the page so I can find it later. If you use MediaWiki and especialy OWW, I think you'll like this method. Here are two more links to DPL uses I have on OWW public wiki:

• An aggregation of ideas I have for possibly improving our Junior Lab course next year.
• Various pages with ideas for improvements to the wiki (I don't really use this very often, but it's a good example of what can be done easily).
  

Posting our research principles on OpenWetWare

Image from wordle.net
Welcome to my research blog! If you've read any of my teaching or science blog, you may know that I made the highly questionable decision of having three separate blogs. So far, this decision is bothering me, and not a single post has fallen nicely into any of those areas. I'm still hoping this isn't a big deal or too annoying to any readers I may have.

This research blog has been the hardest one to start, because I am still wrestling with issues of open science. Being partially open is difficult (and this time consuming and distracting difficulty is a big argument in favor of being completely open). But I am still concerned about taking a full-on dive into open science. I truly feel that my main worries are the risks it presents my students and (and to a much lesser extent, my collaborators). This risk arises because the students' careers are tied strongly to the success of our lab. So, any risks to our publications, and particularly our funding directly impact the students (and of course my career). It's actually not a fear, but I think it's a consciously planned precaution. My instinct, actually is to just say "f it" and to immediately put everything into the open--my gut tells me that this would actually lead to much more and quicker research success. I think my students would mostly agree too. However, I think this may add risk to our lab's success. My students are already taking a big risk working with the new un-tenured, un-funded guy...so I'm hesitant to pile on extra risks. Thus is my justification for our open science baby steps.

Given that sort of thought process, I thought my first blog post here could be about our research lab principles (including thoughts on openness), as opposed to directly about the science we are pursuing. To do so, I posted on our public wiki an old (very old by web standards) early 2007 draft of a lab philosophy that I have had posted on our private wiki. I'm hoping it can evlove into a set of principles that more than just me try to improve on.

I always cringe when I re-read something I've written some time ago (so I guess I should group all of my blogs together into one called "Steve Koch Cringes"). This draft of our lab principles is no exception. But I'm resisting the urge to edit it now and I've just posted it as is. If nothing else, it's a good snapshot of what was going through my mind as a new, confused, but concerned professor. Now I am a rapidly aging, still confused, still concerned professor...whose having a lot of fun with all of the great students he has.

I feel like it's a waste or confusing to copy over the entire post to this blog entry. I also fee like it would be difficult to relink everything, so if you're interested, please do follow this link.