NSERC reviewing: giving out federal science funding

This year I am serving on the one of the Federal granting councils that awards the basic NSERC grant to computer science researchers. It's a lot of work but it has already provided and interesting overview of the research scene in Canada. As opposed to a typical conference program committee or journal editorial board, this kinds of applications is very diverse both with respect to research topic and level of seniority.

The first phases of the process involved the identification of possible conflicts of interest and the subsequent assignment of reviewers. It is streamlined to work very smoothly and to be as fair as possible. Since each candidate proposes a set of reviewers, we try and use some of their suggestions, as well as throwing in a few other reviewers they have not suggested. This addresses the possibility that the suggested reviewers are all biased to be especially favorable, which is probably a commonplace selection bias.

One obvious reality is that via the assignment of biased reviewers one could indirectly bias the outcome unfairly. This is true with any reviewing process and this is no exception. The very diversity of the applications seem to make bias even more of a risk since for some esoteric research areas there might not be multiple applications to compare againt one another, and the match between the proposals and the particular specialities of the panel are not as finely tuned. Also, unlike a conference, the results of this process cannot be overturned for a long time (the review cycle is typically every 5 years). All this makes the assignment of reviewers very time consuming, but rather interesting. All in all I spent a lot more time considering and assigning reviewers that I had expected or than I might have done in other contexts. That was especially painful since I had other issues keeping me busy.