14 October

This year I am serving on the one of the Federal granting councils that awards the basic NSERC grant to computer science researchers. It's a lot of work but it has already provided and interesting overview of the research scene in Canada. As opposed to a typical conference program committee or journal editorial board, this kinds of applications is very diverse both with respect to research topic and level of seniority.

The first phases of the process involved the identification of possible conflicts of interest and the subsequent assignment of reviewers. It is streamlined to work very smoothly and to be as fair as possible. Since each candidate proposes a set of reviewers, we try and use some of their suggestions, as well as throwing in a few other reviewers they have not suggested. This addresses the possibility that the suggested reviewers are all biased to be especially favorable, which is probably a commonplace selection bias.

One obvious reality is that via the assignment of biased reviewers one could indirectly bias the outcome unfairly. This is true with any reviewing process and this is no exception. The very diversity of the applications seem to make bias even more of a risk since for some esoteric research areas there might not be multiple applications to compare againt one another, and the match between the proposals and the particular specialities of the panel are not as finely tuned. Also, unlike a conference, the results of this process cannot be overturned for a long time (the review cycle is typically every 5 years). All this makes the assignment of reviewers very time consuming, but rather interesting. All in all I spent a lot more time considering and assigning reviewers that I had expected or than I might have done in other contexts. That was especially painful since I had other issues keeping me busy.

By Gregory Dudek at | Leave a comment |    
There are no comments.
Please send trackback to:/blog/81/tbping
There are no trackbacks.
Post your own response

Each comment is manually screened for the presence of appropriate and substantive content, due to a constant onslaught of comment-spam. This means there may be a delay before your comment appears.

(Some kind of name is required, will be visible)

Required, whatever you enter will be visible to other users.

(Optional, used for "mailto" link)

Your email address is not required, but if you insert it it will be displayed so people can contact you.

Answer this question correctly to demonstrate that you are not a dumb spambot.

The title for your comment.

Your comment goes here. All relevant comments are welcome, except for those that simply promote an irrelevant product or else are used to fraudulently inflate the link count to an irrelevant web page. They appear after moderation. Don't forget to also fill in the captcha below or your text will be rejected automatically!

You must answer this question to prove you are human
The governator said (when in a movie as T2): 'I'll be ... (pick a number): 1:tuckered, 2:broke, 3:back, 4:broken, 5:swartzneggered, 6:eaten, 7:you?

Answer this question correctly to demonstrate that you are not a dumb spambot.