Specification Updates, March 5, 2021
This feature was only briefly mentioned in the Proposal Submission section, under #11. "BIRS also maintains a list of experts that it calls upon to do proposal reviews. Each "Reviewer" should have subject area expertise associated to them."
This feature is flushed out here, and is now referenced in the Proposal Reviews section, #18.
An important piece of functionality will be the database of Proposal Reviewers. This database should include their names, email addresses, affiliations -- normal Person records -- but must also include their areas of expertise, their general availability (are they retired? deceased?), and scoring of their past performance as reviewers.
The American Mathematical Society (AMS) maintains a list of mathematical subject codes and subject descriptions that are intended to encompass the range of subjects in the mathematical sciences. The reviewers should be "tagged" with their areas of expertise. In addition to the AMS subject codes, BIRS maintains it's own list of subjects and codes, which are more specific and up-to-date with the cutting edge of research in the mathematical sciences.
Proposals currently have 3 subject code fields: Subject 1, Subject 2, and BIRS Subject. We ask proposal submitters to select a primary AMS subject, a secondary AMS subject, and a BIRS Subject. This scheme should also apply to Proposal Reviewers.
The BIRS legacy database has a group of about 4000 people who are invited to give proposal reviews. However, they currently have no subject codes attached to them. BIRS is seeking to acquire an additional database of potential scientific reviewers, with up-to-date associations to areas of expertise. The new Proposals app should be able to import this data for use in its review process.
The app should automatically suggest a list of potential reviewers for each proposal, based on the subject code matching. BIRS Staff should be able to click a button which sends an email to the suggested reviewers, inviting them to review all of the proposals that match their expertise. The current legacy system does this, but without the subject-code matching, reviewers must be manually assigned to proposals by the BIRS Director -- a laborious process.
The invitation email that is sent to potential proposal reviewers should include a special OTP URL that grants access to the proposals online, without the need for them to setup an account.
The majority of the potential reviewers in the current BIRS database do not actually participate in the review process, despite being invited to every year. There needs to be a way to track which invited experts actually participate in the review process, so that the list can be kept up-to-date.
In addition to tracking which reviewers are actually participating, the quality of their participation also needs to be tracked. Are they leaving only 1 review out of the 5 assigned to them? What is their level of engagement? Are the reviews they leave adequate? i.e. adding scores, but no comments. Or adding one sentence comments instead of in-depth analysis.
Journal review systems such as EditFlow have a ranking and scoring system for scientific reviewers, so that the journal editors can gauge the quality of reviews the can expect of the invited reviewers.
