Specification Updates, Jan 13, 2021
The following features will be added to the spec on the Home page:
- User records should be associated to Person records. People should have at least the following properties: first name, last name, affiliation (place of work), email address, subject areas of expertise (hash of "code": "subject description"), and boolean fields for: deceased, retired.
- In addition to collecting EDI data from the potential participants who indicate their interest in attending, the data must be collected in an anonymized way, but associated with the proposal. It should be made clear to the user that the information will not be directly associated to their identity, but is collected for statistical purposes. These statistics should be available to the organizers of the proposal, as well as staff and review committees.
10.5. Some of the fields on the proposal submission form concern the subject matter of the proposal. Submitters will be asked to choose the AMS Subject Classification that their proposal falls under. They are given a primary subject, a secondary subject, and a third, "BIRS Subject", which is our local list of subjects that change often (see Staff Features #11).
BIRS also maintains a list of experts that it calls upon to do proposal reviews. Each "Reviewer" should have subject area expertise associated to them.
After the Proposal Submitter has selected the subjects of their proposal, they should be asked to select which member of the Scientific Advisory Board (SAB) committee should be the "Handling Editor" for their proposal. Each member of the SAB will have a list of subject areas associated to them, so that the proposals subject areas can be matched with the SAB members' subject areas, and the Proposal Submitter can be shown a list of candidates to be the Handling Editor.
Additionally, the proposal submitter should be able to mark any potential SAB members as "not applicable" for handling this proposal. This could be due to conflict of interest, or some other reason. (Nice to have feature: allow them to attach a note to explain why they marked certain SAB members as "not applicable"). There will be further discussion of the "applicability" of reviewers in the Proposal Reviews section, #...
10.6. Part of the Proposal Submission form will be preferred location for the event to take place, should the proposal be accepted after the review process. BIRS currently has 3, possibly 4 locations. The available locations should be configurable by Staff.
10.7. Another part of the Proposal Submission form will be preferred and impossible dates of the event, should the proposal be accepted. The selection of dates available to choose from is a function of the preferred location. Each location has different weeks of the year available for hosting events. The available weeks in each location also needs to be configurable by staff. i.e., Banff location has weeks 1-48, but Oaxaca location has weeks 20-24, 28-32, etc.. The Organizers will be able to select up to 5 preferred dates and 2 impossible dates -- these will act as constraint parameters to the schedule optimization program that will be run at the end of the review process.
- The review process should proceed in two phases:
a) An initial survey is sent out to Reviewers, asking for initial impressions of the proposals, a "Quick Review". This would just be a textarea for them to leave comments, and form elements for them to recommend someone else to review this proposal (name, email, affiliation).
b) The second phase will be more detailed reviews, where they add a grade to the proposal, as well as comments.
- Proposal reviewers should be presented with a list of submitted proposals that they can review, named their "Work List". Some committee members can see all proposals, and others only the proposals assigned to them. Some may be in a group that allows viewing all proposals, and also have assigned proposals. The assigned proposals should be highlighted in the list of proposals, in some way.
14.5. Proposal reviewers should be able to indicate whether they give consent to sharing their review with the organizer. Staff or Scientific Advisory Board (a Committee) members may want to share a particular review with a proposal submitter, and this field gives them permission from the Reviewer to do that.
14.6. The system should record statistics on Proposal Reviewers: how many reviews they have left, ratio of invitations to review to accepted/declined, how many proposal review cycles (years) they've participated in, and possibly others. The idea is to be able to gauge the reliability of reviewers. The "reliability" score of the reviewer should be indicated in the interface where SAB members or Staff can invite them to do reviews.
-
A button for Staff or certain Committee members that would share a particular review with the Contact Organizer of a Proposal.
-
The review form should allow uploads of files (e.g. PDFs), to be associated with the review.
-
An option for Proposal Reviewers to receive e-mail (or other types) of notifications when their "Work List" changes. i.e. they've been assigned new proposals.
- They should see lists of proposals, and have access to the reviews of proposals. The average scientific grade and average EDI grade for each proposal should be shown, as well as the number of scientific reviews and number of EDI reviews.
2.5. Members of the Scientific Advisory Board (SAB) committee should have the ability to add new Reviewers, and edit existing ones.
2.6. There should be an interface for Reviewers to add/suggest new reviewers. Suggested reviewers confirmed by SAB members or Staff before invitation sent (see Staff Features #8).
12.5. One committee, the Scientific Advisory Board (SAB), should have permission to assign proposals to particular Proposal Reviewers (Staff also have this ability). The relationship between a particular reviewer and the proposal they are assigned should have at least two additional properties:
a) The type of reviewer: One particular reviewer will be marked as a "Handling Editor", who is in charge of the reviews for this proposal.
b) Each Proposal Reviewer's relationship to their assigned proposal should have an "Objectivity" grade, of say, 1-5.
The system should automatically subtract from a Reviewer's objectivity score if, for example, the Reviewer's university affiliation is the same as the university affiliation of the Organizer who submitted the proposal. Or if the Reviewer is on the list of Potential Participants in the Proposal. There will be a list of conditions that affect a Reviewer's objectivity score. Members of the SAB committee should also be able to adjust the objectivity score of Reviewers, as they relate to particular proposals. The SAB members may have special knowledge of conflicts of interest, for example. The Objectivity Score might be called "Arms Length" score -- check with Director.
In the interface, the objectivity score of a reviewer assigned to a proposal should be colour-coded. For example, green for an objectivity score of 5, and red for a score of 1.
- Invitations and replies to external reviewers should be automatically handled, similar to the Workshops RSVP feature described above*. A special link gets emailed to invited reviewers, and they can click the link that goes to a form where they can indicate whether they accept the invitation to be a reviewer, or decline. It should have a text field for them to add a comment, which gets emailed to staff and the "Handling Editor", if there is one assigned to the associated proposal.
8.5. Invited Reviewers should also have a field where they can recommend other reviewers for consideration. The Handling Editor (SAB member) and/or Staff should get a notice if this happens, and there should be an interface for them to allow/decline sending the suggested Reviewer an invitation.
19.5. Log all emails sent by the system, for auditing purposes. Staff will want to know, for example, whether a reviewer was sent an invitation, when, and by whom.
Inviting Reviewers: Early September
Evaluation Committee features: Early October
