Home
"Proposals" will be a web application for accepting proposals for scientific meetings, such as workshops or conferences, and facilitating the peer-review and selection process at The Banff International Research Station (BIRS) for Mathematical Innovation and Discovery.
BIRS opens its Call for Proposals at the start of every summer, soliciting the global scientific community to submit proposals for workshops in all areas of science that intersect with mathematics. It receives 200-300 proposals, which are then reviewed by around 1000 experts on respective subject matter.
At the end of the review process, several committees of 5-20 people, evaluate the proposals and the reviews on them, for various criteria, to select the best 50 - 75 of them, which will become workshops at one of the BIRS research stations, two years hence.
The Guidelines for Proposal Submission may be helpful for getting an idea of what the application needs to accomplish. Although, this new app is meant to be a major improvement over our existing process.
Since the organization already uses Ruby on Rails for it's conference management application, Workshops, RoR is a good choice of framework to develop Proposals with. The application should run in a Docker container, on Ubuntu Linux. It will be hosted on a virtual server at the University of British Columbia.
The application should have the following features:
-
A login / user account system that connects to an identity service provider (IdP), such as Keycloak.
-
Registration of new users.
-
User records should be associated to Person records. People should have at least the following properties: first name, last name, affiliation (place of work), email address, subject areas of expertise (hash of "code": "subject description"), and boolean fields for: deceased, retired.
-
Groups: people can be members of various user-defined groups. For example: Staff, Admin, Reviewers, Submitters, and various Committees. Staff should be able to add and remove groups, and add and remove people from the various groups. People can be members of many groups. Committees (groups) can have sub-committees (sub-groups), that inherit permissions from their parent group.
-
There should be role-based access controls throughout the application for groups. For example, a member of the Scientific Review Committee may be able to access a "Scientific Merit" grade, and a comment field for scientific reviews, and a member of the EDI Review Committee won't see those fields, but will see different ones, relevant to EDI reviews.
The core functionality of the application is to accept proposal submissions from Internet users. This involves:
-
Login or registration, as per above.
-
Several users can submit a single proposal together (collaborators). One of them is assigned the "Contact Organizer" role, 2-4 others are supporting "Organizers".
-
Proposal submitters must select the type of proposal they are submitting. i.e. proposal for a 2-Day Workshop, a 5-Day Workshop, a Summer School, a Research in Teams group, or a Focussed Research Group. Staff should be able to add new proposal types.
-
Each type of proposal will have it's own unique submission form, with different fields.
-
Each submitted proposal gets assigned a unique code, in the format [2-digit year][proposal type code][3-digit-integer]. For example, the first submitted 5-Day Workshop for 2023 would have the code 23w5001. "23" is the year, "w5" is for 5-Day Workshops, and "001" is unique to this proposal.
-
Submission forms will have some mandatory and some optional fields, configurable by staff.
-
Staff users should be able to specify the fields on each form, whether they are mandatory, and the viewing privileges of committee members for each field. e.g. members of the "Foo" committee can see all fields of 5 Day Workshop proposals, but members of the "Bar" committee can see only fields A, B, C, F and G.
-
Text fields should have an editor that supports Markdown and LaTeX. The LaTeX editor could be embedded in the web form, like TeXStudio, or it may have to run as a separate service, such as CoCalc or Overleaf.
-
Submitters should be able to view the rendered version of their LaTeX document (i.e. in PDF), ideally in real-time as they write it.
-
In addition to collecting EDI data from the potential participants who indicate their interest in attending, the data must be collected in an anonymized way, but associated with the proposal. It should be made clear to the user that the information will not be directly associated to their identity, but is collected for statistical purposes. These statistics should be available to the organizers of the proposal, as well as staff and review committees.
-
Some of the fields on the proposal submission form concern the subject matter of the proposal. Submitters will be asked to choose the AMS Subject Classification that their proposal falls under. They are given a primary subject, a secondary subject, and a third, "BIRS Subject", which is our local list of subjects that change often (see Staff Features #11).
BIRS also maintains a list of experts that it calls upon to do proposal reviews. Each "Reviewer" should have subject area expertise associated to them.
After the Proposal Submitter has selected the subjects of their proposal, they should be asked to select which member of the Scientific Advisory Board (SAB) committee should be the "Handling Editor" for their proposal. Each member of the SAB will have a list of subject areas associated to them, so that the proposals subject areas can be matched with the SAB members' subject areas, and the Proposal Submitter can be shown a list of candidates to be the Handling Editor.
Additionally, the proposal submitter should be able to mark any potential SAB members as "not applicable" for handling this proposal. This could be due to conflict of interest, or some other reason. (Nice to have feature: allow them to attach a note to explain why they marked certain SAB members as "not applicable"). There will be further discussion of the "applicability" of reviewers in the Proposal Reviews section, #...
-
Part of the Proposal Submission form will be preferred location for the event to take place, should the proposal be accepted after the review process. BIRS currently has 3, possibly 4 locations. The available locations should be configurable by Staff.
-
Another part of the Proposal Submission form will be preferred and impossible dates of the event, should the proposal be accepted. The selection of dates available to choose from is a function of the preferred location. Each location has different weeks of the year available for hosting events. The available weeks in each location also needs to be configurable by staff. i.e., Banff location has weeks 1-48, but Oaxaca location has weeks 20-24, 28-32, etc.. The Organizers will be able to select up to 5 preferred dates and 2 impossible dates -- these will act as constraint parameters to the schedule optimization program that will be run at the end of the review process.
-
The ability to upload file attachments, such as images that might be referenced in the LaTeX document.
-
Collaborative editing, so that several people submitting a single proposal can work on the document together. For this, integrating something like Overleaf would be appropriate.
-
Calling up previous proposal submissions as a template for a new submission. Proposals previously submitted by a user, but were rejected after peer-review could be called up, and feedback from review committees could be viewed. Then a "resubmit" button would open the submission form pre-filled out with the data from the old proposal.
For each proposal, one committee member will be designated as the "Handling Reviewer" (HR), who will take responsibility for all of the other reviews of that proposal. The HR should be able to assign or remove assignments of reviewers to the proposal, and has special access for suggesting communications with the submitting organizers of the proposals they are responsible for (see #).
-
Proposal reviewers should be presented with a list of submitted proposals that they can review, named their "Work List". The last name of the person designated as the proposal's Handling Reviewer should be mentioned in the proposal list. Some committee members can see all proposals, and others only the proposals assigned to them. Some may be in a group that allows viewing all proposals, and also have assigned proposals. The assigned proposals should be highlighted in the list of proposals, in some way.
-
The review process should proceed in two phases:
a) An initial survey is sent out to Reviewers, asking for initial impressions of the proposals, a "Quick Review". This would just be a textarea for them to leave comments, and form elements for them to recommend someone else to review this proposal (name, email, affiliation).
b) The second phase will be more detailed reviews, where they add a grade to the proposal, as well as comments.
-
Proposals should be sortable by event code (each submission is assigned a unique code), title, or subject code. There should also be a toggle to hide all but assigned proposals. Sorting and viewing options should be remembered (cookies?).
-
The proposal list should indicate which proposals they've already reviewed, and which ones have reviews in progress but are not marked as complete.
-
Some proposal reviewers prefer doing reviews in printed format, so an easy way to compile a PDF of individual, all, selected, or assigned proposals should be provided.
-
Proposals should be presented online in the same order that they are printed in the book of proposals.
-
There should be prominent “review” buttons or links in the list of proposals, so that reviewers can immediately open the review form for that proposal (instead of needing to click on the proposal first, and then review).
-
There are at least 2 types of review: reviews for scientific merit, and reviews for EDI considerations. The review form presented depends on the user's committee membership. EDI reviewers get a form for reviewing EDI merits, and scientific reviewers get a form for reviewing scientific merits. Some users are members of multiple committees, so should be presented with both forms.
-
The review form for scientific reviews should have a grading feature, so that the reviewer can rank the strength of the proposal on a scale of 1 to 5. It should also have a text area (markdown editor) for the reviewer to leave comments on the proposal.
-
The review form for EDI reviews needs 3 grading sections, and 3 text areas.
-
The review form should save the text entered into the text fields automatically, in case the user accidentally leaves the page before saving, or their computer crashes, etc..
-
There should be a "switch" to indicate review in progress / review complete.
-
In the review form interface, there should be a “Go to Next Assigned Proposal” button, to bring up the next assigned proposal for review. We could also show a list of assigned proposals beside or below the review form, so that they could select the proposal to review next, or perhaps they just want to see a previously reviewed proposal.
-
Information from a proposal's content should be presented to reviewers in a way that is tailored to the user's committee membership. For example, members of an "EDI review" committee are interested mainly in the EDI characteristics of the proposal, so fields of the proposal that are relevant to EDI should be most prominent, with statistical summaries of the information. i.e., number of women in the proposed event, number of visible minorities. This should be consistent in printed and online versions.
-
Proposal reviewers should be able to indicate whether they give consent to sharing their review with the organizer. Staff or Scientific Advisory Board (a Committee) members may want to share a particular review with a proposal submitter, and this field gives them permission from the Reviewer to do that.
-
The system should record statistics on Proposal Reviewers: how many reviews they have left, ratio of invitations to review to accepted/declined, how many proposal review cycles (years) they've participated in, and possibly others. The idea is to be able to gauge the reliability of reviewers. The "reliability" score of the reviewer should be indicated in the interface where SAB members or Staff can invite them to do reviews.
-
Staff should have the ability to hide/show various proposal fields, according to the group/committee that the reviewer is a part of.
-
Embedded LaTeX editor in the review field, so that the reviewer can use mathematical expressions in their review.
-
Staff should have the ability to send a special link with a code in the URI to a reviewer, that grants the reviewer access to the review system to see and review their assigned proposals, without having to setup an account.
-
A button for Staff or certain Committee members that would share a particular review with the Contact Organizer of a Proposal.
-
The review form should allow uploads of files (e.g. PDFs), to be associated with the review.
-
An option for Proposal Reviewers to receive e-mail (or other types) of notifications when their "Work List" changes. i.e. they've been assigned new proposals.
-
A feature to summarize data from past proposals, in reference to the current proposal. For example, when viewing a proposal, show a histogram of how many proposals that have the same subject codes have been submitted in the past. How many of those were accepted, and rejected? Arrange by year. (For details inquire about "Louigi's suggestion").
The Program Committees are groups of users who evaluate the reviews, to determine which proposals are to be accepted or rejected.
-
Program Committee members should be able to register and login.
-
They should see lists of proposals, and have access to the reviews of proposals. The average scientific grade and average EDI grade for each proposal should be shown, as well as the number of scientific reviews and number of EDI reviews.
-
Members of the Scientific Advisory Board (SAB) committee should have the ability to add new Reviewers, and edit existing ones.
-
There should be an interface for Reviewers to add/suggest new reviewers. Suggested reviewers confirmed by SAB members or Staff before invitation sent (see Staff Features #8).
-
Users can be on both review committees and program committees. (Question for Director: should their own grades be excluded from the averages?)
-
Some people are on multiple review committees, for example committees that are assigned to evaluate EDI reviews and committees assigned to evaluate scientific reviews. For reviewers who are members of multiple committees and may evaluate reviews of both types, the review evaluation form should present a clear separation between the types of reviews that they are evaluating. Also note that one reviewer may be on multiple review committees, so could leave more than one type of review.
-
Committee members should be able to leave feedback for proposal submitters whose proposals are rejected, to help them improve their proposals for the next competition.
-
There should be a printing (to PDF) feature to compile a book of proposal reviews. The order of the printed proposal reviews should be consistent with the printed book of proposals.
-
Program committee members should be able to leave notes on proposals, and on reviews.
-
Program committee members should be able to rank proposals, from 1 to the total number of proposal submissions. There should be a way to order the proposals by rank.
-
Program committee members should be able to mark proposals as rejected, accepted, or maybe.
-
Whether these annotations, rankings, and accepted/rejected states are viewable by other members, or the Director only, or if all but the accepted/rejected status should be private, should be considered.
-
There should be a mechanism for the Director or Staff member to mark evaluated proposals as either accepted, tentative, or rejected.
-
After the program is finalized by the Director, emails should be sent to organizers with a link to a form in Proposals that allows them to select their preferred and impossible dates for their event.
-
One committee, the Scientific Advisory Board (SAB), should have permission to assign proposals to particular Proposal Reviewers (Staff also have this ability). The relationship between a particular reviewer and the proposal they are assigned should have at least two additional properties:
a) The type of reviewer: One particular reviewer will be marked as a "Handling Editor", who is in charge of the reviews for this proposal.
b) Each Proposal Reviewer's relationship to their assigned proposal should have an "Objectivity" grade, of say, 1-5.
The system should automatically subtract from a Reviewer's objectivity score if, for example, the Reviewer's university affiliation is the same as the university affiliation of the Organizer who submitted the proposal. Or if the Reviewer is on the list of Potential Participants in the Proposal. There will be a list of conditions that affect a Reviewer's objectivity score. Members of the SAB committee should also be able to adjust the objectivity score of Reviewers, as they relate to particular proposals. The SAB members may have special knowledge of conflicts of interest, for example. The Objectivity Score might be called "Arms Length" score -- check with Director.
In the interface, the objectivity score of a reviewer assigned to a proposal should be colour-coded. For example, green for an objectivity score of 5, and red for a score of 1.
-
A feature for a proposal's Handling Reviewer ("handling editor" in journal parlance) to request a change to the proposal, to be sent to the proposal's submitter (Organizer). Staff must approve the message before it is sent to the organizer, with the ability to edit the message (but keep the original note from the handling reviewer). The organizer gets the request, with a deadline for responding. The Handling Reviewer is notified when the organizer submits a revision to the proposal. The Handling Reviewer can indicate "accept", "reject", or "other" with respect to the update from the organizer, and has a textarea for leaving comments on the update.
-
Every proposal should have attached to it the entire history of every message sent in relation to that proposal, every grade, every update, comments from everyone, etc, -- in an easily accessible interface. See the following screenshot from the EditFlow journal review system for an example. Contact Malabika for more details about Editflow-like features.
- A reporting feature to generate a PDF for each proposal that includes the proposal itself, all reviews of the proposal including grades and comments, and a summary of the "scores", as shown in Staff Features #4. These reports are for Program Committee members who like to work on printed materials instead of online.
16x. The open-source collaboration platform Mattermost is an open-source alternative to Slack. It could be setup as official BIRS groupware, and integrated into both Workshops and Proposals. With the help of separate identity provider software, we could have single-sign-on, so that one account could be used for all BIRS’ services.
-
Staff should have the ability to specify that certain proposals must take place on certain dates. Also that certain proposals must always be scheduled the week after another proposal. Or in the case of half-workshops, to specify that a pair of half-workshops should always be scheduled the same week.
-
After proposal submitters have selected their preferred and impossible dates that the event could take place, and staff have applied any scheduling options, there should be a button that submits the accepted proposal unique codes, along with their preferred and impossible dates, and staff scheduling options, to an external service for schedule optimization.
-
The schedule optimizer returns several versions of the optimized schedule (hours later). The scheudle options should be presented within the Proposals online system, in a way that helps the Director/Staff efficiently make a selection. Making a selection would involve an "Apply schedule" button, which would create new events in Workshops.
-
Display a table with the following data: proposal code, title, organizer name, handling reviewer's name, a summary of the review statistics, and the committee member's rank for the proposal. Sortable and searchable. A print to PDF feature. An export to CSV feature. (For details inquire about "Chee's spreadsheet").
-
A post-review process report, summarizing how many proposals were submitted, how many reviews were done, by how many reviewers, and compared with the same stats from previous years. (For details inquire about "Nassif's report").
-
Admin staff need an efficient, cohesive, and intuitive interface for managing all types of users of the proposals systems: reviewers, applicants, committee members. Communications with various groups should be straightforward and easy.
-
The new interface should allow for fast search and at-a-glance associations of people to proposal assignments, committee memberships, proposal submissions, review submissions, and account details. Moving between proposals, reviews, and people associated with them should be as seamless and integrated as possible.
-
There should be a quick way to select proposal reviewers and email them an invitation to be a reviewer, which they can accept or decline by clicking on a link, much like the BIRS RSVP system in Workshops* currently does.
* The BIRS RSVP system allows users to send e-mail invitations to selected people. They receive a special one-time-password link that takes them to a Yes/No/Maybe form to indicate if they will attend, etc..
-
The system should be able to automatically or at least very easily send reviewers their assigned proposals, in PDF format. The order of the proposals in the PDFs should be consistent with what they are presented with online.
-
Invitations and replies to external reviewers should be automatically handled, similar to the Workshops RSVP feature described above*. A special link gets emailed to invited reviewers, and they can click the link that goes to a form where they can indicate whether they accept the invitation to be a reviewer, or decline. It should have a text field for them to add a comment, which gets emailed to staff and the "Handling Editor", if there is one assigned to the associated proposal.
-
Invited Reviewers should also have a field where they can recommend other reviewers for consideration. The Handling Editor (SAB member) and/or Staff should get a notice if this happens, and there should be an interface for them to allow/decline sending the suggested Reviewer an invitation.
-
An interface for managing email templates should be provided. Possibly a separate API-based app for managing email templates, automated scheduled sending, and other features. Workshops is in need of this application as well, so it would be a way to avoid replicated code, and extract complexity from both Proposals and Workshops.
-
There should be an email scheduling feature (possibly part of separate app, as per #9). Admin staff should be able to set a schedule to automatically send selected email templates to given people or groups. This could be based on date, or on when certain conditions are met. For example, schedule a reminder email to all invited External Referees who have not responded to the initial invitation, two weeks after it was sent.
-
Staff need an easy way to print a book of all proposals in a given year, or selected groups of proposals. The printed proposals should be organized into chapters, by subject. The software needs an intuitive interface for staff to manage subjects and their groupings into book chapters. Perhaps a visual interface of the table of contents, that allows drag & drop re-ordering of the subject chapters. The interface should allow for the creation of new chapters that group several subjects into one: subject-groups. For example, moving all proposals with subject “Applied Computer Science” and those with subject “Theoretical Computer Science” into a new chapter named, “Computer Science”. Note that the book chapter subject groups should not change the proposal's subjects; it needs to be managed separately.
-
The interface should have printing for both proposals and reviews. Options for printing, such as “Show the identity of reviewers” or “type of proposal/review: EDI, Scientific”, should be presented in a similar way to how operating systems present printing options: first select what to print, then select what options to print with.
-
Staff should be able to validate all new reviews, and mark them as “OK” before they appear in printed reports or to the evaluation committees. Staff should get an email notification if a review changes after it has been marked OK.
-
Staff should have the ability to set schedules for openings or closures of the proposal submission form, or of logins for various user groups/committees, and of various features within the system. For example, the ability to write proposal reviews opens and closes on certain dates.
-
Staff should be able to assign visibility to fields of the proposal to different reviewer groups. Some committees should see fields related to EDI, whereas other committees do not need to see that. Staff should be able to set visibility parameters. i.e. Members of EDIB committee can see these proposal fields: organizer names, participant list, etc., but not these fields: …
-
Staff need an interface for selecting schedule optimization options, such as location of the events, number of program weeks, date of first week, excluded weeks (e.g. Oaxaca takes July off for summer vacation, and weeks with U.S. & Canadian Thanksgiving). Once options are selected, an “Optimize Schedule” button, when clicked, should send the data to the schedule optimizer (external service).
-
Staff also need to set scheduling options for certain proposals. These could be part of the proposal record itself. For example, that one proposal should always be scheduled after a given other proposal, or that a half proposal should always be scheduled the same week as another half proposal.
-
The schedule optimizer sends back sets of results, proving choices of schedules for the Director/Staff to choose from. This data should be presented in a way that displays the list of weeks, the proposal id, proposal title, and contact organizer last name. A summary of the statistics for the result, such as number of preferred dates satisfied, number of 1st choice dates, 2nd choice dates, etc, should be displayed.
-
A button to select a given result set from the group of potential schedules should send the data, with the dates in the selected schedule applied, to the Workshops API for event creation.
-
Log all emails sent by the system, for auditing purposes. Staff will want to know, for example, whether a reviewer was sent an invitation, when, and by whom.
- Staff should be able to view the proposal system “as” a specific user, to see what they see when logging in. For example, as a member of the program committee who is also an External EDI reviewer.
I anticipate rolling out this project in phases.
Work in progress: MVP for this phase
BIRS opens its call for proposals in the first week of June, so that is a hard deadline for having at least the Proposal Submission functionality up and running. This would include user registration and logins. The ability to accept proposal submissions from the scientific community. Typically, there are a small handful of keeners (0 - 5) who submit proposals at the beginning of the summer. It is not until August when they usually start to be submitted with increasing regularity, up until the close of the Call for Proposals at the end of September. About 80% of them are submitted in the final week before the Call closes.
BIRS Staff will begin organizing the various review and evaluation committees, board members, etc. in July.
BIRS Staff will need to e-mail potential peer reviewers, to ask if they are able to participate in the process for the 2023 program year, in August.
The Call for Proposals typically closes late September, and we want to be sure that the peer review features are available before that happens. For some reviewers, books of assigned proposals are printed (to PDF) and e-mailed to them. A book of all proposals is usually prepared by the end of September, for the full peer review process to begin.
Various boards and evaluation committees will begin reviewing the proposals and their reviews at the end of October. This process will continue into November, with a Scientific Advisory Board meeting mid November, to work out the final selection of proposals for the program year two years hence.
Once the proposals are selected, there is often a couple of weeks until some details and options are worked out, i.e. if certain workshops should be scheduled together or one after another, and then the schedule optimization program is run. The schedule is normally announced early December.
