Design: submission reference letter uploads

[lmkw]

“I was able to log in to the application portal with my NetID, and I see that three letters of reference can be added as PDF files. I’m going to reach out to my recommenders ASAP, and am wondering whether there’s also a more anonymous way to submit recommendations (i.e., without the applicant having access) in case any of my recommenders would prefer to keep their letters more private.”

[jseraf]

One possible design:

• Add "letter of reference" field on grant to store count of letters (defaults to 0)
• Add "letter of reference" question type to form_builder that takes email addresses (see update below)
• Create GrantSubmission::Recommender model. Something like:
  • submission_id
  • email_address (when user doesn't exist?)
  • user_id (when user exists)
  • attached_document
  • created/updated dates
• Outgoing notification to email includes link to upload file (grant/[id]/submissions/[id]/referral/[id] where referrer only sees name of submission and applicant name). Would need to build in reminders both to the referree and the referrer.

[jseraf]

Discussed this further w/ Matt earlier today:

• Add controller to grant/[:id]/submission/[:id]/recommend for recommenders reference-letter/uuid to upload a pdf/word file.
• At minimum this view should show grant name, submission name, applicant name and the form to upload the LoR
• Add Letters of Recommendation section at bottom of Submission view for reviewers/admins to see/download

Other considerations:

• needs dependent: destroy in submission
• how to handle submission.complete if a letter of recommendation hasn't been uploaded. My inclination is to allow it to be complete for purposes of review.
• show/hide to applicants?
  • show after close date?
• should the recommender be able to change the file once it has been uploaded?
• should the recommender see the entire submission when uploading or just the bare minimum lined out above?
  • seems like title and applicant name should be sufficient

[jseraf]

@lmkw Keeping this ticket open. See comments.

[jseraf]

"recommender" vs "reference" vs ...something else?

[lmkw]

Had another request for this feature:
"we hoped there might be a way for the recommenders to upload directly, as well as a way for the platform to send them emails with the link and a body of text with instruction that includes the applicant’s name. Housing the listed recommenders, communications, and gathering of letters, etc in one place with the applications and eventual reviews would be ideal, over separate manual lists and emails in individual’s inboxes."

[jseraf]

As mentioned above, to mirror NIH guidelines, name model ReferenceLetter - https://www.niaid.nih.gov/grants-contracts/reference-letters

[jseraf]

Summarizing all the above:

• Add ReferenceLetter model
  • uuid (rather than incremental id)
  • belongs_to :user, optional:true
  • referrer_email
  • letter
• Add reference_letter_number attribute on Grant. Default 0
• GrantSubmission::Submission has_many :reference_letters, optional: true

Proposed Workflow:
Grant Admin uses Reference Letters Required field to provide a number (or none)

Submitter prompted to provide email address for each referrer after all questions/responses. Emails are required for the Submission to be submitted (i.e. not required for draft submissions).

Emails sent to the emails provided with message like "You've been asked to provide a reference letter for "Submission Title" by Submitter Name for the Grant Name competition. You may upload a file at the following link. This is a unique URL for your reference letter. Please don't share this email with anyone." Use the uuid in the provided link. Include a due date (see questions below)

Referrer visits: reference-letter/[uuid] to upload their letter. Page includes:
GrantName
Submission Name
Submitter Name
List of Applicants where needed.

Questions:
Can a referrer decline?
If we use a no-reply email, what happens if the submitter provides a bad email address?
Or should the email come from the submitter's email address?
When are letters due? e.g. Submission close date? Panel Date?
Do we send reminders to the referrer? Should also they go to the submitter?
Is a submission eligible for review if no or only some submission letters have been submitted?
Add a new My Referrals section when the referrer is already a known Competitions user?
Check for open reference letters when new account is created?

[jseraf]

Misc notes:

• UUIDs won't work as an ActiveStorage record_id unless GrantSubmission::Submission also uses them.
• UUID requires enabling pgcrypto. Does the deploy or competitions user have permission to do this?

[lmkw]

Feedback:

great to see added improvements!

we currently use REDCap for rec letters of student applicants to our summer program and we set it up so that they only need a link to upload the document without a login.

My worry with the requirement to register/login is it adds an additional barrier and may reduce completion rates for recommenders. if there's an "upload document" field that way recommenders could draft their letters and upload when ready I think is the best approach.

I don't think the recommender should see the whole submission to protect the applicant's privacy/info.

another feature that would be nice is if the system could remind the recommenders to complete their submission if they haven't done so.

[jseraf]

There's also this workflow on the NIH page: https://www.niaid.nih.gov/grants-contracts/reference-letters

And the eRA Commons submission page: https://public.era.nih.gov/commonsplus/public/reference/submitReferenceLetter.era

[lmkw]

Should a referrer be able to decline? - Yes, but ideally the submitter verifies their consent to submit (or a way to "undo" in case they reject by mistake)

Should the email come from the submitter's email address?
I don't think so, but maybe as it might end up in the spam folder?

When are letters due? e.g. Submission close date? Panel Date?
Typically we allow a grace period (usually 5 - 10 days) after the application due date for LOS to be received (and yes prior to the review/panel date)

Should we send reminders to the referrer? Should they also go to the submitter?
Yes! and I think both is good in this case. but not too frequently (i.e. more than 1x/week)

Is a submission eligible for review if no or only some submission letters have been submitted? Or must it have all reference letters before it can be reviewed?
This would be a project-by-project basis I believe, our policy is all letters are required (2) for consideration.

Should we add a new My Referrals section when the referrer is already a known Competitions user?
Should we check for open reference letters when a new account is created?
I think this would be a good feature for the first one, not sure about the second.