Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch File Encoding Errors with Package #3414

Closed
wants to merge 20 commits into from

Conversation

elipe17
Copy link

@elipe17 elipe17 commented Jan 9, 2025

Summary of Changes

  • Added new package to the frontend to better handle data file encoding errors
  • Updated FileUpload.jsx to use the new package to detect and subsequently transcode datafiles if they are not utf8 format

Pull request closes N/A

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose up --build
cd tdrs-backend && docker-compose up
  1. Open http://localhost:3000/ and sign in.
  2. Upload/submit a normal datafile to verify functionality
  3. Take the same datafile open it in VS code. In the bottom right of VS code, you will see the encoding of the file. Click that and then select Save with Encoding. Choose a few encodings and submit each file as you change the encoding.
  4. Verify that the file is uploaded, the backend doesn't report an invalid encoding, and ideally no data loss has occurred and the submission history reports the same output as the original submission from step 2. Some obvious new encodings that would break develop but not this PR are: UTF-8 with BOM, UTF-16 LE, and UTF-16 BE.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

  • [insert ACs here]
  • lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

  • Are all areas of code introduced in this PR meaningfully tested?
    • If this PR introduces backend code changes, are they meaningfully tested?
    • If this PR introduces frontend code changes, are they meaningfully tested?
  • Are code coverage minimums met?
    • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
    • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

  • Are backend code style checks passing on CircleCI?
  • Are frontend code style checks passing on CircleCI?
  • Are code maintainability principles being followed?

Deliverable 4: Accessible

  • Does this PR complete the epic?
  • Are links included to any other gov-approved PRs associated with epic?
  • Does PR include documentation for Raft's a11y review?
  • Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

  • Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

  • Does this PR provide background for why coding decisions were made?
  • If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
  • If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
  • If this PR introduces dependencies, are their licenses documented?
  • Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

  • Does the OWASP Scan pass on CircleCI?
  • Do manual code review and manual testing detect any new security issues?
  • If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

  • the purpose of the research
  • methods used to conduct the research
  • who participated in the research
  • what was tested and how
  • impact of research on TDP
  • (if applicable) final design mockups produced for TDP development

@elipe17 elipe17 self-assigned this Jan 9, 2025
@elipe17 elipe17 marked this pull request as ready for review January 9, 2025 21:02
@elipe17 elipe17 added frontend dev raft review This issue is ready for raft review labels Jan 14, 2025
Copy link

codecov bot commented Jan 15, 2025

Codecov Report

Attention: Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 91.34%. Comparing base (25440c5) to head (9f49585).

Files with missing lines Patch % Lines
...-frontend/src/components/FileUpload/FileUpload.jsx 73.33% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3414      +/-   ##
===========================================
- Coverage    91.38%   91.34%   -0.04%     
===========================================
  Files          299      299              
  Lines         8614     8624      +10     
  Branches       640      641       +1     
===========================================
+ Hits          7872     7878       +6     
- Misses         622      626       +4     
  Partials       120      120              
Flag Coverage Δ
dev-backend 91.24% <ø> (ø)
dev-frontend 92.13% <73.33%> (-0.31%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
tdrs-frontend/src/actions/reports.js 100.00% <ø> (ø)
...-frontend/src/components/FileUpload/FileUpload.jsx 87.50% <73.33%> (-5.10%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25440c5...9f49585. Read the comment docs.

Copy link

@raftmsohani raftmsohani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@elipe17
Copy link
Author

elipe17 commented Jan 22, 2025

Closing this in favor of #3438. When #3390 merged, it caused irreparable merge conflicts on this PR which even when accepting in their entirety caused this PR to break in several ways. I decided to re-branch off of develop to see if the same issues still occurred and they did not. It seems there were some conflicting packages in the package-lock.json on this PR. However, it was faster to cherry pick changes to a new branch and create a new PR as opposed to looking for conflicts in ~14,000 changes.

@elipe17 elipe17 closed this Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev frontend raft review This issue is ready for raft review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants