Investigate solutions for handling encoding errors during file upload and parsing

[ADPennington]

Background

An aggregate file was rejected in TDP due to an encoding issue, but the error messages returned were misleading, indicating a different cause:

After further investigation, it was discovered that the file failed because of its encoding format. When the file was re-encoded to UTF-8, the file was processed successfully:

This issue raises questions about how to catch and handle encoding problems early in the process, particularly at the file upload stage, and whether more helpful error messages can be provided. Additionally, there is the possibility that accepting UTF-8 with BOM could bypass the issue, which needs to be explored further.

The purpose of this spike is to explore potential solutions to better handle encoding errors, whether at the file upload stage, pre-parsing stage, or through on-the-fly encoding adjustments.

Tasks

• Investigate whether encoding issues can be detected at the file upload stage (e.g., in FileUpload.jsx) and if more meaningful error messages can be provided to the user.
• Research the feasibility of converting file encoding on-the-fly during the upload or parsing process, ensuring no data loss occurs.
• Investigate if converting to UTF-8 or UTF-8 with BOM could be a viable solution to prevent encoding-related errors.
• Create a set of tests (or integration tests) that simulate different file encodings to ensure that the system can handle various encoding formats without errors.
• Document findings and potential solutions

Acceptance Criteria

• Research findings on blocking file uploads with unsupported encodings (e.g., using nginx or other tools).
• Evaluation of on-the-fly encoding conversion, ensuring no data loss and that files are processed correctly.
• Test cases for different file encodings and UTF-8 with BOM, verifying that files are handled appropriately.
• Documentation summarizing research findings, proposed solutions, and recommendations.

[ADPennington]

this issue came up again for tribal file submitted this week. an audit of rejected files needs to be conducted. suggested criteria for the audit:

• files submitted for FY24Q1 (last quarter)
• file status == Rejected
• error message includes Header length is 24...

cc: @lfrohlich @ysong001 @ttran-hub

[lhuxraft]

May want to explore accepting UTF-8 with BOM, potentially bypass the issue

[elipe17]

@ADPennington There are two paths I see this going.

First, if we detect that the file is not plain old UTF-8 we provide the user an error in the same way we let them know about file extension errors and force them to correct it. We could also provide a KC link on how to change file encoding.

Second, if we detect that the file is not UTF-8 we could give the user a warning that the file is not encoded with UTF-8 and we will be encoding it as UTF-8 before submitting. If we go this route, it would make sense to install another frontend dependency like jschardet. This would help us also cover the case where the file is not encoded as UTF-8 and could not be safely converted to UTF-8 without data loss or corruption. Thus, we would inform the user they have some work to do to fix their file.

Both of these options can be done strictly in the frontend and would not require the backend. Let me know what you think. cc. @reitermb

[ADPennington]

@ADPennington There are two paths I see this going.

First, if we detect that the file is not plain old UTF-8 we provide the user an error in the same way we let them know about file extension errors and force them to correct it. We could also provide a KC link on how to change file encoding.

Second, if we detect that the file is not UTF-8 we could give the user a warning that the file is not encoded with UTF-8 and we will be encoding it as UTF-8 before submitting. If we go this route, it would make sense to install another frontend dependency like jschardet. This would help us also cover the case where the file is not encoded as UTF-8 and could not be safely converted to UTF-8 without data loss or corruption. Thus, we would inform the user they have some work to do to fix their file.

Both of these options can be done strictly in the frontend and would not require the backend. Let me know what you think. cc. @reitermb

If feasible, I like the 2nd option. In most cases I've come across, the data submitter does not know anything about encoding or how to fix it. So warning about this, fixing it if possible, and informing when it can't be fixed would be great.