Handle non-unicode encoded datafiles

[sp35]

Description

Upload CSV feature fails for non utf-8 encoded CSVs. The exception is unhandled so the server returns 500.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 265: invalid start byte

Expected behavior

We can ask for the encoding of the CSV file in the UI and handle the errors if any.

To Reproduce

1. Upload non utf-8 encoded CSV (e.g.- unknown-8bit: https://sample-videos.com/csv/Sample-Spreadsheet-10-rows.csv)
2. Observe error (Internal Server Error)

Environment

Docker

[kgodey]

Thanks for reporting this bug, @sp35!

If possible, I'd like to avoid asking users for the encoding since we're aiming to be friendly to non-technical users. I think we should investigate a solution that tries different encoding options automatically.

[asharonbaltazar]

I've used a small 2kb sample file I found on the web and it failed with this error:

Unable to tabulate data

Here's the file I used:

grades.csv

[eito-fis]

The grades file seems to cause the Unable to tabulate data error due to a missing comma on line 10. However, after fixing that, we still get an error - it seems like having quotes in the column name also breaks things. I'll open a new issue for this.

Edit: #572

[silentninja]

Proposal

---

Use encoding detection libraries like Charset Normalizer or cChardet.

Issues

---

Blocking request - Encoding detection is not instant and must be dealt asynchronously. This is also the case with creating data from csv(will be blocking for huge csv files) and solving it should also solve this problem.
Return null instead of throwing an exception

Remarks

---

I am using charset-normalizer and haven't had issues so far, so I am leaning towards using that library.
I can send a PR if this sounds good.

[kgodey]

@silentninja That sounds good to me. If you're still interested, please feel free to send a PR.

We've put off handling operations asynchronously for now in order to avoid adding additional dependencies such as a task queue, etc. for Mathesar. Let's handle this synchronously for now and create a separate issue for handling it asynchronously.