Replace datapusher with xloader

[adborden]

User Story

As a Data.gov Developer, I want to use ckanext-xloader instead of datapusher so that we are using a well-supported CKAN data loading implementation and have fewer moving parts to document and secure before we can get our ATO.

Acceptance Criteria

• GIVEN I am looking at the data.gov SSP
  WHEN I search for references to datapusher
  THEN I see no results.
• GIVEN I have a working deployment of inventory.data.gov
  AND there is no datapusher application deployed/configured
  WHEN I upload a compatible resource file to inventory.data.gov
  THEN the file contents are available via the CKAN API.

Background

• In looking at our datapusher fork, Datopian recommended using ckanext-xloader instead of datapusher. They also had a working PR
• Note the alternative approach we'll have to take if we continue to use datapusher.
• We may also be able to handle upload of XLSX files as a result of this work... Something to investigate!

Security Considerations (required)

This work will reduce the number of moving parts and internal connections in the inventory.data.gov architecture, simplifying security and compliance. The xloader extension is also Python 3 and CKAN 2.9-compatible.

Sketch

Here's the previous PR implementing xloader with CKAN 2.8.

[pjsharpe07]

See the PR above for current state of xloader code changes. Other notes:

• for local development, tested .csv and .xls with success. I had to use the 'old' metadata form (/new) as the publisher field on the new form wouldn't populate and I couldn't move on without it. Still investigating or will open another ticket if needed
• Attempted .xlsx files by editing the ckanext.xloader.formats config option but still received an error (see screen shot below). So, more work needed for this ticket
• Was able to make changes to push to cloud.gov. App comes up and passes all tests, but xloader functionality untested in UI
• The various config options are in both .ini files, but am not sure what the optimal configuration would be

[pjsharpe07]

Update:

• publisher field in local dev populates so we can test with the metadata form
• Both .csv and .xls formats working and so does .xlsx -- so we may be able to close this ticket within the scope of this work