PDF Inventory where title is PMID

[joshlawrimore]

Josh will upload the existing inventory of PDFs to the S3 bucket osm-pdf-uploads. Each PDF from the IRP inventory will have PMID as its filename. Task is to:

Write code that parses the S3 bucket’s inventory to create a document entry, a works entry, and a identifier entry based on the document filehash and the PMID in the title. The upload should create an entry in the provenance table and the provenance_id should be included in the works, document, and identifier tables.

Code should be written to iterate over S3 inventory to ensure that PDF inventory in S3 and document’s table are synced. Any missing file hashes should be able to be reported and either added or removed from the documents table. If a PDF is removed, then all references to a missing pdf are removed from foreign keys from tables.
Run the oddpub code on the inventory to generate the oddpub metrics table.

The resultant database should be in an AWS postgres database.

[joshlawrimore]

@quang-ng and @leej3 , I will start with 3539 PDFs that I have on my local machine. I will then write code to pull the existing inventory from biowulf to the osm-pd-uploads bucket

[joshlawrimore]

@quang-ng and @leej3, the 3538 pdfs are in the osm-pdf-uploads bucket in the pdfs subdirectory.

[quang-ng]

working on PR: #21

[quang-ng]

Hi @joshlawrimore, I'd like to clarify the second point: if a PDF file is present in the S3 bucket but not in the documents table, it should be added to the table as a new entry. On the other hand, if an entity in the table is not found in S3, both that entity and any related entries will be removed. Is that correct?

[leej3]

On the other hand, if an entity in the table is not found in S3, both that entity and any related entries will be removed. Is that correct?

In this case we would leave those entries untouched. They may be valuable for downstream tasks and at some point the future we may populate associated document table entries.