☄️ Derivative Rodeo Integration Epic

[jeremyf]

The goal of this punchiest is to outline the steps necessary to verify that IIIF print picks up the changes

• Standardize fields display order in the "Descriptions" tab adventist_knapsack#659
• Set/ensure space stone's logger level is :info or more granular
• Mint new patch version of Derivative Rodeo
• Update SpaceStone's Derivative Rodeo version to the new patch version
• Deploy SpaceStone server less with change
• Remove from the bucket the 20121816 entry (https://s3.console.aws.amazon.com/s3/buckets/space-stone-dev-preprocessedbucketf21466dd-bxjjlz4251re?region=us-west-2&tab=objects)
• Switch to the "working-on-posting-to-serverless" branch
• Submit the below CSV to SpaceStone and review logs of the various lambdas; you're looking for the logged information
• Re-submit the below CSV to SpaceStone and see that the logs are mentioning "we already have the file"

With the above SpaceStone and Derivative Rodeo adjustments

• Set IIIF Print's application's logger level to :info
• Update IIIF Print to use above Derivative Rodeo gem version
• Update the IIIF Print configuration to leverage SpaceStone; this will require AWS credentials for the Pre Processed Buckets.
• Run import of 20121816 entry (likely want to get a single CSV of this file)
• Review logs; we should not see generating derivatives but instead should see log entries regarding found location
  • See Split PDF and constituent pages as works with expected derivative files (e.g. thumbnail and JSON)
• Run import for a single image entry that has been ingested

Derivative Rodeo Integration Tests for PDF Splitting

The following are the scenarios I’m working through for integration testing:

• Scenario: PDF Split does not exist

Given a work with a PDF
And SpaceStone has not split the PDF
When we import the PDF
Then the application should split the PDF
And attach the resulting split pages as child works

• Scenario: Thumbnail of PDF does not exist

Given a work with a PDF
And SpaceStone has not pre-processed the thumbnail
When we import the PDF
Then the application generates a thumbnail
And attaches the thumbnail to the work

• Scenario: PDF Split exists

Given a work with a PDF
And SpaceStone has not split the PDF
When we import the PDF
Then the application should split the PDF
And attach the resulting split pages as child works

• Scenario: Thumbnail of PDF exists

Given a work with a PDF
And SpaceStone has pre-processed the thumbnail
When we import the PDF
Then the application retrieves the pre-processed thumbnail
And attaches the thumbnail to the work

After the integration test we will need to:

• ☄️ Go Live Checklist for Adventist Pre-Processing adventist_knapsack#343
• Adjust div class to reflect partial name adventist-dl#67
• Bring in Citation work from BL adventist_knapsack#648
• Check if fix for nested indexing is being backported to 3.4 adventist_knapsack#649

[jeremyf]

I removed the 20121816 folder from the bucket, then ran the following to re-process:

• <2023-07-11 Tue 09:19>: I submitted a single row for processing. This row was for a single PDF with 3 pages.
• <2023-07-11 Tue 09:24>: Reviewed the bucket folder 20121816 and saw each page had been processed.
• <2023-07-11 Tue 09:25>: Reviewed logs to see latest Derivative Rodeo changes in place.
• <2023-07-11 Tue 09:29> : Resubmitted the row for processing, the expected behavior is that none of the files will change. Reviewed the split-ocr-thumbnailWorker and saw the logs mention:

DerivativeRodeo::Generators::PdfSplitGenerator#destination :: input_location file_uri s3://space-stone-dev-preprocessedbucketf21466dd-bxjjlz4251re.s3.us-west-1.amazonaws.com/20121816/20121816.ARCHIVAL–page-1.tiff: Found output_location file_uri s3://space-stone-dev-preprocessedbucketf21466dd-bxjjlz4251re.s3.us-west-1.amazonaws.com/20121816/20121816.ARCHIVAL–page-1.tiff.

In other words, the conditional generation is working.

[jeremyf]

Working Notes from 2023-07-13

First, I am working a local copy that has the following Gemfile adjustment:

gem 'iiif_print', path: 'vendor/gems/iiif_print'
gem 'derivative-rodeo', path: 'vendor/gems/derivative_rodeo'

And then I have cloned the iiif_print and derivative_rodeo repositories into vendor/gems.

I’m working with AARK ID 20121816; which can be found at http://oai.adventistdigitallibrary.org/OAI-script?verb=GetRecord&identifier=20121816&metadataPrefix=oai_adl

The above AARK ID can be found in http://oai.adventistdigitallibrary.org/OAI-script?verb=ListRecords&metadataPrefix=oai_adl&set=adl:other as the 25th entry. I created an ingester for the OAI feed and have been working through problems locally.

I have setup my docker compose to not start the web application nor the workers. Instead I bring it up and then have two terminals: one for rails console and one for good jobs. I'm often restarting the good_job workers/service.

Following on https://playbook-staging.notch8.com/en/samvera/bulkrax/re-running-a-single-entry-from-an-import; I bash into the worker and run rails console and enter the following:

switch!('adventist'); entry = Bulkrax::Entry.find(136)
GoodJob::Job.where(finished_at: nil).where(GoodJob::Job.arel_table['created_at'].lteq(DateTime.current)).joins_advisory_locks.where(pg_locks: { locktype: nil }).destroy_all
entry.factory.find&.destroy(eradicate: true) 
entry.build

In the worker:

export AWS_ACCESS_KEY_ID=xxx ; export AWS_SECRET_ACCESS_KEY=xxx ; export AWS_REGION=us-west-2 ; export AWS_S3_BUCKET=space-stone-dev-preprocessedbucketf21466dd-bxjjlz4251re ;  bundle exec good_job start

Observed Problem

I am encountering an issue where the path_to_hocr is using an image file. As this is a thumbnail, I don’t really need to run derivatives on it. In fact, as part of the OAI import, we should skip attaching this file.

However, the underlying problem remains.

🤠🐮 DerivativeRodeo::Generators::WordCoordinatesGenerator#generated\_files encountered \`RuntimeError':
   “🤠🐮 DerivativeRodeo::Generators::WordCoordinatesGenerator#convert\_to\_coordinates encountered \`ArgumentError' error “invalid byte sequence in UTF-8” for path\_to\_hocr: "*tmp/d20230713-248-1vwuyjg/adl-ebstore-repo.s3.amazonaws.com/20/1218/20121816/20121816.TN.jpg" and path\_to\_coordinate: "/tmp/d20230713-248-1jpg9k1/app/samvera/hyrax-webapp/tmp/derivatives/a6/2f/f0/75*-f/57/d-*43/f7*-b/7d/5-/29/dd/74/74/1d/f1-json.json"”
 
 for input\_uri: "![img](https://adl-ebstore-repo.s3.amazonaws.com/20/1218/20121816/20121816.TN.jpg)",
 output\_location\_template: "<file:///app/samvera/hyrax-webapp/tmp/derivatives/a6/2f/f0/75/-f/57/d-/43/f7/-b/7d/5-/29/dd/74/74/1d/f1-json.json>",
 and preprocessed\_location\_template: "s3://space-stone-dev-preprocessedbucketf21466dd-bxjjlz4251re.s3.us-east-1.amazonaws.com/20121816/20121816.TN.jpg.coordinates.json".

[jeremyf]

Starting from a fresh Hyku tenant. When I import a work, files are associated with that work. When I eradicate the work and children then reimport the work, then sometimes (but not often) no files show in the UI.

One observation in the code is that the child works and file sets have an is_child true but their is_child_of is empty. Which leverages ordered_by_ids.

Order of Operations

• Ensure I have a clean Fedora and SOLR
  • docker compose down -v works
• Check out the main branch
  • git checkout main
• Build and bring up the image
  • docker compose up --force-recreate --build
• Create the new tenant
  • Login to hyku.test and create "adventist" tenant
• Stop the docker environment
  • docker compose stop
• Set the correct branch and conditions
  • This is a mix of stash and different branches.
  • Branch updating-to-test-the-rodeo incorporates skipping the thumbnails
  • Branch without-slug-bug is setup to ignore the slug features of Adventist; it is based on updating-to-test-the-rodeo
  • Git Stash I'm referencing iiif_print and derivative_rodeo in vendor/gems; my stashed Gemfile and lock reflect this.
  • Note: I have bumped the relationship job interval from 10 minutes to 1 minute
  • In the config/initializers/iiif_print.rb I have add the following two lines at the bottom:
    • # DerivativeRodeo::Generators::PdfSplitGenerator.output_extension = 'jpg'
    • DerivativeRodeo.config.logger = Logger.new(Rails.root.join("dr.log").to_s, level: Logger::DEBUG)
  • The above two lines use the TIFF for splitting (the format we used in the initial SpaceStone test) and setting the DerivativeRodeo logger means we send the very chatty information from the rodeo to a durable and separate location. This is helpful for reviewing to see how and what is being used.
• Bring up the images without starting services
  • I don't want to auto-start the Rails server nor Good Job because we need to clean up two rotten jobs
• Bash into Worker and Delete All Jobs
  • There should be two jobs the Embargo and Lease job.
  • In Rails console ru GoodJob::Job.destroy_all
• Start the Rails server and good jobs
  • Bash into web and run bundle exec puma -v -b tcp://0.0.0.0:3000
  • Bash into worker
    • export the following ENV variables
      • AWS_ACCESS_KEY_ID
      • AWS_SECRET_ACCESS_KEY
      • AWS_REGION
      • AWS_S3_BUCKET
  • Run bundle exec good_job start
• Create an importer for the adventist tenant
  • Using the OAI Parser
  • With URL of http://oai.adventistdigitallibrary.org/OAI-script
  • metadataPrefix of "oai_adl"
  • set of "adl:other"
• This import should take somewhere between 5 and 10 minutes
• Find one of the following AARK_ID import entries: 20121816 (3 page PDF) or 20121862 (1 page PDF)
  • Make note of the Bulkrax::Entry ID of the work, hereto referred to as entry_id
• VERIFY WORK STEP: Verify in the UI that the work has at least 3 items: the ARCHIVAL PDF, RAW.txt, and one image per page of the PDF.
• Assuming that is the case, shell into the web console and start the rails console and run the following:
  • switch!('adventist')
  • entry = Bulkrax::Entry.find(entry_id) # NOTE entry_id
  • GoodJob::Job.where(finished_at: nil).where(GoodJob::Job.arel_table['created_at'].lteq(DateTime.current)).joins_advisory_locks.where(pg_locks: { locktype: nil }).destroy_all
  • entry.factory.find&.child_works&.map {|cw| cw.destroy(eradicate: true) }
  • This may output a =ActiveTriples::ParentStrategy::UnmutableParentError= message; if so run again
  • entry.factory.find&.destroy(eradicate: true)
  • This may output a =ActiveTriples::ParentStrategy::UnmutableParentError= message; if so I don't believe running it again will matter.
  • entry.build
• Then repeat the VERIFY WORK STEP; you'll likely want to use the importer entries page for the entry_id

If we do not have all of the same items as in the VERIFY WORK STEP then we continue to have a deep seeded bug; and may need to change something in the code and repeat some aspect of the checklist. Perhaps going all the way back to the first step.

What I have observed is that the parent work (e.g. the object associated with entry.factory.find) has the correct associations assigned.

IiifPrint::LineageService.descendent_member_ids_for(work) returns file_set and work ids that represent the items in the UI. However, any of those ids when reified to an ActiveFedora object as child and then we call IiifPrint::LineageService.ancestor_ids_for(child), we get an empty array. We should get an array that is [work.id]

Digging deeper, each of the child works has is_child of true but their ordered_by_ids is an empty array.

Conjectures

My current conjecture is one of the following is :

• The SlugBug overrides are problematic either alone or in relation to IIIF Print
  • We can look at https://github.com/scientist-softserv/adventist-dl/blob/main/config/initializers/slug_override.rb#L58-L69 to see how we amend the retrieval of ordered_by_ids
• We are missing the often copied config/initializers/active_fedora_override.rb in which we apply some changes to ActiveFedora::Aggregation::ListSource.attribute_will_change!
  • essi
  • louisville-hyku
  • palni-palci
  • nnp
• IIIF Print has some other underlying issue with relationships