Support for Fcrepo 6 in Valkyrie (#6976)

* Convert Ldp::Gone to ObjectNotFoundError

Ldp::Gone is the error raised by the fedora adapter when the resource is a tombstone after being deleted.

* Use the real metadata adapter

* Ensure embargo/lease factory is using DateTime

The valkyrie fedora adapter does not convert RDF Date causing comparison failures.
In real use the embargo transaction converts to a datetime, but this factory sets the value directly, so let's make it match real use.

* Avoid using << on frozen default array in specs

* Sometimes the original file is not first in file_ids

* Convert Ldp::NotFound to Valkyrie::Persistence::ObjectNotFoundError

This fixes file_set_indexer_spec which has a bunch of odd setup and mocking, likely from before our factories were equipped to handle building work/file_set/file_metadata stacks. However, it feels right to include here and be treated as an ObjectNotFoundError.

* Fix specs stubbing Hyrax::TimeService with a string

* More date format spec fixes

* Limit rails versions to speed dependency resolution

* Bump valkyrie, use main until new release

* Unpin faraday, rdf 3.3.2 fixed the bug

* Add sirenia to Github Actions CI; Significant rework of docker compose

- All 3 test apps use the same container image.
- dassie stays in hyrax-webapp, koppie/sirenia live in hyrax-koppie.
- db_migrate service moved to dev-entrypoint in web service.
- Worker waits to start until rails app is up, should reduce bundle install racing.
- hyrax-engine-dev target now based on hyrax-worker-base.
- Chrome is still a pain and randomly stops responding.

* rubocop

* Docker and GHA fixes

Add worker to depends_on to avoid volume copy race.
Fix Dockerfile style issues
Align chrome service config
Corrected container names for log capture

* Relax class matcher strictness

Now that this spec uses the real valkyrie adapter,
the object coming out of wings is really an anonymous
descendant of the valkyrie model class.

* Restore `rake assets:precompile` for javascript tests

* Publish deletion first to avoid encountering tombstones

* Use correct rails root

* rubocop

* Convert custom queries added for statistics to solr

* Optimize admin set list (also fixes sorting issues)

* Sort the admin sets for real life

* :clean_repo for flaky specs

* Bump valkyrie version

* Rename db-wait.sh script to service-wait.sh

* Return Resources instead of SolrHits

* Undo running koppie/sirenia from hyrax-koppie dir

* We are not yet compatible with v1 of openseadragon gem

.dassie/.env

@@ -2,7 +2,7 @@ ANALYTICS_START_DATE=2021-08-21
 BUNDLE_GEMFILE=Gemfile.dassie
 BUNDLE_PATH=/app/bundle
 CH12N_TOOL=fits_servlet
-CHROME_HEADLESS_MODE=false
+CHROME_HEADLESS_MODE=true
 DATABASE_CLEANER_ALLOW_REMOTE_DATABASE_URL=true
 DATABASE_TEST_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax_test?pool=5
 DATABASE_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax?pool=5

.dassie/Gemfile

@@ -30,7 +30,7 @@ gem 'jbuilder', '~> 2.5'
 gem 'jquery-rails'
 gem 'pg', '~> 1.3'
 gem 'puma'
-gem 'rails', '~> 6.1'
+gem 'rails', '~> 6.1', '>= 6.1.7.10'
 gem 'riiif', '~> 2.1'
 gem 'rsolr', '>= 1.0', '< 3'
 gem 'sass-rails', '~> 6.0'

.dockerignore

@@ -14,9 +14,11 @@ artifacts/*
 coverage/*
 chart/*
 
-Gemfile.lock
-.dassie/Gemfile.lock
-.koppie/Gemfile.lock
+Gemfile*lock
+*/Gemfile*lock
 
-node_modules/*
+
+node_modules
+*/node_modules
 yarn.lock
+*/yarn.lock

.github/workflows/lint-build-test.yml

@@ -1,5 +1,5 @@
 name: 'Lint Build Test'
-run-name: Link Build Test of ${{ github.ref_name }} by @${{ github.actor }}
+run-name: Lint Build Test of ${{ github.ref_name }} by @${{ github.actor }}
 on:
   push:
     branches:
@@ -44,39 +44,35 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
-      matrix:
-        ci_test_app: [dassie,koppie]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
       - name: Build and export
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
-          build-args: |
-            APP_PATH=.${{ matrix.ci_test_app }}
           cache-from: type=gha
           cache-to: type=gha, mode=max
           context: .
           target: hyrax-engine-dev
-          tags: samvera/${{ matrix.ci_test_app }}:latest,samvera/${{ matrix.ci_test_app }}:${{ github.sha }}
-          outputs: type=docker,dest=/tmp/${{ matrix.ci_test_app }}-${{ github.sha }}.tar
+          tags: samvera/hyrax-dev:latest,samvera/hyrax-dev:${{ github.sha }}
+          outputs: type=docker,dest=/tmp/hyrax-dev-${{ github.sha }}.tar
       - name: Upload built image artifact
         uses: actions/upload-artifact@v4
         with:
-          name: ${{ matrix.ci_test_app }}-image
-          path: /tmp/${{ matrix.ci_test_app }}-${{ github.sha }}.tar
+          name: hyrax-dev
+          path: /tmp/hyrax-dev-${{ github.sha }}.tar
 
   test:
     needs: build
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        ci_test_app: [dassie,koppie]
-        ci_node_total: [8]
-        ci_node_index: [0,1,2,3,4,5,6,7]
+        ci_test_app: [dassie,koppie,sirenia]
+        ci_node_total: [4]
+        ci_node_index: [0,1,2,3]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -89,11 +85,11 @@ jobs:
       - name: Download built image artifact
         uses: actions/download-artifact@v4
         with:
-          name: ${{ matrix.ci_test_app }}-image
+          name: hyrax-dev
           path: /tmp
       - name: Start containers
         run: |
-          docker load --input /tmp/${{ matrix.ci_test_app }}-${{ github.sha }}.tar
+          docker load --input /tmp/hyrax-dev-${{ github.sha }}.tar
           docker image ls -a
           docker compose -f docker-compose-${{ matrix.ci_test_app }}.yml up -d --quiet-pull --pull missing --no-build
       - name: RSpec
@@ -107,7 +103,7 @@ jobs:
         if: always()
         uses: jwalton/gh-docker-logs@v2
         with:
-          images: 'seleniarm/standalone-chromium'
+          images: 'selenium/standalone-chromium,postgres,fcrepo/fcrepo,solr,bitnami/redis,ghcr.io/samvera/fitsservlet,ghcr.io/samvera/fcrepo4'
       - name: Move Test Files
         if: always()
         env:

.github/workflows/test-results.yml

@@ -1,5 +1,5 @@
 name: Test Results
-
+run-name: Test Results of ${{ github.ref_name }} by @${{ github.actor }}
 on:
   workflow_run:
     workflows: ["Lint Build Test"]

.koppie/.env

@@ -2,7 +2,7 @@ APP_NAME=koppie
 BUNDLE_GEMFILE=Gemfile.koppie
 BUNDLE_PATH=/app/bundle
 CH12N_TOOL=fits_servlet
-CHROME_HEADLESS_MODE=false
+CHROME_HEADLESS_MODE=true
 DATABASE_CLEANER_ALLOW_REMOTE_DATABASE_URL=true
 DATABASE_TEST_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax_test?pool=5
 DATABASE_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax?pool=5
@@ -29,7 +29,7 @@ POSTGRES_PASSWORD=hyrax_password
 POSTGRES_USER=hyrax_user
 RACK_ENV=development
 RAILS_ENV=development
-RAILS_ROOT=.koppie
+RAILS_ROOT=/app/samvera/hyrax-webapp
 RAILS_LOG_TO_STDOUT=false
 RAILS_QUEUE=sidekiq
 REDIS_HOST=redis

.koppie/Gemfile

@@ -30,7 +30,7 @@ gem 'jbuilder', '~> 2.5'
 gem 'jquery-rails'
 gem 'pg', '~> 1.3'
 gem 'puma'
-gem 'rails', '~> 6.1'
+gem 'rails', '~> 6.1', '>= 6.1.7.10'
 gem 'riiif', '~> 2.1'
 gem 'rsolr', '>= 1.0', '< 3'
 gem 'sass-rails', '~> 6.0'

Dockerfile

@@ -1,7 +1,7 @@
 ARG ALPINE_VERSION=3.19
-ARG RUBY_VERSION=3.2.4
+ARG RUBY_VERSION=3.2.6
 
-FROM ruby:$RUBY_VERSION-alpine$ALPINE_VERSION as hyrax-base
+FROM ruby:$RUBY_VERSION-alpine$ALPINE_VERSION AS hyrax-base
 
 ARG DATABASE_APK_PACKAGE="postgresql-dev"
 ARG EXTRA_APK_PACKAGES="git"
@@ -49,7 +49,7 @@ ENTRYPOINT ["hyrax-entrypoint.sh"]
 CMD ["bundle", "exec", "puma", "-v", "-b", "tcp://0.0.0.0:3000"]
 
 
-FROM hyrax-base as hyrax
+FROM hyrax-base AS hyrax
 
 ARG APP_PATH=.
 ARG BUNDLE_WITHOUT="development test"
@@ -59,7 +59,7 @@ ONBUILD RUN bundle install --jobs "$(nproc)"
 ONBUILD RUN RAILS_ENV=production SECRET_KEY_BASE=`bin/rake secret` DB_ADAPTER=nulldb DATABASE_URL='postgresql://fake' bundle exec rake assets:precompile
 
 
-FROM hyrax-base as hyrax-worker-base
+FROM hyrax-base AS hyrax-worker-base
 
 USER root
 RUN apk --no-cache add bash \
@@ -78,10 +78,10 @@ RUN mkdir -p /app/fits && \
     sed -i 's/\(<tool.*TikaTool.*>\)/<!--\1-->/' /app/fits/xml/fits.xml
 ENV PATH="${PATH}:/app/fits"
 
-CMD bundle exec sidekiq
+CMD ["bundle", "exec", "sidekiq"]
 
 
-FROM hyrax-worker-base as hyrax-worker
+FROM hyrax-worker-base AS hyrax-worker
 
 ARG APP_PATH=.
 ARG BUNDLE_WITHOUT="development test"
@@ -91,40 +91,19 @@ ONBUILD RUN bundle install --jobs "$(nproc)"
 ONBUILD RUN RAILS_ENV=production SECRET_KEY_BASE=`bin/rake secret` DB_ADAPTER=nulldb DATABASE_URL='postgresql://fake' bundle exec rake assets:precompile
 
 
-FROM hyrax-base as hyrax-engine-dev
+FROM hyrax-worker-base AS hyrax-engine-dev
 
-USER root
-RUN apk --no-cache add bash \
-  ffmpeg \
-  mediainfo \
-  perl
 USER app
-
-ARG APP_PATH=.dassie
 ARG BUNDLE_WITHOUT=
+ENV HYRAX_ENGINE_PATH=/app/samvera/hyrax-engine
 
-ENV HYRAX_ENGINE_PATH /app/samvera/hyrax-engine
-
-COPY --chown=1001:101 $APP_PATH /app/samvera/hyrax-webapp
+COPY --chown=1001:101 .dassie /app/samvera/hyrax-webapp
 COPY --chown=1001:101 . /app/samvera/hyrax-engine
 
 RUN bundle -v && \
-  bundle install --jobs "$(nproc)" && \
-  cd $HYRAX_ENGINE_PATH && \
-  bundle install --jobs "$(nproc)" && \
-  yarn && yarn cache clean
-
-RUN RAILS_ENV=production SECRET_KEY_BASE='fakesecret1234' DB_ADAPTER=nulldb DATABASE_URL='postgresql://fake' bundle exec rake assets:precompile
-
-
-FROM hyrax-worker-base as hyrax-engine-dev-worker
-
-ARG APP_PATH=.dassie
-ARG BUNDLE_WITHOUT=
-
-ENV HYRAX_ENGINE_PATH /app/samvera/hyrax-engine
-
-COPY --chown=1001:101 $APP_PATH /app/samvera/hyrax-webapp
-COPY --chown=1001:101 . /app/samvera/hyrax-engine
+  BUNDLE_GEMFILE=Gemfile.dassie bundle install --jobs "$(nproc)" && yarn && \
+  cd $HYRAX_ENGINE_PATH && bundle install --jobs "$(nproc)" && yarn && \
+  yarn cache clean
 
-RUN bundle install --jobs "$(nproc)"
+ENTRYPOINT ["dev-entrypoint.sh"]
+CMD ["bundle", "exec", "puma", "-v", "-b", "tcp://0.0.0.0:3000"]

app/controllers/hyrax/my/works_controller.rb

@@ -51,7 +51,21 @@ def managed_works_count
 
       def admin_sets_for_select
         source_ids = Hyrax::Collections::PermissionsService.source_ids_for_deposit(ability: current_ability, source_type: 'admin_set')
-        source_ids.map { |admin_set_id| [Hyrax.query_service.find_by(id: admin_set_id).title.first, admin_set_id] }
+
+        admin_sets_list = Hyrax.query_service.find_many_by_ids(ids: source_ids).map do |source|
+          [source.title.first, source.id]
+        end
+
+        # Sorts the default admin set to be first, then the rest by title.
+        admin_sets_list.sort do |a, b|
+          if Hyrax::AdminSetCreateService.default_admin_set?(id: a[1])
+            -1
+          elsif Hyrax::AdminSetCreateService.default_admin_set?(id: b[1])
+            1
+          else
+            a[0] <=> b[0]
+          end
+        end
       end
     end
   end

app/services/hyrax/custom_queries/find_access_control.rb

@@ -20,7 +20,7 @@ def find_access_control_for(resource:)
           .find_inverse_references_by(resource: resource, property: :access_to)
           .find { |r| r.is_a?(Hyrax::AccessControl) } ||
           raise(Valkyrie::Persistence::ObjectNotFoundError)
-      rescue ArgumentError, Ldp::Gone # some adapters raise ArgumentError for missing resources
+      rescue ArgumentError, Ldp::Gone, Ldp::NotFound # some adapters raise ArgumentError for missing resources
         raise(Valkyrie::Persistence::ObjectNotFoundError)
       end
     end

app/services/hyrax/custom_queries/find_by_date_range.rb

@@ -25,30 +25,13 @@ def initialize(query_service:)
       # @param models [Array]
       # @param start_datetime [DateTime]
       # @param end_datetime [DateTime]
+      # @return [Array<Hyrax::Resource>]
       def find_by_date_range(start_datetime:, end_datetime: nil, models: nil)
-        end_datetime = 1.second.since(Time.zone.now) if end_datetime.blank?
-        if models.present?
-          query_service.run_query(find_models_by_date_range_query, start_datetime.to_s, end_datetime.to_s, models)
-        else
-          query_service.run_query(find_by_date_range_query, start_datetime.to_s, end_datetime.to_s)
-        end
-      end
-
-      def find_models_by_date_range_query
-        <<-SQL
-          SELECT * FROM orm_resources
-          WHERE created_at >= ?
-          AND created_at <= ?
-          AND internal_resource IN (?);
-        SQL
-      end
-
-      def find_by_date_range_query
-        <<-SQL
-          SELECT * FROM orm_resources
-          WHERE created_at >= ?
-          AND created_at <= ?;
-        SQL
+        end_range = end_datetime.blank? ? '*' : end_datetime.utc.xmlschema
+        query = "system_create_dtsi:[#{start_datetime.utc.xmlschema} TO #{end_range}]"
+        query += " AND has_model_ssim: (#{models.map { |m| "\"#{m}\"" }.join(' OR ')})" unless models.empty?
+        ids = Hyrax::SolrService.query_result(query, fl: 'id')['response']['docs'].map { |doc| doc['id'] }
+        Hyrax.query_service.find_many_by_ids(ids: ids)
       end
     end
   end

app/services/hyrax/custom_queries/find_count_by.rb

@@ -25,45 +25,10 @@ def initialize(query_service:)
       # @param hash [Hash] the hash representation of the query
       def find_count_by(hash = {}, models: nil)
         return nil if models.empty? && hash.blank?
-        return find_count_by_af(hash, models: models) unless query_service.respond_to?(:orm_class)
-
-        internal_array = ["{ #{hash.map { |k, v| "\"#{k}\": #{v}" }.join(', ')} }"] if hash.present?
-        if models.empty?
-          query_service.orm_class.count_by_sql(([find_count_by_properties_query] + internal_array))
-        elsif hash.blank?
-          query_service.orm_class.count_by_sql([find_count_by_models_query] + [models])
-        else
-          query_service.orm_class.count_by_sql(([find_count_by_properties_and_models_query] + internal_array + [models]))
-        end
-      end
-
-      def find_count_by_af(hash, models: nil)
         flat_hash = hash.map { |k, v| "#{k}: \"#{v}\"" }.join(' ')
         flat_hash += " has_model_ssim: (#{models.map { |m| "\"#{m}\"" }.join(' OR ')})" unless models.empty?
         Hyrax::SolrService.count(flat_hash)
       end
-
-      def find_count_by_properties_and_models_query
-        <<-SQL
-          SELECT count(*) FROM orm_resources
-          WHERE metadata @> ?
-          AND internal_resource IN (?);
-        SQL
-      end
-
-      def find_count_by_models_query
-        <<-SQL
-          SELECT count(*) FROM orm_resources
-          WHERE internal_resource IN (?);
-        SQL
-      end
-
-      def find_count_by_properties_query
-        <<-SQL
-          SELECT count(*) FROM orm_resources
-          WHERE metadata @> ?;
-        SQL
-      end
     end
   end
 end