Fix "Sort by: Published Date" #185

KatharineV · 2024-03-14T19:09:18Z

Summary

The "Sort by Published Date" (both ascending and descending) feature in the catalog search and in collections does not properly sort based on the "Publication date" metadata field. If this feature is sorting based on a different date field, perhaps that would explain the problem, but if so, then we still would request work on this ticket, as explained in the last paragraph.

If the feature is meant to sort based on the "Publication date" metadata field, then we'd like the functionality to work when dates are entered in that field according to the system's requirements. In this example collection on ADL staging, we used the UI to enter publication date metadata, so we assume the system entered the date in a format that it prefers to read. The back end gave us a calendar widget, which we used. The dates appear properly in each work's metadata. However, when we use the sort by published date feature in the collection, the works do not sort in date order. See screenshots below.

Acceptance Criteria

All Sort By options should function correctly
- on the catalog search page
- and from the collection show page (admin view)

Catalog search

Collection show pg (admin view)

Screenshots or Video

Proper date in work metadata:
https://adl.s2.adventistdigitallibrary.org/concern/published_works/22255888_the_minnesota_worker_december_21_1898?locale=en

Automatic relevant sorting in the collection:

Sort By: Published Date (Ascending):

Sort By: Published Date (Descending):

These examples clearly show that the dates (which also appear in the work titles) are not sorting in proper order. They don't even change between ascending and descending. The behavior is the same in the catalog search, which offers a publication date sort.

If the feature is actually pulling from a different date field, we'd like to know which one is being referenced, what date format the system will recognize and use to sort properly, and we'd like the Sort By label to be renamed to match the metadata field being referenced.

As a sidenote only tangentially related to this ticket (and perhaps headed for its own ticket someday, or could be rolled into general Hyku upgrades of the future), the Publication Date metadata field seems like a bad choice for the catalog and collection search sort by feature. Not all works even have this metadata field in their work type. For example, Images. The sort by date feature would be better if it pulled a metadata field that appears in all work types, and it would be nice if there was a note in the UI that mentions to users that this is the field for date sorting. With soooooo many date fields across work types, it's very confusing what is used for search facets and sort by features etc. Additionally, the UI could contain a note regarding what date formats the system will actually recognize and use to sort. Is this info already included in Hyku documentation?
See this screenshot of the back end of the UI "add new work" for the kind of note I'm suggesting would include format info and info about sort and facet dependency. Hope this makes sense.

Testing Instructions

Testing note: the staging site was partially reindexed resulting in most existing records being updated but not all. Our assumption is that the cut over will handle the update of all works in production. Most records will pass QA.

Ensure that works & collections have "Created Date" filled in. If there is no Created Date, they will sort to the end of the list, whether sorting by ascending or descending.

Go to the catalog index page
Use the drop down menu to sort by the following
- Title
- Author
- Published date (asc)
- Published date (desc)
Go to the Dashboard > Collection > select a collection with associated works > scroll down to works index page
Use the drop down menu to sort by the following
- Title
- Author
- Published date (asc)
- Published date (desc)

Notes

This will require an update of data to remove ["0~"] from blank author (creator) fields.

christhepianist · 2024-03-14T20:12:22Z

[like] Christine Peterson reacted to your message:

…

jillpe · 2024-04-04T16:36:27Z

This was done in pals for a different date (date created)

notch8/palni-palci#867

KatharineV · 2024-04-30T14:40:17Z

Folks, the Sort By feature isn't working at all on a collection right now. When I created this ticket, we didn't have Knapsack. We are post Knapsack now, so I can't be sure if the cut over is responsible for the lack of functionality or if I overlooked and failed to test the other sort by options. Maybe it was always broken.

Here's the collection where I'm having trouble:
https://adl.b2.adventistdigitallibrary.org/collections/b1cb4449-c0ee-4499-82da-32c1246daeea

Automatic relevance sort (which isn't sorted by any metric that I can determine):

I select "Sort By: Upload Date (Descending)" and hit Refresh and this is what I get:

I select "Sort By: Title" and hit Refresh and this is what I get:

orangewolf · 2024-08-29T17:19:23Z

Sorting does work (you have to hit refresh after making change) but sort by title is backward and sort by dates could not be verified easily. Need to look at those

jillpe · 2024-09-04T16:28:10Z

This was fixed in UTK

This commit fixes a bug in the ability to sort works by "Published date", "Upload date", and "Title" in the catalog search page and the collection show page. Ref: - notch8/adventist_knapsack#185

This commit fixes a bug in the ability to sort works by "Published date", "Upload date", and "Title" on the catalog search page and the collection show page. Ref: - notch8/adventist_knapsack#185

This commit adds a guard clause to the `SortTitle` model to fix failing tests due to a nil value being assigned to `@title` on initialization. Ref: - notch8/adventist_knapsack#185

sjproctor · 2024-09-09T22:59:41Z

We identified the source of the sorting problem is coming from Hyku. We are making the changes there first.

https://github.com/samvera/hyku/pull/2316/files

This commit pulls in an update to Hyku that fixes a bug when sorting by title, author, and created date (published date). The Hyku indexing moves from `CollectionResource` to `CollectionResourceIndexer`. Ref: - #185

This commit moves the HykuIndexing module from the CollectionResource to the CollectionResourceIndexer. Ref: - notch8/adventist_knapsack#185

sjproctor · 2024-09-23T18:03:12Z

If a record does not have a author (creator) the field will have ["0~"] due to an older version of the order_already gem.
The newest version of the gem handles this issue. The gem version was updated in Hyku and in the submodule of this application.
However, existing records in ADL with ["0~"] will not register the author (creator) field as blank and therefore do not display or sort properly.
To fix this problem, all records with ["0~"] in the creator field need to be updated. This can be done with the following script:

concerns = Hyrax.config.curation_concerns.map{ |c| "#{c}Resource".safe_constantize } << CollectionResource

concerns.each do |concern|
  works = Hyrax.query_service.find_all_of_model(model: concern)
  works.each do |work|
    next unless work['creator'] == ["0~"]
    work.creator = []
    work.save
  end
end

# Story The "Sort by Published Date" (both ascending and descending) feature in the catalog search and in collections does not properly sort based on the "Publication date" metadata field. Ref: - #185 # Expected Behavior Before Changes Title, author, and publication date sort fields did not sort properly on the catalog search results page or the collection show page. # Expected Behavior After Changes The user is able to sort by title, author, and publication date on the catalog search results page as well as the collection show page. # Screenshots / Video https://github.com/user-attachments/assets/23ca2643-f0f4-498f-91b3-2ccc2b5e9714 # Notes This updated required an update of the submodule as well as reindexing local works and collections.

ShanaLMoore · 2024-09-25T21:26:32Z

QA Results: ❌ NEEDS REWORK

TLDR: sort by published date doesn't appear to be working for Collections. Kirk ran a staging reindex yesterday and in the tested example, I manually edit/saved each work to be sure it's not an indexing problem.

tested on STAGING

Testing note: the staging site was partially reindexed resulting in most existing records being updated but not all. Our assumption is that the cut over will handle the update of all works in production. Most records will pass QA.

CATALOG INDEX PAGE ✅

Go to the catalog index page
Use the drop down menu to sort by the following
- Title
- Author
- Published date (asc)
- Published date (desc)

COLLECTION PAGE ❌

tested on STAGING

Go to the Dashboard > Collection > select a collection with associated works > scroll down to works index page
Use the drop down menu to sort by the following
- Title
- Author
- Published date (asc) ❌
- Published date (desc) ❌

KatharineV · 2024-09-26T13:35:07Z

Team, as you're fixing this ticket, can you also educate me on which date field is used for the Sort By Published Date feature? There are so many date fields and they vary between work types. I have never been clear on which field maps to this feature, so your insight will be valuable.

ShanaLMoore · 2024-09-26T16:33:05Z

cc @sjproctor could you provide some insight here? ^^

sjproctor · 2024-09-26T16:37:02Z

Yeah, there are definitely a lot of date fields! We are using the date_created_ssi for publication date.

laritakr · 2024-09-26T17:00:24Z

This shows as "Date Created" on the work's metadata.

The input form uses a calendar entry that puts a date into mm/dd/yyyy format, but via bulkrax, I am seeing dates such as yyyy-mm-dd. Without a consistent format, we can't do a clean sort.

KatharineV · 2024-09-26T17:40:44Z

@laritakr Thank you! These are the instructions I needed. I've tried to find out before which field and which format the system prefers. The answer wasn't available when I asked the last couple times, so this is really great. I understand why the sort won't work unless we make revisions. We can default to mm/dd/yyyy, but (in case the community ever sits down to think about this) I would love to see modifications in future versions that allow for ISO standard dates, since mm/dd/yyyy is confusing across international boundaries.

Well, forget that for now, and thank you again!

laritakr · 2024-09-26T17:55:03Z

@laritakr Thank you! These are the instructions I needed. I've tried to find out before which field and which format the system prefers. The answer wasn't available when I asked the last couple times, so this is really great.

You're welcome! Glad I could add clarity. Based on what I have found, I am moving this ticket back to QA.

As an added note, I found that anything with no created date will sort at the end, whether using ascending OR descending.

We could put in some date standardization as we index the date used for sorting. This would require a reindex of all works & collections. If this is a desired change, you would need to make a ticket for it.

laritakr · 2024-09-26T18:12:07Z

QA Results: ✅ Passed

(previously passed catalog view QA)

COLLECTION PAGE ✅

tested on STAGING

Created Dates were added to several works - I added the dates into the title so the results are obvious.

Go to the Dashboard > Collection > select a collection with associated works > scroll down to works index page
Use the drop down menu to sort by the following
- Title
- Author
- Published date (asc)
- Published date (desc)

KatharineV · 2024-09-30T15:59:34Z

Folks, does the date_created field also inform the date range facet on the catalog search page?

laritakr · 2024-09-30T16:30:45Z

Folks, does the date_created field also inform the date range facet on the catalog search page?

Yes, date_created is also used to build the sorted date. It expects the date_created to be in the format yyyy-mm-dd to extract the year to be used for sorting.

KatharineV · 2024-10-01T19:04:39Z

I tested this ticket with a set of works (see importer) deposited into a single collection. I put date metadata into the date_created column of the CSV. The final version metadata CSV is attached to that importer. When I formatted dates as 5/1/2023, 5/2/2023, 5/3/2023 and so on, the sort by date feature did NOT work. However, when I entered just the year, the sort worked and put works in the expected order. So, the feature works as described earlier in this ticket.

I tested the sort on the collection dashboard page, the public show page, and in the catalog search.

jillpe added the Knapsack Upgrade label Apr 30, 2024

kirkkwang transferred this issue from notch8/adventist-dl May 10, 2024

jillpe moved this to Ready for Development in Adventist Knapsack May 10, 2024

jillpe removed the status in Adventist Knapsack May 10, 2024

jillpe mentioned this issue May 10, 2024

☄️ EPIC: Knapsack bugs #134

Open

23 tasks

jillpe added the bug something isn't working label May 10, 2024

jillpe moved this to Ready for Development in Adventist Knapsack May 15, 2024

jillpe added this to the Milestone 3 - completion of project milestone Sep 3, 2024

ShanaLMoore added the M3 Milestone 3 label Sep 3, 2024

jillpe added the good first issue label Sep 4, 2024

sjproctor self-assigned this Sep 6, 2024

sjproctor moved this to In Development in adventist-dl Sep 6, 2024

sjproctor moved this to In Development in Adventist Knapsack Sep 6, 2024

ShanaLMoore removed the good first issue label Sep 6, 2024

sjproctor mentioned this issue Sep 9, 2024

🐛 Fix "Sort by: Published Date" samvera/hyku#2316

Merged

sjproctor mentioned this issue Sep 17, 2024

🐛 Sort by: Published Date #814

Merged

sjproctor moved this from In Development to Code Review in adventist-dl Sep 17, 2024

sjproctor moved this from In Development to Code Review in Adventist Knapsack Sep 17, 2024

sjproctor added a commit to samvera/hyku that referenced this issue Sep 18, 2024

🐛 Moves HykuIndexing to CollectionResourceIndexer

ed46919

This commit moves the HykuIndexing module from the CollectionResource to the CollectionResourceIndexer. Ref: - notch8/adventist_knapsack#185

sjproctor mentioned this issue Sep 18, 2024

🐛 Moves HykuIndexing to CollectionResourceIndexer samvera/hyku#2327

Merged

sjproctor moved this from Code Review to Deploy to Staging in adventist-dl Sep 23, 2024

sjproctor moved this from Code Review to Deploy to Staging in Adventist Knapsack Sep 23, 2024

sjproctor moved this from Deploy to Staging to SoftServ QA in adventist-dl Sep 25, 2024

sjproctor moved this from Deploy to Staging to SoftServ QA in Adventist Knapsack Sep 25, 2024

ShanaLMoore assigned laritakr and unassigned sjproctor Sep 25, 2024

ShanaLMoore removed this from Adventist Knapsack Sep 25, 2024

ShanaLMoore moved this from SoftServ QA to In Development in adventist-dl Sep 25, 2024

ShanaLMoore moved this from In Development to Ready for Development in adventist-dl Sep 25, 2024

ShanaLMoore added the needs rework issue needs additional work label Sep 25, 2024

laritakr removed the needs rework issue needs additional work label Sep 26, 2024

laritakr moved this from Ready for Development to Client QA in adventist-dl Sep 26, 2024

laritakr moved this from Client QA to SoftServ QA in adventist-dl Sep 26, 2024

laritakr moved this from SoftServ QA to Client QA in adventist-dl Sep 26, 2024

ShanaLMoore added this to Adventist Knapsack Oct 1, 2024

ShanaLMoore moved this to Client QA in Adventist Knapsack Oct 1, 2024

jillpe moved this from Client QA to Deploy to Production in adventist-dl Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix "Sort by: Published Date" #185

Fix "Sort by: Published Date" #185

KatharineV commented Mar 14, 2024 •

edited by laritakr

Loading

christhepianist commented Mar 14, 2024 via email

jillpe commented Apr 4, 2024 •

edited

Loading

KatharineV commented Apr 30, 2024

orangewolf commented Aug 29, 2024

jillpe commented Sep 4, 2024

sjproctor commented Sep 9, 2024

sjproctor commented Sep 23, 2024

ShanaLMoore commented Sep 25, 2024 •

edited

Loading

KatharineV commented Sep 26, 2024

ShanaLMoore commented Sep 26, 2024

sjproctor commented Sep 26, 2024 •

edited by laritakr

Loading

laritakr commented Sep 26, 2024 •

edited

Loading

KatharineV commented Sep 26, 2024

laritakr commented Sep 26, 2024 •

edited

Loading

laritakr commented Sep 26, 2024 •

edited

Loading

KatharineV commented Sep 30, 2024

laritakr commented Sep 30, 2024

KatharineV commented Oct 1, 2024

Fix "Sort by: Published Date" #185

Fix "Sort by: Published Date" #185

Comments

KatharineV commented Mar 14, 2024 • edited by laritakr Loading

Summary

Acceptance Criteria

Catalog search

Collection show pg (admin view)

Screenshots or Video

Testing Instructions

Notes

christhepianist commented Mar 14, 2024 via email

jillpe commented Apr 4, 2024 • edited Loading

KatharineV commented Apr 30, 2024

orangewolf commented Aug 29, 2024

jillpe commented Sep 4, 2024

sjproctor commented Sep 9, 2024

sjproctor commented Sep 23, 2024

ShanaLMoore commented Sep 25, 2024 • edited Loading

QA Results: ❌ NEEDS REWORK

CATALOG INDEX PAGE ✅

COLLECTION PAGE ❌

KatharineV commented Sep 26, 2024

ShanaLMoore commented Sep 26, 2024

sjproctor commented Sep 26, 2024 • edited by laritakr Loading

laritakr commented Sep 26, 2024 • edited Loading

KatharineV commented Sep 26, 2024

laritakr commented Sep 26, 2024 • edited Loading

laritakr commented Sep 26, 2024 • edited Loading

QA Results: ✅ Passed

COLLECTION PAGE ✅

KatharineV commented Sep 30, 2024

laritakr commented Sep 30, 2024

KatharineV commented Oct 1, 2024

KatharineV commented Mar 14, 2024 •

edited by laritakr

Loading

jillpe commented Apr 4, 2024 •

edited

Loading

ShanaLMoore commented Sep 25, 2024 •

edited

Loading

sjproctor commented Sep 26, 2024 •

edited by laritakr

Loading

laritakr commented Sep 26, 2024 •

edited

Loading

laritakr commented Sep 26, 2024 •

edited

Loading

laritakr commented Sep 26, 2024 •

edited

Loading