Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky distributions by county system spec #4910

Conversation

coalest
Copy link
Collaborator

@coalest coalest commented Jan 3, 2025

Doesn't resolve any issue. Related to #4557

Description

I noticed these distributions by county system specs are still flaky. See this CI failure: https://github.com/rubyforgood/human-essentials/actions/runs/12568036610/job/35035172689?pr=4908

Example rspec output:

   1) Distributions by County handles time ranges properly works for last 12 months
     Failure/Error: expect(page).to have_text(served_area.county.name)
       expected to find text "County 5" in "Need Help?\n2\nDiaper McDiaperface\n \nDashboard\n \nDonations\n \nPurchases\n \nRequests\n \nDistributions\n \nPick Ups & Deliveries\n \nPartner Agencies\n \nInventory\n \nCommunity\n \nReports\n Activity Graph\n Annual Survey\n History\n Distributions - Summary\n Distributions - By County\n Distributions - Itemized\n Distributions - Trends\n Donations - Summary\n Donations - Itemized\n Donations - Manufacturer\n Donations - Trends\n Product Drives - Summary\n Purchases - Summary\n Purchases - Trends\nEstimated Distributions by County for Some Unique Name\n Home\nDistributions\n Filter\nCounty\tEstimated total items\tEstimated total market value\nUnspecified\t0\t$0.00\nHuman Essentials was built with  by Ruby for Good."

     [Screenshot Image]: /home/runner/work/human-essentials/human-essentials/tmp/capybara/failures_r_spec_example_groups_distributions_by_county_handles_time_ranges_properly_works_for_last_12_months_717.png


     # ./spec/system/distributions_by_county_system_spec.rb:78:in `block (4 levels) in <top (required)>'
     # ./spec/system/distributions_by_county_system_spec.rb:77:in `block (3 levels) in <top (required)>'

  2) Distributions by County handles time ranges properly works for prior year
     Failure/Error: expect(page).to have_text(served_area.county.name)
       expected to find text "County 9" in "Need Help?\nDiaper McDiaperface\n \nDashboard\n \nDonations\n \nPurchases\n \nRequests\n \nDistributions\n \nPick Ups & Deliveries\n \nPartner Agencies\n \nInventory\n \nCommunity\n \nReports\n Activity Graph\n Annual Survey\n History\n Distributions - Summary\n Distributions - By County\n Distributions - Itemized\n Distributions - Trends\n Donations - Summary\n Donations - Itemized\n Donations - Manufacturer\n Donations - Trends\n Product Drives - Summary\n Purchases - Summary\n Purchases - Trends\nEstimated Distributions by County for Some Unique Name\n Home\nDistributions\n Filter\nCounty\tEstimated total items\tEstimated total market value\nUnspecified\t0\t$0.00\nHuman Essentials was built with  by Ruby for Good."

     [Screenshot Image]: /home/runner/work/human-essentials/human-essentials/tmp/screenshots/failures_r_spec_example_groups_distributions_by_county_handles_time_ranges_properly_works_for_prior_year_120.png


     # ./spec/system/distributions_by_county_system_spec.rb:56:in `block (4 levels) in <top (required)>'
     # ./spec/system/distributions_by_county_system_spec.rb:55:in `block (3 levels) in <top (required)>'

Reason for flakiness:
We should be returning counties for the valid distribution, but no distribution is found. The test seems to be only flaky during certain times of day, specifically at times when the time difference between PST and UTC results in different days.

The system time zone (for Capybara) and server time zone (Rails) are in PST. But timestamps are stored in the database in UTC time. Normally this isn't an issue because Active Record will convert to UTC for queries. But since this is a raw SQL query we need to convert to UTC. Otherwise the timezone provided is silently ignored. From Postgres docs:

If a time zone is specified in the input for time without time zone, it is silently ignored.

Note this is not an issue for production though, because the timezone in production is the default (UTC).

Another solution would be to change the timezone for the test environment to UTC, but there might be a reason for it being Los Angeles time.

Type of change

  • Flaky test fix

How Has This Been Tested?

In order add a test for this that reliably fails, we would need to add a JS package to mock the time in the browser (because travel_to only sets the time for Rails server, but the headless chrome browser for Capybara will still have the regular time).

Instead to reliably reproduce the failure, I manually set the time to the time at which the CI test failed and indeed this change fixes the issue).

diff --git a/app/javascript/application.js b/app/javascript/application.js
index bb8df687..ab6629c1 100644
--- a/app/javascript/application.js
+++ b/app/javascript/application.js
@@ -78,7 +78,7 @@ $(document).ready(function(){
     if (!rangeElement) {
       return;
     }
-    const today = DateTime.now();
+    const today = DateTime.now(2025, 1, 2, 22, 34, 19);
     const startDate = new Date(rangeElement.dataset["initialStartDate"]);
     const endDate = new Date(rangeElement.dataset["initialEndDate"]);

diff --git a/spec/system/distributions_by_county_system_spec.rb b/spec/system/distributions_by_county_system_spec.rb
index 808e7fa2..b48fe52f 100644
--- a/spec/system/distributions_by_county_system_spec.rb
+++ b/spec/system/distributions_by_county_system_spec.rb
@@ -5,6 +5,7 @@ RSpec.feature "Distributions by County", type: :system do
   let(:issued_at_last_year) { Time.current.change(year: current_year - 1).to_datetime }

   before do
+    travel_to Time.zone.utc(2025, 1, 2, 22, 34, 19)
     sign_in(user)
     @storage_location = create(:storage_location, organization: organization)
     setup_storage_location(@storage_location)

Distribution by county system specs are flaky when run at certain times
of day because the system time zone (for Capybara) and server time zone
(Rails) are in PST. But timestamps are stored in the database in UTC
time. Normally this isn't an issue because Active Record will convert
to UTC for queries. But since this is a raw SQL query we need to convert
to UTC.

This is not an issue for production though, because the timezone in
production is UTC.
@coalest coalest requested a review from dorner January 3, 2025 10:50
Copy link
Collaborator

@dorner dorner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@dorner dorner merged commit 4d80126 into rubyforgood:main Jan 7, 2025
11 checks passed
Copy link
Contributor

@coalest: Your PR Fix flaky distributions by county system spec is part of today's Human Essentials production release: 2025.01.12.
Thank you very much for your contribution!

@coalest coalest deleted the fix-flaky-distributions-by-county-system-spec branch January 26, 2025 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants