Use time strings formatted for the locale for homework set dates. #1615

drgrice1 · 2022-03-17T12:02:03Z

Instead of formatting the unix timestamp for display in the inputs for dates in the instructor tools, just use the unix timestamp in those inputs. The flatpickr javascript now hides those inputs and adds an input that the user actually sees. That input contains a localized time string that is formatted for the locale (language) of the course via native javascript's Intl.DateTimeFormat. The flatpickr javascript takes care of updating the value of the hidden input with a unix timestamp as the user changes the visible input.

To make this work well the dates that are only used for display are formatted with a different approach by Perl. The DateTime format_cldr is used for this instead of strftime. This allows for the use of the DateTime::Locale methods which provides uniform formatting for different languages. A special case is added to the lib/WeBWorK/Utils.pm formatDateTime function to handle this.

To be precise, javascript uses Intl.DateTimeFormat with the options { dateStyle: 'short', timeStyle: 'short' }, and perl uses the formatting string returned by $dt->locale->datetime_format_short.

Edit: Now the javascript instead uses the luxon date time library with carefully chosen formats that match the perl formats.

This also makes things work cleaner in the way that the server handles dates because you don't need to worry about parsing a date string back to a unix timestamp.

For most languages the perl DateTime::Locale formatting and javascript formatting are the same. Unfortunately, for a few languages Perl's DateTime::Locale formatting is not correct. This can be seen when editing a problem set values for a user. The user values are formatted by javascript, and the class values by perl. The languages that seem to have problems here are any of the Asian languages. In Korean javascript gives 22. 3. 16. 오전 12:00 and perl gives 22. 3. 16. AM 12:00. Clearly perl is not properly translating the AM/PM part. For zh_CN javascript gives 2022/3/16 00:00 and perl gives 2022/3/16 上午12:00. Javscript gives a 24 hour time, and perl uses a 12 hour time. For zh_hk javascript gives 16/3/2022 上午12:00 and perl gives 2022/3/16 上午12:00. So perl does not seem to use the correct ordering of the year, month, and day for that locale.

Edit: With the recent changes to this pull request, only the Korean language still has differences between the perl and javascript.

Of course this means that the old strftime format %m/%d/%Y at %I:%M%P is not used for set dates anymore. For English the CLDR format returned by datetime_format_short that is now used is M/d/yy, h:mm a. This means instead of 03/16/2022 at 12:00am you get 3/16/22, 12:00 AM. I would prefer 03/16/2022, 12:00 AM, but that is what our locale gives consistently for javascript and perl. So it is probably best to go with the standard.

Note that the old format is still used for set export and import files. In fact that is the only place that the parseDateTime method is still used. It is used there to ensure that the correct timezone is set for the saved dates.

Note that the useDateTimePicker option has been removed. That could be re-implemented if desired, but it would be a bit tedious to do so. The problem is that the date/time picker does not format and parse dates, but if that option is disabled then you would need to. I don't really see the need for that option anymore though.

There is a minor change in the lib/WeBWorK/Utils/LanguageAndDirection.pm file. That was using the po filenames directly for the html lang attribute. Those filenames use underscores which are not valid for the lang attribute. So the underscores are translated to dashes.

Edit: The above comment is no longer valid, and was removed from the code.

Note that this pull request replaces #1333, and it what I meant in the discussion there as to the correct way to do this.

pstaabp · 2022-03-18T14:51:39Z

This seems to work as much as I can tell. I would love to see someone who uses this in a non-English language test it out though.

robert-marik · 2022-03-22T07:22:56Z

Hello, thank you for working on this issue. I will check the patch. However, I do not have last WeBWorK version running and after upgrade my apache2 did not start. Perhaps some missing dependencies. Need a couple of days to resolve. I will let you know.

drgrice1 · 2022-03-22T10:53:36Z

@robert-marik: Yeah, there are several changes that need to be accounted for to get things up and running. We need to get that documented. Note that there are some changes to webwork2/conf/webwork.apache2.4-config that you will need to make (see the dist file). You will need to run npm install in both webwork2/htdocs and pg/htdocs. You will need to make sure the perl Email::Stuffer package is installed (sudo apt install libemail-stuffer-perl will do this).

robert-marik · 2022-04-01T11:01:30Z

Hello. I succeeded to switch branch and was able to test the patch. Thank you again for working on this. I have Europe/Prague timezone. I tried to edit open/close dates in HW sets editor. I can see correct times and days in local format.

taniwallach · 2022-04-03T18:48:00Z

I'm still new to the new "asset build system".
In order to test this, I found that I needed to manually run ./generate-assets.js in the /opt/webwork/webwork2/htdocs.
For some reason, running npm install did not seem to do that.

When I first tested this, I had not set the timezone for the course in the test course I was using nor in docker-compose.yml in Docker, and as a result the code was using a different time-zone in the "date picker tool" and in what was being shown in the input box. The tool was apparently using my local time zone, and converting into the default America/New_York timezone (7 hours earlier at present) for the value shown in the input box.

After setting $siteDefaults{timezone} for my timezone, the date-picker seems to work properly.

However, intentionally setting a course to a different timezone than the local timezone, which is a use case my server supports - it becomes very confusing to use the date-picker, as the tool is certainly using the local timezone, while the input box/stored setting is using the course time-zone.

drgrice1 · 2022-04-03T19:06:16Z

I am not sure what you are saying. It seems to be working correctly for me. If the course default is the site default in $siteDefaults{timezone}, then I see the time in that time zone. If the course default is something different, then I see the time in that different time zone.

drgrice1 · 2022-04-03T19:12:16Z

Also, you should not need to separately run ./generate-assets.js. It should run when you run npm install. What versions of npm and node are you using? Versions of npm and node prior to the ones on Ubuntu 20.04 may not do this, but I don't think they will work for other things as well. If you are using Ubuntu 18.04 you will need to install newer versions of npm and node than those in the Ubuntu repositories. See the "unpublished" release notes at https://webwork.maa.org/wiki/Release_notes_for_WeBWorK_2.17.

taniwallach

I am not sure what to recommend to fix this, but this code behaves very problematically if a user needs to change a date in a course whose timezone is different from his local (the browsers) timezone.

For example, set a course to a time-zone different that your and then edit a date by changing the values of the minutes by a small shift. After saving the change - the hour will be changed. (Pressing tab has essentially the same effect.) It seems that this is a result of the "text value" being parsed in the browser's local timezone and then converted into the course time-zone.

Apparently, flatpickr is known to be quite problematic in terms of time-zone support.
See:

taniwallach · 2022-04-03T19:29:18Z

Also, you should not need to separately run ./generate-assets.js. It should run when you run npm install. What versions of npm and node are you using? Versions of npm and node prior to the ones on Ubuntu 20.04 may not do this, but I don't think they will work for other things as well. If you are using Ubuntu 18.04 you will need to install newer versions of npm and node than those in the Ubuntu repositories. See the "unpublished" release notes at https://webwork.maa.org/wiki/Release_notes_for_WeBWorK_2.17.

I was running npm install inside the Docker container (Ubuntu 20.04). It did not seem to run the code to generate the assets.

root@myhost:/opt/webwork/webwork2/htdocs# npm --version
6.14.4
root@myhost:/opt/webwork/webwork2/htdocs# npm install
npm WARN lifecycle webwork.javascript_package_manager@~prepare: cannot run in wd webwork.javascript_package_manager@ npm run generate-assets (wd=/opt/webwork/webwork2/htdocs)
npm WARN [email protected] requires a peer of @popperjs/core@^2.10.2 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 128 packages in 5.075s

27 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

taniwallach · 2022-04-03T19:31:40Z

Here is a movie of the unusual behavior when the browser timezone and course timezone are different.

Timezone-issue.mp4

drgrice1 · 2022-04-03T19:40:33Z

I can't see your video. It seems to be corrupt. Although, now I see your problem. If you open the time picker and select a time, that time is not what ends up in the input. It gets adjusted by the time zone. I will see what I can do about that.

drgrice1 · 2022-04-03T19:42:41Z

By the way, it isn't flatpickr that is the problem. flatpickr is already in use in the develop branch, and this does not happen there. It is the locale time formatting that I added in this pull request that is the problem.

drgrice1 · 2022-04-04T19:56:39Z

@taniwallach: You never answered my question about the dir=ltr setting on the date/time inputs. Is that needed? It seems that behavior is quite odd as it is.

drgrice1 · 2022-04-05T14:18:17Z

@taniwallach: As to running npm install inside of a docker container, make sure that you add the --unsafe-perm flag. Then it will run the prepare script. Note that when the image is built npm install --unsafe-perm is run, and the prepare script is executed, so the static assets are generated at that time. This is needed because you are runing npm install as root. Also note that with later versions of npm the --unsafe-perm option has been removed, and will not be needed.

taniwallach · 2022-04-05T20:15:36Z

@taniwallach: You never answered my question about the dir=ltr setting on the date/time inputs. Is that needed? It seems that behavior is quite odd as it is.

I sent a response inside #1615 (comment) .
I'm not sure what is the best thing for a date+time field. Bidirectional behavior is often inconvenient in form fields. I have no objections to adding a dir=ltr setting here. In the Hebrew locale - no Hebrew is actually being used and dir=ltr would probably behave better. It might not be the right thing for Arabic or some other RTL languages, but I think end users will get used to whatever there is.

drgrice1 · 2022-04-05T20:24:56Z

@taniwallach: Sorry, I missed that. Thanks. I will add the dir=ltr setting then.

drgrice1 · 2022-04-05T20:30:15Z

You should note that this only supports the languages that we have translations for. So this does not support Arabic at this point. Each language that we support will now need a format added to datepicker.js and problemsetlist.js. All of our current languages already have formats in those files. The formats should be chosen to be consistent with the formats that perl's DateTime::Locale pacakge uses.

drgrice1 · 2022-04-05T20:30:59Z

@taniwallach: So have you had a chance to test this again since I fixed everything?

taniwallach

In terms of the functionality of the new code to modify dates - everything I tested works properly now. I tested in Hebrew for the various places where a set deadline can be edited for all users or a specific user.

The change to use dir="ltr" certainly makes the date+time input boxes behave more conveniently for Hebrew courses, but the formatting is not fully consistent. I think that for the sake of consistency the places in the instructor tools put the date+time as a fixed string (ex. the table in the list of sets in the set editor) should also have dir="ltr" applied to them. This is already the case in the table of sets assigned to a user ("UserDetail-date-table") for the class-wide fixed date.

If you do make that change - would you also try to force the "Set name" fields to also use dir="ltr" as when there are spaces in the set name, and a RTL language is used the name gets garbled by the RTL direction.

taniwallach · 2022-04-06T05:22:48Z

lib/WeBWorK/Utils/LanguageAndDirection.pm

@@ -63,7 +63,8 @@ Arabic ("ar") trigger the RTL direction setting.
 =cut

 sub get_lang_and_dir {
-	my $lang = shift;
+	# en_us, fr_CA, etc., are not valid html language codes.  They should be en-us, fr-CA, etc.
+	my $lang = shift =~ s/_/-/gr;


taniwallach · 2022-04-06T05:23:13Z

lib/WeBWorK/Utils/LanguageAndDirection.pm

@@ -112,7 +113,8 @@ In some cases, the return result is empty.
 sub get_problem_lang_and_dir {
 	my $pg_flags = shift;
 	my $requested_mode = shift;
-	my $lang = shift;
+	# en_us, fr_CA, etc., are not valid html language codes.  They sould be en-us, fr-CA, etc.
+	my $lang = shift =~ s/_/-/gr;


taniwallach · 2022-04-06T09:44:59Z

htdocs/js/apps/DatePicker/datepicker.js

@@ -26,23 +44,83 @@
 		for (const rule of groupRules) {
 			const orig_value = rule.value;

-			flatpickr(rule.parentNode, {
+			luxon.Settings.defaultLocale = rule.dataset.locale?.replaceAll(/_/g, '-') ?? 'en';


Is the replaceAll needed here after the change to the PO file names made in #1632 ?

taniwallach · 2022-04-06T09:54:51Z

htdocs/js/apps/ProblemSetList/problemsetlist.js

 	// Initialize the date/time picker for the import form.
 	const importDateShift = document.getElementById('import_date_shift');
 	if (importDateShift) {
-		flatpickr(importDateShift.parentNode, {
+
+		luxon.Settings.defaultLocale = importDateShift.dataset.locale?.replaceAll(/_/g, '-') ?? 'en';


drgrice1 · 2022-04-06T11:04:34Z

I see. I missed adding the dir="ltr" on the dates in the ProblemSetList main page (the Hmwk Sets Editor main page).

Yeah, the underscore to space substitutions aren't needed anymore. This pull request came before the one that changed the locale file names, and so that was needed. I will remove that.

I can look into adding dir="ltr" to the set name inputs. Where exactly is that needed?

drgrice1 · 2022-04-06T12:02:34Z

I will deal with the set names in a separate pull request. That is unrelated to this one. Although, I would like to talk to you in general about this. Is there any possibility that you could meet via zoom sometime today? I can meet anytime. I was looking into this, and it may be that instead of adding dir="ltr" we need instead to use the css unicode-bidi.

pstaabp · 2022-04-07T12:43:34Z

Looks like @robert-marik approved this. @drgrice1 let's merge after the conflicts.

drgrice1 · 2022-04-07T12:44:13Z

Working on it now.

Instead of formatting the unix timestamp for display in the inputs for dates in the instructor tools, just use the unix timestamp in those inputs. The flatpickr javascript now hides those inputs and adds an input that the user actually sees. That input contains a localized time string that is formatted for the locale (language) of the course via native javascript's Intl.DateTimeFormat. The flatpickr javascript takes care of updating the value of the hidden input with a unix timestamp as the user changes the visible input. To make this work well the dates that are only used for display are formatted with a different approach by Perl. The DateTime format_cldr is used for this instead of strftime. This allows for the use of the DateTime::Locale methods which provides uniform formatting for different languages. A special case is added to the lib/WeBWorK/Utils.pm formatDateTime function to handle this. To be precise, javascript uses Intl.DateTimeFormat with the options `{ dateStyle: 'short', timeStyle: 'short' }`, and perl uses the formatting string returned by `$dt->locale->datetime_format_short`. This also makes things work cleaner in the way that the server handles dates because you don't need to worry about parsing a date string back to a unix timestamp. For most languages the perl DateTime::Locale formatting and javascript Intl.DateTimeFormat formatting are the same. Unfortunately, for a few languages Perl's DateTime::Locale formatting is not correct. This can be seen when editing a problem set values for a user. The user values are formatted by javascript, and the class values by perl. The languages that seem to have problems here are any of the Asian languages. In Korean javascript gives `22. 3. 16. 오전 12:00` and perl gives `23. 3. 16. AM 12:00`. Clearly perl is not properly translating the AM/PM part. For zh_CN javascript gives `2022/3/16 00:00` and perl gives `2022/3/16 上午12:00`. Javscript gives a 24 hour time, and perl uses a 12 hour time. For zh_hk javascript gives `16/3/2022 上午12:00` and perl gives `2022/3/16 上午12:00`. So perl does not seem to use the correct ordering of the year, month, and day for that locale. Of course this means that the old strftime format `%m/%d/%Y at %I:%M%P` is not used for set dates anymore. For English the CLDR format returned by `datetime_format_short` that is now used is `M/d/yy, h:mm a`. This means instead of `03/16/2022 at 12:00am` you get `3/16/22, 12:00 AM`. I would prefer `03/16/2022, 12:00 AM`, but that is what our locale gives consistently for javascript and perl. So it is probably best to go with the standard. Note that the old format is still used for set export and import files. In fact that is the only place that the parseDateTime method is still used. It is used there to ensure that the correct timezone is set for the saved dates. Note that the `useDateTimePicker` option has been removed. That could be re-implemented if desired, but it would be a bit tedious to do so. The problem is that the date/time picker does not format and parse dates, but if that option is disabled then you would need to. I don't really see the need for that option anymore though. There is a minor change in the lib/WeBWorK/Utils/LanguageAndDirection.pm file. That was using the po filenames directly for the html `lang` attribute. Those filenames use underscores which are not valid for the `lang` attribute. So the underscores are translated to dashes.

the browser's timezone and the course timezone.

instead of using javascript Intl formats for date/times, use format strings chosen to match the perl DateTime::Locale formats. We have to use formats in any case to correctly parse strings back to a unix timestamp. So we might as well use them both ways, and make the perl display match that of the javascript better. There is only one exception. That is for the Korean language. The perl DateTime::Locale does not translate the AM/PM meridian, and the javascript does. There isn't much that can be done about that except for fixing the perl DateTime::Locale data.

accessibility purposes.

languages.

That is not needed anymore now that they have been renamed. Add `dir="ltr"` to the dates in the ProblemSetList.pm main table.

drgrice1 · 2022-04-07T13:21:38Z

This is now updated.

drgrice1 force-pushed the time-locale branch from f270100 to 6f75b37 Compare March 17, 2022 14:35

pstaabp added Multilingual Part of the Multilingual (localization) project Put comment in release notes labels Mar 18, 2022

drgrice1 requested review from robert-marik and taniwallach March 18, 2022 18:01

drgrice1 force-pushed the time-locale branch 2 times, most recently from 7c4f70b to 8a49479 Compare March 21, 2022 23:31

drgrice1 force-pushed the time-locale branch from 8a49479 to 877058d Compare March 22, 2022 20:30

drgrice1 mentioned this pull request Mar 22, 2022

Update the javascript for SetMaker (the library browser). #1619

Merged

drgrice1 force-pushed the time-locale branch 5 times, most recently from f53e98e to b65fc96 Compare March 31, 2022 15:06

pstaabp mentioned this pull request Apr 1, 2022

Use en_US format whenever date and time are intended to be in form fields #1333

Closed

drgrice1 force-pushed the time-locale branch 3 times, most recently from f9de8e0 to 1fbb938 Compare April 1, 2022 19:46

taniwallach requested changes Apr 3, 2022

View reviewed changes

drgrice1 force-pushed the time-locale branch from a47d789 to cb17422 Compare April 4, 2022 20:01

drgrice1 requested a review from taniwallach April 5, 2022 14:12

taniwallach approved these changes Apr 6, 2022

View reviewed changes

drgrice1 force-pushed the time-locale branch from 718be36 to 31fac5b Compare April 6, 2022 12:29

drgrice1 force-pushed the time-locale branch from 31fac5b to 1582405 Compare April 7, 2022 13:02

drgrice1 added 5 commits April 7, 2022 08:02

Adjust the times used by flatpickr to account for the difference between

9ad7ce3

the browser's timezone and the course timezone.

Make the date picker button activatable via the keyboard for

63433b8

accessibility purposes.

Make the date/time inputs always left-to-right, even for right-to-left

c55db23

languages.

drgrice1 force-pushed the time-locale branch from 1582405 to 4c1f6fc Compare April 7, 2022 13:02

Remove the translation from underscore to space in the locale names.

4f20299

That is not needed anymore now that they have been renamed. Add `dir="ltr"` to the dates in the ProblemSetList.pm main table.

drgrice1 force-pushed the time-locale branch from 4c1f6fc to 4f20299 Compare April 7, 2022 13:14

pstaabp merged commit 943664f into openwebwork:develop Apr 7, 2022

drgrice1 deleted the time-locale branch April 7, 2022 13:39

robert-marik mentioned this pull request Jul 1, 2022

Localization of time broken in the set editor #1164

Closed

pstaabp removed the Put comment in release notes label Mar 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use time strings formatted for the locale for homework set dates. #1615

Use time strings formatted for the locale for homework set dates. #1615

drgrice1 commented Mar 17, 2022 •

edited

Loading

pstaabp commented Mar 18, 2022

robert-marik commented Mar 22, 2022

drgrice1 commented Mar 22, 2022

robert-marik commented Apr 1, 2022

taniwallach commented Apr 3, 2022

drgrice1 commented Apr 3, 2022 •

edited

Loading

drgrice1 commented Apr 3, 2022

taniwallach left a comment

taniwallach commented Apr 3, 2022

taniwallach commented Apr 3, 2022

drgrice1 commented Apr 3, 2022

drgrice1 commented Apr 3, 2022

drgrice1 commented Apr 4, 2022

drgrice1 commented Apr 5, 2022 •

edited

Loading

taniwallach commented Apr 5, 2022

drgrice1 commented Apr 5, 2022

drgrice1 commented Apr 5, 2022

drgrice1 commented Apr 5, 2022

taniwallach left a comment

taniwallach Apr 6, 2022

taniwallach Apr 6, 2022

taniwallach Apr 6, 2022

taniwallach Apr 6, 2022

drgrice1 commented Apr 6, 2022

drgrice1 commented Apr 6, 2022

pstaabp commented Apr 7, 2022

drgrice1 commented Apr 7, 2022

drgrice1 commented Apr 7, 2022

Use time strings formatted for the locale for homework set dates. #1615

Use time strings formatted for the locale for homework set dates. #1615

Conversation

drgrice1 commented Mar 17, 2022 • edited Loading

pstaabp commented Mar 18, 2022

robert-marik commented Mar 22, 2022

drgrice1 commented Mar 22, 2022

robert-marik commented Apr 1, 2022

taniwallach commented Apr 3, 2022

drgrice1 commented Apr 3, 2022 • edited Loading

drgrice1 commented Apr 3, 2022

taniwallach left a comment

Choose a reason for hiding this comment

taniwallach commented Apr 3, 2022

taniwallach commented Apr 3, 2022

drgrice1 commented Apr 3, 2022

drgrice1 commented Apr 3, 2022

drgrice1 commented Apr 4, 2022

drgrice1 commented Apr 5, 2022 • edited Loading

taniwallach commented Apr 5, 2022

drgrice1 commented Apr 5, 2022

drgrice1 commented Apr 5, 2022

drgrice1 commented Apr 5, 2022

taniwallach left a comment

Choose a reason for hiding this comment

taniwallach Apr 6, 2022

Choose a reason for hiding this comment

taniwallach Apr 6, 2022

Choose a reason for hiding this comment

taniwallach Apr 6, 2022

Choose a reason for hiding this comment

taniwallach Apr 6, 2022

Choose a reason for hiding this comment

drgrice1 commented Apr 6, 2022

drgrice1 commented Apr 6, 2022

pstaabp commented Apr 7, 2022

drgrice1 commented Apr 7, 2022

drgrice1 commented Apr 7, 2022

drgrice1 commented Mar 17, 2022 •

edited

Loading

drgrice1 commented Apr 3, 2022 •

edited

Loading

drgrice1 commented Apr 5, 2022 •

edited

Loading