MRG: Extract measurement date and age for NIRX files

[rob-luke]

What does this implement/fix?

Extract measurement date and birthday for NIRX files.

Additional information

The vendor software only allows saving of age as an integer. So I have taken the measurement date, subtracted the age integer from the years, and stored that as the birthday. Open to alternative suggestions.

[rob-luke]

@larsoner this is failing due to processing of French date strings. Is there any inbuilt multilingual date string parsers in MNE already or do I need to write one?

[rob-luke]

HHmmmm my changes also seem to break other things, Ill fix that. But still curious about the best way to process French dates.

[agramfort]

what do you mean French dates? like month with letter with accents?

[larsoner]

Like Travis says:

ValueError: time data '"jeu. 13 févr. 2020""09:08:47.511"' does not match format '"%a, %b %d, %Y""%H:%M:%S.%f"'

[agramfort]

is it fixable by setting the LOCALE properly?

…

[larsoner]

The docs suggest that locale is used, so indeed if the locale can be extracted from the file header, then we can make a @contextlib.contextmanager to orig = locale.set_locale(locale.LC_something, whatever) and then set_locale(locale.LC_something, orig) afterward. Even if it isn't embedded in the header (which would be sad), we could start making a list of ones to iterate over and try, and always add others later if need be. (Or maybe it won't be so bad to iterate over all of them, if we can get a list?)

[rob-luke]

Thanks for the helpful suggestions. The locale is not embedded in the header. I had started writing a list of locales to iterate over, that's when I stopped and decided to ask here before progressing too far. So it sounds like there is no solution in MNE for this already, so I will continue to code along this path.

[larsoner]

Indeed AFAIK we have never had to deal with locale when parsing datetime. We can try adding some other LC_ env var to some Travis or Azure run to see if we can get some existing datetime stuff to break. Let's do that after you fix the reading of this particular file, though
-- it will probably tell us which env var(s) are relevant.

[agramfort]

here is an example that works: import locale from datetime import datetime locale.setlocale(locale.LC_TIME, "fr_FR") s = "jeu. 13 fév. 2020""09:08:47.511" f = "%a. %d %b. %Y""%H:%M:%S.%f" datetime.strptime(s, f) however I had to use fév. and févr. which seems to be the standard.

[rob-luke]

however I had to use fév. and févr. which seems to be the standard.

Thanks for the code snippet. I came up with something very similar and ran in to the same problem. It's the févr that is the problem, datetime.strptime seems to expect fev. I considered some sort of string replace approach for different languages, but this seems unwieldily (how many languages to I then support?). I also considered other packages that are made for dealing with dates, but don't want to add more requirements.

[rob-luke]

Next Im going to check where this févr actually comes from. This may be a problem of our own creating, I have never seen a file from France and don't know if the device will actually save as fev or févr, this seems to have been generated somewhere in _test_raw_reader.

[agramfort]

I see with company that produces this file how the come up with févr while it appears the standard across languages is fév I would use try except here to avoid failing to load such files but I agree the mne code cannot fix all IO corner cases. can you see with them how it happens? and maybe how they deal with this internally?

…

[larsoner]

@rob-luke will you have time to revisit this in the next couple of weeks? I'll optimistically mark for 0.21

[rob-luke]

Yes I would like to cross this off. The milestone has sept 15 marked, that should be achievable

[rob-luke]

@larsoner I am not sure what to do here with the non standard French encoding of dates.

It seems to me that fév is what datetime expects, yet we have 'févr' in the tests because that was provided in #7313 (I assume, I don't actually have the file).

So if NIRX is providing non standard french date formats, and if I know all the non standard naming, I could create a translation function. But I don't know all the non standard naming and don't have access to a machine at the moment to determine this. Plus this would become quite unwieldy, what other non standard language formats do we then support.

Would it be acceptable to try and parse the dates in the current locale and throw a warning if it doesn't work. For dates that we can't extract it will set them to 0 or something else? This will get English working and downstream usage of the date, and when someone with access to a French system gets some data they can just tweak the French parsing. Thoughts?

[agramfort]

yes I fear we need something custom. But I would reach out to engineers at the company that produces these files to report the issue.

[rob-luke]

I would reach out to engineers at the company that produces these files to report the issue.

Good idea. But I would like to verify the issue myself first if I am to contact them. Ideally someone could send me some files recorded in France. Failing that, I could set the local of our lab pc to France and record my own file, but Im currently locked out of the lab (at least for the next 3 weeks). So this won't be done by 0.21 release date. I suggest I create an English date PR for 0.21 and raise an issue for the non English dates to be solved when someone sends me some files or I can get in the lab. Acceptable? Or do you prefer I wait and do everything at once?

[agramfort]

acceptable

…

On Mon, Sep 7, 2020 at 10:35 AM Robert Luke ***@***.***> wrote: I would reach out to engineers at the company that produces these files to report the issue. Good idea. But I would like to verify the issue myself first if I am to contact them. Ideally someone could send me some files recorded in France. Failing that, I could set the local of our lab pc to France and record my own file, but Im currently locked out of the lab (at least for the next 3 weeks). So this won't be done by 0.21 release date. I suggest I create an English date PR for 0.21 and raise an issue for the non English dates to be solved when someone sends me some files or I can get in the lab. Acceptable? Or do you prefer I wait and do everything at once? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7891 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABHKHHISNMEPIQE53QCN4DSESLOJANCNFSM4N3636UA> .

[rob-luke]

@agramfort @larsoner can you please review.

[rob-luke]

Thanks for the review. @agramfort and @larsoner could you please review again.
CI fail seems to be a time out error

[agramfort]

thx @rob-luke