Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] Add check_data mode to deal with running with/without nan values #182

Merged
merged 5 commits into from
Aug 9, 2020

Conversation

TomDonoghue
Copy link
Member

Responds to #175

This PR adds the check_data mode to FOOOF objects, which controls if added data is checked for NaN/Inf data-points, and fails out if so. By default, check_data is turned on, in which case objects act the same as before.

However, this mode can be turned off, in which case the object plays nice with having NaN/Inf data. Model fitting will cleanly fail (no error) and so fitting can continue. This allows for fitting groups of spectra in which some spectra are NaN, and makes it easier to deal with messy shaped data in which some spectra may be missing, etc.

So far the decision here is to not change default behaviour, but add an option for users to bypass some of the data checks. This, I think, is a safer option - the user has to choose when to ignore NaN data, and should only do so if they know why it is happening, etc.

Note: this is an issue brought up by @mwprestonjr. MJ - whenever you get a chance, can you have a look at this proposal, and see if it addresses your use case, and check in whether you think this is a good solution for your purposes? Thanks!

Examples

By default, this code will error, as before:

from fooof import FOOOF
from fooof.sim import gen_power_spectrum

freqs, pows = gen_power_spectrum([3, 50], [1, 1], [10, 0.2, 1])
pows[:] = np.nan

fm = FOOOF()
fm.fit(freqs, pows)

But, with explicit toggle of the check data mode, this will no longer throw an explicit error:

fm = FOOOF()
fm.set_check_data_mode(False)
fm.fit(freqs, pows)

This is probably most relevant for cases, with FOOOFGroup, whereby some spectra may be null:

from fooof import FOOOF, FOOOFGroup
from fooof.sim import gen_group_power_spectra

freqs, pows = gen_group_power_spectra(3, [3, 50], [1, 1], [10, 0.2, 1])
pows[1, :] = np.nan

fg = FOOOFGroup()
fg.set_check_data_mode(False)
fg.fit(freqs, pows)

Note, in the above, that by default (or if set_check_data_mode(True) is called, which returns the object to default mode of having check_data on), the above code fails.

@TomDonoghue TomDonoghue requested a review from ryanhammonds July 19, 2020 01:30
@TomDonoghue TomDonoghue changed the title Add check_data mode to deal with running with/without nan values [ENH] Add check_data mode to deal with running with/without nan values Jul 21, 2020
Copy link
Contributor

@ryanhammonds ryanhammonds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @TomDonoghue, I went through this and left a comment about a change.

Do you think it could be useful to implement a ignore_nans option to drop nan values from the spectrum to allow fitting to exit successfully?

fooof/objs/fit.py Outdated Show resolved Hide resolved
@TomDonoghue TomDonoghue added the 2.0 Targetted for the specparam 2.0 release. label Jul 26, 2020
@TomDonoghue
Copy link
Member Author

Just for tracking: I got an update from @mwprestonjr that this does seem to address his use case - so I think we should be all good here.

Merging in.

@TomDonoghue TomDonoghue merged commit 7a520c4 into master Aug 9, 2020
@TomDonoghue TomDonoghue deleted the nans branch August 9, 2020 20:49
@TomDonoghue TomDonoghue added 1.1 Targetted for a fooof 1.1 release. and removed 2.0 Targetted for the specparam 2.0 release. labels Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1.1 Targetted for a fooof 1.1 release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants