Handle EGI MFF files with multiple epochs

[rhancockn]

mne.io.read_raw_egi does not properly process MFF files with multiple epochs. In the worst case, the function returns, but the events are misaligned with respect to the data. In the best case, the function raises an exception as events are created beyond the end of the recording as below. The event start times need to be adjusted to account for any time gaps between the epochs.

 data = mne.io.read_raw_egi('test.mff')
Reading EGI MFF Header from test.mff...
    Reading events ...
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-2-b23e6b31e949> in <module>()
----> 1 data = mne.io.read_raw_egi('test.mff')

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/egi.py in read_raw_egi(input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/utils.py in verbose(function, *args, **kwargs)
    727         with use_log_level(verbose_level):
    728             return function(*args, **kwargs)
--> 729     return function(*args, **kwargs)
    730 
    731 

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/egi.py in read_raw_egi(input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)
    164     if input_fname.endswith('.mff'):
    165         return _read_raw_egi_mff(input_fname, montage, eog, misc, include,
--> 166                                  exclude, preload, channel_naming, verbose)
    167     return RawEGI(input_fname, montage, eog, misc, include, exclude, preload,
    168                   channel_naming, verbose)

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/egimff.py in _read_raw_egi_mff(input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/utils.py in verbose(function, *args, **kwargs)
    727         with use_log_level(verbose_level):
    728             return function(*args, **kwargs)
--> 729     return function(*args, **kwargs)
    730 
    731 

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/egimff.py in _read_raw_egi_mff(input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)
    258     """
    259     return RawMff(input_fname, montage, eog, misc, include, exclude,
--> 260                   preload, channel_naming, verbose)
    261 
    262 

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/egimff.py in __init__(self, input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/utils.py in verbose(function, *args, **kwargs)
    727         with use_log_level(verbose_level):
    728             return function(*args, **kwargs)
--> 729     return function(*args, **kwargs)
    730 
    731 

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/egimff.py in __init__(self, input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)
    278 
    279         logger.info('    Reading events ...')
--> 280         egi_events, egi_info = _read_events(input_fname, egi_info)
    281         gains = _get_gains(op.join(input_fname, egi_info['info_fname']))
    282         if egi_info['value_range'] != 0 and egi_info['bits'] != 0:

/Users/roh17004/anaconda3/lib/python3.6/site-packages/mne/io/egi/events.py in _read_events(input_fname, info)
     30     for n, event in enumerate(event_codes):
     31         for i in mff_events[event]:
---> 32             events[n][i] = n + 1
     33     return events, info
     34 

IndexError: index 413166 is out of bounds for axis 0 with size 413096

[agramfort]

duplicate of #5027 ?

[rhancockn]

I think #5027 is referring to reading files that have been segmented in NetStation and thus have part of the original data blocks missing. This issue deals with reading in raw recordings.

[rhancockn]

Actually a duplicate of #4759 but not fixed by 261a678.

[larsoner]

@rhancockn will you be able to get back to this?

[rhancockn]

@larsoner Yes, I have a first pass at a fix, but it fails some preload=False test cases. I will take another look at getting this integrated and passing.

[mmagnuski]

In the worst case, the function returns, but the events are misaligned with respect to the data.

Oops, I was not aware of this, I will have to check all my preprocessed files for presence of pauses. I never had an error reading mff files, although I am sure some of them had pauses. My attention was drawn to these issues only because I found that in some recordings I lost a lot of events and noticed that these files had pauses in epochs.xml.

[rhancockn]

This issue is likely to affect your data if you are using E-prime with ECI events.

…

On Mar 31, 2019, at 11:08 AM, Mikolaj Magnuski ***@***.***> wrote: In the worst case, the function returns, but the events are misaligned with respect to the data. Oops, I was not aware of this, I will have to check all my preprocessed files for presence of pauses. I never had an error reading mff files, although I am sure some of them had pauses. My attention was drawn to these issues only because I found that in some recordings I lost a lot of events and noticed that these files had pauses in epochs.xml. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mmagnuski]

I have checked and the issue does affects my data - the events are clearly misaligned with the signal in subjects with pauses (no ERP for example). I was using Psychopy with LPT triggers.

[larsoner]

So @mmagnuski should we review+merge this as is, then you (or someone else) make another PR to fix the multiple epoch support?

[mmagnuski]

Yes, I'll just push a cosmetic change to this PR (I have it locally, but I have traumatic memories of pushing to a PR so left it for the evening). Then I would merge as soon as possible and add/fix stuff in subsequent prs. Merging this PR even without a test is much better than having mff reading as it works in master now - small pause at the beginning of the recording removes all time-locked activity without error or warning.

[mmagnuski]

Pushing to a PR turned out to be too much hassle again, I just commented in the PR. I will have to preprocess again one of the files I had problems with and then I would be able to confirm.

[larsoner]

@rhancockn are you able to produce a small file (as small as possible that shows the issue) for testing ?

[rhancockn]

There is a ~4MB file with events in multiple epochs at mne-tools/mne-testing-data#37. I'm not able to generate a smaller file until my University reopens.

[larsoner]

Ahh right forgot about that, thanks @rhancockn . That should be good enough for at least developing a solution for people to test in the meantime.