Not all events included after mne.io.read_raw_eeglab()

[timmidee]

After reading in a dataset from eeglab, I notice not all events have been read.

I do:
raw = mne.io.read_raw_eeglab(raw, event_id=ev_id, preload=True)
raw.drop_channels(['L_GAZE_X','L_GAZE_Y']) # Drop eye tracker channels
montage = mne.channels.read_montage(mfile)
mne.channels.montage._set_montage(raw.info, montage)
evs = mne.find_events(raw,shortest_event = 0,stim_channel = 'STI 014')

With ev_id containing all possible types, as found by unique({EEG.event.type}) in Matlab

the resulting evs is 7 positions shorter than length(EEG.event) in Matlab.

It is NOT that certain types (from EEG.event.type) are not being read in at all, rather that not all events of all types are read (so I'm missing 4 of one type, 1 each of three other types). Any idea what might be the cause of this?

Apart from the obvious potential problems of missing triggers, I want to do something similar to #3837 but I can not line up the information between EEGlab and MNE if not all triggers are imported.

[jaeilepp]

I assume this has something to do with how the event ids are mapped to integers. You could run read_raw_eeglab with verbose=True to see if it throws any warnings.

[timmidee]

Thank you for the suggestion, I just tried this but no warnings are issued.

raw = mne.io.read_raw_eeglab(raw, event_id=ev_id, preload=True, verbose=True)

results in:

Reading /Volumes/(...)/ss0018_ET_EEG.fdt
Reading 0 ... 2582049 = 0.000 ... 2582.049 secs...

edit:
Perhaps it is useful to know that (as I mentioned somewhat cryptically before) I am manually specifying the mapping of all types of events that are in the data (according to EEGlab), to integers, by specifying them in event_id.

[jaeilepp]

Hmm. That's surprising since there is a check for if len(events) < len(types): and it should issue a warning.

I would start debugging by checking which events exactly are excluded and where exactly are they dropped. If you can share the data I can also take a look.

[jasmainak]

@jona-sassenhagen perhaps you can help? It's part of the code you wrote

@timmidee can you share the file + small code to facilitate debugging. Also, take a look at event_id_func. Maybe that will help

[timmidee]

I got some IRL help from @jona-sassenhagen on this. It's not resolved yet, but he has a copy of the file.

As far as I understand this, the triggers in the .set file seem to be ok and to be read correctly by the internal functions in mne.io.eeglab.eeblab, but somehow they seem to end up in the stimulus channel incorrectly, or to be incorrectly retrieved by mne.find_events().

For now I can read out the events with a work around (in case others have this problem):
mat = io.loadmat(raws[0], struct_as_record=False, squeeze_me=True, uint16_codec=None)['EEG'] evs = mne.io.eeglab.eeglab._read_eeglab_events(mat,ev_id)

This gives me evs with all events and that lines up with the custom event info that I want to align from the eeglab events structure.

[mmagnuski]

this is partly related to original_events idea from #3837

[jona-sassenhagen]

Ok, I think I've found the problem. It's due to perfectly overlapping events. `events` is first constructed correctly by _read_eeglab_events. But then, by the conversion to a stim channel, and then by reconstructing an events array from the stim channel, events with identical time points are lost. Proposed solution: - check if len(set(latencies)) == len(latencies) and if not, warn - @timmidee , the problem is not trivial to solve because we simply cannot store two simultaneous events in the stim channel. It's probably easiest to work with the hand-read events array (with _read_eeglab_events) for now

…

On 27 Jan 2017, at 14:35, Mainak Jas ***@***.***> wrote: @jona-sassenhagen <https://github.com/jona-sassenhagen> perhaps you can help? It's part of the code you wrote @timmidee <https://github.com/timmidee> can you share the file + small code to facilitate debugging. Also, take a look at event_id_func. Maybe that will help — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3938 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEHyItaF9FWKnTypCs3Sgc9f6JoAVvArks5rWfKvgaJpZM4LvQOC>.

[jona-sassenhagen]

So what's the plan - I think we discussed this at the sprint, but I forgot the outcome ... How about making _read_eeglab_events public + a warning linking to it?

[timmidee]

Long term I think one way to tackle both this problem and other issues, is if there is additional information one can store with an event. The stim channel can carry the single event that "survives", the overlapping events can be offloaded as "additional information" without getting completely lost.

Such a solution would require a warning and perhaps be much more elaborate, but it would not require making additional functions public. Nor would it be necessary for the user to handle the events separate from the objects and constantly adapt them along with the objects (as I am constantly doing myself at the moment, but that's a different problem from just reading in the events)

May I, by the way, be so bold to ask why the events "time line" has to take the form of a data channel at all?

As an intermediate solution I would be in favour (as far as I have a say in this) of @jona-sassenhagen 's solution of having _read_eeglab_events() available in one form or another. That way at least the events are accessible and the user can do with them as they like.

[larsoner]

Long term I think one way to tackle both this problem and other issues, is if there is additional information one can store with an event... [can I] ask why the events "time line" has to take the form of a data channel at all?

Yeah at the recent code sprint we had some discussions along these lines (see also #3837, #4125). One problem is that our output format (FIF) doesn't have a good way of storing arbitrary event-related information. The events get stored as a data channel because that's how FIF stores events, and until recently, the drawbacks of this approach haven't been so glaringly obvious :) @jona-sassenhagen recently came up with

In expanding our own support for arbitrary event-related information coding, we risk recreating an inferior version of a Pandas DataFrame, which wouldn't be great. But at the same time, telling people "use this other package and have parallel files" isn't entirely satisfying, and also doesn't work well with e.g. read_raw_eeglab.

In the meantime, I agree with @jona-sassenhagen, @timmidee, and others (probably also @mmagnuski) that for RawEEGLAB we can make read_eeglab_events public, or just expose it as a property like raw.eeglab_events or so. The critical thing is that we need to make it clear that such data will not be saved with .save() and so people should save it elsewhere themselves if they need it. This would allow us to punt on the issue of richer events for a little bit longer (hopefully in a future-compatible way) while not holding up @timmidee and @nbara.

Is there some volunteer familiar with EEGLAB for trying this? @jona-sassenhagen?

[jona-sassenhagen]

Yup, me or @jasmainak . IIRC I wrote the function anyways.

@Eric89GXL see the last reply here: #4124 for "rich predictor" data set.

[jasmainak]

Maybe I'm missing something, but overlapping events can now be handled using annotation objects. Why do we not make use of that?

[agramfort]

annotations and events are different concepts. Events refer to paradigm why annotation refer to the data: - presence of blink / movement artifact / - sleep spindle, epileptic spike etc.

[abhiishekpal]

I have currently acquired an .mff data, but upon reading it with mne.io.read_raw_egi() my obtained data length is shorter. My EEG data length is shorter by about 50 seconds.

Can someone please help me with this issue.

[mmagnuski]

@abhiishekpal Are you sure your data is shorter? Maybe your evenst are just shifted because you had a pause in the recording? Take a look at #5374 and #5373 for example. This issue should be fixed soon.

[mmagnuski]

BTW - this issue is not related to reading mff files so its better to move the conversation to some issue that is. :)

[abhiishekpal]

@mmagnuski probably yes, MNE shows a total of 971 events, whereas EEGLAB shows a total of 1004 events(which is the correct count)

[mmagnuski]

Could you check in your mff folder in the epochs.xml file if you have multiple epochs (that would suggest pauses in the recording and thus event alignment problem).