Feature request: EGI MFF reader for discontinuous files

[christian-oreilly]

Code:

raw = mne.io.read_raw_egi(raw_fname)  # already has an average reference

Result:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-4-0a72d29f7e68> in <module>()
      1 raw_fname = '/media/oreilly/ETUDE/suj1 session1/signal1.bin'
      2 
----> 3 raw = mne.io.read_raw_egi(raw_fname)  # already has an average reference
      4 events = mne.find_events(raw, stim_channel='STI 014')
      5 

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/io/egi/egi.py in read_raw_egi(input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/utils.py in verbose(function, *args, **kwargs)
    726         with use_log_level(verbose_level):
    727             return function(*args, **kwargs)
--> 728     return function(*args, **kwargs)
    729 
    730 

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/io/egi/egi.py in read_raw_egi(input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)
    166                                  exclude, preload, channel_naming, verbose)
    167     return RawEGI(input_fname, montage, eog, misc, include, exclude, preload,
--> 168                   channel_naming, verbose)
    169 
    170 

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/io/egi/egi.py in __init__(self, input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/utils.py in verbose(function, *args, **kwargs)
    726         with use_log_level(verbose_level):
    727             return function(*args, **kwargs)
--> 728     return function(*args, **kwargs)
    729 
    730 

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/io/egi/egi.py in __init__(self, input_fname, montage, eog, misc, include, exclude, preload, channel_naming, verbose)
    182         with open(input_fname, 'rb') as fid:  # 'rb' important for py3k
    183             logger.info('Reading EGI header from %s...' % input_fname)
--> 184             egi_info = _read_header(fid)
    185             logger.info('    Reading events ...')
    186             egi_events = _read_events(fid, egi_info)  # update info + jump

~/python_venv/venv-3.4/lib/python3.5/site-packages/mne/io/egi/egi.py in _read_header(fid)
     66         info['event_codes'] = np.array(info['event_codes'])
     67     else:
---> 68         raise NotImplementedError('Only continuous files are supported')
     69     info['unsegmented'] = unsegmented
     70     info['dtype'], info['orig_format'] = {2: ('>i2', 'short'),

NotImplementedError: Only continuous files are supported

[Tested with a fresh !pip3 install --upgrade git+https://github.com/mne-tools/mne-python.git install]

This is a feature request so that we can read these files with MNE-Python.

[larsoner]

Now that we have support for FIF files where acquisition was stopped and restarted in the same FIF file, assuming it's similar for EGI MFF it should be easier to add this.

[agramfort]

I think it's a dupe of #4759

[christian-oreilly]

It seems to be related to the second point of #4759, but the error message is different so I am not totally sure.

[agramfort]

I won't have the bandwidth to look into it. But I would suggest to at least share a file with a link here and hopefully someone will look into it.

[christian-oreilly]

Unfortunately, these recordings are not mine so I am not at liberty to share them. Furthermore, each recording is 30 GB, so not conveniently shared over Internet. FYI, if looking for additional source of inspiration (beside the FIF reader), the Wonambi package (https://github.com/wonambi-python/wonambi) loads these recording correctly.

[agramfort]

thanks for the pointer to wonambi without a test file it will be hard to debug :(

[christian-oreilly]

Yes, I know. I don't have access to EGI NetStation, else I could crop a small part of the recording. I am not sure if there are any free MFF writer available (finding readers is already hard enough), but if I found some, I'll submit a cropped file for debugging.

[mmagnuski]

The file you are trying to load is raw_fname = '/media/oreilly/ETUDE/suj1 session1/signal1.bin', so it does not end with .mff (or .raw), are you sure this is the correct file?

[christian-oreilly]

.RAW are not MFF files, they are an alternative format that can output EGI NetStation. Correct me if I am wrong (and I am not sure that this is not obfuscated by some operating system), but what we call "MFF" files are actually more a repository containing a set of XML files (for the meta data) and a set (often one) of .bin files containing the actual signals. In Wonambi, the path to the "MFF file" would actually be "/media/oreilly/ETUDE/suj1 session1" but MNE seems to expect directly the path to the .bin file. I think the path expected by Wonambi here is more accurate, it should point to the whole folder.

[mmagnuski]

@christian-oreilly You may be correct, I have experience with .raw files only (but I have a few .mff so I could test). I know that mff is a collection of multiple files, however my expectation from the docs was that you should pass the path to the mff, not the bin. But the docs are not very clear on that. Do you pass the path to .bin file based on docs or inspecting the code?

[mmagnuski]

Actually, I just checked and the code does:

if input_fname.endswith('.mff'):
    # read as mff file

so you should pass the path to the mff "file", not the bin.
I you pass the path to the bin it assumes it is a .raw file.
I think the docs should be clarified and maybe the code should specifically test for .mff vs .raw and throw an error if file ending is different.

[mmagnuski]

@christian-oreilly Are you up for a PR? :)

[ViridianForge]

Kind of wandering in here at random, but I'm presently working with Dr. Don Tucker on reading some of these discontinuous .mff files myself.

I can confirm @mmagnuski that an .mff appears to be a folder containing XML describing the data and the signal in .bin files.

Where I've started poking around on our end to try and read a discontinuous file is that epochs.xml - which appears to list the start and stop time of each epoch of data in microseconds (waiting on a Philips Neuro programmer to confirm this), and lists the first and last block contained in that epoch.

If sample data would be helpful, I can consult with Dr. Tucker to see if we have clearance to share one of our smaller .mffs (I think in the realm of 250-500MB).

[christian-oreilly]

@mmagnuski Pointing directly on the .bin file was based on trial and error. More precisely, pointing to the repository gives an exception (IsADirectoryError). But I just realized that this was due to the fact that the repository does not end with ".mff" and the current behavior of the function is to read the file as a .RAW file if there is not .mff extension. Thus, it fails because .raw files are expected not to be a repository. I renamed my folders so that they have a .mff extension, and it seems to work. So, I apologize, reading the comments in the code should have put me in the right direction.

On a side note, isn't it a bit weird to enforce an extension on a repository? Furthermore, although I did not worked much with EGI NetStation, I think this application does not enforce a .mff extension (the data I get from my collaborators do not have an .mff extension), so it may be counter intuitive for some users if MNE fails to read in these files if the repository do not have a .mff extension. Wouldn't it be better to have different functions (e.g., read_raw_mff and read_raw_egi) for the two format and not enforce the extension for the mff reader? Else, if you prefer using the same function, maybe a better way to detect MFF vs RAW would be to check if the path is a directory (MFF) or a file with a .raw extension.

[fraimondo]

1- In our system (NS 4.5, MFF), whenever we pause the recording, it creates another MFF file. So I was never in the case that I have non-continuous recordings with MFF files. Maybe some new netstation version? We need a small 2-10 seconds recording with one or two pauses in between.

2- MFF files are just OSX packages, so they are indeed directories. Maybe its an error to ask for a .mff extension in the directory. I think the ideal way is to check if it is a directory and check the signal1.bin file.

3- RAW files are another file format from netstation. We use it with NS 4.4, to export their proprietary and closed file format. They do not support non-continous data. If you have a pause in the recording, net station will create two different RAW files when exporting.

[ViridianForge]

We've got access to NS 5 at my lab, so once our equipment shows up (been waiting a couple months now), I could do a test recording with pauses in that version to see if multiple .mffs are created. The data I have that's "non-continuous" was collected in 2003 on a much older version of NetStation. That collection has epochs within the data, with the duration of epochs described by the epochs.xml file.

I do have clearance to share one of the older de-identified files if there's an interest.

Also, for what its worth, I've gotten a little bit of time out of the folks at EGI/Philips Neuro regarding the MFF format, and they've given me some small pointers to an older version of the mff standard, but apparently they've moved to a version of MFF that hasn't been fully documented.

[larsoner]

@ViridianForge any chance to do a short test recording (can be dummy data / no subject) and share it? Without this we're stuck

[ViridianForge]

Sure thing @larsoner - I'll talk to the lab tomorrow morning and get a bit of noise collected for you.

[ViridianForge]

@larsoner - reaching out to via e-mail. I've got a file for you, but gapped that Oregon is on "telework if at all possible" at the moment - so our lab is currently not collecting new datasets.

[ViridianForge]

Here's a compressed version of an epoched MFF. Let me know if it doesn't recover well - I have difficulty believe a 20M file reasonably compresses to 60K.

w1337_20191014_105416.mff.zip

[larsoner]

If the signal is all zeros it's not totally unreasonable at least. It at least uncompressed okay to ~20MB.

[larsoner]

Thanks @ViridianForge , after #7507 is in (which simplifies/refactors the MFF reading quite a bit) I can take a look at this