Reports do not contain unfiltered data

[hoechenberger]

Today I showed a report to a colleague of mine and we figured it didn't contain the raw data before filtering. I believe we used to include this... So this seems like a bug to me that it's missing

[larsoner]

Agreed we should add it. Really we should probably have the ten-1s-segment diagnostic plots of raw at each stage (orig, MF, freq-filt).

Looking at #652, I'm a bit bothered by where the find_bads_meg step stuff had to go. I think things would get simpler if we had a scripts/preprocessing/_01_import_data.py whose job it was to load the original data, do any automatic bad channel finding that has been requested, write those results to disk, and do any necessary reporting. That way this doesn't have to happen conditionally in either _01_maxfilter or _02_frequency_filter

[hoechenberger]

Looking at #652, I'm a bit bothered by where the find_bads_meg step stuff had to go. I think things would get simpler if we had a scripts/preprocessing/_01_import_data.py whose job it was to load the original data, do any automatic bad channel finding that has been requested, write those results to disk, and do any necessary reporting. That way this doesn't have to happen conditionally in either _01_maxfilter or _02_frequency_filter

I agree. I believe we once had discussed this, but decided we wanted to avoid writing yet another set of files to disk. But now with the caching and all, I think it could make sense to reconsider this decision!

[larsoner]

We don't even have to copy the files -- I'm just saying that we should write the bad channel lists to disk. We can keep the somewhat complicated logic for which raw files to load in the next preproc steps, it would just be nice to have a dedicated one for this bads-finding step

[hoechenberger]

I think this touches upon an interesting question:
Should we use MNE-BIDS's write_raw_bids() to write those raw files? Then, having a list (TSV) of bad channels comes naturally. And on non-Windows platforms, we even support writing symlinks to the input files, i.e., only the new sidecar files would need to be written in addition to a symlink.

WDYT?

[larsoner]

That could potentially work, too. I'd probably start by implementing it the way we have it now just as a copy-paste to keep things simple, though, and then in another PR try this enhancement in case it breaks things