Preprocess argument for open_mfdataset and threading lock

[shoyer]

Fixes #443
Fixes #444

[shoyer]

Going to merge this shortly, unless anyone has a better name for preprocess

[razcore-rad]

I have a question about this preprocess thing. Would it mean now that... basically xray will load all data in memory? because of the preprocesing step, whereas before... or at least that's what I understood from the documentation, xray would access the data by a need only basis.

[shoyer]

Nope, each dataset is loaded lazily when using open_mfdataset (via dask). As long as you stick to xray operations and don't actually manually load data into memory (e.g., by calling .load()) the data is only accessed and transformed by preprocess on a need only basis.

[razcore-rad]

I need to get my head around this... I know that when you do list comprehension, this isn't lazy so basically it goes through the loop and evaluates for each iteration... so I thought that:

    if preprocess is not None:
        datasets = [preprocess(ds) for ds in datasets]

translates to forcing the application of the preprocess function to the dataset effectively loading it in memory... anyway, this is really cool, I'll definitely try it out 👍