Formalize contract between XArray and the dask.distributed scheduler

[mrocklin]

XArray was designed long before the dask.distributed task scheduler. As a result newer ways of doing things, like asynchronous computing, persist, etc. either don't function well, or were hacked on in a less-than-optimal-way. We should improve this relationship so that XArray can take advantage of newer dask.distributed features today and also adhere to contracts so that it benefits from changes in the future.

There is conversation towards the end of dask/dask#1068 about what such a contract might look like. I think that @jcrist is planning to work on this on the Dask side some time in the next week or two.

[jhamman]

@mrocklin - I'd be game to help push this forward on the xarray side. I've been pretty busy getting xarray 0.10 released but once that's out the door (next week probably), I'll be ready to work on this.

[mrocklin]

I think we need to wait for dask/dask#1068 first. I think that @jcrist wants to take this on. I'm not sure what his upcoming schedule is like. My estimate is that we might have something by the end of the week that could use some concurrent experimentation on the XArray side. I can also do this though if you're busy.

[jcrist]

Apologies for the delay here. See dask/dask#2748.

[mrocklin]

@jhamman did you want to do the work on the XArray side? If so, what is your schedule like? I'm also looking around within Anaconda Inc to see if others might have time/interest to work on this.

[mrocklin]

Ping @jhamman

[jhamman]

@mrocklin - thanks for the additional ping. I thought I had responded to this.

Yes, I do want to work on this. Now would be an good time to get a branch going on this. I'll read through the docs again.

Logistics... Do you or someone at Anaconda anticipate working on xarray directly? Were you thinking I would do the majority of the implementation?

[mrocklin]

Logistics... Do you or someone at Anaconda anticipate working on xarray directly? Were you thinking I would do the majority of the implementation?

I figure whoever has free time first will get started on it first. I suspect that it's a race between you and @jcrist although I'm also hunting around to find other people to get involved. I'm somewhat busy for the next week but will probably start on it then if no one else has done anything. If you prefer not to do this that's fine with me as well, I just want to make sure that folks don't overlap unnecessarily.

[mrocklin]

This is started in pydata/xarray#1674