Managing differential-dataflow dependencies

[ryzhyk]

I am looking for a way to manage the dependency on differential-dataflow in my project's Cargo.toml so that the project does not break as the differential-dataflow master changes.

I do not want to use the crates.io version of differential, as it is somewhat outdated (and, given that the project is still under active development, will probably remain so for the foreseeable future).

An alternative would be to have a dependency on a specific differential revision on github. I could then advance this revision periodically, after testing my crate against the new revision. Problem is, differential dataflow's Cargo.toml itself has dependencies on the master branch of timely and abomonation, meaning that older revisions of differential may not work with the latest timely master.

One solution would be to change differential's Cargo.toml to also depend on specific revisions (or tags) of all crates from github. I realize this will introduce extra manual work for Frank to update Cargo.toml whenever dependencies change. Is there a better way?

[frankmcsherry]

One thing that might help is Cargo's patch directive, which allows you to override stated dependencies in other crates. In principle this can let you point at compatible revisions of each of the libraries, and they will sort themselves out.

If you have comments on the breakage, I can try and work around some of the issues. We do try and prototype things in branches first, but in several cases the plan is to break things and to get people to move onto the changed version. But, for example, which of the following worry you most:

1. Random breaks means demoing software risks embarrassing failure.
2. Regular breaks means constantly fixing things, waste of your time.
3. Breaks don't come with changelog patch notes explaining how to fix.

Each of these could be issues, but if one is more a pain point than another we can try and avoid the most painful experiences.

[ryzhyk]

Thanks, I did not know about the patch directive!

Number 1 is by far the biggest issue for me, especially as I try to get other people to use my software in production. For instance, recently the WorkerGuards type lost the Send marker, which my code depended on. As another example, the master branch of differential recently switched from using the crates.io version of timely to the github version. Since I also have to import timely in my crate, I ended up with two different versions, which of course did not compile.

I have no problems making this kind of fixes to keep up with differential, as long as they don't break already released software. Sounds like the patch directive should solve the issue for me.

[ryzhyk]

Sadly, the [patch] trick does not work due to this bug.

[frankmcsherry]

Ah crud.

Well, another option is to have more regular timely/differential releases. My sense is that this is what you "actually want", in the sense of clear checkpoints where things are consistent and work and immutably so.

I'm happy to push these out more regularly. If there are things that you are missing from master give a holler at any point and we can cut a new release. In the worst case, if things aren't ready to go live yet, we can do a "release" using github tags and a branch with the right Cargo.toml commit.

[ryzhyk]

Yep, that's exactly what I'm after :)

Both crates.io based releases and tag-based releases would work, as long as within the release all dependencies point to crates from the same release.

BTW, I'm wondering if there is a reason you have not released differential 0.6 to crates.io. I've been using it for a while and have not run into any problems.

Thanks!

[frankmcsherry]

I've cut new timely and differential releases, with the differential release pointing at the timely release (rather than at master). I'm happy to do this pretty regularly (on-demand, perhaps?) as opposed to crates.io releases which are a bit more tedious. This will probably land there pretty soon, though!

[ryzhyk]

Great, thank you! As far as I am concerned, tag-based releases are perfect; I don't really need them to go t crates.io.

[ryzhyk]

Just a nit: differential still has an untagged github dependency on abomonation and graph-map, and timely has an untagged dependency on abomonation.

[frankmcsherry]

I've just pushed versions to crates.io, which have all the specific dependencies nailed down. I think the lesson is that pushing things to crates and then doing a github release makes the most sense for consistency, but for higher frequency releases in the future I can try and wing it and nail down the local revisions.