Excluded or overriden transitive dependencies break topological sort

[frenchy64]

A project.clj such as this:

:dependencies [[org.clojure/core.memoize "0.5.6"]
               [org.clojure/core.cache "0.6.4"]]

ignores the fact that core.memoize must be processed before core.cache because pomegranate thinks core.cache does not appear in the dependencies of core.memoize.

This means core.memoize ends up with calls to clojure.core.cache, since core.cache can be processed first.

[benedekfazekas]

yup, i am aware of this problem. my idea to fix it is basically work with an unresolved dependency tree if that makes sense. so if a certain library occures multiple times in the tree say once as a transient dependency and once as a direct dependency i would rewrite and include it twice. hope this makes sense...

[frenchy64]

As in, you wouldn't change the current topological sort, but also pass around another tree that includes the "actual" dependencies?

Do you get this unresolved tree from Aether by just deleting the :exclusions form?

And does this mean you would mangle core.cache requires twice? Once for core.cache, then again when you discover core.memoize also depends on it?

[benedekfazekas]

i want to cheat in the sense that i would loop through the direct depedencies and pretend that they are the only dependency. in other words work with the one tree per direct dependency. (i have not proven that this would work yet)

unresolved tree from aether

i guess not because it would still 'solve' clashing versions etc

mangle core.cache twice

this would only make sense if i separated the processing of the above mentioned trees. can get tricky when i replace the references in the original project code. this might result in a system which does not allow you to use transient dependencies directly in your code -- which may be a good thing tbh

[frenchy64]

Your cheat sounds like the easiest solution.

[benedekfazekas]

let me know if you have some ideas around this tho

[frenchy64]

It seems like if we fiddle with the topological map from Aether this fix would be a one liner. My idea was to grab the resolved topological map, then walk over it, adding dependencies that have been overriden and reordering as needed. Then just plug this new topological map into the original functions.

[benedekfazekas]

i guess the tricky thing with 'cheat' would be (1) seperating the trees (quite a bit hassling with directories etc) and (2) updating the references at the right places only

[benedekfazekas]

the possible problems with my approach:

• even bigger jar as a result
• even slower loading time as the same dependency will be loaded several times if it occurs at multiple places

[benedekfazekas]

I came up with a hybrid solution where as a best effort I order the primary dependencies according to how many times they occur in the dependency tree: more times they occur the later they get resolved. I guess this approach fails if primary dependency A occurs more times than B but A actually depends on B. However, this solution solves the simpler cases (the one which pending on mranderson for long time in refactor-nrepl for example).

Let me know what you think. 0.4.4-SNAPSHOT is on clojars btw if you want to play with it

[benedekfazekas]

had to revert my failing try to sort deps. i guess options still are

• implement proper topological sort
• original idea of processing every first level dependency independently (won't help with performance)

[benedekfazekas]

this is now fixed in two different ways in this commit.

firstly there is unresolved-deps-hierarchy option now which causes mranderson to work on a fully expanded, unresolved dependency tree. this tree is walked in a depth first order when processed. this mode has other implications too (the same dependency even the same version may occur several times in the tree) but fixes the above problem by creating a deeply nested directory structure where no transient dependencies are eliminated by a resolving algorithm.

secondly in the default mode where mranderson works on a resolved dependency tree flatten out the resolved tree and sort it by the topological order of the fully expanded tree. Also use a
slightly different walking algorithm which unzips all dependencies first, collects all the relevant contextual info and then processes the source files in a reverse topological order. this fixes the above problem in a sense drafted in this comment.

not closing this issue as the above is not merged into master yet or released. pretty much alpha really.

[benedekfazekas]

0.5.0 released