Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Skipping" messages get overwhelming #1105

Closed
oubiwann opened this issue Mar 3, 2016 · 9 comments
Closed

"Skipping" messages get overwhelming #1105

oubiwann opened this issue Mar 3, 2016 · 9 comments
Labels
minor bug bug that does not prevent major features from working

Comments

@oubiwann
Copy link
Contributor

oubiwann commented Mar 3, 2016

I've been meaning to bring this up for a while ... now that I'm doing a lot of iterative testing, it's hitting me a bunch -- maybe there's a config option I'm missing ...

Anyway, I have a bunch of plugins many of which have overlapping deps in common. When I run rebar3 commands, I get a great deal of console output about "Skipping XXX ... already been fetched."

I imagine this is due to the various (duplicate) deps being present in:

  • _build/default/lib/
  • ``_build/default/plugins/`
  • `_build/test/lib/``
  • `_build/test/plugins/``

Is there anyway to silence these notices by default?

Or, alternatively, am I doing something wrong?

Here's what happens when I kick of the unit tests for the lcli project (https://github.com/oubiwann/lcli):

$ rebar3 eunit

Output is as follows:

===> Newly added dep clj is locked at a lower level. If you really want to unlock it, use 'rebar3 upgrade clj'
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.9.2"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping ltest (from {git,"git://github.com/lfex/ltest.git",
                                      {tag,"0.7.0"}}) as an app of the same name has already been fetched
===> Skipping lutil (from {git,"git://github.com/lfex/lutil.git",
                                      {tag,"0.7.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.9.2"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping ltest (from {git,"git://github.com/lfex/ltest.git",
                                      {tag,"0.7.0"}}) as an app of the same name has already been fetched
===> Skipping lutil (from {git,"git://github.com/lfex/lutil.git",
                                      {tag,"0.7.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.9.2"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping ltest (from {git,"git://github.com/lfex/ltest.git",
                                      {tag,"0.7.0"}}) as an app of the same name has already been fetched
===> Skipping lutil (from {git,"git://github.com/lfex/lutil.git",
                                      {tag,"0.7.0"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                    {tag,"0.3.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Verifying dependencies...
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                    {tag,"v0.10.0"}}) as an app of the same name has already been fetched
===> Compiling lcli
 ~~>    Finding .lfe files ...
===> Performing EUnit tests...
............

Top 10 slowest tests (0.003 seconds, 2.5% of total time):
  lcli-opts-tests:parse-opts_test/0
    0.002 seconds
  lcli-tests:get-spec_test/0: module 'lcli-tests'
    0.001 seconds
  lcli-args-tests:get-raw-args_test/0
    0.000 seconds
  lcli-cmds-tests:commands?_test/0
    0.000 seconds
  lcli-opts-tests:get-opt_test/0
    0.000 seconds
  lcli-opts-tests:boolean?_test/0
    0.000 seconds
  lcli-args-tests:get-args_test/0
    0.000 seconds
  lcli-opts-tests:filter-specs_test/0
    0.000 seconds
  lcli-args-tests:get-script_test/0
    0.000 seconds
  lcli-cmds-tests:has-commands?_test/0
    0.000 seconds

Finished in 0.122 seconds
12 tests, 0 failures
@tsloughter
Copy link
Collaborator

The skipping is only printed during the first run, since the run will produce a lock file and there are no skipping messages if the dep being skipped is locked.

@oubiwann
Copy link
Contributor Author

oubiwann commented Mar 3, 2016

Hrm ... I don't think I understand. I see those skipping messages every time I run rebar3 eunit.

Here's what I just did:

  • removed rebar.lock and _build
  • ran rebar3 compile (which re-downloaded everything and created a new rebar.lock file)
  • ran rebar3 eunit > output
  • ran $ wc -l output and got 135
  • ran rebar3 eunit > output
  • ran $ wc -l output and got 110

So there are fewer "skipping" messages ... but I'm still getting a lot of them. After the rebar.lock file is created, the output looks just like what I pasted in the description of this ticket.

This doesn't happen for most projects?

@ferd
Copy link
Collaborator

ferd commented Mar 3, 2016

Do your plugins also have a rebar.lock file in their own repos? I believe this may eliminate quite a few too

@oubiwann
Copy link
Contributor Author

oubiwann commented Mar 3, 2016

Ah, this, I think is the problem ... the bit I was missing. None of my projects have a rebar.lock file in them -- that's an entry in their .gitignore file. I will make that change today, run rebar3, and I expect I will be immediately returning to this ticket to close it :-)

Thanks for the help and insights, guys!

@tsloughter
Copy link
Collaborator

Yea, don't ignore those :). But I'd also be open to a patch that doesn't display the skip if the one being skipped is the same as the one already picked. No reason to tell the user it is skipping if they both ask for the same version.

@oubiwann
Copy link
Contributor Author

oubiwann commented Mar 6, 2016

Okay, I think most of the LFE projects I manage that had updated to use rebar3 have had a rebar.lock file added for them. This actually ended up being a little tricky ... one or two of the projects had gotten an out-dated repo in their lock file, and tracking that down took a couple of hours (it was in the dep of a dep and it wasn't obvious why an older version was being cloned ... nor where that was happening. I had actually anticipated issues like that with the use of a lock file, so it wasn't a mystery -- just a matter of hunting and not giving up :-)).

That done, I still do get a fair number of "skipping" messages for projects with lots of LFE project deps. For example, when running the unit tests for the LFE lhc project:

$ rebar3 eunit

===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                   {ref,
                                       "3cc502b150ff2a631719f063c0b8a7a3643f420e"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                   {ref,
                                       "3cc502b150ff2a631719f063c0b8a7a3643f420e"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping clj (from {git,"git://github.com/lfex/clj.git",
                                   {ref,
                                       "3cc502b150ff2a631719f063c0b8a7a3643f420e"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Skipping lfe (from {git,"git://github.com/rvirding/lfe.git",
                                   {ref,
                                       "d7bc80da04aedd0a607f17d8d149eba363872b4b"}}) as an app of the same name has already been fetched
===> Verifying dependencies...
===> Compiling lhc
 ~~>    Finding .lfe files ...
 ~~>    Compiling ./test/unit-lhc-util-tests.lfe ...
===> Performing EUnit tests...
..........................................

Top 10 slowest tests (0.004 seconds, 2.0% of total time):
  unit-lhc-url-tests:->string_test/0
    0.004 seconds
  unit-lhc-url-tests:parse-user-host_test/0
    0.000 seconds
  unit-lhc-url-tests:parse-error_test/0
    0.000 seconds
  unit-lhc-url-tests:get-path_test/0
    0.000 seconds
  unit-lhc-url-tests:parse-no-user-password-short-query_test/0
    0.000 seconds
  unit-lhc-url-tests:parse-all-short-path-query-frag_test/0
    0.000 seconds
  unit-lhc-tests:noop_test/0: module 'unit-lhc-tests'
    0.000 seconds
  unit-lhc-url-tests:parse-netloc-user-pass-host-port_test/0
    0.000 seconds
  unit-lhc-url-tests:parse-netloc-host-port_test/0
    0.000 seconds
  unit-lhc-url-tests:parse-netloc-host_test/0
    0.000 seconds

Finished in 0.202 seconds
42 tests, 0 failures

As a sanity check, I captured the output of that to a file and then did a sort -u on it, and fortunately only got the clj and lfe commit ids (iow, all the same version). Also, I should note that this is actually fewer warnings than I was getting before, so the rebar.lock file additions definitely helped (as expected).

I think this brings us to the last note by @tsloughter indicating that there is no check for skips that are for the same version that has already been picked. As you can see from the output, this is a significant distractor for command line use :-/

I'll put this on my to-do list and keep this ticket open for a patch that addresses this.

Thanks again for your help.

@ferd
Copy link
Collaborator

ferd commented Mar 6, 2016

Yeah. There's also the fact that since we don't generate a lock file for plugins (and Tristan and I briefly tried -- but the model is hard to make work right now given plugins have to be fetched differently from deps since plugins may contain custom dependency providers which ultimately may affect deps!), the top-level resolving is never encoded in lock files and gets to be repeated time and time again.

Hiding the same versions (assuming they have the same sources, since git tags and branches are mutable) could reduce some of the noise, but you just need two top-level plugins with different version dependencies to always get warnings.

I think the full solution for us will be to either figure out a scheme for locking plugins, or a scheme to suppress their output on subsequent fetches which we know could be conflicting and therefore useless since we warned already.

@oubiwann
Copy link
Contributor Author

oubiwann commented Mar 6, 2016

@ferd That sounds great. In that case, whenever the decision is made to follow of one the two options you listed, I guess we can close this ticket.

@ferd ferd added the minor bug bug that does not prevent major features from working label Jun 7, 2016
liamjjmcnamara pushed a commit to liamjjmcnamara/rebar3 that referenced this issue Nov 20, 2017
Allow silencing skip warnings when fetching deps
64056bf 
When fetching deps, if this is a clean repo there will be extensive
messages warning that dependencies which have already been fetched are
being skipped. For large projects being built and tested in a clean
environment this significantly increases the noise level of the build.

This modification adds an additional rebar option
(deps_warning_on_conflict) that will allow disabling these warning
messages. If deps_error_on_conflict is set, an error will still be
thrown. This will not change default behaviour of rebar.

There is a similar outstanding issue:
erlang#1105
However this seems to be a push for not outputting warnings when the dep
version is the same, rather than disabling warnings altogether.
@liamjjmcnamara liamjjmcnamara mentioned this issue Nov 20, 2017
Merged
@ferd
Copy link
Collaborator

ferd commented Nov 20, 2017

See #1664

@ferd ferd closed this as completed Nov 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
minor bug bug that does not prevent major features from working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tsloughter @ferd @oubiwann