Add script to detect missing dependencies

[cds-amal]

I noticed the ChainSafe team uses a dependency check tool in their project workflow and think it might be helpful. This PR adds the dependency-check module to the repo to help identify packages that have missing dependencies.

I also added the identified missing dependencies here, but perhaps those should be different PRs?

yarn depcheck

• checks each workspace in our repo except, contract-tests and dashboard-message-bus-e2e-test
• returns a errcode 1 if it discovers any failures which can be utilized as part of husky's checks. This check is unimplemented in this PR.

Full sample output

$ yarn depcheck
yarn run v1.22.18
$ lerna exec --stream --no-bail dependency-check --ignore @truffle/contract-tests --ignore @truffle/dashboard-message-bus-e2e-test  -- --missing --verbose .
lerna notice cli v4.0.0
lerna info versioning independent
lerna notice filter excluding ["@truffle/contract-tests","@truffle/dashboard-message-bus-e2e-test"]
lerna info filter [
lerna info filter   '!@truffle/contract-tests',
lerna info filter   '!@truffle/dashboard-message-bus-e2e-test'
lerna info filter ]
lerna info Executing command in 47 packages: "dependency-check --missing --verbose ."
@truffle/error: Success! All dependencies used in the code are listed in package.json
@truffle/expect: Success! All dependencies used in the code are listed in package.json
@truffle/blockchain-utils: Success! All dependencies used in the code are listed in package.json
@truffle/promise-tracker: Success! All dependencies used in the code are listed in package.json
@truffle/contract-sources: Success! All dependencies used in the code are listed in package.json
@truffle/dashboard-message-bus-common: Success! All dependencies used in the code are listed in package.json
@truffle/code-utils: Success! All dependencies used in the code are listed in package.json
@truffle/hdwallet-provider: Success! All dependencies used in the code are listed in package.json
@truffle/interface-adapter: Success! All dependencies used in the code are listed in package.json
@truffle/contract-schema: Success! All dependencies used in the code are listed in package.json
@truffle/spinners: Success! All dependencies used in the code are listed in package.json
@truffle/source-fetcher: Success! All dependencies used in the code are listed in package.json
@truffle/dashboard-message-bus: Success! All dependencies used in the code are listed in package.json
@truffle/plugins: Success! All dependencies used in the code are listed in package.json
@truffle/dashboard-message-bus-client: Success! All dependencies used in the code are listed in package.json
@truffle/provider: Fail! Dependencies not listed in package.json: debug
@truffle/compile-common: Success! All dependencies used in the code are listed in package.json
@truffle/abi-utils: Success! All dependencies used in the code are listed in package.json
@truffle/events: Fail! Dependencies not listed in package.json: @truffle/dashboard-message-bus-client, debug
@truffle/dashboard: Fail! Dependencies not listed in package.json: get-port, @truffle/dashboard-message-bus-common
@truffle/config: Success! All dependencies used in the code are listed in package.json
@truffle/provisioner: Success! All dependencies used in the code are listed in package.json
@truffle/external-compile: Fail! Dependencies not listed in package.json: @truffle/compile-common
@truffle/profiler: Fail! Dependencies not listed in package.json: @truffle/expect
@truffle/require: Success! All dependencies used in the code are listed in package.json
@truffle/box: Fail! Dependencies not listed in package.json: debug
@truffle/codec: Success! All dependencies used in the code are listed in package.json
@truffle/source-map-utils: Success! All dependencies used in the code are listed in package.json
@truffle/debug-utils: Success! All dependencies used in the code are listed in package.json
@truffle/contract: Success! All dependencies used in the code are listed in package.json
@truffle/artifactor: Success! All dependencies used in the code are listed in package.json
@truffle/resolver: Fail! Dependencies not listed in package.json: fs-extra, web3-utils, @truffle/compile-solidity
@truffle/environment: Fail! Dependencies not listed in package.json: @truffle/provider
@truffle/compile-vyper: Fail! Dependencies not listed in package.json: @truffle/contract-sources
@truffle/compile-solidity: Success! All dependencies used in the code are listed in package.json
@truffle/db: Success! All dependencies used in the code are listed in package.json
@truffle/fetch-and-compile: Success! All dependencies used in the code are listed in package.json
@truffle/db-loader: Success! All dependencies used in the code are listed in package.json
@truffle/workflow-compile: Success! All dependencies used in the code are listed in package.json
@truffle/deployer: Fail! Dependencies not listed in package.json: debug
@truffle/migrate: Fail! Dependencies not listed in package.json: debug
@truffle/encoder: Success! All dependencies used in the code are listed in package.json
@truffle/decoder: Success! All dependencies used in the code are listed in package.json
@truffle/debugger: Success! All dependencies used in the code are listed in package.json
@truffle/db-kit: Fail! Dependencies not listed in package.json: graphql-tag, web3
@truffle/core: Fail! Dependencies not listed in package.json: @truffle/promise-tracker, lodash, @truffle/compile-common
truffle: Success! All dependencies used in the code are listed in package.json
lerna ERR! Received non-zero exit code 1 during execution
lerna success exec Executed command in 47 packages: "dependency-check --missing --verbose ."
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Filtering failures with grep

$ yarn depcheck | grep Fail\!
@truffle/provider: Fail! Dependencies not listed in package.json: debug
@truffle/events: Fail! Dependencies not listed in package.json: @truffle/dashboard-message-bus-client, debug
@truffle/dashboard: Fail! Dependencies not listed in package.json: get-port, @truffle/dashboard-message-bus-common
@truffle/external-compile: Fail! Dependencies not listed in package.json: @truffle/compile-common
@truffle/box: Fail! Dependencies not listed in package.json: debug
@truffle/profiler: Fail! Dependencies not listed in package.json: @truffle/expect
@truffle/resolver: Fail! Dependencies not listed in package.json: fs-extra, web3-utils, @truffle/compile-solidity
@truffle/environment: Fail! Dependencies not listed in package.json: @truffle/provider
@truffle/compile-vyper: Fail! Dependencies not listed in package.json: @truffle/contract-sources
@truffle/deployer: Fail! Dependencies not listed in package.json: debug
@truffle/migrate: Fail! Dependencies not listed in package.json: debug
@truffle/db-kit: Fail! Dependencies not listed in package.json: graphql-tag, web3
@truffle/core: Fail! Dependencies not listed in package.json: @truffle/promise-tracker, lodash, @truffle/compile-common
lerna ERR! Received non-zero exit code 1 during execution
error Command failed with exit code 1.

Passing check

@truffle/error: Success! All dependencies used in the code are listed in package.json

@truffle/error, according to dependency-check has no missing dependencies.

Failing check

@truffle/dashboard: Fail! Dependencies not listed in package.json: get-port, @truffle/dashboard-message-bus-common

@truffle/dashboard does not have get-port and @truffle/dashboard-message-bus-common listed as dependencies even though they are used.

[haltman-at]

This is excellent! I'm assuming you picked the particular versions sensibly. Obviously #5177 will have to be updated if this is merged (and vice versa)...

What'd be really cool is if this dependency check could be added to Husky or GHA. Guessing it'd probably have to be the latter? Like we could have a depcheck job in addition to the yarncheck job! Husky would be even better if it's workable but I'm assuming it probably isn't...? Obviously this all should be a separate PR from this though. :)

[haltman-at]

Why are these --ignores here?

[haltman-at]

Oh, it's because they have no entry point, I see. Um, so, wait, is this thing only checking the main code and not the test code? (Still an improvement, but worth noting if that caveat is present.)

[cds-amal]

I ignored them because they are both private: true and have no entry points. The tool has a --entry flag if we want to include private packages at some later time.

[haltman-at]

Uh-oh: I just noticed that this PR introduces a cyclic dependency between resolver and compile-solidity. O_o Well, OK -- the cyclic dependency was in a sense already there, just not documented. 😬

...not sure if that should hold up merge. If it can be fixed quickly it should. I'll go take a look at what's causing it, as I'm pretty familiar with those two packages...

[haltman-at]

OK, so the source of the cyclicity is this:

1. @truffle/compile-solidity imports @truffle/resolver as a devdep, because it uses it in tests, and
2. @truffle/resolver, in sources/truffle/Deployed.ts, does the following:

import * as RangeUtils from "@truffle/compile-solidity/dist/compilerSupplier/rangeUtils";

My guess is that probably shouldn't hold up merge? But it will need to be fixed obviously. Maybe the range utils should be split into their own package, as a way of resolving the cyclicity? (Thoughts, @eggplantzzz, @gnidan?)

[eggplantzzz]

Oh, my initial thoughts are that if it is a dev dependency then it shouldn't matter much. Maybe I'm missing something though...

[eggplantzzz]

A lot of these versions use the ^. Don't we want to get rid of those for external packages?

[haltman-at]

Oh, my initial thoughts are that if it is a dev dependency then it shouldn't matter much. Maybe I'm missing something though...

Yeah, lerna doesn't care, it'll cause problems on any cyclicity.

[haltman-at]

A lot of these versions use the ^. Don't we want to get rid of those for external packages?

No? I don't recall us ever deciding on such a policy. Typically thus far we've generally used carets and only pinned when necessary.

[cds-amal]

I'm assuming you picked the particular versions sensibly.

Now that I think about it, we should re-use versions already present, which would reduce yarn.lock complexity.

[cds-amal]

Obviously https://github.com/orgs/trufflesuite/discussions/5117 will have to be updated if this is merged (and vice versa)...

Huh? I don't see the connection.

Edit: oh, probably a typo: #5177 (Upgrade to Web3 1.7.4)

[gnidan]

thanks for correcting my title change :)

[cds-amal]

thanks for correcting my title change :)

Ha, I missed that. I saw the title and thought I got ahead of myself!

[haltman-at]

Well, @gnidan thinks the circular dependency shouldn't hold up merge as long as we fix it right after, so I'm hitting approve on this!

[haltman-at]

Btw, this also only checks the compiled code, right? So it also won't catch missing dependencies that only appear in import types. Basically, it's no good for dev dependencies. Still, better than nothing!

Thought: Maybe we should take out --missing and put in --no-dev? Does it still pass if we do that?

[haltman-at]

Hm, I just tried it with --missing removed and --no-dev added, and got some failures... which likely indicate actual things that should be changed! (Note, though, that in a number of cases what should be changed is that the package should be moved to devDep, not removed from the package.json entirely.)

Btw, do we really want --verbose in there? It makes the failures harder to find. I'm of the opinion we'd be better with that removed.

[cds-amal]

Hm, I just tried it with --missing removed and --no-dev added, and got some failures... which likely indicate actual things that should be changed! (Note, though, that in a number of cases what should be changed is that the package should be moved to devDep, not removed from the package.json entirely.)

I think this is a good idea to do in a follow-up PR. We could have multiple dependency scripts (with different configuration). The configuration you suggest will take some time to reason through, for example:

@truffle/hdwallet-provider: Fail! Modules in package.json not used in code: ethereum-protocol

Identified ethereum-protocol as an unused dep of hdwallet-provider because it is used only referenced via import type {...} from ethereum-protocol. I also noticed we should probably determine if some @types/... packages devDeps should move to regular dependencies (Also, for follow-up work).

Btw, do we really want --verbose in there? It makes the failures harder to find. I'm of the opinion we'd be better with that removed.

Yes, I agree and will add that change to this PR.

[haltman-at]

Identified ethereum-protocol as an unused dep of hdwallet-provider because it is used only referenced via import type {...} from ethereum-protocol. I also noticed we should probably determine if some @types/... packages devDeps should move to regular dependencies (Also, for follow-up work).

Oh, right, I forgot that some type imports are still regular dependencies and not dev dependencies. (As in, deliberately, not accidentally.) OK, ignore that idea, I think the current way is probably best after all! (Other than removing --verbose.)