Reduce dependencies #433

icebob · 2018-12-09T18:41:47Z

Due to the recent event-stream vulnerability issue, I'm thinking to reduce the used NPM dependencies in the Moleculer core.
Currently it is the current dependency tree (29 modules):

`-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected] deduped
  | +-- [email protected]
  | `-- [email protected]
  +-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | |   `-- [email protected]
  | +-- [email protected]
  | `-- [email protected]
  |   `-- [email protected]
  +-- [email protected]
  +-- [email protected]
  +-- [email protected]
  +-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | +-- [email protected] deduped
  | | `-- [email protected]
  | +-- [email protected]
  | +-- [email protected]
  | | `-- [email protected]
  | |   +-- [email protected]
  | |   `-- [email protected]
  | +-- [email protected]
  | | `-- [email protected] deduped
  | `-- [email protected]
  +-- [email protected]
  `-- [email protected]

There are 3 libs which have too much dependencies:

args (4 other deps)
It is used in Moleculer Runner to parse arguments
chalk (6 other deps)
It is used to coloring some log messages
glob (12(!) other deps)
It is used in Moleculer Runner & ServiceBroker to load services from folders.
vorpal
It's a dev dependency but it is not maintained & contains some vulnerabilities.

Possible alternatives:

Important to keep the current functionality, so the changes don't cause breaking changes!

`args`

mri (no deps)
cac (no deps)
tinyargs (no deps)

`chalk`

kleur (no deps)

`glob`

tiny-glob (2 deps)

`vorpal`

?

The text was updated successfully, but these errors were encountered:

icebob · 2018-12-09T18:46:28Z

@lukeed do you have any awesome (fast & no deps) lib to replace the huge glob lib? 😄

lukeed · 2018-12-09T19:36:11Z

Hey, haha, I helped work on tiny-glob. It's the one I use

tinchoz49 · 2019-07-17T03:52:55Z

https://github.com/vadimdemedes/ink could be a replacement for vorpal

lukeed · 2019-07-17T17:03:37Z

Ink is good. I forgot to mention before that sade is also an option for your CLI builder. Its only dependency is mri, which is already in this list

joshmanders · 2019-07-17T18:20:17Z

Hello, we use Moleculer at my company and vorpal's vulnerabilities are a concern to us and we'd like to get this resolved. What can I do to help with this? I can fork and start a PR to replace vorpal with something else.

Thanks.

icebob · 2019-07-18T07:37:26Z

vorpal is more than the ink. It has a REPL mode with command & argument handling and autocomplete feature. Ink (as soon as I can see) is a react based GUI for console apps.
The better option would be to fix vulnerable packages in vorpal. I've created a fork here

joshmanders · 2019-07-18T15:39:23Z

Yeah I was looking over alternatives and none really seem to do what moleculer uses vorpal for.

@icebob do you plan to utilize the fork in moleculer-repl by publishing a package or directly to the github repo in the package.json? I can start working on getting the vulnerabilities patched and submit a PR to the fork.

tinchoz49 · 2019-07-18T16:52:51Z

This is another one more close to the vorpal concept: https://github.com/drew-y/cliffy

But I agree:

The better option would be to fix vulnerable packages in vorpal. I've created a fork here

icebob · 2019-07-18T19:11:28Z

@joshmanders good news. It would be good if you could fix it. I plan to publish the fixed vorpal under the org like: @moleculer/vorpal

icebob · 2019-07-18T19:16:38Z

@tinchoz49 great, yes, cliffy can be an alternative. Btw, I've starred it for a while :P

joshmanders · 2019-07-18T19:21:07Z

@joshmanders good news. It would be good if you could fix it. I plan to publish the fixed vorpal under the org like: @moleculer/vorpal

Excellent. I have a PR waiting right now, just have noticed some tests are timing out so I'm trying to figure that out.

GrosSacASac · 2019-09-05T08:59:41Z

I heard that bluebird is no longer needed

abdavid · 2020-05-20T17:26:33Z

@icebob Could we add lodash to the list of libraries that Moleculer could live without? I dissected the framework a while back while playing around with making it a first-class typescript citizen and noticed a lot of lodash usage that could be replaced by simple native js. The reason I particularly am mentioning this lib is because of the bad pratices like prototype pollution it often entails.

https://github.com/moleculerjs/moleculer/search?q=isObject&unscoped_q=isObject

abdavid · 2020-05-20T18:18:06Z

Latest tree btw

[email protected] /Users/davborre/Source/moleculer
├─┬ [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └─┬ [email protected]
│ │ │   └── [email protected]
│ │ ├── [email protected]
│ │ └─┬ [email protected]
│ │   └── [email protected]
│ ├── [email protected]
│ └── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├─┬ [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └─┬ [email protected]
│ │   ├── [email protected]
│ │   └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ └── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├─┬ [email protected]
│ └── [email protected]
└── [email protected]

icebob · 2020-05-21T12:33:19Z

Yes, it would be good to drop lodash lib from the core repo. The most important method is the _.defaultsDeep. It is used in many places, so it's the most important to find an exact same implementation or copy out from the lodash.

joshmanders · 2020-05-21T16:31:58Z

Yes, it would be good to drop lodash lib from the core repo. The most important method is the _.defaultsDeep. It is used in many places, so it's the most important to find an exact same implementation or copy out from the lodash.

Could just depend on lodash.defaultsDeep which is just the dependencies needed for that function.

icebob · 2020-05-21T18:10:01Z

Hmm, good idea!

icebob · 2020-05-21T18:11:27Z

By the way, I've just found cac which is same as args but no dependencies.

Only need to change the glob to avoid another big branch

abdavid · 2020-05-21T18:13:14Z

_.get used in the cacher implementation is pretty important aslo I believe. It would be great to firstly get an overview of all lodash deps to determine which could easily be switched out with vanilla js and which we need to consider keeping or replace. I’d be happy to contribute on some concrete issues like «replace usage of isObject» etc to we can gradually migrate lodash out. - David

…

21. mai 2020 kl. 18:32 skrev Josh Manders ***@***.***>: Yes, it would be good to drop lodash lib from the core repo. The most important method is the _.defaultsDeep. It is used in many places, so it's the most important to find an exact same implementation or copy out from the lodash. Could just depend on lodash.defaultsDeep which is just the dependencies needed for that function. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

lukeed · 2020-05-21T18:23:03Z

cac has dependencies (including mri) – it's just all inlined into one big pre-built file, which prevents any cross-dep code reuse

icebob · 2020-05-21T18:27:45Z

Ohh, thanks @lukeed, I didn't see it.

lukeed · 2020-05-21T18:38:23Z

It's actually only mri - I thought there were more. This means cac basically the same as sade...good upgrade either way

icebob · 2020-05-22T16:36:57Z

@abdavid Here is a list about the used Lodash methods:

abdavid · 2020-05-22T17:52:30Z

Nice! I see a couple of low hanging ones. I think we can break them down pr function. @icebob should we maybe create an epic and materialize tasks for each function? That way we will have the oppertunity to discuss replacements contextually. Like Josh commented we could just import the complex ones. - David

…

22. mai 2020 kl. 18:37 skrev Icebob ***@***.***>: @abdavid Here is a list about the used Lodash methods: — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

abdavid · 2020-05-22T20:09:31Z

@icebob Do we have any docs / guides on performance testing? Would be necessary to have before and after measurements to ensure we keep things speedy

icebob · 2020-05-22T21:11:51Z

Yes, you can with npm run bench. It executes common performance benchmark tests, so you can measure before & after with it.

Wallacy · 2020-05-26T15:07:43Z

@icebob Another option to replace Vorpal: https://github.com/f/omelette

Has zero dependencies;

icebob · 2020-05-27T12:09:07Z

Thanks @Wallacy but I think it's not a replacement of vorpal. It's just a part of it.

AndreMaz · 2020-06-08T21:37:13Z

One possible replacement for vorpal in moleculer-repl is shargs.

Some features:

Highly customizable command-line arguments parser and usage documentation generator.
35+ opt-in features, e.g. (multiple) subcommands, spelling mistake detection, default values, and (best guess) casting.
Synchronous and Promise-based asynchronous mode with async/await support.
Automatic usage documentation generation with fine-grained control over layouts and styles.
Easily extensible with your own custom parser stages and custom usage layouts.
Extensively documented and very well tested (750+ unit and integration tests).
Modular library layout with zero runtime dependencies.

and @Yord is currently working on REPL mode and autocompletion

icebob · 2020-06-09T06:04:10Z

It sounds great. it has no dependency.

Yord · 2020-06-09T06:22:00Z

Hey all, shargs is meant to replace the command-line parser of another project of mine because it had way too many dependencies (27!). Seems our causes are aligned.

The REPL mode is still very fresh, but I am interested in spending more time.

icebob · 2020-06-09T06:25:40Z

@Yord Do you know vorpal? Are there any other missing parts in shargs? We are using it in our moleculer-repl

Yord · 2020-06-09T07:10:40Z

@icebob I have never worked with vorpal.

The command-line parser parts of shargs are almost at 1.0.0, error message ergonomy is left (and bash autocomplete).

As for REPLs, shargs has basic REPL functionality based on the Node REPL and autocomplete for commands. These two parts are not tested, yet.

I could imagine shargs REPL misses text colors and formatting (aka tables) unless the action output does that.

Is there anything apparent to vorpal users that I miss?

Edited to add: Shargs REPL misses pipes and prompts.

AndreMaz · 2020-06-09T09:17:24Z

I could imagine shargs REPL misses text colors and formatting (aka tables) unless the action output does that.

You are right, the coloring and formatting are done directly in action (e.g., service )

Edited to add: Shargs REPL misses pipes and prompts.

I think that we don't use either of those features. We only use "a small portion" of vorpal

icebob · 2020-06-09T12:53:16Z

Would be good to create a branch in moleculer-repl and trying to add Shargs instead of vorpal. @AndreMaz do you have some capacity to work on it? Maybe @Yord can help if something is missing :)

Yord · 2020-06-09T13:00:22Z

@icebob, @AndreMaz I will gladly ship in if I can be of help.

AndreMaz · 2020-06-09T15:52:26Z

@icebob I will look at it during weekend. In theory it shouldn't be very complex. I just need to find the equivalent for vorpal's syntax. E.g.:

vorpal
  .command('foo <requiredArg> [optionalArg]')
  .autocomplete({
    data() {
        return // Return autocomplete array
  })
  .option('-v, --verbose', 'Print foobar instead.')
  .description('Outputs "bar".')
  .alias('foosball')
  .action(function(args, callback) {
    if (args.options.verbose) {
      this.log('foobar');
    } else {
      this.log('bar');
    }
    callback();
  });

@Yord do you have any tutorial for newbies? Your docs are very good but I didn't found any introductory tutorial.

Yord · 2020-06-09T16:01:06Z

@AndreMaz Fortunately I wrote one just yesterday:

https://github.com/Yord/shargs-tutorial-git/blob/master/README.md

AndreMaz · 2020-06-09T16:03:24Z

Sweet 😄 Thanks

Yord · 2020-06-13T20:02:18Z

@AndreMaz Just wanted to let you know I have incorporated recent updates into shargs-example-repl.

To elaborate on this: The API has only slight changes, the updates concern mainly autocomplete internals.

AndreMaz · 2020-06-14T09:04:57Z

@Yord thanks for the info. Can you add the changelog to your repos? It helps to track the changes

Yord · 2020-06-14T09:29:42Z

@AndreMaz Yes, I will do that. Sorry for the sloppy work. I usually start a changelog with the first stable version, but here, it definitely makes sense to start right now.

Changelogs:

gautaz · 2023-01-15T23:16:59Z

Hello, sorry to barge in but I think that what I raised here might be related to this issue.

My main concern is related to the fact that some of us (I have no idea how many) are using the Moleculer broker without the runner.
I was wondering if moving the broker to something like moleculer-core (or the runner to moleculer-runner) would be an acceptable solution.

The runner is great but it also comes with its cost in term of dependencies and I guess that part of the dependencies raised in this issue is in fact due to the runner (another batch is certainly due to the REPL).
⚠️ This last one is an assumption, I did no check the source code so I may be totally wrong (which would be far from being the first time 😆).

icebob · 2023-01-16T09:21:22Z

I plan to split this repo to multiple packages in v1.0 and release them under @moleculer/....

icebob added Type: Enhancement help wanted Module: Core labels Dec 9, 2018

darky mentioned this issue Apr 25, 2019

Drop bluebird #528

Closed

joshmanders mentioned this issue Jul 18, 2019

Fix vulnerabilities moleculerjs/vorpal#1

Closed

joshmanders mentioned this issue Jul 18, 2019

fix vulnerabilities in deps and update xo moleculerjs/vorpal-repl#1

Closed

This was referenced May 22, 2020

[lodash] Remove usage of _.random #749

Closed

[lodash] Remove usage of _.uniq #750

Open

[lodash] Remove usage of _.flatten #751

Closed

[lodash] Remove usage of _.omit #752

Open

[lodash] Remove usage of _.get #753

Open

This was referenced May 23, 2020

Remove usage of lodash.isObject. #755

Merged

Remove usage of lodash.isArray. #756

Merged

Remove usage of lodash.isFunction. #757

Merged

Wallacy mentioned this issue Feb 2, 2021

Use Deno instead of Nodejs #863

Closed

gautaz mentioned this issue Jan 13, 2023

Use of __proto__ seems discouraged #1169

Closed

4 tasks

Reduce dependencies #433

Reduce dependencies #433

Comments

icebob commented Dec 9, 2018 • edited Loading

Possible alternatives:

args

chalk

glob

vorpal

icebob commented Dec 9, 2018 • edited Loading

lukeed commented Dec 9, 2018

tinchoz49 commented Jul 17, 2019

lukeed commented Jul 17, 2019

joshmanders commented Jul 17, 2019

icebob commented Jul 18, 2019

joshmanders commented Jul 18, 2019

tinchoz49 commented Jul 18, 2019 • edited Loading

icebob commented Jul 18, 2019

icebob commented Jul 18, 2019

joshmanders commented Jul 18, 2019

GrosSacASac commented Sep 5, 2019

abdavid commented May 20, 2020

abdavid commented May 20, 2020

icebob commented May 21, 2020

joshmanders commented May 21, 2020

icebob commented May 21, 2020

icebob commented May 21, 2020

abdavid commented May 21, 2020 via email

lukeed commented May 21, 2020

icebob commented May 21, 2020

lukeed commented May 21, 2020

icebob commented May 22, 2020

abdavid commented May 22, 2020 via email

abdavid commented May 22, 2020 • edited Loading

icebob commented May 22, 2020

Wallacy commented May 26, 2020

icebob commented May 27, 2020

AndreMaz commented Jun 8, 2020

icebob commented Jun 9, 2020

Yord commented Jun 9, 2020

icebob commented Jun 9, 2020

Yord commented Jun 9, 2020 • edited Loading

AndreMaz commented Jun 9, 2020

icebob commented Jun 9, 2020

Yord commented Jun 9, 2020

AndreMaz commented Jun 9, 2020 • edited Loading

Yord commented Jun 9, 2020

AndreMaz commented Jun 9, 2020 • edited Loading

Yord commented Jun 13, 2020 • edited Loading

AndreMaz commented Jun 14, 2020

Yord commented Jun 14, 2020 • edited Loading

gautaz commented Jan 15, 2023 • edited Loading

icebob commented Jan 16, 2023

icebob commented Dec 9, 2018 •

edited

Loading

`args`

`chalk`

`glob`

`vorpal`

icebob commented Dec 9, 2018 •

edited

Loading

tinchoz49 commented Jul 18, 2019 •

edited

Loading

abdavid commented May 22, 2020 •

edited

Loading

Yord commented Jun 9, 2020 •

edited

Loading

AndreMaz commented Jun 9, 2020 •

edited

Loading

AndreMaz commented Jun 9, 2020 •

edited

Loading

Yord commented Jun 13, 2020 •

edited

Loading

Yord commented Jun 14, 2020 •

edited

Loading

gautaz commented Jan 15, 2023 •

edited

Loading