Re-slice the Meteor system and packages

[msavin]

Whereas the NPM world provides a nice way to distribute libraries, I think Meteor's packaging system needs to look at packages a way to add or remove features from the framework.

It's pretty close to doing that, but has a number of issues, since a lot of functionality is coupled with-in a few jumbo packages.

For example:

• The mobile-experience imposes many "packages for a great mobile UX," which can be easily argued against, and since they are mostly third party libraries, should not be handled by Meteor, but perhaps NPM.
• The package mongo bundles the server side MongoDB driver with the client side cache (minimongo). If I try to remove the package, and just use minimongo as a cache, I get errors.
• The package meteor-base is a bit of a mystery box. I would assume this is where things like Pub/Sub, LiveQuery, etc, live, but I have no way to access them
• Some semantics are off. For example, if I want to disable LiveQuery, I should just have to remove the LiveQuery package. Instead, I have to add a package called disable-oplog-tailing 🤔
• If I'm using a different data stack, I might want to disable pub/sub completely. However, I cannot, and Meteor will even force me to use it for some features.

Now, imagine if, when we create a new package, we instead get a package list like this and can add/remove the features we want:

# required / maybe does not even need to be in packages list
meteor-build-tool
es6-babel
dynamic-imports

# server tools
methods
audit-arguments
check 
timers
assets

# data tools 
ddp 				# remove it to disable websockets and fall back to http
pubsub
mongo-server
livequery

# front-end
blaze
tracker
reactive-var
reactive-dict
minimongo

# shared features
hot-reload

That way, I can build a lean app that doesn't use pub/sub and auto-updating, and so on. I think a big part of making this work will be to extend the Meteor package system to be able to say "this package should only work if package X is present", which I believe will serve as yet another differentiator against NPM.

This can also be a great way to provide long term support for packages. Each feature can have some kind of guarantee or lack of guarantee. Since MDG has jumped through big hooks to keep Meteor backwards compatible, they might as well enjoy the marketing benefits of it.

[msavin]

Few more things to add:

• this approach would increase the transparency of a framework, which is desperately needed since "magic" is not viewed as a good thing. By seeing the guts of the framework, people can better study these things and make decisions
• there are more things we can do to bring value to the Meteor package system. For example, we can add package flags, such as kadira:apm@prodOnly. These things have nice benefits that NPM does not, and could be more beneficial to implement than going through the pain of porting everything to NPM
• Perhaps a good way to slice the system, is by the documentation. For example, each part of the documentation, such as Assets, Session, etc, should be represented by a package.

[hwillson]

Hi @msavin - thanks for the feature request. Technically speaking, the Meteor package system already supports all of the functionality you've outlined here. Taking the tech out of this then, what remains is the decoupling of internal Meteor packages. Some decoupling is already under way (see #31 for example), but there is definitely a lot more that can be done. That being said, it's important to note that big picture pie in the sky feature requests like this are very difficult to help move forward. We should instead consider creating focused decoupling feature requests, that target specific areas of the Meteor codebase, in small chunks (like #31 targets disabling the mongo server). Small and focused FR's are not only easier to understand and discuss, but they're also easier to implement, test, review and ship.

[zimme]

Maybe we could use this FR to outline which packages it would make sense to split up and why/how.
This way the community would have a more concrete list of packages to refactor when they have time to help out?

[StorytellerCZ]

@zimme More like the FR should be the central epic and things like #31 would be the issue for the specific part/module. So hear we track the big picture.

[zimme]

So hear we track the big picture.

Pretty much what I was getting at, I guess I could have phrased it better =)