Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(performance): allow creating a faster capsule #2162

Merged
merged 28 commits into from
Dec 6, 2019
Merged

Conversation

imsnif
Copy link
Contributor

@imsnif imsnif commented Nov 26, 2019

Proposed Changes

This integrates the librarian package manager to bit when building components. It allows an environment to:

  1. pass a skipNodeModules argument when isolating, so that the capsule will know not to install with npm/yarn.
  2. use capsule.execNode to run an executable through librarian, which patches the executables fs module with the required node_modules.
  3. allows the environment to pass a keepExistingCapsule flag to bit so that the capsule will not be deleted when it is done (this is so that the next librarian installation will be cached and much faster - it can also help an environment rely on incremental builds).

This change is Reviewable

@GiladShoham
Copy link
Member

@imsnif please configure the circle to be able to install the package so we can see that nothing breaks

src/constants.ts Outdated
@@ -4,6 +4,7 @@ import * as path from 'path';
import format from 'string-format';
import { PathOsBased } from './utils/path';
import { ComponentOrigin } from './consumer/bit-map/component-map';
process.env.MEMFS_DONT_WARN = 'true'; // suppress fs experimental warnings from memfs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure this is the correct place to put it.
maybe in the main file (bin/bit.js)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put it in src/app.ts.

// @ts-ignore AUTO-ADDED-AFTER-MIGRATION-PLEASE-FIX!
this.capsulePackageJson = this.componentWithDependencies.component.packageJsonFile;
// @ts-ignore AUTO-ADDED-AFTER-MIGRATION-PLEASE-FIX!
this.componentRootDir = this.componentWithDependencies.component.writtenPath;
await this._addComponentsToRoot();
logger.debug('ManyComponentsWriter, install packages on capsule');
await this._installWithPeerOption();
if (!opts.skipNodeModules) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like it's the same as the installNpmPackages option. (and it's just a bug it wasn't used here).
if so, let's use the existing one?

@imsnif
Copy link
Contributor Author

imsnif commented Dec 4, 2019

Hey, @GiladShoham: so this is ready for another review round. At the time of this writing the AppVeyor tests are still running, but the e2e circle tests are passing. I'll address the AppVeyor errors if/when they come.

Things I did:

  1. I found a bug where components with dependencies would not be updated if their dependencies changed. This involved some changes in bit (namely a smarter overriding of existing files inside a long lived capsule), as well as in librarian (already pushed to the branch) and in the typescript compiler (passing a proper flag to the isolate function and not doing an incremental typescript build, because tsc does not handle this situation well).
  2. Used the installNpmPackages flag instead of the skipNodeModules flag (which I removed)
  • This also involved adjusting the isolate command to accept a skip-npm-installation instead of a install-packages flag, as well as adjusting some e2e tests to match this behaviour.
  1. Moved the environment variable as per @davidfirst's recommendation
  2. Fixed lint issues
  3. Merged master to here

@AppVeyorBot
Copy link

Build bit 1.0.8499 completed (commit 0e0dddbe01 by @imsnif)

@@ -83,6 +98,8 @@ export default class Isolator {
});
}
const writeToPath = opts.writeToPath;
const installNpmPackages = typeof opts.installNpmPackages === 'undefined' ? true : opts.installNpmPackages;
// default should be true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please move the comment to be above the code line, it's above the code across the whole repo. (relevant for other comments as well)

async _writeFileToCapsule(capsule: Capsule, file: AbstractVinyl) {
if (file.override === false) {
async _writeFileToCapsule(capsule: Capsule, file: AbstractVinyl, opts = { overwriteExisting: false }) {
if (file.override === false || opts.overwriteExisting) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the flow here, it's very confusing:

  1. what is the different between file.override and opts.overwriteExisting? where do I get each one of them? who is stronger?
  2. the condition is very weird, it's sounds something like - if (not override or override get here). like if (false or true) enter.
    (I don't have in mind a better way, but let's try thinking about something more clear)

@AppVeyorBot
Copy link

Build bit 1.0.8504 failed (commit c9e315b674 by @imsnif)

@AppVeyorBot
Copy link

Build bit 1.0.8505 failed (commit 2e2c078bd9 by @imsnif)

@AppVeyorBot
Copy link

Build bit 1.0.8521 failed (commit 7e2351dbaf by @imsnif)

@AppVeyorBot
Copy link

Build bit 1.0.8536 completed (commit 59d529c3c9 by @imsnif)

@AppVeyorBot
Copy link

Build bit 1.0.8541 completed (commit b86eeeae30 by @imsnif)

@imsnif imsnif merged commit 31f428d into master Dec 6, 2019
@GiladShoham GiladShoham deleted the faster-capsule-build branch March 3, 2020 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants