Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kbn/pm] add caching to bootstrap #53622

Merged
merged 8 commits into from
Jan 3, 2020

Conversation

spalger
Copy link
Contributor

@spalger spalger commented Dec 19, 2019

Every time someone bootstraps Kibana it runs the kbn:bootstrap script in any package which has it defined. This is because bootstrapping is what people are supposed to do when they pull changes or switch branches, and once we install updated dependencies some packages might need to be rebuilt in order to use the new dependencies.

The reality though is that most pulls/branch switches/package updates don't actually require rebuilding packages, the build output only needs to be updated if the source o the package has updated, and maybe if the dependencies of that package have updated.

This PR attempts to teach bootstrap how to only run the bootstrap scripts when the source of a package has changed, or when the deps of a package have updated (including dependencies on other local packages). To determine when things have changed the bootstrap script writes a .bootstrap-cache file to the target directory of each package. The contents of that file includes a cache key for this package and all packages this one depends on (directly or indirectly) which is a hash of the following:

  • the latest git sha of the package's directory
  • the modified time of any file modified since the last commit in the package's directory (excluding files that are in another packages directory)
  • the fully resolved version of every dependency in the package's package.json file (resolutions are read from yarn.lock)

To inspect the raw values which produce the cache values, set BOOTSTRAP_CACHE_DEBUG_CHECKSUM=1 and run yarn kbn bootstrap.

To ignore the cache and rebuild every package anyway, you can pass --no-cache.

The changes in this PR are huge because there were relatively significant changes made to the kbn/pm distributable, which can be ignores and there are more like a few hundred lines of changes.

2019-12-22 08 54 02

@spalger spalger force-pushed the implement/kbn-pm-calc-cache-key branch 6 times, most recently from fccbaa7 to 5ad8cde Compare December 20, 2019 08:31
@spalger spalger changed the title [kbn/pm] add ability to calculate a cache key for any project [kbn/pm] add caching to bootstrap Dec 20, 2019
@spalger spalger force-pushed the implement/kbn-pm-calc-cache-key branch from 5ad8cde to 5ef56a4 Compare December 20, 2019 17:24
@spalger spalger force-pushed the implement/kbn-pm-calc-cache-key branch from 5ef56a4 to d235f1b Compare December 20, 2019 17:37
@spalger spalger added release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v7.5.1 v7.6.0 v8.0.0 labels Dec 22, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@spalger spalger marked this pull request as ready for review December 22, 2019 15:23
@spalger spalger requested a review from a team as a code owner December 22, 2019 15:23
@elastic elastic deleted a comment from elasticmachine Dec 22, 2019
@elastic elastic deleted a comment from kibanamachine Jan 3, 2020
@elastic elastic deleted a comment from kibanamachine Jan 3, 2020
@elastic elastic deleted a comment from kibanamachine Jan 3, 2020
@spalger
Copy link
Contributor Author

spalger commented Jan 3, 2020

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@spalger spalger merged commit 7b4278d into elastic:master Jan 3, 2020
spalger pushed a commit to spalger/kibana that referenced this pull request Jan 3, 2020
Co-authored-by: Elastic Machine <[email protected]>

# Conflicts:
#	packages/kbn-pm/dist/index.js
spalger pushed a commit to spalger/kibana that referenced this pull request Jan 3, 2020
Co-authored-by: Elastic Machine <[email protected]>

# Conflicts:
#	packages/kbn-dev-utils/package.json
#	packages/kbn-pm/dist/index.js
#	packages/kbn-pm/src/commands/bootstrap.test.ts
#	yarn.lock
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 3, 2020
* master:
  [kbn/pm] add caching to bootstrap (elastic#53622)
  adds createdAt and updatedAt fields to alerting (elastic#53793)
spalger pushed a commit that referenced this pull request Jan 3, 2020
Co-authored-by: Elastic Machine <[email protected]>

# Conflicts:
#	packages/kbn-dev-utils/package.json
#	packages/kbn-pm/dist/index.js
#	packages/kbn-pm/src/commands/bootstrap.test.ts
#	yarn.lock
spalger pushed a commit that referenced this pull request Jan 3, 2020
Co-authored-by: Elastic Machine <[email protected]>

# Conflicts:
#	packages/kbn-pm/dist/index.js
@spalger
Copy link
Contributor Author

spalger commented Jan 3, 2020

7.x/7.6: 14d8f8a
7.5: 55cf96f

@spalger spalger deleted the implement/kbn-pm-calc-cache-key branch January 3, 2020 19:33
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 6, 2020
* master:
  increase delay to make sure license refetched (elastic#53882)
  Allow custom NP plugin paths in production (elastic#53562)
  [Maps] show custom color ramps in legend (elastic#53780)
  [Lens] Expression type on document can be null (elastic#53883)
  [SIEM] [Detection engine] Add user permission to detection engine (elastic#53778)
  Update dependency @elastic/charts to v16.0.2 (elastic#52619)
  Set consistent EOL symbol in core API docs (elastic#53815)
  [Logs UI] Refactor query bar state to hooks (elastic#52656)
  [Maps] pass getFieldFormatter to DynamicTextProperty (elastic#53937)
  Invalidate alert API Key when generating a new one (elastic#53732)
  [Logs UI] HTTP API for log entries (elastic#53798)
  [kbn/pm] add caching to bootstrap (elastic#53622)
  adds createdAt and updatedAt fields to alerting (elastic#53793)
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 6, 2020
* master:
  increase delay to make sure license refetched (elastic#53882)
  Allow custom NP plugin paths in production (elastic#53562)
  [Maps] show custom color ramps in legend (elastic#53780)
  [Lens] Expression type on document can be null (elastic#53883)
  [SIEM] [Detection engine] Add user permission to detection engine (elastic#53778)
  Update dependency @elastic/charts to v16.0.2 (elastic#52619)
  Set consistent EOL symbol in core API docs (elastic#53815)
  [Logs UI] Refactor query bar state to hooks (elastic#52656)
  [Maps] pass getFieldFormatter to DynamicTextProperty (elastic#53937)
  Invalidate alert API Key when generating a new one (elastic#53732)
  [Logs UI] HTTP API for log entries (elastic#53798)
  [kbn/pm] add caching to bootstrap (elastic#53622)
  adds createdAt and updatedAt fields to alerting (elastic#53793)
  [SR] Enable component integration tests (elastic#53893)
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jan 6, 2020
…nsole-dependencies

* 'master' of github.com:elastic/kibana: (33 commits)
  adds strict types to Alerting Client (elastic#53821)
  [Dashboard] Empty screen redesign (elastic#53681)
  Migrate config deprecations and `ShieldUser` functionality to the New Platform (elastic#53768)
  increase delay to make sure license refetched (elastic#53882)
  Allow custom NP plugin paths in production (elastic#53562)
  [Maps] show custom color ramps in legend (elastic#53780)
  [Lens] Expression type on document can be null (elastic#53883)
  [SIEM] [Detection engine] Add user permission to detection engine (elastic#53778)
  Update dependency @elastic/charts to v16.0.2 (elastic#52619)
  Set consistent EOL symbol in core API docs (elastic#53815)
  [Logs UI] Refactor query bar state to hooks (elastic#52656)
  [Maps] pass getFieldFormatter to DynamicTextProperty (elastic#53937)
  Invalidate alert API Key when generating a new one (elastic#53732)
  [Logs UI] HTTP API for log entries (elastic#53798)
  [kbn/pm] add caching to bootstrap (elastic#53622)
  adds createdAt and updatedAt fields to alerting (elastic#53793)
  [SR] Enable component integration tests (elastic#53893)
  Move index patterns: src/legacy/core_plugins/data 👉 src/plugins/data (elastic#53794)
  moved Task Manager server code under "server" directory (elastic#53777)
  Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886)
  ...

# Conflicts:
#	yarn.lock
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v7.5.1 v7.6.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants