build: conditional CI jobs, dynamic test requirements for release, updated independent package release process

[panzarino]

These changes update our CircleCI configuration to only run tests based on what packages have changed. These changes are intended to reduce CI run time which will reduce costs and improve iteration time as we move towards having more and more packages in the monorepo.

In addition, it improves the release process as packages can dynamically wait for dependent packages' tests to successfully finish before releasing. This adds an extra layer of security to prevent regressions from being released. All packages will be released from a single script which will wait for the correct tests to pass and release packages accordingly.

For determining what tests to run, here's the decision logic that's presented in this PR:

• Running on master or develop -> run all tests
• Use git merge-base develop HEAD for change comparison
  • This commit is the point at which the current branch diverged from develop
    • Since develop is always going to be ahead of master, it works the same as git merge-base master HEAD for branches that were made off of master, with the added benefit that we don't have to figure out what branch it was made off of (I know I didn't explain this super well, but I've gone through just about every situation and it always chooses the correct commit to base off of)
• Determine what packages have changed from this commit
  • For each package that has changed, run all of the tests for that package and those listed in the ciDependents
  • Everything relating to the binary is grouped together as one "package" - if anything relating to the binary is changed we run every test for it

This commit also adds the following custom configuration options to each independent package's package.json:

ciJobs

List of Circle CI jobs that directly test the current package. These tests must pass before the package can be released.

In addition, these tests will run when a PR is created that changes this package. All tests will run on develop and master, regardless of what packages were changed.

Note: CI jobs should be unique to a package. Any jobs that are not listed within a ciJobs field are considered to be part of the binary and will only run when the binary is changed.

This option takes an array of CI job names.

Example

{
  "ciJobs": [
    "npm-react",
    "npm-react-axe",
    "npm-react-next"
  ]
}

ciDependents

List of local independent (npm) packages that are dependent on the current package. The tests specified in these packages' ciJobs must pass before the current package will be released.

When the current package is changed in a PR, it will consider these packages to be changed as well and run CI accordingly.

This option takes an array of package names.

Example

{
  "ciDependents": [
    "@cypress/react",
    "@cypress/vue",
    "@cypress/webpack-preprocessor"
  ]
}

Limitations of this setup:

• Right now ciJobs are mutually exclusive - they can only belong to one package. This can be changed easily, but since jobs not in any ciJobs property are assumed to be part of the binary, it can be confusing if we allow multiple independent packages but can't have independent packages share a job with the binary.
• There is no way to attach certain jobs to only run before release (analogous to our Mac workflow for the binary). This would be a pretty simple update, but I felt it was best to start with simplicity and have no branch specific options. It becomes complicated when we only want to run certain tests for that package's direct release but not if its a dependent, etc.
• There is no way to specify packages dependent on the binary. The binary is tested fairly extensively and since each of these independent libraries use cypress within their tests if they depend on it, it feels like we have decently comprehensive test coverage. Since all tests run on develop, there's no way that the binary could accidentally be released if dependent package tests are failing. However, this would be a fairly simple update if needed and would require us to pick where we want to throw the package.json fields for the binary (probably within cli)

Some common concerns:

• What happens if scripts or something outside of the binary lerna packages is changed?
  We include scripts/*, .node-version, electron-builder.json, package.json, and yarn.lock as part of the binary. We might need to add more files here.
• Why run all tests on develop/master?
  We already run additional tests only on develop so it makes sense that we would at least run all tests here. In addition, it's tough to come up with a good criterion for what base should be used for comparison. We could look at only what's changed in a commit and run those tests, which might make sense since we force a linear commit history and squash and merge. This has issues though since it might obscure failures and make it harder to locate where those issues were introduced. For example, take commit history a -> b. When commit a is added to develop, a server test fails. After that failure, b is then added which doesn't touch the server, and "everything" passes. Now the failure from a is obscured with the appearance of all tests passing. If we don't look at just one commit, we'd have to use the CircleCI API to look at test history or something else of that nature that greatly increases complexity for marginal benefit.

How do we actually stop tests?

We use the circleci-agent step halt command which automatically ends a job and reports it as passing. We use the logic presented above within a job to determine if it should stop running and if so, end it before any tests actually start running.

Won't this be confusing since it reports tests as passing which haven't actually run?

Yeah, it's an imperfect solution. I've added a list-changed-packages job so anyone can see what packages have changed for a CI run.

Isn't there a better solution?

Yes, there is. This was the first approach that I considered and would allow us to actually conditionally run different workflows. However, this would require more configuration and multiple workflows - essentially we would need a "trigger" workflow to run and see what packages have changed, then call the CircleCI API to actually trigger the rest of the workflows by passing pipeline parameters. (There's no way to conditionally trigger jobs, only workflows.) This sounds like the better approach on paper, but has some issues. First, the pipeline has to run twice, once for the trigger workflow and then again for the next workflows. In addition, there's no way to share workspaces between workflows (to my knowledge). This means we'd have to run the build step for each workflow, which kinda defeats the point since the build is a pretty heavy job. We also wouldn't be able to mark jobs as required on GitHub since they might not even get started. We'd need some sort of thing like what we have with Percy to report all the statuses after everything happens. Finally, this workflow breaks completely for external PRs. We need a CircleCI API token to call the API, but we can't pass secrets to external branches for security reasons. We could (maybe, not actually sure about their permissions) move the trigger step to GitHub actions, but that's a lot. Essentially, while this solution looks better on paper it greatly increases complexity and requires significant changes to our CI system, so I propose that we just stick to the simpler solution presented in this PR.

Couple other side notes with this PR - Adds and updates some documentation around the new workflow, removes our patched semantic-release-monorepo since it's not necessary and wouldn't work like it was intended to.

Apologies for the ugly code - probably would be a bit cleaner if I knew lodash better

[cypress-bot]

Thanks for taking the time to open a PR!

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
• Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

[cypress]

---

Test summary

8639 • 0 • 124 • 3 • 1

---

Run details

Project	cypress
Status	Passed
Commit	4099f9d
Started	Oct 28, 2020 9:22 PM
Ended	Oct 28, 2020 9:34 PM
Duration	12:27 💡
OS	Linux Debian - 10.2
Browser	Multiple

View run in Cypress Dashboard ➡️

---

Flakiness

	cypress/integration/retries.ui.spec.js		1
1	runner/cypress retries.ui.spec > opens attempt on each attempt failure for the screenshot, and closes after test passes

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[flotwig]

so, let me check my understanding...

if webpack-preprocessor changes, the binary will be rebuilt, because @packages/driver depends on @cypress/webpack-preprocessor, and that's discovered here?

[panzarino]

Yep. Also (at least for now) since almost all packages are affected by the binary, we just run everything if the binary has been changed. It might be smart to update this to actually verify that though, as some things like eslint-plugin-dev are completely independent of the binary.

[panzarino]

I'd also recommend reviewing the changed-packages script which was kinda squashed in without much attention in a prior PR but is pretty important here.

[JessicaSachs]

I wish that there was a way to reuse the circle definitions. Other than that... this looks good to me.

[JessicaSachs]

Anywhere with runTests and process.exit, return the statement and rename it as per above

[JessicaSachs]

Suggested change

	.filter((tag) => tag)
	.filter(_.isEmpty)