linkless build #955

tsmaeder · 2020-12-17T13:54:25Z

Signed-off-by: Thomas Mäder [email protected]

What does this PR do?

This is a proof of concept of a build process for che-theia that does not involve linking source folders: linking source folders introduces some problems because tools that manipulate paths will either see /projects/theia as a parent of a che-theia package or not, depending on whether the tools uses resolved file system paths or just manipulates the path string.

The idea of the PR is to move the assembly folder as a yarn workspace folder inside the che-theia repo and to yarn link all @Theia packages at the level of the che-theia repo root.

What issues does this PR fix or reference?

eclipse-che/che#16852

How to test this PR?

In a self-hosting workspace, checkout the PR branch of che-theia into /projects
Checkout master of theia into /projects
Run yarn inside theia; wait for the build to finish
cd into /projects/che-theia
run the script linktheia.sh
run yarn
To run the back-end: cd assembly && yarn run start --hostname=0.0.0.0 --port=3010 --plugins=local-dir:/projects/theia/plugins,local-dir:/projects/che-theia/plugins
To run the plugin host:
- run the linktheia.sh script in the plugin host container
- cd /projects/che-theia/extensions/eclipse-che-theia-plugin-remote/lib/node && node plugin-remote.js
Open the theia-dev-flow endpoint

if you want to build against the latest Theia release (instead of against master), you can just remove the @theia folder from /projects/che-theia/node_modules and rerun yarn.

PR Checklist

As the author of this Pull Request I made sure that:

The Eclipse Contributor Agreement is valid

Reviewers

Reviewers, please comment how you tested the PR when approving it.

Happy Path Channel

HAPPY_PATH_CHANNEL=next

che-bot · 2020-12-17T13:58:00Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	docker.io/maxura/che-theia:955
che-theia-endpoint-runtime-binary	docker.io/maxura/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

tsmaeder · 2021-01-11T16:15:35Z

@benoitf I would be interested if you see any major problems with my approach.

benoitf · 2021-01-11T16:24:07Z

@tsmaeder I think you've done two steps in this PR (two birds, one stone) and usually when we mix some concerns it's hard to get it merge easily.

There are linkless and 'generator-less' as well (BTW there is still a script to run: linktheia.sh script)

so maybe target first linkless (as for example an option in generator) so everyone can try and there is no such breakage on all existing jobs.
if committers like it, option becomes default

and if committers don't like generator, we could remove it as well (another PR/step/issue)

tsmaeder · 2021-01-12T08:40:44Z

@benoitf I'm looking for a validation of the concept before I update the generator. I was able to come up with a PoC in 1-2 days: fixing the generator to work would take me longer (I'd have to dive into it first).

benoitf · 2021-01-12T08:44:10Z

@tsmaeder if it's introduced as a new flag I don't see any issue at all

sunix · 2021-01-13T11:56:26Z

@tsmaeder do you have a devfile that would setup that POC ?

tsmaeder · 2021-01-13T12:55:52Z

@sunix the one here works fine

sunix · 2021-01-13T13:22:16Z

@sunix the one here works fine

So that one with theia and che-theia (to this fork/branch) side by side

tsmaeder · 2021-01-13T13:24:11Z

@tsmaeder just rm -rf theia and che-theia if you have them, but the referenced devfile starts with no projects anyway.

sunix · 2021-01-13T13:36:32Z

step 8. maybe having a volume for HOME would help not to link twice.

che-bot · 2021-01-27T17:35:43Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	docker.io/maxura/che-theia:955
che-theia-endpoint-runtime-binary	docker.io/maxura/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-01-29T10:28:37Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	docker.io/maxura/che-theia:955
che-theia-endpoint-runtime-binary	docker.io/maxura/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-02-16T12:41:18Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	docker.io/maxura/che-theia:955
che-theia-endpoint-runtime-binary	docker.io/maxura/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-03-12T08:46:01Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-03-22T13:35:54Z

✅ E2E Happy path tests succeed 🎉

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

benoitf · 2021-03-22T13:51:41Z

assembly/package.json

@@ -70,7 +70,7 @@
  "scripts": {
    "prepare": "yarn run clean && yarn build",
    "clean": "theia clean && rimraf errorShots",
-    "build": "theia build --mode development --config cdn/webpack.config.js --env.cdn=./cdn.json --env.monacopkg=@theia/[email protected] && yarn run override-vs-loader",
+    "build": "NODE_OPTIONS='--max-old-space-size=4096' theia build --mode development --config cdn/webpack.config.js --env.cdn=./cdn.json --env.monacopkg=@theia/[email protected] && yarn run override-vs-loader",


I would move it to .github/workflows/pr.yml ENV variable

I've thought about this a bit more, and I think it's correct to set this only for the theia build: the size of memory needed depends on the task being executed, not the environment. Other tasks don't need that much max memory, and giving it to them can make them grab more than they need and take the memory away from other processes, thus reducing potential concurrency.

codecov · 2021-03-22T14:15:36Z

Codecov Report

Merging #955 (237e589) into master (eeb4526) will increase coverage by 2.95%.
The diff coverage is 54.31%.

@@            Coverage Diff             @@
##           master     #955      +/-   ##
==========================================
+ Coverage   29.45%   32.40%   +2.95%     
==========================================
  Files         277      285       +8     
  Lines        9336     9683     +347     
  Branches     1380     1461      +81     
==========================================
+ Hits         2750     3138     +388     
+ Misses       6487     6447      -40     
+ Partials       99       98       -1

Impacted Files	Coverage Δ
...theia-about/src/browser/about-che-theia-dialog.tsx	`0.00% <ø> (ø)`
...e-sync-tracker/src/browser/sync-process-tracker.ts	`0.00% <0.00%> (ø)`
...ovisioner/src/node/git-configuration-controller.ts	`0.00% <0.00%> (ø)`
...eia-plugin-remote/src/node/hosted-plugin-remote.ts	`0.00% <0.00%> (ø)`
...theia-plugin-remote/src/node/plugin-host-custom.ts	`0.00% <0.00%> (ø)`
...in-remote/src/node/plugin-remote-backend-module.ts	`0.00% <0.00%> (ø)`
...theia-plugin-remote/src/node/plugin-remote-init.ts	`0.00% <0.00%> (ø)`
...-che-theia-plugin-remote/src/node/plugin-remote.ts	`0.00% <0.00%> (ø)`
...ugin-remote/src/node/server-plugin-proxy-runner.ts	`0.00% <0.00%> (ø)`
...plugin-remote/src/node/terminal-container-aware.ts	`0.00% <0.00%> (ø)`
... and 80 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 258d9d5...237e589. Read the comment docs.

che-bot · 2021-03-25T08:53:07Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-03-25T19:49:29Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-03-26T12:34:59Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-03-29T09:41:46Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

che-bot · 2021-03-29T10:28:22Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che Single User on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

benoitf · 2021-05-25T10:13:02Z

FYI I've an error on the link test:

 FAIL  tests/link.spec.ts
  ● Console

    console.error
      Error: Unable to execute the command yarn link @theia/bar: Error: Command failed: yarn link @theia/bar
error No registered package found called "@theia/bar".

          at new CliError (/tmp/che-theia/generator/src/cli-error.ts:17:9)
          at /tmp/che-theia/generator/src/command.ts:34:32
          at ChildProcess.exithandler (child_process.js:315:5)
          at ChildProcess.emit (events.js:314:20)
          at maybeClose (internal/child_process.js:1022:16)
          at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

      49 |         await link(cheTheiaDir, theiaDir, linkDir);
      50 |     } catch (e) {
    > 51 |         console.error(e);
         |                 ^
      52 |     }
      53 | }
      54 |

      at Object.<anonymous> (src/link.ts:51:17)
      at step (src/link.ts:5576:19)
      at Object.throw (src/link.ts:5306:14)
      at rejected (src/link.ts:5209:32)

  ● Test link command › test link

    expect(received).toBeTruthy()

    Received: false

      46 |         const zozPackage = path.resolve(srcFolder, 'node_modules/@theia/zoz/package.json');
      47 |
    > 48 |         expect(fs.existsSync(fooPackage)).toBeTruthy();
         |                                           ^
      49 |         expect(fs.existsSync(barPackage)).toBeTruthy();
      50 |         expect(fs.existsSync(zozPackage)).toBeTruthy();

che-bot · 2021-05-25T10:14:15Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

tsmaeder · 2021-05-31T07:56:27Z

@benoitf I can't reproduce the test failure. How are you running the tests?

benoitf · 2021-05-31T09:27:49Z

@tsmaeder I think it's related to symlinks where theia is cloned
I've created tsmaeder#2 and it works for me with that configuration

tsmaeder · 2021-05-31T09:36:31Z

So are you saying your link folder is itself in a symlinked location?

benoitf · 2021-05-31T09:40:10Z

So are you saying your link folder is itself in a symlinked location?

@tsmaeder yes

benoitf · 2021-05-31T09:42:08Z

it's a bug in yarn but unfortunately it's not yet solved (yarnpkg/yarn#7028 and yarnpkg/yarn#1297)

che-bot · 2021-05-31T14:18:14Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Tested with Eclipse Che on K8S (minikube v1.1.1)

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

benoitf · 2021-05-31T14:26:14Z

@tsmaeder could you please rebase/resolve the conflict ? so we can have PR checks running

Signed-off-by: Thomas Mäder <[email protected]>

benoitf

👍

che-bot · 2021-06-08T19:41:01Z

❌ E2E Happy path tests failed ❗

See Details

name	link
che-theia	quay.io/crw_pr/che-theia:955
che-theia-endpoint-runtime-binary	quay.io/crw_pr/che-theia-endpoint-runtime-binary:955

Test product:

tested with Eclipse Che on K8S (minikube v1.1.1)
E2E test scenario
E2E test pipeline source code

Use comment "[crw-ci-test]" to rerun happy path E2E test.
Use comment "[crw-ci-test --rebuild]" to re-build the images and rerun happy path E2E test.

Eclipse Che QE channel: https://mattermost.eclipse.org/eclipse/channels/eclipse-che-qe

Introduce build process using yarn link Signed-off-by: Thomas Mäder <[email protected]>

tsmaeder requested review from benoitf, azatsarynnyy and ericwill December 17, 2020 13:54

tsmaeder mentioned this pull request Mar 5, 2021

Find a good setup for an extension development against local Theia eclipse-theia/theia#1134

Open

tsmaeder force-pushed the 16852_no_source_links branch from f85e4c2 to cfce3d2 Compare March 12, 2021 08:19

tsmaeder force-pushed the 16852_no_source_links branch from cfce3d2 to 7d24fc2 Compare March 22, 2021 12:54

benoitf reviewed Mar 22, 2021

View reviewed changes

tsmaeder force-pushed the 16852_no_source_links branch from ef8d15e to a18112e Compare March 29, 2021 10:21

tsmaeder force-pushed the 16852_no_source_links branch from 45e61e9 to 6bb4ad7 Compare March 30, 2021 11:44

ericwill mentioned this pull request May 4, 2021

Plugins Sprint 201 eclipse-che/che#19719

Closed

32 tasks

ericwill mentioned this pull request May 20, 2021

Plugins Sprint 202 eclipse-che/che#19840

Closed

29 tasks

tsmaeder and others added 12 commits June 8, 2021 07:04

Introduce build process using yarn link

88a8855

Signed-off-by: Thomas Mäder <[email protected]>

Add test for link command

0a26af7

Signed-off-by: Thomas Mäder <[email protected]>

Generate assembly from a single set of templates

f175f62

Signed-off-by: Thomas Mäder <[email protected]>

Add che-theia extension to assembly

3ee3819

Signed-off-by: Thomas Mäder <[email protected]>

Remove some steps from common code path

31a03c8

Signed-off-by: Thomas Mäder <[email protected]>

Add newlines

46fa2c0

Signed-off-by: Thomas Mäder <[email protected]>

Small fix generating assembly

d12729b

Signed-off-by: Thomas Mäder <[email protected]>

Addresses some comments

66980af

Signed-off-by: Thomas Mäder <[email protected]>

Remove concurrency setting

cea914f

Signed-off-by: Thomas Mäder <[email protected]>

Remove bogus yarn.lock

ed8d1de

Signed-off-by: Thomas Mäder <[email protected]>

fix: symlinks handling of yarn links

683e444

Update yarn.lock

237e589

Signed-off-by: Thomas Mäder <[email protected]>

tsmaeder force-pushed the 16852_no_source_links branch from b61fda7 to 237e589 Compare June 8, 2021 07:25

benoitf approved these changes Jun 8, 2021

View reviewed changes

tsmaeder merged commit 732da85 into eclipse-che:master Jun 8, 2021

che-bot added this to the 7.32 milestone Jun 8, 2021

azatsarynnyy mentioned this pull request Jun 10, 2021

How to run Che-Theia within the Che IDE eclipse-che/che#19792

Closed

dmytro-ndp pushed a commit that referenced this pull request Jul 21, 2021

Introduce build process using yarn link (#955)

7e0a28f

Introduce build process using yarn link Signed-off-by: Thomas Mäder <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linkless build #955

linkless build #955

tsmaeder commented Dec 17, 2020 •

edited

Loading

che-bot commented Dec 17, 2020

tsmaeder commented Jan 11, 2021

benoitf commented Jan 11, 2021

tsmaeder commented Jan 12, 2021

benoitf commented Jan 12, 2021

sunix commented Jan 13, 2021

tsmaeder commented Jan 13, 2021

sunix commented Jan 13, 2021

tsmaeder commented Jan 13, 2021 •

edited

Loading

sunix commented Jan 13, 2021

che-bot commented Jan 27, 2021

che-bot commented Jan 29, 2021

che-bot commented Feb 16, 2021

che-bot commented Mar 12, 2021

che-bot commented Mar 22, 2021

benoitf Mar 22, 2021

tsmaeder Mar 26, 2021

codecov bot commented Mar 22, 2021 •

edited

Loading

che-bot commented Mar 25, 2021

che-bot commented Mar 25, 2021

che-bot commented Mar 26, 2021

che-bot commented Mar 29, 2021

che-bot commented Mar 29, 2021

benoitf commented May 25, 2021

che-bot commented May 25, 2021

tsmaeder commented May 31, 2021

benoitf commented May 31, 2021

tsmaeder commented May 31, 2021

benoitf commented May 31, 2021

benoitf commented May 31, 2021

che-bot commented May 31, 2021

benoitf commented May 31, 2021

benoitf left a comment

che-bot commented Jun 8, 2021

linkless build #955

linkless build #955

Conversation

tsmaeder commented Dec 17, 2020 • edited Loading

What does this PR do?

What issues does this PR fix or reference?

How to test this PR?

PR Checklist

Reviewers

Happy Path Channel

che-bot commented Dec 17, 2020

tsmaeder commented Jan 11, 2021

benoitf commented Jan 11, 2021

tsmaeder commented Jan 12, 2021

benoitf commented Jan 12, 2021

sunix commented Jan 13, 2021

tsmaeder commented Jan 13, 2021

sunix commented Jan 13, 2021

tsmaeder commented Jan 13, 2021 • edited Loading

sunix commented Jan 13, 2021

che-bot commented Jan 27, 2021

che-bot commented Jan 29, 2021

che-bot commented Feb 16, 2021

che-bot commented Mar 12, 2021

che-bot commented Mar 22, 2021

benoitf Mar 22, 2021

Choose a reason for hiding this comment

tsmaeder Mar 26, 2021

Choose a reason for hiding this comment

codecov bot commented Mar 22, 2021 • edited Loading

Codecov Report

che-bot commented Mar 25, 2021

che-bot commented Mar 25, 2021

che-bot commented Mar 26, 2021

che-bot commented Mar 29, 2021

che-bot commented Mar 29, 2021

benoitf commented May 25, 2021

che-bot commented May 25, 2021

tsmaeder commented May 31, 2021

benoitf commented May 31, 2021

tsmaeder commented May 31, 2021

benoitf commented May 31, 2021

benoitf commented May 31, 2021

che-bot commented May 31, 2021

benoitf commented May 31, 2021

benoitf left a comment

Choose a reason for hiding this comment

che-bot commented Jun 8, 2021

tsmaeder commented Dec 17, 2020 •

edited

Loading

tsmaeder commented Jan 13, 2021 •

edited

Loading

codecov bot commented Mar 22, 2021 •

edited

Loading