Different behaviour observed when a build command is executed from terminal vs che.command

[slemeur]

Description

Upstream issue - eclipse-che/che#9621
Different behaviour is observer when a command is executed from terminal vs che.command

Steps to reproduce

1. Create a new application with WildFly Swarm runtime and REST mission
2. Open Che
3. In the terminal, run the command mvn clean install -f ${current.project.path} -Dmaven.test.skip=true
4. Observe that the build works well
5. Create a new command and in the instruction paste the same mvn clean install -f ${current.project.path} -Dmaven.test.skip=true
6. Run the command and observe the build failure.

Attached video

[ibuziuk]

@slemeur AFAIK, it is reproducible only for wildfly, right ?

[ghost]

@slemeur of course it is not intentional but you aren't telling the truth. In a command you pass -f path/to/dockerfile. In the terminal you cd to a directory with a Dockerfile in its root. See: #2152 (comment)

So, probably this issue is a duplicate.

[ghost]

And yes, when running mvn clean install with skipped tests it just hangs on collecting artifacts.

[ibuziuk]

@eivantsov AFAIK 2 identical commands (with skipping tests) behaves differently for wildfly:

• from terminal works fine BUILD SUCCESS
• via command BUILD FAILURE with OOME - 'GreetingServiceTest' from wildfly booster is failing in Che #2152 (comment)

[ghost]

@ibuziuk however, it is not related to this issue, at least with a given description. It's weird why invoking maven build from root and by passing path to pom.xml works in a different way.

As to OOM, I think a good way to oc exec without Che. I wonder if we see the same error.

[ghost]

@ibuziuk Che is not to blame here:

eugene@ivantsoft ~ $ oc exec workspacem93qanu7dl8cf29w.dockerimage -- mvn clean install -DskipTests -f /projects/app-test-1
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building WildFly Swarm Booster - Health Check Mission 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ app-test-1 ---
[INFO] Deleting /projects/app-test-1/target
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-java-version) @ app-test-1 ---
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-maven-version) @ app-test-1 ---
[INFO] 
[INFO] --- buildnumber-maven-plugin:1.4:create (get-scm-revision) @ app-test-1 ---
[INFO] Executing: /bin/sh -c cd '/projects/app-test-1' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /projects/app-test-1
[INFO] Storing buildNumber: b989b63b8401686c2d6f5dc65caf88385c8e2ab7 at timestamp: 1525454471114
[INFO] Storing buildScmBranch: master
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ app-test-1 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /projects/app-test-1/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ app-test-1 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to /projects/app-test-1/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ app-test-1 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ app-test-1 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /projects/app-test-1/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ app-test-1 ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-war-plugin:3.1.0:war (default-war) @ app-test-1 ---
[INFO] Packaging webapp
[INFO] Assembling webapp [app-test-1] in [/projects/app-test-1/target/app-test-1-1.0.0]
[INFO] Processing war project
[INFO] Copying webapp resources [/projects/app-test-1/src/main/webapp]
[INFO] Webapp assembled in [132 msecs]
[INFO] Building war: /projects/app-test-1/target/app-test-1-1.0.0.war
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.110 s
[INFO] Finished at: 2018-05-04T17:21:16+00:00
[INFO] Final Memory: 59M/66M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.1.0:war (default-war) on project app-test-1: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]

To purify the experiment, you may deploy the image yourself, without Che, and give it a try in a shell and in a detached way through exec.

[ibuziuk]

@eivantsov I think you are commenting about #2152 not about this issue. The fact that is wildfly failing in docker is clearly quickstart problem. But we wanted to workaround it by skipping tests and this still fails with OOM when executing as Che command. Isn't it Che related problem (OOM for command execution I mean)?

[ibuziuk]

@eivantsov ah, you mean it fails with OOM in docker! good point

[ghost]

@ibuziuk not on Docker. On OSO. I stated a workspace from OSIO and ran an exec against pod.

[slemeur]

yeah, so there is something weird indeed - and I think @eivantsov you are making a good point.
@bmicklea is actually able to do the build with:
mvn clean install -f ${current.project.path} -Dmaven.test.skip=true in the definition of the command

[ibuziuk]

@bmicklea it is quite interesting that you do not have oom. Could you please tell against which oso cluster you are provisioned ?

[bmicklea]

Odd, now it's failing with the following:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.024 s
[INFO] Finished at: 2018-05-04T22:32:26+00:00
[INFO] Final Memory: 52M/63M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.1.0:war (default-war) on project wildfly-test: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ibuziuk]

yeah, this is what we all observe ;-)

[bmicklea]

Not sure if it's helpful but it was working for me before JF did investigation into my space. It had been working perfectly (including this build working) then I hit one of the Jenkins issues and since then it's been broken. Not sure if there's a correlation... -- Brad Micklea // Director, Product Management // Developer Tools // 416.707.0792

[skabashnyuk]

I believe that this issue related to MAVEN_OPTS that we have in command execution environment.
For OSIO we have

terminal

[user@workspacedvyxxdwxbbf4kfx0 projects]$ env | grep OPTS
JAVA_OPTS=-XX:+UseG1GC -XX:+UseStringDeduplication -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:MaxRAM=1200m -Xms256m
MAVEN_OPTS=-XX:+UseG1GC -XX:+UseStringDeduplication -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:MaxRAM=1200m -Xms256m
[user@workspacedvyxxdwxbbf4kfx0 projects]$

commmand

JAVA_OPTS=-XX:+UseG1GC -XX:+UseStringDeduplication -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:MaxRAM=1200m -Xms256m
MAVEN_OPTS=-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom

Upstream Che

terminal

user@42a2e936baaf:/projects$ env | grep OPTS
JAVA_OPTS=-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom
MAVEN_OPTS=-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom
user@42a2e936baaf:/projects$

command

env | grep OPTS
JAVA_OPTS=-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom 
MAVEN_OPTS=-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom 

[skabashnyuk]

My "Quick and dirty" proposal is to execute commands like this
source ~/.bashrc && your_command_here

[skabashnyuk]

Related to this pr fabric8-services/fabric8-tenant-che#79

[ibuziuk]

@slemeur we could update the wildfy build command according to @skabashnyuk proposal. This would require rebuilding rh-che & yet another rollout on prod. wdyt ?

[slemeur]

We should be testing that it works. I’m afk at this moment.could you validate it works?

If we want to avoid another rollout, the other approach would be to edit the stack in che-server (as an admin).

We would still need to update the stack, so that next rollout, the change will be there and consistent.

Otherwise, if we can handle it, I’m fine to do another rollout now.

[ibuziuk]

had a quick chat with @skabashnyuk . Approach with .bashrc is not reliable since it would set the following:

export MAVEN_OPTS=$JAVA_OPTS

This would potentially kill wsagent since there would not be enough ram allocated in the pod for both processes. Safer option / hack that works for me is the following:

MAVEN_OPTS="-Xmx100m" && scl enable rh-maven33 'mvn clean install -f ${current.project.path} -DskipTests=true

[bmicklea]

@ibuziuk that works for me too (although others should try since mine was working yesterday with the command that wasn't working for anyone else).

For some reason the trailing ' was lost in your command, so I'm repasting:

MAVEN_OPTS="-Xmx100m" && scl enable rh-maven33 'mvn clean install -f ${current.project.path} -DskipTests=true'

[slemeur]

+1 Le sam. 5 mai 2018 à 13:32, Brad Micklea <[email protected]> a écrit :

@ibuziuk <https://github.com/ibuziuk> that works for me too (although others should try since mine was working yesterday with the command that wasn't working for anyone else). For some reason the trailing ' was lost in your command, so I'm repasting: MAVEN_OPTS="-Xmx100m" && scl enable rh-maven33 'mvn clean install -f ${current.project.path} -DskipTests=true' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3463 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABj5oZVhyCcZ-VSgqWrY_WsUIaBqtHxhks5tvY3hgaJpZM4Ty6Ke> .

-- STEVAN LEMEUR PRODUCT MANAGER Red Hat <https://www.redhat.com/> [email protected] M: +336-87-11-27-55 <javascript:void(0);> <https://red.ht/sig>

[ibuziuk]

PR is merged and fixed should be available on prod-preview

[slemeur]

Was not able to test, as I hit an issue while creating the workspace.

[ibuziuk]

@slemeur it looks like rollout of che on prod-preview osd failed: Deployment rhche #47 failed

  | --> Scaling rhche-46 down to zero
-- | --
  | --> Scaling rhche-47 to 1 before performing acceptance check
  | error: update acceptor rejected rhche-47: watch closed before Until timeout

[ibuziuk]

Created housekeeping issue for prod-preview rollout problem - https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/1861

It is probably coupled with che 5 deprecation and removal of all resources, the thing is that both che 5 & che 6 are using the same che secret for obtaining service.account.secret / service.account.id

[ibuziuk]

Created a SEV1 for this #3476

[ibuziuk]

Che is back to prod-preview and available for testing \o/
wildfly build / run commands seem to work fine.
Here is the PR for rollout of those fixes to prod - openshiftio/saas-openshiftio#871

@slemeur if you are ok with the update of prod, I hope @davidfestal & @garagatyi would be able to handle rollout with announcement on Monday.
@jfchevrette told that rhche secret on prod already contains all required entries so it would be just a matter of announcing on devtools-saas & #openshiftio + merging PR

[bmicklea]

Do we need or want to wait for Monday to push this live? Seems like it would be better to push it now, test it and know it's ready than wait...

[ibuziuk]

@bmicklea it is risky to push it now since it could technically be broken if there is smth. wrong from infa side e.g. invalid secret values on prod and there would be no one from che / sd side to fix it. So, I'd better wait till Monday when there would be people from both che & sd team around, announce update as we always did before and merge the PR for rollout. If smth. goes wrong there will be right people to address the problem quickly

[riuvshin]

@ibuziuk I would recommend to increase xmx to 256 at least. we've updated prod and fix worked for @garagatyi and @rhopp but still failed for me... with 256mb it should be more stable.

[slemeur]

@ibuziuk @riuvshin : The fix if not working on prod:

[ghost]

@slemeur it looks like you have not built the project. Why do you expect a jar in target?

[slemeur]

I guess the command is wrong too.
Even running the following command in the stack:
MAVEN_OPTS="-Xmx100m" && scl enable rh-maven33 'mvn clean install -f ${current.project.path} -DskipTests=true' is not working either

Changing to :
`MAVEN_OPTS="-Xmx256m" && scl enable rh-maven33 'mvn clean install -f ${current.project.path} -DskipTests=true'
Works well.

[slemeur]

@eivantsov : I just hit the default run command for the wildfly stack

[slemeur]

Ok - Doing the "build" command and then doing the "run" command is working.

[ghost]

@slemeur yes, this is what I meant. Perhaps, for demo purposes the two should be combined in one.

[slemeur]

yes, here the flow is not consistent. In all other stacks, the "run" command is also doing the "build".

[ibuziuk]

@eivantsov am I correct that we can close this one since the upstream issue is resolved eclipse-che/che#9621 ?

[ghost]

Yes, it can be closed

[ibuziuk]

@eivantsov hmm.. I thought that we could remove now maven options set in the build command [1] for wildfly but it is still failing. And btw it is started to fail also from terminal. So now behavior is pretty consistent in a bad way ;-) setting MAVEN_OPTS='-Xmx500m' ` fixes the issue, so wondering what would be the best way to tackle it by default ?

[1] https://github.com/redhat-developer/rh-che/blob/master/assembly/fabric8-stacks/src/main/resources/stacks.json#L759

[ghost]

Well, the original issue was about consistency. It is consistent now. Yes, MAVEN_OPTS may be required for some projects

[ibuziuk]

Closing, since upstream issue has been closed eclipse-che/che#9621 (comment)
@eivantsov thanks for verifying on osio