Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Migrate hdfs default setup #4140

Merged
merged 7 commits into from
Feb 5, 2024
Merged

ci: Migrate hdfs default setup #4140

merged 7 commits into from
Feb 5, 2024

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Feb 4, 2024

No description provided.

Signed-off-by: Xuanwo <[email protected]>
@Xuanwo
Copy link
Member Author

Xuanwo commented Feb 4, 2024

Test on java binding failed for:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.opendal.test.behavior.AsyncRenameTest
could not find method getRootCauseMessage from class org/apache/commons/lang3/exception/ExceptionUtils with signature (Ljava/lang/Throwable;)Ljava/lang/String;
Aborted (core dumped)
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
Warning:  Corrupted channel by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /home/runner/work/opendal/opendal/bindings/java/target/surefire-reports/2024-02-04T03-57-59_023-jvmRun1.dumpstream
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:35 min
[INFO] Finished at: 2024-02-04T03:58:17Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project opendal-java: 
Error:  
Error:  Please refer to /home/runner/work/opendal/opendal/bindings/java/target/surefire-reports for the individual test results.
Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd '/home/runner/work/opendal/opendal/bindings/java' && '/usr/lib/jvm/temurin-11-jdk-amd64/bin/java' '-jar' '/home/runner/work/opendal/opendal/bindings/java/target/surefire/surefirebooter-20240204035759134_3.jar' '/home/runner/work/opendal/opendal/bindings/java/target/surefire' '2024-02-04T03-57-59_023-jvmRun1' 'surefire-20240204035759134_1tmp' 'surefire_0-20240204035759134_2tmp'
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 134
Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd '/home/runner/work/opendal/opendal/bindings/java' && '/usr/lib/jvm/temurin-11-jdk-amd64/bin/java' '-jar' '/home/runner/work/opendal/opendal/bindings/java/target/surefire/surefirebooter-20240204035759134_3.jar' '/home/runner/work/opendal/opendal/bindings/java/target/surefire' '2024-02-04T03-57-59_023-jvmRun1' 'surefire-20240204035759134_1tmp' 'surefire_0-20240204035759134_2tmp'
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 134
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
Error:  	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
Error:  	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
Error:  	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
Error:  	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
Error:  	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)
Error:  	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
Error:  	at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Error:  	at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:52)
Error:  	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:161)
Error:  	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:73)
Error:  
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error: Process completed with exit code 1.

cc @tisonkun @G-XD, do you have suggestions?

@tisonkun
Copy link
Member

tisonkun commented Feb 4, 2024

[INFO] Running org.apache.opendal.test.behavior.AsyncRenameTest
could not find method getRootCauseMessage from class org/apache/commons/lang3/exception/ExceptionUtils with signature (Ljava/lang/Throwable;)Ljava/lang/String;
Aborted (core dumped)

This is the root cause. Let me take a closer look.

@tisonkun
Copy link
Member

tisonkun commented Feb 4, 2024

I read the docs of the service and it writes:


The path of libjvm.so could be different, please keep an eye on it.

  • If meeting errors like the following:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)

CLASSPATH is not set correctly or your hadoop installation is incorrect.

To set CLASSPATH:

export CLASSPATH=$(find $HADOOP_HOME -iname "*.jar" | xargs echo | tr ' ' ':'):${CLASSPATH}

@tisonkun
Copy link
Member

tisonkun commented Feb 4, 2024

IIRC we don't even turn on "services-hdfs" for java binding.

And so does Node.js binding and Python binding.

Actually Python binding skips all the tests, and I don't know why Node.js gets tests passed. Maybe we can run in verbose mode for more information.

@Xuanwo
Copy link
Member Author

Xuanwo commented Feb 5, 2024

IIRC we don't even turn on "services-hdfs" for java binding.

I don't want to enable services-hdfs for bindings by default. This change aims to streamline the workflows and ensure that everything works properly at a minimum.

Signed-off-by: Xuanwo <[email protected]>
This reverts commit 7c7b85a.
Signed-off-by: Xuanwo <[email protected]>
Signed-off-by: Xuanwo <[email protected]>
@Xuanwo
Copy link
Member Author

Xuanwo commented Feb 5, 2024

Exposing HDFS to Java seems nonsensical and difficult, so I've decided to ignore this case.

Copy link
Member

@tisonkun tisonkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

I still wonder why Node.js tests can pass, there should be some hidden logics to be figured out @suyanhanx

.github/scripts/test_behavior/plan.py Show resolved Hide resolved
@Xuanwo Xuanwo merged commit 1b3c95f into main Feb 5, 2024
13 checks passed
@Xuanwo Xuanwo deleted the refactor-hdfs-setup branch February 5, 2024 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants