This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
[MXNET-1155] Add scala packageTest utility #13046
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
zachgk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| SCALA_VERSION_PROFILE := 2.11 | ||
| SCALA_VERSION := 2.11.8 | ||
| MXNET_VERSION := "[1.3.0-SNAPSHOT,)" | ||
|
|
||
| MXNET_REPO = https://repository.apache.org/content/repositories/snapshots | ||
|
|
||
| ifeq ($(OS),Windows_NT) | ||
| UNAME_S := Windows | ||
| else | ||
| UNAME_S := $(shell uname -s) | ||
| endif | ||
|
|
||
| ifeq ($(UNAME_S), Windows) | ||
| # TODO: currently scala package does not support windows | ||
| SCALA_PKG_PROFILE := windows | ||
| else | ||
| ifeq ($(UNAME_S), Darwin) | ||
| SCALA_PKG_PROFILE := osx-x86_64-cpu | ||
| else | ||
| SCALA_PKG_PROFILE := linux-x86_64 | ||
| ifeq ($(USE_CUDA), 1) | ||
| SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-gpu | ||
| else | ||
| SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-cpu | ||
| endif | ||
| endif | ||
| endif | ||
|
|
||
| PROFILES := -Ptest | ||
| ifeq ($(UNIT), 1) | ||
| PROFILES := "$(PROFILES),unittest" | ||
| endif | ||
| ifeq ($(INTEGRATION), 1) | ||
| PROFILES := "$(PROFILES),integrationtest" | ||
| endif | ||
|
|
||
| ifneq ($(UNIT), 1) | ||
| ifneq ($(INTEGRATION), 1) | ||
| PROFILES := "$(PROFILES),unittest,integrationtest" | ||
| endif | ||
| endif | ||
|
|
||
|
|
||
| clean: | ||
| (mvn clean -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ | ||
| -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ | ||
| -Dmxnet.version=$(MXNET_VERSION) \ | ||
| -Dscala.version=$(SCALA_VERSION)) | ||
|
|
||
| testinstall: | ||
| (mvn integration-test -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ | ||
| $(PROFILES) \ | ||
| -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ | ||
| -Dmxnet.version=$(MXNET_VERSION) \ | ||
| -Dscala.version=$(SCALA_VERSION)) | ||
|
|
||
| testlocal: | ||
| (mvn integration-test -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ | ||
| $(PROFILES),fromLocal \ | ||
| -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ | ||
| -Dmxnet.version=$(MXNET_VERSION) \ | ||
| -Dscala.version=$(SCALA_VERSION)) | ||
|
|
||
| testsnapshot: | ||
| (mvn integration-test -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ | ||
| $(PROFILES),fromSnapshots \ | ||
| -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ | ||
| -Dmxnet.repo=$(MXNET_REPO) \ | ||
| -Dmxnet.version=$(MXNET_VERSION) \ | ||
| -Dscala.version=$(SCALA_VERSION)) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # MXNet Scala Package Test | ||
|
|
||
| This is an project created to run the test suite on a fully packaged mxnet jar. The test suite is found locally but mxnet is from the target jarfile. | ||
|
|
||
| ## General Setup | ||
|
|
||
| To setup the packageTest, you must first build your tests. To build the tests, follow these steps from the mxnet main directory: | ||
|
|
||
| 1. Build MXNet and the scala package from source following the directions [here](https://mxnet.incubator.apache.org/install/scala_setup.html#source) | ||
| 2. Build the tests by running `make scalatestcompile`. | ||
| 3. Follow setup instructions below for your testing goal | ||
|
|
||
| ## Running | ||
|
|
||
| There are three different modes of operation for testing based on the location of the jar and where it is coming from: | ||
|
|
||
| ### Test Installed Jars | ||
|
|
||
| If you have a jar file, you can install it to your maven cache repository(`~/.m2/repository`). This might be useful if you acquire the .jar file from elsewhere. To install, it is easiest to use `mvn install:install-file -Dfile=<path-to-file> -DpomFile=<path-to-pomfile>`. If the pom file is not available, you can also run `mvn install:install-file -Dfile=<path-to-file> -DgroupId=<group-id> -DartifactId=<artifact-id> -Dversion=<version> -Dpackaging=<packaging>`. With the full mxnet jar, this might look like `mvn install:install-file -Dfile=<path-to-file> -DgroupId=org.apache.mxnet -DartifactId=mxnet-full_2.11-linux-x86_64-cpu -Dversion=1.3.0 -Dpackaging=jar`. | ||
|
|
||
| You can also run `make scalainstall` to install from a local build. | ||
|
|
||
| After installing, run `make testinstall` in the package test directory to run the tests. Note that unless you also install an additional mxnetexamples jar, you can only run the unit tests. | ||
|
|
||
zachgk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ### Test Local Deployment | ||
|
|
||
| To test the jars that would be produced by a deployment, you can run `make scaladeploylocal` from the main mxnet directory. This produces a local snapshot located at `scala-package/local-snapshot`. To test this local snapshot, run `make testlocal`. | ||
|
|
||
| ### Remote Repository Snapshot | ||
|
|
||
| This mode is to test a jar located in a remote repository. The default repository is the apache snapshot repisotory located at `https://repository.apache.org/content/repositories/snapshots`. Note that the actual jar in a repisotory should be located at `$repoUrl/org/apache/mxnet/mxnet-full_$scalaVersion-$osMode/$version/*.jar`. | ||
|
|
||
| Test the snapshot repo using `make testsnapshot` or a different repo using `make testsnapshot MXNET_REPO=$NEW_REPO_URL`. | ||
|
|
||
| ### Options | ||
|
|
||
| You are able to run unit tests, integration tests, or both using this utility. To run the unit tests, add the flag `UNIT=1` to make (e.g. `make testsnapshot UNIT=1`). Use `INTEGRATION=1` for integration tests. The default behavior is to run both the unit and integration tests. However, the integration tests require that the mxnet examples be installed in addition to the full mxnet package (see test mode instructions above). | ||
|
|
||
| An additional option, you can specify the mxnet version with `MXNET_VERSION=1.3.1-SNAPSHOT`. | ||
|
|
||
| ## Cleaning Up | ||
|
|
||
| You can clean temporary files and target artifacts by running `make scalaclean`. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Missing Examples | ||
|
|
||
| If you fail with the following error | ||
| ``` | ||
| [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project mxnet-scala-packagetest-examples_2.11: There are test failures -> [Help 1] | ||
| [ERROR] | ||
| [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. | ||
| [ERROR] Re-run Maven using the -X switch to enable full debug logging. | ||
| [ERROR] | ||
| [ERROR] For more information about the errors and possible solutions, please read the following articles: | ||
| [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException | ||
| [ERROR] | ||
| [ERROR] After correcting the problems, you can resume the build with the command | ||
| [ERROR] mvn <goals> -rf :mxnet-scala-packagetest-examples_2.11 | ||
| Makefile:57: recipe for target 'scalaintegrationtest' failed | ||
| make: *** [scalaintegrationtest] Error 1 | ||
| ``` | ||
|
|
||
| and stacktrace begins with the following, | ||
|
|
||
| ``` | ||
| *** RUN ABORTED *** | ||
| java.lang.NoClassDefFoundError: org/apache/mxnetexamples/Util$ | ||
| ``` | ||
|
|
||
| you are missing the mxnetexamples package. See your test mode installation section for details. | ||
|
There was a problem hiding this comment. Can you link to the test mode installation section so that it is easy to navigate ? There was a problem hiding this comment. In document linking is not officially part of markdown. Github added the feature, but it won't work in other viewers such as intellij. I would rather have no link than a link that might not work |
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <parent> | ||
| <groupId>PackageTest</groupId> | ||
| <artifactId>mxnet-scala-packagetest_2.11</artifactId> | ||
| <version>1.0-SNAPSHOT</version> | ||
| <relativePath>../pom.xml</relativePath> | ||
| </parent> | ||
|
|
||
| <artifactId>mxnet-scala-packagetest-core_2.11</artifactId> | ||
| <name>MXNet Scala Package Test</name> | ||
| <packaging>pom</packaging> | ||
|
|
||
| <profiles> | ||
| <profile> | ||
| <id>unittest</id> | ||
| <properties> | ||
| <skipTests>false</skipTests> | ||
| </properties> | ||
| </profile> | ||
| </profiles> | ||
|
|
||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>org.scalatest</groupId> | ||
| <artifactId>scalatest-maven-plugin</artifactId> | ||
| <version>1.0</version> | ||
| <configuration> | ||
| <runpath>${project.build.outputDirectory},${project.build.testOutputDirectory},../../core/target/test-classes</runpath> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
|
|
||
| </project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../core/scripts | ||
zachgk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <parent> | ||
| <groupId>PackageTest</groupId> | ||
| <artifactId>mxnet-scala-packagetest_2.11</artifactId> | ||
| <version>1.0-SNAPSHOT</version> | ||
| <relativePath>../pom.xml</relativePath> | ||
| </parent> | ||
|
|
||
| <artifactId>mxnet-scala-packagetest-examples_2.11</artifactId> | ||
| <name>MXNet Scala Package Test</name> | ||
| <packaging>pom</packaging> | ||
|
|
||
| <profiles> | ||
| <profile> | ||
| <id>integrationtest</id> | ||
| <properties> | ||
| <skipTests>false</skipTests> | ||
| </properties> | ||
| </profile> | ||
| </profiles> | ||
|
|
||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>org.scalatest</groupId> | ||
| <artifactId>scalatest-maven-plugin</artifactId> | ||
| <version>1.0</version> | ||
| <configuration> | ||
| <runpath>${project.build.outputDirectory},${project.build.testOutputDirectory},../../examples/target/test-classes</runpath> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.mxnet</groupId> | ||
| <artifactId>mxnet-examples_${mxnet.scalaprofile}</artifactId> | ||
| <version>${mxnet.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| </project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../examples/scripts |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why we use a
&&instead of;?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
&& executes the second command only if the first succeeds (see #13046 (comment))