Skip dependencies of native tasks with --no-native

[zakkak]

When creating the build schedule, if --no-native is passed as a
parameter to mx build all tasks dominated by an
AbstractNativeBuildTask (including the AbstractNativeBuildTask
itself) are removed from the schedule to avoid fetching and building
unnecessary dependencies.

Closes: oracle/graal#2221

[zakkak]

@dougxc can you please review this?

[gilles-duboscq]

The --no-native should really be called --skip-native since that's what it does.

We should not prune the task DAG we should rather try to only get what we want in the roots in the first place.

How are you using --no-native?

[zakkak]

The --no-native should really be called --skip-native since that's what it does.

I guess that comment is not directly related to the PR since --no-native (as well as --no-java) already exists:

mx/mx.py

Lines 13544 to 13545 in a7df814

	parser.add_argument('--no-java', action='store_false', dest='java', help='do not build Java projects')
	parser.add_argument('--no-native', action='store_false', dest='native', help='do not build native projects')

We should not prune the task DAG we should rather try to only get what we want in the roots in the first place.

True, that would solve other problems as well and speedup builds. It's significantly more work though :)

How are you using --no-native?

We use it to generate the necessary jars that we then use both as maven artifacts and as libraries for mandrel's native-image.
Then in a separate step we build the native dependencies using --projects and combine them all together using https://github.com/graalvm/mandrel-packaging

[gilles-duboscq]

The --no-native should really be called --skip-native since that's what it does.

I guess that comment is not directly related to the PR since --no-native (as well as --no-java) already exists:

Right, I'm just saying that their behaviour is to skip the native parts, still building everything else, regardless of dependencies.

We should not prune the task DAG we should rather try to only get what we want in the roots in the first place.

True, that would solve other problems as well and speedup builds. It's significantly more work though :)

I think it's much simpler in terms of code. One approach could be to add a pseudo dependency such as {JARS} that can be used in the --dependencies option.

[zakkak]

Hi @gilles-duboscq sorry for the late reply.

The following quote from Doug Simon (oracle/graal#2221 (comment)) seems to indicate that this is not "much simpler in terms of code" though:

When creating the build schedule, all projects, dists, library and custom dependencies are considered as roots. The schedule then just sorts them such that dependencies are built before things that depend on them. There is currently no easy way to see that a dependency is a native-only dependency.

It's not clear to me how the use of a pseudo dependency would solve the issue. As Doug points out "all projects, dists, library and custom dependencies are considered as roots", thus even with a JARS pseudo-dependency I am ending up fetching LIBFFI_SOURCES for instance.

[zakkak]

Hi @gilles-duboscq this is a kind reminder :)

[gilles-duboscq]

"all projects, dists, library and custom dependencies are considered as roots"

This a the default when you don't specify what you want to build. If you provide --dependencies then the roots are only what you specify. This is why i proposed the {JARS} pseudo dependencies.
If you use mx build --dependencies '{JARS}' then only what is needed to build the jar distributions will be built. Unless of course one of the jars contains the results of a native build.

Maybe we should start from the beginning: what are you trying to achieve? I have assumed so far that you just want to build the jars of a suite and you would like not to be bothered by unrelated build steps such as the download of some sources for a native build.

[zakkak]

Maybe we should start from the beginning: what are you trying to achieve? I have assumed so far that you just want to build the jars of a suite and you would like not to be bothered by unrelated build steps such as the download of some sources for a native build.

Our ultimate goal is to be able to build Mandrel using mostly mx and without altering suite files. Currently we rely on https://github.com/graalvm/mandrel-packaging for removing some dependencies (e.g. Truffle) and building only the necessary bits using mx.

mandrel-packaging is currently building SVM,SVM_DRIVER, and SVM_AGENT with:

mx --primary-suite-path substratevm build --no-native --dependencies SVM,SVM_DRIVER,SVM_AGENT

which fetches LIBFFI_SOURCES although we are not interested in it.

[gilles-duboscq]

OK, i see. But what is the purpose of --no-native?

I suppose you are trying to work around the dependency of these jars on the libffi project. It seems that the conclusion of #2852 was to not add an option to build against the system libffi so we need another solution.

It seems that mx build --dependencies SVM,SVM_DRIVER,SVM_AGENT brings in too many dependencies for your use-case because that contains the truffle SVM feature.
@olpaw, do you think the Truffle & NFI support could be split out to a separate jar?

[zakkak]

OK, i see. But what is the purpose of --no-native?

Internally we build the Java bits separately from the native bits.

I suppose you are trying to work around the dependency of these jars on the libffi project. It seems that the conclusion of #2852 was to not add an option to build against the system libffi so we need another solution.

Correct.

It seems that mx build --dependencies SVM,SVM_DRIVER,SVM_AGENT brings in too many dependencies for your use-case because that contains the truffle SVM feature.

That's also correct and we explicitly remove some of the dependencies by altering suite.py.

@olpaw, do you think the Truffle & NFI support could be split out to a separate jar?

That would be great.

[zakkak]

@olpaw, do you think the Truffle & NFI support could be split out to a separate jar?

@olpaw doing so would allow us to stop "patching" the suite files to remove the corresponding dependencies.
FWIW given some guidance I am willing to put the effort to do the splitting.

[zakkak]

@olpaw could you also please have a look at this? (the discussion mostly, not necessarily the PR itself)

[olpaw]

From a quick glance, wouldn't this be solved by not having GraalVM component SVM Truffle NFI Support (see substratevm/mx.substratevm/mx_substratevm.py) built by your build command? I.e. by using something like --exclude-components=svmnfi

[zakkak]

The issue of fetching LIBFFI_SOURCES is actually no longer an issue, so sorry if the outdated discussion misled you.

The current issue is that despite using:

mx --primary-suite-path substratevm --trust-http --no-jlinking --native-images=lib:native-image-agent,lib:native-image-diagnostics-agent --exclude-components=nju,svmnfi build --no-native --dependencies SVM,SVM_DRIVER,SVM_AGENT,SVM_DIAGNOSTICS_AGENT

We are still building a bunch of unnecessary dependencies. With some trial and error I was able to manually remove a number of TRUFFLE-related dependencies from Mandrel. So the question now is:

Do you think the Truffle & NFI support could be split out to a separate jar as to reduce our dependencies without manually patching the suite files when building Mandrel?

Thanks

[olpaw]

Do you think the Triffle & NFI support coulf be split out to a separate jar as to reduce our dependencies without manually patching the suite files when building Mandrel?

That would require mx projects com.oracle.svm.graal and com.oracle.svm.truffle to be in a separate distribution.

While that is technically possible, it would require quite some work. Also I worry about breaking compatibility with other projects that (knowingly or not) depend on com.oracle.svm.graal and com.oracle.svm.truffle to be part of svm.jar.

This is a topic best discussed in an upcoming "Native Image Committer Community Meeting"
cc @christianwimmer

[rschatz]

Fwiw, the com.oracle.svm.truffle.nfi* projects are already in a separate jar, svm-libffi.jar. There should be nothing that has a hard dependency on it, it's put on the ImageBuilderClassPath manually by the NFI's native-image.properties.

[zakkak]

Closing as out of date.