Gradle: can't targetExclude files in the buildDir

[tbroyer]

I have a project where some Java source files are generated by a task; they're generated into build/generated-sources/ and added to a source set to later be compiled (like any other source files). I don't want those generated source files to be checked, but setting targetExclude("build/generated-sources/**/*.java") doesn't work.

This is using Gradle 5.4.1 and Spotless 3.23.0 on Linux.

The way java() works, target is set to include allJava sources for all source sets, irrespective of their location (contrary to setting the target to an Ant-style pattern, which will exclude files in the buildDir).
The problem is that targetExclude() works the same as target() and will also exclude the buildDir, so targetExclude("build/generated-sources/**/*.java") will result in an empty FileTree, and my generated sources will be checked by Spotless.

I found a workaround by using a FileTree, which won't go through that code that excludes the buildDir:

targetExclude(fileTree("$buildDir/generated-sources") { include("**/*.java") })

But the special exclusions should only be applied to the target, not targetExclude.

To reproduce, no need for a task generating source code: put some malformed Java source file somewhere in the buildDir (e.g. $buildDir/generated-sources/test/Foo.java) and include it in a source set:

sourceSets {
  test {
    java {
      srcDir("$buildDir/generated-sources/test")
    }
  }
}
spotless {
  java {
    targetExclude("build/generated-sources/**/*.java")
    googleJavaFormat()
  }
}

[nedtwigg]

Hmmm... so JavaExtension sets the target automatically like this:

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/JavaExtension.java

Lines 176 to 198 in 9843130

	@Override
	protected void setupTask(SpotlessTask task) {
	if (target == null) {
	JavaPluginConvention javaPlugin = getProject().getConvention().findPlugin(JavaPluginConvention.class);
	if (javaPlugin == null) {
	throw new GradleException("You must apply the java plugin before the spotless plugin if you are using the java extension.");
	}
	FileCollection union = getProject().files();
	for (SourceSet sourceSet : javaPlugin.getSourceSets()) {
	union = union.plus(sourceSet.getAllJava());
	}
	target = union;
	}
	steps.replaceAll(step -> {
	if (LicenseHeaderStep.name().equals(step.getName())) {
	return step.filterByFile(LicenseHeaderStep.unsupportedJvmFilesFilter());
	} else {
	return step;
	}
	});
	super.setupTask(task);
	}

And then calls super, which is this:

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/FormatExtension.java

Lines 569 to 581 in 9843130

	/** Sets up a format task according to the values in this extension. */
	protected void setupTask(SpotlessTask task) {
	task.setPaddedCell(paddedCell);
	task.setEncoding(getEncoding().name());
	task.setExceptionPolicy(exceptionPolicy);
	if (targetExclude == null) {
	task.setTarget(target);
	} else {
	task.setTarget(target.minus(targetExclude));
	}
	task.setSteps(steps);
	task.setLineEndingsPolicy(getLineEndings().createPolicy(getProject().getProjectDir(), () -> task.target));
	}

So the exclusion should still happen. The reason it's failing is that targetExclude goes through parseTargets:

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/FormatExtension.java

Lines 152 to 154 in 9843130

	public void targetExclude(Object... targets) {
	this.targetExclude = parseTargets(targets);
	}

Which excludes .git and build/ and stuff like that:

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/FormatExtension.java

Lines 190 to 201 in 9843130

	List<String> excludes = new ArrayList<>();
	// no git
	excludes.add(".git");
	// no .gradle
	if (getProject() == getProject().getRootProject()) {
	excludes.add(".gradle");
	}
	// no build folders (flatInclude means that subproject might not be subfolders, see https://github.com/diffplug/spotless/issues/121)
	relativizeIfSubdir(excludes, dir, getProject().getBuildDir());
	for (Project subproject : getProject().getSubprojects()) {
	relativizeIfSubdir(excludes, dir, subproject.getBuildDir());
	}

This is good when we're parsing targets, e.g. target('**/*.ext') is much slower if you're slogging through the .git and build/ directories. But it's bad for cases like this.

By supplying a fileTree, you're bypassing the parsing, which is why the workaround works.

The complicated parse logic is pretty nasty, and has caused surprises before. But there are lots of wildcard **/*.ext inclusions in the wild, which are also nasty, but useful to our users, and those perform terribly without these exclusions.

Maybe a good way to handle this would be if those complicated exclusions only happened if target.startsWith('**/'). Otherwise, no fancy excludes, and the target string is passed straight.

[tbroyer]

How about passing a boolean to parseTargets and down to parseTarget to turn those special exclusions on/off? target would call parseTargets(true, targets), and targetExclude would call parseTargets(false, targets). Call that boolean addSpecialExcludes, or isInclude.

[nedtwigg]

That could work too. I think the first proposal is closer to the root cause, and the second proposal is a bit more of a patch.

The root cause is that the parsing logic assumes the user was lazy, and didn't really mean to include build or .git or subprojects. And the bug is that if you weren't lazy, it's really surprising that the tool stops doing what its docs say that it will do. So I think the rootcause fix is "only assume the user is lazy if the user actually was lazy".

Can we think of a String where the change I have proposed would be a problem for an existing target? The only one I can think of is subprojectdir/**/*.ext. Right now, we automatically exclude the subprojectdir/build, and after my proposed change, we wouldn't. That would be pretty strange usage though, to apply spotless to the root project, and then apply it specifically to a single subproject... It's easy to fix, and the change would fail by applying to too much rather than too little, so it "fails safe".

[tbroyer]

I agree for target("**/*.ext"), but still can't find a single argument for targetExclude("**/*.ext") not actually excluding the build dirs. Why would you want targetExclude("**/*.ext") to mean "exclude all files whose name ends in *.ext except those files in the build dir?

For example, kotlin includes all *.kt and *.kts but I've had issues with *.kts files "missing a top level declaration" (haven't had time to investigate). In case some of those files were similarly generated in the build dir, targetExclude("**/*.kts") would not actually exclude them.

[nedtwigg]

Why would you want targetExclude("**/*.ext") to mean "exclude all files whose name ends in *.ext except those files in the build dir?

For the hypothetical buildscript where I did targetExclude('**/*.ext'), what was the target()? It seems the result would be empty no matter what target was... But I see your point. Probably we should do both - and only exclude if isInclude=true and pattern.startsWith('**/').

[redhead]

is there any workaround how to currently exclude generated sources?

[tbroyer]

@redhead Yes, given in the original report above. Ctrl+F workaround.

(Sorry about the close/reopen, I mistapped on my phone)

[nedtwigg]

Resolved by #457 and released in 3.24.3.

[niels1voo]

I tried to use the OP's original non-working solution, thinking that #457 would make it work, but no luck.
targetExclude("build/*generated-sources/**/*.java")
results in
'Execution failed for task ':helium-data-model:spotlessJava'.

Spotless error! All target files must be within the project dir.
project dir: /Users/niels/repo/sx/helium/helium-main/helium-data-model
target: /Users/niels/repo/sx/helium/helium-main/build/helium-data-model/generated/sources/xjc/java/com/soundexchange/helium/distribution/iso20022/NameAndAddress3.java'

I'm using gradle spotlees plugin 6.4.2

This is a multi-module gradle build, where spotless is configured in the root build file.

[niels1voo]

I stepped through the build in a debugger and found that the problem lies in SpotlessTaskImpl.getOutputFile(). The call to FormatExtension.relativize returns null, which causes the exception to be thrown. relativize returns null because the targetExclude is not a child of the subproject directory in a multi-module build. So targetExclude can't be used to exclude generated sources from a multi-module build, because the build output is at the root directory, not the subproject directory.