"Generating Effective POM" renders the PC unusable on startup and causes Maven releases to fail

[anthonyvdotbe]

Describe the bug
My workspace has 3 Maven projects, with in total 17 modules.

Sometimes when opening VS Code (it doesn't happen on each startup, and I haven't been able to find a logic in it yet. For example, it has already occurred, even though none of the POMs had changed since the previous occurrence), it shows "Generating Effective POM" and the PC becomes effectively unusable for several minutes (to the point where VS Code itself says the windows is no longer responding, see screenshot below).

Another manifestation of this issue, is when doing a release (using Maven's release plugin) of a larger project (1 of the projects has 11 submodules): since all the POMs are changed multiple times in a short period, VS Code goes nuts trying to keep up with generating effective POMs, which causes the build to fail.

There are some similar closed issues related to optimization of effective POM generation, namely #173 and #182

To Reproduce
I can't readily share my projects, but I believe it's a simple combination of large effective POMs (mine are 1000+ lines) and the fact that they're all being generated in parallel.

The PC becomes unusable on VS Code startup, and releases fail.

Expected behavior
The PC remains usable at all times, and releases don't fail.
Some ideas:

• make sure to regenerate effective POMs only when there's been an actual change, and that change might have an impact (e.g. if the project's description was changed, there's no need to regenerate)
• generate effective POMs sequentially
• detect when a Maven release is going on, and avoid generating effective POMs during that time
• provide a folder-specific setting to have VS Code ask me each time it wants to generate an effective POM

Environments (please complete the following information as much as possible):

• OS: Windows 10 1809
• VS Code version: 1.33.1
• Extension version: 0.16.2

Screenshots

[Eskibear]

As we know, the root cause is it's time/CPU consuming to calculate effective-pom.

make sure to regenerate effective POMs only when there's been an actual change

Not like a language server, this extension only has knowledge of file changes, and the effective-pom is calculated by launching a background process mvn help:effective-pom. Currently it seems hard to realize that "if the project's description was changed, there's no need to regenerate"

generate effective POMs sequentially

Yes, it is exactly what the extension does. On activation, in order to know info of plugins and plugin details, lots of background Maven processes are required. To avoid high CPU overload, the tasks are scheduled one by one sequentially. But whenever pom.xml changes, the effective-pom re-calculation seems to be unscheduled, I need to double check on it.

provide a folder-specific setting to have VS Code ask me each time it wants to generate an effective POM

This is the a compromising approach. If in the end we cannot think of a better way of handling effective-pom, we might have to adopt this.

[anthonyvdotbe]

Thanks for your feedback. I thought about this some more and came up with the following:

The extension keeps some kind of database with, for each pom.xml file: the file path, the file content, a hash of the file content, and the effective POM (both the raw data, and the parsed resultant JSON object).

On startup, or whenever a change is detected, the extension calculates the hash of the pom.xml file. If the hash matches the one in the database, nothing is done. However, if it's different, the extension will:

• do an XML diff (e.g. using diff-js-xml) of the file content
• check if one of the changes is in an element that matters (e.g. <parent>, <pluginManagement>, <plugins>, <dependencyManagement>, <dependencies>)
  • if not, it will update the file path/content/hash, but not regenerate the effective POM
  • if yes, it will regenerate the effective POM, and update everything

With this, the extension would typically not have to generate any effective POMs at all on startup. For the issue with doing Maven releases: when doing the XML diff, the extension could check whether the version ends with "-SNAPSHOT": if not, this most likely means a release is ongoing, and the extension should not regenerate the effective POM.

What do you think?

(Another idea I had is taking advantage of Maven inheritance: the POMs of my projects and their child modules are typically very small, since most configuration is in a "master POM". In other words: 95% of the effective POMs is the same for all modules. So if there would be a way to calculate the effective POM just once for the master POM, and then merge that with each of the modules' POMs, it would be a lot more efficient. But I don't see how to implement this in practice)

[Eskibear]

Caching pom.xml and its effective-pom is an acceptable option. As we need to diff XMLs (before and after the change), it's not enough to cache the hash.

• Hash of pom.xml's absolute path can be the key, and we cache the content of pom.xml
• Cache folder can be workspaceStorage folder.
• Whenever changes detected, parse XMLs into objects, do a deep-diff

the extension could check whether the version ends with "-SNAPSHOT": if not, this most likely means a release is ongoing

I don't think it's a solid conclusion. If users don't follow this convention to append "-SNAPSHOT" during development, the effective pom would never get updated.

PS
In fact it's hard to keep the effective-pom updated, even now. Whenever a project's pom changes, all its child projects should re-calculate their effective poms. But at the moment it would be time consuming, without approaches to update effective-pom incrementally.

[anthonyvdotbe]

I don't think it's a solid conclusion. If users don't follow this convention to append "-SNAPSHOT" during development, the effective pom would never get updated.

Yes, I agree this is rather fragile and not a priority at the moment. One way to improve it, would be to verify that the maven-release-plugin is specified under build.plugins in the effective POM.

In fact it's hard to keep the effective-pom updated, even now. Whenever a project's pom changes, all its child projects should re-calculate their effective poms. But at the moment it would be time consuming, without approaches to update effective-pom incrementally.

Can I find an overview somewhere of the POM's sections that are actually used by the extension? Because some elements, like the configuration element of a plugin, are a lot harder to merge & update incrementally than others.

[Eskibear]

Can I find an overview somewhere of the POM's sections that are actually used by the extension?

The extension doesn't calculate effective-pom by itself. It simply calls external command mvn help:effective-pom to generate effective-pom to a temp file, and reads it.

Several users have also complained about the performance issue of effective-pom auto-generating. Now I think before we come up to a better solution, in the coming version we can first disable the feature by default, and allow users to turn it on by some settings.

[blassmegod]

An interesting fact is that each time I open an workspace/folder in which I have Maven projects the plugin is trying to regenerate the effective pom files, but by checking the folder no file is actually created (the command is fine, if I'm running it from a console the file is created).

Spawn {"command":"mvn","args":["help:effective-pom","-Doutput=\"c:\\Users\\username\\AppData\\Roaming\\Code\\User\\workspaceStorage\\a694ce1656fd223a84f33e04410dfef3\\vscjava.vscode-maven\\75c48b9e6171265e041c3b2e02026d6a\"","-Djdk.util.zip.ensureTrailingSlash=false","--offline","-f","\"c:\\Dev\\git\\work12\\java\\maven_module-parent\\maven-module_x1\\pom.xml\""]}
[INFO] Scanning for projects...
...
[INFO] Effective-POM written to: c:\Users\username\AppData\Roaming\Code\User\workspaceStorage\a694ce1656fd223a84f33e04410dfef3\vscjava.vscode-maven\75c48b9e6171265e041c3b2e02026d6a
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.015 s
[INFO] Finished at: 2020-03-19T16:06:40+02:00 

So, after the above was executed the folder "c:\Users\username\AppData\Roaming\Code\User\workspaceStorage\a694ce1656fd223a84f33e04410dfef3\vscjava.vscode-maven" was empty. Maybe this is the reason that each time I'm reopening a folder the Maven plugin kicks in and eats the processor by trying to "re"create the effective poms.

[Eskibear]

It was deleted after the extension read the content. We can preserve the file as a cache though, but you still have to re-run the process to update effective pom whenever you modify your pom.xml

[blassmegod]

It was deleted after the extension read the content. We can preserve the file as a cache though, but you still have to re-run the process to update effective pom whenever you modify your pom.xml

Indeed, but for example even if I don’t do modifications to pom.xml I don’t see why it should redo it each time I open same project. Maybe the cache can be recreated only when nothing is in the cache or when a checksum is different. It takes on larger projects more than 4 min to complete during which the processor is overloaded.

[Eskibear]

It takes on larger projects more than 4 min to complete during which the processor is overloaded.

That's why we turn autoUpdateEffectivePOM off by default. Lacking of a simple way to incremantally update effective pom, the cache becomes dirty whenever you modify pom.xml.

What I can think now is:

1. keep the cache file there after calculating effective pom
2. re-calculate it only if you explicit tell it to, e.g. refresh the project

[Eskibear]

It's optimized with proper cache file.

See:
#579 (comment)
#502