Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lots of InvocationTargetException after web app is deployed #3384

Closed
vladak opened this issue Dec 2, 2020 · 4 comments
Closed

lots of InvocationTargetException after web app is deployed #3384

vladak opened this issue Dec 2, 2020 · 4 comments
Assignees
@vladak
Labels
webapp web application

Comments

@vladak
Copy link
Member

vladak commented Dec 2, 2020

Since some time every deploy of the web application is followed by tens of exceptions like this one:

24-Nov-2020 22:51:59.517 WARNING [localhost-startStop-1] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/var/tomcat8/webapps/source/WEB-INF/lib/annotations-19.0.0.jar] from class
loader hierarchy
        java.io.IOException: java.lang.reflect.InvocationTargetException
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:205)
                at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
                at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
                at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:385)
                at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:320)
                at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:283)
                at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:234)
                at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1916)
                at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1130)
                at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:779)
                at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
                at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5053)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
                at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1125)
                at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1858)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at java.base/java.lang.Thread.run(Thread.java:834)
        Caused by: java.lang.reflect.InvocationTargetException
                at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
                at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:202)
                ... 23 more
        Caused by: java.nio.file.NoSuchFileException: /var/tomcat8/webapps/source/WEB-INF/lib/annotations-19.0.0.jar
                at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
                at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
                at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
                at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
                at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1225)
                at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:727)
                at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:844)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:247)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
                at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
                ... 27 more

At first I thought that this was related to the Java 11 switch over however it might be something else entirely per #2857 (comment).
@vladak vladak mentioned this issue Dec 2, 2020
Open
@vladak vladak added the webapp web application label Dec 2, 2020
@vladak
Copy link
Member Author

vladak commented Dec 2, 2020

Some of the exceptions are thrown because of the double lib path component. For example:

24-Nov-2020 22:52:09.671 WARNING [localhost-startStop-1] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/var/tomcat8/webapps/source/WEB-INF/lib/lib/lucene-queryparser-8.6.2.jar] 
from classloader hierarchy
        java.io.IOException: java.lang.reflect.InvocationTargetException
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:205)
                at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
                at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
                at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:385)
                at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:320)
                at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:283)
                at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:234)
                at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262)
                at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104)
                at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:101)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5144)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
                at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1125)
                at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1858)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at java.base/java.lang.Thread.run(Thread.java:834)
        Caused by: java.lang.reflect.InvocationTargetException
                at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
                at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:202)
                ... 21 more
        Caused by: java.nio.file.NoSuchFileException: /var/tomcat8/webapps/source/WEB-INF/lib/lib/lucene-queryparser-8.6.2.jar
                at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
                at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
                at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
                at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
                at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1225)
                at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:727)
                at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:844)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:247)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
                at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
                ... 25 more

The lucene-queryparser-8.6.2.jar file is present as /var/tomcat8/webapps/source/WEB-INF/lib/lucene-queryparser-8.6.2.jar, not as /var/tomcat8/webapps/source/WEB-INF/lib/lib/lucene-queryparser-8.6.2.jar.

@vladak
Copy link
Member Author

vladak commented Dec 2, 2020

The lib/lib is where the jars are delivered in the release tar ball:

# gtar tvf /var/tmp/opengrok-1.5.9.tar.gz | grep 'lib\/lib' | head
-rw-r--r-- runner/docker    52132 2020-10-27 17:02 opengrok-1.5.9/lib/lib/lucene-memory-8.6.2.jar
-rw-r--r-- runner/docker    29779 2020-10-27 17:02 opengrok-1.5.9/lib/lib/LatencyUtils-2.0.3.jar
-rw-r--r-- runner/docker   131590 2020-10-27 17:02 opengrok-1.5.9/lib/lib/hk2-utils-2.6.1.jar
-rw-r--r-- runner/docker   927734 2020-10-27 17:02 opengrok-1.5.9/lib/lib/jersey-server-2.30.1.jar
-rw-r--r-- runner/docker    69409 2020-10-27 17:02 opengrok-1.5.9/lib/lib/activation-1.1.1.jar
-rw-r--r-- runner/docker   125632 2020-10-27 17:02 opengrok-1.5.9/lib/lib/jaxb-api-2.3.0.jar
-rw-r--r-- runner/docker  1685426 2020-10-27 17:02 opengrok-1.5.9/lib/lib/lucene-analyzers-common-8.6.2.jar
-rw-r--r-- runner/docker  1155887 2020-10-27 17:02 opengrok-1.5.9/lib/lib/hibernate-validator-6.0.17.Final.jar
-rw-r--r-- runner/docker  1167646 2020-10-27 17:02 opengrok-1.5.9/lib/lib/jersey-common-2.30.1.jar
-rw-r--r-- runner/docker    91930 2020-10-27 17:02 opengrok-1.5.9/lib/lib/jakarta.validation-api-2.0.2.jar

@vladak
Copy link
Member Author

vladak commented Dec 2, 2020 •
edited
Loading

Another interesting case:

24-Nov-2020 22:52:09.711 WARNING [localhost-startStop-1] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/var/tomcat8/webapps/source/WEB-INF/lib/lib/activation-1.1.1.jar] from cla
ssloader hierarchy
        java.io.IOException: java.lang.reflect.InvocationTargetException
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:205)
                at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
                at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
                at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:385)
                at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:320)
                at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:283)
                at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:234)
                at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262)
                at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104)
                at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:101)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5144)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
                at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1125)
                at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1858)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at java.base/java.lang.Thread.run(Thread.java:834)
        Caused by: java.lang.reflect.InvocationTargetException
                at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
                at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:202)
                ... 21 more
        Caused by: java.nio.file.NoSuchFileException: /var/tomcat8/webapps/source/WEB-INF/lib/lib/activation-1.1.1.jar
                at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
                at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
                at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
                at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
                at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1225)
                at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:727)
                at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:844)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:247)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
                at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
                ... 25 more

The actual file is /var/tomcat8/webapps/source/WEB-INF/lib/activation-1.1.jar , i.e. this time the problem is not only the double lib path component but also the version - 1.1.1 vs. 1.1.

This seems to be related to how the distribution is constructed. Looking at the fully built repository:

$ find . -type f -name 'activation*.jar'
./opengrok-web/target/source/WEB-INF/lib/activation-1.1.jar
./distribution/target/dist/activation-1.1.1.jar

The version number discrepancy stems from different dependencies in opengrok-web and opengrok-indexer modules. While the former depends on 1.1, the latter depends on 1.1.1.

@vladak vladak mentioned this issue Dec 2, 2020
Merged
@vladak
Copy link
Member Author

vladak commented Dec 2, 2020 •
edited
Loading

Looking at the META-INF/MANIFEST.MF file inside the opengrok.jar it contains Class-Path with some 64 entries (such a weird format, breaking lines in the middle of a path):

Manifest-Version: 1.0
Created-By: Apache Maven 3.6.1
Built-By: vkotal
Build-Jdk: 11.0.7
Class-Path: lib/bcel-6.4.1.jar lib/commons-lang3-3.9.jar lib/commons-com
 press-1.19.jar lib/lucene-core-8.6.2.jar lib/lucene-analyzers-common-8.
 6.2.jar lib/lucene-highlighter-8.6.2.jar lib/lucene-join-8.6.2.jar lib/
 lucene-memory-8.6.2.jar lib/lucene-queries-8.6.2.jar lib/lucene-querypa
...

This corresponds to the files in the exceptions thrown during web app deploy (the parsed output has some 58 files for the lib/lib path).

It seems this has to do with opengrok.jar being included in the source.war archive. What is likely happening is when the Tomcat Jar Scanner (https://tomcat.apache.org/tomcat-8.5-doc/config/jar-scanner.html) is scanning the jar files inside the extracted source.war it processes the opengrok.jar as well, trying to find its dependencies using the class path specified in its META-INF/MANIFEST.MF file. Given that the class paths are relative and have the lib/ prefix and the opengrok.jar is under WEB-INF/lib/ directory, it will try to find them under the WEB-INF/lib/lib directory.

As for the lib/lib I seem to remember there used to be lib symlink inside the distribution somewhere pointing to ../lib which probably provided a workaround.

There has to be a better way how to deal with this.

vladak pushed a commit to vladak/OpenGrok that referenced this issue Dec 2, 2020
flatten library directory
8a36b34 
fixes oracle#3384
@vladak vladak self-assigned this Dec 2, 2020
@vladak vladak mentioned this issue Dec 2, 2020
Open
@vladak vladak changed the title InvocationTargetException after web app is deployed lots of InvocationTargetException after web app is deployed Dec 2, 2020
@vladak vladak closed this as completed in 6356683 Dec 2, 2020
@vladak vladak mentioned this issue Feb 16, 2021
Closed
vladak pushed a commit that referenced this issue Apr 9, 2021
flatten library directory
ffff4dd 
fixes #3384
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vladak vladak

Labels
webapp web application
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vladak