Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make W3S JAVA OPTs configurable via advanced editor #76

Merged
merged 4 commits into from
Nov 27, 2023
Merged

Make W3S JAVA OPTs configurable via advanced editor #76

merged 4 commits into from
Nov 27, 2023

Conversation

alexpeterson91
Copy link
Member

@alexpeterson91 alexpeterson91 commented Nov 25, 2023 •
edited
Loading

Ever since the release of v1.0.8 The W3S no longer works for many users (likely those running higher numbers of validators) and the error appears to be related to OOM issues due to the JAVA OPTs being hardcoded in this package. In every other Java based package, Besu, Teku and all their forks for networks all have a separate field in the Advanced Config for Java OPTS.
Best solution for this appears to be making the JAVA opts configurable from the UI, and not have the values hardcoded in the docker file.
Due to the many reports of others with this issue on other networks than gnosis, i moved the upper limit of the max heap size to 10g from 4g and removed the lower limit to make it work. Before that i tested with 6g max. then 8g max then 8g max and a 1g min, same issues failing at same spots. felt like it was something else but then as a hail mary i tried to just throw 10g at it and remove the lower limit and now Ive got no issues running 93 keys Open to suggestions on what the upper limit should be but clearly 4g is not high enough given the current feedback and error reports and it appears theres a deeper issue as this upstream version shouldnt require more than double the max heap size to run the same amount of keys as the previous version did. Anyways this fix works and this param should be made editable in all signers. (PRs are already open)

Errors detailed below:

2023-11-25 12:59:37.577+00:00 | main | INFO  | Web3SignerApp | Web3Signer has started with args --key-store-path=/data/keyfiles,--http-listen-port=9000,--http-listen-host=0.0.0.0,--http-host-allowlist=web3signer.web3signer-gnosis.dappnode,brain.web3signer-gnosis.dappnode,validator.teku-gnosis.dappnode,--http-cors-origins=*,--metrics-enabled=true,--metrics-host,0.0.0.0,--metrics-port,9091,--metrics-host-allowlist=*,--idle-connection-timeout-seconds=900,eth2,--network=gnosis,--Xnetwork-capella-fork-epoch=648704,--slashing-protection-db-url=jdbc:postgresql://postgres.web3signer-gnosis.dappnode:5432/web3signer-gnosis,--slashing-protection-db-username=postgres,--slashing-protection-db-password=gnosis,--slashing-protection-pruning-enabled=true,--slashing-protection-pruning-epochs-to-keep=500,--key-manager-api-enabled=true
2023-11-25 12:59:37.589+00:00 | main | INFO  | Web3SignerApp | Version = web3signer/v23.11.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
2023-11-25 12:59:38.485+00:00 | main | INFO  | Eth2SubCommand | Network: gnosis
Spec Name: PHASE0, Fork Epoch: 0, First Slot: 0
Spec Name: ALTAIR, Fork Epoch: 512, First Slot: 8192
Spec Name: BELLATRIX, Fork Epoch: 385536, First Slot: 6168576
Spec Name: CAPELLA, Fork Epoch: 648704, First Slot: 10379264

2023-11-25 12:59:38.528+00:00 | main | INFO  | HikariDataSource | HikariPool-1 - Starting...
2023-11-25 12:59:38.656+00:00 | main | INFO  | HikariPool | HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@69cd1085
2023-11-25 12:59:38.657+00:00 | main | INFO  | HikariDataSource | HikariPool-1 - Start completed.
2023-11-25 12:59:38.845+00:00 | main | INFO  | HikariDataSource | HikariPool-2 - Starting...
2023-11-25 12:59:38.856+00:00 | main | INFO  | HikariPool | HikariPool-2 - Added connection org.postgresql.jdbc.PgConnection@6588b715
2023-11-25 12:59:38.857+00:00 | main | INFO  | HikariDataSource | HikariPool-2 - Start completed.
2023-11-25 12:59:39.009+00:00 | main | INFO  | MetricsHttpService | Starting metrics http service on 0.0.0.0:9091
2023-11-25 12:59:39.105+00:00 | vert.x-eventloop-thread-0 | INFO  | MetricsHttpService | Metrics service started and listening on 0.0.0.0:9091
2023-11-25 12:59:39.120+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Loading signer configuration metadata files from /data/keyfiles
2023-11-25 12:59:39.138+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Signer configuration metadata files read in memory 93 in 00:00:00.016
2023-11-25 12:59:39.208+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Converting signing metadata to Artifact Signer using parallel streams ...
2023-11-25 12:59:39.208+00:00 | ForkJoinPool.commonPool-worker-22 | INFO  | SignerLoader | 10 signing metadata processed
2023-11-25 12:59:39.208+00:00 | ForkJoinPool.commonPool-worker-29 | INFO  | SignerLoader | 20 signing metadata processed
2023-11-25 12:59:39.208+00:00 | ForkJoinPool.commonPool-worker-21 | INFO  | SignerLoader | 30 signing metadata processed
2023-11-25 12:59:46.475+00:00 | ForkJoinPool.commonPool-worker-15 | INFO  | SignerLoader | 40 signing metadata processed
2023-11-25 12:59:46.483+00:00 | ForkJoinPool.commonPool-worker-13 | INFO  | SignerLoader | 50 signing metadata processed
2023-11-25 12:59:46.578+00:00 | ForkJoinPool.commonPool-worker-17 | INFO  | SignerLoader | 60 signing metadata processed
2023-11-25 12:59:46.579+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Java heap space
2023-11-25 12:59:46.483+00:00 | main | ERROR | Runner | Error loading signers
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
	at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
	at tech.pegasys.web3signer.core.Runner.run(Runner.java:121) [web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.core.Eth2Runner.run(Eth2Runner.java:401) [web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.commandline.subcommands.ModeSubCommand.run(ModeSubCommand.java:33) [web3signer-commandline-23.11.0.jar:23.11.0]
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine.access$1300(CommandLine.java:145) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine.execute(CommandLine.java:2078) [picocli-4.6.2.jar:4.6.2]
	at tech.pegasys.web3signer.commandline.CommandlineParser.parseCommandLine(CommandlineParser.java:85) [web3signer-commandline-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.Web3SignerApp.executeWithEnvironment(Web3SignerApp.java:50) [web3signer-app-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.Web3SignerApp.main(Web3SignerApp.java:35) [web3signer-app-23.11.0.jar:23.11.0]
Caused by: java.lang.OutOfMemoryError
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
	at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source) ~[?:?]
	at java.util.concurrent.ForkJoinTask.reportException(Unknown Source) ~[?:?]
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
	at tech.pegasys.web3signer.signing.config.SignerLoader.mapToArtifactSigner(SignerLoader.java:241) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.signing.config.SignerLoader.mapMetadataToArtifactSigner(SignerLoader.java:208) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.signing.config.SignerLoader.load(SignerLoader.java:90) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.core.Eth2Runner.loadSignersFromKeyConfigFiles(Eth2Runner.java:304) ~[web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.core.Eth2Runner.lambda$createArtifactSignerProvider$2(Eth2Runner.java:259) ~[web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.signing.config.DefaultArtifactSignerProvider.lambda$load$1(DefaultArtifactSignerProvider.java:51) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
2023-11-25 12:59:46.648+00:00 | main | INFO  | Runner | Web3Signer has started with TLS disabled, and ready to handle signing requests on 0.0.0.0:9000
2023-11-25 12:59:46.982+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-4-thread-1"
2023-11-25 12:59:52.250+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:52.250+00:00 | ForkJoinPool.commonPool-worker-15 | INFO  | SignerLoader | 70 signing metadata processed
2023-11-25 12:59:52.256+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
Exception in thread "vertx-blocked-thread-checker" java.lang.OutOfMemoryError: Java heap space
2023-11-25 12:59:52.263+00:00 | ForkJoinPool.commonPool-worker-28 | INFO  | SignerLoader | 80 signing metadata processed
2023-11-25 12:59:52.380+00:00 | ForkJoinPool.commonPool-worker-30 | INFO  | SignerLoader | 90 signing metadata processed
2023-11-25 12:59:52.391+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:52.392+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:52.394+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:52.403+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:52.414+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:52.416+00:00 | vert.x-eventloop-thread-0 | ERROR | ConnectionBase | Could not initialize class io.netty.handler.codec.compression.ZlibCodecFactory
2023-11-25 12:59:57.212+00:00 | vert.x-acceptor-thread-0 | WARN  | SingleThreadEventExecutor | Unexpected exception from an event executor: 
java.lang.OutOfMemoryError: Java heap space
2023-11-25 12:59:57.822+00:00 | ForkJoinPool.commonPool-worker-1 | INFO  | BLS | BLS: loaded BLST library
2023-11-25 19:45:01.783+00:00 | main | INFO  | Web3SignerApp | Web3Signer has started with args --key-store-path=/data/keyfiles,--http-listen-port=9000,--http-listen-host=0.0.0.0,--http-host-allowlist=web3signer.web3signer-gnosis.dappnode,brain.web3signer-gnosis.dappnode,validator.teku-gnosis.dappnode,--http-cors-origins=*,--metrics-enabled=true,--metrics-host,0.0.0.0,--metrics-port,9091,--metrics-host-allowlist=*,--idle-connection-timeout-seconds=900,eth2,--network=gnosis,--Xnetwork-capella-fork-epoch=648704,--slashing-protection-db-url=jdbc:postgresql://postgres.web3signer-gnosis.dappnode:5432/web3signer-gnosis,--slashing-protection-db-username=postgres,--slashing-protection-db-password=gnosis,--slashing-protection-pruning-enabled=true,--slashing-protection-pruning-epochs-to-keep=500,--key-manager-api-enabled=true
2023-11-25 19:45:01.795+00:00 | main | INFO  | Web3SignerApp | Version = web3signer/v23.11.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
2023-11-25 19:45:02.720+00:00 | main | INFO  | Eth2SubCommand | Network: gnosis
Spec Name: PHASE0, Fork Epoch: 0, First Slot: 0
Spec Name: ALTAIR, Fork Epoch: 512, First Slot: 8192
Spec Name: BELLATRIX, Fork Epoch: 385536, First Slot: 6168576
Spec Name: CAPELLA, Fork Epoch: 648704, First Slot: 10379264

2023-11-25 19:45:02.790+00:00 | main | INFO  | HikariDataSource | HikariPool-1 - Starting...
2023-11-25 19:45:02.918+00:00 | main | INFO  | HikariPool | HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@7c2924d7
2023-11-25 19:45:02.919+00:00 | main | INFO  | HikariDataSource | HikariPool-1 - Start completed.
2023-11-25 19:45:03.118+00:00 | main | INFO  | HikariDataSource | HikariPool-2 - Starting...
2023-11-25 19:45:03.136+00:00 | main | INFO  | HikariPool | HikariPool-2 - Added connection org.postgresql.jdbc.PgConnection@6b357eb6
2023-11-25 19:45:03.137+00:00 | main | INFO  | HikariDataSource | HikariPool-2 - Start completed.
2023-11-25 19:45:03.372+00:00 | main | INFO  | MetricsHttpService | Starting metrics http service on 0.0.0.0:9091
2023-11-25 19:45:03.499+00:00 | vert.x-eventloop-thread-0 | INFO  | MetricsHttpService | Metrics service started and listening on 0.0.0.0:9091
2023-11-25 19:45:03.523+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Loading signer configuration metadata files from /data/keyfiles
2023-11-25 19:45:03.551+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Signer configuration metadata files read in memory 93 in 00:00:00.026
2023-11-25 19:45:03.619+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Converting signing metadata to Artifact Signer using parallel streams ...
2023-11-25 19:45:03.620+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | SignerLoader | 10 signing metadata processed
2023-11-25 19:45:03.620+00:00 | ForkJoinPool.commonPool-worker-16 | INFO  | SignerLoader | 30 signing metadata processed
2023-11-25 19:45:03.620+00:00 | ForkJoinPool.commonPool-worker-32 | INFO  | SignerLoader | 20 signing metadata processed

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "vertx-blocked-thread-checker"
2023-11-25 19:45:11.285+00:00 | ForkJoinPool.commonPool-worker-20 | INFO  | SignerLoader | 40 signing metadata processed
2023-11-25 19:45:11.305+00:00 | ForkJoinPool.commonPool-worker-30 | INFO  | SignerLoader | 50 signing metadata processed
2023-11-25 19:45:11.386+00:00 | ForkJoinPool.commonPool-worker-14 | INFO  | SignerLoader | 60 signing metadata processed
2023-11-25 19:45:11.389+00:00 | ForkJoinPool.commonPool-worker-1 | INFO  | SignerLoader | 70 signing metadata processed
2023-11-25 19:45:11.314+00:00 | main | ERROR | Runner | Error loading signers
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
	at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
	at tech.pegasys.web3signer.core.Runner.run(Runner.java:121) [web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.core.Eth2Runner.run(Eth2Runner.java:401) [web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.commandline.subcommands.ModeSubCommand.run(ModeSubCommand.java:33) [web3signer-commandline-23.11.0.jar:23.11.0]
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine.access$1300(CommandLine.java:145) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316) [picocli-4.6.2.jar:4.6.2]
	at picocli.CommandLine.execute(CommandLine.java:2078) [picocli-4.6.2.jar:4.6.2]
	at tech.pegasys.web3signer.commandline.CommandlineParser.parseCommandLine(CommandlineParser.java:85) [web3signer-commandline-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.Web3SignerApp.executeWithEnvironment(Web3SignerApp.java:50) [web3signer-app-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.Web3SignerApp.main(Web3SignerApp.java:35) [web3signer-app-23.11.0.jar:23.11.0]
Caused by: java.lang.OutOfMemoryError
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
	at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source) ~[?:?]
	at java.util.concurrent.ForkJoinTask.reportException(Unknown Source) ~[?:?]
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
	at tech.pegasys.web3signer.signing.config.SignerLoader.mapToArtifactSigner(SignerLoader.java:241) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.signing.config.SignerLoader.mapMetadataToArtifactSigner(SignerLoader.java:208) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.signing.config.SignerLoader.load(SignerLoader.java:90) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.core.Eth2Runner.loadSignersFromKeyConfigFiles(Eth2Runner.java:304) ~[web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.core.Eth2Runner.lambda$createArtifactSignerProvider$2(Eth2Runner.java:259) ~[web3signer-core-23.11.0.jar:23.11.0]
	at tech.pegasys.web3signer.signing.config.DefaultArtifactSignerProvider.lambda$load$1(DefaultArtifactSignerProvider.java:51) ~[web3signer-signing-23.11.0.jar:23.11.0]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
2023-11-25 19:45:11.732+00:00 | main | INFO  | Runner | Web3Signer has started with TLS disabled, and ready to handle signing requests on 0.0.0.0:9000
2023-11-25 19:45:17.619+00:00 | ForkJoinPool.commonPool-worker-26 | INFO  | SignerLoader | 80 signing metadata processed
2023-11-25 19:45:18.372+00:00 | ForkJoinPool.commonPool-worker-20 | INFO  | SignerLoader | 90 signing metadata processed
Exception in thread "vert.x-eventloop-thread-0" java.lang.OutOfMemoryError: Java heap space
2023-11-25 19:45:23.161+00:00 | ForkJoinPool.commonPool-worker-28 | INFO  | BLS | BLS: loaded BLST library
2023-11-25 19:45:23.598+00:00 | vert.x-acceptor-thread-0 | WARN  | AbstractChannel | Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xa1d86567, L:/127.0.0.1:9000 - R:/127.0.0.1:38700]
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:89) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:83) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.vertx.core.net.impl.VertxEventLoopGroup.register(VertxEventLoopGroup.java:53) [vertx-core-4.4.4.jar:4.4.4]
	at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:215) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:553) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
2023-11-25 19:45:23.599+00:00 | vert.x-acceptor-thread-0 | ERROR | rejectedExecution | Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:215) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:553) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
2023-11-25 19:45:28.649+00:00 | vert.x-acceptor-thread-0 | WARN  | AbstractChannel | Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x7d7146d6, L:/127.0.0.1:9000 - R:/127.0.0.1:49190]
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
	```

@alexpeterson91 alexpeterson91 added the P0 Top Priority, Needs Immediate attention label Nov 25, 2023
This was referenced Nov 25, 2023
Merged
Merged
Merged
Merged
@alexpeterson91
Merge pull request #1 from Vis-Nova-Labs/test-workflow
787014d 
bump workflow checkout to v3 and add workflow dispatch
@alexpeterson91
Copy link
Member Author

the last commit here 787014d was not intended to merge into this PR, it was merged on my package to generate a release on my CI but i forgot that it would edit this PR as well since its a PR from my master branch not a new branch as I did with every other PR. IDK if i should revert it or just close this PR and open a new PR without from a new branch, so any other edits i make trying to tweak this build dont accidentally get pulled into this PR as well.

Converting to a Draft now since ive been having so much trouble getting a non officially signed DNP installed so far and this PR got jumbled by pulling changes from my master branch in another repo. but this remains high priority.

@alexpeterson91 alexpeterson91 marked this pull request as draft November 26, 2023 00:56
@alexpeterson91
Copy link
Member Author

UPDATE: build resolved itself, same hash was installed without incident but still had same issues even doubling the Max heap size to 8g from 4g. was going to give up but just tested no minimum heap size and a 10g max. all working as soon as I finished. Definitely need to make this a confgurable field as some power users running lots of keys may run into similar issues, but it appears unless this upstream was supposed to need more than double the max heap size as the previous version for the same amount of validators, there's something else wrong, perhaps a memory leak somwhere but regardless I was able to resolve this issue for me with this PR and JAVA_OPTS set to -Xmx10g

@alexpeterson91
bump max heap to 10g from 6
b050838 
bumping max heap size to 10g testing with lower values, 6g, 8g no improvement but when i made the max 10 and removed the low end everything started working for me again.
@alexpeterson91
Copy link
Member Author

alexpeterson91 commented Nov 26, 2023 •
edited
Loading

/ipfs/QmUK42NVuP4tVT6UbpNK1WnyYJeiNZ3eL1wM67QvdvMSmd
from https://github.com/Vis-Nova-Labs/DAppNodePackage-web3signer-gnosis/releases/tag/v1.0.9 commit tag Vis-Nova-Labs@787014d
is the test image i have been using where the heap size is configurable and messing with settings made the latest upstream work, though this version did not work out of the box for me and i needed to increase the heap size beyond the 6g max to 10g to make it work so i updated the max heap to 10g in the commit just above to match what worked in practice for me. I dont think it will be the same numbers for everyone since not everyone experiences this issue at least thats what discord looks like. but many do.

@alexpeterson91 alexpeterson91 marked this pull request as ready for review November 26, 2023 02:10
@pablomendezroyo pablomendezroyo merged commit e7703e7 into dappnode:master Nov 27, 2023
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pablomendezroyo pablomendezroyo pablomendezroyo approved these changes

@3alpha 3alpha Awaiting requested review from 3alpha

Assignees
No one assigned
Labels
P0 Top Priority, Needs Immediate attention
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alexpeterson91 @pablomendezroyo