Does Kurtosis-CDK have support for the real prover for testing?

[tiagorvmartins]

System information

Ubuntu 22.04

Commit id

af3798e

Description & steps to reproduce

Is there support for running the real prover with the Kurtosis-CDK stack?
Found the flag zkevm_use_real_verifier in

kurtosis-cdk/input_parser.star

Line 239 in 2edcc2f

"zkevm_use_real_verifier": False,

However when changing to true it doesn't deploy the real prover, it actually just uses a stateless executor, is this expected?
Currently running the stack (fork13) with cdk validium option and erigon sequencer in strict mode.

Also can't see the memory bumping (more than ~10gb), was actually expecting a lot more memory comsuption (around 300GB+) as the comment describes in:

kurtosis-cdk/input_parser.star

Line 238 in 2edcc2f

# Note: This will require a lot of memory to run!

The reason, I believe, its because of this if condition:

kurtosis-cdk/cdk_central_environment.star

Line 30 in 2edcc2f

not args["zkevm_use_real_verifier"]

However would like to get more insights on this, since I need to test the real prover using kurtosis-cdk framework.

Thank you in advance!

Desired behavior

Running kurtosis-cdk with the real prover.

What is the severity of this bug?

Painful; This is causing significant friction in my workflow.

[doutv]

I also want to test it.
I think we need to start a real prover separately.
See zkevm-prover config https://github.com/0xPolygonHermez/zkevm-prover/blob/main/src/config/README.md

[praetoriansentry]

Yes, I test with a real prover. @doutv is right.

When I do this, I set the zkevm_use_real_veriifer flag like what you're doing. Then I bring up the full enclave. At this point, you should have everything running except the prover. Then I would start a prover separately using a binary or docker or whatever. You'll need to configure the prover to connect to the aggregator port that will be exposed by the CDK.

[tiagorvmartins]

Beside turning the flag 'zkevm_use_real_verifer' to True and configuring the zkevm-prover outside of the kurtosis enclave while connected to the aggregator (cdk node of kurtosis enclave) and postgres prover db (also from kurtosis enclave), I see some errors stating:

zkError: ExecutorServiceImpl::ProcessStatelessBatchV2() called with config.hashDBSingleton=true

Not only this, but the zkevm_verifiedBatchNumber is not going up, only the zkevm_batchNumber and zkevm_virtualBatchNumber, I believe this is not expected.

At this moment, the first batch sequenced was 15h ago and still wasn't verified, wondering if I have something badly configured (related to the error shown above) or something else is off / requires more time to be verified?!

Let me know,
Thank you!

[doutv]

Same as you. The prover is running but zkevm_verifiedBatchNumber is not going up.

@praetoriansentry It's better to create a markdown document to show how to connect a real prover to kurtosis-cdk

[seromenho]

I have identified the source of the error: it originates from the executor_service.cpp https://github.com/0xPolygonHermez/zkevm-prover/blob/v9.0.0-RC3/src/service/executor/executor_service.cpp#L1724-L1731

However, I’m confused by the presence of this check, as from what I understand by reading the documentation and the prover configuration file, in a production environment the prover is expected to be configured with databaseURL set and stateManager enabled. Specifically in the trusted-node template configuration https://github.com/0xPolygon/kurtosis-cdk/blob/main/templates/trusted-node/prover-config.json#L167-L179
The erroring check seems more applicable to the stateless executor than the prover.

What should the correct configuration look like for a production prover setup?

Additionally, I am using the version of the components corresponding to fork ID 13 from the CDK version matrix.

Thank you

[tiagorvmartins]

Any news on this? Thank you
@doutv @praetoriansentry

[doutv]

I haven't figured out the reason and I switch to other things. perhaps you can check aggregator log.