[Feature request] Do not batch/aggregate when basefee is lower than "BatchPreCommitAboveBaseFee/AggregateAboveBaseFee"

[f8-ptrk]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

version
Daemon:  1.13.2-rc4+calibnet+git.e05fdf892+api1.4.0
Local: lotus version 1.13.2-rc4+calibnet+git.e05fdf892

Daemon:  1.13.2-rc4+calibnet+git.e05fdf892+api1.2.0
Local: lotus-miner version 1.13.2-rc4+calibnet+git.e05fdf892

Describe the Bug

We have the following settings in the config.toml for the lotus-miner

[...]
BatchPreCommits = true
MaxPreCommitBatch = 16
PreCommitBatchWait = "8h0m0s"
#BatchPreCommitAboveBaseFee = "0.00000000032 FIL"
[...]

The calibnet baseFee is at 100 aFIL - the minimal possible value.

Now we start sealing sectors and the lotus-miner gathers 16 precommits before trying to submit the batch - with the baseFee being way below the configured #BatchPreCommitAboveBaseFee = "0.00000000032 FIL" (assuming this defaults to the default value) it sends 16 individual messages.

This comes unexpected. We expected the lotus-miner to send the message on the spot if the baseFee is below the baseFee threshold and not gather them until MaxPreCommitBatch is reached.

We are not sure if this is actually intended behavior - the expectations out there seem to be that the messages get send on the spot (see minerX3 chan on slack: https://filecoinproject.slack.com/archives/C02G9S9T91Q/p1640215123223200). If this is intended what is the reasoning behind it? We doubt it will be cheaper to wait until the batch is full instead of sending the precommit messages on the spot as soon as the precommit finished.

We are testing the commit aggregation now. Will update this issue if the same thing occurs there.

Logging Information

not needed - it's a logic bug most likely. if logs are needed let me know.

Repo Steps

See bug description.

[f8-ptrk]

without having restarted the lotus-miner:

• uncommenting the #BatchPreCommitAboveBaseFee = "0.00000000032 FIL" doesn't help

[f8-ptrk]

without having restarted the lotus-miner, changed config.toml:

before:

[...]
FinalizeEarly = true
AggregateCommits = false
#MinCommitBatch = 4
#MaxCommitBatch = 819
#CommitBatchWait = "24h0m0s"
#AggregateAboveBaseFee = "0.00000000032 FIL"
[...]

changed to:

[...]
FinalizeEarly = true
AggregateCommits = true
MinCommitBatch = 8
MaxCommitBatch = 16
CommitBatchWait = "8h0m0s"
#AggregateAboveBaseFee = "0.00000000032 FIL"
[...]

at the same 100aFIL baseFee leads to commit aggregation.

[f8-ptrk]

manual --publish-now's obviously leads to:

lotus-miner sectors batching commit --publish-now


Batch 0:
        Message: bafy2bzaceb2aebwsotwyfngxznuncw2mlrqe4shu5hbxtvnvbmmsnt6dnpusm
        Sectors:
                198     OK
Batch 1:
        Message: bafy2bzacedfxrl64czmylkmoidozelpgpqhydzmmazw4cklanjlfz2aierdmc
        Sectors:
                196     OK
Batch 2:
        Message: bafy2bzacebbj6zf7x35na27yigcxy7v6jumflnxavagyafsf3d2mmzgm25nec
        Sectors:
                197     OK
Batch 3:
        Message: bafy2bzacea4l44pyltuklajbawitk7zorndgmcevq4afvlrklqh7qkosxvz4k
        Sectors:
                203     OK

and

lotus-miner sectors batching precommit --publish-now


Batch 0:
        Message: bafy2bzaced6vtwrhzqqrqlwc2gdtxrld4s67ldtizq6dem32a7iinnkawz6lw
        Sectors:
                207     OK
Batch 1:
        Message: bafy2bzaceduluecszmaakmyn2h7nbk32izxqb66h3gzgbcp2otmdq3jecixbq
        Sectors:
                212     OK
Batch 2:
        Message: bafy2bzacecmzj4axq2fo7grythlsom2adsxvggh4ylbhxu2ryuc2llpk4ddbo
        Sectors:
                209     OK
Batch 3:
        Message: bafy2bzacecs5anevmdmk5x4vvgmmse6mlfrfor26yq3dzxafn6stosc7ahzgu
        Sectors:
                213     OK
Batch 4:
        Message: bafy2bzacecnlj7rx4h7n4zn6zbhv5qyf7d6u3rzxqf26sa6rciajlifd4m6vy
        Sectors:
                210     OK
Batch 5:
        Message: bafy2bzacec7te5pkorh7py6ybnt4iwp3mks3scmn67xwkq5xyawwtjswopknm
        Sectors:
                211     OK
Batch 6:
        Message: bafy2bzaceatctq5fqizq7numflwkbu5bs5qoqfmdjqnhhdkssxezj2tnof5rq
        Sectors:
                208     OK

[RobQuistNL]

This batching "logic" made me always lose the latest few added sectors (20 to 40) in my deadlines. Disabling it (manually
forcing the publishes with those commands) fixed my issue

The batching logic needs to be edited - this is costing people a lot of burn fees. +100 internet points for @f8-ptrk for finding this issue!
EDIT: it did not fix my issue

[Reiers]

Hello again @f8-ptrk

Thanks for taking the time to find this bug and creating a the report.
I have added labels and will assign it to the right team for analysis.
We might tag you again for more information.

[jennijuju]

@magik6k suggested maybe another thing we can do is, add a new configuration variable to say, if the base fee is below it, auto disable batching & aggregation.

[rjan90]

We expected the lotus-miner to send the message on the spot if the baseFee is below the baseFee threshold and not gather them until MaxPreCommitBatch is reached.

I´m not entirely sure the suggest logic here takes into account scenarioes when the BaseFee is consistently above 0.32nFIL and falls under 0.32nFIL for a short period of time (xx-minutes / xx-hours). It´s potentially uneconomically to send the messages in these scenarios, especially for larger amounts of sectors in aggregation/batching (needs further investigation and calculation).

What would the wanted behaviour when you have:

• XX sectors ready to be batched/aggregated from when the baseFee was > 0.32nFIL
• BaseFee falls under 0.32nFIL

[f8-ptrk]

can the basefee be below the threshhold? yes.

#7002 this is a direct consequence of this issue here. it needs to be fixed or it stays a gamble to do batching/aggregation

[rjan90]

an the basefee be below the threshhold? yes.

I was rather trying to get to the bottom of what the wanted behaviour from your point of view would be when the baseFee falls below the threshold when you already have sectors that are in currently in batching/aggregation. And that is totally a seperate question from #7002 which is a bug, which should be resolved. This is a feature request/enhacement.

[f8-ptrk]

glad you asked: a config flag that changes between always batch and never batch below threshold

[f8-ptrk]

If AggregateCommits is set to true, lotus will aggregate and batch pre-commitments until any of MaxCommitBatch, CommitBatchWait or CommitBatchSlack is hit:

AggregateAboveBaseFee is the network base fee to start aggregating proofs. When the network base fee is lower than this value, the prove commits will be submitted individually via ProveCommitSector. According to the Batch Incentive Alignment introduced in FIP-0013, we recommend you to set this value to 0.15 nanoFIL to avoid unexpected aggregation fee in burn.

https://lotus.filecoin.io/storage-providers/advanced-configurations/sealing/#provecommitaggregate

you're right - this is a feature request. i cannot find any specification documents for lotus that would describe the feature so i use the docs.

• docs stil lsay 0.15 nFIL - is that still valid? i though we are at 0.32 and the config default values seem to confirm that
• the "When the network base fee is lower than this value, the prove commits will be submitted individually via ProveCommitSector" is kinda confusing.

If AggregateCommits is set to true, lotus will aggregate and batch pre-commitments until any of MaxCommitBatch, CommitBatchWait or CommitBatchSlack is hit:

• c+p errors in here i assume ?!? "pre-commitment"

[rjan90]

a config flag that changes between always batch and never batch below threshold

That sounds possible to implement. What would you believe is the ideal feature when you have started batching/agreggating some sectors, but then then Basfee drops below the threshold?

[rjan90]

docs stil lsay 0.15 nFIL - is that still valid? i though we are at 0.32 and the config default values seem to confirm that

That seems to stem from an old config-file yes, and should be fixed.

the "When the network base fee is lower than this value, the prove commits will be submitted individually via ProveCommitSector" is kinda confusing.

c+p errors in here i assume ?!? "pre-commitment"

Thanks! As an open source software you always more then welcome to add PRs for fixing this.

For the Lotus docs I would recommend keeping the issues to the relevant repo: https://github.com/filecoin-project/lotus-docs/issues. There is also a very convenient "edit this page on Github" link at the bottom of each docs page that makes it very easy to make easy PRs for fixing c+p errors when you find them

[f8-ptrk]

That sounds possible to implement. What would you believe is the ideal feature when you have started batching/agreggating some sectors, but then then Basfee drops below the threshold?

then behave as before, now.

Thanks! As an open source software you always more then welcome to add PRs for fixing this.

fun story. the github logo on the docs page leads me to the lotus repo. and then you loose me as finding the docs repo needs 2-3 clicks and even typing.

[rjan90]

Again, please add these as issues in https://github.com/filecoin-project/lotus-docs/issues whenever you find them.

[rjan90]

fun story. the github logo on the docs page leads me to the lotus repo. and then you loose me as finding the docs repo needs 2-3 clicks and even typing.

The link at the bottom brings you right to the relevant page so you can edit it:

For the Github logo that has already been an longer discussion, and it´s unclear what people that are actively looking at the Lotus documentation would want to get linked to with the Github logo. We landed that most probably want to see the Lotus code repo, which is similar to most other open source software documentation. If we see evidence of otherwise, I´m happy to link it back to the Lotus docs github repo.