Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

occasionally transactions are submitted but not broadcast #2084

Closed
terianil opened this issue Dec 18, 2023 · 10 comments
Closed

occasionally transactions are submitted but not broadcast #2084

terianil opened this issue Dec 18, 2023 · 10 comments
Assignees
@weiihann
Labels
X-network task filter for network issues, features, improve

Comments

@terianil
Copy link

System information

Geth version: geth_linux version 1.3.5-fe928d47-20231208
OS & Version: Linux

Expected behaviour

Sent transactions are broadcasted properly.

Actual behaviour

Once every several months we have issues with broadcasting transactions.
It has happened across multiple versions of 1.2.x and now we've noticed that issue on 1.3.5.

After 3 days of proper work and multiple transactions being sent BSC stopped broadcasting transactions.

In logs transaction submission is visible, but nothing more happens:

t=2023-12-17T16:27:29+0000 lvl=info msg="Submitted transaction"                  hash=xyz from=xyz nonce=xxx recipient=xxx value=0                      x-forward-ip=nil

I was able to send the same transactions (RAW hex) through our other nodes (as well as using bscscan.com). So it's not a wrong gas price estimation.

After restarting geth - everything works fine again - transactions are sent and broadcasted correctly.
Before restart geth had 66 connected peers (and the same number after restart), so it's probably not a connection problem.
@weiihann
Copy link
Contributor

weiihann commented Dec 19, 2023 •
edited
Loading

Hello! Thanks for raising this issue. Let me just restate your problem to make sure I understand it fully:

So you're facing a situation where occasionally, transactions are submitted through the BSC node but it's not included in the block, even though the logs say that it's submitted. Am I understanding it correctly?

Few questions to get more clarity:

  1. Each time you face the problem, is it only 1 transaction out of many that's not broadcasted? Or is it all transactions are not broadcasted until you restart the node?
  2. How many transactions does your node send per second (TPS)?

Can I also get the following details:

  1. geth startup command
  2. config.toml file
  3. [Optional] Peer list, just the "name" will do

If possible, please show the debugging logs as well. Thanks!

@Wbsba
Copy link

Wbsba commented Dec 22, 2023 •
edited
Loading

I've had the same problem many times
geth version 1.3.7

./geth --config ./config.toml --datadir ./datadir --cache 10240 --rpc.allow-unprotected-txs --txlookuplimit 0 --tries-verify-mode none --db.engine=pebble --history.transactions=0 --pruneancient --syncmode=full
config.toml
`
[Eth]
NetworkId = 56
LightPeers = 100
TrieTimeout = 150000000000

[Eth.Miner]
GasCeil = 140000000
GasPrice = 3000000000
Recommit = 10000000000

[Eth.TxPool]
Locals = []
NoLocals = true
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 3000000000
PriceBump = 10
AccountSlots = 200
GlobalSlots = 8000
AccountQueue = 200
GlobalQueue = 4000

[Eth.GPO]
Blocks = 20
Percentile = 60
OracleThreshold = 1000

[Node]
IPCPath = "geth.ipc"
HTTPHost = "0.0.0.0"
InsecureUnlockAllowed = false
HTTPPort = 8545
HTTPVirtualHosts = ["0.0.0.0"]
HTTPModules = ["eth", "net", "web3", "txpool", "parlia"]
WSPort = 8546
WSModules = ["net", "web3", "eth"]

[Node.P2P]
MaxPeers = 200
NoDiscovery = false
StaticNodes = []
ListenAddr = ":30311"
EnableMsgEvents = false

[Node.LogConfig]
FilePath = "bsc.log"
MaxBytesSize = 10485760
Level = "info"
FileRoot = ""

`

@Brightside56
Copy link

Have same issue on 1.2.15/1.3.7. Pending TXs are rebroadcasted normally using bscscan

@Joter271
Copy link

Joter271 commented Dec 26, 2023 •
edited
Loading

Same issue here, Geth 1.3.7 on Windows

geth_windows --config "D:\geth bnb\geth\config.toml" --syncmode=full --tries-verify-mode=none --pruneancient=true --diffblock=5000 --http --http.port 8998 --http.addr X --http.api eth,net,personal,admin --http.corsdomain '*' --nat extip:X --datadir "D:\geth bnb" --port 30302 --ipcdisable

Config is default

After some more digging we find that estimated transaction fee is too low.

@weiihann
Copy link
Contributor

weiihann commented Jan 2, 2024

Hey guys, try setting NoLocals = false in your config.toml files and do it again.

@terianil
Copy link
Author

terianil commented Jan 2, 2024

Hello! Thanks for raising this issue. Let me just restate your problem to make sure I understand it fully:

So you're facing a situation where occasionally, transactions are submitted through the BSC node but it's not included in the block, even though the logs say that it's submitted. Am I understanding it correctly?

Few questions to get more clarity:

  1. Each time you face the problem, is it only 1 transaction out of many that's not broadcasted? Or is it all transactions are not broadcasted until you restart the node?
  2. How many transactions does your node send per second (TPS)?

Can I also get the following details:

  1. geth startup command
  2. config.toml file
  3. [Optional] Peer list, just the "name" will do

If possible, please show the debugging logs as well. Thanks!

  1. Most of our transactions are from one address - so all transactions with higher nonce wait. But after I had used bscscan.com to broadcast the first several transactions - the others were broadcasted successfully.
  2. We send about 50 transactions a day

Details:

  1. startup command
geth_linux \
  --http \
  --graphql \
  --http.addr=0.0.0.0 \
  --http.api "admin,eth,net,web3,personal,txpool" \
  --datadir={{ coin_dir }} \
  --config {{ coin_dir }}/config.toml \
  --cache 8192 \
  --snapshot=true \
  --verbosity 5
  1. config.toml
[Eth]
NetworkId = 56
LightPeers = 100
TrieTimeout = 150000000000

[Eth.Miner]
GasCeil = 140000000
GasPrice = 3000000000
Recommit = 10000000000

[Eth.TxPool]
Locals = []
NoLocals = true
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 3000000000
PriceBump = 10
AccountSlots = 200
GlobalSlots = 8000
AccountQueue = 200
GlobalQueue = 4000

[Eth.GPO]
Blocks = 20
Percentile = 60
OracleThreshold = 1000

[Node]
IPCPath = "geth.ipc"
HTTPHost = "0.0.0.0"
NoUSB = true
InsecureUnlockAllowed = false
HTTPPort = 8545
HTTPVirtualHosts = ["localhost"]
HTTPModules = ["eth", "net", "web3", "txpool", "parlia"]
WSPort = 8546
WSModules = ["net", "web3", "eth"]

[Node.P2P]
MaxPeers = 200
NoDiscovery = false
StaticNodes = []
ListenAddr = ":30311"
EnableMsgEvents = false

[Node.LogConfig]
FilePath = "bsc.log"
MaxBytesSize = 10485760
Level = "info"
FileRoot = ""
  1. peers - admin.peers.map(x => x.name)
[
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.3.4-df449c2e/linux-amd64/go1.20.11",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Meganode/v1.2.15-p2p/linux-amd64/go1.20.3",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.3.4-df449c2e-20231204/linux-amd64/go1.20.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.3.4-313850a8-20231215/linux-amd64/go1.20.5",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.3.7/linux-amd64/go1.20.11",
    "Geth/v1.2.15/linux-amd64/go1.20.11",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15/linux-amd64/go1.19",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "Geth/v1.2.15-bfe000bd-20231204/linux-amd64/go1.19.13",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "Geth/v1.2.15/linux-amd64/go1.20.11",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7",
    "erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7"
]
  1. Unfortunately I don't have debug logs

@terianil
Copy link
Author

terianil commented Jan 2, 2024

ethereum/go-ethereum#28097

Maybe this PR from go-ethereum could solve the issue?

@weiihann
Copy link
Contributor

weiihann commented Jan 3, 2024 •
edited
Loading

@terianil Thanks for pointing out! I'll look into it.

By the way, did you still face the same issue after disabling NoLocals (i.e. set NoLocals = false)?

@weiihann weiihann mentioned this issue Jan 8, 2024
Merged
@weiihann weiihann added the X-network task filter for network issues, features, improve label Jan 8, 2024
@weiihann weiihann closed this as completed Jan 9, 2024
@Joter271
Copy link

Joter271 commented Jan 9, 2024 •
edited
Loading

This issue wasn't resolved. Setting NoLocals = false doesn't fix the issue.

Why was this closed?

@weiihann
Copy link
Contributor

eth/fetcher: allow underpriced transactions in after timeout (#28097) #2140

It's closed because there's a PR to fix it already. Let's wait for the next release and try it again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiihann weiihann

Labels
X-network task filter for network issues, features, improve
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Brightside56 @terianil @Joter271 @weiihann @Wbsba