Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

My node always showing 100 blocks behind #258

Closed
urkishan opened this issue May 26, 2021 · 54 comments
Closed

My node always showing 100 blocks behind #258

urkishan opened this issue May 26, 2021 · 54 comments

Comments

@urkishan
Copy link

> eth.syncing

{
  currentBlock: 7738573,
  highestBlock: 7738689,
  knownStates: 370745564,
  pulledStates: 370690961,
  startingBlock: 7716912
}

Is there any way to boostup the syncing?

@soltrac
Copy link

soltrac commented May 26, 2021

Better hardware, nvme required.

@urkishan
Copy link
Author

I am using the better hardware, with 8 vCPU 32 GB of RAM and 5GB of Bandwidth and SSD 3000 IOPS. Please let me know if I need to increase this.

@soltrac
Copy link

soltrac commented May 26, 2021

3000 IOPS is nothing. A typical NVME SSD is doing around 250000 - 500000 IOPS

@urkishan
Copy link
Author

I am using AWS kindly please let me know which type of disk I need to use.

General Purpose SSD (gp2)
General Purpose SSD (gp3)
Provisioned Purpose SSD (io1)
Provisioned Purpose SSD (io2)

And as per my understanding we are using nvme type of SSD in aws.

@DefiDebauchery
Copy link

Just because they're selling you an NVME doesn't mean that you are getting all of that power.
In any case, someone in another ticket stated they were using 15K IOPS on AWS and it was syncing. That gives you a target.

@bencao
Copy link

bencao commented May 27, 2021

My node has been 60~100 blocks behind for more than a week.
Previous IOPS was set to 5000, but just updated to 16K to see whether the situation will improve.

Also the below syncing result on my node can give you an idea the lower boundary of # of knownStates, as of 2021-05-27.

{
  currentBlock: 7770055,
  highestBlock: 7770172,
  knownStates: 670047752,
  pulledStates: 669969232,
  startingBlock: 7564631
}

Update: please check out how I finally synced with a different instance type in comment

@urkishan
Copy link
Author

@DefiDebauchery I have increased my IOPS to 16k, and I am hoping it will be synced asap. @bencao do you know how much knownStates we have to wait for.

@x1957
Copy link

x1957 commented May 27, 2021

update gp3 IOPS to 16k too...hope some improve

@urkishan
Copy link
Author

update gp3 IOPS to 16k too...hope some improve

Yes I've changed the disk type to gp3 and IOPS is 16k, but still the known state is behind, and I don't know how much time this is gonna take. As I am trying to sync it from last 2 weeks. And this is making me really sick.

As I also downloaded the snapshot I thought that will work but I don't see any help on that too, as currently I am syncing BSC on two instance one with snapshot and other one without snapshot.

@x1957
Copy link

x1957 commented May 28, 2021

update gp3 IOPS to 16k too...hope some improve

Yes I've changed the disk type to gp3 and IOPS is 16k, but still the known state is behind, and I don't know how much time this is gonna take. As I am trying to sync it from last 2 weeks. And this is making me really sick.

As I also downloaded the snapshot I thought that will work but I don't see any help on that too, as currently I am syncing BSC on two instance one with snapshot and other one without snapshot.

I have the same problem ...T_T IOPS 16k help nothing...

@bgdsh
Copy link

bgdsh commented May 28, 2021

same issue here. Just followed the recommended instance type.

Screen Shot 2021-05-28 at 10 40 48 AM

@urkishan
Copy link
Author

Is this repo has contributors or not, I don't see anyone want to help us to resolve this issue.

@bencao
Copy link

bencao commented May 28, 2021

I had an AWS c6g.2xlarge instance even set IOPS to 16k doesn't seem to help after one day. I've started a new AWS EC2 i3 instance with 900G ephemeral NVME SSD to verify whether an even higher IOPS can help sync eventually. Will update this thread if I got some progress. Date when the comment was posted: 2021-05-28.

Update 1

Some raw numbers to check during syncing:

i3.xlarge instance with ephemeral native NVME, IOPS 70K, iostat MB_wrtn/s was averaged to be about 10M/s

expand to see details
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
{
  currentBlock: 5045035,
  highestBlock: 7796190,
  knownStates: 96385243,
  pulledStates: 95904052,
  startingBlock: 0
}

# iostat -m -d 10 10
Linux 4.14.231-173.361.amzn2.x86_64 (ip-172-31-66-40.ec2.internal) 	05/28/2021 	_x86_64_	(4 CPU)

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda              9.06         0.04         0.10        236        587
nvme0n1         125.74         0.00       164.96          6     982969

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda              0.30         0.00         0.00          0          0
nvme0n1          69.80         0.00         4.22          0         42

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda              0.00         0.00         0.00          0          0
nvme0n1         202.20         0.00        18.09          0        180

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda             39.20         0.00         0.34          0          3
nvme0n1         117.30         0.00         6.66          0         66

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda              0.20         0.00         0.00          0          0
nvme0n1          84.00         0.00         4.57          0         45

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda             35.60         0.00         0.26          0          2
nvme0n1         192.90         0.00        18.79          0        187

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda             31.20         0.10         0.18          1          1
nvme0n1          87.20         0.00         4.41          0         44

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda              0.40         0.00         0.00          0          0
nvme0n1          55.70         0.00         3.03          0         30

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda             36.50         0.00         0.28          0          2
nvme0n1         185.50         0.00        18.31          0        183

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
xvda              0.00         0.00         0.00          0          0
nvme0n1          97.10         0.00         6.20          0         62

c5.2xlarge instance with gp3 volume, IOPS 16K, MB_wrtn/s was almost 0

expand to see details
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
{
  currentBlock: 7797374,
  highestBlock: 7797452,
  knownStates: 687576703,
  pulledStates: 687420045,
  startingBlock: 7794295
}

# iostat -m -d 10 10
Linux 4.14.231-173.361.amzn2.aarch64 (ip-172-31-3-204.ec2.internal) 	05/28/2021 	_aarch64_	(8 CPU)

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1        1631.77        15.09         1.67   10541477    1166468

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1        1503.30        12.04         0.66        120          6

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1         759.20         5.85         0.00         58          0

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1        1555.24        12.45         0.00        124          0

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1        1356.30        10.76         0.00        107          0

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1         750.30         6.12         1.07         61         10

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1         837.80         6.93         0.00         69          0

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1         927.60         7.53         0.00         75          0

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1        1528.40        12.61         0.06        126          0

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
nvme0n1        1484.80        12.07         0.00        120          0

Update 2

Finally synced in 10 hours!
Based on geth process log, those are some final signals before the node became synced.

  • Indexing transactions, it kept printing out percentage until it reaches 100, took about 20mins for this may be, during this period eth.syncing result was frozen, but we can notice that pulledStates was the same as knownStates.
  • Generating state snapshot, this was a new feature in 1.10 for a faster snapshot sync mode, it kept saying resuming/aborting. Just let it running for a while and don't worry about it.
expand to see details
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
{
  currentBlock: 7808427,
  highestBlock: 7808533,
  knownStates: 340871401,
  pulledStates: 340871401,
  startingBlock: 7803641
}

# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
false

# date
Fri May 28 17:48:12 UTC 2021

# journalctl -f -u geth
May 28 17:50:38 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:38.371] Imported new chain segment               blocks=1    txs=104     mgas=14.633   elapsed=271.241ms   mgasps=53.948  number=7,808,617 hash=d33527..30585e dirty=1019.97MiB
May 28 17:50:38 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:38.373] Unindexed transactions                   blocks=1    txs=77      tail=5,458,618 elapsed=1.835ms
May 28 17:50:41 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:41.471] Imported new chain segment               blocks=1    txs=212     mgas=22.106   elapsed=392.647ms   mgasps=56.300  number=7,808,618 hash=a543ea..e0b103 dirty=1020.60MiB
May 28 17:50:41 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:41.472] Unindexed transactions                   blocks=1    txs=59      tail=5,458,619 elapsed=1.510ms

Update 3

I'll share my AWS EC2 based configuration here hope that can help others struggling with building a fully synced node.

1. EC2 Instance Config

I used a i3.xlarge EC2 instance with 4 CPUs, 32GB Memory, 900G native NVME and 8G gp3 volume for hosting the operating system. The AMI was Amazon Linux 2 AMI (HVM), SSD Volume Type, in 64-bit (x86) arch.

When the instance first started you have to create file system on the 900G blk device by mkfs -t xfs /dev/nvme0n1 (check device id by running lsblk). And then mkdir /opt/bsc and mount /dev/nvme0n1 /opt/bsc

2. /opt/bsc/node/config.toml
[Eth]
NetworkId = 56
SyncMode = "fast"
NoPruning = false
NoPrefetch = false
LightPeers = 100
DatabaseCache = 24000
DatabaseFreezer = ""
TrieCleanCache = 256
TrieDirtyCache = 256
UltraLightFraction = 75
TrieTimeout = 5000000000000
EnablePreimageRecording = false
EWASMInterpreter = ""
EVMInterpreter = ""

[Eth.Miner]
DelayLeftOver = 50000000
GasFloor = 8000000
GasCeil = 8000000
GasPrice = 1000000000
Recommit = 3000000000
Noverify = false

[Eth.Ethash]
CacheDir = "ethash"
CachesInMem = 2
CachesOnDisk = 3
CachesLockMmap = false
DatasetDir = "/opt/bsc/node/ethash"
DatasetsInMem = 1
DatasetsOnDisk = 2
DatasetsLockMmap = false
PowMode = 0

[Eth.TxPool]
Locals = []
NoLocals = false
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 1000000000
PriceBump = 10
AccountSlots = 512
GlobalSlots = 10000
AccountQueue = 256
GlobalQueue = 5000
Lifetime = 10800000000000

[Eth.GPO]
Blocks = 20
Percentile = 60
OracleThreshold = 1000

[Node]
DataDir = "/opt/bsc/node/data"
HTTPHost = "0.0.0.0"
NoUSB = true
InsecureUnlockAllowed = false
IPCPath = "geth.ipc"
HTTPPort = 8545
HTTPVirtualHosts = ["localhost"]
HTTPModules = ["net", "web3", "eth", "debug"]
WSPort = 8546
WSModules = ["net", "web3", "eth", "debug"]

[Node.P2P]
MaxPeers = 1000
NoDiscovery = false
BootstrapNodes = ["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]
StaticNodes = ["enode://f3cfd69f2808ef64838abd8786342c0b22fdd28268703c8d6812e26e109f9a7cb2b37bd49724ebb46c233289f22da82991c87345eb9a2dadeddb8f37eeb259ac@18.180.28.21:30311","enode://ae74385270d4afeb953561603fcedc4a0e755a241ffdea31c3f751dc8be5bf29c03bf46e3051d1c8d997c45479a92632020c9a84b96dcb63b2259ec09b4fde38@54.178.30.104:30311","enode://d1cabe083d5fc1da9b510889188f06dab891935294e4569df759fc2c4d684b3b4982051b84a9a078512202ad947f9240adc5b6abea5320fb9a736d2f6751c52e@54.238.28.14:30311","enode://f420209bac5324326c116d38d83edfa2256c4101a27cd3e7f9b8287dc8526900f4137e915df6806986b28bc79b1e66679b544a1c515a95ede86f4d809bd65dab@54.178.62.117:30311","enode://c0e8d1abd27c3c13ca879e16f34c12ffee936a7e5d7b7fb6f1af5cc75c6fad704e5667c7bbf7826fcb200d22b9bf86395271b0f76c21e63ad9a388ed548d4c90@54.65.247.12:30311","enode://f1b49b1cf536e36f9a56730f7a0ece899e5efb344eec2fdca3a335465bc4f619b98121f4a5032a1218fa8b69a5488d1ec48afe2abda073280beec296b104db31@13.114.199.41:30311","enode://4924583cfb262b6e333969c86eab8da009b3f7d165cc9ad326914f576c575741e71dc6e64a830e833c25e8c45b906364e58e70cdf043651fd583082ea7db5e3b@18.180.17.171:30311","enode://4d041250eb4f05ab55af184a01aed1a71d241a94a03a5b86f4e32659e1ab1e144be919890682d4afb5e7afd837146ce584d61a38837553d95a7de1f28ea4513a@54.178.99.222:30311","enode://b5772a14fdaeebf4c1924e73c923bdf11c35240a6da7b9e5ec0e6cbb95e78327690b90e8ab0ea5270debc8834454b98eca34cc2a19817f5972498648a6959a3a@54.170.158.102:30311","enode://f329176b187cec87b327f82e78b6ece3102a0f7c89b92a5312e1674062c6e89f785f55fb1b167e369d71c66b0548994c6035c6d85849eccb434d4d9e0c489cdd@34.253.94.130:30311","enode://cbfd1219940d4e312ad94108e7fa3bc34c4c22081d6f334a2e7b36bb28928b56879924cf0353ad85fa5b2f3d5033bbe8ad5371feae9c2088214184be301ed658@54.75.11.3:30311","enode://c64b0a0c619c03c220ea0d7cac754931f967665f9e148b92d2e46761ad9180f5eb5aaef48dfc230d8db8f8c16d2265a3d5407b06bedcd5f0f5a22c2f51c2e69f@54.216.208.163:30311","enode://352a361a9240d4d23bb6fab19cc6dc5a5fc6921abf19de65afe13f1802780aecd67c8c09d8c89043ff86947f171d98ab06906ef616d58e718067e02abea0dda9@79.125.105.65:30311","enode://bb683ef5d03db7d945d6f84b88e5b98920b70aecc22abed8c00d6db621f784e4280e5813d12694c7a091543064456ad9789980766f3f1feb38906cf7255c33d6@54.195.127.237:30311","enode://11dc6fea50630b68a9289055d6b0fb0e22fb5048a3f4e4efd741a7ab09dd79e78d383efc052089e516f0a0f3eacdd5d3ffbe5279b36ecc42ad7cd1f2767fdbdb@46.137.182.25:30311","enode://21530e423b42aed17d7eef67882ebb23357db4f8b10c94d4c71191f52955d97dc13eec03cfeff0fe3a1c89c955e81a6970c09689d21ecbec2142b26b7e759c45@54.216.119.18:30311","enode://d61a31410c365e7fcd50e24d56a77d2d9741d4a57b295cc5070189ad90d0ec749d113b4b0432c6d795eb36597efce88d12ca45e645ec51b3a2144e1c1c41b66a@34.204.129.242:30311","enode://bb91215b1d77c892897048dd58f709f02aacb5355aa8f50f00b67c879c3dffd7eef5b5a152ac46cdfb255295bec4d06701a8032456703c6b604a4686d388ea8f@75.101.197.198:30311","enode://786acbdf5a3cf91b99047a0fd8305e11e54d96ea3a72b1527050d3d6f8c9fc0278ff9ef56f3e56b3b70a283d97c309065506ea2fc3eb9b62477fd014a3ec1a96@107.23.90.162:30311","enode://4653bc7c235c3480968e5e81d91123bc67626f35c207ae4acab89347db675a627784c5982431300c02f547a7d33558718f7795e848d547a327abb111eac73636@54.144.170.236:30311","enode://c6ffd994c4ef130f90f8ee2fc08c1b0f02a6e9b12152092bf5a03dd7af9fd33597d4b2e2000a271cc0648d5e55242aeadd6d5061bb2e596372655ba0722cc704@54.147.151.108:30311","enode://99b07e9dc5f204263b87243146743399b2bd60c98f68d1239a3461d09087e6c417e40f1106fa606ccf54159feabdddb4e7f367559b349a6511e66e525de4906e@54.81.225.170:30311","enode://1479af5ea7bda822e8747d0b967309bced22cad5083b93bc6f4e1d7da7be067cd8495dc4c5a71579f2da8d9068f0c43ad6933d2b335a545b4ae49a846122b261@52.7.247.132:30311"]
ListenAddr = ":30311"
EnableMsgEvents = false

[Node.HTTPTimeouts]
ReadTimeout = 30000000000
WriteTimeout = 30000000000
IdleTimeout = 120000000000
3. /etc/systemd/system/geth.service
[Unit]
Description=BSC geth go client
After=syslog.target network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/geth --config /opt/bsc/node/config.toml --metrics --metrics.addr 0.0.0.0
KillMode=process
KillSignal=SIGINT
TimeoutStopSec=90
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target
4. chain initialization steps
# follow the guide to setup AWS EC2 time sync
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

# use /tmp as the working directory
cd /tmp

# download bsc 1.1.0-beta
wget https://github.com/binance-chain/bsc/releases/download/v1.1.0-beta/geth_linux
chmod +x geth_linux
mv geth_linux /usr/local/bin/geth

# initialize genesis
wget https://github.com/binance-chain/bsc/releases/download/v1.1.0-beta/mainnet.zip
unzip mainnet.zip 
/usr/local/bin/geth --datadir /opt/bsc/node/data init genesis.json

# start service
systemctl enable geth
systemctl start geth

# check syncing progress
/usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"

# check peer count
curl -s 127.0.0.1:6060/debug/metrics | grep peer

# check geth daemon logs
journalctl -f -u geth

Update 4

Cost

According to AWS Pricing Calculator, the cost for a i3.xlarge instance is about $228.40 per month.

Caveats

i3.xlarge instance NVME disk was ephemeral which means the data would be gone if you ever stopped the EC2 instance. Since EC2 SLA is not particularly high (99.99% which translates into 52m 35s downtime per year), it is recommended to run at least 2 of the above instances behind a load balancer to ensure continuous service.

@seonggwonyoon
Copy link

seonggwonyoon commented May 28, 2021

I'm using ap-northeast-2 region on aws and my instance type is m5a.2xlarge Disk IOPS is 3600 on gp2 nvme ssd I also using geth v1.1.0 version Full syncmode. (It is not an archive mode.)

However, the sync has never been slow. and I used this snapshot to get the initial data. Please note.

@urkishan
Copy link
Author

@seonggwonyoon your syncing is completed? Can you please let us know the highest knownStates you know.

@seonggwonyoon
Copy link

@urkishan Yes sync is completed.

{
  currentBlock: 7803509,
  highestBlock: 7803510,
  knownStates: 297473485,
  pulledStates: 297473485,
  startingBlock: 7803508
}

@n8twj
Copy link

n8twj commented May 28, 2021 via email

@seonggwonyoon
Copy link

seonggwonyoon commented May 28, 2021

@n8twj Thank you for letting me know json object returns false, but it stops syncing for a while to see the knownStates and print it.

@n8twj
Copy link

n8twj commented May 28, 2021 via email

@bencao
Copy link

bencao commented May 28, 2021

@urkishan @bgdsh @x1957 @seonggwonyoon Good news! Node synced in 10 hours. Details posted in the above comment

@x1957
Copy link

x1957 commented May 31, 2021

@bencao thanks, synced using i3.xlarge

@urkishan
Copy link
Author

urkishan commented May 31, 2021

Thank you @bencao, as one of the node of mine is also synced. Here is the details of AWS instance:
Instance type: t3.2xlarge

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
Stepping:            7
CPU MHz:             2500.004
BogoMIPS:            5000.00
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-7

With disk 1TB:

16K IOPS NVME Disk

Disk type: gp3

@x1957
Copy link

x1957 commented May 31, 2021

Thank you @bencao, as one of the node of mine is also synced. Here is the details of AWS instance:
Instance type: t3.2xlarge

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
Stepping:            7
CPU MHz:             2500.004
BogoMIPS:            5000.00
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-7

With disk 1TB:

16K IOPS NVME Disk

which type of nvme disk? gp2,gp3,io2?

@urkishan
Copy link
Author

@x1957 it is gp3.

@souilh
Copy link

souilh commented Jun 13, 2021

@bencao first thanks a lot for your sharing and i follow all the steps that you describe and now is many day and i dont have full sync i have this result :

{
currentBlock: 8270051,
highestBlock: 8270143,
knownStates: 488187192,
pulledStates: 488085742,
startingBlock: 0
}
any help will be appreciate

@muyinliu
Copy link

muyinliu commented Jul 4, 2021

BSC node sync test report of AWS EC2

Note: Region is Asia Pacific(Tokyo)

instant type cpu memory BSC data disk network price block fall behind at init sync take time
i3.large 2 cores 15.25 GB 1 × 0475 GB native NVMe SSD up to 10 Gigabit $0.1830/h 63809(8834170-8770361), 2d5h44m 09:15:28
i3.xlarge 4 cores 30.50 GB 1 × 0950 GB native NVMe SSD up to 10 Gigabit $0.3660/h
i3en.2xlarge 8 cores 64.00 GB 2 × 2500 GB native NVMe SSD Up to 25 Gigabit $1.0640/h 9282512(0-9282512) 08:01:55
m5zn.xlarge 4 cores 16.00 GB 1 × 1000 GB io2 SSD Up to 25 Gigabit $0.4267/h sync never complete !
m5zn.xlarge 4 cores 16.00 GB 1 × 1000 GB gp3 SSD Up to 25 Gigabit $0.4267/h sync never complete !

Note: sync take time calc from BSC node startup to log Fast sync complete, auto disabling

Note: i3.large instance(with option --cache 12000) sync from 2021-07-03T13:13:14+0000(startup time) to 2021-07-03T22:28:42+0000(log Fast sync complete, auto disabling), but current size of BSC node is 308 GB now(at 2021-07-04T02:31:28+0000), increase about 10 GB/day and 475 GB disk will full after about 2 weeks, so i3.xlarge is recommended(be aware: make sure do backup data of native NVMe SSD of i3.xlarge/i3.xlarge before terminated).

Note: i3.xlarge instance sync complete after about 6-7 hours, but I forgot to record the exact time, sorry for that.

Note: i3en.2xlarge instance use CPU up to 3.1 GHz for all cores

Note: gp3/io2 volumes never complete the sync, even if I config 10000 IOPS, keep logging Pivot became stale

Hope this report will help others who want to setup a BSC node on AWS EC2.

thanks @bencao

@evilfenggg
Copy link

/usr/local/bin/geth --rpc --ws

@muyinliu
Copy link

Looks like that you forgot to init with genesis.json

Init is required at the first time, command example:

geth --datadir /path/to/data/bsc-node init /etc/bsc/genesis.json

Then run command begin to sync:

geth --config /etc/bsc/config.toml --datadir /path/to/data/bsc-node console

Note: genesis.json and config.toml are download and unpack from https://github.com/binance-chain/bsc/releases/download/v1.1.0-beta/mainnet.zip

@evilfenggg
Copy link

@bencao 哥们帮忙看一下

@evilfenggg
Copy link

Still can't connect to http and ws @bencao @muyinliu

@evilfenggg
Copy link

http server started
websocket enabled

The firewall is also off
The security group has opened the port

why still can't connect?
Can anyone help me?

@bencao @n8twj @muyinliu

@muyinliu
Copy link

http server started
websocket enabled

The firewall is also off
The security group has opened the port

why still can't connect?
Can anyone help me?

@bencao @n8twj @muyinliu

Please show me the curl command that you used when testing the BSC http server.

@evilfenggg
Copy link

@muyinliu Can't sync, always 100 blocks slow

who can help me?currentBlock often doesn't move { currentBlock: 9405781, highestBlock: 9405898, knownStates: 476110330, pulledStates: 475889046, startingBlock: 9403727 }

@muyinliu
Copy link

@muyinliu Can't sync, always 100 blocks slow

who can help me?currentBlock often doesn't move { currentBlock: 9405781, highestBlock: 9405898, knownStates: 476110330, pulledStates: 475889046, startingBlock: 9403727 }

Show me the specification of your machine, please. It looks like that you are NOT using native NVMe SSD(PCIe 3.0×4 or higher)

@evilfenggg
Copy link

@muyinliu i3.xlarge aws ec2

@evilfenggg
Copy link

@muyinliu 哥们留个wx 或者q 可以吗 谢了

@n8twj
Copy link

n8twj commented Jul 23, 2021 via email

@jayboy-mabushi
Copy link

@n8twj I am having the same issue. I have been syncing for about 16 hours and its always about 100 block behind.
image

{
currentBlock: 9586006,
highestBlock: 9586131,
knownStates: 507448125,
pulledStates: 507397025,
startingBlock: 0
}

@evilfenggg
Copy link

I am having the same issue. I have been syncing for about 16 hours and its always about 100 block behind.

ec2

i3.xlarge

@bencao @muyinliu @n8twj

@n8twj
Copy link

n8twj commented Jul 30, 2021 via email

@jayboy-mabushi
Copy link

why is this issue closed when a lot of people are experiencing the same thing? nodes that were synced in before are suddenly lagging.

@jayboy-mabushi
Copy link

I have 12 core, 64g ram and 1T nvme ssd and still have this issoue. thats to tell u its not a hardware thing

@yyyandyou
Copy link

I have 12 core, 64g ram and 1T nvme ssd and still have this issoue. thats to tell u its not a hardware thing

Yes, i guess the connected peer could be the issue. As when i try, it seems only able to connect 1 peer, which cause the syncing process slow. As this is P2P network, i think the more nodes it can connect the faster it could be.

So, if it can connect more peers, i believe it can be faster. So updating the BootstrapNodes and StaticNodes may help more instead of paying x2 on buying the expensive SSD hard drive.

How do we get the latest and stable node address?
Why Binance never update their web site or Github on the latest enode?

@Jolly23
Copy link
Contributor

Jolly23 commented Oct 7, 2021

Found something here: https://ethereum.stackexchange.com/a/87931/76294
It says this situation is caused by FAST SYNC.
I think he explained clearly.

If you see that you are 64 blocks behind mainnet, you aren’t yet synchronized, not even close. You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You’ll need to wait that out too before your node comes truly online.

@izidorit
Copy link

I restarted my BSC node, and I experience the same, always behind by ~houndred blocks...

> eth.syncing
{
  currentBlock: 12569623,
  highestBlock: 12569732,
  knownStates: 493843508,
  pulledStates: 493798595,
  startingBlock: 12567541
}
> eth.syncing
{
  currentBlock: 12570008,
  highestBlock: 12570075,
  knownStates: 500897422,
  pulledStates: 500877686,
  startingBlock: 12567541
}
> eth.syncing
{
  currentBlock: 12579380,
  highestBlock: 12579492,
  knownStates: 725075801,
  pulledStates: 725034981,
  startingBlock: 12567541
}

Does anyone have any idea what's going on?

@n8twj
Copy link

n8twj commented Nov 12, 2021 via email

@izidorit
Copy link

izidorit commented Nov 12, 2021

Make sure you have updated to 1.1.4 and have enabled diffsync, that seems to help those servers who may not have the latest generation of NVMe drives.

Hi, I'm on 1.1.4, could you help me with this diffsync? On https://geth.ethereum.org/docs/interface/command-line-options I cannot see such a switch to enable :\

Update:
Found it:
https://github.com/binance-chain/bsc/releases/tag/v1.1.3

@n8twj
Copy link

n8twj commented Nov 12, 2021 via email

@izidorit
Copy link

Make sure you have updated to 1.1.4 and have enabled diffsync, that seems to help those servers who may not have the latest generation of NVMe drives.

Tried it: /usr/bin/geth --config /root/.ethereum/BSC-config.toml --diffsync --cache 131072
Still the same:

{
  currentBlock: 12580876,
  highestBlock: 12580994,
  knownStates: 752010831,
  pulledStates: 751997306,
  startingBlock: 12574756
}
{
  currentBlock: 12580943,
  highestBlock: 12581009,
  knownStates: 752129783,
  pulledStates: 752109410,
  startingBlock: 12574756
}
{
  currentBlock: 12581007,
  highestBlock: 12581084,
  knownStates: 752818157,
  pulledStates: 752805094,
  startingBlock: 12574756
}
{
  currentBlock: 12581071,
  highestBlock: 12581148,
  knownStates: 753455415,
  pulledStates: 753442809,
  startingBlock: 12574756
}

Still ~100 blocks away :(

@n8twj
Copy link

n8twj commented Nov 12, 2021 via email

@izidorit
Copy link

izidorit commented Nov 12, 2021

It is going to stay ~100 blocks away until/if it syncs up.
On Fri, Nov 12, 2021 at 03:24 izidorit @.***> wrote: Make sure you have updated to 1.1.4 and have enabled diffsync, that seems to help those servers who may not have the latest generation of NVMe drives. Tried it: /usr/bin/geth --config /root/.ethereum/BSC-config.toml --diffsync --cache 131072 Still the same: { currentBlock: 12580876, highestBlock: 12580994, knownStates: 752010831, pulledStates: 751997306, startingBlock: 12574756 } { currentBlock: 12580943, highestBlock: 12581009, knownStates: 752129783, pulledStates: 752109410, startingBlock: 12574756 } { currentBlock: 12581007, highestBlock: 12581084, knownStates: 752818157, pulledStates: 752805094, startingBlock: 12574756 } { currentBlock: 12581071, highestBlock: 12581148, knownStates: 753455415, pulledStates: 753442809, startingBlock: 12574756 } Still ~100 blocks away :(
-- Typos courtesy of my iPhone...

Sad fact, the same machine one month ago did the sync around 8 hours. Yesterday it started to fall back by 10-20 blocks, which caused slowness to the clients, who were using the node. Thus I removed geth folder, inited again, and started sync again...and it's not done after 34 hours, which is a pain in the ass...

@ib0b
Copy link

ib0b commented Nov 13, 2021

@izidorit
same here just got started with running nodes
tried a 8cpu 32gb 16,000 ips gp3 ~ never synced
tried an i3.xlarge ~still not syncing

keeps showing pending state entries

by any chance did you update your geth client to 1.1.4

I suspect people who have synced are probably running an old client

@gangadhar-devops
Copy link

--rpc --ws

it would work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests