too many pending ledger saves

[nelisky]

After a power failure my rippled was constantly going unavailable, server_info saying state was "full" and then "connected" and then "full" again and so on.

Looking closer at the logs I find this repeating:

2013-Mar-26 15:32:33 LedgerMaster:NFO Advancing accepted ledger to 412475 with >= 2 validations
2013-Mar-26 15:32:33 LedgerMaster:DBG Publishing ledger 412475
2013-Mar-26 15:32:33 LedgerMaster:DBG Ledger 412475 accepted :928B67A85986A12D4CE999B6A6E9FC16DFB3EB3DD27411F3E2EDAD353A18BAB5
2013-Mar-26 15:32:33 NetworkOPs:DBG recvValidation 928B67A85986A12D4CE999B6A6E9FC16DFB3EB3DD27411F3E2EDAD353A18BAB5 from 174.129.1
34.81
2013-Mar-26 15:32:33 ValidationCollection:DBG Val for 928B67A85986A12D4CE999B6A6E9FC16DFB3EB3DD27411F3E2EDAD353A18BAB5 from n9KPnV
Ln7ewVzHvn218DcEYsnWLzKerTDwhpofhk4Ym1RUq4TeGw added trusted/current
2013-Mar-26 15:32:34 Ledger:DBG no resume, too many pending ledger saves
2013-Mar-26 15:32:34 LedgerAcquire:DBG Done: complete 412475
2013-Mar-26 15:32:35 Application:NFO Root master seed: snoPBrXtMeMyMHUVTgbuqAfg1SUTb
2013-Mar-26 15:32:35 Application:NFO Root account: rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh
2013-Mar-26 15:32:35 LedgerMaster:NFO PushLedger: AB868A6CFEEC779C2FF845C0AF00A642259986AF40C01976A7F842B6918936C7
2013-Mar-26 15:32:35 LedgerMaster:NFO StashAccepted: AFC49AC1E7AA317C67E712EDBC3A2DD3C5040928F7B150BE1EF9025F24C9D341
2013-Mar-26 15:32:35 OrderBookDB:DBG OrderBookDB>
2013-Mar-26 15:32:35 OrderBookDB:DBG OrderBookDB<
2013-Mar-26 15:32:35 UniqueNodeList:DBG Validator fetch updated: 2013-Mar-26 15:32:20
2013-Mar-26 15:32:35 UniqueNodeList:DBG Validator score updated: 2013-Mar-26 15:32:24
2013-Mar-26 15:32:35 UniqueNodeList:NFO Bootstrapping UNL: loading from 'validators.txt'.
2013-Mar-26 15:32:35 UniqueNodeList:WRN validators.txt not found: "validators.txt"
2013-Mar-26 15:32:35 UniqueNodeList:NFO Bootstrapping UNL: loading from '"/home/cn/src/NewCoin/build/rippled.cfg"'.
2013-Mar-26 15:32:35 UniqueNodeList:NFO Node Public: n9KPnVLn7ewVzHvn218DcEYsnWLzKerTDwhpofhk4Ym1RUq4TeGw RIP1
2013-Mar-26 15:32:35 UniqueNodeList:NFO Node Public: n9LFzWuhKNvXStHAuemfRKFVECLApowncMAM5chSCL9R5ECHGN4V RIP2
2013-Mar-26 15:32:36 UniqueNodeList:NFO Node Public: n94rSdgTyBNGvYg8pZXGuNt59Y5bGAZGxbxyvjDaqD9ceRAgD85P RIP3
2013-Mar-26 15:32:36 UniqueNodeList:NFO Node Public: n9LeQeDcLDMZKjx1TZtrXoLBLo5q1bR1sUQrWG7tEADFU6R27UBp RIP4
2013-Mar-26 15:32:36 TaggedCache:DBG Validations target size set to 256

So after Ledger:DBG no resume, too many pending ledger saves it just restarts, and this continues in an infinite loop.

[JoelKatz]

"Too many pending ledger saves" is perfectly normal, that's why it's at debug level. It just means the server was I/O limited and couldn't continue fetching historical ledgers because it already had too much information in memory waiting to write to disk. It's not clear from this log what the issue is.

[nelisky]

Ok, I didn't even check the log level tbh, I just assumed as that was the last thing before the restart consistently.

I have grabbed a clean log with -v that shows rippled restarting every 20 secs or so:
https://dl.dropbox.com/u/38488881/debug.log

I'm going to try a clean DB, but will keep the old one in case it is needed for debugging. I am simply not able to use rippled currently, and this was triggered by a power fail so some db inconsistency seems likely.