cmd/integration: collection of commands to export db data into csv #11991
Conversation
|
FYI: geth has |
@AskAlexSharov thanks, I'll check it out. I'm exporting tables in CSV because it is easy to visualise in an IDE. I find it quite helpful for debugging purposes. E.g. I just run
|
| label := kv.ChainDB | ||
| db, err := openDB(dbCfg(label, path.Join(datadirCli, label.String())), true, logger) | ||
| if err != nil { | ||
| logger.Error(err.Error()) |
There was a problem hiding this comment.
i would advise move all logic to a func like doExportHeaderTD - which will do usual return err and here log error once.
| copy(blockHash[8:], k) | ||
|
|
||
| td := new(big.Int) | ||
| if err := rlp.Decode(bytes.NewReader(v), td); err != nil { |
There was a problem hiding this comment.
use rlp.DecodeBytes - it will do less allocs. (see: #12013 )
| getter := seg.MakeGetter() | ||
| for getter.HasNext() { | ||
| var buf []byte | ||
| buf, _ = getter.Next(buf) |
There was a problem hiding this comment.
probably you want getter.Next(buf[:0])
| seg := eventSegments[i] | ||
| getter := seg.MakeGetter() | ||
| for getter.HasNext() { | ||
| var buf []byte |
There was a problem hiding this comment.
declare buf outside of loop. then passing it to getter.Next will give an effect (it will not alloc buf on every iteration)
| var buf []byte | ||
| buf, _ = getter.Next(buf) | ||
|
|
||
| eventBytes := rlp.RawValue(libcommon.Copy(buf[length.Hash+length.BlockNum+8:])) |
There was a problem hiding this comment.
you dont use eventBytes outside of loop iteration, then no reason to common.Copy
| return | ||
| } | ||
|
|
||
| out, err := os.Create(outputCsvFile) |
There was a problem hiding this comment.
probably you want defer f.Close()
| snapDir := path.Join(datadirCli, "snapshots") | ||
| logger.Info("snapshot dir info", "dir", snapDir) | ||
|
|
||
| allSnapshots := freezeblocks.NewRoSnapshots(ethconfig.BlocksFreezing{}, snapDir, 0, logger) |
There was a problem hiding this comment.
pls use allSnapshots() helper in integration. because it using global singletones - and in future they may conflict with your objects (theoreticaly)
| } | ||
|
|
||
| defer tx.Rollback() | ||
| snapDir := path.Join(datadirCli, "snapshots") |
There was a problem hiding this comment.
pls use:
dirs := datadir.New(datadirCli)
dirs.Snap
| from := make([]byte, 8) | ||
| binary.BigEndian.PutUint64(from, max(fromNum, lastFrozenEventId+1)) | ||
| var k, v []byte | ||
| for k, v, err = c.Seek(from); iterateDb && err == nil && k != nil; k, v, err = c.Next() { |
There was a problem hiding this comment.
theoretically logic of db read and snapshots read - better add to BlockReader object (like: .ForEachBorEvent()) - but maybe you need very-custom logic - then fine.
Commands to export table+snapshot data for various tables:
The csv outputs can then be opened in an IDE. Makes life easier when debugging issues. Example, heimdall events table + snapshots:
