Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Reintroduce the MapStore interface and SimpleMap and make the Badger a submodules #33

Merged
merged 33 commits into from
Jan 3, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
bb08623
feat: move benchmarks into docs folder, tidy readme and docs
h5law Dec 23, 2023
89dee09
feat: update gitignore
h5law Dec 23, 2023
dedc8f3
feat: cleanup and update workflow file
h5law Dec 23, 2023
2d215eb
feat: make codecov reports informational only
h5law Dec 23, 2023
63a9c3a
feat: add badger as a submodule
h5law Dec 22, 2023
83a994a
chore: makefile changes
h5law Dec 22, 2023
bcc569f
chore: update workflow to cover submodule tests
h5law Dec 22, 2023
4280986
chore: revert go version to 1.20.12
h5law Dec 23, 2023
0ce48aa
chore: unexport simpleMap MapStore implementation
h5law Dec 23, 2023
27dd854
chore: add kvstore package godoc
h5law Dec 23, 2023
66beac1
chore: s/badger/v4/g
h5law Dec 23, 2023
a0ebc3f
chore: use custom errors and wrap badgerv4 errors
h5law Dec 23, 2023
719ea55
squash: merged feat/doc-changes
h5law Jan 3, 2024
7a0f8fb
chore: add executable code examples
h5law Jan 3, 2024
17ccf68
feat: move interface out of submodule declarations and add simplemap …
h5law Jan 3, 2024
e4670f1
chore: better submodule support
h5law Jan 3, 2024
4518387
chore: grammar
h5law Jan 3, 2024
afeb02e
chore: fix example tests
h5law Jan 3, 2024
a64495a
s/name/desc/g
h5law Jan 3, 2024
f88a178
chore: clarify godoc
h5law Jan 3, 2024
81ff1b8
chore: refactor to use simplemap submodules
h5law Jan 3, 2024
e9892b1
chore: fix tests to cover simplemap submodule
h5law Jan 3, 2024
2d70eb0
chore: update godocs
h5law Jan 3, 2024
71ec219
feat: add SMT specific errors
h5law Jan 3, 2024
9bcd021
chore: add simplestore MapStore complaiance check
h5law Jan 3, 2024
f927cf2
chore: address comments
h5law Jan 3, 2024
6850a8a
chore: interface commennts
h5law Jan 3, 2024
fee3061
chore: s/name/desc/g
h5law Jan 3, 2024
a7c793c
chore: address comments
h5law Jan 3, 2024
9b1371c
chore: indentation
h5law Jan 3, 2024
27461f3
Merge branch 'main' into feat/badger-submodule
h5law Jan 3, 2024
538dcb5
chore: add simple to simplemap description
h5law Jan 3, 2024
58cc541
chore: address final comments
h5law Jan 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ jobs:
# Run each of the tests (excluding benchmarks) outputting the JSON
# to the test_results.json file for usage in later steps.
set -euo pipefail
go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee -a test_results.json
go test -v -json -p 1 ./kvstore/... -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json
go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee test_results.json
go test -v -json -p 1 ./kvstore/simplemap/... -mod=readonly -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json
go test -v -json -p 1 ./kvstore/badger/... -mod=readonly -race -coverprofile=coverage3.txt -covermode=atomic 2>&1 | tee -a test_results.json
# Combine coverage reports
gocovmerge coverage1.txt coverage2.txt coverage3.txt > coverage.txt
Expand Down
12 changes: 11 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,22 @@ go_docs: check_godoc ## Generate documentation for the project
.PHONY: test_all
test_all: ## runs the test suite
go test -v -p 1 ./ -mod=readonly -race
go test -v -p 1 ./kvstore/... -race
go test -v -p 1 ./kvstore/simplemap/... -mod=readonly -race

.PHONY: test_badger
test_badger: ## runs the badger KVStore submodule's test suite
go test -v -p 1 ./kvstore/badger/... -mod=readonly -race


#####################
### go mod ###
#####################
.PHONY: mod_tidy
mod_tidy: ## runs go mod tidy for all (sub)modules
h5law marked this conversation as resolved.
Show resolved Hide resolved
go mod tidy
cd kvstore/simplemap && go mod tidy
cd kvstore/badger && go mod tidy

#####################
### Benchmark ###
h5law marked this conversation as resolved.
Show resolved Hide resolved
#####################
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

<!-- tocstop -->

**NOTE: Requires Go 1.20.12**
**NOTE: Requires Go 1.20.12+**

## Overview

Expand All @@ -41,7 +41,7 @@ make test_all
```

To test the `badger` submodule that provides a more fully featured key-value
store run the following command:
store, run the following command:

```sh
make test_badger
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/bench_leaf_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ import (
"github.com/stretchr/testify/require"

"github.com/pokt-network/smt"
"github.com/pokt-network/smt/kvstore"
"github.com/pokt-network/smt/kvstore/simplemap"
)

func BenchmarkSMTLeafSizes_Fill(b *testing.B) {
trieSizes := []int{100000, 500000, 1000000, 5000000, 10000000} // number of leaves
leafSizes := []int{256, 512, 1024, 2048, 4096, 8192, 16384} // number of bytes per leaf
nodes := kvstore.NewSimpleMap()
nodes := simplemap.NewSimpleMap()
for _, trieSize := range trieSizes {
for _, leafSize := range leafSizes {
leaf := make([]byte, leafSize)
Expand Down Expand Up @@ -44,7 +44,7 @@ func BenchmarkSMTLeafSizes_Fill(b *testing.B) {
func BenchmarkSMSTLeafSizes_Fill(b *testing.B) {
trieSizes := []int{100000, 500000, 1000000, 5000000, 10000000} // number of leaves
leafSizes := []int{256, 512, 1024, 2048, 4096, 8192, 16384} // number of bytes per leaf
nodes := kvstore.NewSimpleMap()
nodes := simplemap.NewSimpleMap()
for _, trieSize := range trieSizes {
for _, leafSize := range leafSizes {
leaf := make([]byte, leafSize)
Expand Down
70 changes: 35 additions & 35 deletions benchmarks/bench_smst_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,65 +11,65 @@ import (

func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) {
testCases := []struct {
name string
desc string
trieSize int
commit bool
}{
{
name: "Fill (100000)",
desc: "Fill (100000)",
trieSize: 100000,
commit: false,
},
{
name: "Fill & Commit (100000)",
desc: "Fill & Commit (100000)",
trieSize: 100000,
commit: true,
},
{
name: "Fill (500000)",
desc: "Fill (500000)",
trieSize: 500000,
commit: false,
},
{
name: "Fill & Commit (500000)",
desc: "Fill & Commit (500000)",
trieSize: 500000,
commit: true,
},
{
name: "Fill (1000000)",
desc: "Fill (1000000)",
trieSize: 1000000,
commit: false,
},
{
name: "Fill & Commit (1000000)",
desc: "Fill & Commit (1000000)",
trieSize: 1000000,
commit: true,
},
{
name: "Fill (5000000)",
desc: "Fill (5000000)",
trieSize: 5000000,
commit: false,
},
{
name: "Fill & Commit (5000000)",
desc: "Fill & Commit (5000000)",
trieSize: 5000000,
commit: true,
},
{
name: "Fill (10000000)",
desc: "Fill (10000000)",
trieSize: 10000000,
commit: false,
},
{
name: "Fill & Commit (10000000)",
desc: "Fill & Commit (10000000)",
trieSize: 10000000,
commit: true,
},
}

for _, tc := range testCases {
b.ResetTimer()
b.Run(tc.name, func(b *testing.B) {
b.Run(tc.desc, func(b *testing.B) {
trie := setupSMST(b, tc.trieSize)
b.ResetTimer()
b.StartTimer()
Expand All @@ -90,67 +90,67 @@ func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) {

func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) {
testCases := []struct {
name string
desc string
trieSize int
commit bool
fn func(*smt.SMST, uint64) error
}{
{
name: "Update (Prefilled: 100000)",
desc: "Update (Prefilled: 100000)",
trieSize: 100000,
commit: false,
fn: updSMST,
},
{
name: "Update & Commit (Prefilled: 100000)",
desc: "Update & Commit (Prefilled: 100000)",
trieSize: 100000,
commit: true,
fn: updSMST,
},
{
name: "Update (Prefilled: 500000)",
desc: "Update (Prefilled: 500000)",
trieSize: 500000,
commit: false,
fn: updSMST,
},
{
name: "Update & Commit (Prefilled: 500000)",
desc: "Update & Commit (Prefilled: 500000)",
trieSize: 500000,
commit: true,
fn: updSMST,
},
{
name: "Update (Prefilled: 1000000)",
desc: "Update (Prefilled: 1000000)",
trieSize: 1000000,
commit: false,
fn: updSMST,
},
{
name: "Update & Commit (Prefilled: 1000000)",
desc: "Update & Commit (Prefilled: 1000000)",
trieSize: 1000000,
commit: true,
fn: updSMST,
},
{
name: "Update (Prefilled: 5000000)",
desc: "Update (Prefilled: 5000000)",
trieSize: 5000000,
commit: false,
fn: updSMST,
},
{
name: "Update & Commit (Prefilled: 5000000)",
desc: "Update & Commit (Prefilled: 5000000)",
trieSize: 5000000,
commit: true,
fn: updSMST,
},
{
name: "Update (Prefilled: 10000000)",
desc: "Update (Prefilled: 10000000)",
trieSize: 10000000,
commit: false,
fn: updSMST,
},
{
name: "Update & Commit (Prefilled: 10000000)",
desc: "Update & Commit (Prefilled: 10000000)",
trieSize: 10000000,
commit: true,
fn: updSMST,
Expand All @@ -159,7 +159,7 @@ func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) {

for _, tc := range testCases {
b.ResetTimer()
b.Run(tc.name, func(b *testing.B) {
b.Run(tc.desc, func(b *testing.B) {
trie := setupSMST(b, tc.trieSize)
benchmarkSMST(b, trie, tc.commit, tc.fn)
})
Expand All @@ -168,37 +168,37 @@ func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) {

func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) {
testCases := []struct {
name string
desc string
trieSize int
commit bool
fn func(*smt.SMST, uint64) error
}{
{
name: "Get (Prefilled: 100000)",
desc: "Get (Prefilled: 100000)",
trieSize: 100000,
commit: false,
fn: getSMST,
},
{
name: "Get (Prefilled: 500000)",
desc: "Get (Prefilled: 500000)",
trieSize: 500000,
commit: false,
fn: getSMST,
},
{
name: "Get (Prefilled: 1000000)",
desc: "Get (Prefilled: 1000000)",
trieSize: 1000000,
commit: false,
fn: getSMST,
},
{
name: "Get (Prefilled: 5000000)",
desc: "Get (Prefilled: 5000000)",
trieSize: 5000000,
commit: false,
fn: getSMST,
},
{
name: "Get (Prefilled: 10000000)",
desc: "Get (Prefilled: 10000000)",
trieSize: 10000000,
commit: false,
fn: getSMST,
Expand All @@ -207,7 +207,7 @@ func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) {

for _, tc := range testCases {
b.ResetTimer()
b.Run(tc.name, func(b *testing.B) {
b.Run(tc.desc, func(b *testing.B) {
trie := setupSMST(b, tc.trieSize)
benchmarkSMST(b, trie, tc.commit, tc.fn)
})
Expand All @@ -216,25 +216,25 @@ func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) {

func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) {
testCases := []struct {
name string
desc string
trieSize int
commit bool
fn func(*smt.SMST, uint64) error
}{
{
name: "Prove (Prefilled: 100000)",
desc: "Prove (Prefilled: 100000)",
trieSize: 100000,
commit: false,
fn: proSMST,
},
{
name: "Prove (Prefilled: 500000)",
desc: "Prove (Prefilled: 500000)",
trieSize: 500000,
commit: false,
fn: proSMST,
},
{
name: "Prove (Prefilled: 1000000)",
desc: "Prove (Prefilled: 1000000)",
trieSize: 1000000,
commit: false,
fn: proSMST,
Expand Down
Loading
Loading