Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Onetechnical/relstable2.2.0 #1681

Merged
merged 149 commits into from
Nov 2, 2020
Merged

Onetechnical/relstable2.2.0 #1681

merged 149 commits into from
Nov 2, 2020

Conversation

onetechnical
Copy link
Contributor

@onetechnical onetechnical commented Oct 31, 2020

Summary

Highlights

  • Miscellaneous bug fixes
  • Some slight performance optimizations
  • Additional tests and fixes to existing tests
  • Work to implement new packaging pipeline
  • Initial backend support for compact certificate development

Changes

  1. Agreement
    • Enhancements
      • Add consensus-driven agreement filter timeout
      • Vote validation error refactor
      • Deprecate algolambdamsec environment variable
  2. Algod
    • Bug Fix - exit algod with a non-zero in case of a startup error
  3. Catchup
    • Enhancements
      • Minor catchpoint REST handler error code cleanup
      • Improve RestoringFromCatchpointFile benchmark precision
    • Bug Fix - ensure syncStartNS is managed correctly across catchpoint file writing
    • Add detailed error logging in case of failed blocks download
  4. Crypto
    • Enhancement - add more inputs into hashCoin() for better hygiene
  5. General
    • New Features
      • Core crypto primitives for compact certificates
      • Added a transaction type to store compact certificates
    • Bug Fix - modify the error messages for parseIntcblock to distinguish from parseBytecblock
  6. Goal
    • Enhancements
      • Fix inconsistencies in goal asset command
      • Add goal account info command
      • Output error when 'goal node stop' is run on invalid/non-existing directory
    • Bug Fix - Avoid printing control characters from goal
  7. Ledger
    • New Feature - commit to Merkle tree of online participants in block header
    • Enhancements
      • Implement tracking of top-N online accounts in the ledger
      • Add metrics for the various ledger components
      • Cache the most recent Seed/Circulation calls to reduce pressure on ledger
      • Remove the Lookup from the ledgerForEvaluator interface
      • Decouple database round and acctupdate round
      • Fix typos in ledger package
      • Optimize listCreatables implementation
      • Repack merkle trie pages before flush
      • Extend account lookup to include validity range
    • Bug Fixes
      • Calculate normalized balance using genesis proto
      • Add syncMu to AlgorandFullNode OnNewBlock
  8. Transaction Pool
    • Bug Fixes
      • Refactor AssembleBlock error
      • Limit txpool size by transaction count rather then transaction groups count
      • Correct the accounting of transactions in checkPendingQueueSize
      • Correctly account for bloom filter binary marshaling size
  9. Network
    • Enhancements
      • Remove expected ReadError events
      • Add ASCII filtering for remote requests logged data
  10. REST API
    • New Feature - new endpoint /genesis.json
    • Enhancements
      • Don't return an error when requesting the same catchpoint multiple times
      • Add new line characters allowing proper Markdown formatting when docs are generated from this source file
      • REST docs update: /v1/account/{addr}/transactions
    • Bug Fixes
      • Add versions endpoint to spec, fix versions model objects
      • Allow max and assetIndex to be optional
      • Return 404 if v1/transaction/{txid} doesn't find the txn.
      • Set content type header for health endpoint
  11. Telemetry
    • Bug Fixes
      • Correct Telemetry uploading message reported in node.log
      • Change telemetryConfig from global variable to a member of the logger
  12. Tools
    • Enhancements
      • Added a new recipe for performing network disruption tests.
      • Simplifed dryrunning and debugging group transactions
      • Add support for algorelay deleting unused SRV and DNS entries
      • Add txn.Accounts and txn.ApplicationArgs support to tealdbg
      • Improve goal node start error reporting in case of algod errored
      • Tealdbg: listen for upcoming spinoff connections
      • Netgoal general templates
    • Bug Fixes
      • Improve network template validation
  13. Tests
    • New Feature - performance testing with realilistic network performance
    • Enhancements
      • Add concurrent read and write test from database table
      • Add a unit test to confirm BinaryMarshalLength correctness
      • Added more deb and rpm package tests
      • Coverage tests for ledger/acctupdates
      • Cleanup buggy dead code
      • Implement e2e test for tealdbg
      • Increase code coverage of tealdbg unit tests
      • Tweak the mainnet-model recipe
      • Added auth middleware tests
      • Added debugger coverage test
      • Add a benchmark to test restoring from catchpoint performance
      • Add unit tests for equal functions in teal.go
      • Add a benchmark for writeCatchpointStagingBalances
      • Improve db insertion benchmark - add ordered insertions
      • Add additional ledger tests
      • Updates to support testing TEAL application size
    • Bug Fixes
      • Extend expect timeout when checking account balance
      • Fix random failure in goalNodeTest.exp
      • Fix random failure in goalAppAccountAddressTest.exp
      • Fix unreliable network unit test TestSlowPeerDisconnection
      • Fix data race in TestPing unit test
      • Remove close from eof pattern match to remove error message
      • Test for process return in expect tests
  14. Other
    • New Feature - add support for building native Windows binaries
    • Enhancements
      • Update new issue template
      • Update rpm packaging to allow for more scenarios
      • Update deploy scripts to reflect current deploy strategy
      • Formatting updates with make sanity
      • Added S3_SOURCE env var to specify source of packages for signing
      • Implement mule packaging pipeline
      • Remove devtools binaries from algorand package (rpm)
      • Remove hardcoded dependency package names from package scirpts
      • Fix pathing issue to installer directory in packaging scripts
      • Simplify mule signing task
      • Update mule docs for new pipeline
      • Add support for Windows compilation on travis
      • Remove unused dockerfiles
      • Added package-upload mule task
      • Create mule tasks for aptly, releases page and docker hub images
    • Bug Fixes
      • Remove commas from mule yaml files
      • Revert change to grep recommended by shellcheck
      • Remove Travis token, which is not needed for open source repos
      • Many fixes for implementing new packaging pipeline

Test Plan

Standard release testing.

winder and others added 30 commits September 2, 2020 10:17
* Auth middleware unit tests.

* Add pprof test to e2e rest test script.
Added a new recipe for performing network disruption tests.
…le times (#1439)

No need to return an error when requesting the same catchpoint multiple times.
Improve error object creation for catchpoint errors
Extend clerk dryrun so it can dump dryrun state object suitable for REST, tealdbg and python tests
Add unit tests to logic debugger
Add support for algorelay deleting unused SRV and DNS entries
Add a benchmark to test restoring from catchpoint performance
Allow max and assetIndex to be optional
-- Add new REST API endpoint GET /genesis
Compact merkle trie nodes memory. Changing the underlying data structure is expected to reduce the practical memory utilization during a fast catchup by 50-60 percent.
Add unit tests for equal functions in teal.go
Implement core crypto primitives for compact certificates
Implement tracking of top-N online accounts in the ledger
algojohnlee and others added 18 commits October 27, 2020 14:04
This sets all the scripts to use the same method to get branch name as the deb package script. This allows the BRANCH var to get the TRAVIS_BRANCH value, if set.
Since jenkins doesn't pull the full git tree with refs, this will have compute_branch_network use the current branch instead of the branch parent when that rev parse fails.
…1663)

We weren't checking nulls appropriately when setting BRANCHPARENT in compute_branch_network.sh.
Remove quotes from /test/muleCI/mule.yaml env vars. These get propogated to the actual env vars.
The RPM packaging script wasn't pulling the right network due to hardcoding. Modified this and the deb script to use the passed branch to calculate the correct network. Also modified the compute branch network script to return betanet if passed rel/beta.
Improved the signing shell script in the following ways:

Fixed gpg warning by setting correct permissions in the docker container.
Instead of assuming the build status file is present, check for it first.
Fix issues that came up during betanet release

- RPM/arch issue when installing yum
- Missing genesisfiles for tarballs
- missing algorand-devtools-beta package
While we need to change the arch type for other reasons, we do not need to rename the RPM. A separate commit is reverting the name change, so this is reverting the related changes.
This supports multiple OS and ARCH by downloading assets recursively from s3. It then will determine multiple OS by inspecting the direct child directories of tmp/node_pkgs to determine how to proceed.

Assets are separated by ARCH as usual, i.e. tmp/node_pkgs/OS/ARCH, but when uploaded to staging they all go in the same object (flat directory structure).
Added ARCH_TYPE to file name so they won't overwrite each other.
Miscellaneous sign and test pipeline fixes:

- fix status file name
- change name of image in mule yaml
- add vars to dynamically account for both algorand and algorand-devtools packages (all channels)
@onetechnical onetechnical self-assigned this Oct 31, 2020
@ian-algorand ian-algorand added this to the Sprint 12 milestone Nov 2, 2020
@onetechnical onetechnical marked this pull request as ready for review November 2, 2020 22:07
Copy link
Contributor

@egieseke egieseke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@algojohnlee algojohnlee merged commit c048b4a into algorand:rel/stable Nov 2, 2020
@onetechnical onetechnical deleted the onetechnical/relstable2.2.0 branch April 16, 2021 21:31
tsachiherman pushed a commit to tsachiherman/go-algorand that referenced this pull request Jul 7, 2021
PhearZero pushed a commit to PhearNet/crypto that referenced this pull request Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.