Bump version 8.0.0b1 (#1034)

CHANGELOG.md

@@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
-## [Unreleased]
+## [8.0.0b1] - 2024-06-18
 
 ### Added
 
@@ -1491,7 +1491,8 @@ This release contains no changes except for the version number.
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/8.0.0a1...HEAD
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/8.0.0b1...HEAD
+[8.0.0b1]: https://github.com/dipdup-io/dipdup/compare/8.0.0a1...8.0.0b1
 [7.5.7]: https://github.com/dipdup-io/dipdup/compare/7.5.6...7.5.7
 [7.5.6]: https://github.com/dipdup-io/dipdup/compare/7.5.5...7.5.6
 [8.0.0a1]: https://github.com/dipdup-io/dipdup/compare/7.5.7...8.0.0a1

Makefile

@@ -13,6 +13,10 @@ FRONTEND_PATH=../interface
 help:           ## Show this help (default)
 	@grep -Fh "##" $(MAKEFILE_LIST) | grep -Fv grep -F | sed -e 's/\\$$//' | sed -e 's/##//'
 
+##
+##-- CI
+##
+
 all:            ## Run an entire CI pipeline
 	make format lint test
 
@@ -25,6 +29,9 @@ lint:           ## Lint with all tools
 test:           ## Run tests
 	COVERAGE_CORE=sysmon pytest --cov-report=term-missing --cov=dipdup --cov-report=xml -n auto -s -v tests
 
+image:          ## Build Docker image
+	docker buildx build . -t ${PACKAGE}:${TAG}
+
 ##
 
 black:          ## Format with black
@@ -37,20 +44,11 @@ mypy:           ## Lint with mypy
 	mypy ${SOURCE}
 
 ##
-
-image:          ## Build Docker image
-	docker buildx build . -t ${PACKAGE}:${TAG}
-
+##-- Docs
 ##
 
-demos:          ## Recreate demo projects from templates
-	python scripts/demos.py render ${DEMO}
-	python scripts/demos.py init ${DEMO}
-	make format lint
-
-docs: docs_build
-
-docs_build:     ## Build docs
+docs_build: docs
+docs:           ## Build docs
 	python scripts/docs.py check-links --source docs
 	python scripts/docs.py dump-references
 	python scripts/docs.py dump-demos
@@ -65,8 +63,18 @@ docs_serve:     ## Build docs and start frontend server
 docs_watch:     ## Build docs and watch for changes
 	python scripts/docs.py build --source docs --destination ${FRONTEND_PATH}/content/docs --watch
 
-fixme:          ## Find FIXME and TODO comments
-	grep -r -e 'FIXME: ' -e 'TODO: ' -e 'type: ignore' -n src/dipdup --color
+##
+
+fixme: todo
+todo:           ## Find FIXME and TODO comments
+	grep -r -e 'FIXME: ' -e 'TODO: ' -n src/dipdup --color
+
+typeignore:     ## Find type:ignore comments
+	grep -r -e 'type: ignore' -n src/dipdup --color
+
+##
+##-- Release
+##
 
 update:         ## Update dependencies and dump requirements.txt
 	pdm update
@@ -80,5 +88,5 @@ before_release: ## Prepare for a new release after updating version in pyproject
 	make demos
 	make test
 	make docs_build
-	echo "🎉 Commit changes, merge aux/X.Y.Z branch, tag release on next"
+
 ##

docs/14.contributing.md

@@ -10,48 +10,66 @@ DipDup is a free and open-source software licensed under the [MIT License](#mit-
 
 ## Contributor Guide
 
+Thanks for considering contributing to DipDup! We're happy to have you here. This document is meant to help you get started with the project. If you have any questions, feel free to open an issue on [GitHub](https://github.com/dipdup-io/dipdup/issues/new/choose) or join our [Discord server](https://discord.gg/aG8XKuwsQd). Have fun!
+
+### Workflow
+
+1. Fork the repository, clone it, and `git checkout next`.
+2. Create a new branch with `git checkout -b <your-branch-name>`.
+3. Make your changes. Run `make format lint` to perform basic code checks.
+4. When finished, push your branch with `git push origin --set-upstream <your-branch-name>`.
+5. Create a pull request to merge `<your-branch-name>` into `next`. Maintainers will review your pull request and may make comments, ask questions, or request changes. When all feedback has been addressed the pull request will be approved, and after all checks have passed it will be merged by a maintainer.
+
+### Development environment
+
+You'll need Python 3.12 and PDM package manager to run DipDup locally. To set up the development environment, run `pdm venv create python3.12 && pdm sync && $(pdm venv activate)`. To see the list of development commands, run `make help`.
+
+## Maintainer Guide
+
+::banner{type="warning"}
+This section is intended for DipDup maintainers only! If you're a regular contributor, you don't need to read it (but welcome to do so).
+::
+
 The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
 
 ### General
 
 - All code in this repository MUST be licensed under the [MIT License](#mit-license).
 - Python code in this repository MUST run on Python 3.12. Using modern language features is encouraged.
-- Python code in this repository MUST run in Linux, macOS, Docker, and environments on `amd64` and `arm64` architectures.
-- We use the PDM package manager to set up the development environment. You SHOULD install it and run `pdm run -l` to see available shortcuts.
-- Have fun!
+- Python code in this repository MUST run in Linux, macOS, and Docker environments, and on `amd64`, `arm64` architectures. CI will help you with that.
 
 ### Git workflow
 
-- Branch names MUST follow `prefix/short-description` format. Prefixes currently in use: `feat` for features, `fix` for bugfixes, `docs` for documentation, `exp` for experiments, `ci` for GHA and Docker stuff, `aux` for everything else.
+- Default branches are `next` for the latest release and `current` for the previous one. `next` is the default branch for pull requests.
+- Branch names SHOULD follow the `prefix/short-description` format. Prefixes currently in use: `feat` for features, `fix` for bugfixes, `docs` for documentation, `exp` for experiments, `ref` for refactoring, `ci` for GitHub Actions, scripts and Docker stuff, `aux` for everything else.
 - Commits in pull requests MUST be squashed when merging to `next`.
 - Issues and pull requests MUST have a descriptive title; they SHOULD be linked to each other, appropriately labeled, and assigned to maintainers while in progress.
 
 ### Codestyle
 
-We use several tools to enforce codestyle and code quality: `black` for autoformatting, `ruff` for linting, and `mypy` for typechecking. All checks MUST pass before merging the code to the default branch. Everything not enforced by these tools is up to the developer. But here are some recommendations:
+We use several tools to enforce the code style and decent code quality: `black` for autoformatting, `ruff` for linting, and `mypy` for type checking. All checks MUST pass before the code is merged to the default branch. Anything not enforced by these tools is up to the developer. But here are some recommendations:
 
-- Consistency is the key. If you see a pattern in the codebase, follow it.
-- Use `NOTE`, `TODO`, and `FIXME` prefixes for meaningful comments. They help a lot to navigate the codebase.
-- Lazy imports are important to keep startup time low for tiny commands. We also do it for project imports, so not a big deal.
-- Some methods and attributes are made private to avoid polluting the public API. Feel free to access them from the outside if you know what you are doing.
-- Finally, about exact language features. f-string formatting is preferred over other syntax. Be careful with the walrus operator. Don't forget else in conditional expressions. Listen to your mom. We have no consensus about the match-case yet.
+- Consistency is the key. If you see a pattern in the codebase, follow it. Use meaningful names for variables, and avoid abbreviations.
+- Use `NOTE/TODO/FIXME` prefixes for meaningful comments, Avoid inline comments. It helps a lot in navigating the codebase.
+- Lazy imports are important to keep startup time low for tiny commands. We also do this for project imports.
+- Some methods and attributes are made private to avoid cluttering the public API. Feel free to access them from the outside if you know what you are doing.
+- Finally, about exact language features. f-string formatting is preferred over other syntax. Be careful with the walrus operator. Don't forget `else` in conditional expressions. Listen to your mom. We do not yet have a consensus on match-case.
 
 ### Changelog
 
-- All changes that affect user (developer) experience MUST be documented in the CHANGELOG.md file.
-- Changes that significantly affect DipDup maintainers' experience MAY be documented in the CHANGELOG.md file.
-- The changelog MUST conform to the "Keep a Changelog" specification (CI will break otherwise). Group order we use: Added, Fixed, Changed, Deprecated, Removed, Performance, Security, Other.
-- Lines describing changes MUST be sorted and begin with the component name (usually Python module name).
+- All changes that affect the user (developer) experience MUST be documented in the CHANGELOG.md file.
+- The changelog MUST follow the "Keep a Changelog" specification (CI will break otherwise). Group order we use: Added, Fixed, Changed, Deprecated, Removed, Performance, Security, Other.
+- Lines describing changes MUST be sorted and start with the component name (usually the Python module name).
 
 ### Documentation
 
-- A page in "Release notes" section MUST accompany all major releases. Minor releases SHOULD be documented as well. Avoid includes in Release notes pages as they are not intended to change over time.
-
-## Maintainer Guide
+- We have a pretty complex process of building and deploying docs. It starts with Markdown files in the `docs` directory. Then `scripts/docs.py` script generates several dynamic pages, API references, processes custom Cookiecutter-style macros, and so on. The resulting Markdown is pushed to the private frontend repository via the `docs.yml` GitHub Action. `docs.py` script code should answer most of your questions.
+- All public APIs MUST be documented using docstrings. We use the reStructuredText (reST) syntax.
+- A page in the "Release notes" section MUST accompany all major and minor releases. Avoid using `#include` macro in Release notes as they should not change after the publication.
 
 ### Dependencies
 
-- All dependencies MUST be declared in `pyproject.toml` file and pinned to non-breaking versions (e.g. `~1.2`).
+- All dependencies MUST be declared in `pyproject.toml` file and pinned to non-breaking versions. We are more of an application than a library, so no asterisks, please.
 
 ### Security
 
@@ -69,38 +87,32 @@ We use several tools to enforce codestyle and code quality: `black` for autoform
 - Docker images for stable releases MUST be published on Docker Hub and GitHub Container Registry.
 - Maintainers MAY publish arbitrary images on GHCR and remove them when not needed.
 
-### Installer
-
-- Installer module MUST depend on Python stdlib only.
-
 ### Scaffolding
 
 - Project templates SHOULD cover all index types available in DipDup.
 - They also MAY contain additional features and integrations.
 
 ### Demo projects
 
-- Demos are stored in `src` directory. They MUST be generated automatically from project templates using replay files.
+- Demos are stored in the `src` directory. They MUST be generated automatically from project templates using replay files.
 - Maintainers SHOULD run `pdm demos` command regularly to ensure that demo projects are up to date.
 
 ### Releases
 
 - Release versions MUST conform to [Semantic Versioning](https://semver.org/). Releases that introduce breaking changes MUST be major ones.
-- Only the latest major version is supported in general. Important fixes SHOULD be backported to the previous major release (currently, 6.5)
+- The latest major version is the only one supported in general. Important fixes SHOULD be backported to the previous major release.
+
+### Release process
 
 Releasing a new version currently requires some manual actions:
 
 1. Ensure that all GH issues and PRs are closed and linked to the milestone.
-2. Checkout to `aux/X.Y.Z` branch from `next` (or `master` for 6.5). Update DipDup version in `pyproject.toml`.
+2. Checkout to `aux/X.Y.Z` branch from `next` (or `current` for backports). Update DipDup version in `pyproject.toml`.
 3. Run `make before_release` to lock dependencies, dump `requirements.txt` files, generate demo projects etc.
 4. Commit and push all changes with msg like `Bump version X.Y.Z`. Open a PR, and link it to the milestone.
 5. Now you may want to switch Docker images of demos we host to `aux-X.Y.Z` tag as a smoke test.
 6. Merge the PR, then `git tag X.Y.Z && git push origin X.Y.Z`. Wait for `release.yml` and `docs.yml` pipelines to finish.
-
-### 6.5 branch
-
-- DipDup 6.5 is supported until March 2024. Maintainers MUST backport bugfixes from the main branch until then. All Tezos and TzKT-related code was synced with `next`, so it should be a relatively easy task.
-- 6.5 docs and installer are hosted on GH Pages at `docs.dipdup.io`.
+7. Don't forget an announcement on Twitter and Discord.
 
 ## MIT License
 

docs/8.examples/_demos_table.md

@@ -1,22 +1,22 @@
 <!-- markdownlint-disable first-line-h1 -->
 | name | network | description | source |
 |-|-|-|-|
-| demo_blank |  | Empty config for a fresh start | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_blank) |
-| demo_evm_events | EVM | ERC-20 token transfers (from event logs) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_evm_events) |
-| demo_evm_transactions | EVM | ERC-20 token transfers (from transactions) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_evm_transactions) |
-| demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_evm_uniswap) |
-| demo_starknet_events | EVM | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_starknet_events) |
-| demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_auction) |
-| demo_tezos_big_maps | Tezos | Indexing specific big maps | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_big_maps) |
-| demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_dao) |
-| demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_dex) |
-| demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_domains) |
-| demo_tezos_etherlink | Tezos | Etherlink smart rollup transactions | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_etherlink) |
-| demo_tezos_events | Tezos | Processing contract events | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_events) |
-| demo_tezos_factories | Tezos | Example of spawning indexes in runtime | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_factories) |
-| demo_tezos_head | Tezos | Processing head block metadata (realtime only) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_head) |
-| demo_tezos_nft_marketplace | Tezos | NFT marketplace (hic at nunc) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_nft_marketplace) |
-| demo_tezos_raw | Tezos | Process raw operations without filtering and typed payloads | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_raw) |
-| demo_tezos_token | Tezos | FA1.2 token contract operations | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_token) |
-| demo_tezos_token_balances | Tezos | FA1.2 token balances | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_token_balances) |
-| demo_tezos_token_transfers | Tezos | FA1.2 token transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0a1/src/demo_tezos_token_transfers) |
+| demo_blank |  | Empty config for a fresh start | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_blank) |
+| demo_evm_events | EVM | ERC-20 token transfers (from event logs) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_evm_events) |
+| demo_evm_transactions | EVM | ERC-20 token transfers (from transactions) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_evm_transactions) |
+| demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_evm_uniswap) |
+| demo_starknet_events | EVM | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_starknet_events) |
+| demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_auction) |
+| demo_tezos_big_maps | Tezos | Indexing specific big maps | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_big_maps) |
+| demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_dao) |
+| demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_dex) |
+| demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_domains) |
+| demo_tezos_etherlink | Tezos | Etherlink smart rollup transactions | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_etherlink) |
+| demo_tezos_events | Tezos | Processing contract events | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_events) |
+| demo_tezos_factories | Tezos | Example of spawning indexes in runtime | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_factories) |
+| demo_tezos_head | Tezos | Processing head block metadata (realtime only) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_head) |
+| demo_tezos_nft_marketplace | Tezos | NFT marketplace (hic at nunc) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_nft_marketplace) |
+| demo_tezos_raw | Tezos | Process raw operations without filtering and typed payloads | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_raw) |
+| demo_tezos_token | Tezos | FA1.2 token contract operations | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_token) |
+| demo_tezos_token_balances | Tezos | FA1.2 token balances | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_token_balances) |
+| demo_tezos_token_transfers | Tezos | FA1.2 token transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0b1/src/demo_tezos_token_transfers) |

docs/9.release-notes/1.v8.0.md

@@ -1,5 +1,5 @@
 ---
-title: 8.0.0a1
+title: 8.0.0b1
 description: DipDup 8.0 release notes
 ---
 
@@ -64,7 +64,7 @@ Existing projects require semi-automatic migration. Please follow the steps belo
 
 <!-- TODO: Remove arguments after dash and subdomain with a stable release -->
 1. Make sure you have Python 3.12 installed.
-2. Update the current DipDup installation. Run `dipdup self uninstall`, then `curl -Lsf https://dev.dipdup.io/install.py | python3.12 - -f -v 8.0.0a1`.
+2. Update the current DipDup installation. Run `dipdup self uninstall`, then `curl -Lsf https://dev.dipdup.io/install.py | python3.12 - -f -v 8.0.0b1`.
 3. Enter the project directory, but do not activate the virtual environment. Run the `dipdup migrate` command. It will update your config files and generate a new package structure. Modules requiring manual migration will be moved to `<module>.old` path; leave them as is for now. Review and commit the changes.
 4. Run `dipdup init --base --force` command to update pyproject.toml and other metadata files. Recreate and enter the virtual environment. For PDM run `rm -rf .venv pdm.lock && pdm venv create python3.12 && pdm install && $(pdm venv activate)`. For Poetry run `rm -rf .venv poetry.lock && poetry install && poetry shell`. Review and commit the changes.
 5. Move the callback function bodies from `<module>.old` to `<module>` files. Review and commit the changes.