Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(#27): address ambiguity for score fields, allow assign the priority manually #29

Merged
merged 21 commits into from
Jun 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ jobs:

services:
mongodb:
image: mongo:6.0
image: mongo:latest
ports:
- 27017:27017

redis:
image: redis:7.0
image: redis:latest
ports:
- 6379:6379

Expand Down
67 changes: 67 additions & 0 deletions .github/workflows/test_engine_compatibility.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Test Engine Compatibility

on:
pull_request:
branches:
- main
push:
branches:
- main

env:
GO_VERSION: 1.18

permissions:
checks: write
contents: read

jobs:
test-compatibility:
runs-on: ubuntu-latest

services:
mongodb:
image: mongo:5.0.18
ports:
- '27017:27017'

redis:
image: redis:5.0.14
ports:
- 6379:6379

steps:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: ${{ env.GO_VERSION }}

- uses: actions/checkout@v3

- name: Cache Go modules
uses: actions/cache@v3
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: ${{ runner.os }}-golang-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-golang-

- name: Install mockgen
run: go install github.com/golang/mock/mockgen@latest

- name: Install gotestsum
run: go install gotest.tools/gotestsum@latest

- name: Generate Mocks
run: make gen-mocks

- name: Generate TLS certificates
run: make gen-cert

- name: Go mod tidy
run: go mod tidy

- name: Make test
run: make integration-test
5 changes: 5 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# TODO: configure linter correctly
# Disabling staticcheck since we deprecated proto field
linters:
disable:
- staticcheck
39 changes: 16 additions & 23 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,22 @@ You may also download the latest release from the [releases](https://github.com/
>
> To change the default configuration see the [configuration section](/README.md?#configuration).

### Mocks

We use the [GoMock](https://github.com/golang/mock) project to generate mocks.

To update the mocks you must run the following command:
```shell
make gen-mocks
```

Mocks are generated in the [/internal/mocks](/internal/mocks) folder.

When creating interfaces with the need to generate mocks, you must add the following directive to the interface file:
```go
//go:generate mockgen -destination=<path_to_mocks>/mock_<file>.go -package=mocks -source=<file>.go
```

## Running tests

To run project tests you must first generate all mock files with the following command:
Expand Down Expand Up @@ -202,29 +218,6 @@ make gen-cert
>
> Check the [generation script](/internal/service/cert/gen.sh) for more details.

### Mocks

We use the [GoMock](https://github.com/golang/mock) project to generate mocks.

To update the mocks you must run the following command:
```shell
make gen-mocks
```

Mocks are generated in the [/internal/mocks](/internal/mocks) folder.

When creating interfaces with the need to generate mocks, you must add the following directive to the interface file:
```go
//go:generate mockgen -destination=<path_to_mocks>/mock_<file>.go -package=mocks -source=<file>.go
```

## Documentation

If you made any changes in the `.proto` file, you must generate the documentation with the following command:
```shell
make gen-docs
```

## Docker Image

To generate the Deckard image we use the [ko](https://github.com/google/ko) tool.
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,17 @@

[![Slack](https://img.shields.io/badge/slack-Gophers_%28Deckard%29-blue?logo=slack&link=https://gophers.slack.com/archives/C05E1TMS1FY)](https://gophers.slack.com/archives/C05E1TMS1FY)

Deckard is a priority queue system inspired by projects such as Google Cloud PubSub, Nats, Kafka, and others. Its main distinction lies in its ability to associate a priority with each message and have a queue that can be optionally cyclic. This means that messages can be delivered again after a user-managed time. Additionally, Deckard implements a locking mechanism to prevent message processing for a specified duration.
Deckard is a priority queue system inspired by projects such as Google Cloud PubSub, Nats, Kafka, and others. Its main distinction lies in its ability to associate a priority score with each message and have a queue that can be optionally cyclic. This means that messages can be delivered again after a user-managed time. Additionally, Deckard implements a locking mechanism to prevent message processing for a specified duration.

![deckard](docs/deckard_cartoon.webp)

Briefly:
- An application inserts a message to be queued and its configuration (TTL, metadata, payload, etc).
- The message will be prioritized with a default timestamp-based algorithm. The priority can also be provided by the application.
- An application inserts a message to be queued and its configuration (TTL, metadata, payload, priority score, etc).
- The message will be prioritized with a default timestamp-based algorithm if the provided score is 0 (the default value).
- A worker application pull messages from Deckard at regular intervals and performs any processing.
- When it finishes processing a message, the application must notify with the processing result.
- When notifying, the application may provide a lock time, to lock the message for a certain duration of time before being requeued and delivered again.
- It is also possible to notify a message changing its priority.
- It is also possible to notify a message changing its priority score.
- When the message's TTL is reached, it stops being delivered;
- For some use cases the TTL can be set as infinite.
- An application can also remove the message when notifying.
Expand Down
Loading