Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SpiceDB community integration #2538

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
6381c23
Initial commit
tstirrat15 Nov 15, 2024
374f023
Add configuration spec based on cockroach spec
tstirrat15 Nov 15, 2024
0f6b69e
Regen config_models off of spec
tstirrat15 Nov 15, 2024
6f2a378
Regen example config off of spec
tstirrat15 Nov 15, 2024
0f0db17
Add metrics definition file in the style of cockroach and add checks …
tstirrat15 Nov 15, 2024
4ac152f
Copy service checks from cockroach
tstirrat15 Dec 2, 2024
c848aa9
Put spicedb version in hatch.toml
tstirrat15 Dec 2, 2024
069bcdd
Add authzed as an optional dependency
tstirrat15 Dec 2, 2024
0186d70
Add test files
tstirrat15 Dec 2, 2024
c24ff3a
Add a test util file
tstirrat15 Dec 2, 2024
3e93005
Add a docker folder
tstirrat15 Dec 2, 2024
f8d508c
Add a fixtures file
tstirrat15 Dec 2, 2024
6ae324c
Start fleshing out conftest
tstirrat15 Dec 2, 2024
5472127
Add a todo
tstirrat15 Dec 2, 2024
d785a02
Finish fleshing out metrics
tstirrat15 Dec 2, 2024
6773455
Set histogram configuration by default
tstirrat15 Dec 2, 2024
707d172
Run ruff format
tstirrat15 Dec 2, 2024
4dc4a33
Make histogram as distribution on by default
tstirrat15 Dec 2, 2024
54df4c9
Fix reference
tstirrat15 Dec 3, 2024
f4ee182
Fix test error by disabling chekc
tstirrat15 Dec 3, 2024
fd9e314
Re-add deps to keep hatch happy
tstirrat15 Dec 3, 2024
729536f
Fix references
tstirrat15 Dec 3, 2024
c64b4b0
Add get_expected_metrics
tstirrat15 Dec 3, 2024
7e95ded
use get_expected_metrics
tstirrat15 Dec 3, 2024
6803970
Reorganize metrics
tstirrat15 Dec 3, 2024
b8e53a1
Fill out metadata csv
tstirrat15 Dec 4, 2024
c3892e0
Reorganize metrics to provide types
tstirrat15 Dec 4, 2024
d86185c
Fix tag name
tstirrat15 Dec 4, 2024
a596a28
Get unit tests running
tstirrat15 Dec 4, 2024
8cd73f5
Fix port, split up histogram and non-histogram metrics
tstirrat15 Dec 4, 2024
d1bf884
Fix fixture invocation and key
tstirrat15 Dec 4, 2024
f49eed6
Add integration test
tstirrat15 Dec 4, 2024
3d555ab
Add e2e test
tstirrat15 Dec 4, 2024
8418320
Add fleshed-out dockerfile
tstirrat15 Dec 4, 2024
5391196
Sync ci
tstirrat15 Dec 4, 2024
a016e2a
Undo formatting changse
tstirrat15 Dec 4, 2024
82c12b8
Add images to repo
tstirrat15 Dec 4, 2024
0f0d3c9
Add TODO
tstirrat15 Dec 4, 2024
435e8bc
Add codeowner line for spicedb
tstirrat15 Dec 4, 2024
9c13b90
Add starter dashboard
tstirrat15 Dec 4, 2024
038a183
Flesh out spicedb manifest file
tstirrat15 Dec 4, 2024
9cea1e4
Shorten description
tstirrat15 Dec 4, 2024
ea185e7
Fix unit
tstirrat15 Dec 5, 2024
b80596e
Flesh out readme
tstirrat15 Dec 5, 2024
b9e6393
Update spicedb/README.md
tstirrat15 Dec 6, 2024
2ee3c3b
Update spicedb/README.md
tstirrat15 Dec 6, 2024
ac77f8d
Update spicedb/README.md
tstirrat15 Dec 6, 2024
a1cfb59
Update spicedb/README.md
tstirrat15 Dec 6, 2024
3023db1
Update spicedb/README.md
tstirrat15 Dec 6, 2024
ef9ec6b
Format dashboard
tstirrat15 Dec 6, 2024
3718eba
Remove readme
tstirrat15 Dec 6, 2024
da8ae0b
Update spicedb/README.md
tstirrat15 Dec 7, 2024
8cfe890
Update spicedb/CHANGELOG.md
tstirrat15 Dec 7, 2024
8093de5
Use open-source image for thumper
tstirrat15 Dec 12, 2024
d54b74f
Merge branch 'add-spicedb-integration' of github.com:tstirrat15/integ…
tstirrat15 Dec 12, 2024
1867002
Fix merge conflicts
tstirrat15 Dec 12, 2024
7df9e90
Run black
tstirrat15 Dec 18, 2024
8df39f9
Ruff as well
tstirrat15 Dec 18, 2024
c2735ba
Fix import sort
tstirrat15 Dec 18, 2024
31c683b
Add noqa to stop ruff and black from fighting
tstirrat15 Dec 18, 2024
8ba8171
Last one
tstirrat15 Dec 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,10 @@ coverage:
target: 75
flags:
- sortdb
SpiceDB:
target: 75
flags:
- spicedb
Stardog:
target: 70
flags:
Expand Down Expand Up @@ -613,6 +617,11 @@ flags:
paths:
- speedtest/datadog_checks/speedtest
- speedtest/tests
spicedb:
carryforward: true
paths:
- spicedb/datadog_checks/spicedb
- spicedb/tests
stardog:
carryforward: true
paths:
Expand Down
2 changes: 2 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -1147,6 +1147,8 @@

/oceanbasecloud/ @kyle-oceanbase [email protected] @DataDog/ecosystems-review

/spicedb/ @tstirrat15 @DataDog/ecosystems-review

# LEAVE THE FOLLOWING LOG OWNERSHIP LAST IN THE FILE
# Make sure logs team is the full owner for all logs related files
**/assets/logs/ @DataDog/logs-backend
19 changes: 19 additions & 0 deletions .github/workflows/test-all.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1227,6 +1227,25 @@ jobs:
test-py3: ${{ inputs.test-py3 }}
setup-env-vars: "${{ inputs.setup-env-vars }}"
secrets: inherit
jc9d4106:
uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
with:
job-name: SpiceDB
target: spicedb
platform: linux
runner: '["ubuntu-22.04"]'
repo: "${{ inputs.repo }}"
python-version: "${{ inputs.python-version }}"
standard: ${{ inputs.standard }}
latest: ${{ inputs.latest }}
agent-image: "${{ inputs.agent-image }}"
agent-image-py2: "${{ inputs.agent-image-py2 }}"
agent-image-windows: "${{ inputs.agent-image-windows }}"
agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
test-py2: ${{ inputs.test-py2 }}
test-py3: ${{ inputs.test-py3 }}
setup-env-vars: "${{ inputs.setup-env-vars }}"
secrets: inherit
je0445d5:
uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
with:
Expand Down
8 changes: 8 additions & 0 deletions spicedb/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# CHANGELOG - SpiceDB

## 1.0.0 / YYYY-MM-DD
tstirrat15 marked this conversation as resolved.
Show resolved Hide resolved

***Added***:

* Initial Release

77 changes: 77 additions & 0 deletions spicedb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Agent Check: SpiceDB

## Overview

[SpiceDB][1]is an open source, [Google Zanzibar][zanzibar]-inspired database system for creating and managing security-critical application permissions.
tstirrat15 marked this conversation as resolved.
Show resolved Hide resolved

Developers create a schema that models their permissions requirements. Then, they use any of the official or community maintained client libraries to apply the schema to the database and insert data into the database. They can query the data to efficiently check permissions in their applications.

SpiceDB metrics allow developers and SREs to monitor their SpiceDB deployments, including request latency metrics, cache metrics such as size and hit/miss metrics, and datastore connection and query metrics. These metrics allow developers and SREs to diagnose performance problems and tune performance characteristics of their SpiceDB clusters.

## Setup

### Installation

#### Host-level installation
To install the SpiceDB check on your host:

1. Download and install the [Datadog Agent][2].
1. Install the SpiceDB integration on the Agent on the host where it's running.
```shell
datadog-agent integration install -t datadog-spicedb==<INTEGRATION_VERSION>
```

#### With the Agent sidecar container
1. Build a custom image with the Agent installed:
```dockerfile
FROM gcr.io/datadoghq/agent:latest
RUN agent integration install -t datadog-spicedb==<INTEGRATION_VERSION>
# Optionally include the configuration in the image
COPY spicedb.yaml /conf.d/spicedb.d/conf.yaml
```
1. Deploy that image as a sidecar (or whatever makes sense for your topology).

### Configuration

A full list of configuration options are available in the [example configuration][example-conf].
This integration wraps the Datadog Openmetrics configuration, so the [openmetrics documentation][openmentrics-docs]
may be helpful as well.

### Validation

[Run the Agent's status subcommand][6] and look for `spicedb` under the Checks section.

## Data Collected

### Metrics

The integration collects metrics through the [SpiceDB prometheus endpoint][spicedb-prometheus].

For a full list of metrics this integration provides, see the [metadata.csv][metadata.csv] file.

### Service Checks

Service check `spicedb.openmetrics.health` is submitted in the base check.

### Events

SpiceDB does not include any events.

## Troubleshooting

Need help? Contact [Datadog support][3].

[1]: https://authzed.com/spicedb
[2]: https://app.datadoghq.com/account/settings/agent/latest
[3]: https://docs.datadoghq.com/agent/kubernetes/integrations/
[4]: https://github.com/DataDog/integrations-extras/blob/master/spicedb/datadog_checks/spicedb/data/conf.yaml.example
[5]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
[6]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
[7]: https://github.com/DataDog/integrations-extras/blob/master/spicedb/metadata.csv
[8]: https://github.com/DataDog/integrations-extras/blob/master/spicedb/assets/service_checks.json
[9]: https://docs.datadoghq.com/help/
[zanzibar]: https://authzed.com/zanzibar
[metadata.csv]: https://github.com/DataDog/integrations-extras/blob/master/spicedb/metadata.csv
[example-conf]: https://github.com/DataDog/integrations-extras/blob/master/spicedb/datadog_checks/spicedb/data/conf.yaml.example
[spicedb-prometheus]: https://authzed.com/docs/spicedb/ops/observability#prometheus
[openmetrics-docs]: https://docs.datadoghq.com/integrations/openmetrics/
22 changes: 22 additions & 0 deletions spicedb/assets/configuration/spec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
name: "SpiceDB"
files:
- name: "spicedb.yaml"
options:
- template: "init_config"
options:
- template: "init_config/openmetrics"
- template: "instances"
options:
- template: "instances/openmetrics"
overrides:
openmetrics_endpoint.required: false
openmetrics_endpoint.value.example: "http://localhost:9090/metrics"
# This is required to make pXX graphs in the default dashboard.
histogram_buckets_as_distributions.value.example: true
tags.value.example:
- "node:<NODE>"
- "cluster:<CLUSTER>"
- "region:<REGION>"
- "<KEY_1>:<VALUE_1>"
- "<KEY_2>:<VALUE_2>"
Loading
Loading