Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add offspot org and update repos #72

Merged
merged 5 commits into from
Aug 25, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 29 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,33 @@
# Metrics for Kiwix
Kiwix & openZIM software development processes monitoring

We use [Grimoirelab](https://chaoss.github.io/grimoirelab/) to visualise several metrics on Kiwix developement evolution. Data are collected from our Kiwix and openZIM repositories, also called "projects" in Grimoirelab.

Grimoirelab uses [Elasticsearch and Kibana](https://www.elastic.co) to store data and visualise these on web dashboards. To populate effectively this database from several sources (Git, Github, Gitlab, Mediawiki, RSS, Jenkins, Slack ...), Grimoirelab has developed modules to create a [toolchain](https://chaoss.github.io/grimoirelab-tutorial/basics/components.html) to optimally update data collection. In addition to Elasticsearch, we use an SQL database (MariaDB) to store identities collected from GitHub. All of this is orchestrated with the [Grimoire-Sirmordered](https://github.com/chaoss/grimoirelab-sirmordred) daemon. We can also use micro-mordered as command line interface to manually update data.

We propose a docker instance based on a [docker image](https://github.com/chaoss/grimoirelab/tree/master/docker) provided by Grimoirelab and including Elasticksearch, MariaDB and Kibana (Dockerfile-full).
Kiwix & openZIM software development processes monitoring

We use [Grimoirelab](https://chaoss.github.io/grimoirelab/) to
visualise several metrics on Kiwix developement evolution. Data are
collected from our Kiwix and openZIM repositories, also called
"projects" in Grimoirelab.

Grimoirelab uses [Elasticsearch and Kibana](https://www.elastic.co) to
store data and visualise these on web dashboards. To populate
effectively this database from several sources (Git, Github, Gitlab,
Mediawiki, RSS, Jenkins, Slack ...), Grimoirelab has developed modules
to create a
[toolchain](https://chaoss.github.io/grimoirelab-tutorial/basics/components.html)
to optimally update data collection. In addition to Elasticsearch, we
use an SQL database (MariaDB) to store identities collected from
GitHub. All of this is orchestrated with the
[Grimoire-Sirmordered](https://github.com/chaoss/grimoirelab-sirmordred)
daemon. We can also use micro-mordered as command line interface to
manually update data.

We propose a docker instance based on a [docker
image](https://github.com/chaoss/grimoirelab/tree/master/docker)
provided by Grimoirelab and including Elasticksearch, MariaDB and
Kibana (Dockerfile-full).

## Run

```
```bash
docker run -p 127.0.0.1:9200:9200 -p 127.0.0.1:5601:5601 -e GITHUB_TOKEN=<your token> -e PROJECT_NAME='Kiwix development metrics' -v $(pwd)/logs:/logs -v $(pwd)/es-data:/var/lib/elasticsearch -t kiwix/metrics
```

Expand All @@ -32,7 +50,7 @@ The list of repositories to analyze. The first level is the project. The second

### orgs.json

Lists of known organizations.
Lists of known organizations.

### identities.yaml

Expand All @@ -51,7 +69,9 @@ The dashboard show in first. To change this dashboard, fisrt turn off read-only

`curl -X PUT "http://localhost:9200/.kibana/_settings" -H'Content-Type: application/json' -d '{ "index.blocks.read_only" : false }'`

Change the dashboard with [user interface](https://metrics.kiwix.org/app/kibana#/dashboard/Overview?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1y,mode:quick,to:now))&_a=(viewMode:edit)) Don't forget to save.
Change the dashboard with [user
interface](https://metrics.kiwix.org/app/kibana#/dashboard/Overview?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1y,mode:quick,to:now))&_a=(viewMode:edit))
Don't forget to save.

Turn on read-only mode :

Expand Down
83 changes: 62 additions & 21 deletions config/projects.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,145 +3,186 @@
"git": [
"https://github.com/kiwix/apple",
"https://github.com/kiwix/borg-backup",
"https://github.com/kiwix/cardshop",
"https://github.com/kiwix/ipfs-portal",
"https://github.com/kiwix/java-libkiwix",
"https://github.com/kiwix/kiwix-android",
"https://github.com/kiwix/kiwix-android-custom",
"https://github.com/kiwix/kiwix-build",
"https://github.com/kiwix/kiwix-desktop",
"https://github.com/kiwix/kiwix-hotspot",
"https://github.com/kiwix/kiwix-js",
"https://github.com/kiwix/kiwix-js-windows",
"https://github.com/kiwix/kiwix-lib",
"https://github.com/kiwix/kiwix-tools",
"https://github.com/kiwix/libkiwix",
"https://github.com/kiwix/maintenance",
"https://github.com/kiwix/metrics",
"https://github.com/kiwix/overview",
"https://github.com/kiwix/python_storagelib",
"https://github.com/kiwix/web"
],
"github": [
"https://github.com/kiwix/apple",
"https://github.com/kiwix/borg-backup",
"https://github.com/kiwix/cardshop",
"https://github.com/kiwix/ipfs-portal",
"https://github.com/kiwix/java-libkiwix",
"https://github.com/kiwix/kiwix-android",
"https://github.com/kiwix/kiwix-android-custom",
"https://github.com/kiwix/kiwix-build",
"https://github.com/kiwix/kiwix-desktop",
"https://github.com/kiwix/kiwix-hotspot",
"https://github.com/kiwix/kiwix-js",
"https://github.com/kiwix/kiwix-js-windows",
"https://github.com/kiwix/kiwix-lib",
"https://github.com/kiwix/kiwix-tools",
"https://github.com/kiwix/libkiwix",
"https://github.com/kiwix/maintenance",
"https://github.com/kiwix/metrics",
"https://github.com/kiwix/overview",
"https://github.com/kiwix/python_storagelib",
"https://github.com/kiwix/web"
],
"cocom": [
"https://github.com/kiwix/apple",
"https://github.com/kiwix/borg-backup",
"https://github.com/kiwix/cardshop",
"https://github.com/kiwix/ipfs-portal",
"https://github.com/kiwix/java-libkiwix",
"https://github.com/kiwix/kiwix-android",
"https://github.com/kiwix/kiwix-android-custom",
"https://github.com/kiwix/kiwix-build",
"https://github.com/kiwix/kiwix-desktop",
"https://github.com/kiwix/kiwix-hotspot",
"https://github.com/kiwix/kiwix-js",
"https://github.com/kiwix/kiwix-js-windows",
"https://github.com/kiwix/kiwix-lib",
"https://github.com/kiwix/kiwix-tools",
"https://github.com/kiwix/libkiwix",
"https://github.com/kiwix/maintenance",
"https://github.com/kiwix/metrics",
"https://github.com/kiwix/overview",
"https://github.com/kiwix/python_storagelib",
"https://github.com/kiwix/web"
]
},
"openzim": {
"git": [
"https://github.com/openzim/cms",
"https://github.com/openzim/docker-publish-action",
"https://github.com/openzim/education-numerique",
"https://github.com/openzim/gutenberg",
"https://github.com/openzim/khan-chef",
"https://github.com/openzim/kolibri2zim",
"https://github.com/openzim/libretexts-sushi-chef",
"https://github.com/openzim/librechef",
"https://github.com/openzim/libzim",
"https://github.com/openzim/mediawiki-docker",
"https://github.com/openzim/mwoffliner",
"https://github.com/openzim/nautilus",
"https://github.com/openzim/node-libzim",
"https://github.com/openzim/openedx",
"https://github.com/openzim/overview",
"https://github.com/openzim/python-libzim",
"https://github.com/openzim/python-scraperlib",
"https://github.com/openzim/python-storagelib",
"https://github.com/openzim/phet",
"https://github.com/openzim/sotoki",
"https://github.com/openzim/ted",
"https://github.com/openzim/warc2zim",
"https://github.com/openzim/wikifundi",
"https://github.com/openzim/wikihow",
"https://github.com/openzim/wp1",
"https://github.com/openzim/wp1_selection_tools",
"https://github.com/openzim/youtube",
"https://github.com/openzim/zim-requests",
"https://github.com/openzim/zim-tools",
"https://github.com/openzim/zimfarm",
"https://github.com/openzim/zimit-frontend",
"https://github.com/openzim/zim-testing-suite",
"https://github.com/openzim/zimit",
"https://github.com/openzim/zimit-frontend",
"https://github.com/openzim/zip2zim"
],
"github": [
"https://github.com/openzim/cms",
"https://github.com/openzim/docker-publish-action",
"https://github.com/openzim/education-numerique",
"https://github.com/openzim/gutenberg",
"https://github.com/openzim/khan-chef",
"https://github.com/openzim/kolibri2zim",
"https://github.com/openzim/libretexts-sushi-chef",
"https://github.com/openzim/librechef",
"https://github.com/openzim/libzim",
"https://github.com/openzim/mediawiki-docker",
"https://github.com/openzim/mwoffliner",
"https://github.com/openzim/nautilus",
"https://github.com/openzim/node-libzim",
"https://github.com/openzim/openedx",
"https://github.com/openzim/overview",
"https://github.com/openzim/python-libzim",
"https://github.com/openzim/python-scraperlib",
"https://github.com/openzim/python-storagelib",
"https://github.com/openzim/phet",
"https://github.com/openzim/sotoki",
"https://github.com/openzim/ted",
"https://github.com/openzim/warc2zim",
"https://github.com/openzim/wikifundi",
"https://github.com/openzim/wikihow",
"https://github.com/openzim/wp1",
"https://github.com/openzim/wp1_selection_tools",
"https://github.com/openzim/youtube",
"https://github.com/openzim/zim-requests",
"https://github.com/openzim/zim-tools",
"https://github.com/openzim/zimfarm",
"https://github.com/openzim/zimit-frontend",
"https://github.com/openzim/zim-testing-suite",
"https://github.com/openzim/zimit",
"https://github.com/openzim/zimit-frontend",
"https://github.com/openzim/zip2zim"
],
"cocom": [
"https://github.com/openzim/cms",
"https://github.com/openzim/docker-publish-action",
"https://github.com/openzim/education-numerique",
"https://github.com/openzim/gutenberg",
"https://github.com/openzim/khan-chef",
"https://github.com/openzim/kolibri2zim",
"https://github.com/openzim/libretexts-sushi-chef",
"https://github.com/openzim/librechef",
"https://github.com/openzim/libzim",
"https://github.com/openzim/mediawiki-docker",
"https://github.com/openzim/mwoffliner",
"https://github.com/openzim/nautilus",
"https://github.com/openzim/node-libzim",
"https://github.com/openzim/openedx",
"https://github.com/openzim/overview",
"https://github.com/openzim/python-libzim",
"https://github.com/openzim/python-scraperlib",
"https://github.com/openzim/python-storagelib",
"https://github.com/openzim/phet",
"https://github.com/openzim/sotoki",
"https://github.com/openzim/ted",
"https://github.com/openzim/warc2zim",
"https://github.com/openzim/wikifundi",
"https://github.com/openzim/wikihow",
"https://github.com/openzim/wp1",
"https://github.com/openzim/wp1_selection_tools",
"https://github.com/openzim/youtube",
"https://github.com/openzim/zim-requests",
"https://github.com/openzim/zim-tools",
"https://github.com/openzim/zimfarm",
"https://github.com/openzim/zimit-frontend",
"https://github.com/openzim/zim-testing-suite",
"https://github.com/openzim/zimit",
"https://github.com/openzim/zimit-frontend",
"https://github.com/openzim/zip2zim"
]
},
"offspot": {
"git": [
"https://github.com/offspot/cardshop",
"https://github.com/offspot/mediawiki-docker",
"https://github.com/offspot/wikifundi",
"https://github.com/offspot/operations",
"https://github.com/offspot/kiwix-hotspot",
"https://github.com/offspot/testbench"
],
"github": [
"https://github.com/offspot/cardshop",
"https://github.com/offspot/mediawiki-docker",
"https://github.com/offspot/wikifundi",
"https://github.com/offspot/operations",
"https://github.com/offspot/kiwix-hotspot",
"https://github.com/offspot/testbench"
],
"cocom": [
"https://github.com/offspot/cardshop",
"https://github.com/offspot/mediawiki-docker",
"https://github.com/offspot/wikifundi",
"https://github.com/offspot/operations",
"https://github.com/offspot/kiwix-hotspot",
"https://github.com/offspot/testbench"
]
}
}
26 changes: 26 additions & 0 deletions workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: Docker

on:
push:
branches:
- master
tags:
- v*

jobs:
build-and-push:
name: Deploy Docker Image
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- name: Build and push
uses: openzim/docker-publish-action@v5
with:
registries: ghcr.io
image-name: kiwix/metrics
on-master: latest
tag-pattern: /^v([0-9.]+)$/
restrict-to: kiwix/metrics
credentials:
GHCRIO_USERNAME=${{ secrets.GHCR_USERNAME }}
GHCRIO_TOKEN=${{ secrets.GHCR_TOKEN }}