Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document cml runner Bitbucket support #243

Merged
merged 6 commits into from
May 26, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions content/docs/ref/runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,6 @@ Any [generic option](/doc/ref) in addition to:

## FAQs and Known Issues

- Bitbucket: Support for
[self-hosted runners for Bitbucket Pipelines](https://support.atlassian.com/bitbucket-cloud/docs/runners)
is [coming soon](https://github.com/iterative/cml/pull/798).
- GitHub Actions by default timeout after a few hours. You can request up to
[72 hours](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#usage-limits)
via
Expand Down
75 changes: 48 additions & 27 deletions content/docs/self-hosted-runners.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
# Self-hosted (On-premise or Cloud) Runners

GitHub Actions and GitLab CI workflows are run on GitHub- and GitLab- hosted
runners by default. However, there are many great reasons to use your own
runners: to take advantage of GPUs, orchestrate your team's shared computing
resources, or train in the cloud.
GitHub Actions, GitLab CI/CD and Bitbucket Pipelines workflows are run on
GitHub-, GitLab- and Bitbucket- hosted runners by default. However, there are
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved
many great reasons to use your own runners: to take advantage of GPUs,
orchestrate your team's shared computing resources, or train in the cloud.

☝️ **Tip!** Check out the official documentation from
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved
[GitHub](https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners)
and [GitLab](https://docs.gitlab.com/runner) for more information on self-hosted
runners.

⚠️ Using
[self-hosted runners for Bitbucket Pipelines](https://support.atlassian.com/bitbucket-cloud/docs/runners)
is not yet supported.
[GitHub](https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners),
[GitLab](https://docs.gitlab.com/runner) and
[Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/runners) for more
information on self-hosted runners.
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved

## Allocating Cloud Compute Resources with CML

Expand Down Expand Up @@ -71,7 +68,7 @@ jobs:
- uses: actions/checkout@v2
- name: Train model
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
dacbd marked this conversation as resolved.
Show resolved Hide resolved
REPO_TOKEN: ${{ secrets.REPO_TOKEN }}
run: |
pip install -r requirements.txt
python train.py
Expand Down Expand Up @@ -104,17 +101,40 @@ train-model:
script:
- pip install -r requirements.txt
- python train.py
image: iterativeai/cml:0-dvc2-base1
script:
- pip install -r requirements.txt
- python train.py

# Create CML report
- cat metrics.txt >> report.md
- cml publish plot.png --md --title="Confusion Matrix" >> report.md
- cml send-comment report.md
```

</tab>
<tab title="Bitbucket">

```yaml
pipelines:
default:
- step:
image: iterativeai/cml:0-dvc2-base1
script:
- |
cml runner \
--cloud=aws \
--cloud-region=us-west \
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved
--cloud-type=p2.xlarge \
--cloud-spot \
--labels=cml-gpu
- step:
runs-on: [self.hosted, cml-gpu]
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved
image: iterativeai/cml:0-dvc2-base1-gpu
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved
script:
- pip install -r requirements.txt
- python train.py
# Create CML report
- cat metrics.txt >> report.md
- cml publish plot.png --md --title="Confusion Matrix" >> report.md
- cml send-comment report.md
```

</tab>
</toggle>

Expand Down Expand Up @@ -178,16 +198,16 @@ The `cml runner` command supports many options (see the

Sensitive values like cloud and repository credentials can be provided through
environment variables with the aid of GitHub
[secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository)
or GitLab
[masked variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project);
the latter also supports
[external secrets](https://docs.gitlab.com/ee/ci/secrets) for added security.
[secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository),
GitLab
[masked variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project)
(or [external secrets](https://docs.gitlab.com/ee/ci/secrets) for added
security), or Bitbucket
[secured user-defined variables](https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/#User-defined-variables).

⚠️ You will need to create a
[personal access token (PAT)](#personal-access-token) with repository read/write
access. In the example workflow above, this token is stored as
`PERSONAL_ACCESS_TOKEN`.
⚠️ You will need to create a [personal access token](#personal-access-token)
with enough permissions to register self-hosted runners. In the example workflow
above, this token is stored as `REPO_TOKEN`.
0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved

🛈 If using the `--cloud` option, you will also need to provide access
credentials for your cloud compute resources as secrets. In the above example,
Expand Down Expand Up @@ -314,7 +334,8 @@ Use either:

- your username and a
[Bitbucket Cloud App Password](https://bitbucket.org/account/settings/app-passwords/)
with `Write` permission for Pull requests, Pipelines, and Runners, or
with `Read` permission for Account and `Write` permission for Pull requests,
Pipelines, and Runners, or
- create a designated "CI/CD" _bot account_ for CML authentication. Bot accounts
are the same as normal user accounts, with the only difference being the
intended use case: you limit the account to only access the repositories where
Expand Down