-
Notifications
You must be signed in to change notification settings - Fork 104
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Add document to setup snapshotter by daemonset
Signed-off-by: ChengyuZhu6 <[email protected]>
- Loading branch information
ChengyuZhu6
committed
Dec 19, 2023
1 parent
63470dd
commit 2210061
Showing
1 changed file
with
248 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| # Setup Nydus Snapshotter by DaemonSet | ||
|
|
||
| This document will guide you through the simple steps of setting up and cleaning up the nydus snapshotter in a kubernetes cluster that runs on the host. | ||
|
|
||
| ## Steps for Setting up Nydus Snapshotter | ||
|
|
||
| To begin, let's clone the Nydus Snapshotter repository. | ||
|
|
||
| ```bash | ||
| git clone https://github.com/containerd/nydus-snapshotter | ||
| cd nydus-snapshotter | ||
| ``` | ||
|
|
||
| Next, we can configure access control for nydus snapshotter. | ||
| ```bash | ||
| kubectl apply -f nydus-snapshotter-rbac.yaml | ||
| ``` | ||
|
|
||
| Afterward, we can deploy a DaemonSet for nydus snapshotter. | ||
|
|
||
| ```bash | ||
| kubectl apply -f nydus-snapshotter.yaml | ||
| ``` | ||
|
|
||
| Then, we can confirm that nydus snapshotter is running through the DaemonSet. | ||
| ```bash | ||
| $ kubectl get pods -n nydus-snapshotter | ||
| NAME READY STATUS RESTARTS AGE | ||
| nydus-snapshotter-bfhmr 1/1 Running 0 18s | ||
| ``` | ||
|
|
||
| Finally, we can view the logs in the pod. | ||
| ```bash | ||
| $ kubectl logs nydus-snapshotter-bfhmr -n nydus-snapshotter | ||
| there is no proxy plugin! | ||
| time="2023-12-19T23:15:04.315888431Z" level=info msg="Start nydus-snapshotter. Version: v0.7.0-306-gde77420.m, PID: 1505719, FsDriver: fusedev, DaemonMode: dedicated" | ||
| time="2023-12-19T23:15:04.317256726Z" level=info msg="Run daemons monitor..." | ||
| ``` | ||
|
|
||
| ## Steps for Cleaning up Nydus Snapshotter | ||
|
|
||
| We use `preStop`` hook in the DaemonSet to uninstall nydus snapshotter and roll back the containerd configuration. | ||
|
|
||
| ```bash | ||
| $ kubectl delete -f nydus-snapshotter.yaml | ||
| $ kubectl delete -f nydus-snapshotter-rbac.yaml | ||
| $ systemd restart containerd.service | ||
| ``` | ||
|
|
||
| ## Customized Setup | ||
|
|
||
| As we know, nydus snapshotter supports four filesystem drivers (fs_driver): `fusedev`, `fscache`, `blockdev`, `proxy`. Within the container image, we have included configurations for these snapshotter drivers, as well as the corresponding nydusd configurations. By default, the fusedev driver is enabled in the nydus snapshotter, using the snapshotter configuration [`config-fusedev.toml`](../misc/snapshotter/config-fusedev.toml) and the nydusd configuration [`nydusd-config.fusedev.json`](../misc/snapshotter/nydusd-config.fusedev.json). | ||
|
|
||
| ### Other filesystem driver with related default configuration | ||
|
|
||
| If we want to setup the nydus snapshotter with the default configuration for different fs_driver (such as `proxy`), we can modify the values in the `Configmap` in `nydus-snapshotter.yaml`: | ||
| ```yaml | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: nydus-snapshotter-configs | ||
| labels: | ||
| app: nydus-snapshotter | ||
| namespace: nydus-snapshotter | ||
| data: | ||
| FS_DRIVER: "proxy" | ||
| NYDUSD_DAEMON_MODE: "none" | ||
| ``` | ||
| Then we can run the nydus snapshotter enabling `proxy` `fs_driver` with the snapshotter configuration [`config-proxy.toml`](../misc/snapshotter/config-proxy.toml). | ||
|
|
||
| **NOTE:** The fs_driver (`blockdev` and `proxy`) do not need nydusd, so they do not need nydusd config. | ||
|
|
||
| ### Same filesystem with different nydusd configuration | ||
|
|
||
| If we want to setup the nydus snapshotter for the same fs_driver (such as `fusedev`) with same snapshotter configuration and different nydusd configuration, we can enable `ENABLE_NYDUSD_CONFIG_FROM_VOLUME` and add the nydusd configuration (named `nydusd-config.json`) in the `Configmap` in `nydus-snapshotter.yaml`: | ||
|
|
||
| ```yaml | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: nydus-snapshotter-configs | ||
| labels: | ||
| app: nydus-snapshotter | ||
| namespace: nydus-snapshotter | ||
| data: | ||
| FS_DRIVER: "fusedev" | ||
| ENABLE_NYDUSD_CONFIG_FROM_VOLUME: "true" | ||
| nydusd-config.json: |- | ||
| { | ||
| "device": { | ||
| "backend": { | ||
| "type": "registry", | ||
| "config": { | ||
| "timeout": 5, | ||
| "connect_timeout": 5, | ||
| "retry_limit": 2 | ||
| } | ||
| }, | ||
| "cache": { | ||
| "type": "blobcache" | ||
| } | ||
| }, | ||
| "mode": "direct", | ||
| "digest_validate": false, | ||
| "iostats_files": false, | ||
| "enable_xattr": true, | ||
| "amplify_io": 1048576, | ||
| "fs_prefetch": { | ||
| "enable": true, | ||
| "threads_count": 8, | ||
| "merging_size": 1048576, | ||
| "prefetch_all": true | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Same filesystem with different snapshotter configuration | ||
|
|
||
| If we want to setup the nydus snapshotter for the same fs_driver (such as `fusedev`) with different snapshotter configuration and same nydusd configuration, we can enable `ENABLE_SNAPSHOTTER_CONFIG_FROM_VOLUME` and add the snapshotter configuration (named `config.toml`) in the `Configmap` in `nydus-snapshotter.yaml`: | ||
|
|
||
| ```yaml | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: nydus-snapshotter-configs | ||
| labels: | ||
| app: nydus-snapshotter | ||
| namespace: nydus-snapshotter | ||
| data: | ||
| ENABLE_SNAPSHOTTER_CONFIG_FROM_VOLUME: "true" | ||
| config.toml: |- | ||
| version = 1 | ||
| # Snapshotter's own home directory where it stores and creates necessary resources | ||
| root = "/var/lib/containerd-nydus" | ||
| # The snapshotter's GRPC server socket, containerd will connect to plugin on this socket | ||
| address = "/run/containerd-nydus/containerd-nydus-grpc.sock" | ||
| # The nydus daemon mode can be one of the following options: multiple, dedicated, shared, or none. | ||
| # If `daemon_mode` option is not specified, the default value is multiple. | ||
| daemon_mode = "multiple" | ||
|
|
||
| [daemon] | ||
| # Specify a configuration file for nydusd | ||
| nydusd_config = "/etc/nydus/nydusd-fusedev.json" | ||
| nydusd_path = "/usr/bin/nydusd" | ||
| nydusimage_path = "/usr/bin/nydus-image" | ||
| # fusedev or fscache | ||
| fs_driver = "fusedev" | ||
|
|
||
| [log] | ||
| # Print logs to stdout rather than logging files | ||
| log_to_stdout = true | ||
| # Snapshotter's log level | ||
| level = "info" | ||
| ``` | ||
|
|
||
| ### Same filesystem with different snapshotter configuration and different nydusd configuration | ||
|
|
||
| If we want to setup the nydus snapshotter for the same fs_driver (such as `fusedev`) with different snapshotter configuration and different nydusd configuration, we can enable `ENABLE_SNAPSHOTTER_CONFIG_FROM_VOLUME` and add the snapshotter configuration (named `config.toml`) in the `Configmap` in `nydus-snapshotter.yaml`: | ||
|
|
||
| ```yaml | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: nydus-snapshotter-configs | ||
| labels: | ||
| app: nydus-snapshotter | ||
| namespace: nydus-snapshotter | ||
| data: | ||
| ENABLE_SNAPSHOTTER_CONFIG_FROM_VOLUME: "true" | ||
|
|
||
| config.toml: |- | ||
| version = 1 | ||
| # Snapshotter's own home directory where it stores and creates necessary resources | ||
| root = "/var/lib/containerd-nydus" | ||
| # The snapshotter's GRPC server socket, containerd will connect to plugin on this socket | ||
| address = "/run/containerd-nydus/containerd-nydus-grpc.sock" | ||
| # The nydus daemon mode can be one of the following options: multiple, dedicated, shared, or none. | ||
| # If `daemon_mode` option is not specified, the default value is multiple. | ||
| daemon_mode = "multiple" | ||
| [daemon] | ||
| # Specify a configuration file for nydusd | ||
| nydusd_config = "/etc/nydus-snapshotter/nydusd-config.json" | ||
| nydusd_path = "/usr/bin/nydusd" | ||
| nydusimage_path = "/usr/bin/nydus-image" | ||
| # fusedev or fscache | ||
| fs_driver = "fusedev" | ||
| [log] | ||
| # Print logs to stdout rather than logging files | ||
| log_to_stdout = true | ||
| # Snapshotter's log level | ||
| level = "info" | ||
| nydusd-config.json: |- | ||
| { | ||
| "device": { | ||
| "backend": { | ||
| "type": "registry", | ||
| "config": { | ||
| "timeout": 5, | ||
| "connect_timeout": 5, | ||
| "retry_limit": 2 | ||
| } | ||
| }, | ||
| "cache": { | ||
| "type": "blobcache" | ||
| } | ||
| }, | ||
| "mode": "direct", | ||
| "digest_validate": false, | ||
| "iostats_files": false, | ||
| "enable_xattr": true, | ||
| "amplify_io": 1048576, | ||
| "fs_prefetch": { | ||
| "enable": true, | ||
| "threads_count": 8, | ||
| "merging_size": 1048576, | ||
| "prefetch_all": true | ||
| } | ||
| } | ||
| ``` | ||
| **NOTE:** We need to set `nydusd_config` to `/etc/nydus-snapshotter/nydusd-config.json` in the `config.toml`, so that snapshotter can find the nydusd configuration from configmap. | ||
|
|
||
| ### Customized Options | ||
|
|
||
| | Options | Type | Default |Comment | ||
| | --------------- | --------------- | --------------- | --------------- | | ||
| | FS_DRIVER | string | "fusedev" | the filesystem driver of snapshotter| | ||
| | LOG_LEVEL | string | "info" | logging level | | ||
| | NYDUSD_DAEMON_MODE | string | "multiple" | nydusd daemon mode | | ||
| | ENABLE_KATA_VOLUME | bool | true | enabling to construct kata virtual volume, only worked when `fs_driver`=`blockdev` | | ||
| | ENABLE_TARFS | bool | true | enabling to convert image to tarfs, only worked when `fs_driver`=`blockdev` | | ||
| | MOUNT_TARFS_ON_HOST | bool | true | enabling to mount tarfs on the host | | ||
| | EXPORT_MODE | string | "image_block_with_verity" | enabling to export an image to one or more disk images | | ||
| | NYDUSD_CONFIG | string | "/etc/nydus/nydusd-fusedev.json" | path to the nydusd configuration | | ||
| | SNAPSHOTTER_CONFIG | string | "/etc/nydus/config-fusdev.toml" | path to the snapshotter configuration | | ||
| | ENABLE_SNAPSHOTTER_CONFIG_FROM_VOLUME | bool | false | enabling to use the snapshotter configuration from volume | | ||
| | ENABLE_NYDUSD_CONFIG_FROM_VOLUME | bool | false | enabling to use the nydusd configuration from volume | | ||
| | ENABLE_RUNTIME_SPECIFIC_SNAPSHOTTER | bool | false | enabling to skip to set `plugins."io.containerd.grpc.v1.cri".containerd` to `nydus` for runtime specific snapshotter feature in containerd 1.7+ | |