| sidebar_position | title | description |
|---|---|---|
2 |
Config Examples |
Example configs with explainations |
import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
NativeLink uses JSON files for configuration of the CAS and Scheduler.
This section will break down all of the customizable fields in these JSON files so they can be modified to suit your needs.
The Basic CAS configuration is ideal for running NativeLink locally. It provides a straightforward setup that is easy to understand and implement, making it perfect for development and testing environments.
```json5 { "stores": { "AC_MAIN_STORE": { "filesystem": { "content_path": "/tmp/nativelink/data-worker-test/content_path-ac", "temp_path": "/tmp/nativelink/data-worker-test/tmp_path-ac", "eviction_policy": { // 1gb. "max_bytes": 1000000000, } } }, "WORKER_FAST_SLOW_STORE": { "fast_slow": { // "fast" must be a "filesystem" store because the worker uses it to make // hardlinks on disk to a directory where the jobs are running. "fast": { "filesystem": { "content_path": "/tmp/nativelink/data-worker-test/content_path-cas", "temp_path": "/tmp/nativelink/data-worker-test/tmp_path-cas", "eviction_policy": { // 10gb. "max_bytes": 10000000000, } } }, "slow": { /// Discard data. /// This example usage has the CAS and the Worker live in the same place, /// so they share the same underlying CAS. Since workers require a fast_slow /// store, we use the fast store as our primary data store, and the slow store /// is just a noop, since there's no shared storage in this config. "noop": {} } } } }, "schedulers": { "MAIN_SCHEDULER": { "simple": { "supported_platform_properties": { "cpu_count": "minimum", "memory_kb": "minimum", "network_kbps": "minimum", "disk_read_iops": "minimum", "disk_read_bps": "minimum", "disk_write_iops": "minimum", "disk_write_bps": "minimum", "shm_size": "minimum", "gpu_count": "minimum", "gpu_model": "exact", "cpu_vendor": "exact", "cpu_arch": "exact", "cpu_model": "exact", "kernel_version": "exact", "OSFamily": "priority", "container-image": "priority", // Example of how to set which docker images are available and set // them in the platform properties. // "docker_image": "priority", } } } }, "workers": [{ "local": { "worker_api_endpoint": { "uri": "grpc://127.0.0.1:50061", }, "cas_fast_slow_store": "WORKER_FAST_SLOW_STORE", "upload_action_result": { "ac_store": "AC_MAIN_STORE", }, "work_directory": "/tmp/nativelink/work", "platform_properties": { "cpu_count": { "values": ["16"], }, "memory_kb": { "values": ["500000"], }, "network_kbps": { "values": ["100000"], }, "cpu_arch": { "values": ["x86_64"], }, "OSFamily": { "values": [""] }, "container-image": { "values": [""] }, // Example of how to set which docker images are available and set // them in the platform properties. // "docker_image": { // "query_cmd": "docker images --format {{.Repository}}:{{.Tag}}", // } } } }], "servers": [{ "name": "public", "listener": { "http": { "socket_address": "0.0.0.0:50051" } }, "services": { "cas": { "main": { "cas_store": "WORKER_FAST_SLOW_STORE" } }, "ac": { "main": { "ac_store": "AC_MAIN_STORE" } }, "execution": { "main": { "cas_store": "WORKER_FAST_SLOW_STORE", "scheduler": "MAIN_SCHEDULER", } }, "capabilities": { "main": { "remote_execution": { "scheduler": "MAIN_SCHEDULER", } } }, "bytestream": { "cas_stores": { "main": "WORKER_FAST_SLOW_STORE", } } } }, { "name": "private_workers_servers", "listener": { "http": { "socket_address": "0.0.0.0:50061" } }, "services": { "experimental_prometheus": { "path": "/metrics" }, // Note: This should be served on a different port, because it has // a different permission set than the other services. // In other words, this service is a backend api. The ones above // are a frontend api. "worker_api": { "scheduler": "MAIN_SCHEDULER", }, "admin": {}, "health": {}, } }], "global": { "max_open_files": 512 } } ```
The Filesystem CAS configuration serves as a robust local alternative to the Basic CAS. It is designed to store all data on disk, allowing for service restarts without data loss. This setup is optimized for performance with features like compression and deduplication. Additionally, it can be used in a cloud environment, providing flexibility for both local and cloud-based deployments.
```json5 // This configuration will place objects in various folders in // `/tmp/nativelink/data`. It will store all data on disk and // allows for restarts of the underlying service. It is optimized // so objects are compressed, deduplicated and uses some in-memory // optimizations for certain hot paths. { "stores": { "FS_CONTENT_STORE": { "compression": { "compression_algorithm": { "lz4": {} }, "backend": { "filesystem": { "content_path": "/tmp/nativelink/data/content_path-cas", "temp_path": "/tmp/nativelink/data/tmp_path-cas", "eviction_policy": { // 2gb. "max_bytes": 2000000000, } } } } }, "CAS_MAIN_STORE": { "verify": { "backend": { // Because we are using a dedup store, we can bypass small objects // and put those objects directly into the content store without // having to be indexed. This greatly improves performance of serving // general build content, since many objects are quite small and by // putting this size distinguish store in place will prevent 1+ index // read/write per small object request. "size_partitioning": { "size": 262144, // 256k. "lower_store": { "ref_store": { "name": "FS_CONTENT_STORE" } }, "upper_store": { "dedup": { "index_store": { // Since our index store is queried so much, we use a fast_slow // store so it will keep in memory objects that are accessed // frequently before going to disk. // Note: indexes are generally quite small, but accessed frequently. "fast_slow": { "fast": { "memory": { "eviction_policy": { // 10mb. "max_bytes": 10000000, } } }, "slow": { "filesystem": { "content_path": "/tmp/nativelink/data/content_path-index", "temp_path": "/tmp/nativelink/data/tmp_path-index", "eviction_policy": { // 500mb. "max_bytes": 500000000, } } } } }, "content_store": { "ref_store": { "name": "FS_CONTENT_STORE" } } } } } }, "verify_size": true, "hash_verification_function": "sha256" } }, "AC_MAIN_STORE": { "filesystem": { "content_path": "/tmp/nativelink/data/content_path-ac", "temp_path": "/tmp/nativelink/data/tmp_path-ac", "eviction_policy": { // 500mb. "max_bytes": 500000000, } } } }, "schedulers": { "MAIN_SCHEDULER": { "simple": { "supported_platform_properties": { "cpu_count": "minimum", "memory_kb": "minimum", "network_kbps": "minimum", "disk_read_iops": "minimum", "disk_read_bps": "minimum", "disk_write_iops": "minimum", "disk_write_bps": "minimum", "shm_size": "minimum", "gpu_count": "minimum", "gpu_model": "exact", "cpu_vendor": "exact", "cpu_arch": "exact", "cpu_model": "exact", "kernel_version": "exact", "docker_image": "priority", } } } }, "servers": [{ "listener": { "http": { "socket_address": "0.0.0.0:50051" } }, "services": { "cas": { "main": { "cas_store": "CAS_MAIN_STORE" } }, "ac": { "main": { "ac_store": "AC_MAIN_STORE" } }, "execution": { "main": { "cas_store": "CAS_MAIN_STORE", "scheduler": "MAIN_SCHEDULER", } }, "capabilities": { "main": { "remote_execution": { "scheduler": "MAIN_SCHEDULER", } } }, "bytestream": { "cas_stores": { "main": "CAS_MAIN_STORE", } } } }, { "listener": { "http": { "socket_address": "0.0.0.0:50061" } }, "services": { // Note: This should be served on a different port, because it has // a different permission set than the other services. // In other words, this service is a backend api. The ones above // are a frontend api. "worker_api": { "scheduler": "MAIN_SCHEDULER", }, "health": {}, } }] }
</p>
## S3 Backend with Local Fast CAS
The S3 Backend configuration is an excellent choice when transitioning to a production-grade
hosted cloud environment. It leverages Amazon S3 for scalable and reliable storage,
combined with a local fast CAS for a simple performant solution.
<p style={{
maxHeight: '600px',
overflowY: 'auto',
borderRadius: '10px 10px 10px 10px'
}}>
```json5
{
"stores": {
"CAS_MAIN_STORE": {
"verify": {
"backend": {
"dedup": {
"index_store": {
"fast_slow": {
"fast": {
"filesystem": {
"content_path": "/tmp/nativelink/data/content_path-index",
"temp_path": "/tmp/nativelink/data/tmp_path-index",
"eviction_policy": {
// 500mb.
"max_bytes": 500000000,
}
}
},
"slow": {
"experimental_s3_store": {
"region": "eu-north-1",
"bucket": "crossplane-bucket-af79aeca9",
"key_prefix": "test-prefix-index/",
"retry": {
"max_retries": 6,
"delay": 0.3,
"jitter": 0.5,
},
"additional_max_concurrent_requests": 10
}
}
}
},
"content_store": {
"compression": {
"compression_algorithm": {
"lz4": {}
},
"backend": {
"fast_slow": {
"fast": {
"filesystem": {
"content_path": "/tmp/nativelink/data/content_path-content",
"temp_path": "/tmp/nativelink/data/tmp_path-content",
"eviction_policy": {
// 2gb.
"max_bytes": 2000000000,
}
}
},
"slow": {
"experimental_s3_store": {
"region": "eu-north-1",
"bucket": "crossplane-bucket-af79aeca9",
"key_prefix": "test-prefix-dedup-cas/",
"retry": {
"max_retries": 6,
"delay": 0.3,
"jitter": 0.5,
},
"additional_max_concurrent_requests": 10
}
}
}
}
}
}
}
},
"verify_size": true,
"hash_verification_function": "sha256"
}
},
"AC_MAIN_STORE": {
"fast_slow": {
"fast": {
"memory": {
"eviction_policy": {
// 100mb.
"max_bytes": 100000000,
}
},
"filesystem": {
"content_path": "/tmp/nativelink/data/content_path-ac",
"temp_path": "/tmp/nativelink/data/tmp_path-ac",
"eviction_policy": {
// 500mb.
"max_bytes": 500000000,
}
}
},
"slow": {
"experimental_s3_store": {
"region": "eu-north-1",
"bucket": "crossplane-bucket-af79aeca9",
"key_prefix": "test-prefix-ac/",
"retry": {
"max_retries": 6,
"delay": 0.3,
"jitter": 0.5,
},
"additional_max_concurrent_requests": 10
}
}
}
}
},
"schedulers": {
"MAIN_SCHEDULER": {
"simple": {
"supported_platform_properties": {
"cpu_count": "minimum",
"memory_kb": "minimum",
"network_kbps": "minimum",
"disk_read_iops": "minimum",
"disk_read_bps": "minimum",
"disk_write_iops": "minimum",
"disk_write_bps": "minimum",
"shm_size": "minimum",
"gpu_count": "minimum",
"gpu_model": "exact",
"cpu_vendor": "exact",
"cpu_arch": "exact",
"cpu_model": "exact",
"kernel_version": "exact",
"docker_image": "priority",
}
}
}
},
"servers": [{
"listener": {
"http": {
"socket_address": "0.0.0.0:50051"
}
},
"services": {
"cas": {
"main": {
"cas_store": "CAS_MAIN_STORE"
}
},
"ac": {
"main": {
"ac_store": "AC_MAIN_STORE"
}
},
"execution": {
"main": {
"cas_store": "CAS_MAIN_STORE",
"scheduler": "MAIN_SCHEDULER",
}
},
"capabilities": {
"main": {
"remote_execution": {
"scheduler": "MAIN_SCHEDULER",
}
}
},
"bytestream": {
"cas_stores": {
"main": "CAS_MAIN_STORE",
}
},
"health": {},
}
}]
}
The Redis CAS configuration is designed for high-performance cloud environments. It uses Redis as the fast-end storage, providing quick access to frequently used data. This configuration can be effectively paired with S3 for the slow store, creating a highly efficient and scalable production setup. It is particularly well-suited for use with Kubernetes, enabling seamless integration and management of cloud resources.
```json5 { "stores": { "CAS_FAST_SLOW_STORE": { "fast_slow": { "fast": { "redis_store": { "addresses": [ "redis://127.0.0.1:6379/", // Master node 1 "redis://127.0.0.1:6380/", // Master node 2 "redis://127.0.0.1:6381/", // Master node 3 "redis://127.0.0.1:6382/", // Master node 3 "redis://127.0.0.1:6383/", // Master node 3 "redis://127.0.0.1:6384/" // Master node 3 //"redis://172.18.0.2:6379/" // Master node 3 // "redis://172.18.0.3:6379/", // Master node 3 // "redis://172.18.0.4:6379/" // Master node 3 ] } }, "slow": { "filesystem": { "content_path": "/tmp/nativelink/data/content_path-index", "temp_path": "/tmp/nativelink/data/tmp_path-index", "eviction_policy": { "max_bytes": 120000000000 } } } } }, "AC_FAST_SLOW_STORE": { "fast_slow": { "fast": { "filesystem": { "content_path": "/tmp/nativelink/data/content_path-index", "temp_path": "/tmp/nativelink/data/tmp_path-index", "eviction_policy": { "max_bytes": 120000000000 } } }, "slow": { "filesystem": { "content_path": "/tmp/nativelink/data/content_path-ac", "temp_path": "/tmp/nativelink/data/tmp_path-ac", "eviction_policy": { "max_bytes": 5000000000 } } } } }, "AC_MAIN_STORE": { "completeness_checking": { "backend": { "ref_store": { "name": "AC_FAST_SLOW_STORE" } }, "cas_store": { "ref_store": { "name": "CAS_MAIN_STORE" } } } }, "CAS_MAIN_STORE": { "existence_cache": { "backend": { "compression": { "compression_algorithm": { "lz4": {} }, "backend": { "ref_store": { "name": "CAS_FAST_SLOW_STORE" } } } } } } }, "servers": [ { "listener": { "http": { "socket_address": "0.0.0.0:50051" } }, "services": { "cas": { "main": { "cas_store": "CAS_MAIN_STORE" } }, "ac": { "main": { "ac_store": "AC_MAIN_STORE" } }, "capabilities": {}, "bytestream": { "cas_stores": { "main": "CAS_MAIN_STORE" } } } }, { "listener": { "http": { "socket_address": "0.0.0.0:50061" } }, "services": { "experimental_prometheus": { "path": "/metrics" } } } ] } ```