From a470a37ac9f0f1be48e5dadf796be97abefaf712 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Korn=C3=A9l=20D=C3=A1vid?= <47784437+davidkornel@users.noreply.github.com> Date: Fri, 28 Jul 2023 11:22:37 +0200 Subject: [PATCH 001/243] docs(Gateway): Document external ip specification (#93) --- docs/GATEWAY.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 51efe6bf..e95c114d 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -148,6 +148,7 @@ Below is a quick reference of the most important fields of the Gateway [`spec`]( | :--- | :---: | :--- | :---: | | `gatewayClassName` | `string` | The name of the GatewayClass that provides the root of the hierarchy the Gateway is attached to. | Yes | | `listeners` | `list` | The list of TURN listeners. | Yes | +| `addresses` | `list` | The list of manually hinted external IP addresses for the rendered service (only the first one is used). | No | Each TURN `listener` is defined by a unique name, a transport protocol and a port. In addition, a `tls` configuration is required for TLS and DTLS listeners. @@ -162,7 +163,20 @@ Each TURN `listener` is defined by a unique name, a transport protocol and a por For TLS/DTLS listeners, `tls.mode` must be set to `Terminate` or omitted (`Passthrough` does not make sense for TURN), and `tls.certificateRefs` must be a [reference to a Kubernetes Secret](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig) of type `tls` or `opaque` with exactly two keys: `tls.crt` must hold the TLS PEM certificate and `tls.key` must hold the TLS PEM key. -STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN listeners on UDP and DTLS protocols are exposed as UDP services, TCP and TLS listeners are exposed as TCP. +STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to +clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be +assigned a single externally reachable IP address. If you want multiple TURN listeners on different +public IPs, create multiple Gateways. TURN listeners on UDP and DTLS protocols are exposed as UDP +services, TCP and TLS listeners are exposed as TCP. + +Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway, will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. +> **Warning** +Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated, thus will be ignored if the cloud-provider or your Kubernetes install does not support the feature. Also the `externalIPs` field is denied by some cloud-providers and will fail the resource creation. Be thorough when using this feature. + +| Field | Type | Description | Required | +| :--- | :---: | :--- | :---: | +| `type` | `string` | Type of the address. Currently we only support IPAddress. | Yes | +| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | Mixed multi-protocol Gateways are supported: this means if you want to expose a UDP and a TCP port on the same LoadBalancer service you can do it with a single Gateway. By default, the STUNner gateway-operator disables the use of mixed-protocol LBs for compatibility reasons. However, it can be enabled by annotating a Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. The below Gateway will expose both ports with their respective protocols. From a0e0cd4d6080575008b2f03c1e717f9f879bd5cf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Wed, 2 Aug 2023 11:57:31 +0200 Subject: [PATCH 002/243] chore(CI/CD): Bump Go version to 1.20 (#94) * chore(CI/CD): Bump Go version to 1.20 * chore(CI/CD): Bump actions/setup-go to v4 * chore(CI/CD): Fix go version --- .github/workflows/e2e-test.yml | 4 ++-- .github/workflows/lint.yml | 4 ++-- .github/workflows/test.yml | 8 ++++---- Dockerfile | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/e2e-test.yml b/.github/workflows/e2e-test.yml index 546b0d10..5b9c3b58 100644 --- a/.github/workflows/e2e-test.yml +++ b/.github/workflows/e2e-test.yml @@ -12,9 +12,9 @@ jobs: - uses: actions/checkout@v3 - name: Set up Go - uses: actions/setup-go@v3 + uses: actions/setup-go@v4 with: - go-version: 1.19 + go-version: '1.20' - name: Download modules run: go mod download diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 45ac2fd1..b14063ff 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -18,9 +18,9 @@ jobs: runs-on: ubuntu-latest steps: - name: Install Go - uses: actions/setup-go@v3 + uses: actions/setup-go@v4 with: - go-version: 1.19 + go-version: '1.20' - name: Checkout code uses: actions/checkout@v3 - name: Run linters diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 7ab2719c..ab6c93d9 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -19,9 +19,9 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Go - uses: actions/setup-go@v3 + uses: actions/setup-go@v4 with: - go-version: 1.19 + go-version: '1.20' - name: Download modules run: go mod download - name: Go install @@ -33,9 +33,9 @@ jobs: steps: - name: Install Go if: success() - uses: actions/setup-go@v3 + uses: actions/setup-go@v4 with: - go-version: 1.19 + go-version: '1.20' - name: Checkout code uses: actions/checkout@v3 - name: Calc coverage diff --git a/Dockerfile b/Dockerfile index f1b6fd44..e907a606 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ ########### # BUILD -FROM golang:1.19-alpine as builder +FROM golang:1.20-alpine as builder WORKDIR /app From 2459ff21e1b6fd0543e1aa0172d83c28ebf3015b Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Wed, 9 Aug 2023 11:45:11 +0200 Subject: [PATCH 003/243] fix(rtd): Fix code rendering --- mkdocs.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 1882d45e..0abc27df 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -46,3 +46,10 @@ markdown_extensions: - markdown_include.include: base_path: . - admonition + - pymdownx.highlight: + anchor_linenums: true + line_spans: __span + pygments_lang_class: true + - pymdownx.inlinehilite + - pymdownx.snippets + - pymdownx.superfences From c26fb446bf523ac783a074f9bcc9d432a39837c1 Mon Sep 17 00:00:00 2001 From: Kornel David Date: Fri, 11 Aug 2023 13:01:08 +0200 Subject: [PATCH 004/243] fix(livekit-demo): fix api version --- docs/examples/livekit/livekit-call-stunner.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/examples/livekit/livekit-call-stunner.yaml b/docs/examples/livekit/livekit-call-stunner.yaml index f99aa44f..fd4e703a 100644 --- a/docs/examples/livekit/livekit-call-stunner.yaml +++ b/docs/examples/livekit/livekit-call-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1alpha2 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1alpha2 kind: Gateway metadata: name: udp-gateway From 234767fa08e3b5e5db4a663575e6dab9b58c68e1 Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Thu, 17 Aug 2023 11:49:09 +0200 Subject: [PATCH 005/243] chore(rtd): .readthedocs build deprecation --- .readthedocs.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index d9222bb5..c7006c23 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -6,10 +6,10 @@ version: 2 # Set the version of Python and other tools you might need -# build: -# os: ubuntu-22.04 -# tools: -# python: "3.10" +build: + os: "ubuntu-22.04" + tools: + python: "3.11" mkdocs: configuration: mkdocs.yml From 89221ee2c22ce1667599328fa523e26896d86ff6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Korn=C3=A9l=20D=C3=A1vid?= <47784437+davidkornel@users.noreply.github.com> Date: Mon, 4 Sep 2023 14:09:05 +0200 Subject: [PATCH 006/243] Update stunnerd.conf min_port and max_port was used falsely instead of min_relay_port and max_relay_port https://github.com/l7mp/stunner/blob/234767fa08e3b5e5db4a663575e6dab9b58c68e1/pkg/apis/v1alpha1/listener.go#L29 --- cmd/stunnerd/stunnerd.conf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cmd/stunnerd/stunnerd.conf b/cmd/stunnerd/stunnerd.conf index 03403d72..8d130fa6 100644 --- a/cmd/stunnerd/stunnerd.conf +++ b/cmd/stunnerd/stunnerd.conf @@ -25,8 +25,8 @@ listeners: address: "$STUNNER_ADDR" protocol: udp port: $STUNNER_PORT - min_port: $STUNNER_MIN_PORT - max_port: $STUNNER_MAX_PORT + min_relay_port: $STUNNER_MIN_PORT + max_relay_port: $STUNNER_MAX_PORT routes: - open-cluster # - media-server-cluster @@ -35,8 +35,8 @@ listeners: address: "$STUNNER_ADDR" protocol: tcp port: $STUNNER_PORT - min_port: $STUNNER_MIN_PORT - max_port: $STUNNER_MAX_PORT + min_relay_port: $STUNNER_MIN_PORT + max_relay_port: $STUNNER_MAX_PORT routes: - open-cluster # - media-server-cluster From b5b2b40d9c2c0adcfed79f09401be9815bc8148e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 11 Sep 2023 22:40:19 +0200 Subject: [PATCH 007/243] feat: Implement and integrate a Config Discovery Service (CDS) client This change makes it possible for the STUNner dataplane component `stunnerd` to autodiscover its own configuration. This is achieved by implementing a CDS client for `stunnerd`: on startup the CDS client opens a WebSocket connection to a preconfigured CDS server and grabs an initial configuration. As long as the connection is open, the CDS server will push new configurations down to the client as they become available. This makes it possible to remove the unstability and latency associated with obtaining the fresh config by watching a config file mapped into the filesystem of `stunnerd` (the config file watcher remains available for the "legacy" dataplane mode). Major changes: - implement a generic config watcher interface and make the config file watcher and the CDS config poller two implementations of the interface - obtain the CDS client id from the environment as /, or fall back to the default (hostname) if the environment is not set - the initial zeroconfig is now emitted by the config file watcher (the CDS client does not use a zeroconfig to init `stunnerd` but rather waits for a valid config from the CDS server) - the ZeroConfig() and ParseConfig() functions are moved to the CDS client - stunnerd id is now avaiable in the config (`Admin.Name`) - tests are implemented at the CDS server side (the operator) that imports the CDS client --- cmd/stunnerd/main.go | 72 ++-- config.go | 268 +------------ config_test.go | 24 +- deploy/manifests/default-dataplane.yaml | 24 ++ go.mod | 2 + go.sum | 3 + pkg/apis/v1alpha1/admin.go | 3 + pkg/config/client/client.go | 500 ++++++++++++++++++++++++ pkg/config/client/config.go | 66 ++++ stunner.go | 20 + 10 files changed, 680 insertions(+), 302 deletions(-) create mode 100644 deploy/manifests/default-dataplane.yaml create mode 100644 pkg/config/client/client.go create mode 100644 pkg/config/client/config.go diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index e5454c3a..cd493ea3 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -1,8 +1,8 @@ package main import ( - // "fmt" "context" + "fmt" "os" "os/signal" "syscall" @@ -17,13 +17,16 @@ import ( // usage: stunnerd -v turn://user1:passwd1@127.0.0.1:3478?transport=udp const ( - defaultLoglevel = "all:INFO" - confUpdatePeriod = 1 * time.Second + defaultLoglevel = "all:INFO" + // environment for the config poller + // defaultDiscoveryAddress = "ws://localhost:13478/api/v1/config/watch" + envVarName = "STUNNER_NAME" + envVarNamespace = "STUNNER_NAMESPACE" ) func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", "Config file.") + var config = flag.StringP("config", "c", "", "Config origin. If starts with ws or http then it is considered as the URL of a config discivery server which is polled for config, otherwise considered as a file from which to read the config.") var level = flag.StringP("log", "l", "", "Log level (default: all:INFO).") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false).") var udpThreadNum = flag.IntP("udp-thread-num", "u", 0, @@ -37,12 +40,22 @@ func main() { // verbose mode on, override any loglevel logLevel = "all:DEBUG" } + if *level != "" { // loglevel set on the comman line, use that one instead logLevel = *level } + // default id is hostname + var id string + name, ok1 := os.LookupEnv(envVarName) + namespace, ok2 := os.LookupEnv(envVarNamespace) + if ok1 && ok2 { + id = fmt.Sprintf("%s/%s", namespace, name) + } + st := stunner.NewStunner(stunner.Options{ + Id: id, LogLevel: logLevel, DryRun: *dryRun, UDPListenerThreadNum: *udpThreadNum, @@ -51,10 +64,12 @@ func main() { log := st.GetLogger().NewLogger("stunnerd") + log.Infof("starting stunnerd instance %q", id) + conf := make(chan v1alpha1.StunnerConfig, 1) defer close(conf) - var cancelWatcher context.CancelFunc + var cancelConfigLoader context.CancelFunc if *config == "" && flag.NArg() == 1 { log.Infof("starting %s with default configuration at TURN URI: %s", os.Args[0], flag.Arg(0)) @@ -68,9 +83,9 @@ func main() { conf <- *c } else if *config != "" && !*watch { - log.Infof("loading configuration from config file %q", *config) + log.Infof("loading configuration from origin %q", *config) - c, err := stunner.LoadConfig(*config) + c, err := st.LoadConfig(*config) if err != nil { log.Error(err.Error()) os.Exit(1) @@ -79,27 +94,14 @@ func main() { conf <- *c } else if *config != "" && *watch { - log.Infof("watching configuration file at %q", *config) - - // init stunnerd with an empty config: this bootstraps it with the default - // resources (above all, starts the health-checker) - initConf := stunner.NewZeroConfig() - log.Debug("bootstrapping with zero reconciliation") - if err := st.Reconcile(*initConf); err != nil { - log.Errorf("could not reconcile initial configuratoin: %s", err.Error()) - os.Exit(1) - } + log.Infof("watching configuration at origin %q", *config) ctx, cancel := context.WithCancel(context.Background()) defer cancel() - cancelWatcher = cancel - - if err := stunner.WatchConfig(ctx, stunner.Watcher{ - ConfigFile: *config, - ConfigChannel: conf, - Logger: st.GetLogger(), - }); err != nil { - log.Errorf("could not create config file watcher: %s", err.Error()) + cancelConfigLoader = cancel + + if err := st.WatchConfig(ctx, *config, conf); err != nil { + log.Errorf("could not run config watcher: %s", err.Error()) os.Exit(1) } } else { @@ -107,13 +109,9 @@ func main() { os.Exit(1) } - sigint := make(chan os.Signal, 1) - defer close(sigint) - signal.Notify(sigint, syscall.SIGINT) - sigterm := make(chan os.Signal, 1) defer close(sigterm) - signal.Notify(sigterm, syscall.SIGTERM) + signal.Notify(sigterm, syscall.SIGTERM, syscall.SIGINT) exit := make(chan bool, 1) defer close(exit) @@ -124,18 +122,14 @@ func main() { log.Info("normal exit on graceful shutdown") os.Exit(0) - case <-sigint: - log.Info("normal exit") - os.Exit(0) - case <-sigterm: - log.Info("caught SIGTERM: performing a graceful shutdown") + log.Info("performing a graceful shutdown") st.Shutdown() - // cancel the config watcher - if cancelWatcher != nil { - log.Info("canceling config watcher") - cancelWatcher() + if cancelConfigLoader != nil { + log.Info("canceling config loader") + cancelConfigLoader() + cancelConfigLoader = nil } go func() { diff --git a/config.go b/config.go index a13d12d2..60504448 100644 --- a/config.go +++ b/config.go @@ -3,28 +3,20 @@ package stunner import ( "context" "encoding/base64" - "encoding/json" - "errors" "fmt" - "os" - "regexp" - "strconv" "strings" - "time" - "github.com/fsnotify/fsnotify" - "github.com/pion/logging" "github.com/pion/transport/v2" - "sigs.k8s.io/yaml" "github.com/l7mp/stunner/internal/resolver" "github.com/l7mp/stunner/pkg/apis/v1alpha1" + cds "github.com/l7mp/stunner/pkg/config/client" ) -const confUpdatePeriod = 1 * time.Second - // Options defines various options for the STUNner server. type Options struct { + // Id is the identifier of this stunnerd daemon instance. Defaults to hostname. + Id string // DryRun suppresses sideeffects: STUNner will not initialize listener sockets and bring up // the TURN server, and it will not fire up the health-check and the metrics // servers. Intended for testing, default is false. @@ -52,25 +44,6 @@ type Options struct { Net transport.Net } -// NewZeroConfig builds a zero configuration useful for bootstrapping STUNner. It starts with -// plaintext authentication and opens no listeners and clusters. -func NewZeroConfig() *v1alpha1.StunnerConfig { - return &v1alpha1.StunnerConfig{ - ApiVersion: v1alpha1.ApiVersion, - Admin: v1alpha1.AdminConfig{}, - Auth: v1alpha1.AuthConfig{ - Type: "plaintext", - Realm: v1alpha1.DefaultRealm, - Credentials: map[string]string{ - "username": "dummy-username", - "password": "dummy-password", - }, - }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{}, - } -} - // NewDefaultConfig builds a default configuration from a TURN server URI. Example: the URI // `turn://user:pass@127.0.0.1:3478?transport=udp` will be parsed into a STUNner configuration with // a server running on the localhost at UDP port 3478, with plain-text authentication using the @@ -131,45 +104,6 @@ func NewDefaultConfig(uri string) (*v1alpha1.StunnerConfig, error) { return c, nil } -// LoadConfig loads a configuration from a file, substituting environment variables for -// placeholders in the configuration file. Returns the new configuration or error if load fails. -func LoadConfig(config string) (*v1alpha1.StunnerConfig, error) { - c, err := os.ReadFile(config) - if err != nil { - return nil, fmt.Errorf("could not read config: %s\n", err.Error()) - } - - // substitute environtment variables - // default port: STUNNER_PUBLIC_PORT -> STUNNER_PORT - re := regexp.MustCompile(`^[0-9]+$`) - port, ok := os.LookupEnv("STUNNER_PORT") - if !ok || (ok && port == "") || (ok && !re.Match([]byte(port))) { - publicPort := v1alpha1.DefaultPort - publicPortStr, ok := os.LookupEnv("STUNNER_PUBLIC_PORT") - if ok { - if p, err := strconv.Atoi(publicPortStr); err == nil { - publicPort = p - } - } - os.Setenv("STUNNER_PORT", fmt.Sprintf("%d", publicPort)) - } - - e := os.ExpandEnv(string(c)) - - s := v1alpha1.StunnerConfig{} - // try YAML first - if err = yaml.Unmarshal([]byte(e), &s); err != nil { - // if it fails, try to json - if errJ := json.Unmarshal([]byte(e), &s); err != nil { - return nil, fmt.Errorf("could not parse config file at '%s': "+ - "YAML parse error: %s, JSON parse error: %s\n", - config, err.Error(), errJ.Error()) - } - } - - return &s, nil -} - // GetConfig returns the configuration of the running STUNner daemon. func (s *Stunner) GetConfig() *v1alpha1.StunnerConfig { s.log.Tracef("GetConfig") @@ -208,196 +142,22 @@ func (s *Stunner) GetConfig() *v1alpha1.StunnerConfig { return &c } -type Watcher struct { - // ConfigFile specifies the config file name to watch. - ConfigFile string - // ConfigChannel is used to return the configs read. - ConfigChannel chan<- v1alpha1.StunnerConfig - // Logger is a logger factory as returned by, e.g., stunner.GetLogger(). - Logger logging.LoggerFactory - // Log is a leveled logger used to report progress. Either Logger or Log must be specified. - Log logging.LeveledLogger -} - -// WatchConfig will watch a configuration file specified in the `Watcher.ConfigFile` parameter for -// changes and emit a new `StunnerConfig` on `Watcher.ConfigChannel` each time the file changes. If -// no file exists at the given path, then WatchConfig will periodically retry until the file -// appears. The configuration sent through the channel is not validated, make sure to check for -// syntax errors on the receiver side. Use the `context` to cancel the watcher. -func WatchConfig(ctx context.Context, w Watcher) error { - if w.ConfigChannel == nil { - return errors.New("uninitialized config channel") - } - - if w.ConfigFile == "" { - return errors.New("uninitialized config file path") - } - - if w.Log == nil { - w.Log = w.Logger.NewLogger("watch-config") - } - w.Log.Tracef("WatchConfig") - - go func() { - for { - // try to watch - if ok := configWatcher(ctx, w); !ok { - return - } - - if ok := tryWatchConfig(ctx, w); !ok { - return - } - } - - }() - - return nil -} - -// tryWatchConfig runs a timer to look for the config file at the given path and returns it -// immediately once found. Returns true if further action is needed (configWatcher has to be -// started) or false on normal exit. -func tryWatchConfig(ctx context.Context, w Watcher) bool { - w.Log.Tracef("tryWatchConfig") - config := w.ConfigFile - - ticker := time.NewTicker(confUpdatePeriod) - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return false - - case <-ticker.C: - w.Log.Debugf("trying to read config file %q from periodic timer", - config) - - // check if config file exists and it is readable - if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { - w.Log.Debugf("config file %q does not exist", config) - - // report status in every 10th second - if time.Now().Second()%10 == 0 { - w.Log.Warnf("waiting for config file %q", config) - } - - continue - } - - return true - } - } -} - -// configWatcher actually watches the config and emits the configs found on the specified -// channel. Returns true if further action is needed (tryWatachConfig is to be started) or false on -// normal exit. -func configWatcher(ctx context.Context, w Watcher) bool { - w.Log.Tracef("configWatcher") - prev := v1alpha1.StunnerConfig{} - - // create a new watcher - watcher, err := fsnotify.NewWatcher() +// LoadConfig loads a configuration from an origin. This is a shim wrapper around ConfigOrigin.Load. +func (s *Stunner) LoadConfig(config string) (*v1alpha1.StunnerConfig, error) { + client, err := cds.NewClient(config, s.id, s.logger) if err != nil { - return true + return nil, err } - defer watcher.Close() - - config := w.ConfigFile - ch := w.ConfigChannel - if err := watcher.Add(config); err != nil { - w.Log.Debugf("could not add config file %q watcher: %s", config, err.Error()) - return true - } + return client.Load() +} - // emit an initial config - c, err := LoadConfig(config) +// WatchConfig watches a configuration from an origin. This is a shim wrapper around ConfigOrigin.Watch. +func (s *Stunner) WatchConfig(ctx context.Context, config string, ch chan<- v1alpha1.StunnerConfig) error { + client, err := cds.NewClient(config, s.id, s.logger) if err != nil { - w.Log.Warnf("could not load config file %q: %s", config, err.Error()) - return true + return err } - w.Log.Debugf("config file successfully loaded from %q", config) - - // send a deepcopy over the channel - copy := v1alpha1.StunnerConfig{} - c.DeepCopyInto(©) - ch <- copy - - // save deepcopy so that we can filter repeated events - c.DeepCopyInto(&prev) - - for { - select { - case <-ctx.Done(): - return false - - case e, ok := <-watcher.Events: - if !ok { - w.Log.Debug("config watcher event handler received invalid event") - return true - } - - w.Log.Debugf("received watcher event: %s", e.String()) - - if e.Has(fsnotify.Remove) { - w.Log.Warnf("config file deleted %q, disabling watcher", e.Op.String()) - - if err := watcher.Remove(config); err != nil { - w.Log.Debugf("could not remove config file %q watcher: %s", - config, err.Error()) - } - - return true - } - - if !e.Has(fsnotify.Write) { - w.Log.Debugf("unhandled notify op on config file %q (ignoring): %s", - e.Name, e.Op.String()) - continue - } - - w.Log.Debugf("loading configuration file: %s", config) - c, err = LoadConfig(config) - if err != nil { - // assume it is a YAML/JSON syntax error (LoadConfig does not - // validate): report and ignore - w.Log.Warnf("could not load config file %q: %s", config, err.Error()) - continue - } - - // suppress repeated events - if c.DeepEqual(&prev) { - w.Log.Debugf("ignoring recurrent notify event for the same config file") - continue - } - - w.Log.Debugf("config file successfully loaded from %q", config) - - copy := v1alpha1.StunnerConfig{} - c.DeepCopyInto(©) - ch <- copy - - // save deepcopy so that we can filter repeated events - c.DeepCopyInto(&prev) - - case err, ok := <-watcher.Errors: - if !ok { - w.Log.Debugf("config watcher error handler received invalid error") - return true - } - - w.Log.Debugf("watcher error, deactivating watcher: %s", err.Error()) - - if err := watcher.Remove(config); err != nil { - w.Log.Debugf("could not remove config file %q watcher: %s", - config, err.Error()) - } - - return true - } - } + return client.Watch(ctx, ch) } diff --git a/config_test.go b/config_test.go index b83faf34..ecd9c87f 100644 --- a/config_test.go +++ b/config_test.go @@ -1,10 +1,9 @@ package stunner import ( + "context" "fmt" "net" - // "reflect" - "context" "os" "testing" "time" @@ -14,6 +13,7 @@ import ( "sigs.k8s.io/yaml" "github.com/l7mp/stunner/pkg/apis/v1alpha1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) @@ -149,11 +149,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { log.Debug("init watcher with nonexistent config file") ctx, cancel := context.WithCancel(context.Background()) defer cancel() - err = WatchConfig(ctx, Watcher{ - ConfigFile: file, - ConfigChannel: conf, - Logger: loggerFactory, - }) + err = stunner.WatchConfig(ctx, file, conf) assert.NoError(t, err, "creating config watcher") // nothing should happen here: wait a bit so that the watcher has comfortable time to start @@ -186,8 +182,14 @@ func TestStunnerConfigFileWatcher(t *testing.T) { // wait a bit so that the watcher has time to react time.Sleep(50 * time.Millisecond) - // read back result - c2 := <-conf + // first read should yield a zeroconfig + c2, ok := <-conf + assert.True(t, ok, "zeroconfig emitted") + checkZeroConfig(t, &c2, stunner.GetId()) + + // second read yields the real config + c2, ok = <-conf + assert.True(t, ok, "config emitted") checkDefaultConfig(t, &c2, "UDP") log.Debug("write a wrong config file (WatchConfig does not validate)") @@ -244,3 +246,7 @@ func checkDefaultConfig(t *testing.T, c *v1alpha1.StunnerConfig, proto string) { assert.Len(t, c.Clusters[0].Endpoints, 1, "cluster endpoint len") assert.Equal(t, "0.0.0.0/0", c.Clusters[0].Endpoints[0], "endpoint") } + +func checkZeroConfig(t *testing.T, c *v1alpha1.StunnerConfig, id string) { + assert.True(t, c.DeepEqual(cdsclient.ZeroConfig(id)), "zeroconfig ok") +} diff --git a/deploy/manifests/default-dataplane.yaml b/deploy/manifests/default-dataplane.yaml new file mode 100644 index 00000000..77f14e10 --- /dev/null +++ b/deploy/manifests/default-dataplane.yaml @@ -0,0 +1,24 @@ +apiVersion: stunner.l7mp.io/v1alpha1 +kind: Dataplane +metadata: + name: default +spec: + replicas: 1 + image: l7mp/stunnerd:dev + imagePullPolicy: Always + command: + - stunnerd + args: + - -w + - -c + - /etc/stunnerd/stunnerd.conf + - --udp-thread-num=1 + hostNetwork: false + resources: + limits: + cpu: 250m + memory: 120Mi + requests: + cpu: 250m + memory: 120Mi + terminationGracePeriodSeconds: 3600 diff --git a/go.mod b/go.mod index ae8bb475..6c2a3a69 100644 --- a/go.mod +++ b/go.mod @@ -19,6 +19,8 @@ require ( ) require ( + github.com/google/uuid v1.1.2 + github.com/gorilla/websocket v1.5.0 github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb github.com/pion/randutil v0.1.0 github.com/pion/transport v0.14.1 diff --git a/go.sum b/go.sum index 156f09b0..66b80915 100644 --- a/go.sum +++ b/go.sum @@ -175,11 +175,14 @@ github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= +github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc= +github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= diff --git a/pkg/apis/v1alpha1/admin.go b/pkg/apis/v1alpha1/admin.go index 9bfdce7a..336ee463 100644 --- a/pkg/apis/v1alpha1/admin.go +++ b/pkg/apis/v1alpha1/admin.go @@ -82,6 +82,9 @@ func (req *AdminConfig) DeepCopyInto(dst Config) { // String stringifies the configuration. func (req *AdminConfig) String() string { status := []string{} + if req.Name != "" { + status = append(status, fmt.Sprintf("name=%q", req.Name)) + } if req.LogLevel != "" { status = append(status, fmt.Sprintf("logLevel=%q", req.LogLevel)) } diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go new file mode 100644 index 00000000..45409e10 --- /dev/null +++ b/pkg/config/client/client.go @@ -0,0 +1,500 @@ +package configdiscoveryclient + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "os" + "strings" + "time" + + "github.com/fsnotify/fsnotify" + "github.com/gorilla/websocket" + "github.com/pion/logging" + + "github.com/l7mp/stunner/pkg/apis/v1alpha1" +) + +var ( + // Time + RetryPeriod = 1 * time.Second + + // Time allowed to write a message to the CDS server. + WriteWait = 2 * time.Second + + // Time allowed to read the next pong message from the CDS server. + PongWait = 8 * time.Second + + // Send pings to the CDS server with this period. Must be less than PongWait. + PingPeriod = 5 * time.Second +) + +// Client represents a generic config client. Currently supported config providers: http, ws, or +// file. Configuration obtained through the client are not validated, make sure to validate on the +// receiver side. +type Client interface { + // Load grabs a new configuration from the config client. + Load() (*v1alpha1.StunnerConfig, error) + // Watch listens to new configurations and returns them on the channel ch. ctx cancels the + // watcher. + Watch(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error + fmt.Stringer +} + +// GetConfig returns the configuration of the running STUNner daemon. +func NewClient(origin string, id string, logger logging.LoggerFactory) (Client, error) { + u, err := url.Parse(origin) + if err != nil { + return nil, fmt.Errorf("could not parse config origin address %q: %w", origin, err) + } + + var client Client + switch strings.ToLower(u.Scheme) { + case "http", "ws", "https", "wss": + client = &configDiscoveryClient{ + serverAddress: origin, + id: id, + log: logger.NewLogger("config-poller"), + } + default: + client = &configFileClient{ + configFile: origin, + id: id, + log: logger.NewLogger("config-watcher"), + } + } + + return client, nil +} + +// configFileClient is the implementation of the Client interface for config files. +type configFileClient struct { + // configFile specifies the config file name to watch. + configFile string + // id is the name of the stunnerd instance. + id string + // log is a leveled logger used to report progress. Either Logger or Log must be specified. + log logging.LeveledLogger +} + +func (w *configFileClient) String() string { + return fmt.Sprintf("config client using file %q", w.configFile) +} + +func (w *configFileClient) Load() (*v1alpha1.StunnerConfig, error) { + b, err := os.ReadFile(w.configFile) + if err != nil { + return nil, fmt.Errorf("could not read config file %q: %s", w.configFile, err.Error()) + } + + return ParseConfig(b) +} + +// WatchConfig watches a configuration file for changes. If no file exists at the given path, +// WatchConfig will periodically retry until the file appears. +func (w *configFileClient) Watch(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error { + if w.configFile == "" { + return errors.New("uninitialized config file path") + } + + // emit an empty config: this bootstraps stunner the default resources (above all, starts + // the health-checker) + w.log.Debug("bootstrapping with zero configuration") + initConf := ZeroConfig(w.id) + ch <- *initConf + + go func() { + for { + // try to watch + if !w.configWatcher(ctx, ch) { + return + } + + if !w.tryWatchConfig(ctx) { + return + } + } + + }() + + return nil +} + +// tryWatchConfig runs a timer to look for the config file at the given path and returns it +// immediately once found. Returns true if further action is needed (configWatcher has to be +// started) or false on normal exit. +func (w *configFileClient) tryWatchConfig(ctx context.Context) bool { + w.log.Tracef("tryWatchConfig") + config := w.configFile + + ticker := time.NewTicker(RetryPeriod) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return false + + case <-ticker.C: + w.log.Debugf("trying to read config file %q from periodic timer", + config) + + // check if config file exists and it is readable + if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { + w.log.Debugf("config file %q does not exist", config) + + // report status in every 10th second + if time.Now().Second()%10 == 0 { + w.log.Warnf("waiting for config file %q", config) + } + + continue + } + + return true + } + } +} + +// configWatcher watches the config file and emits new configs on the specified channel. Returns +// true if further action is needed (tryWatchConfig is to be started) or false on normal exit. +func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) bool { + w.log.Tracef("configWatcher") + + // create a new watcher + watcher, err := fsnotify.NewWatcher() + if err != nil { + return true + } + defer watcher.Close() + + config := w.configFile + if err := watcher.Add(config); err != nil { + w.log.Debugf("could not add watcher for config file %q: %s", config, err.Error()) + return true + } + + // emit an initial config + c, err := w.Load() + if err != nil { + w.log.Warnf("cannot load config file: %s", err.Error()) + return true + } + + w.log.Debugf("config file successfully loaded from %q", config) + + // send a deepcopy over the channel + copy := v1alpha1.StunnerConfig{} + c.DeepCopyInto(©) + ch <- copy + + // save deepcopy so that we can filter repeated events + prev := v1alpha1.StunnerConfig{} + c.DeepCopyInto(&prev) + + for { + select { + case <-ctx.Done(): + return false + + case e, ok := <-watcher.Events: + if !ok { + w.log.Debug("config watcher event handler received invalid event") + return true + } + + w.log.Debugf("received watcher event: %s", e.String()) + + if e.Has(fsnotify.Remove) { + w.log.Warnf("config file deleted %q, disabling watcher", e.Op.String()) + + if err := watcher.Remove(config); err != nil { + w.log.Debugf("could not remove config file %q watcher: %s", + config, err.Error()) + } + + return true + } + + if !e.Has(fsnotify.Write) { + w.log.Debugf("unhandled notify op on config file %q (ignoring): %s", + e.Name, e.Op.String()) + continue + } + + w.log.Debugf("loading configuration file: %s", config) + c, err := w.Load() + if err != nil { + w.log.Warnf("error loading config file: %s", err.Error()) + return true + } + + // suppress repeated events + if c.DeepEqual(&prev) { + w.log.Debugf("ignoring recurrent notify event for the same config file") + continue + } + + w.log.Debugf("config file successfully loaded from %q", config) + + copy := v1alpha1.StunnerConfig{} + c.DeepCopyInto(©) + ch <- copy + + // save deepcopy so that we can filter repeated events + c.DeepCopyInto(&prev) + + case err, ok := <-watcher.Errors: + if !ok { + w.log.Debugf("config watcher error handler received invalid error") + return true + } + + w.log.Debugf("watcher error, deactivating watcher: %s", err.Error()) + + if err := watcher.Remove(config); err != nil { + w.log.Debugf("could not remove config file %q watcher: %s", + config, err.Error()) + } + + return true + } + } +} + +// configDiscoveryClient is the the implementation of the config discovery service. +type configDiscoveryClient struct { + // serverAddress is the URL of the config discovery server. + serverAddress string + // Id is the name of the stunnerd instance that is used to bootstrap the connection + // poller. Set to namespace/name of the pod when using the stunner gateway operator for + // config discovery. + id string + // Log is a leveled logger used to report progress. + log logging.LeveledLogger +} + +func (p *configDiscoveryClient) String() string { + return fmt.Sprintf("config discovery service using server %q", p.serverAddress) +} + +func (p *configDiscoveryClient) Load() (*v1alpha1.StunnerConfig, error) { + location, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, false) + if err != nil { + return nil, err + } + + resp, err := http.Get(location) + if err != nil { + return nil, err + } + + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("invalid HTTP response status: %s", resp.Status) + } + + defer resp.Body.Close() + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, err + } + + if len(body) == 0 { + // empty config: this is an error for the config loader + return nil, errors.New("empty config received") + } + + // fmt.Println("++++++++++++++++++++") + // fmt.Println(string(body)) + + return ParseConfig(body) +} + +// Watch polls a config discovery server for a configuration file by sending a config request +// and then waits for the server to push a valid `StunnerConfig`. Use the `context` to cancel the +// watcher. +func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error { + _, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, true) + if err != nil { + return err + } + + // Note: we do not emit an initial config but rather wait for the CDS server to send one, + // so that pod will not be able to bootstrap the healthchecker and keep on restarting until + // it finds the CDS server + + go func() { + for { + // try to watch + if err := p.configPoller(ctx, ch); err != nil { + p.log.Errorf("config file discovery service: %s", err.Error()) + } else { + // context got cancelled + return + } + + // wait between each attempt + time.Sleep(RetryPeriod) + } + }() + + return nil +} + +func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error { + p.log.Tracef("configPoller: trying to open connection to config discovery server at %q", p.serverAddress) + + location, origin, _ := getConfigDiscoveryLocation(p.serverAddress, p.id, true) + header := http.Header{} + header.Set("origin", origin) + conn, _, err := websocket.DefaultDialer.DialContext(ctx, location, header) + if err != nil { + return err + } + + p.log.Infof("connection sucessfully opened to config discovery server at %q", location) + + // this will close the poller goroutine + defer conn.Close() + + // pinger + resCh := make(chan v1alpha1.StunnerConfig) + errCh := make(chan error) + + pingTicker := time.NewTicker(PingPeriod) + closePinger := make(chan any) + defer close(closePinger) + + go func() { + defer pingTicker.Stop() + for { + select { + case <-pingTicker.C: + // p.log.Tracef("++++ PING ++++ for CDS server %q at client %q", location, p.id) + conn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck + if err := conn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { + errCh <- fmt.Errorf("could not ping CDS server at %q: %w", + conn.RemoteAddr(), err) + return + } + case <-closePinger: + p.log.Tracef("closing ping handler to config discovery server at %q at client %q", + location, p.id) + return + } + } + }() + + // poller + go func() { + defer close(resCh) + defer close(errCh) + + // the next pong must arrive within the PongWait period + conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + // reinit the deadline when we get a pong + conn.SetPongHandler(func(string) error { + // p.log.Tracef("++++ PONG ++++ from CDS server %q at client %q", location, p.id) + conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + return nil + }) + + for { + // ping-pong deadline misses will end up being caught here as a read beyond + // the deadline + msgType, msg, err := conn.ReadMessage() + if err != nil { + errCh <- err + return + } + + if msgType != websocket.TextMessage { + errCh <- fmt.Errorf("unexpected message type (code: %d) from client %q", + msgType, conn.RemoteAddr().String()) + return + } + + // fmt.Println("++++++++++++++++++++") + // fmt.Println(n) + // fmt.Println("++++++++++++++++++++") + // fmt.Println(msg) + // fmt.Println("++++++++++++++++++++") + + // msg = bytes.TrimRight(msg, "\x00") + + c, err := ParseConfig(msg) + if err != nil { + // assume it is a YAML/JSON syntax error: report and ignore + p.log.Warnf("could not parse config: %s", err.Error()) + continue + } + + p.log.Debugf("new config received: %s", c.String()) + + copy := v1alpha1.StunnerConfig{} + c.DeepCopyInto(©) + + resCh <- copy + } + }() + + // wait fo cancel + for { + select { + case <-ctx.Done(): + // cancel: normal return + closePinger <- struct{}{} + return nil + case err := <-errCh: + // error: return it + closePinger <- struct{}{} + return err + case conf := <-resCh: + // new config: pass it along and move on + ch <- conf + continue + } + } +} + +// getConfigLocation returns a valid URL from config server address, either for a single HTTP GET +// to query the config discovery server for a single config file or a websocket URL and client for +// polling config file updates +func getConfigDiscoveryLocation(addr, id string, ws bool) (string, string, error) { + u, err := url.Parse(addr) + if err != nil { + return "", "", fmt.Errorf("invalid config discovery server URL %q: %w", addr, err) + } + + q, err := url.ParseQuery(u.RawQuery) + if err != nil { + return "", "", fmt.Errorf("invalid config discovery query server URL %q: %w", addr, err) + } + + // add our id as a query parameter + q.Set("id", id) + u.RawQuery = q.Encode() + + // TODO: share between server and client + u.Path = "/api/v1/config" + if ws { + u.Scheme = "ws" + u.Path = u.Path + "/watch" + } else { + u.Scheme = "http" + } + + // target URL + location := u.String() + + // client + u.Scheme = "http" + u.RawQuery = "" + client := u.String() + + return location, client, nil +} diff --git a/pkg/config/client/config.go b/pkg/config/client/config.go new file mode 100644 index 00000000..e3538f06 --- /dev/null +++ b/pkg/config/client/config.go @@ -0,0 +1,66 @@ +package configdiscoveryclient + +import ( + "encoding/json" + "fmt" + "os" + "regexp" + "strconv" + + "sigs.k8s.io/yaml" + + "github.com/l7mp/stunner/pkg/apis/v1alpha1" +) + +// ZeroConfig builds a zero configuration useful for bootstrapping STUNner. It starts with +// plaintext authentication and opens no listeners and clusters. +func ZeroConfig(id string) *v1alpha1.StunnerConfig { + return &v1alpha1.StunnerConfig{ + ApiVersion: v1alpha1.ApiVersion, + Admin: v1alpha1.AdminConfig{Name: id}, + Auth: v1alpha1.AuthConfig{ + Type: "plaintext", + Realm: v1alpha1.DefaultRealm, + Credentials: map[string]string{ + "username": "dummy-username", + "password": "dummy-password", + }, + }, + Listeners: []v1alpha1.ListenerConfig{}, + Clusters: []v1alpha1.ClusterConfig{}, + } +} + +// ParseConfig parses a raw buffer holding a configuration, substituting environment variables for +// placeholders in the configuration. Returns the new configuration or error if parsing fails. +func ParseConfig(c []byte) (*v1alpha1.StunnerConfig, error) { + // substitute environtment variables + // default port: STUNNER_PUBLIC_PORT -> STUNNER_PORT + re := regexp.MustCompile(`^[0-9]+$`) + port, ok := os.LookupEnv("STUNNER_PORT") + if !ok || port == "" || !re.Match([]byte(port)) { + publicPort := v1alpha1.DefaultPort + publicPortStr, ok := os.LookupEnv("STUNNER_PUBLIC_PORT") + if ok { + if p, err := strconv.Atoi(publicPortStr); err == nil { + publicPort = p + } + } + os.Setenv("STUNNER_PORT", fmt.Sprintf("%d", publicPort)) + } + + e := os.ExpandEnv(string(c)) + + s := v1alpha1.StunnerConfig{} + // try YAML first + if err := yaml.Unmarshal([]byte(e), &s); err != nil { + // if it fails, try to json + if errJ := json.Unmarshal([]byte(e), &s); err != nil { + return nil, fmt.Errorf("could not parse config file: "+ + "YAML parse error: %s, JSON parse error: %s\n", + err.Error(), errJ.Error()) + } + } + + return &s, nil +} diff --git a/stunner.go b/stunner.go index 52038ec6..883f7a9d 100644 --- a/stunner.go +++ b/stunner.go @@ -3,8 +3,10 @@ package stunner import ( "fmt" + "os" "strings" + "github.com/google/uuid" "github.com/pion/logging" "github.com/pion/transport/v2" "github.com/pion/transport/v2/stdnet" @@ -19,8 +21,11 @@ import ( const DefaultLogLevel = "all:WARN" +var DefaultInstanceId = fmt.Sprintf("stunnerd-%s", uuid.New().String()) + // Stunner is an instance of the STUNner deamon. type Stunner struct { + id string version string adminManager, authManager, listenerManager, clusterManager manager.Manager suppressRollback, dryRun bool @@ -68,7 +73,17 @@ func NewStunner(options Options) *Stunner { udpThreadNum = options.UDPListenerThreadNum } + id := options.Id + if id == "" { + if h, err := os.Hostname(); err != nil { + id = DefaultInstanceId + } else { + id = h + } + } + s := &Stunner{ + id: id, version: v1alpha1.ApiVersion, logger: logger, log: log, @@ -100,6 +115,11 @@ func NewStunner(options Options) *Stunner { return s } +// GetId returns the id of the current stunnerd instance. +func (s *Stunner) GetId() string { + return s.id +} + // GetVersion returns the STUNner API version. func (s *Stunner) GetVersion() string { return s.version From 6f16d0067d29f04c5b714c5952a416709584eb97 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 20 Sep 2023 10:22:21 +0200 Subject: [PATCH 008/243] chore(CI/CD): Add e2e test for stable version too Scheduled to run on every second Tuesday --- .../{e2e-test.yml => e2e-test-dev.yml} | 2 +- .github/workflows/e2e-test-stable.yml | 91 +++++++++++++++++++ 2 files changed, 92 insertions(+), 1 deletion(-) rename .github/workflows/{e2e-test.yml => e2e-test-dev.yml} (98%) create mode 100644 .github/workflows/e2e-test-stable.yml diff --git a/.github/workflows/e2e-test.yml b/.github/workflows/e2e-test-dev.yml similarity index 98% rename from .github/workflows/e2e-test.yml rename to .github/workflows/e2e-test-dev.yml index 5b9c3b58..045a4556 100644 --- a/.github/workflows/e2e-test.yml +++ b/.github/workflows/e2e-test-dev.yml @@ -1,4 +1,4 @@ -name: Run End-to-End Test +name: Run End-to-End Test (dev) on: workflow_dispatch: diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml new file mode 100644 index 00000000..e640278b --- /dev/null +++ b/.github/workflows/e2e-test-stable.yml @@ -0,0 +1,91 @@ +name: Run End-to-End Test (stable) + +on: + workflow_dispatch: + schedule: + - cron: '0 11 1-7,15-21 * 2' + +jobs: + e2e_test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Set up Go + uses: actions/setup-go@v4 + with: + go-version: '1.20' + + - name: Download modules + run: go mod download + - name: Go install + run: go install + - name: Build turncat + run: CGO_ENABLED=0 go build -ldflags="-w -s" -o turncat cmd/turncat/main.go + + - name: Start minikube + uses: medyagh/setup-minikube@master + with: + driver: docker + container-runtime: containerd + wait: all + cache: false + + - name: Start minikube tunnel + run: minikube tunnel &>mktunnel.log & + + - name: Set up Helm + uses: azure/setup-helm@v3 + with: + version: v3.11.3 + + - name: Install STUNner + run: | + helm repo add stunner https://l7mp.io/stunner + helm repo update + helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner + helm install stunner stunner/stunner --create-namespace --namespace=stunner + + - name: Deploy iperf server + run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml + + - name: Configure STUNner + run: | + kubectl apply -f docs/examples/simple-tunnel/iperf-stunner.yaml + sleep 75 + + - name: Install iperf client + run: | + sudo apt-get update + sudo apt-get -y install iperf + + - name: Wait for LoadBalancer IP + run: | + while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done + kubectl get all -A + + - name: Start turncat + run: | + ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & + sleep 1 + + - name: Run iperf client + run: | + iperf -c 127.0.0.1 -p 5000 -u -l 100 -b 5M -t 5 | tee iperf.log + + - name: Show logs + run: | + echo "* IPERF" + echo "** Client" + cat iperf.log + echo "** Server" + kubectl logs $(kubectl get pods -l app=iperf-server -o jsonpath='{.items[0].metadata.name}') + echo "* MINIKUBE TUNNEL" + cat mktunnel.log + echo "* TURNCAT" + cat turncat.log + echo "* STUNNER" + kubectl logs -n stunner $(kubectl get pods -n stunner -l app=stunner -o jsonpath='{.items[0].metadata.name}') + + - name: Check iperf conectivity + run: grep "Server Report" iperf.log From 219970a4e18c45d9f40336adaa5ddf00611d075b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 14 Sep 2023 20:39:12 +0200 Subject: [PATCH 009/243] feature: Let id and CDS server address to be passed in env vars --- cmd/stunnerd/main.go | 36 +++++++++++++++---------- deploy/manifests/default-dataplane.yaml | 8 +++--- 2 files changed, 25 insertions(+), 19 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index cd493ea3..f594418d 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -20,14 +20,17 @@ const ( defaultLoglevel = "all:INFO" // environment for the config poller // defaultDiscoveryAddress = "ws://localhost:13478/api/v1/config/watch" - envVarName = "STUNNER_NAME" - envVarNamespace = "STUNNER_NAMESPACE" + envVarName = "STUNNER_NAME" + envVarNamespace = "STUNNER_NAMESPACE" + envVarId = "STUNNER_ID" + envVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" ) func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", "Config origin. If starts with ws or http then it is considered as the URL of a config discivery server which is polled for config, otherwise considered as a file from which to read the config.") - var level = flag.StringP("log", "l", "", "Log level (default: all:INFO).") + var config = flag.StringP("config", "c", "", "Config origin, either a valid URL to the CDS server or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: none).") + var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO).") + var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: ).") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false).") var udpThreadNum = flag.IntP("udp-thread-num", "u", 0, "Number of readloop threads (CPU cores) per UDP listener. Zero disables UDP multithreading (default: 0).") @@ -37,25 +40,30 @@ func main() { logLevel := defaultLoglevel if *verbose { - // verbose mode on, override any loglevel logLevel = "all:DEBUG" } if *level != "" { - // loglevel set on the comman line, use that one instead logLevel = *level } - // default id is hostname - var id string - name, ok1 := os.LookupEnv(envVarName) - namespace, ok2 := os.LookupEnv(envVarNamespace) - if ok1 && ok2 { - id = fmt.Sprintf("%s/%s", namespace, name) + if *config == "" { + origin, ok := os.LookupEnv(envVarConfigOrigin) + if ok { + *config = origin + } + } + + if *id == "" { + name, ok1 := os.LookupEnv(envVarName) + namespace, ok2 := os.LookupEnv(envVarNamespace) + if ok1 && ok2 { + *id = fmt.Sprintf("%s/%s", namespace, name) + } } st := stunner.NewStunner(stunner.Options{ - Id: id, + Id: *id, LogLevel: logLevel, DryRun: *dryRun, UDPListenerThreadNum: *udpThreadNum, @@ -64,7 +72,7 @@ func main() { log := st.GetLogger().NewLogger("stunnerd") - log.Infof("starting stunnerd instance %q", id) + log.Infof("starting stunnerd instance %q", *id) conf := make(chan v1alpha1.StunnerConfig, 1) defer close(conf) diff --git a/deploy/manifests/default-dataplane.yaml b/deploy/manifests/default-dataplane.yaml index 77f14e10..c90e392d 100644 --- a/deploy/manifests/default-dataplane.yaml +++ b/deploy/manifests/default-dataplane.yaml @@ -7,12 +7,10 @@ spec: image: l7mp/stunnerd:dev imagePullPolicy: Always command: - - stunnerd + - "stunnerd" args: - - -w - - -c - - /etc/stunnerd/stunnerd.conf - - --udp-thread-num=1 + - "-w" + - "--udp-thread-num=1" hostNetwork: false resources: limits: From e59e201141b6a81d6d235b80ad33f4bd165163bc Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 21 Sep 2023 17:46:24 +0200 Subject: [PATCH 010/243] fix: Protocol name disambiguation Rename listener protocols to harmonize with Gateway API protocol names. In particular, introduce: - TURN-UDP to mean "TURN over UDP", - TURN-TCP to mean "TURN over TCP", - TURN-TLS to mean "TURN over TLS", - TURN-DTLS to mean "TURN over DTLS", and - remove old shorthand names (UDP, TCP, TLS, DTLS). For more info on this change, see l7mp/stunner-gateway-operator#28 --- config.go | 3 +- config_test.go | 10 ++-- handlers_test.go | 10 ++-- internal/object/listener.go | 6 +- pkg/apis/v1alpha1/default.go | 2 +- pkg/apis/v1alpha1/listener.go | 5 +- pkg/apis/v1alpha1/util.go | 36 +++++++++-- reconcile_test.go | 109 +++++++++++++++++----------------- server.go | 8 +-- server_unix_test.go | 12 ++-- stunner_test.go | 50 ++++++++-------- turncat.go | 90 +++++++++++++++------------- turncat_test.go | 45 +++++++------- uri.go | 59 +++++++++++++----- 14 files changed, 250 insertions(+), 195 deletions(-) diff --git a/config.go b/config.go index 60504448..33d48b02 100644 --- a/config.go +++ b/config.go @@ -88,7 +88,8 @@ func NewDefaultConfig(uri string) (*v1alpha1.StunnerConfig, error) { }}, } - if strings.ToUpper(u.Protocol) == "TLS" || strings.ToUpper(u.Protocol) == "DTLS" { + p := strings.ToUpper(u.Protocol) + if p == "TLS" || p == "DTLS" || p == "TURN-TLS" || p == "TURN-DTLS" { certPem, keyPem, err := GenerateSelfSignedKey() if err != nil { return nil, err diff --git a/config_test.go b/config_test.go index ecd9c87f..82047f36 100644 --- a/config_test.go +++ b/config_test.go @@ -49,7 +49,7 @@ func TestStunnerDefaultServerVNet(t *testing.T) { // patch in the loglevel c.Admin.LogLevel = stunnerTestLoglevel - checkDefaultConfig(t, c, "UDP") + checkDefaultConfig(t, c, "TURN-UDP") // patch in the vnet log.Debug("building virtual network") @@ -104,7 +104,7 @@ func TestStunnerConfigFileRoundTrip(t *testing.T) { // patch in the loglevel c.Admin.LogLevel = stunnerTestLoglevel - checkDefaultConfig(t, c, "UDP") + checkDefaultConfig(t, c, "TURN-UDP") file, err2 := yaml.Marshal(c) assert.NoError(t, err2, "marschal config fike") @@ -190,7 +190,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { // second read yields the real config c2, ok = <-conf assert.True(t, ok, "config emitted") - checkDefaultConfig(t, &c2, "UDP") + checkDefaultConfig(t, &c2, "TURN-UDP") log.Debug("write a wrong config file (WatchConfig does not validate)") c2.Listeners[0].Protocol = "dummy" @@ -213,7 +213,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { checkDefaultConfig(t, &c3, "dummy") log.Debug("update the config file and check") - c3.Listeners[0].Protocol = "TCP" + c3.Listeners[0].Protocol = "TURN-TCP" y, err = yaml.Marshal(c3) assert.NoError(t, err, "marshal config file") err = f.Truncate(0) @@ -228,7 +228,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { // read back result c4 := <-conf - checkDefaultConfig(t, &c4, "TCP") + checkDefaultConfig(t, &c4, "TURN-TCP") stunner.Close() } diff --git a/handlers_test.go b/handlers_test.go index 513a76e0..7ba4762b 100644 --- a/handlers_test.go +++ b/handlers_test.go @@ -62,7 +62,7 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, @@ -89,7 +89,7 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, @@ -119,7 +119,7 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, @@ -151,7 +151,7 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, @@ -183,7 +183,7 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, diff --git a/internal/object/listener.go b/internal/object/listener.go index bd22f41d..7ad639ac 100644 --- a/internal/object/listener.go +++ b/internal/object/listener.go @@ -138,7 +138,7 @@ func (l *Listener) Reconcile(conf v1alpha1.Config) error { l.Addr = ipAddr l.rawAddr = req.Addr l.Port, l.MinPort, l.MaxPort = req.Port, req.MinRelayPort, req.MaxRelayPort - if proto == v1alpha1.ListenerProtocolTLS || proto == v1alpha1.ListenerProtocolDTLS { + if proto == v1alpha1.ListenerProtocolTURNTLS || proto == v1alpha1.ListenerProtocolTURNDTLS { cert, err := base64.StdEncoding.DecodeString(req.Cert) if err != nil { return fmt.Errorf("invalid TLS certificate: base64-decode error: %w", err) @@ -204,7 +204,7 @@ func (l *Listener) Close() error { for _, c := range l.Conns { switch l.Proto { - case v1alpha1.ListenerProtocolUDP: + case v1alpha1.ListenerProtocolTURNUDP: l.log.Tracef("closing %s packet socket at %s", l.Proto.String(), l.Addr) conn, ok := c.(turn.PacketConnConfig) @@ -216,7 +216,7 @@ func (l *Listener) Close() error { if err := conn.PacketConn.Close(); err != nil && !util.IsClosedErr(err) { return err } - case v1alpha1.ListenerProtocolTCP, v1alpha1.ListenerProtocolTLS, v1alpha1.ListenerProtocolDTLS: + case v1alpha1.ListenerProtocolTURNTCP, v1alpha1.ListenerProtocolTURNTLS, v1alpha1.ListenerProtocolTURNDTLS: l.log.Tracef("closing %s listener socket at %s", l.Proto.String(), l.Addr) conn, ok := c.(turn.ListenerConfig) diff --git a/pkg/apis/v1alpha1/default.go b/pkg/apis/v1alpha1/default.go index 680c2bc8..49ee6927 100644 --- a/pkg/apis/v1alpha1/default.go +++ b/pkg/apis/v1alpha1/default.go @@ -2,7 +2,7 @@ package v1alpha1 const ApiVersion string = "v1alpha1" const DefaultStunnerName = "default-stunnerd" -const DefaultProtocol = "udp" +const DefaultProtocol = "turn-udp" const DefaultClusterProtocol = "udp" const DefaultPort int = 3478 const DefaultLogLevel = "all:INFO" diff --git a/pkg/apis/v1alpha1/listener.go b/pkg/apis/v1alpha1/listener.go index f4ec64ee..8be57e51 100644 --- a/pkg/apis/v1alpha1/listener.go +++ b/pkg/apis/v1alpha1/listener.go @@ -72,7 +72,8 @@ func (req *ListenerConfig) Validate() error { } } - if proto == ListenerProtocolTLS || proto == ListenerProtocolDTLS { + if proto == ListenerProtocolTURNTLS || proto == ListenerProtocolTURNDTLS || + proto == ListenerProtocolTLS || proto == ListenerProtocolDTLS { if req.Cert == "" { return fmt.Errorf("empty TLS cert for %s listener", proto.String()) } @@ -115,7 +116,7 @@ func (req *ListenerConfig) String() string { pr, a, p := "udp", "-", "-" if req.Protocol != "" { - pr = req.Protocol + pr = strings.ToLower(req.Protocol) } if req.Addr != "" { a = req.Addr diff --git a/pkg/apis/v1alpha1/util.go b/pkg/apis/v1alpha1/util.go index dc779f0b..95865b6d 100644 --- a/pkg/apis/v1alpha1/util.go +++ b/pkg/apis/v1alpha1/util.go @@ -47,18 +47,26 @@ func (a AuthType) String() string { type ListenerProtocol int const ( - ListenerProtocolUDP ListenerProtocol = iota + 1 + ListenerProtocolUnknown ListenerProtocol = iota + ListenerProtocolUDP ListenerProtocolTCP ListenerProtocolTLS ListenerProtocolDTLS - ListenerProtocolUnknown + ListenerProtocolTURNUDP + ListenerProtocolTURNTCP + ListenerProtocolTURNTLS + ListenerProtocolTURNDTLS ) const ( - listenerProtocolUDPStr = "UDP" - listenerProtocolTCPStr = "TCP" - listenerProtocolTLSStr = "TLS" - listenerProtocolDTLSStr = "DTLS" + listenerProtocolUDPStr = "UDP" + listenerProtocolTCPStr = "TCP" + listenerProtocolTLSStr = "TLS" + listenerProtocolDTLSStr = "DTLS" + listenerProtocolTURNUDPStr = "TURN-UDP" + listenerProtocolTURNTCPStr = "TURN-TCP" + listenerProtocolTURNTLSStr = "TURN-TLS" + listenerProtocolTURNDTLSStr = "TURN-DTLS" ) // NewListenerProtocol parses the protocol specification @@ -72,6 +80,14 @@ func NewListenerProtocol(raw string) (ListenerProtocol, error) { return ListenerProtocolTLS, nil case listenerProtocolDTLSStr: return ListenerProtocolDTLS, nil + case listenerProtocolTURNUDPStr: + return ListenerProtocolTURNUDP, nil + case listenerProtocolTURNTCPStr: + return ListenerProtocolTURNTCP, nil + case listenerProtocolTURNTLSStr: + return ListenerProtocolTURNTLS, nil + case listenerProtocolTURNDTLSStr: + return ListenerProtocolTURNDTLS, nil default: return ListenerProtocol(ListenerProtocolUnknown), fmt.Errorf("unknown listener protocol: \"%s\"", raw) @@ -89,6 +105,14 @@ func (l ListenerProtocol) String() string { return listenerProtocolTLSStr case ListenerProtocolDTLS: return listenerProtocolDTLSStr + case ListenerProtocolTURNUDP: + return listenerProtocolTURNUDPStr + case ListenerProtocolTURNTCP: + return listenerProtocolTURNTCPStr + case ListenerProtocolTURNTLS: + return listenerProtocolTURNTLSStr + case ListenerProtocolTURNDTLS: + return listenerProtocolTURNDTLSStr default: return "" } diff --git a/reconcile_test.go b/reconcile_test.go index e8b21b73..c8140b80 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -87,7 +87,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -330,7 +330,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -416,7 +416,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -506,7 +506,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -592,7 +592,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -677,7 +677,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -766,7 +766,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -816,7 +816,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "default-listener", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.2", Port: 12345, MinRelayPort: 10, @@ -842,7 +842,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTCP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 12345, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -898,7 +898,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "newlistener", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.2", Port: 1, MinRelayPort: 10, @@ -923,7 +923,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTCP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -972,7 +972,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "newlistener", - Protocol: "tls", + Protocol: "turn-tls", Addr: "127.0.0.2", Port: 1, MinRelayPort: 10, @@ -1007,7 +1007,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Routes: []string{"allow-any"}, }, { Name: "newlistener", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.2", Port: 1, MinRelayPort: 10, @@ -1028,7 +1028,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ l := s.GetListener("default-listener") assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolUDP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") @@ -1063,7 +1063,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTCP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -1104,13 +1104,13 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Listeners: []v1alpha1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", - Protocol: "DTLS", + Protocol: "TURN-DTLS", Cert: dummyCert64, Key: dummyKey64, Routes: []string{"allow-any"}, }, { Name: "newlistener", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.2", Port: 1, MinRelayPort: 10, @@ -1135,7 +1135,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ l := s.GetListener("default-listener") assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolDTLS, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNDTLS, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, bytes.Compare(l.Cert, []byte("dummy-cert")), 0, "listener cert ok") assert.Equal(t, bytes.Compare(l.Key, []byte("dummy-key")), 0, "listener key ok") @@ -1172,7 +1172,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTCP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -1213,13 +1213,13 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Listeners: []v1alpha1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", - Protocol: "TLS", + Protocol: "TURN-TLS", Cert: dummyCert64, Key: dummyKey64, Routes: []string{"allow-any"}, }, { Name: "newlistener", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.2", Port: 1, MinRelayPort: 10, @@ -1244,7 +1244,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ l := s.GetListener("default-listener") assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTLS, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTLS, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, bytes.Compare(l.Cert, []byte("dummy-cert")), 0, "listener cert ok") assert.Equal(t, bytes.Compare(l.Key, []byte("dummy-key")), 0, "listener key ok") @@ -1281,7 +1281,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTCP, "listener proto ok") + assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -1656,7 +1656,6 @@ func TestStunnerReconcile(t *testing.T) { log.Debug("creating a stunnerd") conf, err := NewDefaultConfig("turn://user:pass@127.0.0.1:3478") assert.NoError(t, err, err) - conf.Admin.LogLevel = stunnerTestLoglevel log.Debug("creating a stunnerd") @@ -1826,7 +1825,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3480, Routes: []string{ @@ -1856,7 +1855,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3480, Routes: []string{ @@ -1891,7 +1890,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -1899,7 +1898,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3480, Routes: []string{ @@ -1934,7 +1933,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -1942,7 +1941,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -1977,7 +1976,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -1985,7 +1984,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2020,7 +2019,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2028,7 +2027,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2064,7 +2063,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2072,7 +2071,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2108,7 +2107,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2116,7 +2115,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2152,7 +2151,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2160,7 +2159,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2195,7 +2194,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2204,7 +2203,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2243,7 +2242,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2252,7 +2251,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2286,7 +2285,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2295,7 +2294,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, }, { Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3479, Routes: []string{ @@ -2334,7 +2333,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2373,7 +2372,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2414,7 +2413,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2450,7 +2449,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2485,7 +2484,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2522,7 +2521,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2557,7 +2556,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-ok", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2623,7 +2622,7 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "default-listener", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -2659,7 +2658,7 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "default-listener", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ diff --git a/server.go b/server.go index 0292a04e..bf481c2d 100644 --- a/server.go +++ b/server.go @@ -36,7 +36,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { addr := fmt.Sprintf("%s:%d", l.Addr.String(), l.Port) switch l.Proto { - case v1alpha1.ListenerProtocolUDP: + case v1alpha1.ListenerProtocolTURNUDP: socketPool := util.NewPacketConnPool(l.Net, s.udpThreadNum) s.log.Infof("setting up UDP listener socket pool at %s with %d readloop threads", @@ -58,7 +58,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { pConns = append(pConns, conn) } - case v1alpha1.ListenerProtocolTCP: + case v1alpha1.ListenerProtocolTURNTCP: s.log.Debugf("setting up TCP listener at %s", addr) tcpListener, err := net.Listen("tcp", addr) @@ -78,7 +78,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { l.Conns = append(l.Conns, conn) // cannot test this on vnet, no TLS in vnet.Net - case v1alpha1.ListenerProtocolTLS: + case v1alpha1.ListenerProtocolTURNTLS: s.log.Debugf("setting up TLS/TCP listener at %s", addr) cer, err := tls.X509KeyPair(l.Cert, l.Key) @@ -105,7 +105,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { lConns = append(lConns, conn) l.Conns = append(l.Conns, conn) - case v1alpha1.ListenerProtocolDTLS: + case v1alpha1.ListenerProtocolTURNDTLS: s.log.Debugf("setting up DTLS/UDP listener at %s", addr) cer, err := tls.X509KeyPair(l.Cert, l.Key) diff --git a/server_unix_test.go b/server_unix_test.go index b94f32f5..2515d4d5 100644 --- a/server_unix_test.go +++ b/server_unix_test.go @@ -33,7 +33,7 @@ var TestStunnerConfigsMultithreadedUDP = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "127.0.0.1", Port: 23478, PublicAddr: "1.2.3.4", @@ -127,7 +127,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { log.Debug("creating a turncat client") stunnerURI := fmt.Sprintf("turn://127.0.0.1:23478?transport=%s", proto) clientProto := "tcp" - if proto == "udp" || proto == "dtls" { + if proto == "turn-udp" || proto == "turn-dtls" { clientProto = "udp" } testTurncatConfig := TurncatConfig{ @@ -196,7 +196,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { func BenchmarkUDPServer(b *testing.B) { for i := 1; i <= 4; i++ { b.Run(fmt.Sprintf("udp:thread_num=%d", i), func(b *testing.B) { - RunBenchmarkServer(b, "udp", i) + RunBenchmarkServer(b, "turn-udp", i) }) } } @@ -205,7 +205,7 @@ func BenchmarkUDPServer(b *testing.B) { // threads. Setup: `client --tcp--> turncat --tcp--> stunner --udp--> sink` func BenchmarkTCPServer(b *testing.B) { b.Run("tcp", func(b *testing.B) { - RunBenchmarkServer(b, "tcp", 0) + RunBenchmarkServer(b, "turn-tcp", 0) }) } @@ -213,7 +213,7 @@ func BenchmarkTCPServer(b *testing.B) { // threads. Setup: `client --tcp--> turncat --tls--> stunner --udp--> sink` func BenchmarkTLSServer(b *testing.B) { b.Run("tls", func(b *testing.B) { - RunBenchmarkServer(b, "tls", 0) + RunBenchmarkServer(b, "turn-tls", 0) }) } @@ -221,6 +221,6 @@ func BenchmarkTLSServer(b *testing.B) { // threads. Setup: `client --udp--> turncat --dtls--> stunner --udp--> sink` func BenchmarkDTLSServer(b *testing.B) { b.Run("dtls", func(b *testing.B) { - RunBenchmarkServer(b, "dtls", 0) + RunBenchmarkServer(b, "turn-dtls", 0) }) } diff --git a/stunner_test.go b/stunner_test.go index 1244d6b1..984fd61f 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -282,7 +282,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "127.0.0.1", Port: 23478, PublicAddr: "1.2.3.4", @@ -311,7 +311,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "127.0.0.1", Port: 23478, PublicAddr: "1.2.3.4", @@ -341,7 +341,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "tcp", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.1", Port: 23478, PublicAddr: "1.2.3.4", @@ -370,7 +370,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "tcp", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.1", Port: 23478, PublicAddr: "1.2.3.4", @@ -400,7 +400,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "tls", - Protocol: "tls", + Protocol: "turn-tls", Addr: "127.0.0.1", PublicAddr: "1.2.3.4", PublicPort: 3478, @@ -431,7 +431,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "tls", - Protocol: "tls", + Protocol: "turn-tls", Addr: "127.0.0.1", Port: 23478, PublicAddr: "1.2.3.4", @@ -463,7 +463,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "dtls", - Protocol: "dtls", + Protocol: "turn-dtls", Addr: "127.0.0.1", PublicAddr: "1.2.3.4", PublicPort: 3478, @@ -493,7 +493,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ // }, // Listeners: []v1alpha1.ListenerConfig{{ // Name: "dtls", - // Protocol: "dtls", + // Protocol: "turn-dtls", // Addr: "127.0.0.1", // Port: 23478, // Routes: []string{"allow-any"}, @@ -576,14 +576,14 @@ func testStunnerLocalhost(t *testing.T, udpThreadNum int, tests []TestStunnerCon log.Debug("creating a client") var lconn net.PacketConn switch proto { - case "udp": - lconn, err = net.ListenPacket("udp4", "0.0.0.0:0") + case "turn-udp": + lconn, err = net.ListenPacket("udp", "0.0.0.0:0") assert.NoError(t, err, "cannot create UDP client socket") - case "tcp": + case "turn-tcp": conn, cErr := net.Dial("tcp", stunnerAddr) assert.NoError(t, cErr, "cannot create TCP client socket") lconn = turn.NewSTUNConn(conn) - case "tls": + case "turn-tls": cer, err := tls.X509KeyPair(certPem, keyPem) assert.NoError(t, err, "cannot create certificate for TLS client socket") conn, err := tls.Dial("tcp", stunnerAddr, &tls.Config{ @@ -593,7 +593,7 @@ func testStunnerLocalhost(t *testing.T, udpThreadNum int, tests []TestStunnerCon }) assert.NoError(t, err, "cannot create TLS client socket") lconn = turn.NewSTUNConn(conn) - case "dtls": + case "turn-dtls": cer, err := tls.X509KeyPair(certPem, keyPem) assert.NoError(t, err, "cannot create certificate for DTLS client socket") // for some reason dtls.Listen requires a UDPAddr and not an addr string @@ -668,7 +668,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"echo-server-cluster"}, @@ -700,7 +700,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -733,7 +733,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -767,7 +767,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -801,7 +801,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -842,7 +842,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -883,7 +883,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -921,7 +921,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -958,7 +958,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -992,7 +992,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -1026,7 +1026,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ @@ -1061,7 +1061,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp", - Protocol: "udp", + Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{ diff --git a/turncat.go b/turncat.go index 7f836d3a..94e460ef 100644 --- a/turncat.go +++ b/turncat.go @@ -5,6 +5,7 @@ import ( "crypto/tls" "fmt" "net" + "net/url" "os" "strings" "sync" @@ -78,8 +79,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { log.Tracef("Resolving TURN server address: %s", config.ServerAddr) server, sErr := ParseUri(config.ServerAddr) if sErr != nil { - return nil, fmt.Errorf("error resolving server address %s: %s", - config.ServerAddr, sErr.Error()) + return nil, fmt.Errorf("error resolving server address %s: %w", config.ServerAddr, sErr) } if server.Address == "" || server.Port == 0 { return nil, fmt.Errorf("error resolving TURN server address %s: empty address (\"%s\") "+ @@ -87,26 +87,21 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { } log.Tracef("Resolving listener address: %s", config.ListenerAddr) - listener, lErr := ParseUri(config.ListenerAddr) + listener, lErr := url.Parse(config.ListenerAddr) if lErr != nil { - return nil, fmt.Errorf("error resolving listener address %s: %s", - config.ListenerAddr, lErr.Error()) - } - if listener.Port == 0 { - return nil, fmt.Errorf("error resolving listener address %s: invalid port (%d)", - config.ListenerAddr, listener.Port) + return nil, fmt.Errorf("error parsing listener address %q: %w", config.ListenerAddr, lErr) } + listenerProtocol := strings.ToLower(listener.Scheme) log.Tracef("Resolving peer address: %s", config.PeerAddr) - peer, pErr := ParseUri(config.PeerAddr) + peer, pErr := url.Parse(config.PeerAddr) if pErr != nil { - return nil, fmt.Errorf("error resolving peer address %s: %s", - config.PeerAddr, pErr.Error()) + return nil, fmt.Errorf("error parsing peer address %q: %w", config.PeerAddr, pErr) } - if peer.Address == "" || peer.Port == 0 || !strings.HasPrefix(peer.Protocol, "udp") { - return nil, fmt.Errorf("error resolving peer address %s: invalid protocol (\"%s\"), "+ - "empty address (\"%s\") or invalid port (%d)", config.PeerAddr, - peer.Protocol, peer.Address, peer.Port) + // default to UDP + peerAddress, err := net.ResolveUDPAddr("udp", peer.Host) + if err != nil { + return nil, fmt.Errorf("error resolving peer address %q: %w", config.PeerAddr, err) } if config.Realm == "" { @@ -119,34 +114,45 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { var listenerConn interface{} listenerConf := &net.ListenConfig{Control: reuseAddr} - switch listener.Protocol { + var listenerAddress net.Addr + switch listenerProtocol { case "file": listenerConn = util.NewFileConn(os.Stdin) - case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6": - l, err := listenerConf.ListenPacket(context.Background(), listener.Addr.Network(), - listener.Addr.String()) + case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6", "turn-udp", "turn-dtls": + addr, err := net.ResolveUDPAddr("udp", listener.Host) + if err != nil { + return nil, fmt.Errorf("error resolving listener address %q: %w", config.ListenerAddr, err) + } + + l, err := listenerConf.ListenPacket(context.Background(), addr.Network(), addr.String()) if err != nil { return nil, fmt.Errorf("cannot create listening client packet socket at %s: %s", config.ListenerAddr, err) } + listenerAddress = addr listenerConn = l - case "tcp", "tcp4", "tcp6", "unix", "unixpacket": - l, err := listenerConf.Listen(context.Background(), listener.Addr.Network(), listener.Addr.String()) + case "tcp", "tcp4", "tcp6", "unix", "unixpacket", "turn-tcp", "turn-tls": + addr, err := net.ResolveTCPAddr("tcp", listener.Host) + if err != nil { + return nil, fmt.Errorf("error resolving listener address %q: %w", config.ListenerAddr, err) + } + + l, err := listenerConf.Listen(context.Background(), addr.Network(), addr.String()) if err != nil { return nil, fmt.Errorf("cannot create listening client socket at %s: %s", config.ListenerAddr, err) } + listenerAddress = addr listenerConn = l default: - return nil, fmt.Errorf("unknown client protocol %s for client %s", - listener.Addr.Network(), config.ListenerAddr) + return nil, fmt.Errorf("unknown client protocol %s", listenerProtocol) } t := &Turncat{ - listenerAddr: listener.Addr, + listenerAddr: listenerAddress, serverAddr: server.Addr, serverProto: server.Protocol, - peerAddr: peer.Addr, + peerAddr: peerAddress, listenerConn: listenerConn, connTrack: make(map[string]*connection), lock: new(sync.Mutex), @@ -157,7 +163,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { log: log, } - switch t.listenerAddr.Network() { + switch listenerAddress.Network() { case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6": // client connection is a packet conn, write our own Listen/Accept loop for UDP // main loop: for every new packet we create a new connection and connect it back to the client @@ -169,12 +175,14 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { // client connection is file go t.runListenFile() default: - t.log.Errorf("internal error: unknown client protocol %s for client %s:%s", - t.listenerAddr.Network(), t.listenerAddr.Network(), t.listenerAddr.String()) + t.log.Errorf("internal error: unknown client protocol %q for client %s:%s", + listenerAddress.Network(), listenerAddress.Network(), listenerAddress.String()) } - log.Infof("Turncat client listening on %s, TURN server: %s, peer: %s", - config.ListenerAddr, config.ServerAddr, config.PeerAddr) + log.Infof("Turncat client listening on %s:%s, TURN server: %s, peer: %s:%s", + listenerAddress.Network(), listenerAddress.String(), + config.ServerAddr, + peerAddress.Network(), peerAddress.String()) return t, nil } @@ -219,8 +227,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { conn.clientAddr = clientAddr conn.clientConn = clientConn - t.log.Tracef("Setting up TURN client to server %s:%s", - t.serverAddr.Network(), t.serverAddr.String()) + t.log.Tracef("Setting up TURN client to server %s:%s", t.serverAddr.Network(), t.serverAddr.String()) user, passwd, errAuth := t.authGen() if errAuth != nil { @@ -230,22 +237,22 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { // connection for the TURN client var turnConn net.PacketConn - switch t.serverProto { - case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6": + switch strings.ToLower(t.serverProto) { + case "turn-udp": t, err := net.ListenPacket(t.serverAddr.Network(), "0.0.0.0:0") if err != nil { return nil, fmt.Errorf("cannot allocate TURN listening packet socket for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } turnConn = t - case "tcp", "tcp4", "tcp6", "unix", "unixpacket": + case "turn-tcp": c, err := net.Dial(t.serverAddr.Network(), t.serverAddr.String()) if err != nil { return nil, fmt.Errorf("cannot allocate TURN socket for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } turnConn = turn.NewSTUNConn(c) - case "tls": + case "turn-tls": // cert, err := tls.LoadX509KeyPair(certFile.Name(), keyFile.Name()) // assert.NoError(t, err, "cannot create certificate for TLS client socket") c, err := tls.Dial("tcp", t.serverAddr.String(), &tls.Config{ @@ -257,7 +264,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { clientAddr.Network(), clientAddr.String(), err) } turnConn = turn.NewSTUNConn(c) - case "dtls": + case "turn-dtls": // cert, err := tls.LoadX509KeyPair(certFile.Name(), keyFile.Name()) // assert.NoError(t, err, "cannot create certificate for DTLS client socket") udpAddr, _ := net.ResolveUDPAddr("udp", t.serverAddr.String()) @@ -270,8 +277,8 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { } turnConn = turn.NewSTUNConn(conn) default: - return nil, fmt.Errorf("unknown TURN server protocol %s for client %s:%s", - t.serverAddr.Network(), clientAddr.Network(), clientAddr.String()) + return nil, fmt.Errorf("unknown TURN server protocol %q for client %s:%s", + t.serverProto, clientAddr.Network(), clientAddr.String()) } turnClient, err := turn.NewClient(&turn.ClientConfig{ @@ -450,8 +457,7 @@ func (t *Turncat) runListenPacket() { conn, err := t.newConnection(clientConn) if err != nil { t.lock.Unlock() - t.log.Warnf("relay setup failed for client %s, dropping client connection", - caddr) + t.log.Warnf("relay setup failed for client %s: %s", caddr, err.Error()) continue } diff --git a/turncat_test.go b/turncat_test.go index 7b59e29b..0fb5e938 100644 --- a/turncat_test.go +++ b/turncat_test.go @@ -3,6 +3,7 @@ package stunner import ( "fmt" "net" + "net/url" "strings" "testing" "time" @@ -19,7 +20,7 @@ import ( var turncatTestLoglevel string = "all:ERROR" -//var turncatTestLoglevel string = "all:TRACE" +// var turncatTestLoglevel string = "all:TRACE" var sharedSecret = "my-secret" var defaultDuration = "10m" @@ -38,7 +39,7 @@ type turncatEchoTestConfig struct { // client lconn net.Conn // peer - peer *StunnerUri + peer net.Addr loggerFactory logging.LoggerFactory } @@ -46,8 +47,7 @@ func turncatEchoTest(conf turncatEchoTestConfig) { t := conf.t log := conf.loggerFactory.NewLogger("test") - peerAddr := fmt.Sprintf("%s:%d", conf.peer.Address, conf.peer.Port) - echoConn, err := net.ListenPacket("udp4", peerAddr) + echoConn, err := net.ListenPacket(conf.peer.Network(), conf.peer.String()) assert.NoError(t, err, "cannot allocate echo server connection") go func() { @@ -115,7 +115,8 @@ func TestTurncatPlaintext(t *testing.T) { err := stunner.Reconcile(v1alpha1.StunnerConfig{ ApiVersion: "v1alpha1", Admin: v1alpha1.AdminConfig{ - LogLevel: turncatTestLoglevel, + LogLevel: turncatTestLoglevel, + MetricsEndpoint: "", }, Auth: v1alpha1.AuthConfig{ Type: "plaintext", @@ -126,13 +127,13 @@ func TestTurncatPlaintext(t *testing.T) { }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-listener-23478", - Protocol: "udp", + Protocol: "turn-udp", Addr: "127.0.0.1", Port: 23478, Routes: []string{"allow-any"}, }, { Name: "tcp-listener-23478", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.1", Port: 23478, Routes: []string{"allow-any"}, @@ -178,35 +179,32 @@ func TestTurncatPlaintext(t *testing.T) { } for _, c := range testTurncatConfigs { - listener, err := ParseUri(c.ListenerAddr) + listener, err := url.Parse(c.ListenerAddr) assert.NoError(t, err, "cannot parse turncat listener URI") server, err := ParseUri(c.ServerAddr) assert.NoError(t, err, "cannot parse server URI") testName := fmt.Sprintf("TestTurncat_NewTurncat_Plaintext_client:%s_server:%s", - listener.Protocol, server.Protocol) + listener.Scheme, server.Protocol) t.Run(testName, func(t *testing.T) { log.Debugf("-------------- Running test: %s -------------", testName) - peer, err := ParseUri(c.PeerAddr) - assert.NoError(t, err, "cannot parse peer URI") log.Debug("creating turncat relay") turncat, err := NewTurncat(&c) assert.NoError(t, err, "cannot create turncat relay") - lconn, err := net.Dial(listener.Protocol, - fmt.Sprintf("%s:%d", listener.Address, listener.Port)) + lconn, err := net.Dial(turncat.listenerAddr.Network(), turncat.listenerAddr.String()) assert.NoError(t, err, "cannot create client socket") - if strings.HasPrefix(listener.Protocol, "tcp") { + if strings.HasPrefix(turncat.listenerAddr.Network(), "tcp") { // prevent "addess already in use" errors: close sends RST assert.NoError(t, lconn.(*net.TCPConn).SetLinger(0), "cannnot set TCP linger") } - testConfig := turncatEchoTestConfig{t, stunner, lconn, peer, logger} + testConfig := turncatEchoTestConfig{t, stunner, lconn, turncat.peerAddr, logger} turncatEchoTest(testConfig) turncat.Close() @@ -243,13 +241,13 @@ func TestTurncatLongterm(t *testing.T) { }, Listeners: []v1alpha1.ListenerConfig{{ Name: "udp-listener-23478", - Protocol: "udp", + Protocol: "turn-udp", Addr: "127.0.0.1", Port: 23478, Routes: []string{"allow-any"}, }, { Name: "tcp-listener-23478", - Protocol: "tcp", + Protocol: "turn-tcp", Addr: "127.0.0.1", Port: 23478, Routes: []string{"allow-any"}, @@ -295,35 +293,32 @@ func TestTurncatLongterm(t *testing.T) { } for _, c := range testTurncatConfigs { - listener, err := ParseUri(c.ListenerAddr) + listener, err := url.Parse(c.ListenerAddr) assert.NoError(t, err, "cannot parse turncat listener URI") server, err := ParseUri(c.ServerAddr) assert.NoError(t, err, "cannot parse server URI") testName := fmt.Sprintf("TestTurncat_NewTurncat_Longterm_client:%s_server:%s", - listener.Protocol, server.Protocol) + listener.Scheme, server.Protocol) t.Run(testName, func(t *testing.T) { log.Debugf("-------------- Running test: %s -------------", testName) - peer, err := ParseUri(c.PeerAddr) - assert.NoError(t, err, "cannot parse peer URI") log.Debug("creating turncat relay") turncat, err := NewTurncat(&c) assert.NoError(t, err, "cannot create turncat relay") - lconn, err := net.Dial(listener.Protocol, - fmt.Sprintf("%s:%d", listener.Address, listener.Port)) + lconn, err := net.Dial(turncat.listenerAddr.Network(), turncat.listenerAddr.String()) assert.NoError(t, err, "cannot create client socket") - if strings.HasPrefix(listener.Protocol, "tcp") { + if strings.HasPrefix(turncat.listenerAddr.Network(), "tcp") { // prevent "addess already in use" errors: close sends RST assert.NoError(t, lconn.(*net.TCPConn).SetLinger(0), "cannnot set TCP linger") } - testConfig := turncatEchoTestConfig{t, stunner, lconn, peer, logger} + testConfig := turncatEchoTestConfig{t, stunner, lconn, turncat.peerAddr, logger} turncatEchoTest(testConfig) turncat.Close() diff --git a/uri.go b/uri.go index f61bed00..7386688a 100644 --- a/uri.go +++ b/uri.go @@ -45,28 +45,23 @@ func ParseUri(uri string) (*StunnerUri, error) { s.Password = password } - proto := strings.ToLower(u.Scheme) - if proto == "turn" { - q := u.Query() - if len(q["transport"]) > 0 { - proto = strings.ToLower(q["transport"][0]) - } else { - proto = "udp" - } + proto, err := getStunnerProtoForURI(u) + if err != nil { + return nil, err } s.Protocol = proto port, _ := strconv.Atoi(u.Port()) s.Port = port - switch proto { - case "udp", "udp4", "udp6", "dtls": + switch strings.ToLower(proto) { + case "udp", "udp4", "udp6", "dtls", "turn-udp", "turn-dtls": a, err := net.ResolveUDPAddr("udp", s.Address+":"+u.Port()) if err != nil { return nil, err } s.Addr = a - case "tcp", "tcp4", "tcp6", "tls": + case "tcp", "tcp4", "tcp6", "tls", "turn-tcp", "turn-tls": a, err := net.ResolveTCPAddr("tcp", s.Address+":"+u.Port()) if err != nil { return nil, err @@ -100,16 +95,16 @@ func GetUriFromListener(req *v1alpha1.ListenerConfig) (string, error) { service, protocol := "", "" switch proto { - case v1alpha1.ListenerProtocolUDP: + case v1alpha1.ListenerProtocolTURNUDP: service = "turn" protocol = "udp" - case v1alpha1.ListenerProtocolTCP: + case v1alpha1.ListenerProtocolTURNTCP: service = "turn" protocol = "tcp" - case v1alpha1.ListenerProtocolDTLS: + case v1alpha1.ListenerProtocolTURNDTLS: service = "turns" protocol = "udp" - case v1alpha1.ListenerProtocolTLS: + case v1alpha1.ListenerProtocolTURNTLS: service = "turns" protocol = "tcp" } @@ -144,6 +139,40 @@ func GetTurnUris(req *v1alpha1.StunnerConfig) ([]string, error) { return ret, nil } +func getStunnerProtoForURI(u *url.URL) (string, error) { + scheme := strings.ToLower(u.Scheme) + if scheme == "" { + scheme = "turn" + } + + proto := "udp" + q := u.Query() + if len(q["transport"]) > 0 { + proto = strings.ToLower(q["transport"][0]) + } + + // fully specified protocol names (ignore "turns" scheme for compatibility) + switch proto { + case "tls": + return "TURN-TLS", nil + case "dtls": + return "TURN-DTLS", nil + } + + // using RFC7065 compatible URIs + if scheme == "turn" || proto == "udp" { + return "TURN-UDP", nil + } else if scheme == "turn" || proto == "tcp" { + return "TURN-TCP", nil + } else if scheme == "turns" || proto == "udp" { + return "TURN-DTLS", nil + } else if scheme == "turns" || proto == "tcp" { + return "TURN-TLS", nil + } + + return "", fmt.Errorf("Invalid scheme/protocol in URI %q", u.String()) +} + func reuseAddr(network, address string, conn syscall.RawConn) error { return conn.Control(func(descriptor uintptr) { _ = syscall.SetsockoptInt(int(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1) From 549b6ff76d8a961f05ce997ee53bb5590841905f Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 21 Sep 2023 22:23:38 +0200 Subject: [PATCH 011/243] fix: Streamline TURN URI parsing --- cmd/turncat/main.go | 3 +- config_test.go | 58 ++++++++++++++++++++++++++++- pkg/apis/v1alpha1/listener.go | 59 +++++++++++++++++++++++------ uri.go | 70 ++++++++++++++++------------------- 4 files changed, 136 insertions(+), 54 deletions(-) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 844c72d6..bfe83759 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -282,8 +282,7 @@ func getStunnerURI(config *stunnerv1alpha1.StunnerConfig) (string, error) { return "", fmt.Errorf("no protocol for listener %q", l.Name) } - return fmt.Sprintf("%s://%s:%d", strings.ToLower(l.Protocol), l.PublicAddr, - l.PublicPort), nil + return stunner.GetStandardURLFromListener(&l) } func parseK8sDef(def string) (string, string, string, error) { diff --git a/config_test.go b/config_test.go index 82047f36..d9fd7d79 100644 --- a/config_test.go +++ b/config_test.go @@ -5,6 +5,7 @@ import ( "fmt" "net" "os" + "strings" "testing" "time" @@ -35,8 +36,8 @@ func TestStunnerDefaultServerVNet(t *testing.T) { for _, conf := range []string{ "turn://user1:passwd1@1.2.3.4:3478?transport=udp", - "udp://user1:passwd1@1.2.3.4:3478?transport=udp", - "udp://user1:passwd1@1.2.3.4:3478", + "turn://user1:passwd1@1.2.3.4?transport=udp", + "turn://user1:passwd1@1.2.3.4:3478", } { testName := fmt.Sprintf("TestStunner_NewDefaultConfig_URI:%s", conf) t.Run(testName, func(t *testing.T) { @@ -233,6 +234,59 @@ func TestStunnerConfigFileWatcher(t *testing.T) { stunner.Close() } +func TestStunnerURIParser(t *testing.T) { + lim := test.TimeOut(time.Second * 30) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + // loggerFactory := logger.NewLoggerFactory("all:TRACE") + loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) + log := loggerFactory.NewLogger("test") + + for _, conf := range []struct { + uri string + su StunnerUri + }{ + // udp + {"turn://user1:passwd1@1.2.3.4:3478?transport=udp", StunnerUri{"turn-udp", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turn://user1:passwd1@1.2.3.4?transport=udp", StunnerUri{"turn-udp", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turn://user1:passwd1@1.2.3.4:3478", StunnerUri{"turn-udp", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + // tcp + {"turn://user1:passwd1@1.2.3.4:3478?transport=tcp", StunnerUri{"turn-tcp", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turn://user1:passwd1@1.2.3.4?transport=tcp", StunnerUri{"turn-tcp", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + // tls - old style + {"turn://user1:passwd1@1.2.3.4:3478?transport=tls", StunnerUri{"turn-tls", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turn://user1:passwd1@1.2.3.4?transport=tls", StunnerUri{"turn-tls", "1.2.3.4", "user1", "passwd1", 443, nil}}, + // tls - RFC style + {"turns://user1:passwd1@1.2.3.4:3478?transport=tcp", StunnerUri{"turn-tls", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turns://user1:passwd1@1.2.3.4?transport=tcp", StunnerUri{"turn-tls", "1.2.3.4", "user1", "passwd1", 443, nil}}, + // dtls - old style + {"turn://user1:passwd1@1.2.3.4:3478?transport=dtls", StunnerUri{"turn-dtls", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turn://user1:passwd1@1.2.3.4?transport=dtls", StunnerUri{"turn-dtls", "1.2.3.4", "user1", "passwd1", 443, nil}}, + // dtls - RFC style + {"turns://user1:passwd1@1.2.3.4:3478?transport=udp", StunnerUri{"turn-dtls", "1.2.3.4", "user1", "passwd1", 3478, nil}}, + {"turns://user1:passwd1@1.2.3.4?transport=udp", StunnerUri{"turn-dtls", "1.2.3.4", "user1", "passwd1", 443, nil}}, + // no cred + {"turn://1.2.3.4:3478?transport=udp", StunnerUri{"turn-udp", "1.2.3.4", "", "", 3478, nil}}, + {"turn://1.2.3.4?transport=udp", StunnerUri{"turn-udp", "1.2.3.4", "", "", 3478, nil}}, + {"turn://1.2.3.4", StunnerUri{"turn-udp", "1.2.3.4", "", "", 3478, nil}}, + } { + testName := fmt.Sprintf("TestStunnerURIParser:%s", conf.uri) + t.Run(testName, func(t *testing.T) { + log.Debugf("-------------- Running test: %s -------------", testName) + u, err := ParseUri(conf.uri) + assert.NoError(t, err, "URI parser") + assert.Equal(t, strings.ToLower(conf.su.Protocol), strings.ToLower(u.Protocol), "uri protocol") + assert.Equal(t, conf.su.Address, u.Address, "uri address") + assert.Equal(t, conf.su.Username, u.Username, "uri username") + assert.Equal(t, conf.su.Password, u.Password, "uri password") + assert.Equal(t, conf.su.Port, u.Port, "uri port") + }) + } +} + func checkDefaultConfig(t *testing.T, c *v1alpha1.StunnerConfig, proto string) { assert.Equal(t, "plaintext", c.Auth.Type, "auth-type") assert.Equal(t, "user1", c.Auth.Credentials["username"], "username") diff --git a/pkg/apis/v1alpha1/listener.go b/pkg/apis/v1alpha1/listener.go index 8be57e51..7a8904d0 100644 --- a/pkg/apis/v1alpha1/listener.go +++ b/pkg/apis/v1alpha1/listener.go @@ -114,16 +114,6 @@ func (req *ListenerConfig) String() string { n = req.Name } - pr, a, p := "udp", "-", "-" - if req.Protocol != "" { - pr = strings.ToLower(req.Protocol) - } - if req.Addr != "" { - a = req.Addr - } - if req.Port != 0 { - p = fmt.Sprintf("%d", req.Port) - } min, max := 0, 65535 if req.MinRelayPort != 0 { min = req.MinRelayPort @@ -131,9 +121,10 @@ func (req *ListenerConfig) String() string { if req.MaxRelayPort != 0 { max = req.MaxRelayPort } - status = append(status, fmt.Sprintf("%s://%s:%s<%d-%d>", pr, a, p, min, max)) + uri, _ := req.GetListenerURI(false) //nolint:errcheck + status = append(status, fmt.Sprintf("%s<%d-%d>", uri, min, max)) - a, p = "-", "-" + a, p := "-", "-" if req.PublicAddr != "" { a = req.PublicAddr } @@ -154,3 +145,47 @@ func (req *ListenerConfig) String() string { return fmt.Sprintf("%q:{%s}", n, strings.Join(status, ",")) } + +// GetListenerURI is a helper that can output two types of Listener URIs: one with :// after the scheme or one with only : (as per RFC7065). +func (req *ListenerConfig) GetListenerURI(rfc7065 bool) (string, error) { + proto, err := NewListenerProtocol(req.Protocol) + if err != nil { + return "", err + } + + service, protocol := "", "" + switch proto { + case ListenerProtocolTURNUDP: + service = "turn" + protocol = "udp" + case ListenerProtocolTURNTCP: + service = "turn" + protocol = "tcp" + case ListenerProtocolTURNDTLS: + service = "turns" + protocol = "udp" + case ListenerProtocolTURNTLS: + service = "turns" + protocol = "tcp" + } + + addr := req.PublicAddr + if addr == "" { + // fallback to server addr + addr = req.Addr + } + + port := req.PublicPort + if port == 0 { + // fallback to server addr + port = req.Port + } + + var uri string + if rfc7065 { + uri = fmt.Sprintf("%s:%s:%d?transport=%s", service, addr, port, protocol) + } else { + uri = fmt.Sprintf("%s://%s:%d?transport=%s", service, addr, port, protocol) + } + return uri, nil +} diff --git a/uri.go b/uri.go index 7386688a..be9bafc8 100644 --- a/uri.go +++ b/uri.go @@ -51,9 +51,6 @@ func ParseUri(uri string) (*StunnerUri, error) { } s.Protocol = proto - port, _ := strconv.Atoi(u.Port()) - s.Port = port - switch strings.ToLower(proto) { case "udp", "udp4", "udp6", "dtls", "turn-udp", "turn-dtls": a, err := net.ResolveUDPAddr("udp", s.Address+":"+u.Port()) @@ -83,46 +80,43 @@ func ParseUri(uri string) (*StunnerUri, error) { return nil, fmt.Errorf("invalid protocol: %s", proto) } - return &s, nil -} + defaultPort := 3478 + if strings.ToLower(proto) == "turn-tls" || strings.ToLower(proto) == "turn-dtls" { + defaultPort = 443 + } -// GetUriFromListener returns a standard TURN URI from a listener config -func GetUriFromListener(req *v1alpha1.ListenerConfig) (string, error) { - proto, err := v1alpha1.NewListenerProtocol(req.Protocol) + port, err := strconv.Atoi(u.Port()) if err != nil { - return "", err + port = defaultPort } + s.Port = port - service, protocol := "", "" - switch proto { - case v1alpha1.ListenerProtocolTURNUDP: - service = "turn" - protocol = "udp" - case v1alpha1.ListenerProtocolTURNTCP: - service = "turn" - protocol = "tcp" - case v1alpha1.ListenerProtocolTURNDTLS: - service = "turns" - protocol = "udp" - case v1alpha1.ListenerProtocolTURNTLS: - service = "turns" - protocol = "tcp" - } + return &s, nil +} - addr := req.PublicAddr - if addr == "" { - // fallback to server addr - addr = req.Addr +func (u *StunnerUri) String() string { + req := v1alpha1.ListenerConfig{ + Protocol: u.Protocol, + PublicAddr: u.Address, + PublicPort: u.Port, } - port := req.PublicPort - if port == 0 { - // fallback to server addr - port = req.Port + uri, err := GetStandardURLFromListener(&req) + if err != nil { + return "" } - uri := fmt.Sprintf("%s:%s:%d?transport=%s", service, addr, port, protocol) - return uri, nil + return uri +} + +// GetUriFromListener returns a standard TURN URI as per RFC7065from a listener config. +func GetUriFromListener(req *v1alpha1.ListenerConfig) (string, error) { + return req.GetListenerURI(true) +} + +// GetStandardURLFromListener returns a standard URL (that can be parsed using net/url) from a listener config. +func GetStandardURLFromListener(req *v1alpha1.ListenerConfig) (string, error) { + return req.GetListenerURI(false) } // GetUriFromListener returns a standard TURN URI from a listener config @@ -160,13 +154,13 @@ func getStunnerProtoForURI(u *url.URL) (string, error) { } // using RFC7065 compatible URIs - if scheme == "turn" || proto == "udp" { + if scheme == "turn" && proto == "udp" { return "TURN-UDP", nil - } else if scheme == "turn" || proto == "tcp" { + } else if scheme == "turn" && proto == "tcp" { return "TURN-TCP", nil - } else if scheme == "turns" || proto == "udp" { + } else if scheme == "turns" && proto == "udp" { return "TURN-DTLS", nil - } else if scheme == "turns" || proto == "tcp" { + } else if scheme == "turns" && proto == "tcp" { return "TURN-TLS", nil } From 0fe3d2f6c5b1b1dc415638e3df6a64c0d961c4f9 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 22 Sep 2023 17:24:53 +0200 Subject: [PATCH 012/243] fix: Make sure STUNner listens on all interfaces So far stunnerd has bound TURN server sockets to the corresponding listener address and used the same listener address to create the relayed transport address. This works well as long as it is deployed in a Kubernetes pod that has only a single IP address, but breaks on baremetal servers with multiple IPs. This change causes stunnerd to bind listener sockets to 0.0.0.0: and only use the listener address to create the relayed transport address. --- pkg/apis/v1alpha1/listener.go | 3 +-- reconcile_test.go | 9 ++++++--- server.go | 10 +++++++--- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/pkg/apis/v1alpha1/listener.go b/pkg/apis/v1alpha1/listener.go index 7a8904d0..e4c74ef2 100644 --- a/pkg/apis/v1alpha1/listener.go +++ b/pkg/apis/v1alpha1/listener.go @@ -121,8 +121,7 @@ func (req *ListenerConfig) String() string { if req.MaxRelayPort != 0 { max = req.MaxRelayPort } - uri, _ := req.GetListenerURI(false) //nolint:errcheck - status = append(status, fmt.Sprintf("%s<%d-%d>", uri, min, max)) + status = append(status, fmt.Sprintf("turn://0.0.0.0:%d<%d-%d>", req.Port, min, max)) a, p := "-", "-" if req.PublicAddr != "" { diff --git a/reconcile_test.go b/reconcile_test.go index c8140b80..d0d8d2f8 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -2643,8 +2643,9 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ echoResult: true, }, { - // tcp will fail on vnet: must rollback for the test to succeed - testName: "reconcile listener with a changed protocol", + // this will trigger an error at a later stage of reconciliation that the + // validation phase cannot catch and cause a rollback + testName: "reconcile listener with an invalid TLS cert/key", config: v1alpha1.StunnerConfig{ ApiVersion: "v1alpha1", Admin: v1alpha1.AdminConfig{ @@ -2658,9 +2659,11 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ }, Listeners: []v1alpha1.ListenerConfig{{ Name: "default-listener", - Protocol: "turn-tcp", + Protocol: "turn-tls", Addr: "1.2.3.4", Port: 3478, + Key: "ZHVtbXkK", // base64: dummy + Cert: "ZHVtbXkK", // base64: dummy Routes: []string{ "echo-server-cluster", }, diff --git a/server.go b/server.go index bf481c2d..f00fa030 100644 --- a/server.go +++ b/server.go @@ -22,10 +22,11 @@ func (s *Stunner) StartServer(l *object.Listener) error { var pConns []turn.PacketConnConfig var lConns []turn.ListenerConfig + // listen on all IPs, relay to the listener address relay := &telemetry.RelayAddressGenerator{ Name: l.Name, RelayAddress: l.Addr, - Address: l.Addr.String(), + Address: "0.0.0.0", MinPort: uint16(l.MinPort), MaxPort: uint16(l.MaxPort), Net: l.Net, @@ -33,7 +34,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { permissionHandler := s.NewPermissionHandler(l) - addr := fmt.Sprintf("%s:%d", l.Addr.String(), l.Port) + addr := fmt.Sprintf("0.0.0.0:%d", l.Port) switch l.Proto { case v1alpha1.ListenerProtocolTURNUDP: @@ -115,7 +116,10 @@ func (s *Stunner) StartServer(l *object.Listener) error { } // for some reason dtls.Listen requires a UDPAddr and not an addr string - udpAddr := &net.UDPAddr{IP: l.Addr, Port: l.Port} + udpAddr, err := net.ResolveUDPAddr("udp", addr) + if err != nil { + return fmt.Errorf("failed to parse DTLS listener address %s: %s", addr, err) + } dtlsListener, err := dtls.Listen("udp", udpAddr, &dtls.Config{ Certificates: []tls.Certificate{cer}, // ExtendedMasterSecret: dtls.RequireExtendedMasterSecret, From 48e59a817f1394f5dc2cc46f19945704e2778892 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Sep 2023 15:15:52 +0200 Subject: [PATCH 013/243] doc: Document the StaticService CRD --- docs/GATEWAY.md | 42 ++++++++++++++++++++++++++++++++++++++++++ docs/SECURITY.md | 14 ++++---------- 2 files changed, 46 insertions(+), 10 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index e95c114d..c2779fa0 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -239,6 +239,48 @@ Below is a quick reference of the most important fields of the UDPRoute [`spec`] UDPRoute resources are safe for modification: `stunnerd` knows how to reconcile modified routes without restarting any listeners/TURN servers. +## StaticService + +When the target backend of a UDPRoute is running *inside* Kubernetes then the backend is always a proper Kubernetes Service. However, when the target is deployed *outside* Kubernetes then there is no Kubernetes Service that could be configured as a backend. This is particularly important when STUNner is used as a public TURN service. The StaticService resource provides a way to assign a routable IP address range to a UDPRoute for these cases. + +The below StaticService represents a hypothetical Kubernetes Service backing a set of pods with IP +addresses in the range `192.0.2.0/24` or `198.51.100.0/24`. + +```yaml +apiVersion: stunner.l7mp.io/v1alpha1 +kind: StaticService +metadata: + name: static-svc + namespace: stunner +spec: + prefixes: + - "192.0.2.0/24" + - "198.51.100.0/24" +``` + +Assigning this StaticService to a UDPRoute will make sure allows access to *any* IP address in the specified ranges. + +```yaml +apiVersion: gateway.networking.k8s.io/v1alpha2 +kind: UDPRoute +metadata: + name: media-plane-route + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - group: stunner.l7mp.io + kind: StaticService + name: static-svc +``` + +The StaticService `spec.prefixes` must be a list of proper IPv4 prefixes: any IP address in any of the listed prefixes will be whitelisted. Use the single prefix `0.0.0.0/0` to provide wildcard access via an UDPRoute. + +> **Warning** +Never use StaticServices to access Services running *inside* Kubernetes, this may open up an unintended backdoor to your cluster. Use StaticServices only with *external* target backends. + ## Status Most Kubernetes resources contain a `status` subresource that describes the current state of the resource, supplied and updated by the Kubernetes system and its components. The Kubernetes control plane continually and actively manages every object's actual state to match the desired state you supplied and updates the status field to indicate whether any error was encountered during the reconciliation process. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 806d696a..776f3e97 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -144,16 +144,10 @@ spec: - namespace: media-plane ``` -To avoid potential misuse, STUNner disables open wildcard access to the entire cluster. (Note that -in the [standalone mode](OBSOLETE.md) the user can still explicitly create an open `stunnerd` -cluster, but this is discouraged). - -For hardened deployments, it is possible to add a second level of isolation between STUNner and the -rest of the workload using the Kubernetes NetworkPolicy facility. Creating a NetworkPolicy will -essentially implement a firewall, blocking all access from the source to the target workload except -the services explicitly whitelisted by the user. The below example allows access from STUNner to -*any* media server pod labeled as `app=media-server` in the `default` namespace over the UDP port -range `[10000:20000]`, but nothing else. +> **Note** +To avoid potential misuse, STUNner disables open wildcard access to the entire cluster unless explicitly requested to do so by a [specifying an open StaticService as a backend](GATEWAY.md#staticservice). + +For hardened deployments, it is possible to add a second level of isolation between STUNner and the rest of the workload using the Kubernetes NetworkPolicy facility. Creating a NetworkPolicy will essentially implement a firewall, blocking all access from the source to the target workload except the services explicitly whitelisted by the user. The below example allows access from STUNner to *any* media server pod labeled as `app=media-server` in the `default` namespace over the UDP port range `[10000:20000]`, but nothing else. ```yaml apiVersion: networking.k8s.io/v1 From 06bd535aeb1d76fcc082ae742932ebee7e9e2375 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Sep 2023 15:16:28 +0200 Subject: [PATCH 014/243] doc: Update docs as per the upcoming v0.16.0 release --- README.md | 72 +++++++++---------- deploy/manifests/stunner-expose-kube-dns.yaml | 4 +- docs/AUTH.md | 46 ++++++------ docs/CONCEPTS.md | 2 +- docs/DEPLOYMENT.md | 36 +++++----- docs/GATEWAY.md | 65 ++++++++--------- docs/MONITORING.md | 33 ++++----- docs/SCALING.md | 2 +- docs/SECURITY.md | 11 ++- docs/WHY.md | 7 +- docs/examples/direct-one2one-call/README.md | 2 +- .../direct-one2one-call-stunner.yaml | 6 +- docs/examples/kurento-one2one-call/README.md | 2 +- .../kurento-one2one-call-stunner.yaml | 7 +- docs/examples/simple-tunnel/README.md | 8 +-- .../examples/simple-tunnel/iperf-stunner.yaml | 10 +-- 16 files changed, 151 insertions(+), 162 deletions(-) diff --git a/README.md b/README.md index 9585a46f..f34f28b6 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ Ever wondered how to [deploy your WebRTC infrastructure into the cloud](https://webrtchacks.com/webrtc-media-servers-in-the-cloud)? Frightened away by the complexities of Kubernetes container networking, and the surprising ways in which it may interact -with your UDP/RTP media? Tried to read through the endless stream of [Stack +with your UDP/RTP media? Read through the endless stream of [Stack Overflow](https://stackoverflow.com/search?q=kubernetes+webrtc) [questions](https://stackoverflow.com/questions/61140228/kubernetes-loadbalancer-open-a-wide-range-thousands-of-port) [asking](https://stackoverflow.com/questions/64232853/how-to-use-webrtc-with-rtcpeerconnection-on-kubernetes) @@ -88,15 +88,12 @@ raise security concerns, and come with a non-trivial price tag. The main goal of STUNner is to allow *anyone* to deploy their own WebRTC infrastructure into Kubernetes, without relying on any external service other than the cloud-provider's standard hosted -Kubernetes offering. This is achieved by STUNner acting as a gateway for ingesting WebRTC media -traffic into the Kubernetes cluster, exposing a public-facing STUN/TURN server that WebRTC clients -can connect to. - -STUNner can act as a STUN/TURN server that WebRTC clients and media servers can use as a scalable -NAT traversal facility (headless model), or it can serve as a fully-fledged ingress gateway for -clients to reach a media server deployed behind STUNner (media-plane model). This makes it possible -to deploy WebRTC application servers and media servers into ordinary Kubernetes pods, taking -advantage of Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC +Kubernetes offering. STUNner can act as a standalone STUN/TURN server that WebRTC clients and media +servers can use as a scalable NAT traversal facility (headless model), or it can act as a gateway +for ingesting WebRTC media traffic into the Kubernetes cluster by exposing a public-facing +STUN/TURN server that WebRTC clients can connect to (media-plane model). This makes it possible to +deploy WebRTC application servers and media servers into ordinary Kubernetes pods, taking advantage +of Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC infrastructure like any other cloud-bound workload. ![STUNner media-plane deployment architecture](./docs/img/stunner_arch.svg) @@ -130,7 +127,7 @@ way. [hacks](https://kubernetes.io/docs/concepts/configuration/overview), like privileged pods and `hostNetwork`/`hostPort` services, typically recommended as a prerequisite to containerizing your WebRTC media plane. Using STUNner a WebRTC deployment needs only two public-facing ports, one - HTTPS port for the application server and a *single* UDP port for *all* your media. + HTTPS port for signaling and a *single* UDP port for *all* your media. * **No reliance on external services for NAT traversal.** Can't afford a [hosted TURN service](https://bloggeek.me/webrtc-turn) for client-side NAT traversal? Can't get decent @@ -142,7 +139,7 @@ way. * **Easily scale your WebRTC infrastructure.** Tired of manually provisioning your WebRTC media servers? STUNner lets you deploy the entire WebRTC infrastructure into ordinary Kubernetes pods, thus [scaling the media plane](docs/SCALING.md) is as easy as issuing a `kubectl scale` - command. Even better, use the built in Kubernetes horizontal autoscaler to *automatically* resize + command. Or you can use the built in Kubernetes horizontal autoscaler to *automatically* resize your workload based on demand. * **Secure perimeter defense.** No need to open thousands of UDP/TCP ports on your media server for @@ -155,7 +152,7 @@ way. * **Simple code and extremely small size.** Written in pure Go using the battle-tested [pion/webrtc](https://github.com/pion/webrtc) framework, STUNner is just a couple of hundred lines of fully open-source code. The server is extremely lightweight: the typical STUNner - container image size is only about 5 Mbytes. + container image size is only 15 Mbytes. ## Getting Started @@ -196,16 +193,14 @@ Find out more about the charts in the [STUNner-helm repository](https://github.c ### Configuration The standard way to interact with STUNner is via the standard Kubernetes [Gateway - API](https://gateway-api.sigs.k8s.io) version - [v1alpha2](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec). This is much akin to the - way you configure *all* Kubernetes workloads: specify your intents in YAML files and issue a - `kubectl apply`, and the [STUNner gateway - operator](https://github.com/l7mp/stunner-gateway-operator) will automatically reconcile the - STUNner dataplane for the new configuration. + API](https://gateway-api.sigs.k8s.io). This is much akin to the way you configure *all* + Kubernetes workloads: specify your intents in YAML files and issue a `kubectl apply`, and the + [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) will automatically + reconcile the STUNner dataplane for the new configuration. 1. Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway API. This amounts to creating a - [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1alpha2.GatewayClass), + [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.GatewayClass), which serves as the [root level configuration](/docs/GATEWAY.md#gatewayclass) for your STUNner deployment. @@ -217,7 +212,7 @@ The standard way to interact with STUNner is via the standard Kubernetes [Gatewa ``` console kubectl apply -f - < Considering the above example: even if the `default/media-plane` Service was created for the TCP:80 port, STUNner will allow connections via any protocol-port pair, say, via UDP:10000 or any other UDP port for that matter. This hack remains our only viable way to support WebRTC workloads in Kubernetes until [support for port ranges is implemented in Kubernetes services](https://github.com/kubernetes/kubernetes/issues/23864). Note that this affects only the *internal* backend services: STUNner is still exposed *externally* via a *single* protocol-port, but it can demultiplex incoming client media connections to any *internal* backend ports via a single UDPRoute. + And that's all. You don't need to worry about client-side NAT traversal and WebRTC media routing because STUNner has you covered! Even better, every time you change a Gateway API resource in @@ -372,7 +369,7 @@ STUN/TURN authentication type: plaintext STUN/TURN username: user-1 STUN/TURN password: pass-1 Listener: udp-listener -Protocol: UDP +Protocol: TURN-UDP Public address: 34.118.36.108 Public port: 3478 ``` @@ -395,10 +392,9 @@ a heartwarming welcome message. 1. Fire up the UDP greeter service. The below manifest spawns the service in the `default` namespace and wraps it in a Kubernetes - service called `media-plane`. Recall, this is the target service STUNner will route connections - to. Note that the type of the `media-plane` service is `ClusterIP`, which means that Kubernetes - will *not* expose it to the Internet: the only way for clients to obtain a response is via - STUNner. + service called `media-plane`. Recall, this is the target service in our UDPRoute. Note that the + type of the `media-plane` service is `ClusterIP`, which means that Kubernetes will *not* expose + it to the Internet: the only way for clients to obtain a response is via STUNner. ```console kubectl apply -f deploy/manifests/udp-greeter.yaml @@ -454,11 +450,11 @@ greeter) by STUNner. ``` 1. Add the new TLS Gateway. Notice how the `tls-listener` now contains a `tls` object that refers - the above Secret, this way assigning the TLS certificate to use with our TLS listener. + the above Secret, this way assigning the TLS certificate to use with our TURN-TLS listener. ```console kubectl apply -f - < var pc = new RTCPeerConnection(iceConfig); ``` @@ -96,9 +92,10 @@ The intended authentication workflow in STUNner is as follows. ## Static authentication In STUNner, `static` authentication is the simplest and least secure authentication mode, basically -corresponding to a traditional "log-in" username and password pair given to users. STUNner accepts -(and sometimes reports) the alias `plaintext` to mean the `static` authentication mode; the use of -`plaintext` is deprecated and will be removed in a later release. +corresponding to a traditional "log-in" username and password pair given to users. + +> **Note** +STUNner accepts (and sometimes reports) the alias `plaintext` to mean the `static` authentication mode; the use of `plaintext` is deprecated and will be removed in a later release. When STUNner is configured to use `static` authentication only a single username/password pair is used for *all* clients. This makes configuration easy; e.g., the ICE server configuration can be @@ -109,7 +106,7 @@ credentials, see below). The first step of configuring STUNner for the `static` authentication mode is to create a Kubernetes Secret to hold the username/password pair. The below will set the username to `my-user` -and the password to `my-password`. Note that if no `type` is set then STUNner defaults to `static` +and the password to `my-password`. If no `type` is set then STUNner defaults to `static` authentication. ```console @@ -146,9 +143,8 @@ kubectl -n stunner edit secret stunner-auth-secret > **Warning** Modifying STUNner's credentials goes *without* restarting the TURN server but may affect existing -sessions, in that existing sessions will not be able to refresh the active TURN allocation with the -old credentials. The application server may also need to be restarted to learn the new TURN -credentials. +sessions, in that active sessions will not be able to refresh the TURN allocation established with +the old credentials. ## Ephemeral authentication @@ -158,10 +154,12 @@ with a pre-configured lifetime and, once the lifetime expires, the credential ca authenticate (or refresh) with STUNner any more. This authentication mode is more secure since credentials are not shared between clients and come with a limited lifetime. Configuring `ephemeral` authentication may be more complex though, since credentials must be dynamically -generated for each session and properly returned to clients. STUNner accepts (and sometimes -reports) the alias `longterm` to mean the `ephemeral` authentication mode; the use of `longterm` is -deprecated and will be removed in a later release. Note also that the alias `timewindowed` is also -accepted. +generated for each session and properly returned to clients. + +> **Note** +STUNner accepts (and sometimes reports) the alias `longterm` to mean the `ephemeral` authentication +mode; the use of `longterm` is deprecated and will be removed in a later release. The alias +`timewindowed` is also accepted. To implement this mode, STUNner adopts the [quasi-standard time-windowed TURN authentication credential format](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00). In this @@ -178,8 +176,8 @@ The advantage of this mechanism is that it is enough to know the shared secret f able to check the validity of a credential. > **Warning** -The user-id is used only for the integrity check but STUNner in no way checks whether it identifies -a valid user-id in the system. +The user-id is to ensure that the password generated per user-id is unique, but STUNner in no way +checks whether it identifies a valid user-id in the system. In order to switch from `static` mode to `ephemeral` authentication, it is enough to update the Secret that holds the credentials. The below will set the shared secret `my-shared-secret` for the diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md index f09412fa..7fb4f26f 100644 --- a/docs/CONCEPTS.md +++ b/docs/CONCEPTS.md @@ -4,7 +4,7 @@ In this guide we describe STUNner's architecture and the most important componen ## Architecture -A STUNner installation consists of two parts, a *control plane* and a *dataplane*. The control plane consists of declarative policies specifying the way STUNner should route WebRTC media traffic to the media servers, plus a gateway operator that renders the high-level policies into an actual dataplane configuration. The dataplane in turn comprises one or more `stunnerd` pods, responsible for actually ingesting media traffic into the cluster through a STUN/TURN server. Since the TURN service underlying STUNner is agnostic to NATs, STUNner can inject clients' media traffic into the private Kubernetes pod network, addressing all NAT traversal steps (client-side and server-side) in a single go. +A STUNner installation consists of two parts, a *control plane* and a *dataplane*. The control plane consists of declarative policies specifying the way STUNner should route WebRTC media traffic to the media servers, plus a gateway operator that renders the high-level policies into an actual dataplane configuration. The dataplane in turn comprises one or more `stunnerd` pods, responsible for actually ingesting media traffic into the cluster through a STUN/TURN server. ![STUNner architecture](img/stunner_arch_big.svg) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 66b9b808..3e0dd317 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,14 +1,14 @@ # Deployment models -STUNner can be deployed in many combinations to support a wide range of operational +STUNner can be deployed in many different ways, supporting a wide range of operational requirements. First, it supports multiple [architectural models](#architectural-models) where it can act either as a simple headless STUN/TURN server or a fully fledged ingress gateway in front of an entire Kubernetes-based media server pool. Second, when STUNner is configured as an ingress gateway then there are multiple [ICE models](#ice-models), based on whether only the client connects via STUNner or both clients and media servers use STUNner to set up the media-plane -connection. Third, STUNner can run in one of two [control plane models](#control-plane-models), +connection. Third, STUNner can run in one of several [control plane models](#control-plane-models), based on whether the user manually supplies STUNner configuration or there is a separate STUNner -control plane that automatically reconciles the dataplane state based on a high-level [declarative +control plane that automatically reconciles the dataplane based on a high-level [declarative API](https://gateway-api.sigs.k8s.io). ## Architectural models @@ -84,9 +84,8 @@ are as below: > - on the client, set STUNner as the *only* TURN server and configure *no* STUN servers, whereas > - on the server do *not* configure *any* STUN or TURN servers whatsoever. -Most users will want to deploy STUNner using the asymmetric ICE mode. In the rest of the docs, -unless noted otherwise we will assume the asymmetric ICE mode with the media plane deployment -model. +Most users will want to deploy STUNner using the asymmetric ICE mode. In the rest of the docs we +assume the asymmetric ICE mode with the media plane deployment model, unless noted otherwise. > **Warning** Deviating from the above rules *might* work in certain cases, but may have uncanny and @@ -130,13 +129,18 @@ you from adopting STUNner. ## Control plane models -STUNner can run in one of two modes: in the default mode STUNner configuration is controlled by a -*gateway-operator* component based on high-level intent encoded in [Kubernetes Gateway API -resources](https://gateway-api.sigs.k8s.io), while in the *standalone model* the user configures -STUNner manually. The standalone mode provides perfect control over the way STUNner ingests media, -but at the same time it requires users to deal with the subtleties of internal STUNner APIs that -are subject to change between subsequent releases. As of v0.14, STUNner's operator-ful mode is -feature complete and the standalone model is considered obsolete. If still interested, -comprehensive documentation for the standalone can be found [here](OBSOLETE.md), but this mode -is no longer supported. - +STUNner can run in one of several modes. + +In the default mode STUNner configuration is controlled by a *gateway-operator* component based on +high-level intent encoded in [Kubernetes Gateway API resources](https://gateway-api.sigs.k8s.io), +while in the *standalone model* the user configures STUNner manually. The standalone mode provides +perfect control over the way STUNner ingests media, but at the same time it requires users to deal +with the subtleties of internal STUNner APIs that are subject to change between subsequent +releases. As of v0.16, STUNner's operator-ful mode is feature complete and the standalone model is +considered obsolete. If still interested, comprehensive documentation for the standalone can be +found [here](OBSOLETE.md), but this mode is no longer supported. + +In addition, STUNner supports two dataplane provisioning modes. In the *legacy* mode the dataplane +is supposed to be deployed by the user manually (by installing the `stunner/stunner` Helm chart +into the target namespaces) while in the *managed* mode the dataplane pods are provisioned by the +gateway operator automatically. As of STUNner v0.16.0, the default is the *legacy* dataplane mode. diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index c2779fa0..16742336 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -8,7 +8,7 @@ The main unit of the control plane configuration is the *gateway hierarchy*. Her ![Gateway hierarchy](img/gateway_api.svg) -In general, the scope of a gateway hierarchy is a single namespace, but this is not strictly enforced: e.g., the GatewayClass is [cluster-scoped](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions) so it is outside the namespace, GatewayClasses can refer to GatewayConfigs across namespaces, Routes can attach to Gateways across a namespace boundary (if the Gateway allows this), etc. Still, it is a good practice to keep all control plane configuration, plus the actual dataplane pods, in a single namespace as much as possible. +In general, the scope of a gateway hierarchy is a single namespace, but this is not strictly enforced: e.g., the GatewayClass is [cluster-scoped](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions) so it is outside the namespace, GatewayClasses can refer to GatewayConfigs across namespaces, Routes can attach to Gateways across a namespace boundary (if the Gateway [allows](https://gateway-api.sigs.k8s.io/guides/multiple-ns) this), etc. Still, it is a good practice to keep all control plane configuration, plus the actual dataplane pods, in a single namespace as much as possible. ## GatewayClass @@ -17,7 +17,7 @@ The GatewayClass resource provides the root of the gateway hierarchy. GatewayCla Below is a sample GatewayClass resource. Each GatewayClass must specify a controller that will manage the Gateway objects created under the hierarchy; this must be set to `stunner.l7mp.io/gateway-operator` for the STUNner gateway operator to pick up the GatewayClass. In addition, a GatewayClass can refer to further implementation-specific configuration via a `parametersRef`; in the case of STUNner this will always be a GatewayConfig object (see [below](#gatewayconfig)). ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -31,7 +31,7 @@ spec: description: "STUNner is a WebRTC ingress gateway for Kubernetes" ``` -Below is a quick reference of the most important fields of the GatewayClass [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects) +Below is a quick reference of the most important fields of the GatewayClass [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects). | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | @@ -66,7 +66,7 @@ Below is a quick reference of the most important fields of the GatewayConfig [`s | :--- | :---: | :--- | :---: | | `stunnerConfig` | `string` | The name of the ConfigMap into which the operator renders the `stunnerd` running configuration. Default: `stunnerd-config`. | No | | `logLevel` | `string` | Logging level for the dataplane daemon pods (`stunnerd`). Default: `all:INFO`. | No | -| `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and `-`, and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | +| `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | | `authRef` | `reference` | Reference to a Secret (`namespace` and `name`) that defines the STUN/TURN authentication mechanism and the credentials. | No | | `authType` | `string` | Type of the STUN/TURN authentication mechanism. Valid only if `authRef` is not set. Default: `static`. | No | | `username` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | @@ -86,10 +86,10 @@ Except the TURN authentication realm, all GatewayConfig resources are safe for m Gateways describe the STUN/TURN server listeners exposed to clients. -The below Gateway will configure STUNner to open a STUN/TURN listener on the UDP port 3478 and automatically expose it on a public IP address and port by creating a [LoadBalancer service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). The name and namespace of the automatically provisioned service are the same as those of the Gateway, and the service is automatically updated if the Gateway changes (e.g., a port changes). +The below Gateway will configure STUNner to open a STUN/TURN listener over the UDP port 3478 and automatically expose it on a public IP address and port by creating a [LoadBalancer service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). The name and namespace of the automatically provisioned service are the same as those of the Gateway, and the service is automatically updated if the Gateway changes (e.g., a port changes). ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -99,13 +99,13 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP ``` -The below more complex example defines two TURN listeners: a UDP listener at port 3478 that accepts routes from any namespace, and a TLS/TCP listener at port 443 that accepts routes from all namespaces labeled as `app:dev`. +The below more complex example defines two TURN listeners: a TURN listener at the UDP:3478 port that accepts routes from any namespace, and a TURN listener at port TLS/TCP:443 that accepts routes from all namespaces labeled as `app:dev`. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: complex-gateway @@ -121,13 +121,13 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP allowedRoutes: namespaces: from: All - name: tls-listener port: 443 - protocol: TLS + protocol: TURN-TLS tls: mode: Terminate certificateRefs: @@ -151,37 +151,34 @@ Below is a quick reference of the most important fields of the Gateway [`spec`]( | `addresses` | `list` | The list of manually hinted external IP addresses for the rendered service (only the first one is used). | No | Each TURN `listener` is defined by a unique name, a transport protocol and a port. In addition, a -`tls` configuration is required for TLS and DTLS listeners. +`tls` configuration is required for TURN-TLS and TURN-DTLS listeners. | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | | `name` | `string` | Name of the TURN listener. | Yes | | `port` | `int` | Network port for the TURN listener. | Yes | -| `protocol` | `string` | Transport protocol for the TURN listener. Either UDP, TCP, TLS or DTLS. | Yes | -| `tls` | `object` | [TLS configuration](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig).| Yes (for TLS/DTLS) | +| `protocol` | `string` | Transport protocol for the TURN listener. Either TURN-UDP, TURN-TCP, TURN-TLS or TURN-DTLS. | Yes | +| `tls` | `object` | [TLS configuration](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig).| Yes (for TURN-TLS/TURN-DTLS) | | `allowedRoutes.from` | `object` | [Route attachment policy](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.AllowedRoutes), either `All`, `Selector`, or `Same` (default is `Same`) | No | -For TLS/DTLS listeners, `tls.mode` must be set to `Terminate` or omitted (`Passthrough` does not make sense for TURN), and `tls.certificateRefs` must be a [reference to a Kubernetes Secret](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig) of type `tls` or `opaque` with exactly two keys: `tls.crt` must hold the TLS PEM certificate and `tls.key` must hold the TLS PEM key. +For TURN-TLS/TURN-DTLS listeners, `tls.mode` must be set to `Terminate` or omitted (`Passthrough` does not make sense for TURN), and `tls.certificateRefs` must be a [reference to a Kubernetes Secret](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig) of type `tls` or `opaque` with exactly two keys: `tls.crt` must hold the TLS PEM certificate and `tls.key` must hold the TLS PEM key. -STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to -clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be -assigned a single externally reachable IP address. If you want multiple TURN listeners on different -public IPs, create multiple Gateways. TURN listeners on UDP and DTLS protocols are exposed as UDP -services, TCP and TLS listeners are exposed as TCP. +STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN over UDP and TURN over DTLS listeners are exposed as UDP services, TURN-TCP and TURN-TLS listeners are exposed as TCP. -Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway, will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. -> **Warning** -Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated, thus will be ignored if the cloud-provider or your Kubernetes install does not support the feature. Also the `externalIPs` field is denied by some cloud-providers and will fail the resource creation. Be thorough when using this feature. +Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. -| Field | Type | Description | Required | -| :--- | :---: | :--- | :---: | -| `type` | `string` | Type of the address. Currently we only support IPAddress. | Yes | -| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | +| Field | Type | Description | Required | +|:--------|:--------:|:--------------------------------------------------------------|:--------:| +| `type` | `string` | Type of the address. Currently only `IPAddress` is supported. | Yes | +| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | + +> **Warning** +Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if the cloud-provider or your Kubernetes install do not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. -Mixed multi-protocol Gateways are supported: this means if you want to expose a UDP and a TCP port on the same LoadBalancer service you can do it with a single Gateway. By default, the STUNner gateway-operator disables the use of mixed-protocol LBs for compatibility reasons. However, it can be enabled by annotating a Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. The below Gateway will expose both ports with their respective protocols. +Mixed multi-protocol Gateways are supported: this means if you want to expose a UDP and a TCP port on the same LoadBalancer service you can do it with a single Gateway. The below Gateway will expose both ports with their respective protocols. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: mixed-protocol-gateway @@ -192,20 +189,20 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP - name: tcp-listener port: 3479 - protocol: TCP + protocol: TURN-TCP ``` > **Warning** -> Note that the mixed-protocol LB feature might not be supported in your Kubernetes version. +> Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature for compatibility reasons. You can re-enable mixed-protocol LBs by annotating your Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. -STUNner implements two ways to customize the automatically created Service, both involving setting certain [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) to the Service. First, if any annotation is set in the GatewayConfig `loadBalancerServiceAnnotations` object then those will be copied verbatim into the Service. Note that `loadBalancerServiceAnnotations` affect *all* LoadBalancer Services created by STUNner. Second, Service annotations can be customized on a per-Gateway basis as well by adding the annotations to Gateway resources. STUNner then copies all annotations from the Gateway verbatim into the Service, overwriting the annotations specified in the GatewayConfig on conflict. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. Value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). +STUNner implements two ways to customize the automatically created Service, both involving adding certain [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) to the Service. First, if any annotation is set in the GatewayConfig `loadBalancerServiceAnnotations` field then those will be copied verbatim into the Service. Note that `loadBalancerServiceAnnotations` affect *all* LoadBalancer Services created by STUNner under the current Gateway hierarchy. Second, Service annotations can be customized on a per-Gateway basis as well by adding the annotations to Gateway resources. STUNner then copies all annotations from the Gateway verbatim into the Service, overwriting the annotations specified in the GatewayConfig on conflict. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). > **Warning** Gateway resources are *not* safe for modification. This means that certain changes to a Gateway will restart the underlying TURN server listener, causing all active client sessions to terminate. The particular rules are as follows: -> - adding or removing a listener will start/stop *only* the TURN server to be started/stopped, without affecting the rest of the listeners; +> - adding or removing a listener will start/stop *only* the TURN listener to be started/stopped, without affecting the rest of the listeners on the same Gateway; > - changing the transport protocol, port or TLS keys/certs of an *existing* listener will restart the TURN listener but leave the rest of the listeners intact; > - changing the TURN authentication realm will restart *all* TURN listeners. diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 73ff4740..9608d94d 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -57,33 +57,32 @@ STUNner provides deep visibility into the amount of traffic sent and received on | :--- | :--- | :--- | :--- | | `stunner_listener_connections` | Number of *active* downstream connections at a listener. | gauge | `name=` | | `stunner_listener_connections_total` | Number of downstream connections at a listener. | counter | `name=` | -| `stunner_listener_packets_total` | Number of datagrams sent or received at a listener. Unreliable for listeners running on a connection-oriented a protocol (TCP/TLS). | counter | `direction=`, `name=`| +| `stunner_listener_packets_total` | Number of datagrams sent or received at a listener. Unreliable for listeners running on a connection-oriented transport protocol (TCP/TLS). | counter | `direction=`, `name=`| | `stunner_listener_bytes_total` | Number of bytes sent or received at a listener. | counter | `direction=`, `name=` | | `stunner_cluster_connections` | Number of *active* upstream connections on behalf of a listener. | gauge | `name=` | | `stunner_cluster_connections_total` | Number of upstream connections on behalf of a listener. | counter | `name=` | -| `stunner_cluster_packets_total` | Number of datagrams sent to backends or received from backends on behalf of a listener. Unreliable for clusters running on a connection-oriented a protocol (TCP/TLS).| counter | `direction=`, `name=` | +| `stunner_cluster_packets_total` | Number of datagrams sent to backends or received from backends on behalf of a listener. Unreliable for clusters running on a connection-oriented transport protocol (TCP/TLS).| counter | `direction=`, `name=` | | `stunner_cluster_bytes_total` | Number of bytes sent to backends or received from backends on behalf of a listener. | counter | `direction=`, `name=` | ## Integration with Prometheus and Grafana -Collection and visualization of STUNner relies on Prometheus and Grafana services. The STUNer helm repository provides a ready-to-use Prometheus and Grafana stack. See [Installation](#installation) for installation steps. Metrics visualization requires user input on configuring the plots. Refer to [Configuration and Usage](#configuration-and-usage) for details. +Collection and visualization of STUNner relies on Prometheus and Grafana services. The STUNer helm repository provides a way to [install](#installation) a ready-to-use Prometheus and Grafana stack. In addition, metrics visualization requires [user input](#configuration-and-usage) on configuring the plots; see below. ### Installation A full-fledged Prometheus+Grafana helm chart is available in the STUNner helm repo. To use this chart, the installation steps involve enabling monitoring in STUNner, and installing the Prometheus+Grafana stack with helm. -1. **Configure STUNner to expose the metrics** +1. Install STUNner with Prometheus support: -- Deploy STUNner with monitoring enabled to enable the monitoring port of STUNner pods -```console -helm install stunner stunner/stunner --create-namespace --namespace=stunner --set stunner.deployment.monitoring.enabled=true -``` + ```console + helm install stunner stunner/stunner --create-namespace --namespace=stunner --set stunner.deployment.monitoring.enabled=true + ``` -- [Expose the STUNner metrics-collection server in the GatewayConfig](#configuration) +2. Configure STUNner to expose the metrics by [exposing the STUNner metrics-collection server in the GatewayConfig](#configuration). -2. **Install the Prometheus+Grafana stack with a helm chart** +3. Install the Prometheus+Grafana stack with a helm chart. -This helm chart creates a ready-to-use Prometheus+Grafana stack in the `monitoring` namespace: installs Prometheus along with the prometheus-operator, and Grafana; configures PodMonitor for monitoring STUNner pods, and sets up Prometheus as a datasource for Grafana. + The below creates a ready-to-use Prometheus+Grafana stack in the `monitoring` namespace: Prometheus, along with the prometheus-operator, is installed for metrics scarping, Grafana is set up for visualization, and the Prometheus is configured as a datasource for Grafana. ```console helm repo add stunner https://l7mp.io/stunner @@ -92,7 +91,6 @@ helm repo update helm install prometheus stunner/stunner-prometheus ``` - ### Configuration and Usage The helm chart deploys a ready-to-use Prometheus and Grafana stack, but leaves the Grafana dashboard empty to let the user pick metrics and configure their visualization. An interactive way to visualize STUNner metrics is to use the Grafana dashboard. @@ -115,11 +113,11 @@ Click on *Add panel* (1), then *Add a new panel* (2): The *Add a new panel* will open the panel configuration. The configuration steps are the following. -1. Set the datasource: **prometheus** +1. Set the datasource: **prometheus**. 2. Choose a metric. In this example, this is the `stunner_listener_connections`. -3. Click on *Run queries* (this will update the figure) +3. Click on *Run queries* (this will update the figure). 4. Fine-tune plot parameters. For example, set the title. -5. Click *Apply* +5. Click *Apply*. ![Grafana Panel Configuration](img/grafana-add-panel-config_0.png) @@ -129,7 +127,6 @@ Below is an example dashboard with data collected from the [simple-tunnel](examp ![Grafana Dashboard with the New Panel](img/grafana-add-panel-dashboard_1.png) - ### Troubleshooting Prometheus and Grafana both provide a dashboard to troubleshoot a running system, and to check the flow of metrics from STUNner to Prometheus, and from Prometheus to Grafana. @@ -141,8 +138,8 @@ The dashboard enables checking running Prometheus configuration and testing the For example, to observe the `stunner_listener_connections` metric on the Prometheus dashboard: -1. Write `stunner_listener_connections` to the marked field (next to the looking glass icon) -2. Click on the `Execute` button +1. Write `stunner_listener_connections` to the marked field (next to the looking glass icon). +2. Click on the `Execute` button. 3. Switch to `Graph` view tab. ![Prometheus Dashboard](img/prometheus-dashboard.png) diff --git a/docs/SCALING.md b/docs/SCALING.md index c591f42f..b615c501 100644 --- a/docs/SCALING.md +++ b/docs/SCALING.md @@ -65,7 +65,7 @@ connections. As usual, however, some caveats apply: even if active allocations would last longer. You can always set this by adjusting the `terminationGracePeriod` on your `stunnerd` pods. 2. STUNner pods may remain alive well after the last client connection goes away. This occurs when - an TURN/UDP allocation is left open by a client (spontaneous UDP client-side connection closure + an TURN-UDP allocation is left open by a client (spontaneous UDP client-side connection closure cannot be reliably detected by the server). As the default TURN refresh lifetime is [10 minutes](https://www.rfc-editor.org/rfc/rfc8656#section-3.2-3), it may take 10 minutes until all allocations time out, letting `stunnerd` to finally terminate. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 776f3e97..49ba8fb0 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -71,7 +71,7 @@ prerequisites for this: 2. the target service *must* be wrapped with a UDPRoute 3. the attacker *must* know at least one pod address or the ClusterIP for the targeted service. -Should any of these prerequisites miss, STUNner will block access to the target service. +Should any of these prerequisites fail, STUNner will block access to the target service. Now rewrite the backend service in the UDPRoute to an arbitrary non-existent service. @@ -90,7 +90,7 @@ spec: ``` Repeat the above `dig` command to query the Kubernetes DNS service again and observe how the query -times out. This demonstrates that a properly locked down STUNner installation blocks all accesses +times out. This demonstrates that a properly locked down STUNner installation blocks all access outside of the backend services explicitly opened up via a UDPRoute. ## Locking down STUNner @@ -102,7 +102,7 @@ services exposed via STUNner. STUNner's basic security model is as follows: > In a properly configured deployment, STUNner provides the same level of security as a media -server pool exposed to the Internet over public IP addresses, protected by a firewall that admits +server pool exposed to the Internet over a public IP address, protected by a firewall that admits only UDP access. A malicious attacker, even possessing a valid TURN credential, can reach only the media servers deployed behind STUNner, but no other services. @@ -196,7 +196,4 @@ though: remember, none of these private IP addresses can be reached externally. can be further reduced to the STUNner pods' private IP addresses by using the [symmetric ICE mode](DEPLOYMENT.md#symmetric-ice-mode). -Nevertheless, if worried about information exposure then STUNner may not be the best option at the -moment. In later releases, we plan to implement a feature to obscure the transport relay connection -addresses returned by STUNner, which would lock down external scanning attempts. Feel free to open -an issue if you think this limitation is a blocker for you. +Nevertheless, if worried about information exposure then STUNner may not be the best option at the moment. In later releases, we plan to implement a feature to obscure the relay transport addresses returned by STUNner. Please file an issue if you think this limitation is a blocker for your use case. diff --git a/docs/WHY.md b/docs/WHY.md index 348a560a..ce632d1b 100644 --- a/docs/WHY.md +++ b/docs/WHY.md @@ -5,8 +5,7 @@ STUNner is a *WebRTC media gateway for Kubernetes*. All words matter here: indee encapsulations, it is a *media gateway* so its job is to ingest WebRTC audio/video streams into a virtualized media plane, and it is *opinionated towards Kubernetes*, so everything around STUNner is designed and built to fit into the Kubernetes ecosystem. That being said, STUNner can easily be -used outside of this context (e.g., as a regular STUN/TURN server), but these deployment options -are not the main focus. +used outside of this context (e.g., as a regular STUN/TURN server), but this is not the main focus. ## The problem @@ -20,7 +19,7 @@ private IP address and the network dataplane applies several rounds of Network A (NAT) steps to ingest media traffic into this private pod network. Most cloud load-balancers apply a DNAT step to route packets to a node and then an SNAT step to put the packet to the private pod network, so that by the time a media packet reaches a pod essentially all header fields in the [IP -5-tuple](https://www.techopedia.com/definition/28190/5-tuple) are modified, except the destination +5-tuple](https://www.techopedia.com/definition/28190/5-tuple) are modified except the destination port. Then, if any pod sends the packet over to another pod via a Kubernetes service load-balancer then the packet will again undergo a DNAT step, and so on. @@ -68,7 +67,7 @@ There are *lots* of reasons why this deployment model is less than ideal: - **It is a security nightmare.** Given today's operational reality, exposing a fleet of media servers to the Internet over a public IP address, and opening up all UDP ports for potentially - malicious access, is an adventurous undertaking, to say the least. Wouldn't it be nice to hide + malicious access, is an adventurous undertaking to say the least. Wouldn't it be nice to hide your media servers behind a secure perimeter defense mechanism and lock down *all* uncontrolled access and nefarious business by running it over a private IP? diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index a9db3f1b..a0caa56b 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -220,7 +220,7 @@ The most important component in the STUNner configuration is the TURN Gateway: t public TURN server on the UDP port 3478 through which clients will connect to each other. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway diff --git a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml index c0df1a8e..ce7aa1cb 100644 --- a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml +++ b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -37,7 +37,7 @@ spec: protocol: UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: tcp-gateway diff --git a/docs/examples/kurento-one2one-call/README.md b/docs/examples/kurento-one2one-call/README.md index b90f6c49..05c67c6c 100644 --- a/docs/examples/kurento-one2one-call/README.md +++ b/docs/examples/kurento-one2one-call/README.md @@ -345,7 +345,7 @@ STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer s application server. ``` console -export WEBRTC_SERVER_IP=$(kubectl get service -n stunner webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +export WEBRTC_SERVER_IP=$(kubectl get service webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, diff --git a/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml b/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml index 91f4c8ef..9930fd99 100644 --- a/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml +++ b/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -23,7 +23,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -33,7 +33,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: gateway.networking.k8s.io/v1alpha2 @@ -48,4 +48,3 @@ spec: - backendRefs: - name: kms namespace: default - diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index 1f87a266..25337d15 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -65,7 +65,7 @@ that the UDPRoute specifies the `iperf-server` service as the `backendRef`, whic STUNner will forward the client connections received in any of the Gateways to the iperf server. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -75,10 +75,10 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: tcp-gateway @@ -88,7 +88,7 @@ spec: listeners: - name: tcp-listener port: 3478 - protocol: TCP + protocol: TURN-TCP --- apiVersion: gateway.networking.k8s.io/v1alpha2 diff --git a/docs/examples/simple-tunnel/iperf-stunner.yaml b/docs/examples/simple-tunnel/iperf-stunner.yaml index f1a295cf..d30ff989 100644 --- a/docs/examples/simple-tunnel/iperf-stunner.yaml +++ b/docs/examples/simple-tunnel/iperf-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -34,10 +34,10 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: tcp-gateway @@ -47,7 +47,7 @@ spec: listeners: - name: tcp-listener port: 3478 - protocol: TCP + protocol: TURN-TCP --- apiVersion: gateway.networking.k8s.io/v1alpha2 From f0f42126371f5eb6c5674df645ba54fda2630eec Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Sep 2023 15:31:44 +0200 Subject: [PATCH 015/243] chore: Update pion dependencies --- cmd/turncat/main.go | 2 +- config.go | 2 +- config_test.go | 2 +- go.mod | 47 ++++++-------- go.sum | 106 ++++++++++++++++--------------- handlers_test.go | 4 +- internal/object/listener.go | 4 +- internal/telemetry/relay.go | 4 +- internal/util/conn.go | 2 +- internal/util/socketpool.go | 2 +- internal/util/socketpool_unix.go | 4 +- pkg/authentication/auth.go | 4 +- reconcile_test.go | 2 +- server.go | 2 +- stunner.go | 4 +- stunner_test.go | 10 +-- turncat.go | 2 +- turncat_test.go | 4 +- 18 files changed, 104 insertions(+), 103 deletions(-) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index bfe83759..155057eb 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -11,7 +11,7 @@ import ( "time" "github.com/pion/logging" - "github.com/pion/turn/v2" + "github.com/pion/turn/v3" flag "github.com/spf13/pflag" corev1 "k8s.io/api/core/v1" diff --git a/config.go b/config.go index 33d48b02..b7076c40 100644 --- a/config.go +++ b/config.go @@ -6,7 +6,7 @@ import ( "fmt" "strings" - "github.com/pion/transport/v2" + "github.com/pion/transport/v3" "github.com/l7mp/stunner/internal/resolver" "github.com/l7mp/stunner/pkg/apis/v1alpha1" diff --git a/config_test.go b/config_test.go index d9fd7d79..0161ae9f 100644 --- a/config_test.go +++ b/config_test.go @@ -9,7 +9,7 @@ import ( "testing" "time" - "github.com/pion/transport/test" + "github.com/pion/transport/v3/test" "github.com/stretchr/testify/assert" "sigs.k8s.io/yaml" diff --git a/go.mod b/go.mod index 6c2a3a69..e349b737 100644 --- a/go.mod +++ b/go.mod @@ -4,29 +4,24 @@ go 1.19 require ( github.com/fsnotify/fsnotify v1.6.0 - github.com/pion/dtls/v2 v2.2.6 + github.com/google/uuid v1.3.1 + github.com/gorilla/websocket v1.5.0 + github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb + github.com/pion/dtls/v2 v2.2.7 github.com/pion/logging v0.2.2 - github.com/pion/transport/v2 v2.0.2 - // replace from l7mp/turn - github.com/pion/turn/v2 v2.1.0 - github.com/prometheus/client_golang v1.14.0 + github.com/pion/randutil v0.1.0 + github.com/pion/transport/v3 v3.0.1 + github.com/pion/turn/v3 v3.0.1 + github.com/prometheus/client_golang v1.16.0 github.com/spf13/pflag v1.0.5 - github.com/stretchr/testify v1.8.1 + github.com/stretchr/testify v1.8.4 + golang.org/x/sys v0.12.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 sigs.k8s.io/controller-runtime v0.12.3 sigs.k8s.io/yaml v1.3.0 ) -require ( - github.com/google/uuid v1.1.2 - github.com/gorilla/websocket v1.5.0 - github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb - github.com/pion/randutil v0.1.0 - github.com/pion/transport v0.14.1 - golang.org/x/sys v0.6.0 -) - require ( github.com/PuerkitoBio/purell v1.1.1 // indirect github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect @@ -51,21 +46,21 @@ require ( github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/pion/stun v0.4.0 // indirect - github.com/pion/udp/v2 v2.0.1 // indirect + github.com/pion/stun/v2 v2.0.0 // indirect + github.com/pion/transport/v2 v2.2.1 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect - github.com/prometheus/client_model v0.3.0 // indirect - github.com/prometheus/common v0.42.0 // indirect - github.com/prometheus/procfs v0.9.0 // indirect - golang.org/x/crypto v0.7.0 // indirect - golang.org/x/net v0.8.0 // indirect - golang.org/x/oauth2 v0.5.0 // indirect - golang.org/x/term v0.6.0 // indirect - golang.org/x/text v0.8.0 // indirect + github.com/prometheus/client_model v0.4.0 // indirect + github.com/prometheus/common v0.44.0 // indirect + github.com/prometheus/procfs v0.12.0 // indirect + golang.org/x/crypto v0.13.0 // indirect + golang.org/x/net v0.15.0 // indirect + golang.org/x/oauth2 v0.8.0 // indirect + golang.org/x/term v0.12.0 // indirect + golang.org/x/text v0.13.0 // indirect golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/protobuf v1.30.0 // indirect + google.golang.org/protobuf v1.31.0 // indirect gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/go.sum b/go.sum index 66b80915..4a0c650d 100644 --- a/go.sum +++ b/go.sum @@ -175,8 +175,9 @@ github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= -github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4= +github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= @@ -204,8 +205,8 @@ github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/X github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= -github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs= github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= @@ -241,37 +242,35 @@ github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7J github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= -github.com/pion/dtls/v2 v2.2.6 h1:yXMxKr0Skd+Ub6A8UqXTRLSywskx93ooMRHsQUtd+Z4= -github.com/pion/dtls/v2 v2.2.6/go.mod h1:t8fWJCIquY5rlQZwA2yWxUS1+OCrAdXrhVKXB5oD/wY= +github.com/pion/dtls/v2 v2.2.7 h1:cSUBsETxepsCSFSxC3mc/aDo14qQLMSL+O6IjG28yV8= +github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8= -github.com/pion/stun v0.4.0 h1:vgRrbBE2htWHy7l3Zsxckk7rkjnjOsSM7PHZnBwo8rk= -github.com/pion/stun v0.4.0/go.mod h1:QPsh1/SbXASntw3zkkrIk3ZJVKz4saBY2G7S10P3wCw= -github.com/pion/transport v0.14.1 h1:XSM6olwW+o8J4SCmOBb/BpwZypkHeyM0PGFCxNQBr40= -github.com/pion/transport v0.14.1/go.mod h1:4tGmbk00NeYA3rUa9+n+dzCCoKkcy3YlYb99Jn2fNnI= -github.com/pion/transport/v2 v2.0.0/go.mod h1:HS2MEBJTwD+1ZI2eSXSvHJx/HnzQqRy2/LXxt6eVMHc= -github.com/pion/transport/v2 v2.0.2 h1:St+8o+1PEzPT51O9bv+tH/KYYLMNR5Vwm5Z3Qkjsywg= -github.com/pion/transport/v2 v2.0.2/go.mod h1:vrz6bUbFr/cjdwbnxq8OdDDzHf7JJfGsIRkxfpZoTA0= -github.com/pion/turn/v2 v2.1.0 h1:5wGHSgGhJhP/RpabkUb/T9PdsAjkGLS6toYz5HNzoSI= -github.com/pion/turn/v2 v2.1.0/go.mod h1:yrT5XbXSGX1VFSF31A3c1kCNB5bBZgk/uu5LET162qs= -github.com/pion/udp/v2 v2.0.1 h1:xP0z6WNux1zWEjhC7onRA3EwwSliXqu1ElUZAQhUP54= -github.com/pion/udp/v2 v2.0.1/go.mod h1:B7uvTMP00lzWdyMr/1PVZXtV3wpPIxBRd4Wl6AksXn8= +github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= +github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= +github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= +github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= +github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= +github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= +github.com/pion/turn/v3 v3.0.1 h1:wLi7BTQr6/Q20R0vt/lHbjv6y4GChFtC33nkYbasoT8= +github.com/pion/turn/v3 v3.0.1/go.mod h1:MrJDKgqryDyWy1/4NT9TWfXWGMC7UHT6pJIv1+gMeNE= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.14.0 h1:nJdhIvne2eSX/XRAFV9PcvFFRbrjbcTUj0VP62TMhnw= -github.com/prometheus/client_golang v1.14.0/go.mod h1:8vpkKitgIVNcqrRBWh1C4TIUQgYNtG/XQE4E/Zae36Y= +github.com/prometheus/client_golang v1.16.0 h1:yk/hx9hDbrGHovbci4BY+pRMfSuuat626eFsHb7tmT8= +github.com/prometheus/client_golang v1.16.0/go.mod h1:Zsulrv/L9oM40tJ7T815tM89lFEugiJ9HzIqaAx4LKc= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/client_model v0.3.0 h1:UBgGFHqYdG/TPFD1B1ogZywDqEkwp3fBMvqdiQ7Xew4= -github.com/prometheus/client_model v0.3.0/go.mod h1:LDGWKZIo7rky3hgvBe+caln+Dr3dPggB5dvjtD7w9+w= -github.com/prometheus/common v0.42.0 h1:EKsfXEYo4JpWMHH5cg+KOUWeuJSov1Id8zGR8eeI1YM= -github.com/prometheus/common v0.42.0/go.mod h1:xBwqVerjNdUDjgODMpudtOMwlOwf2SaTr1yjz4b7Zbc= -github.com/prometheus/procfs v0.9.0 h1:wzCHvIvM5SxWqYvwgVL7yJY8Lz3PKn49KQtpgMYJfhI= -github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY= +github.com/prometheus/client_model v0.4.0 h1:5lQXD3cAg1OXBf4Wq03gTrXHeaV0TQvGfUooCfx1yqY= +github.com/prometheus/client_model v0.4.0/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU= +github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY= +github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY= +github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= +github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= @@ -286,8 +285,9 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= -github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -312,9 +312,10 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.5.0/go.mod h1:NK/OQwhpMQP3MwtdjgLlYHnH9ebylxKWv3e0fK+mkQU= -golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= -golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= +golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= +golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= +golang.org/x/crypto v0.13.0 h1:mvySKfSWJ+UKUii46M40LOvyWfN0s2U+46/jDd0e6Ck= +golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -350,6 +351,7 @@ golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -390,11 +392,12 @@ golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96b golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= -golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= -golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws= -golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= -golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= +golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= +golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= +golang.org/x/net v0.15.0 h1:ugBLEUaxABaB5AJqW9enI0ACdci2RUd4eP51NTBvuJ8= +golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -407,8 +410,8 @@ golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.5.0 h1:HuArIo48skDwlrvM3sEdHXElYslAMsf3KwRkkW4MC4s= -golang.org/x/oauth2 v0.5.0/go.mod h1:9/XBHVqLaWO3/BRHs5jbpYCnOZVjj5V0ndyaAM7KB4I= +golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8= +golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -421,6 +424,7 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -470,19 +474,20 @@ golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= -golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.12.0 h1:CM0HF96J0hcLAwsHPJZjfdNzs0gftsLfgKt57wWHJ0o= +golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= -golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= -golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= +golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= +golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= +golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= +golang.org/x/term v0.12.0 h1:/ZfYdc3zq+q02Rv9vGqTeSItdzZTSNDmfTi0mBAuidU= +golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -492,11 +497,11 @@ golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68= -golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= +golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -553,6 +558,7 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -658,8 +664,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng= -google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 h1:FVCohIoYO7IJoDDVpV2pdq7SgrMH6wHnuTyrdrxJNoY= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0/go.mod h1:OdE7CF6DbADk7lN8LIKRzRJTTZXIjtWgA5THM5lhBAw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/handlers_test.go b/handlers_test.go index 7ba4762b..0e2edc42 100644 --- a/handlers_test.go +++ b/handlers_test.go @@ -10,8 +10,8 @@ import ( "testing" "time" - "github.com/pion/transport/test" - "github.com/pion/turn/v2" + "github.com/pion/transport/v3/test" + "github.com/pion/turn/v3" "github.com/stretchr/testify/assert" "github.com/l7mp/stunner/pkg/apis/v1alpha1" diff --git a/internal/object/listener.go b/internal/object/listener.go index 7ad639ac..f46183ee 100644 --- a/internal/object/listener.go +++ b/internal/object/listener.go @@ -9,8 +9,8 @@ import ( "strings" "github.com/pion/logging" - "github.com/pion/transport/v2" - "github.com/pion/turn/v2" + "github.com/pion/transport/v3" + "github.com/pion/turn/v3" "github.com/l7mp/stunner/internal/util" "github.com/l7mp/stunner/pkg/apis/v1alpha1" diff --git a/internal/telemetry/relay.go b/internal/telemetry/relay.go index eb7c0a76..a18c4e50 100644 --- a/internal/telemetry/relay.go +++ b/internal/telemetry/relay.go @@ -8,8 +8,8 @@ import ( "net" "github.com/pion/randutil" - "github.com/pion/transport/v2" - "github.com/pion/transport/v2/stdnet" + "github.com/pion/transport/v3" + "github.com/pion/transport/v3/stdnet" ) var ( diff --git a/internal/util/conn.go b/internal/util/conn.go index 1479b630..3e3e50dd 100644 --- a/internal/util/conn.go +++ b/internal/util/conn.go @@ -6,7 +6,7 @@ import ( "os" "time" - "github.com/pion/transport/v2" + "github.com/pion/transport/v3" ) type FileConnAddr struct { diff --git a/internal/util/socketpool.go b/internal/util/socketpool.go index 08fd05f1..b2b4e5e5 100644 --- a/internal/util/socketpool.go +++ b/internal/util/socketpool.go @@ -3,7 +3,7 @@ package util import ( - "github.com/pion/transport/v2" + "github.com/pion/transport/v3" ) // NewPacketConnPool creates a new packet connection pool which is fixed to a single connection, diff --git a/internal/util/socketpool_unix.go b/internal/util/socketpool_unix.go index c0710b37..d5c61765 100644 --- a/internal/util/socketpool_unix.go +++ b/internal/util/socketpool_unix.go @@ -10,8 +10,8 @@ import ( "golang.org/x/sys/unix" - "github.com/pion/transport/v2" - "github.com/pion/transport/v2/stdnet" + "github.com/pion/transport/v3" + "github.com/pion/transport/v3/stdnet" ) // unixPacketConPool implements socketpools for unix with full support for SO_REUSEPORT diff --git a/pkg/authentication/auth.go b/pkg/authentication/auth.go index 55768263..8441d70b 100644 --- a/pkg/authentication/auth.go +++ b/pkg/authentication/auth.go @@ -10,7 +10,7 @@ import ( "strings" "time" - "github.com/pion/turn/v2" + "github.com/pion/turn/v3" ) // UsernameSeparator is the separator character used in time-windowed TURN authentication as @@ -72,7 +72,7 @@ func GetLongTermCredential(username string, sharedSecret string) (string, error) } // GenerateAuthKey is a convenience function to easily generate keys in the format used by -// AuthHandler. Re-exported from `pion/turn/v2` so that our callers will have a single import. +// AuthHandler. Re-exported from `pion/turn` so that our callers will have a single import. func GenerateAuthKey(username, realm, password string) []byte { return turn.GenerateAuthKey(username, realm, password) } diff --git a/reconcile_test.go b/reconcile_test.go index d0d8d2f8..b7a57b1c 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -8,7 +8,7 @@ import ( "testing" "time" - "github.com/pion/transport/test" + "github.com/pion/transport/v3/test" "github.com/stretchr/testify/assert" "github.com/l7mp/stunner/internal/object" diff --git a/server.go b/server.go index f00fa030..15c60a98 100644 --- a/server.go +++ b/server.go @@ -6,7 +6,7 @@ import ( "net" "github.com/pion/dtls/v2" - "github.com/pion/turn/v2" + "github.com/pion/turn/v3" "github.com/l7mp/stunner/internal/object" "github.com/l7mp/stunner/internal/telemetry" diff --git a/stunner.go b/stunner.go index 883f7a9d..681ae439 100644 --- a/stunner.go +++ b/stunner.go @@ -8,8 +8,8 @@ import ( "github.com/google/uuid" "github.com/pion/logging" - "github.com/pion/transport/v2" - "github.com/pion/transport/v2/stdnet" + "github.com/pion/transport/v3" + "github.com/pion/transport/v3/stdnet" "github.com/l7mp/stunner/internal/manager" "github.com/l7mp/stunner/internal/object" diff --git a/stunner_test.go b/stunner_test.go index 984fd61f..a81147a8 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -13,11 +13,11 @@ import ( "github.com/pion/dtls/v2" "github.com/pion/logging" - "github.com/pion/transport/v2" - "github.com/pion/transport/v2/stdnet" - "github.com/pion/transport/v2/test" - "github.com/pion/transport/v2/vnet" - "github.com/pion/turn/v2" + "github.com/pion/transport/v3" + "github.com/pion/transport/v3/stdnet" + "github.com/pion/transport/v3/test" + "github.com/pion/transport/v3/vnet" + "github.com/pion/turn/v3" "github.com/stretchr/testify/assert" "github.com/l7mp/stunner/internal/resolver" diff --git a/turncat.go b/turncat.go index 94e460ef..db998b07 100644 --- a/turncat.go +++ b/turncat.go @@ -12,7 +12,7 @@ import ( "github.com/pion/dtls/v2" "github.com/pion/logging" - "github.com/pion/turn/v2" + "github.com/pion/turn/v3" "github.com/l7mp/stunner/internal/util" "github.com/l7mp/stunner/pkg/apis/v1alpha1" diff --git a/turncat_test.go b/turncat_test.go index 0fb5e938..7f2d81bd 100644 --- a/turncat_test.go +++ b/turncat_test.go @@ -9,8 +9,8 @@ import ( "time" "github.com/pion/logging" - "github.com/pion/transport/test" - "github.com/pion/turn/v2" + "github.com/pion/transport/v3/test" + "github.com/pion/turn/v3" "github.com/stretchr/testify/assert" "github.com/l7mp/stunner/pkg/logger" From a9698fbf142c9ae81fbd02556073d4df62a3ff01 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Sep 2023 21:29:05 +0200 Subject: [PATCH 016/243] fix: Fix handling /dev/stdin client connections in turncat --- turncat.go | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/turncat.go b/turncat.go index db998b07..d65d8200 100644 --- a/turncat.go +++ b/turncat.go @@ -87,6 +87,10 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { } log.Tracef("Resolving listener address: %s", config.ListenerAddr) + // special case the "-" client address + if config.ListenerAddr == "-" { + config.ListenerAddr = "file://stdin" + } listener, lErr := url.Parse(config.ListenerAddr) if lErr != nil { return nil, fmt.Errorf("error parsing listener address %q: %w", config.ListenerAddr, lErr) @@ -118,7 +122,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { switch listenerProtocol { case "file": listenerConn = util.NewFileConn(os.Stdin) - case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6", "turn-udp", "turn-dtls": + case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6": addr, err := net.ResolveUDPAddr("udp", listener.Host) if err != nil { return nil, fmt.Errorf("error resolving listener address %q: %w", config.ListenerAddr, err) @@ -131,7 +135,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { } listenerAddress = addr listenerConn = l - case "tcp", "tcp4", "tcp6", "unix", "unixpacket", "turn-tcp", "turn-tls": + case "tcp", "tcp4", "tcp6", "unix", "unixpacket": addr, err := net.ResolveTCPAddr("tcp", listener.Host) if err != nil { return nil, fmt.Errorf("error resolving listener address %q: %w", config.ListenerAddr, err) @@ -163,7 +167,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { log: log, } - switch listenerAddress.Network() { + switch listenerProtocol { case "udp", "udp4", "udp6", "unixgram", "ip", "ip4", "ip6": // client connection is a packet conn, write our own Listen/Accept loop for UDP // main loop: for every new packet we create a new connection and connect it back to the client @@ -179,9 +183,8 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { listenerAddress.Network(), listenerAddress.Network(), listenerAddress.String()) } - log.Infof("Turncat client listening on %s:%s, TURN server: %s, peer: %s:%s", - listenerAddress.Network(), listenerAddress.String(), - config.ServerAddr, + log.Infof("Turncat client listening on %s, TURN server: %s, peer: %s:%s", + config.ListenerAddr, config.ServerAddr, peerAddress.Network(), peerAddress.String()) return t, nil From 1291326c873e21dc4f14bf7ee3d7b7df0c191bcc Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Sep 2023 21:34:14 +0200 Subject: [PATCH 017/243] chore: Report active connections on graceful shutdown --- cmd/stunnerd/main.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index f594418d..792d204c 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -131,7 +131,8 @@ func main() { os.Exit(0) case <-sigterm: - log.Info("performing a graceful shutdown") + log.Infof("performing a graceful shutdown with %d active connections", + st.AllocationCount()) st.Shutdown() if cancelConfigLoader != nil { From 626faf93d2ad4c8d7febe473f95b939dd48c901f Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Sep 2023 22:04:57 +0200 Subject: [PATCH 018/243] doc: Update the direct-1to1 call tutorial as per v0.16.0 --- docs/examples/direct-one2one-call/README.md | 8 ++++---- .../direct-one2one-call/direct-one2one-call-stunner.yaml | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index a0caa56b..28d27a48 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -230,7 +230,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP ``` For later convenience we also create a TCP Gateway that runs on port TCP:3478. @@ -292,13 +292,13 @@ STUN/TURN password: pass-1 Listener 1 Name: udp-listener Listener: udp-listener - Protocol: UDP + Protocol: TURN-UDP Public address: 34.118.82.225 Public port: 3478 Listener 2 Name: tcp-listener Listener: tcp-listener - Protocol: TCP + Protocol: TURN-TCP Public address: 34.118.89.139 Public port: 3478 ``` @@ -310,7 +310,7 @@ STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer s application server. ``` console -export WEBRTC_SERVER_IP=$(kubectl get service -n stunner webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +export WEBRTC_SERVER_IP=$(kubectl get service webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, diff --git a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml index ce7aa1cb..f77bb462 100644 --- a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml +++ b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml @@ -34,7 +34,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: gateway.networking.k8s.io/v1beta1 @@ -47,7 +47,7 @@ spec: listeners: - name: tcp-listener port: 3478 - protocol: TCP + protocol: TURN-TCP --- apiVersion: gateway.networking.k8s.io/v1alpha2 From 5a720413e2790db5157b80cdf3669c2178477422 Mon Sep 17 00:00:00 2001 From: Kornel David Date: Tue, 26 Sep 2023 11:12:01 +0200 Subject: [PATCH 019/243] doc: Update LiveKit tutorial as per v0.16.0 --- docs/examples/livekit/README.md | 2 +- docs/examples/livekit/livekit-call-stunner.yaml | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/examples/livekit/README.md b/docs/examples/livekit/README.md index 1b9691f0..7f47cc9b 100644 --- a/docs/examples/livekit/README.md +++ b/docs/examples/livekit/README.md @@ -169,7 +169,7 @@ This will make sure that LiveKit is started with STUNner as the STUN/TURN server ``` console cd stunner -cmd/stunnerctl/stunnerctl running-config default/stunnerd-config +cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config STUN/TURN authentication type: plaintext STUN/TURN username: user-1 STUN/TURN password: pass-1 diff --git a/docs/examples/livekit/livekit-call-stunner.yaml b/docs/examples/livekit/livekit-call-stunner.yaml index fd4e703a..2fe6a7ce 100644 --- a/docs/examples/livekit/livekit-call-stunner.yaml +++ b/docs/examples/livekit/livekit-call-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -34,7 +34,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: gateway.networking.k8s.io/v1alpha2 kind: UDPRoute From ce226bc262892a3923f7a3163be25e8259519510 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 26 Sep 2023 13:19:24 +0200 Subject: [PATCH 020/243] doc: Initial installation docs for the managed mode --- docs/INSTALL.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index b9cc4077..e61a71d7 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -73,3 +73,41 @@ it at your own risk: we do not promise any stability for STUNner installed from helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner ``` + +## Managed mode + +From v0.16.0 STUNner provides a new way to provision dataplane pods that is called the *managed mode*. In the traditional operational model (called the *legacy mode*), the user was responsible for provisioning both the control plane, by installing the `stunner-gateway-operator` Helm chart, and the dataplane(s), by helm-installing the `stunner` chart [possibly multiple times](#parallel-deployments). In the managed mode the operator *automatically* provisions the necessary dataplanes by creating a separate `stunnerd` Deployment per each Gateway, plus the usual LoadBalancer service to expose it. This substantially simplifies operations and removes lot of manual and repetitive work. + +To install the gateway operator using the new manged mode, start with a clean Kubernetes cluster and install the `stunner-gateway-operator` Helm chart, setting the flag `stunnerGatewayOperator.dataplane.mode` to `managed`. Observe that we do not install the `stunner` Helm chart separately; the operator will readily create the `stunnerd` pods as needed. + +```console +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace \ + --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=managed +``` + +The `stunnerd` pods created by the operator can be customized using the Dataplane CR: for instance you can specify the `stunnerd` container image version to be used as the dataplane, provision resources for each `stunenrd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1alpha1#DataplaneSpec). All gateways will use the `default` Dataplane; you can override this by creating a new Dataplane CR and setting the name in the [`spec.dataplane` field](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator@v0.15.2/api/v1alpha1#GatewayConfigSpec) of the corresponding GatewayConfig. + +```console +kubectl get dataplanes.stunner.l7mp.io default -o yaml +apiVersion: stunner.l7mp.io/v1alpha1 +kind: Dataplane +metadata: + name: default +spec: + args: + - -w + - --udp-thread-num=16 + command: + - stunnerd + hostNetwork: false + image: l7mp/stunnerd:latest + imagePullPolicy: Always + resources: + limits: + cpu: 2 + memory: 512Mi + requests: + cpu: 500m + memory: 128Mi + terminationGracePeriodSeconds: 3600 +``` From f5c7b52166f3e10c1a0523c2cb0c99a8a329a691 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 27 Sep 2023 08:33:48 +0200 Subject: [PATCH 021/243] docs: Advertise RTD in the main README --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f34f28b6..e1df936d 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ modification to your existing WebRTC codebase. STUNner implements the standard API](https://gateway-api.sigs.k8s.io) so you can configure it in the familiar YAML-engineering style via Kubernetes manifests. -See the full documentation [here](https://docs.l7mp.io/en/latest). +See the full documentation [here](https://docs.l7mp.io). ## Table of Contents 1. [Description](#description) @@ -594,7 +594,8 @@ applications into Kubernetes. ## Documentation -See the full documentation [here](/docs/README.md). +The documentation of the stable release can be found [here](https://docs.l7mp.io). The +documentation for the latest development release can be found [here](/docs/README.md). ## Caveats From fc86ff4a4f2909c0894f3b04c122553bea282f1d Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 27 Sep 2023 21:34:57 +0200 Subject: [PATCH 022/243] doc: Update Neko tutorial as per v0.16.0 Changes: - use full image name for neko - update stunner config as per v0.16.0 - do not specify iceTransportPolicy to make it work with Firefox - mention managed mode steps in the README - improve README formatting Tested on k8s v1.27 (minikube & GKE autopilot). --- docs/examples/neko/README.md | 21 +++++++++++---------- docs/examples/neko/neko.yaml | 4 ++-- docs/examples/neko/stunner.yaml | 10 ++++++---- 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/examples/neko/README.md b/docs/examples/neko/README.md index cc81c51d..4ad1fcd6 100644 --- a/docs/examples/neko/README.md +++ b/docs/examples/neko/README.md @@ -9,7 +9,7 @@ However, integrating Neko into Kubernetes is far from trivial. In this demo you will learn the following steps to: - integrate a typical WebRTC application server to be used with STUNner, -- deploy the Neko into Kubernetes behind STUNner, +- deploy Neko into Kubernetes behind STUNner, ## Installation @@ -19,9 +19,7 @@ Consult the [STUNner installation and configuration guide](../../INSTALL.md) to ### Quick installation -The simplest way to deploy the demo is to clone the [STUNner git -repository](https://github.com/l7mp/stunner) and deploy the -[manifest](neko.yaml) packaged with STUNner. +The simplest way to deploy the demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy the [manifest](neko.yaml) packaged with STUNner. Install the STUNner gateway operator and STUNner ([more info](https://github.com/l7mp/stunner-helm)): @@ -34,31 +32,34 @@ helm install stunner stunner/stunner Configure STUNner to act as a STUN server towards clients, and to let media reach the media server. -``` +```console git clone https://github.com/l7mp/stunner cd stunner/docs/examples/neko kubectl apply -f stunner.yaml ``` +> **Warning** +> In case of [managed mode](../../INSTALL.md#managed-mode), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. + This will expose STUNner on a public IP on UDP port 3478. A Kubernetes `LoadBalancer` assigns an ephemeral public IP address to the service, so first we need to learn the external IP. -``` +```console kubectl get service udp-gateway -n default -o jsonpath='{.status.loadBalancer.ingress[0].ip}' STUNNERIP=$(kubectl get service udp-gateway -n default -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` -NOTE: this IP should be accessible from your browser. If that "public IP" is behind a NAT, you can overwrite it with the actual -public IP that routes to the service by hand (e.g. `STUNNERIP=`). +> **Note** +> This IP should be accessible from your browser. If that "public IP" is behind a NAT, you can overwrite it with the actual public IP that routes to the service by hand (e.g. `STUNNERIP=`). We need to give this public IP the Neko configuration in the `NEKO_ICESERVERS` environment variable, inside the `json` content (basically this will tell you browser to use STUNner as a STUN/TURN server). You can do that by hand, or by this fancy `sed` command: -``` +```console sed -i "s/1.1.1.1/$STUNNERIP/g" neko.yaml ``` Now apply the Neko manifests: -``` +```console kubectl apply -f neko.yaml kubectl get pods ``` diff --git a/docs/examples/neko/neko.yaml b/docs/examples/neko/neko.yaml index 53b69600..7292dbcd 100644 --- a/docs/examples/neko/neko.yaml +++ b/docs/examples/neko/neko.yaml @@ -16,7 +16,7 @@ spec: spec: containers: - name: neko - image: m1k1o/neko:firefox + image: docker.io/m1k1o/neko:firefox env: - name: NEKO_SCREEN value: 1280x720@30 @@ -35,7 +35,7 @@ spec: fieldRef: fieldPath: status.podIP - name: NEKO_ICESERVERS - value: '[{"urls": ["turn:1.1.1.1:3478?transport=udp"], "username": "user-1", "credential": "pass-1","iceTransportPolicy": "relay"}]' + value: '[{"urls": ["turn:1.1.1.1:3478?transport=udp"], "username": "user-1", "credential": "pass-1","iceTransportPolicy": "all"}]' ports: - name: http containerPort: 8080 diff --git a/docs/examples/neko/stunner.yaml b/docs/examples/neko/stunner.yaml index 776d6372..98d57d13 100644 --- a/docs/examples/neko/stunner.yaml +++ b/docs/examples/neko/stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -34,7 +34,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: gateway.networking.k8s.io/v1alpha2 @@ -48,4 +48,6 @@ spec: rules: - backendRefs: - name: neko-udp - namespace: default \ No newline at end of file + namespace: default + - name: stunner + namespace: default From 22c01f4c3392c2de8f3d653ce1e110446a788482 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Wed, 27 Sep 2023 21:42:10 +0200 Subject: [PATCH 023/243] doc(examples/neko): Use abs. path of INSTALL.md --- docs/examples/neko/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/examples/neko/README.md b/docs/examples/neko/README.md index 4ad1fcd6..fd776da7 100644 --- a/docs/examples/neko/README.md +++ b/docs/examples/neko/README.md @@ -15,7 +15,7 @@ In this demo you will learn the following steps to: ### Prerequisites -Consult the [STUNner installation and configuration guide](../../INSTALL.md) to set up STUNner. +Consult the [STUNner installation and configuration guide](/docs/INSTALL.md) to set up STUNner. ### Quick installation @@ -39,7 +39,7 @@ kubectl apply -f stunner.yaml ``` > **Warning** -> In case of [managed mode](../../INSTALL.md#managed-mode), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. +> In case of [managed mode](/docs/INSTALL.md#managed-mode), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. This will expose STUNner on a public IP on UDP port 3478. A Kubernetes `LoadBalancer` assigns an ephemeral public IP address to the service, so first we need to learn the external IP. From 07136f3b9f5b022476d986e9bb8a68e8e19ef598 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Wed, 27 Sep 2023 21:43:07 +0200 Subject: [PATCH 024/243] doc(examples/neko): Fix broken section link --- docs/examples/neko/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/examples/neko/README.md b/docs/examples/neko/README.md index fd776da7..56628c27 100644 --- a/docs/examples/neko/README.md +++ b/docs/examples/neko/README.md @@ -39,7 +39,7 @@ kubectl apply -f stunner.yaml ``` > **Warning** -> In case of [managed mode](/docs/INSTALL.md#managed-mode), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. +> In case of [managed mode](/docs/INSTALL.md), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. This will expose STUNner on a public IP on UDP port 3478. A Kubernetes `LoadBalancer` assigns an ephemeral public IP address to the service, so first we need to learn the external IP. From aa9fc40e2ad9671afaba2d1154ede5a8b1ac5883 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 27 Sep 2023 21:52:30 +0200 Subject: [PATCH 025/243] chore(CI/CD): Add e2e test for managed mode too --- ...e-test-dev.yml => e2e-test-dev-legacy.yml} | 2 +- .github/workflows/e2e-test-dev-managed.yaml | 90 +++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) rename .github/workflows/{e2e-test-dev.yml => e2e-test-dev-legacy.yml} (98%) create mode 100644 .github/workflows/e2e-test-dev-managed.yaml diff --git a/.github/workflows/e2e-test-dev.yml b/.github/workflows/e2e-test-dev-legacy.yml similarity index 98% rename from .github/workflows/e2e-test-dev.yml rename to .github/workflows/e2e-test-dev-legacy.yml index 045a4556..4ec99333 100644 --- a/.github/workflows/e2e-test-dev.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -1,4 +1,4 @@ -name: Run End-to-End Test (dev) +name: Run End-to-End Test [legacy] (dev) on: workflow_dispatch: diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml new file mode 100644 index 00000000..225c483b --- /dev/null +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -0,0 +1,90 @@ +name: Run End-to-End Test [managed] (dev) + +on: + workflow_dispatch: + schedule: + - cron: '0 11 * * 1' + +jobs: + e2e_test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Set up Go + uses: actions/setup-go@v4 + with: + go-version: '1.20' + + - name: Download modules + run: go mod download + - name: Go install + run: go install + - name: Build turncat + run: CGO_ENABLED=0 go build -ldflags="-w -s" -o turncat cmd/turncat/main.go + + - name: Start minikube + uses: medyagh/setup-minikube@master + with: + driver: docker + container-runtime: containerd + wait: all + cache: false + + - name: Start minikube tunnel + run: minikube tunnel &>mktunnel.log & + + - name: Set up Helm + uses: azure/setup-helm@v3 + with: + version: v3.11.3 + + - name: Install STUNner + run: | + helm repo add stunner https://l7mp.io/stunner + helm repo update + helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed + + - name: Deploy iperf server + run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml + + - name: Configure STUNner + run: | + kubectl apply -f docs/examples/simple-tunnel/iperf-stunner.yaml + sleep 75 + + - name: Install iperf client + run: | + sudo apt-get update + sudo apt-get -y install iperf + + - name: Wait for LoadBalancer IP + run: | + while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done + kubectl get all -A + + - name: Start turncat + run: | + ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & + sleep 1 + + - name: Run iperf client + run: | + iperf -c 127.0.0.1 -p 5000 -u -l 100 -b 5M -t 5 | tee iperf.log + + - name: Show logs + run: | + echo "* IPERF" + echo "** Client" + cat iperf.log + echo "** Server" + kubectl logs $(kubectl get pods -l app=iperf-server -o jsonpath='{.items[0].metadata.name}') + echo "* MINIKUBE TUNNEL" + cat mktunnel.log + echo "* TURNCAT" + cat turncat.log + echo "* STUNNER" + kubectl logs -n stunner $(kubectl get pods -n stunner -l app=stunner -o jsonpath='{.items[0].metadata.name}') + + - name: Check iperf conectivity + run: grep "Server Report" iperf.log From 1f779fcc45a653967d611654fa681259c1c35d94 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 27 Sep 2023 21:54:49 +0200 Subject: [PATCH 026/243] chore(CI/CD): Update to actions/checkout@v4 --- .github/workflows/e2e-test-dev-legacy.yml | 2 +- .github/workflows/e2e-test-dev-managed.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- .github/workflows/lint.yml | 2 +- .github/workflows/publish-dev.yaml | 2 +- .github/workflows/publish.yml | 2 +- .github/workflows/test.yml | 4 ++-- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index 4ec99333..d3cfdee3 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -9,7 +9,7 @@ jobs: e2e_test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Go uses: actions/setup-go@v4 diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index 225c483b..522633b5 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -9,7 +9,7 @@ jobs: e2e_test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Go uses: actions/setup-go@v4 diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index e640278b..0d6c56b6 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -9,7 +9,7 @@ jobs: e2e_test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Go uses: actions/setup-go@v4 diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index b14063ff..a79ebf49 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -22,7 +22,7 @@ jobs: with: go-version: '1.20' - name: Checkout code - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Run linters uses: golangci/golangci-lint-action@v3 with: diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 800e1609..d90a620e 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Docker meta id: meta diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index b2005e75..02e18b8e 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Docker meta id: meta diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ab6c93d9..9dd15524 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -17,7 +17,7 @@ jobs: test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Go uses: actions/setup-go@v4 with: @@ -37,7 +37,7 @@ jobs: with: go-version: '1.20' - name: Checkout code - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Calc coverage run: | go test -v -covermode=count -coverprofile=coverage.out From 395e6475c6a3af3f58d23f4db69709208b3bcd03 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 27 Sep 2023 21:58:19 +0200 Subject: [PATCH 027/243] chore(CI/CD): Update docker actions in publish --- .github/workflows/publish-dev.yaml | 12 ++++++------ .github/workflows/publish.yml | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index d90a620e..cb99a403 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -1,4 +1,4 @@ -name: "release-dev" +name: Release (dev) on: workflow_dispatch: @@ -21,26 +21,26 @@ jobs: - name: Docker meta id: meta - uses: docker/metadata-action@v4 + uses: docker/metadata-action@v5 with: images: l7mp/stunnerd tags: | type=raw,value=dev - name: Set up QEMU - uses: docker/setup-qemu-action@v2 + uses: docker/setup-qemu-action@v3 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v2 + uses: docker/setup-buildx-action@v3 - name: Login to Docker Hub - uses: docker/login-action@v2 + uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USER }} password: ${{ secrets.DOCKER_TOKEN }} - name: Build and Push - uses: docker/build-push-action@v3 + uses: docker/build-push-action@v5 with: context: . platforms: linux/amd64,linux/arm64 diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 02e18b8e..ca42d533 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -1,4 +1,4 @@ -name: "release" +name: Release on: push: @@ -15,7 +15,7 @@ jobs: - name: Docker meta id: meta - uses: docker/metadata-action@v4 + uses: docker/metadata-action@v5 with: images: l7mp/stunnerd tags: | @@ -23,19 +23,19 @@ jobs: type=raw,value=latest - name: Set up QEMU - uses: docker/setup-qemu-action@v2 + uses: docker/setup-qemu-action@v3 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v2 + uses: docker/setup-buildx-action@v3 - name: Login to Docker Hub - uses: docker/login-action@v2 + uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USER }} password: ${{ secrets.DOCKER_TOKEN }} - name: Build and Push - uses: docker/build-push-action@v3 + uses: docker/build-push-action@v5 with: context: . platforms: linux/amd64,linux/arm64 From fe4b93e19041d9f7dae3e64ae74e414d6f68d948 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 07:32:25 +0200 Subject: [PATCH 028/243] chore(CI/CD): Install Dataplane in e2e managed dev --- .github/workflows/e2e-test-dev-managed.yaml | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index 522633b5..b1589095 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -51,7 +51,22 @@ jobs: - name: Configure STUNner run: | kubectl apply -f docs/examples/simple-tunnel/iperf-stunner.yaml - sleep 75 + sleep 5 + echo "apiVersion: stunner.l7mp.io/v1alpha1" > dataplane.yaml + echo "kind: Dataplane" >> dataplane.yaml + echo "metadata:" >> dataplane.yaml + echo " name: default" >> dataplane.yaml + echo "spec:" >> dataplane.yaml + echo " image: l7mp/stunnerd:dev" >> dataplane.yaml + echo " command:" >> dataplane.yaml + echo " - stunnerd" >> dataplane.yaml + echo " args:" >> dataplane.yaml + echo " - -w" >> dataplane.yaml + echo " - --udp-thread-num=16" >> dataplane.yaml + cat dataplane.yaml + kubectl apply -f dataplane.yaml + rm dataplane.yaml + sleep 35 - name: Install iperf client run: | From 146fbce78a16f7da9c3af151cde2f2e8b55d76f5 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 10:33:17 +0200 Subject: [PATCH 029/243] chore(CI/CD): Remove Dataplane config in e2e dev Helm is fixed in l7mp/stunner-helm#d5387bf. --- .github/workflows/e2e-test-dev-managed.yaml | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index b1589095..522633b5 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -51,22 +51,7 @@ jobs: - name: Configure STUNner run: | kubectl apply -f docs/examples/simple-tunnel/iperf-stunner.yaml - sleep 5 - echo "apiVersion: stunner.l7mp.io/v1alpha1" > dataplane.yaml - echo "kind: Dataplane" >> dataplane.yaml - echo "metadata:" >> dataplane.yaml - echo " name: default" >> dataplane.yaml - echo "spec:" >> dataplane.yaml - echo " image: l7mp/stunnerd:dev" >> dataplane.yaml - echo " command:" >> dataplane.yaml - echo " - stunnerd" >> dataplane.yaml - echo " args:" >> dataplane.yaml - echo " - -w" >> dataplane.yaml - echo " - --udp-thread-num=16" >> dataplane.yaml - cat dataplane.yaml - kubectl apply -f dataplane.yaml - rm dataplane.yaml - sleep 35 + sleep 75 - name: Install iperf client run: | From ae52fd050dcdd1fa21f3317ea9b750a5ebf336ba Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 10:39:47 +0200 Subject: [PATCH 030/243] chore(CI/CD): Wait for deployment in e2e mngd dev --- .github/workflows/e2e-test-dev-managed.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index 522633b5..fc6dc19f 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -61,6 +61,7 @@ jobs: - name: Wait for LoadBalancer IP run: | while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done + kubectl wait -A --for=condition=Ready deployment.apps/udp-gateway --timeout 180 kubectl get all -A - name: Start turncat From eb786a55aae326ecd5931c1ea74b1af1efd65910 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 10:45:57 +0200 Subject: [PATCH 031/243] chore(CI/CD): Fix deployment wait in e2e mngd dev --- .github/workflows/e2e-test-dev-managed.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index fc6dc19f..ccaba2e2 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -61,7 +61,7 @@ jobs: - name: Wait for LoadBalancer IP run: | while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done - kubectl wait -A --for=condition=Ready deployment.apps/udp-gateway --timeout 180 + kubectl wait -A --for=condition=Ready deployment.apps/udp-gateway --timeout 180s kubectl get all -A - name: Start turncat From 9fb16825f4e8b6378bdfd9af16c07f2192794f94 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 10:51:45 +0200 Subject: [PATCH 032/243] chore(CI/CD): Fix kubectl wait in e2e mngd dev --- .github/workflows/e2e-test-dev-managed.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index ccaba2e2..37c6db96 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -61,7 +61,7 @@ jobs: - name: Wait for LoadBalancer IP run: | while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done - kubectl wait -A --for=condition=Ready deployment.apps/udp-gateway --timeout 180s + kubectl wait -n stunner --for=condition=Ready deployment.apps/udp-gateway --timeout 180s kubectl get all -A - name: Start turncat From 1e940cd952187a0f3ae277edd31ddb2dcb5f5f5b Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 11:01:40 +0200 Subject: [PATCH 033/243] chore(CI/CD): Increase deployment wait time --- .github/workflows/e2e-test-dev-managed.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index 37c6db96..e61fb001 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -61,7 +61,10 @@ jobs: - name: Wait for LoadBalancer IP run: | while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done - kubectl wait -n stunner --for=condition=Ready deployment.apps/udp-gateway --timeout 180s + echo "* wait for the deployment" + kubectl get all -A + kubectl wait -n stunner --for=condition=Ready deployment.apps/udp-gateway --timeout 10m + echo "* EVERYTHING UP" kubectl get all -A - name: Start turncat From d2ef41a3157e1810b35e4445aa6f0bb0487d9fa8 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 11:14:30 +0200 Subject: [PATCH 034/243] chore(CI/CD): Add timeout for LB IP wait (e2e,dev) --- .github/workflows/e2e-test-dev-legacy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index d3cfdee3..1323ae04 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -61,7 +61,7 @@ jobs: - name: Wait for LoadBalancer IP run: | - while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done + for n in {1..300}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done kubectl get all -A - name: Start turncat From afc0c703520a9a852e680ba88a509670d05f2df4 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 11:27:21 +0200 Subject: [PATCH 035/243] chore(CI/CD): Add timeout for LB IP wait (e2e,stable) --- .github/workflows/e2e-test-stable.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index 0d6c56b6..f4d6a689 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -61,7 +61,7 @@ jobs: - name: Wait for LoadBalancer IP run: | - while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done + for n in {1..300}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done kubectl get all -A - name: Start turncat From 848ca10377fb182ef5fa8aaeec7ab250cabcc6be Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 28 Sep 2023 11:48:59 +0200 Subject: [PATCH 036/243] chore(CI/CD): Add timeout for LB IP wait (e2e,managed-dev) --- .github/workflows/e2e-test-dev-managed.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index e61fb001..c4b1b95e 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -60,7 +60,7 @@ jobs: - name: Wait for LoadBalancer IP run: | - while [[ -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]]; do echo "Waiting for LoadBalancer IP"; sleep 2; done + for n in {1..300}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done echo "* wait for the deployment" kubectl get all -A kubectl wait -n stunner --for=condition=Ready deployment.apps/udp-gateway --timeout 10m From 209ec04d51d752a3b52a299178e0588246b15cae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Korn=C3=A9l=20D=C3=A1vid?= <47784437+davidkornel@users.noreply.github.com> Date: Mon, 2 Oct 2023 11:11:50 +0200 Subject: [PATCH 037/243] docs: Add mediasoup demo (#103) --- docs/examples/mediasoup/README.md | 211 ++++++++++++++++++ .../mediasoup/mediasoup-call-stunner.yaml | 52 +++++ docs/examples/mediasoup/mediasoup-server.yaml | 111 +++++++++ 3 files changed, 374 insertions(+) create mode 100644 docs/examples/mediasoup/README.md create mode 100644 docs/examples/mediasoup/mediasoup-call-stunner.yaml create mode 100644 docs/examples/mediasoup/mediasoup-server.yaml diff --git a/docs/examples/mediasoup/README.md b/docs/examples/mediasoup/README.md new file mode 100644 index 00000000..2e4ee26e --- /dev/null +++ b/docs/examples/mediasoup/README.md @@ -0,0 +1,211 @@ +# STUNner demo: Video-conferencing with mediasoup + +This document guides you through the installation of [mediasoup](https://mediasoup.org/) into Kubernetes, when it is used together with the STUNner WebRTC media gateway. + +In this demo you will learn to: + +- integrate a typical WebRTC application with STUNner, +- obtain a valid TLS certificate to secure the signaling plane, +- deploy the mediasoup server into Kubernetes, and +- configure STUNner to expose mediasoup to clients. + +## Prerequisites + +The below installation instructions require an operational cluster running a supported version of Kubernetes (>1.22). Most hosted or private Kubernetes cluster services will work, but make sure that the cluster comes with a functional load-balancer integration (all major hosted Kubernetes services should support this). Otherwise, STUNner will not be able to allocate a public IP address for clients to reach your WebRTC infra. As a regrettable exception, Minikube is unfortunately not supported for this demo. The reason is that [Let's Encrypt certificate issuance is not available with nip.io when using local/private IPs](https://medium.com/@EmiiKhaos/there-is-no-possibility-that-you-can-get-lets-encrypt-certificate-with-nip-io-7483663e0c1b); later on, you will learn more about why this is crucial above. + +## Setup + +The recommended way to install mediasoup ([link](https://mediasoup.discourse.group/t/server-in-kubernetes-with-turn/3434),[link](https://www.reddit.com/r/kubernetes/comments/sdkhwn/deploying_mediasoup_webrtc_sfu_in_kubernetes/)) into Kubernetes is deploying the media servers into the host-network namespace of the Kubernetes nodes (`hostNetwork: true`). This deployment model, however, comes with a set of uncanny [operational limitations and security concerns](../../WHY.md). Using STUNner, however, media servers can be deployed into ordinary Kubernetes pods and run over a private IP network, like any "normal" Kubernetes workload. + +The figure below shows mediasoup deployed into regular Kubernetes pods behind STUNner without the host-networking hack. Here, mediasoup is deployed behind STUNner in the [*media-plane deployment model*](../../DEPLOYMENT.md), so that STUNner acts as a "local" STUN/TURN server for mediasoup, saving the overhead of using public a 3rd party STUN/TURN server for NAT traversal. + +In this tutorial we deploy a video room example using [mediasoup's demo application](https://github.com/versatica/mediasoup-demo/) with slight modifications (more on these below), the [mediasoup server](https://github.com/versatica/mediasoup/) for media exchange, a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the LiveKit server pool to clients. + +### Modifications on the mediasoup demo + +Below are the modification that has been done starting from [mediasoup-demo](https://github.com/versatica/mediasoup-demo/): + +- Added a multistage Dockerfile + + - stage 0: run gulp dist to create the frontend app file (they will be served by nodejs from the backend) + - stage 1: build the image for the mediasoup-server and copy the mediasoup-client file + +- Added a simple script that gathers the internal/private IP of the running pod, this is not foolproof, however, +with an additional environment variable we can load the pod's private IP into the code + +- Added the following in server.js in the function "async function createExpressApp()" to serve the mediasoup-client file + +``` +147: expressApp.use(express.static('public')) +``` + +- Added the parsing of url parameters to configure TURN server and a simple if/else in server/app/lib/RoomClient.js. The mediasoup clients will use the configured TURN server to gather the ICE candidates. Example: https://mediasoup-demo.example.com/?enableIceServer=yes&iceServerHost=100.100.100.100&iceServerPort=3478&iceServerProto=udp&iceServerUser=user-1&iceServerPass=pass-1 + +## Installation + +Let's start with a disclaimer. The mediasoup demo example must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. According to the [documentation](https://github.com/versatica/mediasoup-demo/blob/a59c6ab8e50fb950c3df54f4b85167a4e3f8497a/README.md?plain=1#L96) mediasoup should work with self-signed certs, however this haven't been tested. In the following we will deploy mediasoup configured with a valid signed TLS certificate. This will have the unfortunate consequence that the majority of the below installation guide will be about securing client connections to mediasoup over TLS; as it turns out, once HTTPS is correctly working integrating mediasoup with STUNner is very simple. + +In the below example, STUNner will be installed into the identically named namespace, while mediasoup and the Ingress gateway will live in the default namespace. + +### TLS certificates + +As mentioned above, the mediasoup server will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't then [nip.io](https://nip.io) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for mediasoup. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. + +Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of `nip.io`. + +### Ingress + +The first step of obtaining a valid cert is to install a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. + +Install an ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but this is not required. + +```console +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm repo update +helm install ingress-nginx ingress-nginx/ingress-nginx +``` + +Wait until Kubernetes assigns an external IP to the Ingress. + +```console +until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +``` + +Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. + +```console +kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' +export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') +``` + +### Cert manager + +We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. + +First, install cert-manager's CRDs. + +```console +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml +``` + +Then add the Helm repository, which contains the cert-manager Helm chart, and install the charts: + +```console +helm repo add cert-manager https://charts.jetstack.io +helm repo update +helm install my-cert-manager cert-manager/cert-manager \ + --create-namespace \ + --namespace cert-manager \ + --version v1.8.0 +``` + +At this point we have all the necessary boilerplate set up to automate TLS issuance for mediasoup. + +### STUNner + +Now comes the fun part. + +Install the STUNner gateway operator using the [managed dataplane mode](https://github.com/l7mp/stunner/blob/main/docs/INSTALL.md#managed-mode) via [Helm](https://github.com/l7mp/stunner-helm): + +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace + --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed --set stunnerGatewayOperator.dataplane.spec.image=l7mp/stunnerd:latest +``` + +Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the mediasoup server pods. + +```console +git clone https://github.com/l7mp/stunner +cd stunner +kubectl apply -f docs/examples/mediasoup/mediasoup-call-stunner.yaml +``` + +The relevant parts here are the STUNner [Gateway definition](../../GATEWAY.md), which exposes the STUNner STUN/TURN server over UDP:3478 to the Internet, and the [UDPRoute definition](../../GATEWAY.md), which takes care of routing media to the pods running behind the `mediasoup-server` service. + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: TURN-UDP +--- +apiVersion: gateway.networking.k8s.io/v1alpha2 +kind: UDPRoute +metadata: + name: livekit-media-plane + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - group: "" + kind: Service + name: mediasoup-server + namespace: mediasoup +``` + +Once the Gateway resource is installed into Kubernetes, STUNner will create a Kubernetes LoadBalancer for the Gateway to expose the TURN server on UDP:3478 to clients. It can take up to a minute for Kubernetes to allocate a public external IP for the service. + +Wait until Kubernetes assigns an external IP and store the external IP assigned by Kubernetes to +STUNner in an environment variable for later use. + +```console +until [ -n "$(kubectl get svc udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +export STUNNERIP=$(kubectl get service udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +``` + +### mediasoup + +The crucial step of integrating *any* WebRTC media server with STUNner is to ensure that the server instructs the clients to use STUNner as the STUN/TURN server. However, there is a slight issue. In this deployment it's not the server that instructs the clients to use STUNner but the user itself. Obviously, it is not the optimal way but for the sake of the demo purpose we keep it that way. In case anyone would want to create a production ready deployment, they would need to add extra capabilities to the mediasoup server: +- first to make sure turn servers can be configured in the server's config.js file +- second to make sure that clients can fetch (or get automatically) the configured turn servers from the mediasoup server + +We need the Ingress external IP address we have stored previously: this will make sure that the TLS certificate created by cert-manager will be bound to the proper `nip.io` domain and IP address. + +```console +sed -i "s/ingressserviceip/$INGRESSIP/g" docs/examples/mediasoup/mediasoup-server.yaml +``` + +Finally, fire up mediasoup. + +```console +kubectl apply -f docs/examples/mediasoup/mediasoup-server.yaml +``` + +The demo installation bundle includes a lot of resources to deploy mediasoup: + +- a mediasoup-server, +- an application server serving the landing page using [mediasoup-demo](https://github.com/versatica/mediasoup-demo/) +- a cluster issuer for the TLS certificates, +- an Ingress resource to terminate the secure connections between your browser and the Kubernetes cluster. + +Wait until all pods become operational and jump right into testing! + +## Test + +After installing everything, execute the following command to retrieve the URL of your fresh mediasoup demo app: + +```console +echo "https://mediasoup-$INGRESSIP.nip.io:443?enableIceServer=yes&iceServerHost=$STUNNERIP&iceServerPort=3478&iceServerProto=udp&iceServerUser=user-1&iceServerPass=pass-1" +``` + +Opening the output in a browser should get the mediasoup client demo app + +In case you changed something additionally in the STUNner configuration during deployment watch out for the URL parameters: + - `enableIceServer` must be `yes` in order to use STUNner as a TURN server + - `iceServerHost` should point to the public IP that was allocated for the STUNner load balancer service + - `iceServerPort` is the port of your TURN server configured in the Gateway resource + - `iceServerProto` is the expected protocol on the port configured above + - `iceServerUser` is the username used for authentication in STUNner + - `iceServerPass` is the credential used for authentication in STUNner + diff --git a/docs/examples/mediasoup/mediasoup-call-stunner.yaml b/docs/examples/mediasoup/mediasoup-call-stunner.yaml new file mode 100644 index 00000000..85862586 --- /dev/null +++ b/docs/examples/mediasoup/mediasoup-call-stunner.yaml @@ -0,0 +1,52 @@ +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: GatewayClass +metadata: + name: stunner-gatewayclass +spec: + controllerName: "stunner.l7mp.io/gateway-operator" + parametersRef: + group: "stunner.l7mp.io" + kind: GatewayConfig + name: stunner-gatewayconfig + namespace: stunner + description: "STUNner is a WebRTC ingress gateway for Kubernetes" +--- +apiVersion: stunner.l7mp.io/v1alpha1 +kind: GatewayConfig +metadata: + name: stunner-gatewayconfig + namespace: stunner +spec: + realm: stunner.l7mp.io + authType: plaintext + userName: "user-1" + password: "pass-1" +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: TURN-UDP +--- +apiVersion: gateway.networking.k8s.io/v1alpha2 +kind: UDPRoute +metadata: + name: livekit-media-plane + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - group: "" + kind: Service + name: mediasoup-server + namespace: mediasoup +--- \ No newline at end of file diff --git a/docs/examples/mediasoup/mediasoup-server.yaml b/docs/examples/mediasoup/mediasoup-server.yaml new file mode 100644 index 00000000..67117d93 --- /dev/null +++ b/docs/examples/mediasoup/mediasoup-server.yaml @@ -0,0 +1,111 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app.kubernetes.io/name: mediasoup-server + name: mediasoup-server + namespace: mediasoup +spec: + progressDeadlineSeconds: 600 + replicas: 1 + revisionHistoryLimit: 10 + selector: + matchLabels: + app.kubernetes.io/name: mediasoup-server + strategy: + rollingUpdate: + maxSurge: 25% + maxUnavailable: 25% + type: RollingUpdate + template: + metadata: + labels: + app.kubernetes.io/name: mediasoup-server + spec: + containers: + - env: + - name: PROTOO_LISTEN_PORT + value: "443" + - name: MEDIASOUP_ANNOUNCED_IP + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: status.podIP + image: l7mp/mediasoup-demo-docker:latest + imagePullPolicy: IfNotPresent + name: mediasoup-server + ports: + - containerPort: 80 + name: http + protocol: TCP + - containerPort: 443 + name: https + protocol: TCP + resources: {} + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + dnsPolicy: ClusterFirst + restartPolicy: Always + schedulerName: default-scheduler + securityContext: {} + terminationGracePeriodSeconds: 30 +--- +apiVersion: v1 +kind: Service +metadata: + name: mediasoup-server + namespace: mediasoup +spec: + ports: + - name: https-443 + port: 443 + protocol: TCP + targetPort: 443 + selector: + app.kubernetes.io/name: mediasoup-server + type: ClusterIP +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + kubernetes.io/ingress.class: nginx + nginx.ingress.kubernetes.io/backend-protocol: HTTPS + nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_roomId" + name: mediasoup-server + namespace: mediasoup +spec: + rules: + - host: mediasoup-ingressserviceip.nip.io + http: + paths: + - backend: + service: + name: mediasoup-server + port: + number: 443 + path: / + pathType: Prefix + tls: + - hosts: + - mediasoup-ingressserviceip.nip.io + secretName: mediasoup-demo-tls +--- +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + generation: 1 + name: letsencrypt-prod +spec: + acme: + email: info@yourdomain.com + privateKeySecretRef: + name: letsencrypt-secret-prod + server: https://acme-v02.api.letsencrypt.org/directory + solvers: + - http01: + ingress: + class: nginx +--- \ No newline at end of file From 697449eb7606f27488ae1800babd939654b06060 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 2 Oct 2023 11:48:04 +0200 Subject: [PATCH 038/243] docs(mediasoup): Add Help and Acknowledgments --- docs/examples/mediasoup/README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/examples/mediasoup/README.md b/docs/examples/mediasoup/README.md index 2e4ee26e..709a0f74 100644 --- a/docs/examples/mediasoup/README.md +++ b/docs/examples/mediasoup/README.md @@ -209,3 +209,11 @@ In case you changed something additionally in the STUNner configuration during d - `iceServerUser` is the username used for authentication in STUNner - `iceServerPass` is the credential used for authentication in STUNner + +## Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). + +## Acknowledgments + +This demo is adopted from [damhau/mediasoup-demo-docker](https://github.com/damhau/mediasoup-demo-docker). Huge thanks to @damhau for the great demo! From b67efdc50ef260d5f2a878bd7d7847ebc9d41497 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 2 Oct 2023 22:07:11 +0200 Subject: [PATCH 039/243] chore(CI/CD): Fix end2end tests w/ dev versions Changes: - Start minikube w/ max available CPUs - Decrease deployment requirements to prevent overusing CPUs - Update helm version - Wait for udp-gateway deployment to be available (sometimes the svc LB IP was assigned without the backend pod being running) - Fix service name in turncat command - Run managed test 1 hour after the legacy --- .github/workflows/e2e-test-dev-legacy.yml | 15 ++++++++++----- .github/workflows/e2e-test-dev-managed.yaml | 13 +++++++------ 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index 1323ae04..1624bff4 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -28,6 +28,7 @@ jobs: with: driver: docker container-runtime: containerd + cpus: max wait: all cache: false @@ -37,14 +38,14 @@ jobs: - name: Set up Helm uses: azure/setup-helm@v3 with: - version: v3.11.3 + version: v3.12.3 - name: Install STUNner run: | helm repo add stunner https://l7mp.io/stunner helm repo update - helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner - helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner + helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m + helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner --set stunner.deployment.container.stunnerd.resources.requests.cpu=200m - name: Deploy iperf server run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml @@ -61,12 +62,16 @@ jobs: - name: Wait for LoadBalancer IP run: | - for n in {1..300}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done + for n in {1..60}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done + echo "* wait for the deployment" + kubectl get all -A + kubectl wait -n stunner --for=condition=Available deployment udp-gateway --timeout 5m + echo "* EVERYTHING UP" kubectl get all -A - name: Start turncat run: | - ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & + ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & sleep 1 - name: Run iperf client diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index c4b1b95e..beeac794 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -3,7 +3,7 @@ name: Run End-to-End Test [managed] (dev) on: workflow_dispatch: schedule: - - cron: '0 11 * * 1' + - cron: '0 12 * * 1' jobs: e2e_test: @@ -29,6 +29,7 @@ jobs: driver: docker container-runtime: containerd wait: all + cpus: max cache: false - name: Start minikube tunnel @@ -37,13 +38,13 @@ jobs: - name: Set up Helm uses: azure/setup-helm@v3 with: - version: v3.11.3 + version: v3.12.3 - name: Install STUNner run: | helm repo add stunner https://l7mp.io/stunner helm repo update - helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed + helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m --set stunnerGatewayOperator.dataplane.spec.resources.requests.cpu=100m - name: Deploy iperf server run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml @@ -60,16 +61,16 @@ jobs: - name: Wait for LoadBalancer IP run: | - for n in {1..300}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done + for n in {1..60}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done echo "* wait for the deployment" kubectl get all -A - kubectl wait -n stunner --for=condition=Ready deployment.apps/udp-gateway --timeout 10m + kubectl wait -n stunner --for=condition=Available deployment udp-gateway --timeout 5m echo "* EVERYTHING UP" kubectl get all -A - name: Start turncat run: | - ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & + ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & sleep 1 - name: Run iperf client From a610f36854e067eefa54fec96423b7611d7d8796 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 2 Oct 2023 22:18:14 +0200 Subject: [PATCH 040/243] chore(CI/CD): Fix end2end legacy dev test --- .github/workflows/e2e-test-dev-legacy.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index 1624bff4..5f4048bb 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -65,13 +65,13 @@ jobs: for n in {1..60}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done echo "* wait for the deployment" kubectl get all -A - kubectl wait -n stunner --for=condition=Available deployment udp-gateway --timeout 5m + kubectl wait -n stunner --for=condition=Available deployment stunner --timeout 5m echo "* EVERYTHING UP" kubectl get all -A - name: Start turncat run: | - ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & + ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & sleep 1 - name: Run iperf client From 548af70162d3d1f7a5a2bb43e29dc7715fd3e701 Mon Sep 17 00:00:00 2001 From: Kornel David Date: Tue, 3 Oct 2023 17:12:09 +0200 Subject: [PATCH 041/243] doc: Link mediasoup demo in root directory README --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index e1df936d..4f0be215 100644 --- a/README.md +++ b/README.md @@ -582,6 +582,11 @@ applications into Kubernetes. STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling connections, using [cert-manager](https://cert-manager.io), [nip.io](https://nip.io) and [Let's Encrypt](https://letsencrypt.org). +* [Video-conferencing with mediasoup](/docs/examples/mediasoup/README.md): This tutorial helps you deploy + the [mediasoup](https://mediasoup.org/) WebRTC media server behind STUNner. The docs also show how to + obtain a valid TLS certificate to secure your signaling connections, courtesy of the + [cert-manager](https://cert-manager.io) project, [nip.io](https://nip.io) and [Let's + Encrypt](https://letsencrypt.org). * [Cloud-gaming with Cloudretro](/docs/examples/cloudretro/README.md): This tutorial lets you play Super Mario or Street Fighter in your browser, courtesy of the amazing [CloudRetro](https://cloudretro.io) project and, of course, STUNner. The demo also presents a From 4a3ef85d5a952d6f819ec28e5d07ac1951fde62e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 3 Oct 2023 17:43:54 +0200 Subject: [PATCH 042/243] doc: Dataplane CRD docs updated --- docs/INSTALL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index e61a71d7..48abb57a 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -94,14 +94,14 @@ kind: Dataplane metadata: name: default spec: + image: l7mp/stunnerd:latest + imagePullPolicy: Always + command: + - stunnerd args: - -w - --udp-thread-num=16 - command: - - stunnerd hostNetwork: false - image: l7mp/stunnerd:latest - imagePullPolicy: Always resources: limits: cpu: 2 From 9867fc3d32d571154c7792ca74a0272149106e7f Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 4 Oct 2023 14:47:33 +0200 Subject: [PATCH 043/243] doc: Add a disclaimer that the inline docs track the bleeding edge --- README.md | 2 ++ docs/README.md | 2 ++ docs/RELEASE.md | 2 ++ 3 files changed, 6 insertions(+) diff --git a/README.md b/README.md index 4f0be215..978a3fac 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 546d1cdc..dd02c537 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,8 @@ STUNner

+*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io).* + ## Overview * [Why STUNner](WHY.md) diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 58c41c83..ce2baaa7 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -4,6 +4,8 @@ Let the new version be vX.Y.Z. ## STUNner +- remove the doc disclaimers ("this doc is for the dev version, see RTD for stable docs") from the + README.md and docs/README.md - write release notes for vX.Y.Z - `git pull` - `go mod tidy` From 44a8b44b18441db43497da2c3aec0dc1a70c546d Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Mon, 9 Oct 2023 16:04:34 +0200 Subject: [PATCH 044/243] fix(rtd): Fix navigation --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 0abc27df..c34fb616 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,7 +9,7 @@ theme: name: readthedocs highlightjs: true nav: -- 'README.md' +- 'STUNner documentation': 'README.md' - 'Why STUNner': 'WHY.md' - 'Deployment models': 'DEPLOYMENT.md' - 'Concepts': 'CONCEPTS.md' From 7903b6d6c04c9686cb499774a626d31892fa1b04 Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Tue, 10 Oct 2023 14:11:43 +0200 Subject: [PATCH 045/243] fix(rtd): Add index.md --- docs/index.md | 1 + 1 file changed, 1 insertion(+) create mode 120000 docs/index.md diff --git a/docs/index.md b/docs/index.md new file mode 120000 index 00000000..42061c01 --- /dev/null +++ b/docs/index.md @@ -0,0 +1 @@ +README.md \ No newline at end of file From 6056daa27e942422a8efc282dcdafc720a5c74a3 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Sun, 15 Oct 2023 19:14:49 +0200 Subject: [PATCH 046/243] chore(CI/CD): Bump go version to v1.21 --- .github/workflows/e2e-test-dev-legacy.yml | 2 +- .github/workflows/e2e-test-dev-managed.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 4 ++-- .github/workflows/lint.yml | 2 +- .github/workflows/test.yml | 4 ++-- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index 5f4048bb..1d5f5eac 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v4 with: - go-version: '1.20' + go-version: '1.21' - name: Download modules run: go mod download diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index beeac794..414d6a21 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v4 with: - go-version: '1.20' + go-version: '1.21' - name: Download modules run: go mod download diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index f4d6a689..e289a1f0 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v4 with: - go-version: '1.20' + go-version: '1.21' - name: Download modules run: go mod download @@ -37,7 +37,7 @@ jobs: - name: Set up Helm uses: azure/setup-helm@v3 with: - version: v3.11.3 + version: v3.12.3 - name: Install STUNner run: | diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index a79ebf49..2e4b455b 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -20,7 +20,7 @@ jobs: - name: Install Go uses: actions/setup-go@v4 with: - go-version: '1.20' + go-version: '1.21' - name: Checkout code uses: actions/checkout@v4 - name: Run linters diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9dd15524..c0b05703 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -21,7 +21,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v4 with: - go-version: '1.20' + go-version: '1.21' - name: Download modules run: go mod download - name: Go install @@ -35,7 +35,7 @@ jobs: if: success() uses: actions/setup-go@v4 with: - go-version: '1.20' + go-version: '1.21' - name: Checkout code uses: actions/checkout@v4 - name: Calc coverage From deee808b9f8337aee1e073030c49c0c3e90be3a9 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:02:53 +0200 Subject: [PATCH 047/243] chore(ci/cd): Do not discard linter issues --- .github/workflows/lint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 2e4b455b..26b35311 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -26,4 +26,4 @@ jobs: - name: Run linters uses: golangci/golangci-lint-action@v3 with: - args: --timeout 3m --issues-exit-code=0 + args: --timeout 5m From 971aae5d09990709d6897685326e9e266b44e4d2 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:03:10 +0200 Subject: [PATCH 048/243] chore(ci/cd): Set publish step dependencies --- .github/workflows/publish-dev.yaml | 14 +++++++++++++- .github/workflows/publish.yml | 13 ++++++++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index cb99a403..f26edf86 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -12,8 +12,19 @@ on: - 'main' jobs: + run_tests: + name: Run tests + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Run tests + uses: ./.github/workflows/test.yml + push_to_registry: name: Push Docker image to DockerHub + needs: run_tests runs-on: ubuntu-latest steps: - name: Checkout @@ -49,7 +60,8 @@ jobs: labels: ${{ steps.meta.outputs.labels }} push_chart: - name: Push helm charts to the + name: Push helm charts to the repo + needs: push_to_registry runs-on: ubuntu-latest steps: - name: Triggering release workflow in the stunner-helm repo diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index ca42d533..899c9e5b 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -6,8 +6,19 @@ on: - 'v[0-9]+.[0-9]+.0' jobs: + run_tests: + name: Run tests + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Run tests + uses: ./.github/workflows/test.yml + push_to_registry: name: Push Docker image to DockerHub + needs: run_tests runs-on: ubuntu-latest steps: - name: Checkout @@ -45,9 +56,9 @@ jobs: push_chart: name: Push charts to the web + needs: push_to_registry runs-on: ubuntu-latest steps: - - name: Get version id: vars run: echo tag=$(echo ${GITHUB_REF:11}) >> $GITHUB_OUTPUT From 1b0edff42d77a2ea0fe6242b7d4c07d2364b5804 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:09:23 +0200 Subject: [PATCH 049/243] chore(CI/CD): Fix dev publish workflow --- .github/workflows/publish-dev.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index f26edf86..5a9895e0 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -20,7 +20,7 @@ jobs: uses: actions/checkout@v4 - name: Run tests - uses: ./.github/workflows/test.yml + uses: ./.github/workflows/test push_to_registry: name: Push Docker image to DockerHub From ea2174034c63b9025c2c912c0c4dfd6d47b24a72 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:11:07 +0200 Subject: [PATCH 050/243] chore(CI/CD): Revert referencing test action --- .github/workflows/publish-dev.yaml | 16 +++++++++++----- .github/workflows/publish.yml | 16 +++++++++++----- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 5a9895e0..c7f271a4 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -16,11 +16,17 @@ jobs: name: Run tests runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Run tests - uses: ./.github/workflows/test + - uses: actions/checkout@v4 + - name: Set up Go + uses: actions/setup-go@v4 + with: + go-version: '1.21' + - name: Download modules + run: go mod download + - name: Go install + run: go install + - name: Test + run: go test -v -covermode=count push_to_registry: name: Push Docker image to DockerHub diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 899c9e5b..113308ac 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -10,11 +10,17 @@ jobs: name: Run tests runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Run tests - uses: ./.github/workflows/test.yml + - uses: actions/checkout@v4 + - name: Set up Go + uses: actions/setup-go@v4 + with: + go-version: '1.21' + - name: Download modules + run: go mod download + - name: Go install + run: go install + - name: Test + run: go test -v -covermode=count push_to_registry: name: Push Docker image to DockerHub From fdf5993505ea2acc54940ac6aebbf13615f282ed Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:53:01 +0200 Subject: [PATCH 051/243] chore(CI/CD): Fix test action ref in publish --- .github/workflows/publish-dev.yaml | 14 +------------- .github/workflows/publish.yml | 14 +------------- 2 files changed, 2 insertions(+), 26 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index c7f271a4..3988fd7e 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -14,19 +14,7 @@ on: jobs: run_tests: name: Run tests - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Set up Go - uses: actions/setup-go@v4 - with: - go-version: '1.21' - - name: Download modules - run: go mod download - - name: Go install - run: go install - - name: Test - run: go test -v -covermode=count + use: l7mp/stunner/.github/workflows/test.yml@main push_to_registry: name: Push Docker image to DockerHub diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 113308ac..031a3a68 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -8,19 +8,7 @@ on: jobs: run_tests: name: Run tests - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Set up Go - uses: actions/setup-go@v4 - with: - go-version: '1.21' - - name: Download modules - run: go mod download - - name: Go install - run: go install - - name: Test - run: go test -v -covermode=count + use: l7mp/stunner/.github/workflows/test.yml@main push_to_registry: name: Push Docker image to DockerHub From 6f04aa63e6f9a6d552c8fa42963ef5dd28171eac Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:55:21 +0200 Subject: [PATCH 052/243] chore(CI/CD): Fix test action use in publish --- .github/workflows/publish-dev.yaml | 2 +- .github/workflows/publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 3988fd7e..ca0f16c9 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -14,7 +14,7 @@ on: jobs: run_tests: name: Run tests - use: l7mp/stunner/.github/workflows/test.yml@main + uses: l7mp/stunner/.github/workflows/test.yml@main push_to_registry: name: Push Docker image to DockerHub diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 031a3a68..e8537c6b 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -8,7 +8,7 @@ on: jobs: run_tests: name: Run tests - use: l7mp/stunner/.github/workflows/test.yml@main + uses: l7mp/stunner/.github/workflows/test.yml@main push_to_registry: name: Push Docker image to DockerHub From e5312081e584447b1ff2a1e4197815259b3985f8 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Oct 2023 16:57:38 +0200 Subject: [PATCH 053/243] chore(CI/CD): Make test action callable --- .github/workflows/test.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index c0b05703..796cb920 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -1,6 +1,8 @@ name: Tests on: + workflow_call: + workflow_dispatch: push: paths: - '**.go' From 74a436523fd1e32dfb181bad94a6de371d89a904 Mon Sep 17 00:00:00 2001 From: Kornel David Date: Tue, 24 Oct 2023 20:56:04 +0200 Subject: [PATCH 054/243] chore(manifests/static): Update static manifest #85 - by default the legacy dataplane mode is used, however a dataplane CR sample is also provided --- deploy/manifests/static/dataplane.yaml | 24 + deploy/manifests/static/gateway-api-crd.yaml | 8969 ++++++++++++++++- deploy/manifests/static/stunner-crd.yaml | 1418 ++- .../stunner-gateway-operator-manifests.yaml | 81 +- .../manifests/static/stunner-manifests.yaml | 20 +- 5 files changed, 10170 insertions(+), 342 deletions(-) create mode 100644 deploy/manifests/static/dataplane.yaml diff --git a/deploy/manifests/static/dataplane.yaml b/deploy/manifests/static/dataplane.yaml new file mode 100644 index 00000000..3c8319f7 --- /dev/null +++ b/deploy/manifests/static/dataplane.yaml @@ -0,0 +1,24 @@ +--- +apiVersion: stunner.l7mp.io/v1alpha1 +kind: Dataplane +metadata: + name: default +spec: + replicas: 1 + image: l7mp/stunnerd:latest + imagePullPolicy: IfNotPresent + command: + - stunnerd + args: + - -w + - --udp-thread-num=16 + resources: + limits: + cpu: 2 + memory: 512Mi + requests: + cpu: 500m + memory: 128Mi + terminationGracePeriodSeconds: 3600 + hostNetwork: false +--- \ No newline at end of file diff --git a/deploy/manifests/static/gateway-api-crd.yaml b/deploy/manifests/static/gateway-api-crd.yaml index 60e9d1c6..646684fd 100644 --- a/deploy/manifests/static/gateway-api-crd.yaml +++ b/deploy/manifests/static/gateway-api-crd.yaml @@ -3,8 +3,8 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/1538 - gateway.networking.k8s.io/bundle-version: v0.6.2 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: gatewayclasses.gateway.networking.k8s.io @@ -37,7 +37,7 @@ spec: type: string # deprecated: true # deprecationWarning: The v1alpha2 version of GatewayClass has been deprecated and - # will be removed in a future release of the API. Please upgrade to v1beta1. + # will be removed in a future release of the API. Please upgrade to v1beta1. name: v1alpha2 schema: openAPIV3Schema: @@ -50,7 +50,7 @@ spec: to GatewayClass or associated parameters. If implementations choose to propagate GatewayClass changes to existing Gateways, that MUST be clearly documented by the implementation. \n Whenever one or more Gateways are using a GatewayClass, - implementations MUST add the `gateway-exists-finalizer.gateway.networking.k8s.io` + implementations SHOULD add the `gateway-exists-finalizer.gateway.networking.k8s.io` finalizer on the associated GatewayClass. This ensures that a GatewayClass associated with a Gateway is not deleted while in use. \n GatewayClass is a Cluster level resource." @@ -79,6 +79,9 @@ spec: minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf description: description: Description helps describe a GatewayClass with more details. maxLength: 64 @@ -133,7 +136,9 @@ spec: reason: Waiting status: Unknown type: Accepted - description: Status defines the current state of GatewayClass. + description: "Status defines the current state of GatewayClass. \n Implementations + MUST populate status on all GatewayClass resources which specify their + controller name." properties: conditions: default: @@ -149,14 +154,12 @@ spec: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions. For example, - \n \ttype FooStatus struct{ \t // Represents the observations - of a foo's current state. \t // Known .status.conditions.type - are: \"Available\", \"Progressing\", and \"Degraded\" \t // - +patchMergeKey=type \t // +patchStrategy=merge \t // +listType=map - \t // +listMapKey=type \t Conditions []metav1.Condition + \n type FooStatus struct{ // Represents the observations of a + foo's current state. // Known .status.conditions.type are: \"Available\", + \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other fields - \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -252,7 +255,7 @@ spec: to GatewayClass or associated parameters. If implementations choose to propagate GatewayClass changes to existing Gateways, that MUST be clearly documented by the implementation. \n Whenever one or more Gateways are using a GatewayClass, - implementations MUST add the `gateway-exists-finalizer.gateway.networking.k8s.io` + implementations SHOULD add the `gateway-exists-finalizer.gateway.networking.k8s.io` finalizer on the associated GatewayClass. This ensures that a GatewayClass associated with a Gateway is not deleted while in use. \n GatewayClass is a Cluster level resource." @@ -281,6 +284,9 @@ spec: minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf description: description: Description helps describe a GatewayClass with more details. maxLength: 64 @@ -335,7 +341,9 @@ spec: reason: Waiting status: Unknown type: Accepted - description: Status defines the current state of GatewayClass. + description: "Status defines the current state of GatewayClass. \n Implementations + MUST populate status on all GatewayClass resources which specify their + controller name." properties: conditions: default: @@ -351,14 +359,12 @@ spec: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions. For example, - \n \ttype FooStatus struct{ \t // Represents the observations - of a foo's current state. \t // Known .status.conditions.type - are: \"Available\", \"Progressing\", and \"Degraded\" \t // - +patchMergeKey=type \t // +patchStrategy=merge \t // +listType=map - \t // +listMapKey=type \t Conditions []metav1.Condition + \n type FooStatus struct{ // Represents the observations of a + foo's current state. // Known .status.conditions.type are: \"Available\", + \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other fields - \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -432,15 +438,15 @@ status: acceptedNames: kind: "" plural: "" - conditions: [] - storedVersions: [] + conditions: null + storedVersions: null --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/1538 - gateway.networking.k8s.io/bundle-version: v0.6.2 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: gateways.gateway.networking.k8s.io @@ -510,10 +516,24 @@ spec: manner, assigning an appropriate set of Addresses. \n The implementation MUST bind all Listeners to every GatewayAddress that it assigns to the Gateway and add a corresponding entry in GatewayStatus.Addresses. - \n Support: Extended" + \n Support: Extended \n " items: description: GatewayAddress describes an address that can be bound to a Gateway. + oneOf: + - properties: + type: + enum: + - IPAddress + value: + anyOf: + - format: ipv4 + - format: ipv6 + - properties: + type: + not: + enum: + - IPAddress properties: type: default: IPAddress @@ -532,8 +552,20 @@ spec: required: - value type: object + x-kubernetes-validations: + - message: Hostname value must only contain valid characters (matching + ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$) + rule: 'self.type == ''Hostname'' ? self.value.matches(r"""^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"""): + true' maxItems: 16 type: array + x-kubernetes-validations: + - message: IPAddress values must be unique + rule: 'self.all(a1, a1.type == ''IPAddress'' ? self.exists_one(a2, + a2.type == a1.type && a2.value == a1.value) : true )' + - message: Hostname values must be unique + rule: 'self.all(a1, a1.type == ''Hostname'' ? self.exists_one(a2, + a2.type == a1.type && a2.value == a1.value) : true )' gatewayClassName: description: GatewayClassName used for this Gateway. This is the name of a GatewayClass resource. @@ -545,27 +577,34 @@ spec: logical endpoints that are bound on this Gateway's addresses. At least one Listener MUST be specified. \n Each listener in a Gateway must have a unique combination of Hostname, Port, and Protocol. - \n An implementation MAY group Listeners by Port and then collapse - each group of Listeners into a single Listener if the implementation - determines that the Listeners in the group are \"compatible\". An - implementation MAY also group together and collapse compatible Listeners - belonging to different Gateways. \n For example, an implementation - might consider Listeners to be compatible with each other if all - of the following conditions are met: \n 1. Either each Listener - within the group specifies the \"HTTP\" Protocol or each Listener - within the group specifies either the \"HTTPS\" or \"TLS\" Protocol. - \n 2. Each Listener within the group specifies a Hostname that is - unique within the group. \n 3. As a special case, one Listener - within a group may omit Hostname, in which case this Listener - matches when no other Listener matches. \n If the implementation - does collapse compatible Listeners, the hostname provided in the - incoming client request MUST be matched to a Listener to find the - correct set of Routes. The incoming hostname MUST be matched using - the Hostname field for each Listener in order of most to least specific. - That is, exact matches must be processed before wildcard matches. - \n If this field specifies multiple Listeners that have the same - Port value but are not compatible, the implementation must raise - a \"Conflicted\" condition in the Listener status. \n Support: Core" + \n Within the HTTP Conformance Profile, the below combinations of + port and protocol are considered Core and MUST be supported: \n + 1. Port: 80, Protocol: HTTP 2. Port: 443, Protocol: HTTPS \n Within + the TLS Conformance Profile, the below combinations of port and + protocol are considered Core and MUST be supported: \n 1. Port: + 443, Protocol: TLS \n Port and protocol combinations not listed + above are considered Extended. \n An implementation MAY group Listeners + by Port and then collapse each group of Listeners into a single + Listener if the implementation determines that the Listeners in + the group are \"compatible\". An implementation MAY also group together + and collapse compatible Listeners belonging to different Gateways. + \n For example, an implementation might consider Listeners to be + compatible with each other if all of the following conditions are + met: \n 1. Either each Listener within the group specifies the \"HTTP\" + Protocol or each Listener within the group specifies either the + \"HTTPS\" or \"TLS\" Protocol. \n 2. Each Listener within the group + specifies a Hostname that is unique within the group. \n 3. As a + special case, one Listener within a group may omit Hostname, in + which case this Listener matches when no other Listener matches. + \n If the implementation does collapse compatible Listeners, the + hostname provided in the incoming client request MUST be matched + to a Listener to find the correct set of Routes. The incoming hostname + MUST be matched using the Hostname field for each Listener in order + of most to least specific. That is, exact matches must be processed + before wildcard matches. \n If this field specifies multiple Listeners + that have the same Port value but are not compatible, the implementation + must raise a \"Conflicted\" condition in the Listener status. \n + Support: Core" items: description: Listener embodies the concept of a logical endpoint where a Gateway accepts network connections. @@ -582,19 +621,18 @@ spec: determined in order of the following criteria: \n * The most specific match as defined by the Route type. * The oldest Route based on creation timestamp. For example, a Route with - \ a creation timestamp of \"2020-09-08 01:02:03\" is given - precedence over a Route with a creation timestamp of \"2020-09-08 - 01:02:04\". * If everything else is equivalent, the Route - appearing first in alphabetical order (namespace/name) should - be given precedence. For example, foo/bar is given precedence - over foo/baz. \n All valid rules within a Route attached to - this Listener should be implemented. Invalid Route rules can - be ignored (sometimes that will mean the full Route). If a - Route rule transitions from valid to invalid, support for - that Route rule should be dropped to ensure consistency. For - example, even if a filter specified by a Route rule is invalid, - the rest of the rules within that Route should still be supported. - \n Support: Core" + a creation timestamp of \"2020-09-08 01:02:03\" is given precedence + over a Route with a creation timestamp of \"2020-09-08 01:02:04\". + * If everything else is equivalent, the Route appearing first + in alphabetical order (namespace/name) should be given precedence. + For example, foo/bar is given precedence over foo/baz. \n + All valid rules within a Route attached to this Listener should + be implemented. Invalid Route rules can be ignored (sometimes + that will mean the full Route). If a Route rule transitions + from valid to invalid, support for that Route rule should + be dropped to ensure consistency. For example, even if a filter + specified by a Route rule is invalid, the rest of the rules + within that Route should still be supported. \n Support: Core" properties: kinds: description: "Kinds specifies the groups and kinds of Routes @@ -639,12 +677,12 @@ spec: from: default: Same description: "From indicates where Routes will be selected - for this Gateway. Possible values are: * All: Routes + for this Gateway. Possible values are: \n * All: Routes in all namespaces may be used by this Gateway. * Selector: Routes in namespaces selected by the selector may - be used by this Gateway. * Same: Only Routes in - the same namespace may be used by this Gateway. \n - Support: Core" + be used by this Gateway. * Same: Only Routes in the + same namespace may be used by this Gateway. \n Support: + Core" enum: - All - Selector @@ -700,6 +738,7 @@ spec: requirements are ANDed. type: object type: object + x-kubernetes-map-type: atomic type: object type: object hostname: @@ -711,19 +750,18 @@ spec: following protocols: \n * TLS: The Listener Hostname MUST match the SNI. * HTTP: The Listener Hostname MUST match the Host header of the request. * HTTPS: The Listener Hostname - SHOULD match at both the TLS and HTTP protocol layers as - described above. If an implementation does not ensure that - both the SNI and Host header match the Listener hostname, - \ it MUST clearly document that. \n For HTTPRoute and TLSRoute - resources, there is an interaction with the `spec.hostnames` - array. When both listener and route specify hostnames, there - MUST be an intersection between the values for a Route to - be accepted. For more information, refer to the Route specific - Hostnames documentation. \n Hostnames that are prefixed with - a wildcard label (`*.`) are interpreted as a suffix match. - That means that a match for `*.example.com` would match both - `test.example.com`, and `foo.test.example.com`, but not `example.com`. - \n Support: Core" + SHOULD match at both the TLS and HTTP protocol layers as described + above. If an implementation does not ensure that both the + SNI and Host header match the Listener hostname, it MUST clearly + document that. \n For HTTPRoute and TLSRoute resources, there + is an interaction with the `spec.hostnames` array. When both + listener and route specify hostnames, there MUST be an intersection + between the values for a Route to be accepted. For more information, + refer to the Route specific Hostnames documentation. \n Hostnames + that are prefixed with a wildcard label (`*.`) are interpreted + as a suffix match. That means that a match for `*.example.com` + would match both `test.example.com`, and `foo.test.example.com`, + but not `example.com`. \n Support: Core" maxLength: 253 minLength: 1 pattern: ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ @@ -804,7 +842,7 @@ spec: kind: default: Secret description: Kind is kind of the referent. For example - "HTTPRoute" or "Service". + "Secret". maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -817,11 +855,12 @@ spec: namespace: description: "Namespace is the namespace of the backend. When unspecified, the local namespace is inferred. - \n Note that when a namespace is specified, a ReferenceGrant - object is required in the referent namespace to - allow that namespace's owner to accept the reference. - See the ReferenceGrant documentation for details. - \n Support: Core" + \n Note that when a namespace different than the + local namespace is specified, a ReferenceGrant object + is required in the referent namespace to allow that + namespace's owner to accept the reference. See the + ReferenceGrant documentation for details. \n Support: + Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -836,13 +875,13 @@ spec: description: "Mode defines the TLS behavior for the TLS session initiated by the client. There are two possible modes: \n - Terminate: The TLS session between the downstream - client and the Gateway is terminated at the Gateway. - This mode requires certificateRefs to be set and contain - at least one element. - Passthrough: The TLS session is - NOT terminated by the Gateway. This implies that the - Gateway can't decipher the TLS stream except for the - ClientHello message of the TLS protocol. CertificateRefs - field is ignored in this mode. \n Support: Core" + client and the Gateway is terminated at the Gateway. This + mode requires certificateRefs to be set and contain at + least one element. - Passthrough: The TLS session is NOT + terminated by the Gateway. This implies that the Gateway + can't decipher the TLS stream except for the ClientHello + message of the TLS protocol. CertificateRefs field is + ignored in this mode. \n Support: Core" enum: - Terminate - Passthrough @@ -869,6 +908,11 @@ spec: maxProperties: 16 type: object type: object + x-kubernetes-validations: + - message: certificateRefs must be specified when TLSModeType + is Terminate + rule: 'self.mode == ''Terminate'' ? size(self.certificateRefs) + > 0 : true' required: - name - port @@ -880,6 +924,24 @@ spec: x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map + x-kubernetes-validations: + - message: tls must be specified for protocols ['HTTPS', 'TLS'] + rule: 'self.all(l, l.protocol in [''HTTPS'', ''TLS''] ? has(l.tls) + : true)' + - message: tls must not be specified for protocols ['HTTP', 'TCP', + 'UDP'] + rule: 'self.all(l, l.protocol in [''HTTP'', ''TCP'', ''UDP''] ? + !has(l.tls) : true)' + - message: hostname must not be specified for protocols ['TCP', 'UDP'] + rule: 'self.all(l, l.protocol in [''TCP'', ''UDP''] ? (!has(l.hostname) + || l.hostname == '''') : true)' + - message: Listener name must be unique within the Gateway + rule: self.all(l1, self.exists_one(l2, l1.name == l2.name)) + - message: Combination of port, protocol and hostname must be unique + for each listener + rule: 'self.all(l1, self.exists_one(l2, l1.port == l2.port && l1.protocol + == l2.protocol && (has(l1.hostname) && has(l2.hostname) ? l1.hostname + == l2.hostname : true)))' required: - gatewayClassName - listeners @@ -889,19 +951,38 @@ spec: conditions: - lastTransitionTime: "1970-01-01T00:00:00Z" message: Waiting for controller - reason: NotReconciled + reason: Pending status: Unknown type: Accepted + - lastTransitionTime: "1970-01-01T00:00:00Z" + message: Waiting for controller + reason: Pending + status: Unknown + type: Programmed description: Status defines the current state of Gateway. properties: addresses: - description: Addresses lists the IP addresses that have actually been - bound to the Gateway. These addresses may differ from the addresses + description: "Addresses lists the IP addresses that have actually + been bound to the Gateway. These addresses may differ from the addresses in the Spec, e.g. if the Gateway automatically assigns an address - from a reserved pool. + from a reserved pool. \n " items: - description: GatewayAddress describes an address that can be bound + description: GatewayStatusAddress describes an address that is bound to a Gateway. + oneOf: + - properties: + type: + enum: + - IPAddress + value: + anyOf: + - format: ipv4 + - format: ipv6 + - properties: + type: + not: + enum: + - IPAddress properties: type: default: IPAddress @@ -920,6 +1001,11 @@ spec: required: - value type: object + x-kubernetes-validations: + - message: Hostname value must only contain valid characters (matching + ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$) + rule: 'self.type == ''Hostname'' ? self.value.matches(r"""^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"""): + true' maxItems: 16 type: array conditions: @@ -939,19 +1025,17 @@ spec: the `GatewayConditionType` and `GatewayConditionReason` constants so that operators and tools can converge on a common vocabulary to describe Gateway state. \n Known condition types are: \n * \"Accepted\" - * \"Ready\"" + * \"Programmed\" * \"Ready\"" items: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions. For example, - \n \ttype FooStatus struct{ \t // Represents the observations - of a foo's current state. \t // Known .status.conditions.type - are: \"Available\", \"Progressing\", and \"Degraded\" \t // - +patchMergeKey=type \t // +patchStrategy=merge \t // +listType=map - \t // +listMapKey=type \t Conditions []metav1.Condition + \n type FooStatus struct{ // Represents the observations of a + foo's current state. // Known .status.conditions.type are: \"Available\", + \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other fields - \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -1020,8 +1104,8 @@ spec: description: ListenerStatus is the status associated with a Listener. properties: attachedRoutes: - description: AttachedRoutes represents the total number of Routes - that have been successfully attached to this Listener. + description: AttachedRoutes represents the total number of accepted + Routes that have been successfully attached to this Listener. format: int32 type: integer conditions: @@ -1031,15 +1115,14 @@ spec: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path - .status.conditions. For example, \n \ttype FooStatus struct{ - \t // Represents the observations of a foo's current - state. \t // Known .status.conditions.type are: \"Available\", - \"Progressing\", and \"Degraded\" \t // +patchMergeKey=type - \t // +patchStrategy=merge \t // +listType=map \t - \ // +listMapKey=type \t Conditions []metav1.Condition + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other - fields \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -1211,10 +1294,24 @@ spec: manner, assigning an appropriate set of Addresses. \n The implementation MUST bind all Listeners to every GatewayAddress that it assigns to the Gateway and add a corresponding entry in GatewayStatus.Addresses. - \n Support: Extended" + \n Support: Extended \n " items: description: GatewayAddress describes an address that can be bound to a Gateway. + oneOf: + - properties: + type: + enum: + - IPAddress + value: + anyOf: + - format: ipv4 + - format: ipv6 + - properties: + type: + not: + enum: + - IPAddress properties: type: default: IPAddress @@ -1233,8 +1330,20 @@ spec: required: - value type: object + x-kubernetes-validations: + - message: Hostname value must only contain valid characters (matching + ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$) + rule: 'self.type == ''Hostname'' ? self.value.matches(r"""^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"""): + true' maxItems: 16 type: array + x-kubernetes-validations: + - message: IPAddress values must be unique + rule: 'self.all(a1, a1.type == ''IPAddress'' ? self.exists_one(a2, + a2.type == a1.type && a2.value == a1.value) : true )' + - message: Hostname values must be unique + rule: 'self.all(a1, a1.type == ''Hostname'' ? self.exists_one(a2, + a2.type == a1.type && a2.value == a1.value) : true )' gatewayClassName: description: GatewayClassName used for this Gateway. This is the name of a GatewayClass resource. @@ -1246,27 +1355,34 @@ spec: logical endpoints that are bound on this Gateway's addresses. At least one Listener MUST be specified. \n Each listener in a Gateway must have a unique combination of Hostname, Port, and Protocol. - \n An implementation MAY group Listeners by Port and then collapse - each group of Listeners into a single Listener if the implementation - determines that the Listeners in the group are \"compatible\". An - implementation MAY also group together and collapse compatible Listeners - belonging to different Gateways. \n For example, an implementation - might consider Listeners to be compatible with each other if all - of the following conditions are met: \n 1. Either each Listener - within the group specifies the \"HTTP\" Protocol or each Listener - within the group specifies either the \"HTTPS\" or \"TLS\" Protocol. - \n 2. Each Listener within the group specifies a Hostname that is - unique within the group. \n 3. As a special case, one Listener - within a group may omit Hostname, in which case this Listener - matches when no other Listener matches. \n If the implementation - does collapse compatible Listeners, the hostname provided in the - incoming client request MUST be matched to a Listener to find the - correct set of Routes. The incoming hostname MUST be matched using - the Hostname field for each Listener in order of most to least specific. - That is, exact matches must be processed before wildcard matches. - \n If this field specifies multiple Listeners that have the same - Port value but are not compatible, the implementation must raise - a \"Conflicted\" condition in the Listener status. \n Support: Core" + \n Within the HTTP Conformance Profile, the below combinations of + port and protocol are considered Core and MUST be supported: \n + 1. Port: 80, Protocol: HTTP 2. Port: 443, Protocol: HTTPS \n Within + the TLS Conformance Profile, the below combinations of port and + protocol are considered Core and MUST be supported: \n 1. Port: + 443, Protocol: TLS \n Port and protocol combinations not listed + above are considered Extended. \n An implementation MAY group Listeners + by Port and then collapse each group of Listeners into a single + Listener if the implementation determines that the Listeners in + the group are \"compatible\". An implementation MAY also group together + and collapse compatible Listeners belonging to different Gateways. + \n For example, an implementation might consider Listeners to be + compatible with each other if all of the following conditions are + met: \n 1. Either each Listener within the group specifies the \"HTTP\" + Protocol or each Listener within the group specifies either the + \"HTTPS\" or \"TLS\" Protocol. \n 2. Each Listener within the group + specifies a Hostname that is unique within the group. \n 3. As a + special case, one Listener within a group may omit Hostname, in + which case this Listener matches when no other Listener matches. + \n If the implementation does collapse compatible Listeners, the + hostname provided in the incoming client request MUST be matched + to a Listener to find the correct set of Routes. The incoming hostname + MUST be matched using the Hostname field for each Listener in order + of most to least specific. That is, exact matches must be processed + before wildcard matches. \n If this field specifies multiple Listeners + that have the same Port value but are not compatible, the implementation + must raise a \"Conflicted\" condition in the Listener status. \n + Support: Core" items: description: Listener embodies the concept of a logical endpoint where a Gateway accepts network connections. @@ -1283,19 +1399,18 @@ spec: determined in order of the following criteria: \n * The most specific match as defined by the Route type. * The oldest Route based on creation timestamp. For example, a Route with - \ a creation timestamp of \"2020-09-08 01:02:03\" is given - precedence over a Route with a creation timestamp of \"2020-09-08 - 01:02:04\". * If everything else is equivalent, the Route - appearing first in alphabetical order (namespace/name) should - be given precedence. For example, foo/bar is given precedence - over foo/baz. \n All valid rules within a Route attached to - this Listener should be implemented. Invalid Route rules can - be ignored (sometimes that will mean the full Route). If a - Route rule transitions from valid to invalid, support for - that Route rule should be dropped to ensure consistency. For - example, even if a filter specified by a Route rule is invalid, - the rest of the rules within that Route should still be supported. - \n Support: Core" + a creation timestamp of \"2020-09-08 01:02:03\" is given precedence + over a Route with a creation timestamp of \"2020-09-08 01:02:04\". + * If everything else is equivalent, the Route appearing first + in alphabetical order (namespace/name) should be given precedence. + For example, foo/bar is given precedence over foo/baz. \n + All valid rules within a Route attached to this Listener should + be implemented. Invalid Route rules can be ignored (sometimes + that will mean the full Route). If a Route rule transitions + from valid to invalid, support for that Route rule should + be dropped to ensure consistency. For example, even if a filter + specified by a Route rule is invalid, the rest of the rules + within that Route should still be supported. \n Support: Core" properties: kinds: description: "Kinds specifies the groups and kinds of Routes @@ -1340,12 +1455,12 @@ spec: from: default: Same description: "From indicates where Routes will be selected - for this Gateway. Possible values are: * All: Routes + for this Gateway. Possible values are: \n * All: Routes in all namespaces may be used by this Gateway. * Selector: Routes in namespaces selected by the selector may - be used by this Gateway. * Same: Only Routes in - the same namespace may be used by this Gateway. \n - Support: Core" + be used by this Gateway. * Same: Only Routes in the + same namespace may be used by this Gateway. \n Support: + Core" enum: - All - Selector @@ -1401,6 +1516,7 @@ spec: requirements are ANDed. type: object type: object + x-kubernetes-map-type: atomic type: object type: object hostname: @@ -1412,19 +1528,18 @@ spec: following protocols: \n * TLS: The Listener Hostname MUST match the SNI. * HTTP: The Listener Hostname MUST match the Host header of the request. * HTTPS: The Listener Hostname - SHOULD match at both the TLS and HTTP protocol layers as - described above. If an implementation does not ensure that - both the SNI and Host header match the Listener hostname, - \ it MUST clearly document that. \n For HTTPRoute and TLSRoute - resources, there is an interaction with the `spec.hostnames` - array. When both listener and route specify hostnames, there - MUST be an intersection between the values for a Route to - be accepted. For more information, refer to the Route specific - Hostnames documentation. \n Hostnames that are prefixed with - a wildcard label (`*.`) are interpreted as a suffix match. - That means that a match for `*.example.com` would match both - `test.example.com`, and `foo.test.example.com`, but not `example.com`. - \n Support: Core" + SHOULD match at both the TLS and HTTP protocol layers as described + above. If an implementation does not ensure that both the + SNI and Host header match the Listener hostname, it MUST clearly + document that. \n For HTTPRoute and TLSRoute resources, there + is an interaction with the `spec.hostnames` array. When both + listener and route specify hostnames, there MUST be an intersection + between the values for a Route to be accepted. For more information, + refer to the Route specific Hostnames documentation. \n Hostnames + that are prefixed with a wildcard label (`*.`) are interpreted + as a suffix match. That means that a match for `*.example.com` + would match both `test.example.com`, and `foo.test.example.com`, + but not `example.com`. \n Support: Core" maxLength: 253 minLength: 1 pattern: ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ @@ -1505,7 +1620,7 @@ spec: kind: default: Secret description: Kind is kind of the referent. For example - "HTTPRoute" or "Service". + "Secret". maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -1518,11 +1633,12 @@ spec: namespace: description: "Namespace is the namespace of the backend. When unspecified, the local namespace is inferred. - \n Note that when a namespace is specified, a ReferenceGrant - object is required in the referent namespace to - allow that namespace's owner to accept the reference. - See the ReferenceGrant documentation for details. - \n Support: Core" + \n Note that when a namespace different than the + local namespace is specified, a ReferenceGrant object + is required in the referent namespace to allow that + namespace's owner to accept the reference. See the + ReferenceGrant documentation for details. \n Support: + Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -1537,13 +1653,13 @@ spec: description: "Mode defines the TLS behavior for the TLS session initiated by the client. There are two possible modes: \n - Terminate: The TLS session between the downstream - client and the Gateway is terminated at the Gateway. - This mode requires certificateRefs to be set and contain - at least one element. - Passthrough: The TLS session is - NOT terminated by the Gateway. This implies that the - Gateway can't decipher the TLS stream except for the - ClientHello message of the TLS protocol. CertificateRefs - field is ignored in this mode. \n Support: Core" + client and the Gateway is terminated at the Gateway. This + mode requires certificateRefs to be set and contain at + least one element. - Passthrough: The TLS session is NOT + terminated by the Gateway. This implies that the Gateway + can't decipher the TLS stream except for the ClientHello + message of the TLS protocol. CertificateRefs field is + ignored in this mode. \n Support: Core" enum: - Terminate - Passthrough @@ -1570,6 +1686,11 @@ spec: maxProperties: 16 type: object type: object + x-kubernetes-validations: + - message: certificateRefs must be specified when TLSModeType + is Terminate + rule: 'self.mode == ''Terminate'' ? size(self.certificateRefs) + > 0 : true' required: - name - port @@ -1581,6 +1702,24 @@ spec: x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map + x-kubernetes-validations: + - message: tls must be specified for protocols ['HTTPS', 'TLS'] + rule: 'self.all(l, l.protocol in [''HTTPS'', ''TLS''] ? has(l.tls) + : true)' + - message: tls must not be specified for protocols ['HTTP', 'TCP', + 'UDP'] + rule: 'self.all(l, l.protocol in [''HTTP'', ''TCP'', ''UDP''] ? + !has(l.tls) : true)' + - message: hostname must not be specified for protocols ['TCP', 'UDP'] + rule: 'self.all(l, l.protocol in [''TCP'', ''UDP''] ? (!has(l.hostname) + || l.hostname == '''') : true)' + - message: Listener name must be unique within the Gateway + rule: self.all(l1, self.exists_one(l2, l1.name == l2.name)) + - message: Combination of port, protocol and hostname must be unique + for each listener + rule: 'self.all(l1, self.exists_one(l2, l1.port == l2.port && l1.protocol + == l2.protocol && (has(l1.hostname) && has(l2.hostname) ? l1.hostname + == l2.hostname : true)))' required: - gatewayClassName - listeners @@ -1590,19 +1729,38 @@ spec: conditions: - lastTransitionTime: "1970-01-01T00:00:00Z" message: Waiting for controller - reason: NotReconciled + reason: Pending status: Unknown type: Accepted + - lastTransitionTime: "1970-01-01T00:00:00Z" + message: Waiting for controller + reason: Pending + status: Unknown + type: Programmed description: Status defines the current state of Gateway. properties: addresses: - description: Addresses lists the IP addresses that have actually been - bound to the Gateway. These addresses may differ from the addresses + description: "Addresses lists the IP addresses that have actually + been bound to the Gateway. These addresses may differ from the addresses in the Spec, e.g. if the Gateway automatically assigns an address - from a reserved pool. + from a reserved pool. \n " items: - description: GatewayAddress describes an address that can be bound + description: GatewayStatusAddress describes an address that is bound to a Gateway. + oneOf: + - properties: + type: + enum: + - IPAddress + value: + anyOf: + - format: ipv4 + - format: ipv6 + - properties: + type: + not: + enum: + - IPAddress properties: type: default: IPAddress @@ -1621,6 +1779,11 @@ spec: required: - value type: object + x-kubernetes-validations: + - message: Hostname value must only contain valid characters (matching + ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$) + rule: 'self.type == ''Hostname'' ? self.value.matches(r"""^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"""): + true' maxItems: 16 type: array conditions: @@ -1640,19 +1803,17 @@ spec: the `GatewayConditionType` and `GatewayConditionReason` constants so that operators and tools can converge on a common vocabulary to describe Gateway state. \n Known condition types are: \n * \"Accepted\" - * \"Ready\"" + * \"Programmed\" * \"Ready\"" items: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions. For example, - \n \ttype FooStatus struct{ \t // Represents the observations - of a foo's current state. \t // Known .status.conditions.type - are: \"Available\", \"Progressing\", and \"Degraded\" \t // - +patchMergeKey=type \t // +patchStrategy=merge \t // +listType=map - \t // +listMapKey=type \t Conditions []metav1.Condition + \n type FooStatus struct{ // Represents the observations of a + foo's current state. // Known .status.conditions.type are: \"Available\", + \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other fields - \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -1721,8 +1882,8 @@ spec: description: ListenerStatus is the status associated with a Listener. properties: attachedRoutes: - description: AttachedRoutes represents the total number of Routes - that have been successfully attached to this Listener. + description: AttachedRoutes represents the total number of accepted + Routes that have been successfully attached to this Listener. format: int32 type: integer conditions: @@ -1732,15 +1893,14 @@ spec: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path - .status.conditions. For example, \n \ttype FooStatus struct{ - \t // Represents the observations of a foo's current - state. \t // Known .status.conditions.type are: \"Available\", - \"Progressing\", and \"Degraded\" \t // +patchMergeKey=type - \t // +patchStrategy=merge \t // +listType=map \t - \ // +listMapKey=type \t Conditions []metav1.Condition + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other - fields \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -1866,39 +2026,63 @@ status: acceptedNames: kind: "" plural: "" - conditions: [] - storedVersions: [] + conditions: null + storedVersions: null --- +# +# config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml +# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/1538 - gateway.networking.k8s.io/bundle-version: v0.6.2 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null - name: udproutes.gateway.networking.k8s.io + name: grpcroutes.gateway.networking.k8s.io spec: group: gateway.networking.k8s.io names: categories: - gateway-api - kind: UDPRoute - listKind: UDPRouteList - plural: udproutes - singular: udproute + kind: GRPCRoute + listKind: GRPCRouteList + plural: grpcroutes + singular: grpcroute scope: Namespaced versions: - additionalPrinterColumns: + - jsonPath: .spec.hostnames + name: Hostnames + type: string - jsonPath: .metadata.creationTimestamp name: Age type: date name: v1alpha2 schema: openAPIV3Schema: - description: UDPRoute provides a way to route UDP traffic. When combined with - a Gateway listener, it can be used to forward traffic on the port specified - by the listener to a set of backends specified by the UDPRoute. + description: "GRPCRoute provides a way to route gRPC requests. This includes + the capability to match requests by hostname, gRPC service, gRPC method, + or HTTP/2 header. Filters can be used to specify additional processing steps. + Backends specify where matching requests will be routed. \n GRPCRoute falls + under extended support within the Gateway API. Within the following specification, + the word \"MUST\" indicates that an implementation supporting GRPCRoute + must conform to the indicated requirement, but an implementation not supporting + this route type need not follow the requirement unless explicitly indicated. + \n Implementations supporting `GRPCRoute` with the `HTTPS` `ProtocolType` + MUST accept HTTP/2 connections without an initial upgrade from HTTP/1.1, + i.e. via ALPN. If the implementation does not support this, then it MUST + set the \"Accepted\" condition to \"False\" for the affected listener with + a reason of \"UnsupportedProtocol\". Implementations MAY also accept HTTP/2 + connections with an upgrade from HTTP/1. \n Implementations supporting `GRPCRoute` + with the `HTTP` `ProtocolType` MUST support HTTP/2 over cleartext TCP (h2c, + https://www.rfc-editor.org/rfc/rfc7540#section-3.1) without an initial upgrade + from HTTP/1.1, i.e. with prior knowledge (https://www.rfc-editor.org/rfc/rfc7540#section-3.4). + If the implementation does not support this, then it MUST set the \"Accepted\" + condition to \"False\" for the affected listener with a reason of \"UnsupportedProtocol\". + Implementations MAY also accept HTTP/2 connections with an upgrade from + HTTP/1, i.e. without prior knowledge." properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation @@ -1913,20 +2097,85 @@ spec: metadata: type: object spec: - description: Spec defines the desired state of UDPRoute. + description: Spec defines the desired state of GRPCRoute. properties: + hostnames: + description: "Hostnames defines a set of hostnames to match against + the GRPC Host header to select a GRPCRoute to process the request. + This matches the RFC 1123 definition of a hostname with 2 notable + exceptions: \n 1. IPs are not allowed. 2. A hostname may be prefixed + with a wildcard label (`*.`). The wildcard label MUST appear by + itself as the first label. \n If a hostname is specified by both + the Listener and GRPCRoute, there MUST be at least one intersecting + hostname for the GRPCRoute to be attached to the Listener. For example: + \n * A Listener with `test.example.com` as the hostname matches + GRPCRoutes that have either not specified any hostnames, or have + specified at least one of `test.example.com` or `*.example.com`. + * A Listener with `*.example.com` as the hostname matches GRPCRoutes + that have either not specified any hostnames or have specified at + least one hostname that matches the Listener hostname. For example, + `test.example.com` and `*.example.com` would both match. On the + other hand, `example.com` and `test.example.net` would not match. + \n Hostnames that are prefixed with a wildcard label (`*.`) are + interpreted as a suffix match. That means that a match for `*.example.com` + would match both `test.example.com`, and `foo.test.example.com`, + but not `example.com`. \n If both the Listener and GRPCRoute have + specified hostnames, any GRPCRoute hostnames that do not match the + Listener hostname MUST be ignored. For example, if a Listener specified + `*.example.com`, and the GRPCRoute specified `test.example.com` + and `test.example.net`, `test.example.net` MUST NOT be considered + for a match. \n If both the Listener and GRPCRoute have specified + hostnames, and none match with the criteria above, then the GRPCRoute + MUST NOT be accepted by the implementation. The implementation MUST + raise an 'Accepted' Condition with a status of `False` in the corresponding + RouteParentStatus. \n If a Route (A) of type HTTPRoute or GRPCRoute + is attached to a Listener and that listener already has another + Route (B) of the other type attached and the intersection of the + hostnames of A and B is non-empty, then the implementation MUST + accept exactly one of these two routes, determined by the following + criteria, in order: \n * The oldest Route based on creation timestamp. + * The Route appearing first in alphabetical order by \"{namespace}/{name}\". + \n The rejected Route MUST raise an 'Accepted' condition with a + status of 'False' in the corresponding RouteParentStatus. \n Support: + Core" + items: + description: "Hostname is the fully qualified domain name of a network + host. This matches the RFC 1123 definition of a hostname with + 2 notable exceptions: \n 1. IPs are not allowed. 2. A hostname + may be prefixed with a wildcard label (`*.`). The wildcard label + must appear by itself as the first label. \n Hostname can be \"precise\" + which is a domain name without the terminating dot of a network + host (e.g. \"foo.example.com\") or \"wildcard\", which is a domain + name prefixed with a single wildcard label (e.g. `*.example.com`). + \n Note that as per RFC1035 and RFC1123, a *label* must consist + of lower case alphanumeric characters or '-', and must start and + end with an alphanumeric character. No other punctuation is allowed." + maxLength: 253 + minLength: 1 + pattern: ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + maxItems: 16 + type: array parentRefs: description: "ParentRefs references the resources (usually Gateways) that a Route wants to be attached to. Note that the referenced parent resource needs to allow this for the attachment to be complete. For Gateways, that means the Gateway needs to allow attachment from - Routes of this kind and namespace. \n The only kind of parent resource - with \"Core\" support is Gateway. This API may be extended in the - future to support additional kinds of parent resources such as one - of the route kinds. \n It is invalid to reference an identical parent + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) \n This + API may be extended in the future to support additional kinds of + parent resources. \n It is invalid to reference an identical parent more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as 2 Listeners within a Gateway. - \n It is possible to separately reference multiple distinct objects + within the same parent resource, such as two separate Listeners + on the same Gateway or two separate ports on the same Service. \n + It is possible to separately reference multiple distinct objects that may be collapsed by an implementation. For example, some implementations may choose to merge compatible Gateway Listeners together. If that is the case, the list of routes attached to those resources should @@ -1934,16 +2183,25 @@ spec: boundaries, there are specific rules. Cross-namespace references are only valid if they are explicitly allowed by something in the namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable any other - kind of cross-namespace reference." + field, and ReferenceGrant provides a generic way to enable other + kinds of cross-namespace reference. \n ParentRefs from a Route to + a Service in the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any namespace + to the Service. \n ParentRefs from a Route to a Service in a different + namespace are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same namespace + as the Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually - a route). The only kind of parent resource with \"Core\" support - is Gateway. This API may be extended in the future to support - additional kinds of parent resources, such as HTTPRoute. \n The - API object must be valid in the cluster; the Group and Kind must - be registered in the cluster for this reference to be valid." + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." properties: group: default: gateway.networking.k8s.io @@ -1957,8 +2215,11 @@ spec: type: string kind: default: Gateway - description: "Kind is kind of the referent. \n Support: Core - (Gateway) \n Support: Implementation-specific (Other Resources)" + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -1978,7 +2239,15 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n Support: Core" + \n ParentRefs from a Route to a Service in the same namespace + are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. + \n ParentRefs from a Route to a Service in a different namespace + are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same + namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -1993,18 +2262,22 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n Implementations MAY choose - to support other parent resources. Implementations supporting - other types of parent resources MUST clearly document how/if - Port is interpreted. \n For the purpose of status, an attachment - is considered successful as long as the parent resource accepts - it partially. For example, Gateway listeners can restrict - which Routes can attach to them by Route kind, namespace, - or hostname. If 1 of 2 Gateway listeners accept attachment - from the referencing Route, the Route MUST be considered successfully - attached. If no Gateway listeners accept attachment from this - Route, the Route MUST be considered detached from the Gateway. - \n Support: Extended \n " + must match both specified values. \n When the parent resource + is a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified + values. \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes can + attach to them by Route kind, namespace, or hostname. If 1 + of 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. \n + Support: Extended \n " format: int32 maximum: 65535 minimum: 1 @@ -2015,19 +2288,23 @@ spec: interpreted as the following: \n * Gateway: Listener Name. When both Port (experimental) and SectionName are specified, the name and port of the selected listener must match both - specified values. \n Implementations MAY choose to support - attaching Routes to other resources. If that is the case, - they MUST clearly document how SectionName is interpreted. - \n When unspecified (empty string), this will reference the - entire resource. For the purpose of status, an attachment - is considered successful if at least one section in the parent - resource accepts it. For example, Gateway listeners can restrict - which Routes can attach to them by Route kind, namespace, - or hostname. If 1 of 2 Gateway listeners accept attachment - from the referencing Route, the Route MUST be considered successfully - attached. If no Gateway listeners accept attachment from this - Route, the Route MUST be considered detached from the Gateway. - \n Support: Core" + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" maxLength: 253 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ @@ -2037,29 +2314,511 @@ spec: type: object maxItems: 32 type: array + x-kubernetes-validations: + - message: sectionName or port must be specified when parentRefs includes + 2 or more references to the same parent + rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ + == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) + && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) + || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName + != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) + && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) + ) ) ): true ))' + - message: sectionName or port must be unique when parentRefs includes + 2 or more references to the same parent + rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) + || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ + == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) + || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName + == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName + == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port + == p2.port)))) rules: - description: Rules are a list of UDP matchers and actions. + description: Rules are a list of GRPC matchers, filters and actions. items: - description: UDPRouteRule is the configuration for a given rule. + description: GRPCRouteRule defines the semantics for matching a + gRPC request based on conditions (matches), processing it (filters), + and forwarding the request to an API object (backendRefs). properties: backendRefs: description: "BackendRefs defines the backend(s) where matching - requests should be sent. If unspecified or invalid (refers - to a non-existent resource or a Service with no endpoints), - the underlying implementation MUST actively reject connection - attempts to this backend. Packet drops must respect weight; - if an invalid backend is requested to have 80% of the packets, - then 80% of packets must be dropped instead. \n Support: Core - for Kubernetes Service Support: Implementation-specific for - any other resource \n Support for weight: Extended" + requests should be sent. \n Failure behavior here depends + on how many BackendRefs are specified and how many are invalid. + \n If *all* entries in BackendRefs are invalid, and there + are also no filters specified in this route rule, *all* traffic + which matches this rule MUST receive an `UNAVAILABLE` status. + \n See the GRPCBackendRef definition for the rules about what + makes a single GRPCBackendRef invalid. \n When a GRPCBackendRef + is invalid, `UNAVAILABLE` statuses MUST be returned for requests + that would have otherwise been routed to an invalid backend. + If multiple backends are specified, and some are invalid, + the proportion of requests that would otherwise have been + routed to an invalid backend MUST receive an `UNAVAILABLE` + status. \n For example, if two backends are specified with + equal weights, and one is invalid, 50 percent of traffic MUST + receive an `UNAVAILABLE` status. Implementations may choose + how that 50 percent is determined. \n Support: Core for Kubernetes + Service \n Support: Implementation-specific for any other + resource \n Support for weight: Core" items: - description: "BackendRef defines how a Route should forward - a request to a Kubernetes resource. \n Note that when a - namespace is specified, a ReferenceGrant object is required - in the referent namespace to allow that namespace's owner - to accept the reference. See the ReferenceGrant documentation - for details." + description: GRPCBackendRef defines how a GRPCRoute forwards + a gRPC request. properties: + filters: + description: "Filters defined at this level MUST be executed + if and only if the request is being forwarded to the + backend defined here. \n Support: Implementation-specific + (For broader support of filters, use the Filters field + in GRPCRouteRule.)" + items: + description: GRPCRouteFilter defines processing steps + that must be completed during the request or response + lifecycle. GRPCRouteFilters are meant as an extension + point to express processing that may be done in Gateway + implementations. Some examples include request or + response modification, implementing authentication + strategies, rate-limiting, and traffic shaping. API + guarantee/conformance is defined based on the type + of the filter. + properties: + extensionRef: + description: "ExtensionRef is an optional, implementation-specific + extension to the \"filter\" behavior. For example, + resource \"myroutefilter\" in group \"networking.example.net\"). + ExtensionRef MUST NOT be used for core and extended + filters. \n Support: Implementation-specific \n + This filter can be used multiple times within + the same rule." + properties: + group: + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API + group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For + example "HTTPRoute" or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + requestHeaderModifier: + description: "RequestHeaderModifier defines a schema + for a filter that modifies request headers. \n + Support: Core" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It + appends to any existing values associated + with the header name. \n Input: GET /foo HTTP/1.1 + my-header: foo \n Config: add: - name: \"my-header\" + value: \"bar,baz\" \n Output: GET /foo HTTP/1.1 + my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from + the HTTP request before the action. The value + of Remove is a list of HTTP header names. + Note that the header names are case-insensitive + (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo + my-header2: bar my-header3: baz \n Config: + remove: [\"my-header1\", \"my-header3\"] \n + Output: GET /foo HTTP/1.1 my-header2: bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with + the given header (name, value) before the + action. \n Input: GET /foo HTTP/1.1 my-header: + foo \n Config: set: - name: \"my-header\" + value: \"bar\" \n Output: GET /foo HTTP/1.1 + my-header: bar" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + requestMirror: + description: "RequestMirror defines a schema for + a filter that mirrors requests. Requests are sent + to the specified destination, but responses from + that destination are ignored. \n This filter can + be used multiple times within the same rule. Note + that not all implementations will be able to support + mirroring to multiple backends. \n Support: Extended" + properties: + backendRef: + description: "BackendRef references a resource + where mirrored requests are sent. \n Mirrored + requests must be sent only to a single destination + endpoint within this BackendRef, irrespective + of how many endpoints are present within this + BackendRef. \n If the referent cannot be found, + this BackendRef is invalid and must be dropped + from the Gateway. The controller must ensure + the \"ResolvedRefs\" condition on the Route + status is set to `status: False` and not configure + this backend in the underlying implementation. + \n If there is a cross-namespace reference + to an *existing* object that is not allowed + by a ReferenceGrant, the controller must ensure + the \"ResolvedRefs\" condition on the Route + is set to `status: False`, with the \"RefNotPermitted\" + reason and not configure this backend in the + underlying implementation. \n In either error + case, the Message of the `ResolvedRefs` Condition + should be used to provide more detail about + the problem. \n Support: Extended for Kubernetes + Service \n Support: Implementation-specific + for any other resource" + properties: + group: + default: "" + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". + When unspecified or empty string, core + API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource + kind of the referent. For example \"Service\". + \n Defaults to \"Service\" when not specified. + \n ExternalName services can refer to + CNAME DNS records that may live outside + of the cluster and as such are difficult + to reason about in terms of conformance. + They also may not be safe to forward to + (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName + Services. \n Support: Core (Services with + a type other than ExternalName) \n Support: + Implementation-specific (Services with + type ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace + of the backend. When unspecified, the + local namespace is inferred. \n Note that + when a namespace different than the local + namespace is specified, a ReferenceGrant + object is required in the referent namespace + to allow that namespace's owner to accept + the reference. See the ReferenceGrant + documentation for details. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination + port number to use for this resource. + Port is required when the referent is + a Kubernetes Service. In this case, the + port number is the service port number, + not the target port. For other resources, + destination port might be derived from + the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind + == ''Service'') ? has(self.port) : true' + required: + - backendRef + type: object + responseHeaderModifier: + description: "ResponseHeaderModifier defines a schema + for a filter that modifies response headers. \n + Support: Extended" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It + appends to any existing values associated + with the header name. \n Input: GET /foo HTTP/1.1 + my-header: foo \n Config: add: - name: \"my-header\" + value: \"bar,baz\" \n Output: GET /foo HTTP/1.1 + my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from + the HTTP request before the action. The value + of Remove is a list of HTTP header names. + Note that the header names are case-insensitive + (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo + my-header2: bar my-header3: baz \n Config: + remove: [\"my-header1\", \"my-header3\"] \n + Output: GET /foo HTTP/1.1 my-header2: bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with + the given header (name, value) before the + action. \n Input: GET /foo HTTP/1.1 my-header: + foo \n Config: set: - name: \"my-header\" + value: \"bar\" \n Output: GET /foo HTTP/1.1 + my-header: bar" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + type: + description: "Type identifies the type of filter + to apply. As with other API fields, types are + classified into three conformance levels: \n - + Core: Filter types and their corresponding configuration + defined by \"Support: Core\" in this package, + e.g. \"RequestHeaderModifier\". All implementations + supporting GRPCRoute MUST support core filters. + \n - Extended: Filter types and their corresponding + configuration defined by \"Support: Extended\" + in this package, e.g. \"RequestMirror\". Implementers + are encouraged to support extended filters. \n + - Implementation-specific: Filters that are defined + and supported by specific vendors. In the future, + filters showing convergence in behavior across + multiple implementations will be considered for + inclusion in extended or core conformance levels. + Filter-specific configuration for such filters + is specified using the ExtensionRef field. `Type` + MUST be set to \"ExtensionRef\" for custom filters. + \n Implementers are encouraged to define custom + implementation types to extend the core API with + implementation-specific behavior. \n If a reference + to a custom filter type cannot be resolved, the + filter MUST NOT be skipped. Instead, requests + that would have been processed by that filter + MUST receive a HTTP error response. \n " + enum: + - ResponseHeaderModifier + - RequestHeaderModifier + - RequestMirror + - ExtensionRef + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: filter.requestHeaderModifier must be nil + if the filter.type is not RequestHeaderModifier + rule: '!(has(self.requestHeaderModifier) && self.type + != ''RequestHeaderModifier'')' + - message: filter.requestHeaderModifier must be specified + for RequestHeaderModifier filter.type + rule: '!(!has(self.requestHeaderModifier) && self.type + == ''RequestHeaderModifier'')' + - message: filter.responseHeaderModifier must be nil + if the filter.type is not ResponseHeaderModifier + rule: '!(has(self.responseHeaderModifier) && self.type + != ''ResponseHeaderModifier'')' + - message: filter.responseHeaderModifier must be specified + for ResponseHeaderModifier filter.type + rule: '!(!has(self.responseHeaderModifier) && self.type + == ''ResponseHeaderModifier'')' + - message: filter.requestMirror must be nil if the filter.type + is not RequestMirror + rule: '!(has(self.requestMirror) && self.type != ''RequestMirror'')' + - message: filter.requestMirror must be specified for + RequestMirror filter.type + rule: '!(!has(self.requestMirror) && self.type == + ''RequestMirror'')' + - message: filter.extensionRef must be nil if the filter.type + is not ExtensionRef + rule: '!(has(self.extensionRef) && self.type != ''ExtensionRef'')' + - message: filter.extensionRef must be specified for + ExtensionRef filter.type + rule: '!(!has(self.extensionRef) && self.type == ''ExtensionRef'')' + maxItems: 16 + type: array + x-kubernetes-validations: + - message: RequestHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'RequestHeaderModifier').size() + <= 1 + - message: ResponseHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'ResponseHeaderModifier').size() + <= 1 group: default: "" description: Group is the group of the referent. For example, @@ -2070,9 +2829,17 @@ spec: type: string kind: default: Service - description: Kind is kind of the referent. For example - "HTTPRoute" or "Service". Defaults to "Service" when - not specified. + description: "Kind is the Kubernetes resource kind of + the referent. For example \"Service\". \n Defaults to + \"Service\" when not specified. \n ExternalName services + can refer to CNAME DNS records that may live outside + of the cluster and as such are difficult to reason about + in terms of conformance. They also may not be safe to + forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName Services. + \n Support: Core (Services with a type other than ExternalName) + \n Support: Implementation-specific (Services with type + ExternalName)" maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -2085,11 +2852,11 @@ spec: namespace: description: "Namespace is the namespace of the backend. When unspecified, the local namespace is inferred. \n - Note that when a namespace is specified, a ReferenceGrant - object is required in the referent namespace to allow - that namespace's owner to accept the reference. See - the ReferenceGrant documentation for details. \n Support: - Core" + Note that when a namespace different than the local + namespace is specified, a ReferenceGrant object is required + in the referent namespace to allow that namespace's + owner to accept the reference. See the ReferenceGrant + documentation for details. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -2127,18 +2894,580 @@ spec: required: - name type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' maxItems: 16 - minItems: 1 + type: array + filters: + description: "Filters define the filters that are applied to + requests that match this rule. \n The effects of ordering + of multiple behaviors are currently unspecified. This can + change in the future based on feedback during the alpha stage. + \n Conformance-levels at this level are defined based on the + type of filter: \n - ALL core filters MUST be supported by + all implementations that support GRPCRoute. - Implementers + are encouraged to support extended filters. - Implementation-specific + custom filters have no API guarantees across implementations. + \n Specifying the same filter multiple times is not supported + unless explicitly indicated in the filter. \n If an implementation + can not support a combination of filters, it must clearly + document that limitation. In cases where incompatible or unsupported + filters are specified and cause the `Accepted` condition to + be set to status `False`, implementations may use the `IncompatibleFilters` + reason to specify this configuration error. \n Support: Core" + items: + description: GRPCRouteFilter defines processing steps that + must be completed during the request or response lifecycle. + GRPCRouteFilters are meant as an extension point to express + processing that may be done in Gateway implementations. + Some examples include request or response modification, + implementing authentication strategies, rate-limiting, and + traffic shaping. API guarantee/conformance is defined based + on the type of the filter. + properties: + extensionRef: + description: "ExtensionRef is an optional, implementation-specific + extension to the \"filter\" behavior. For example, + resource \"myroutefilter\" in group \"networking.example.net\"). + ExtensionRef MUST NOT be used for core and extended + filters. \n Support: Implementation-specific \n This + filter can be used multiple times within the same rule." + properties: + group: + description: Group is the group of the referent. For + example, "gateway.networking.k8s.io". When unspecified + or empty string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For example + "HTTPRoute" or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + requestHeaderModifier: + description: "RequestHeaderModifier defines a schema for + a filter that modifies request headers. \n Support: + Core" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It appends + to any existing values associated with the header + name. \n Input: GET /foo HTTP/1.1 my-header: foo + \n Config: add: - name: \"my-header\" value: \"bar,baz\" + \n Output: GET /foo HTTP/1.1 my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from the + HTTP request before the action. The value of Remove + is a list of HTTP header names. Note that the header + names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo my-header2: + bar my-header3: baz \n Config: remove: [\"my-header1\", + \"my-header3\"] \n Output: GET /foo HTTP/1.1 my-header2: + bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with the + given header (name, value) before the action. \n + Input: GET /foo HTTP/1.1 my-header: foo \n Config: + set: - name: \"my-header\" value: \"bar\" \n Output: + GET /foo HTTP/1.1 my-header: bar" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + requestMirror: + description: "RequestMirror defines a schema for a filter + that mirrors requests. Requests are sent to the specified + destination, but responses from that destination are + ignored. \n This filter can be used multiple times within + the same rule. Note that not all implementations will + be able to support mirroring to multiple backends. \n + Support: Extended" + properties: + backendRef: + description: "BackendRef references a resource where + mirrored requests are sent. \n Mirrored requests + must be sent only to a single destination endpoint + within this BackendRef, irrespective of how many + endpoints are present within this BackendRef. \n + If the referent cannot be found, this BackendRef + is invalid and must be dropped from the Gateway. + The controller must ensure the \"ResolvedRefs\" + condition on the Route status is set to `status: + False` and not configure this backend in the underlying + implementation. \n If there is a cross-namespace + reference to an *existing* object that is not allowed + by a ReferenceGrant, the controller must ensure + the \"ResolvedRefs\" condition on the Route is + set to `status: False`, with the \"RefNotPermitted\" + reason and not configure this backend in the underlying + implementation. \n In either error case, the Message + of the `ResolvedRefs` Condition should be used to + provide more detail about the problem. \n Support: + Extended for Kubernetes Service \n Support: Implementation-specific + for any other resource" + properties: + group: + default: "" + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". When + unspecified or empty string, core API group + is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource + kind of the referent. For example \"Service\". + \n Defaults to \"Service\" when not specified. + \n ExternalName services can refer to CNAME + DNS records that may live outside of the cluster + and as such are difficult to reason about in + terms of conformance. They also may not be safe + to forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName + Services. \n Support: Core (Services with a + type other than ExternalName) \n Support: Implementation-specific + (Services with type ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the + backend. When unspecified, the local namespace + is inferred. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace + to allow that namespace's owner to accept the + reference. See the ReferenceGrant documentation + for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port + number to use for this resource. Port is required + when the referent is a Kubernetes Service. In + this case, the port number is the service port + number, not the target port. For other resources, + destination port might be derived from the referent + resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + required: + - backendRef + type: object + responseHeaderModifier: + description: "ResponseHeaderModifier defines a schema + for a filter that modifies response headers. \n Support: + Extended" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It appends + to any existing values associated with the header + name. \n Input: GET /foo HTTP/1.1 my-header: foo + \n Config: add: - name: \"my-header\" value: \"bar,baz\" + \n Output: GET /foo HTTP/1.1 my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from the + HTTP request before the action. The value of Remove + is a list of HTTP header names. Note that the header + names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo my-header2: + bar my-header3: baz \n Config: remove: [\"my-header1\", + \"my-header3\"] \n Output: GET /foo HTTP/1.1 my-header2: + bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with the + given header (name, value) before the action. \n + Input: GET /foo HTTP/1.1 my-header: foo \n Config: + set: - name: \"my-header\" value: \"bar\" \n Output: + GET /foo HTTP/1.1 my-header: bar" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + type: + description: "Type identifies the type of filter to apply. + As with other API fields, types are classified into + three conformance levels: \n - Core: Filter types and + their corresponding configuration defined by \"Support: + Core\" in this package, e.g. \"RequestHeaderModifier\". + All implementations supporting GRPCRoute MUST support + core filters. \n - Extended: Filter types and their + corresponding configuration defined by \"Support: Extended\" + in this package, e.g. \"RequestMirror\". Implementers + are encouraged to support extended filters. \n - Implementation-specific: + Filters that are defined and supported by specific vendors. + In the future, filters showing convergence in behavior + across multiple implementations will be considered for + inclusion in extended or core conformance levels. Filter-specific + configuration for such filters is specified using the + ExtensionRef field. `Type` MUST be set to \"ExtensionRef\" + for custom filters. \n Implementers are encouraged to + define custom implementation types to extend the core + API with implementation-specific behavior. \n If a reference + to a custom filter type cannot be resolved, the filter + MUST NOT be skipped. Instead, requests that would have + been processed by that filter MUST receive a HTTP error + response. \n " + enum: + - ResponseHeaderModifier + - RequestHeaderModifier + - RequestMirror + - ExtensionRef + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: filter.requestHeaderModifier must be nil if the + filter.type is not RequestHeaderModifier + rule: '!(has(self.requestHeaderModifier) && self.type != + ''RequestHeaderModifier'')' + - message: filter.requestHeaderModifier must be specified + for RequestHeaderModifier filter.type + rule: '!(!has(self.requestHeaderModifier) && self.type == + ''RequestHeaderModifier'')' + - message: filter.responseHeaderModifier must be nil if the + filter.type is not ResponseHeaderModifier + rule: '!(has(self.responseHeaderModifier) && self.type != + ''ResponseHeaderModifier'')' + - message: filter.responseHeaderModifier must be specified + for ResponseHeaderModifier filter.type + rule: '!(!has(self.responseHeaderModifier) && self.type + == ''ResponseHeaderModifier'')' + - message: filter.requestMirror must be nil if the filter.type + is not RequestMirror + rule: '!(has(self.requestMirror) && self.type != ''RequestMirror'')' + - message: filter.requestMirror must be specified for RequestMirror + filter.type + rule: '!(!has(self.requestMirror) && self.type == ''RequestMirror'')' + - message: filter.extensionRef must be nil if the filter.type + is not ExtensionRef + rule: '!(has(self.extensionRef) && self.type != ''ExtensionRef'')' + - message: filter.extensionRef must be specified for ExtensionRef + filter.type + rule: '!(!has(self.extensionRef) && self.type == ''ExtensionRef'')' + maxItems: 16 + type: array + x-kubernetes-validations: + - message: RequestHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'RequestHeaderModifier').size() + <= 1 + - message: ResponseHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'ResponseHeaderModifier').size() + <= 1 + matches: + description: "Matches define conditions used for matching the + rule against incoming gRPC requests. Each match is independent, + i.e. this rule will be matched if **any** one of the matches + is satisfied. \n For example, take the following matches configuration: + \n ``` matches: - method: service: foo.bar headers: values: + version: 2 - method: service: foo.bar.v2 ``` \n For a request + to match against this rule, it MUST satisfy EITHER of the + two conditions: \n - service of foo.bar AND contains the header + `version: 2` - service of foo.bar.v2 \n See the documentation + for GRPCRouteMatch on how to specify multiple match conditions + to be ANDed together. \n If no matches are specified, the + implementation MUST match every gRPC request. \n Proxy or + Load Balancer routing configuration generated from GRPCRoutes + MUST prioritize rules based on the following criteria, continuing + on ties. Merging MUST not be done between GRPCRoutes and HTTPRoutes. + Precedence MUST be given to the rule with the largest number + of: \n * Characters in a matching non-wildcard hostname. * + Characters in a matching hostname. * Characters in a matching + service. * Characters in a matching method. * Header matches. + \n If ties still exist across multiple Routes, matching precedence + MUST be determined in order of the following criteria, continuing + on ties: \n * The oldest Route based on creation timestamp. + * The Route appearing first in alphabetical order by \"{namespace}/{name}\". + \n If ties still exist within the Route that has been given + precedence, matching precedence MUST be granted to the first + matching rule meeting the above criteria." + items: + description: "GRPCRouteMatch defines the predicate used to + match requests to a given action. Multiple match types are + ANDed together, i.e. the match will evaluate to true only + if all conditions are satisfied. \n For example, the match + below will match a gRPC request only if its service is `foo` + AND it contains the `version: v1` header: \n ``` matches: + - method: type: Exact service: \"foo\" headers: - name: + \"version\" value \"v1\" \n ```" + properties: + headers: + description: Headers specifies gRPC request header matchers. + Multiple match values are ANDed together, meaning, a + request MUST match all the specified headers to select + the route. + items: + description: GRPCHeaderMatch describes how to select + a gRPC route by matching gRPC request headers. + properties: + name: + description: "Name is the name of the gRPC Header + to be matched. \n If multiple entries specify + equivalent header names, only the first entry + with an equivalent name MUST be considered for + a match. Subsequent entries with an equivalent + header name MUST be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + type: + default: Exact + description: Type specifies how to match against + the value of the header. + enum: + - Exact + - RegularExpression + type: string + value: + description: Value is the value of the gRPC Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + method: + description: Method specifies a gRPC request service/method + matcher. If this field is not specified, all services + and methods will match. + properties: + method: + description: "Value of the method to match against. + If left empty or omitted, will match all services. + \n At least one of Service and Method MUST be a + non-empty string." + maxLength: 1024 + type: string + service: + description: "Value of the service to match against. + If left empty or omitted, will match any service. + \n At least one of Service and Method MUST be a + non-empty string." + maxLength: 1024 + type: string + type: + default: Exact + description: "Type specifies how to match against + the service and/or method. Support: Core (Exact + with service and method specified) \n Support: Implementation-specific + (Exact with method specified but no service specified) + \n Support: Implementation-specific (RegularExpression)" + enum: + - Exact + - RegularExpression + type: string + type: object + x-kubernetes-validations: + - message: One or both of 'service' or 'method' must be + specified + rule: 'has(self.type) ? has(self.service) || has(self.method) + : true' + - message: service must only contain valid characters + (matching ^(?i)\.?[a-z_][a-z_0-9]*(\.[a-z_][a-z_0-9]*)*$) + rule: '(!has(self.type) || self.type == ''Exact'') && + has(self.service) ? self.service.matches(r"""^(?i)\.?[a-z_][a-z_0-9]*(\.[a-z_][a-z_0-9]*)*$"""): + true' + - message: method must only contain valid characters (matching + ^[A-Za-z_][A-Za-z_0-9]*$) + rule: '(!has(self.type) || self.type == ''Exact'') && + has(self.method) ? self.method.matches(r"""^[A-Za-z_][A-Za-z_0-9]*$"""): + true' + type: object + maxItems: 8 type: array type: object maxItems: 16 - minItems: 1 type: array - required: - - rules type: object status: - description: Status defines the current state of UDPRoute. + description: Status defines the current state of GRPCRoute. properties: parents: description: "Parents is a list of parent resources (usually Gateways) @@ -2178,15 +3507,14 @@ spec: description: "Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path - .status.conditions. For example, \n \ttype FooStatus struct{ - \t // Represents the observations of a foo's current - state. \t // Known .status.conditions.type are: \"Available\", - \"Progressing\", and \"Degraded\" \t // +patchMergeKey=type - \t // +patchStrategy=merge \t // +listType=map \t - \ // +listMapKey=type \t Conditions []metav1.Condition + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n \t // other - fields \t}" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" properties: lastTransitionTime: description: lastTransitionTime is the last time the condition @@ -2283,9 +3611,11 @@ spec: type: string kind: default: Gateway - description: "Kind is kind of the referent. \n Support: - Core (Gateway) \n Support: Implementation-specific (Other - Resources)" + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -2305,7 +3635,16 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n Support: Core" + reference. \n ParentRefs from a Route to a Service in + the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any + namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for + which the intended destination of the connections are + a Service targeted as a ParentRef of the Route. \n Support: + Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -2321,8 +3660,12 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n Implementations MAY choose to - support other parent resources. Implementations supporting + both specified values. \n When the parent resource is + a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are + specified, the name and port of the selected port must + match both specified values. \n Implementations MAY choose + to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered successful as long as the @@ -2333,7 +3676,7 @@ spec: the Route MUST be considered successfully attached. If no Gateway listeners accept attachment from this Route, the Route MUST be considered detached from the Gateway. - \n Support: Extended \n " + \n Support: Extended \n " format: int32 maximum: 65535 minimum: 1 @@ -2344,8 +3687,7028 @@ spec: is interpreted as the following: \n * Gateway: Listener Name. When both Port (experimental) and SectionName are specified, the name and port of the selected listener - must match both specified values. \n Implementations MAY - choose to support attaching Routes to other resources. + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this + will reference the entire resource. For the purpose of + status, an attachment is considered successful if at least + one section in the parent resource accepts it. For example, + Gateway listeners can restrict which Routes can attach + to them by Route kind, namespace, or hostname. If 1 of + 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + required: + - controllerName + - parentRef + type: object + maxItems: 32 + type: array + required: + - parents + type: object + type: object + served: true + storage: true + subresources: + status: {} +status: + acceptedNames: + kind: "" + plural: "" + conditions: null + storedVersions: null +--- +# +# config/crd/experimental/gateway.networking.k8s.io_httproutes.yaml +# +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 + gateway.networking.k8s.io/channel: experimental + creationTimestamp: null + name: httproutes.gateway.networking.k8s.io +spec: + group: gateway.networking.k8s.io + names: + categories: + - gateway-api + kind: HTTPRoute + listKind: HTTPRouteList + plural: httproutes + singular: httproute + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.hostnames + name: Hostnames + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + deprecated: true + deprecationWarning: The v1alpha2 version of HTTPRoute has been deprecated and + will be removed in a future release of the API. Please upgrade to v1beta1. + name: v1alpha2 + schema: + openAPIV3Schema: + description: HTTPRoute provides a way to route HTTP requests. This includes + the capability to match requests by hostname, path, header, or query param. + Filters can be used to specify additional processing steps. Backends specify + where matching requests should be routed. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of HTTPRoute. + properties: + hostnames: + description: "Hostnames defines a set of hostnames that should match + against the HTTP Host header to select a HTTPRoute used to process + the request. Implementations MUST ignore any port value specified + in the HTTP Host header while performing a match and (absent of + any applicable header modification configuration) MUST forward this + header unmodified to the backend. \n Valid values for Hostnames + are determined by RFC 1123 definition of a hostname with 2 notable + exceptions: \n 1. IPs are not allowed. 2. A hostname may be prefixed + with a wildcard label (`*.`). The wildcard label must appear by + itself as the first label. \n If a hostname is specified by both + the Listener and HTTPRoute, there must be at least one intersecting + hostname for the HTTPRoute to be attached to the Listener. For example: + \n * A Listener with `test.example.com` as the hostname matches + HTTPRoutes that have either not specified any hostnames, or have + specified at least one of `test.example.com` or `*.example.com`. + * A Listener with `*.example.com` as the hostname matches HTTPRoutes + that have either not specified any hostnames or have specified at + least one hostname that matches the Listener hostname. For example, + `*.example.com`, `test.example.com`, and `foo.test.example.com` + would all match. On the other hand, `example.com` and `test.example.net` + would not match. \n Hostnames that are prefixed with a wildcard + label (`*.`) are interpreted as a suffix match. That means that + a match for `*.example.com` would match both `test.example.com`, + and `foo.test.example.com`, but not `example.com`. \n If both the + Listener and HTTPRoute have specified hostnames, any HTTPRoute hostnames + that do not match the Listener hostname MUST be ignored. For example, + if a Listener specified `*.example.com`, and the HTTPRoute specified + `test.example.com` and `test.example.net`, `test.example.net` must + not be considered for a match. \n If both the Listener and HTTPRoute + have specified hostnames, and none match with the criteria above, + then the HTTPRoute is not accepted. The implementation must raise + an 'Accepted' Condition with a status of `False` in the corresponding + RouteParentStatus. \n In the event that multiple HTTPRoutes specify + intersecting hostnames (e.g. overlapping wildcard matching and exact + matching hostnames), precedence must be given to rules from the + HTTPRoute with the largest number of: \n * Characters in a matching + non-wildcard hostname. * Characters in a matching hostname. \n If + ties exist across multiple Routes, the matching precedence rules + for HTTPRouteMatches takes over. \n Support: Core" + items: + description: "Hostname is the fully qualified domain name of a network + host. This matches the RFC 1123 definition of a hostname with + 2 notable exceptions: \n 1. IPs are not allowed. 2. A hostname + may be prefixed with a wildcard label (`*.`). The wildcard label + must appear by itself as the first label. \n Hostname can be \"precise\" + which is a domain name without the terminating dot of a network + host (e.g. \"foo.example.com\") or \"wildcard\", which is a domain + name prefixed with a single wildcard label (e.g. `*.example.com`). + \n Note that as per RFC1035 and RFC1123, a *label* must consist + of lower case alphanumeric characters or '-', and must start and + end with an alphanumeric character. No other punctuation is allowed." + maxLength: 253 + minLength: 1 + pattern: ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + maxItems: 16 + type: array + parentRefs: + description: "ParentRefs references the resources (usually Gateways) + that a Route wants to be attached to. Note that the referenced parent + resource needs to allow this for the attachment to be complete. + For Gateways, that means the Gateway needs to allow attachment from + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) \n This + API may be extended in the future to support additional kinds of + parent resources. \n It is invalid to reference an identical parent + more than once. It is valid to reference multiple distinct sections + within the same parent resource, such as two separate Listeners + on the same Gateway or two separate ports on the same Service. \n + It is possible to separately reference multiple distinct objects + that may be collapsed by an implementation. For example, some implementations + may choose to merge compatible Gateway Listeners together. If that + is the case, the list of routes attached to those resources should + also be merged. \n Note that for ParentRefs that cross namespace + boundaries, there are specific rules. Cross-namespace references + are only valid if they are explicitly allowed by something in the + namespace they are referring to. For example, Gateway has the AllowedRoutes + field, and ReferenceGrant provides a generic way to enable other + kinds of cross-namespace reference. \n ParentRefs from a Route to + a Service in the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any namespace + to the Service. \n ParentRefs from a Route to a Service in a different + namespace are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same namespace + as the Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. \n " + items: + description: "ParentReference identifies an API object (usually + a Gateway) that can be considered a parent of this resource (usually + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the core + API group (such as for a \"Service\" kind referent), Group + must be explicitly set to \"\" (empty string). \n Support: + Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. When + unspecified, this refers to the local namespace of the Route. + \n Note that there are specific rules for ParentRefs which + cross namespace boundaries. Cross-namespace references are + only valid if they are explicitly allowed by something in + the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + \n ParentRefs from a Route to a Service in the same namespace + are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. + \n ParentRefs from a Route to a Service in a different namespace + are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same + namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of + the Route. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. It + can be interpreted differently based on the type of parent + resource. \n When the parent resource is a Gateway, this targets + all listeners listening on the specified port that also support + this kind of Route(and select this Route). It's not recommended + to set `Port` unless the networking behaviors specified in + a Route must apply to a specific port as opposed to a listener(s) + whose port(s) may be changed. When both Port and SectionName + are specified, the name and port of the selected listener + must match both specified values. \n When the parent resource + is a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified + values. \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes can + attach to them by Route kind, namespace, or hostname. If 1 + of 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. \n + Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within the + target resource. In the following resources, SectionName is + interpreted as the following: \n * Gateway: Listener Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match both + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + maxItems: 32 + type: array + x-kubernetes-validations: + - message: sectionName or port must be specified when parentRefs includes + 2 or more references to the same parent + rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ + == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) + && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) + || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName + != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) + && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) + ) ) ): true ))' + - message: sectionName or port must be unique when parentRefs includes + 2 or more references to the same parent + rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) + || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ + == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) + || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName + == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName + == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port + == p2.port)))) + rules: + default: + - matches: + - path: + type: PathPrefix + value: / + description: Rules are a list of HTTP matchers, filters and actions. + items: + description: HTTPRouteRule defines semantics for matching an HTTP + request based on conditions (matches), processing it (filters), + and forwarding the request to an API object (backendRefs). + properties: + backendRefs: + description: "BackendRefs defines the backend(s) where matching + requests should be sent. \n Failure behavior here depends + on how many BackendRefs are specified and how many are invalid. + \n If *all* entries in BackendRefs are invalid, and there + are also no filters specified in this route rule, *all* traffic + which matches this rule MUST receive a 500 status code. \n + See the HTTPBackendRef definition for the rules about what + makes a single HTTPBackendRef invalid. \n When a HTTPBackendRef + is invalid, 500 status codes MUST be returned for requests + that would have otherwise been routed to an invalid backend. + If multiple backends are specified, and some are invalid, + the proportion of requests that would otherwise have been + routed to an invalid backend MUST receive a 500 status code. + \n For example, if two backends are specified with equal weights, + and one is invalid, 50 percent of traffic must receive a 500. + Implementations may choose how that 50 percent is determined. + \n Support: Core for Kubernetes Service \n Support: Extended + for Kubernetes ServiceImport \n Support: Implementation-specific + for any other resource \n Support for weight: Core" + items: + description: HTTPBackendRef defines how a HTTPRoute should + forward an HTTP request. + properties: + filters: + description: "Filters defined at this level should be + executed if and only if the request is being forwarded + to the backend defined here. \n Support: Implementation-specific + (For broader support of filters, use the Filters field + in HTTPRouteRule.)" + items: + description: HTTPRouteFilter defines processing steps + that must be completed during the request or response + lifecycle. HTTPRouteFilters are meant as an extension + point to express processing that may be done in Gateway + implementations. Some examples include request or + response modification, implementing authentication + strategies, rate-limiting, and traffic shaping. API + guarantee/conformance is defined based on the type + of the filter. + properties: + extensionRef: + description: "ExtensionRef is an optional, implementation-specific + extension to the \"filter\" behavior. For example, + resource \"myroutefilter\" in group \"networking.example.net\"). + ExtensionRef MUST NOT be used for core and extended + filters. \n This filter can be used multiple times + within the same rule. \n Support: Implementation-specific" + properties: + group: + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API + group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For + example "HTTPRoute" or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + requestHeaderModifier: + description: "RequestHeaderModifier defines a schema + for a filter that modifies request headers. \n + Support: Core" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It + appends to any existing values associated + with the header name. \n Input: GET /foo HTTP/1.1 + my-header: foo \n Config: add: - name: \"my-header\" + value: \"bar,baz\" \n Output: GET /foo HTTP/1.1 + my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from + the HTTP request before the action. The value + of Remove is a list of HTTP header names. + Note that the header names are case-insensitive + (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo + my-header2: bar my-header3: baz \n Config: + remove: [\"my-header1\", \"my-header3\"] \n + Output: GET /foo HTTP/1.1 my-header2: bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with + the given header (name, value) before the + action. \n Input: GET /foo HTTP/1.1 my-header: + foo \n Config: set: - name: \"my-header\" + value: \"bar\" \n Output: GET /foo HTTP/1.1 + my-header: bar" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + requestMirror: + description: "RequestMirror defines a schema for + a filter that mirrors requests. Requests are sent + to the specified destination, but responses from + that destination are ignored. \n This filter can + be used multiple times within the same rule. Note + that not all implementations will be able to support + mirroring to multiple backends. \n Support: Extended" + properties: + backendRef: + description: "BackendRef references a resource + where mirrored requests are sent. \n Mirrored + requests must be sent only to a single destination + endpoint within this BackendRef, irrespective + of how many endpoints are present within this + BackendRef. \n If the referent cannot be found, + this BackendRef is invalid and must be dropped + from the Gateway. The controller must ensure + the \"ResolvedRefs\" condition on the Route + status is set to `status: False` and not configure + this backend in the underlying implementation. + \n If there is a cross-namespace reference + to an *existing* object that is not allowed + by a ReferenceGrant, the controller must ensure + the \"ResolvedRefs\" condition on the Route + is set to `status: False`, with the \"RefNotPermitted\" + reason and not configure this backend in the + underlying implementation. \n In either error + case, the Message of the `ResolvedRefs` Condition + should be used to provide more detail about + the problem. \n Support: Extended for Kubernetes + Service \n Support: Implementation-specific + for any other resource" + properties: + group: + default: "" + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". + When unspecified or empty string, core + API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource + kind of the referent. For example \"Service\". + \n Defaults to \"Service\" when not specified. + \n ExternalName services can refer to + CNAME DNS records that may live outside + of the cluster and as such are difficult + to reason about in terms of conformance. + They also may not be safe to forward to + (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName + Services. \n Support: Core (Services with + a type other than ExternalName) \n Support: + Implementation-specific (Services with + type ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace + of the backend. When unspecified, the + local namespace is inferred. \n Note that + when a namespace different than the local + namespace is specified, a ReferenceGrant + object is required in the referent namespace + to allow that namespace's owner to accept + the reference. See the ReferenceGrant + documentation for details. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination + port number to use for this resource. + Port is required when the referent is + a Kubernetes Service. In this case, the + port number is the service port number, + not the target port. For other resources, + destination port might be derived from + the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind + == ''Service'') ? has(self.port) : true' + required: + - backendRef + type: object + requestRedirect: + description: "RequestRedirect defines a schema for + a filter that responds to the request with an + HTTP redirection. \n Support: Core" + properties: + hostname: + description: "Hostname is the hostname to be + used in the value of the `Location` header + in the response. When empty, the hostname + in the `Host` header of the request is used. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines parameters used to + modify the path of the incoming request. The + modified path is then used to construct the + `Location` header. When empty, the request + path is used as-is. \n Support: Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the + value with which to replace the full path + of a request during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies + the value with which to replace the prefix + match of a request during a rewrite or + redirect. For example, a request to \"/foo/bar\" + with a prefix match of \"/foo\" and a + ReplacePrefixMatch of \"/xyz\" would be + modified to \"/xyz/bar\". \n Note that + this matches the behavior of the PathPrefix + match type. This matches full path elements. + A path element refers to the list of labels + in the path split by the `/` separator. + When specified, a trailing `/` is ignored. + For example, the paths `/abc`, `/abc/`, + and `/abc/def` would all match the prefix + `/abc`, but the path `/abcd` would not. + \n ReplacePrefixMatch is only compatible + with a `PathPrefix` HTTPRouteMatch. Using + any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the + Route to `status: False`. \n Request Path + | Prefix Match | Replace Prefix | Modified + Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | + /xyz/ | /xyz/bar /foo/bar | + /foo/ | /xyz | /xyz/bar + /foo/bar | /foo/ | /xyz/ | + /xyz/bar /foo | /foo | + /xyz | /xyz /foo/ | /foo + \ | /xyz | /xyz/ /foo/bar + \ | /foo | | + /bar /foo/ | /foo | | / /foo | /foo | + | / /foo/ | /foo + \ | / | / /foo | + /foo | / | /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path + modifier. Additional types may be added + in a future release of the API. \n Note + that values may be added to this enum, + implementations must ensure that unknown + values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the + Route to `status: False`, with a Reason + of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified + when type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? + has(self.replaceFullPath) : true' + - message: type must be 'ReplaceFullPath' when + replaceFullPath is set + rule: 'has(self.replaceFullPath) ? self.type + == ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified + when type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' + ? has(self.replacePrefixMatch) : true' + - message: type must be 'ReplacePrefixMatch' + when replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + port: + description: "Port is the port to be used in + the value of the `Location` header in the + response. \n If no port is specified, the + redirect port MUST be derived using the following + rules: \n * If redirect scheme is not-empty, + the redirect port MUST be the well-known port + associated with the redirect scheme. Specifically + \"http\" to port 80 and \"https\" to port + 443. If the redirect scheme does not have + a well-known port, the listener port of the + Gateway SHOULD be used. * If redirect scheme + is empty, the redirect port MUST be the Gateway + Listener port. \n Implementations SHOULD NOT + add the port number in the 'Location' header + in the following cases: \n * A Location header + that will use HTTP (whether that is determined + via the Listener protocol or the Scheme field) + _and_ use port 80. * A Location header that + will use HTTPS (whether that is determined + via the Listener protocol or the Scheme field) + _and_ use port 443. \n Support: Extended" + format: int32 + maximum: 65535 + minimum: 1 + type: integer + scheme: + description: "Scheme is the scheme to be used + in the value of the `Location` header in the + response. When empty, the scheme of the request + is used. \n Scheme redirects can affect the + port of the redirect, for more information, + refer to the documentation for the port field + of this filter. \n Note that values may be + added to this enum, implementations must ensure + that unknown values will not cause a crash. + \n Unknown values here must result in the + implementation setting the Accepted Condition + for the Route to `status: False`, with a Reason + of `UnsupportedValue`. \n Support: Extended" + enum: + - http + - https + type: string + statusCode: + default: 302 + description: "StatusCode is the HTTP status + code to be used in response. \n Note that + values may be added to this enum, implementations + must ensure that unknown values will not cause + a crash. \n Unknown values here must result + in the implementation setting the Accepted + Condition for the Route to `status: False`, + with a Reason of `UnsupportedValue`. \n Support: + Core" + enum: + - 301 + - 302 + type: integer + type: object + responseHeaderModifier: + description: "ResponseHeaderModifier defines a schema + for a filter that modifies response headers. \n + Support: Extended" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It + appends to any existing values associated + with the header name. \n Input: GET /foo HTTP/1.1 + my-header: foo \n Config: add: - name: \"my-header\" + value: \"bar,baz\" \n Output: GET /foo HTTP/1.1 + my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from + the HTTP request before the action. The value + of Remove is a list of HTTP header names. + Note that the header names are case-insensitive + (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo + my-header2: bar my-header3: baz \n Config: + remove: [\"my-header1\", \"my-header3\"] \n + Output: GET /foo HTTP/1.1 my-header2: bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with + the given header (name, value) before the + action. \n Input: GET /foo HTTP/1.1 my-header: + foo \n Config: set: - name: \"my-header\" + value: \"bar\" \n Output: GET /foo HTTP/1.1 + my-header: bar" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + type: + description: "Type identifies the type of filter + to apply. As with other API fields, types are + classified into three conformance levels: \n - + Core: Filter types and their corresponding configuration + defined by \"Support: Core\" in this package, + e.g. \"RequestHeaderModifier\". All implementations + must support core filters. \n - Extended: Filter + types and their corresponding configuration defined + by \"Support: Extended\" in this package, e.g. + \"RequestMirror\". Implementers are encouraged + to support extended filters. \n - Implementation-specific: + Filters that are defined and supported by specific + vendors. In the future, filters showing convergence + in behavior across multiple implementations will + be considered for inclusion in extended or core + conformance levels. Filter-specific configuration + for such filters is specified using the ExtensionRef + field. `Type` should be set to \"ExtensionRef\" + for custom filters. \n Implementers are encouraged + to define custom implementation types to extend + the core API with implementation-specific behavior. + \n If a reference to a custom filter type cannot + be resolved, the filter MUST NOT be skipped. Instead, + requests that would have been processed by that + filter MUST receive a HTTP error response. \n + Note that values may be added to this enum, implementations + must ensure that unknown values will not cause + a crash. \n Unknown values here must result in + the implementation setting the Accepted Condition + for the Route to `status: False`, with a Reason + of `UnsupportedValue`." + enum: + - RequestHeaderModifier + - ResponseHeaderModifier + - RequestMirror + - RequestRedirect + - URLRewrite + - ExtensionRef + type: string + urlRewrite: + description: "URLRewrite defines a schema for a + filter that modifies a request during forwarding. + \n Support: Extended" + properties: + hostname: + description: "Hostname is the value to be used + to replace the Host header value during forwarding. + \n Support: Extended" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines a path rewrite. \n + Support: Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the + value with which to replace the full path + of a request during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies + the value with which to replace the prefix + match of a request during a rewrite or + redirect. For example, a request to \"/foo/bar\" + with a prefix match of \"/foo\" and a + ReplacePrefixMatch of \"/xyz\" would be + modified to \"/xyz/bar\". \n Note that + this matches the behavior of the PathPrefix + match type. This matches full path elements. + A path element refers to the list of labels + in the path split by the `/` separator. + When specified, a trailing `/` is ignored. + For example, the paths `/abc`, `/abc/`, + and `/abc/def` would all match the prefix + `/abc`, but the path `/abcd` would not. + \n ReplacePrefixMatch is only compatible + with a `PathPrefix` HTTPRouteMatch. Using + any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the + Route to `status: False`. \n Request Path + | Prefix Match | Replace Prefix | Modified + Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | + /xyz/ | /xyz/bar /foo/bar | + /foo/ | /xyz | /xyz/bar + /foo/bar | /foo/ | /xyz/ | + /xyz/bar /foo | /foo | + /xyz | /xyz /foo/ | /foo + \ | /xyz | /xyz/ /foo/bar + \ | /foo | | + /bar /foo/ | /foo | | / /foo | /foo | + | / /foo/ | /foo + \ | / | / /foo | + /foo | / | /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path + modifier. Additional types may be added + in a future release of the API. \n Note + that values may be added to this enum, + implementations must ensure that unknown + values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the + Route to `status: False`, with a Reason + of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified + when type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? + has(self.replaceFullPath) : true' + - message: type must be 'ReplaceFullPath' when + replaceFullPath is set + rule: 'has(self.replaceFullPath) ? self.type + == ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified + when type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' + ? has(self.replacePrefixMatch) : true' + - message: type must be 'ReplacePrefixMatch' + when replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + type: object + required: + - type + type: object + x-kubernetes-validations: + - message: filter.requestHeaderModifier must be nil + if the filter.type is not RequestHeaderModifier + rule: '!(has(self.requestHeaderModifier) && self.type + != ''RequestHeaderModifier'')' + - message: filter.requestHeaderModifier must be specified + for RequestHeaderModifier filter.type + rule: '!(!has(self.requestHeaderModifier) && self.type + == ''RequestHeaderModifier'')' + - message: filter.responseHeaderModifier must be nil + if the filter.type is not ResponseHeaderModifier + rule: '!(has(self.responseHeaderModifier) && self.type + != ''ResponseHeaderModifier'')' + - message: filter.responseHeaderModifier must be specified + for ResponseHeaderModifier filter.type + rule: '!(!has(self.responseHeaderModifier) && self.type + == ''ResponseHeaderModifier'')' + - message: filter.requestMirror must be nil if the filter.type + is not RequestMirror + rule: '!(has(self.requestMirror) && self.type != ''RequestMirror'')' + - message: filter.requestMirror must be specified for + RequestMirror filter.type + rule: '!(!has(self.requestMirror) && self.type == + ''RequestMirror'')' + - message: filter.requestRedirect must be nil if the + filter.type is not RequestRedirect + rule: '!(has(self.requestRedirect) && self.type != + ''RequestRedirect'')' + - message: filter.requestRedirect must be specified + for RequestRedirect filter.type + rule: '!(!has(self.requestRedirect) && self.type == + ''RequestRedirect'')' + - message: filter.urlRewrite must be nil if the filter.type + is not URLRewrite + rule: '!(has(self.urlRewrite) && self.type != ''URLRewrite'')' + - message: filter.urlRewrite must be specified for URLRewrite + filter.type + rule: '!(!has(self.urlRewrite) && self.type == ''URLRewrite'')' + - message: filter.extensionRef must be nil if the filter.type + is not ExtensionRef + rule: '!(has(self.extensionRef) && self.type != ''ExtensionRef'')' + - message: filter.extensionRef must be specified for + ExtensionRef filter.type + rule: '!(!has(self.extensionRef) && self.type == ''ExtensionRef'')' + maxItems: 16 + type: array + x-kubernetes-validations: + - message: May specify either httpRouteFilterRequestRedirect + or httpRouteFilterRequestRewrite, but not both + rule: '!(self.exists(f, f.type == ''RequestRedirect'') + && self.exists(f, f.type == ''URLRewrite''))' + - message: May specify either httpRouteFilterRequestRedirect + or httpRouteFilterRequestRewrite, but not both + rule: '!(self.exists(f, f.type == ''RequestRedirect'') + && self.exists(f, f.type == ''URLRewrite''))' + - message: RequestHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'RequestHeaderModifier').size() + <= 1 + - message: ResponseHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'ResponseHeaderModifier').size() + <= 1 + - message: RequestRedirect filter cannot be repeated + rule: self.filter(f, f.type == 'RequestRedirect').size() + <= 1 + - message: URLRewrite filter cannot be repeated + rule: self.filter(f, f.type == 'URLRewrite').size() + <= 1 + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty + string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource kind of + the referent. For example \"Service\". \n Defaults to + \"Service\" when not specified. \n ExternalName services + can refer to CNAME DNS records that may live outside + of the cluster and as such are difficult to reason about + in terms of conformance. They also may not be safe to + forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName Services. + \n Support: Core (Services with a type other than ExternalName) + \n Support: Implementation-specific (Services with type + ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the backend. + When unspecified, the local namespace is inferred. \n + Note that when a namespace different than the local + namespace is specified, a ReferenceGrant object is required + in the referent namespace to allow that namespace's + owner to accept the reference. See the ReferenceGrant + documentation for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port number + to use for this resource. Port is required when the + referent is a Kubernetes Service. In this case, the + port number is the service port number, not the target + port. For other resources, destination port might be + derived from the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + weight: + default: 1 + description: "Weight specifies the proportion of requests + forwarded to the referenced backend. This is computed + as weight/(sum of all weights in this BackendRefs list). + For non-zero values, there may be some epsilon from + the exact proportion defined here depending on the precision + an implementation supports. Weight is not a percentage + and the sum of weights does not need to equal 100. \n + If only one backend is specified and it has a weight + greater than 0, 100% of the traffic is forwarded to + that backend. If weight is set to 0, no traffic should + be forwarded for this entry. If unspecified, weight + defaults to 1. \n Support for this field varies based + on the context where used." + format: int32 + maximum: 1000000 + minimum: 0 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + maxItems: 16 + type: array + filters: + description: "Filters define the filters that are applied to + requests that match this rule. \n The effects of ordering + of multiple behaviors are currently unspecified. This can + change in the future based on feedback during the alpha stage. + \n Conformance-levels at this level are defined based on the + type of filter: \n - ALL core filters MUST be supported by + all implementations. - Implementers are encouraged to support + extended filters. - Implementation-specific custom filters + have no API guarantees across implementations. \n Specifying + the same filter multiple times is not supported unless explicitly + indicated in the filter. \n All filters are expected to be + compatible with each other except for the URLRewrite and RequestRedirect + filters, which may not be combined. If an implementation can + not support other combinations of filters, they must clearly + document that limitation. In cases where incompatible or unsupported + filters are specified and cause the `Accepted` condition to + be set to status `False`, implementations may use the `IncompatibleFilters` + reason to specify this configuration error. \n Support: Core" + items: + description: HTTPRouteFilter defines processing steps that + must be completed during the request or response lifecycle. + HTTPRouteFilters are meant as an extension point to express + processing that may be done in Gateway implementations. + Some examples include request or response modification, + implementing authentication strategies, rate-limiting, and + traffic shaping. API guarantee/conformance is defined based + on the type of the filter. + properties: + extensionRef: + description: "ExtensionRef is an optional, implementation-specific + extension to the \"filter\" behavior. For example, + resource \"myroutefilter\" in group \"networking.example.net\"). + ExtensionRef MUST NOT be used for core and extended + filters. \n This filter can be used multiple times within + the same rule. \n Support: Implementation-specific" + properties: + group: + description: Group is the group of the referent. For + example, "gateway.networking.k8s.io". When unspecified + or empty string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For example + "HTTPRoute" or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + requestHeaderModifier: + description: "RequestHeaderModifier defines a schema for + a filter that modifies request headers. \n Support: + Core" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It appends + to any existing values associated with the header + name. \n Input: GET /foo HTTP/1.1 my-header: foo + \n Config: add: - name: \"my-header\" value: \"bar,baz\" + \n Output: GET /foo HTTP/1.1 my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from the + HTTP request before the action. The value of Remove + is a list of HTTP header names. Note that the header + names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo my-header2: + bar my-header3: baz \n Config: remove: [\"my-header1\", + \"my-header3\"] \n Output: GET /foo HTTP/1.1 my-header2: + bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with the + given header (name, value) before the action. \n + Input: GET /foo HTTP/1.1 my-header: foo \n Config: + set: - name: \"my-header\" value: \"bar\" \n Output: + GET /foo HTTP/1.1 my-header: bar" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + requestMirror: + description: "RequestMirror defines a schema for a filter + that mirrors requests. Requests are sent to the specified + destination, but responses from that destination are + ignored. \n This filter can be used multiple times within + the same rule. Note that not all implementations will + be able to support mirroring to multiple backends. \n + Support: Extended" + properties: + backendRef: + description: "BackendRef references a resource where + mirrored requests are sent. \n Mirrored requests + must be sent only to a single destination endpoint + within this BackendRef, irrespective of how many + endpoints are present within this BackendRef. \n + If the referent cannot be found, this BackendRef + is invalid and must be dropped from the Gateway. + The controller must ensure the \"ResolvedRefs\" + condition on the Route status is set to `status: + False` and not configure this backend in the underlying + implementation. \n If there is a cross-namespace + reference to an *existing* object that is not allowed + by a ReferenceGrant, the controller must ensure + the \"ResolvedRefs\" condition on the Route is + set to `status: False`, with the \"RefNotPermitted\" + reason and not configure this backend in the underlying + implementation. \n In either error case, the Message + of the `ResolvedRefs` Condition should be used to + provide more detail about the problem. \n Support: + Extended for Kubernetes Service \n Support: Implementation-specific + for any other resource" + properties: + group: + default: "" + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". When + unspecified or empty string, core API group + is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource + kind of the referent. For example \"Service\". + \n Defaults to \"Service\" when not specified. + \n ExternalName services can refer to CNAME + DNS records that may live outside of the cluster + and as such are difficult to reason about in + terms of conformance. They also may not be safe + to forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName + Services. \n Support: Core (Services with a + type other than ExternalName) \n Support: Implementation-specific + (Services with type ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the + backend. When unspecified, the local namespace + is inferred. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace + to allow that namespace's owner to accept the + reference. See the ReferenceGrant documentation + for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port + number to use for this resource. Port is required + when the referent is a Kubernetes Service. In + this case, the port number is the service port + number, not the target port. For other resources, + destination port might be derived from the referent + resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + required: + - backendRef + type: object + requestRedirect: + description: "RequestRedirect defines a schema for a filter + that responds to the request with an HTTP redirection. + \n Support: Core" + properties: + hostname: + description: "Hostname is the hostname to be used + in the value of the `Location` header in the response. + When empty, the hostname in the `Host` header of + the request is used. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines parameters used to modify + the path of the incoming request. The modified path + is then used to construct the `Location` header. + When empty, the request path is used as-is. \n Support: + Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the value + with which to replace the full path of a request + during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies the + value with which to replace the prefix match + of a request during a rewrite or redirect. For + example, a request to \"/foo/bar\" with a prefix + match of \"/foo\" and a ReplacePrefixMatch of + \"/xyz\" would be modified to \"/xyz/bar\". + \n Note that this matches the behavior of the + PathPrefix match type. This matches full path + elements. A path element refers to the list + of labels in the path split by the `/` separator. + When specified, a trailing `/` is ignored. For + example, the paths `/abc`, `/abc/`, and `/abc/def` + would all match the prefix `/abc`, but the path + `/abcd` would not. \n ReplacePrefixMatch is + only compatible with a `PathPrefix` HTTPRouteMatch. + Using any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the Route + to `status: False`. \n Request Path | Prefix + Match | Replace Prefix | Modified Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | /xyz/ + \ | /xyz/bar /foo/bar | /foo/ | + /xyz | /xyz/bar /foo/bar | /foo/ + \ | /xyz/ | /xyz/bar /foo | + /foo | /xyz | /xyz /foo/ | + /foo | /xyz | /xyz/ /foo/bar + \ | /foo | | /bar + /foo/ | /foo | + | / /foo | /foo | + | / /foo/ | /foo | / | + / /foo | /foo | / | + /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path modifier. + Additional types may be added in a future release + of the API. \n Note that values may be added + to this enum, implementations must ensure that + unknown values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the Route + to `status: False`, with a Reason of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified when + type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? has(self.replaceFullPath) + : true' + - message: type must be 'ReplaceFullPath' when replaceFullPath + is set + rule: 'has(self.replaceFullPath) ? self.type == + ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified when + type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' ? has(self.replacePrefixMatch) + : true' + - message: type must be 'ReplacePrefixMatch' when + replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + port: + description: "Port is the port to be used in the value + of the `Location` header in the response. \n If + no port is specified, the redirect port MUST be + derived using the following rules: \n * If redirect + scheme is not-empty, the redirect port MUST be the + well-known port associated with the redirect scheme. + Specifically \"http\" to port 80 and \"https\" to + port 443. If the redirect scheme does not have a + well-known port, the listener port of the Gateway + SHOULD be used. * If redirect scheme is empty, the + redirect port MUST be the Gateway Listener port. + \n Implementations SHOULD NOT add the port number + in the 'Location' header in the following cases: + \n * A Location header that will use HTTP (whether + that is determined via the Listener protocol or + the Scheme field) _and_ use port 80. * A Location + header that will use HTTPS (whether that is determined + via the Listener protocol or the Scheme field) _and_ + use port 443. \n Support: Extended" + format: int32 + maximum: 65535 + minimum: 1 + type: integer + scheme: + description: "Scheme is the scheme to be used in the + value of the `Location` header in the response. + When empty, the scheme of the request is used. \n + Scheme redirects can affect the port of the redirect, + for more information, refer to the documentation + for the port field of this filter. \n Note that + values may be added to this enum, implementations + must ensure that unknown values will not cause a + crash. \n Unknown values here must result in the + implementation setting the Accepted Condition for + the Route to `status: False`, with a Reason of `UnsupportedValue`. + \n Support: Extended" + enum: + - http + - https + type: string + statusCode: + default: 302 + description: "StatusCode is the HTTP status code to + be used in response. \n Note that values may be + added to this enum, implementations must ensure + that unknown values will not cause a crash. \n Unknown + values here must result in the implementation setting + the Accepted Condition for the Route to `status: + False`, with a Reason of `UnsupportedValue`. \n + Support: Core" + enum: + - 301 + - 302 + type: integer + type: object + responseHeaderModifier: + description: "ResponseHeaderModifier defines a schema + for a filter that modifies response headers. \n Support: + Extended" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It appends + to any existing values associated with the header + name. \n Input: GET /foo HTTP/1.1 my-header: foo + \n Config: add: - name: \"my-header\" value: \"bar,baz\" + \n Output: GET /foo HTTP/1.1 my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from the + HTTP request before the action. The value of Remove + is a list of HTTP header names. Note that the header + names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo my-header2: + bar my-header3: baz \n Config: remove: [\"my-header1\", + \"my-header3\"] \n Output: GET /foo HTTP/1.1 my-header2: + bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with the + given header (name, value) before the action. \n + Input: GET /foo HTTP/1.1 my-header: foo \n Config: + set: - name: \"my-header\" value: \"bar\" \n Output: + GET /foo HTTP/1.1 my-header: bar" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + type: + description: "Type identifies the type of filter to apply. + As with other API fields, types are classified into + three conformance levels: \n - Core: Filter types and + their corresponding configuration defined by \"Support: + Core\" in this package, e.g. \"RequestHeaderModifier\". + All implementations must support core filters. \n - + Extended: Filter types and their corresponding configuration + defined by \"Support: Extended\" in this package, e.g. + \"RequestMirror\". Implementers are encouraged to support + extended filters. \n - Implementation-specific: Filters + that are defined and supported by specific vendors. + In the future, filters showing convergence in behavior + across multiple implementations will be considered for + inclusion in extended or core conformance levels. Filter-specific + configuration for such filters is specified using the + ExtensionRef field. `Type` should be set to \"ExtensionRef\" + for custom filters. \n Implementers are encouraged to + define custom implementation types to extend the core + API with implementation-specific behavior. \n If a reference + to a custom filter type cannot be resolved, the filter + MUST NOT be skipped. Instead, requests that would have + been processed by that filter MUST receive a HTTP error + response. \n Note that values may be added to this enum, + implementations must ensure that unknown values will + not cause a crash. \n Unknown values here must result + in the implementation setting the Accepted Condition + for the Route to `status: False`, with a Reason of `UnsupportedValue`." + enum: + - RequestHeaderModifier + - ResponseHeaderModifier + - RequestMirror + - RequestRedirect + - URLRewrite + - ExtensionRef + type: string + urlRewrite: + description: "URLRewrite defines a schema for a filter + that modifies a request during forwarding. \n Support: + Extended" + properties: + hostname: + description: "Hostname is the value to be used to + replace the Host header value during forwarding. + \n Support: Extended" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines a path rewrite. \n Support: + Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the value + with which to replace the full path of a request + during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies the + value with which to replace the prefix match + of a request during a rewrite or redirect. For + example, a request to \"/foo/bar\" with a prefix + match of \"/foo\" and a ReplacePrefixMatch of + \"/xyz\" would be modified to \"/xyz/bar\". + \n Note that this matches the behavior of the + PathPrefix match type. This matches full path + elements. A path element refers to the list + of labels in the path split by the `/` separator. + When specified, a trailing `/` is ignored. For + example, the paths `/abc`, `/abc/`, and `/abc/def` + would all match the prefix `/abc`, but the path + `/abcd` would not. \n ReplacePrefixMatch is + only compatible with a `PathPrefix` HTTPRouteMatch. + Using any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the Route + to `status: False`. \n Request Path | Prefix + Match | Replace Prefix | Modified Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | /xyz/ + \ | /xyz/bar /foo/bar | /foo/ | + /xyz | /xyz/bar /foo/bar | /foo/ + \ | /xyz/ | /xyz/bar /foo | + /foo | /xyz | /xyz /foo/ | + /foo | /xyz | /xyz/ /foo/bar + \ | /foo | | /bar + /foo/ | /foo | + | / /foo | /foo | + | / /foo/ | /foo | / | + / /foo | /foo | / | + /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path modifier. + Additional types may be added in a future release + of the API. \n Note that values may be added + to this enum, implementations must ensure that + unknown values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the Route + to `status: False`, with a Reason of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified when + type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? has(self.replaceFullPath) + : true' + - message: type must be 'ReplaceFullPath' when replaceFullPath + is set + rule: 'has(self.replaceFullPath) ? self.type == + ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified when + type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' ? has(self.replacePrefixMatch) + : true' + - message: type must be 'ReplacePrefixMatch' when + replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + type: object + required: + - type + type: object + x-kubernetes-validations: + - message: filter.requestHeaderModifier must be nil if the + filter.type is not RequestHeaderModifier + rule: '!(has(self.requestHeaderModifier) && self.type != + ''RequestHeaderModifier'')' + - message: filter.requestHeaderModifier must be specified + for RequestHeaderModifier filter.type + rule: '!(!has(self.requestHeaderModifier) && self.type == + ''RequestHeaderModifier'')' + - message: filter.responseHeaderModifier must be nil if the + filter.type is not ResponseHeaderModifier + rule: '!(has(self.responseHeaderModifier) && self.type != + ''ResponseHeaderModifier'')' + - message: filter.responseHeaderModifier must be specified + for ResponseHeaderModifier filter.type + rule: '!(!has(self.responseHeaderModifier) && self.type + == ''ResponseHeaderModifier'')' + - message: filter.requestMirror must be nil if the filter.type + is not RequestMirror + rule: '!(has(self.requestMirror) && self.type != ''RequestMirror'')' + - message: filter.requestMirror must be specified for RequestMirror + filter.type + rule: '!(!has(self.requestMirror) && self.type == ''RequestMirror'')' + - message: filter.requestRedirect must be nil if the filter.type + is not RequestRedirect + rule: '!(has(self.requestRedirect) && self.type != ''RequestRedirect'')' + - message: filter.requestRedirect must be specified for RequestRedirect + filter.type + rule: '!(!has(self.requestRedirect) && self.type == ''RequestRedirect'')' + - message: filter.urlRewrite must be nil if the filter.type + is not URLRewrite + rule: '!(has(self.urlRewrite) && self.type != ''URLRewrite'')' + - message: filter.urlRewrite must be specified for URLRewrite + filter.type + rule: '!(!has(self.urlRewrite) && self.type == ''URLRewrite'')' + - message: filter.extensionRef must be nil if the filter.type + is not ExtensionRef + rule: '!(has(self.extensionRef) && self.type != ''ExtensionRef'')' + - message: filter.extensionRef must be specified for ExtensionRef + filter.type + rule: '!(!has(self.extensionRef) && self.type == ''ExtensionRef'')' + maxItems: 16 + type: array + x-kubernetes-validations: + - message: May specify either httpRouteFilterRequestRedirect + or httpRouteFilterRequestRewrite, but not both + rule: '!(self.exists(f, f.type == ''RequestRedirect'') && + self.exists(f, f.type == ''URLRewrite''))' + - message: RequestHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'RequestHeaderModifier').size() + <= 1 + - message: ResponseHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'ResponseHeaderModifier').size() + <= 1 + - message: RequestRedirect filter cannot be repeated + rule: self.filter(f, f.type == 'RequestRedirect').size() <= + 1 + - message: URLRewrite filter cannot be repeated + rule: self.filter(f, f.type == 'URLRewrite').size() <= 1 + matches: + default: + - path: + type: PathPrefix + value: / + description: "Matches define conditions used for matching the + rule against incoming HTTP requests. Each match is independent, + i.e. this rule will be matched if **any** one of the matches + is satisfied. \n For example, take the following matches configuration: + \n ``` matches: - path: value: \"/foo\" headers: - name: \"version\" + value: \"v2\" - path: value: \"/v2/foo\" ``` \n For a request + to match against this rule, a request must satisfy EITHER + of the two conditions: \n - path prefixed with `/foo` AND + contains the header `version: v2` - path prefix of `/v2/foo` + \n See the documentation for HTTPRouteMatch on how to specify + multiple match conditions that should be ANDed together. \n + If no matches are specified, the default is a prefix path + match on \"/\", which has the effect of matching every HTTP + request. \n Proxy or Load Balancer routing configuration generated + from HTTPRoutes MUST prioritize matches based on the following + criteria, continuing on ties. Across all rules specified on + applicable Routes, precedence must be given to the match having: + \n * \"Exact\" path match. * \"Prefix\" path match with largest + number of characters. * Method match. * Largest number of + header matches. * Largest number of query param matches. \n + Note: The precedence of RegularExpression path matches are + implementation-specific. \n If ties still exist across multiple + Routes, matching precedence MUST be determined in order of + the following criteria, continuing on ties: \n * The oldest + Route based on creation timestamp. * The Route appearing first + in alphabetical order by \"{namespace}/{name}\". \n If ties + still exist within an HTTPRoute, matching precedence MUST + be granted to the FIRST matching rule (in list order) with + a match meeting the above criteria. \n When no rules matching + a request have been successfully attached to the parent a + request is coming from, a HTTP 404 status code MUST be returned." + items: + description: "HTTPRouteMatch defines the predicate used to + match requests to a given action. Multiple match types are + ANDed together, i.e. the match will evaluate to true only + if all conditions are satisfied. \n For example, the match + below will match a HTTP request only if its path starts + with `/foo` AND it contains the `version: v1` header: \n + ``` match: \n path: value: \"/foo\" headers: - name: \"version\" + value \"v1\" \n ```" + properties: + headers: + description: Headers specifies HTTP request header matchers. + Multiple match values are ANDed together, meaning, a + request must match all the specified headers to select + the route. + items: + description: HTTPHeaderMatch describes how to select + a HTTP route by matching HTTP request headers. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case insensitive. + (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent header + names, only the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST be + ignored. Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered equivalent. + \n When a header is repeated in an HTTP request, + it is implementation-specific behavior as to how + this is represented. Generally, proxies should + follow the guidance from the RFC: https://www.rfc-editor.org/rfc/rfc7230.html#section-3.2.2 + regarding processing a repeated header, with special + handling for \"Set-Cookie\"." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + type: + default: Exact + description: "Type specifies how to match against + the value of the header. \n Support: Core (Exact) + \n Support: Implementation-specific (RegularExpression) + \n Since RegularExpression HeaderMatchType has + implementation-specific conformance, implementations + can support POSIX, PCRE or any other dialects + of regular expressions. Please read the implementation's + documentation to determine the supported dialect." + enum: + - Exact + - RegularExpression + type: string + value: + description: Value is the value of HTTP Header to + be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + method: + description: "Method specifies HTTP method matcher. When + specified, this route will be matched only if the request + has the specified method. \n Support: Extended" + enum: + - GET + - HEAD + - POST + - PUT + - DELETE + - CONNECT + - OPTIONS + - TRACE + - PATCH + type: string + path: + default: + type: PathPrefix + value: / + description: Path specifies a HTTP request path matcher. + If this field is not specified, a default prefix match + on the "/" path is provided. + properties: + type: + default: PathPrefix + description: "Type specifies how to match against + the path Value. \n Support: Core (Exact, PathPrefix) + \n Support: Implementation-specific (RegularExpression)" + enum: + - Exact + - PathPrefix + - RegularExpression + type: string + value: + default: / + description: Value of the HTTP path to match against. + maxLength: 1024 + type: string + type: object + x-kubernetes-validations: + - message: value must be an absolute path and start with + '/' when type one of ['Exact', 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? self.value.startsWith(''/'') + : true' + - message: must not contain '//' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''//'') + : true' + - message: must not contain '/./' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''/./'') + : true' + - message: must not contain '/../' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''/../'') + : true' + - message: must not contain '%2f' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''%2f'') + : true' + - message: must not contain '%2F' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''%2F'') + : true' + - message: must not contain '#' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''#'') + : true' + - message: must not end with '/..' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.endsWith(''/..'') + : true' + - message: must not end with '/.' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.endsWith(''/.'') + : true' + - message: type must be one of ['Exact', 'PathPrefix', + 'RegularExpression'] + rule: self.type in ['Exact','PathPrefix'] || self.type + == 'RegularExpression' + - message: must only contain valid characters (matching + ^(?:[-A-Za-z0-9/._~!$&'()*+,;=:@]|[%][0-9a-fA-F]{2})+$) + for types ['Exact', 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? self.value.matches(r"""^(?:[-A-Za-z0-9/._~!$&''()*+,;=:@]|[%][0-9a-fA-F]{2})+$""") + : true' + queryParams: + description: "QueryParams specifies HTTP query parameter + matchers. Multiple match values are ANDed together, + meaning, a request must match all the specified query + parameters to select the route. \n Support: Extended" + items: + description: HTTPQueryParamMatch describes how to select + a HTTP route by matching HTTP query parameters. + properties: + name: + description: "Name is the name of the HTTP query + param to be matched. This must be an exact string + match. (See https://tools.ietf.org/html/rfc7230#section-2.7.3). + \n If multiple entries specify equivalent query + param names, only the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent query param name MUST + be ignored. \n If a query param is repeated in + an HTTP request, the behavior is purposely left + undefined, since different data planes have different + capabilities. However, it is *recommended* that + implementations should match against the first + value of the param if the data plane supports + it, as this behavior is expected in other load + balancing contexts outside of the Gateway API. + \n Users SHOULD NOT route traffic based on repeated + query params to guard themselves against potential + differences in the implementations." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + type: + default: Exact + description: "Type specifies how to match against + the value of the query parameter. \n Support: + Extended (Exact) \n Support: Implementation-specific + (RegularExpression) \n Since RegularExpression + QueryParamMatchType has Implementation-specific + conformance, implementations can support POSIX, + PCRE or any other dialects of regular expressions. + Please read the implementation's documentation + to determine the supported dialect." + enum: + - Exact + - RegularExpression + type: string + value: + description: Value is the value of HTTP query param + to be matched. + maxLength: 1024 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + maxItems: 8 + type: array + type: object + x-kubernetes-validations: + - message: RequestRedirect filter must not be used together with + backendRefs + rule: '(has(self.backendRefs) && size(self.backendRefs) > 0) ? + (!has(self.filters) || self.filters.all(f, !has(f.requestRedirect))): + true' + - message: When using RequestRedirect filter with path.replacePrefixMatch, + exactly one PathPrefix match must be specified + rule: '(has(self.filters) && self.filters.exists_one(f, has(f.requestRedirect) + && has(f.requestRedirect.path) && f.requestRedirect.path.type + == ''ReplacePrefixMatch'' && has(f.requestRedirect.path.replacePrefixMatch))) + ? ((size(self.matches) != 1 || !has(self.matches[0].path) || + self.matches[0].path.type != ''PathPrefix'') ? false : true) + : true' + - message: When using URLRewrite filter with path.replacePrefixMatch, + exactly one PathPrefix match must be specified + rule: '(has(self.filters) && self.filters.exists_one(f, has(f.urlRewrite) + && has(f.urlRewrite.path) && f.urlRewrite.path.type == ''ReplacePrefixMatch'' + && has(f.urlRewrite.path.replacePrefixMatch))) ? ((size(self.matches) + != 1 || !has(self.matches[0].path) || self.matches[0].path.type + != ''PathPrefix'') ? false : true) : true' + - message: Within backendRefs, when using RequestRedirect filter + with path.replacePrefixMatch, exactly one PathPrefix match must + be specified + rule: '(has(self.backendRefs) && self.backendRefs.exists_one(b, + (has(b.filters) && b.filters.exists_one(f, has(f.requestRedirect) + && has(f.requestRedirect.path) && f.requestRedirect.path.type + == ''ReplacePrefixMatch'' && has(f.requestRedirect.path.replacePrefixMatch))) + )) ? ((size(self.matches) != 1 || !has(self.matches[0].path) + || self.matches[0].path.type != ''PathPrefix'') ? false : true) + : true' + - message: Within backendRefs, When using URLRewrite filter with + path.replacePrefixMatch, exactly one PathPrefix match must be + specified + rule: '(has(self.backendRefs) && self.backendRefs.exists_one(b, + (has(b.filters) && b.filters.exists_one(f, has(f.urlRewrite) + && has(f.urlRewrite.path) && f.urlRewrite.path.type == ''ReplacePrefixMatch'' + && has(f.urlRewrite.path.replacePrefixMatch))) )) ? ((size(self.matches) + != 1 || !has(self.matches[0].path) || self.matches[0].path.type + != ''PathPrefix'') ? false : true) : true' + maxItems: 16 + type: array + type: object + status: + description: Status defines the current state of HTTPRoute. + properties: + parents: + description: "Parents is a list of parent resources (usually Gateways) + that are associated with the route, and the status of the route + with respect to each parent. When this route attaches to a parent, + the controller that manages the parent must add an entry to this + list when the controller first sees the route and should update + the entry as appropriate when the route or gateway is modified. + \n Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this + API can only populate Route status for the Gateways/parent resources + they are responsible for. \n A maximum of 32 Gateways will be represented + in this list. An empty list means the route has not been attached + to any Gateway." + items: + description: RouteParentStatus describes the status of a route with + respect to an associated Parent. + properties: + conditions: + description: "Conditions describes the status of the route with + respect to the Gateway. Note that the route's availability + is also subject to the Gateway's own status conditions and + listener status. \n If the Route's ParentRef specifies an + existing Gateway that supports Routes of this kind AND that + Gateway's controller has sufficient access, then that Gateway's + controller MUST set the \"Accepted\" condition on the Route, + to indicate whether the route has been accepted or rejected + by the Gateway, and why. \n A Route MUST be considered \"Accepted\" + if at least one of the Route's rules is implemented by the + Gateway. \n There are a number of cases where the \"Accepted\" + condition may not be set due to lack of controller visibility, + that includes when: \n * The Route refers to a non-existent + parent. * The Route is of a type that the controller does + not support. * The Route is in a namespace the controller + does not have access to." + items: + description: "Condition contains details for one aspect of + the current state of this API Resource. --- This struct + is intended for direct use as an array at the field path + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition + `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should + be when the underlying condition changed. If that is + not known, then using the time when the API field changed + is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation + that the condition was set based upon. For instance, + if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration + is 9, the condition is out of date with respect to the + current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: reason contains a programmatic identifier + indicating the reason for the condition's last transition. + Producers of specific condition types may define expected + values and meanings for this field, and whether the + values are considered a guaranteed API. The value should + be a CamelCase string. This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + --- Many .condition.type values are consistent across + resources like Available, but because arbitrary conditions + can be useful (see .node.status.conditions), the ability + to deconflict is important. The regex it matches is + (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + controllerName: + description: "ControllerName is a domain/path string that indicates + the name of the controller that wrote this status. This corresponds + with the controllerName field on GatewayClass. \n Example: + \"example.net/gateway-controller\". \n The format of this + field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid + Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + \n Controllers MUST populate this field when writing status. + Controllers should ensure that entries to status populated + with their ControllerName are cleaned up when they are no + longer necessary." + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ + type: string + parentRef: + description: ParentRef corresponds with a ParentRef in the spec + that this RouteParentStatus struct describes the status of. + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the + core API group (such as for a \"Service\" kind referent), + Group must be explicitly set to \"\" (empty string). \n + Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. + When unspecified, this refers to the local namespace of + the Route. \n Note that there are specific rules for ParentRefs + which cross namespace boundaries. Cross-namespace references + are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides + a generic way to enable any other kind of cross-namespace + reference. \n ParentRefs from a Route to a Service in + the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any + namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for + which the intended destination of the connections are + a Service targeted as a ParentRef of the Route. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. + It can be interpreted differently based on the type of + parent resource. \n When the parent resource is a Gateway, + this targets all listeners listening on the specified + port that also support this kind of Route(and select this + Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to + a specific port as opposed to a listener(s) whose port(s) + may be changed. When both Port and SectionName are specified, + the name and port of the selected listener must match + both specified values. \n When the parent resource is + a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are + specified, the name and port of the selected port must + match both specified values. \n Implementations MAY choose + to support other parent resources. Implementations supporting + other types of parent resources MUST clearly document + how/if Port is interpreted. \n For the purpose of status, + an attachment is considered successful as long as the + parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them + by Route kind, namespace, or hostname. If 1 of 2 Gateway + listeners accept attachment from the referencing Route, + the Route MUST be considered successfully attached. If + no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within + the target resource. In the following resources, SectionName + is interpreted as the following: \n * Gateway: Listener + Name. When both Port (experimental) and SectionName are + specified, the name and port of the selected listener + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this + will reference the entire resource. For the purpose of + status, an attachment is considered successful if at least + one section in the parent resource accepts it. For example, + Gateway listeners can restrict which Routes can attach + to them by Route kind, namespace, or hostname. If 1 of + 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + required: + - controllerName + - parentRef + type: object + maxItems: 32 + type: array + required: + - parents + type: object + required: + - spec + type: object + served: false + storage: false + subresources: + status: {} + - additionalPrinterColumns: + - jsonPath: .spec.hostnames + name: Hostnames + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: HTTPRoute provides a way to route HTTP requests. This includes + the capability to match requests by hostname, path, header, or query param. + Filters can be used to specify additional processing steps. Backends specify + where matching requests should be routed. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of HTTPRoute. + properties: + hostnames: + description: "Hostnames defines a set of hostnames that should match + against the HTTP Host header to select a HTTPRoute used to process + the request. Implementations MUST ignore any port value specified + in the HTTP Host header while performing a match and (absent of + any applicable header modification configuration) MUST forward this + header unmodified to the backend. \n Valid values for Hostnames + are determined by RFC 1123 definition of a hostname with 2 notable + exceptions: \n 1. IPs are not allowed. 2. A hostname may be prefixed + with a wildcard label (`*.`). The wildcard label must appear by + itself as the first label. \n If a hostname is specified by both + the Listener and HTTPRoute, there must be at least one intersecting + hostname for the HTTPRoute to be attached to the Listener. For example: + \n * A Listener with `test.example.com` as the hostname matches + HTTPRoutes that have either not specified any hostnames, or have + specified at least one of `test.example.com` or `*.example.com`. + * A Listener with `*.example.com` as the hostname matches HTTPRoutes + that have either not specified any hostnames or have specified at + least one hostname that matches the Listener hostname. For example, + `*.example.com`, `test.example.com`, and `foo.test.example.com` + would all match. On the other hand, `example.com` and `test.example.net` + would not match. \n Hostnames that are prefixed with a wildcard + label (`*.`) are interpreted as a suffix match. That means that + a match for `*.example.com` would match both `test.example.com`, + and `foo.test.example.com`, but not `example.com`. \n If both the + Listener and HTTPRoute have specified hostnames, any HTTPRoute hostnames + that do not match the Listener hostname MUST be ignored. For example, + if a Listener specified `*.example.com`, and the HTTPRoute specified + `test.example.com` and `test.example.net`, `test.example.net` must + not be considered for a match. \n If both the Listener and HTTPRoute + have specified hostnames, and none match with the criteria above, + then the HTTPRoute is not accepted. The implementation must raise + an 'Accepted' Condition with a status of `False` in the corresponding + RouteParentStatus. \n In the event that multiple HTTPRoutes specify + intersecting hostnames (e.g. overlapping wildcard matching and exact + matching hostnames), precedence must be given to rules from the + HTTPRoute with the largest number of: \n * Characters in a matching + non-wildcard hostname. * Characters in a matching hostname. \n If + ties exist across multiple Routes, the matching precedence rules + for HTTPRouteMatches takes over. \n Support: Core" + items: + description: "Hostname is the fully qualified domain name of a network + host. This matches the RFC 1123 definition of a hostname with + 2 notable exceptions: \n 1. IPs are not allowed. 2. A hostname + may be prefixed with a wildcard label (`*.`). The wildcard label + must appear by itself as the first label. \n Hostname can be \"precise\" + which is a domain name without the terminating dot of a network + host (e.g. \"foo.example.com\") or \"wildcard\", which is a domain + name prefixed with a single wildcard label (e.g. `*.example.com`). + \n Note that as per RFC1035 and RFC1123, a *label* must consist + of lower case alphanumeric characters or '-', and must start and + end with an alphanumeric character. No other punctuation is allowed." + maxLength: 253 + minLength: 1 + pattern: ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + maxItems: 16 + type: array + parentRefs: + description: "ParentRefs references the resources (usually Gateways) + that a Route wants to be attached to. Note that the referenced parent + resource needs to allow this for the attachment to be complete. + For Gateways, that means the Gateway needs to allow attachment from + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) \n This + API may be extended in the future to support additional kinds of + parent resources. \n It is invalid to reference an identical parent + more than once. It is valid to reference multiple distinct sections + within the same parent resource, such as two separate Listeners + on the same Gateway or two separate ports on the same Service. \n + It is possible to separately reference multiple distinct objects + that may be collapsed by an implementation. For example, some implementations + may choose to merge compatible Gateway Listeners together. If that + is the case, the list of routes attached to those resources should + also be merged. \n Note that for ParentRefs that cross namespace + boundaries, there are specific rules. Cross-namespace references + are only valid if they are explicitly allowed by something in the + namespace they are referring to. For example, Gateway has the AllowedRoutes + field, and ReferenceGrant provides a generic way to enable other + kinds of cross-namespace reference. \n ParentRefs from a Route to + a Service in the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any namespace + to the Service. \n ParentRefs from a Route to a Service in a different + namespace are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same namespace + as the Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. \n " + items: + description: "ParentReference identifies an API object (usually + a Gateway) that can be considered a parent of this resource (usually + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the core + API group (such as for a \"Service\" kind referent), Group + must be explicitly set to \"\" (empty string). \n Support: + Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. When + unspecified, this refers to the local namespace of the Route. + \n Note that there are specific rules for ParentRefs which + cross namespace boundaries. Cross-namespace references are + only valid if they are explicitly allowed by something in + the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + \n ParentRefs from a Route to a Service in the same namespace + are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. + \n ParentRefs from a Route to a Service in a different namespace + are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same + namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of + the Route. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. It + can be interpreted differently based on the type of parent + resource. \n When the parent resource is a Gateway, this targets + all listeners listening on the specified port that also support + this kind of Route(and select this Route). It's not recommended + to set `Port` unless the networking behaviors specified in + a Route must apply to a specific port as opposed to a listener(s) + whose port(s) may be changed. When both Port and SectionName + are specified, the name and port of the selected listener + must match both specified values. \n When the parent resource + is a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified + values. \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes can + attach to them by Route kind, namespace, or hostname. If 1 + of 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. \n + Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within the + target resource. In the following resources, SectionName is + interpreted as the following: \n * Gateway: Listener Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match both + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + maxItems: 32 + type: array + x-kubernetes-validations: + - message: sectionName or port must be specified when parentRefs includes + 2 or more references to the same parent + rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ + == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) + && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) + || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName + != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) + && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) + ) ) ): true ))' + - message: sectionName or port must be unique when parentRefs includes + 2 or more references to the same parent + rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) + || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ + == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) + || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName + == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName + == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port + == p2.port)))) + rules: + default: + - matches: + - path: + type: PathPrefix + value: / + description: Rules are a list of HTTP matchers, filters and actions. + items: + description: HTTPRouteRule defines semantics for matching an HTTP + request based on conditions (matches), processing it (filters), + and forwarding the request to an API object (backendRefs). + properties: + backendRefs: + description: "BackendRefs defines the backend(s) where matching + requests should be sent. \n Failure behavior here depends + on how many BackendRefs are specified and how many are invalid. + \n If *all* entries in BackendRefs are invalid, and there + are also no filters specified in this route rule, *all* traffic + which matches this rule MUST receive a 500 status code. \n + See the HTTPBackendRef definition for the rules about what + makes a single HTTPBackendRef invalid. \n When a HTTPBackendRef + is invalid, 500 status codes MUST be returned for requests + that would have otherwise been routed to an invalid backend. + If multiple backends are specified, and some are invalid, + the proportion of requests that would otherwise have been + routed to an invalid backend MUST receive a 500 status code. + \n For example, if two backends are specified with equal weights, + and one is invalid, 50 percent of traffic must receive a 500. + Implementations may choose how that 50 percent is determined. + \n Support: Core for Kubernetes Service \n Support: Extended + for Kubernetes ServiceImport \n Support: Implementation-specific + for any other resource \n Support for weight: Core" + items: + description: HTTPBackendRef defines how a HTTPRoute should + forward an HTTP request. + properties: + filters: + description: "Filters defined at this level should be + executed if and only if the request is being forwarded + to the backend defined here. \n Support: Implementation-specific + (For broader support of filters, use the Filters field + in HTTPRouteRule.)" + items: + description: HTTPRouteFilter defines processing steps + that must be completed during the request or response + lifecycle. HTTPRouteFilters are meant as an extension + point to express processing that may be done in Gateway + implementations. Some examples include request or + response modification, implementing authentication + strategies, rate-limiting, and traffic shaping. API + guarantee/conformance is defined based on the type + of the filter. + properties: + extensionRef: + description: "ExtensionRef is an optional, implementation-specific + extension to the \"filter\" behavior. For example, + resource \"myroutefilter\" in group \"networking.example.net\"). + ExtensionRef MUST NOT be used for core and extended + filters. \n This filter can be used multiple times + within the same rule. \n Support: Implementation-specific" + properties: + group: + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API + group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For + example "HTTPRoute" or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + requestHeaderModifier: + description: "RequestHeaderModifier defines a schema + for a filter that modifies request headers. \n + Support: Core" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It + appends to any existing values associated + with the header name. \n Input: GET /foo HTTP/1.1 + my-header: foo \n Config: add: - name: \"my-header\" + value: \"bar,baz\" \n Output: GET /foo HTTP/1.1 + my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from + the HTTP request before the action. The value + of Remove is a list of HTTP header names. + Note that the header names are case-insensitive + (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo + my-header2: bar my-header3: baz \n Config: + remove: [\"my-header1\", \"my-header3\"] \n + Output: GET /foo HTTP/1.1 my-header2: bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with + the given header (name, value) before the + action. \n Input: GET /foo HTTP/1.1 my-header: + foo \n Config: set: - name: \"my-header\" + value: \"bar\" \n Output: GET /foo HTTP/1.1 + my-header: bar" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + requestMirror: + description: "RequestMirror defines a schema for + a filter that mirrors requests. Requests are sent + to the specified destination, but responses from + that destination are ignored. \n This filter can + be used multiple times within the same rule. Note + that not all implementations will be able to support + mirroring to multiple backends. \n Support: Extended" + properties: + backendRef: + description: "BackendRef references a resource + where mirrored requests are sent. \n Mirrored + requests must be sent only to a single destination + endpoint within this BackendRef, irrespective + of how many endpoints are present within this + BackendRef. \n If the referent cannot be found, + this BackendRef is invalid and must be dropped + from the Gateway. The controller must ensure + the \"ResolvedRefs\" condition on the Route + status is set to `status: False` and not configure + this backend in the underlying implementation. + \n If there is a cross-namespace reference + to an *existing* object that is not allowed + by a ReferenceGrant, the controller must ensure + the \"ResolvedRefs\" condition on the Route + is set to `status: False`, with the \"RefNotPermitted\" + reason and not configure this backend in the + underlying implementation. \n In either error + case, the Message of the `ResolvedRefs` Condition + should be used to provide more detail about + the problem. \n Support: Extended for Kubernetes + Service \n Support: Implementation-specific + for any other resource" + properties: + group: + default: "" + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". + When unspecified or empty string, core + API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource + kind of the referent. For example \"Service\". + \n Defaults to \"Service\" when not specified. + \n ExternalName services can refer to + CNAME DNS records that may live outside + of the cluster and as such are difficult + to reason about in terms of conformance. + They also may not be safe to forward to + (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName + Services. \n Support: Core (Services with + a type other than ExternalName) \n Support: + Implementation-specific (Services with + type ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace + of the backend. When unspecified, the + local namespace is inferred. \n Note that + when a namespace different than the local + namespace is specified, a ReferenceGrant + object is required in the referent namespace + to allow that namespace's owner to accept + the reference. See the ReferenceGrant + documentation for details. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination + port number to use for this resource. + Port is required when the referent is + a Kubernetes Service. In this case, the + port number is the service port number, + not the target port. For other resources, + destination port might be derived from + the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind + == ''Service'') ? has(self.port) : true' + required: + - backendRef + type: object + requestRedirect: + description: "RequestRedirect defines a schema for + a filter that responds to the request with an + HTTP redirection. \n Support: Core" + properties: + hostname: + description: "Hostname is the hostname to be + used in the value of the `Location` header + in the response. When empty, the hostname + in the `Host` header of the request is used. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines parameters used to + modify the path of the incoming request. The + modified path is then used to construct the + `Location` header. When empty, the request + path is used as-is. \n Support: Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the + value with which to replace the full path + of a request during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies + the value with which to replace the prefix + match of a request during a rewrite or + redirect. For example, a request to \"/foo/bar\" + with a prefix match of \"/foo\" and a + ReplacePrefixMatch of \"/xyz\" would be + modified to \"/xyz/bar\". \n Note that + this matches the behavior of the PathPrefix + match type. This matches full path elements. + A path element refers to the list of labels + in the path split by the `/` separator. + When specified, a trailing `/` is ignored. + For example, the paths `/abc`, `/abc/`, + and `/abc/def` would all match the prefix + `/abc`, but the path `/abcd` would not. + \n ReplacePrefixMatch is only compatible + with a `PathPrefix` HTTPRouteMatch. Using + any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the + Route to `status: False`. \n Request Path + | Prefix Match | Replace Prefix | Modified + Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | + /xyz/ | /xyz/bar /foo/bar | + /foo/ | /xyz | /xyz/bar + /foo/bar | /foo/ | /xyz/ | + /xyz/bar /foo | /foo | + /xyz | /xyz /foo/ | /foo + \ | /xyz | /xyz/ /foo/bar + \ | /foo | | + /bar /foo/ | /foo | | / /foo | /foo | + | / /foo/ | /foo + \ | / | / /foo | + /foo | / | /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path + modifier. Additional types may be added + in a future release of the API. \n Note + that values may be added to this enum, + implementations must ensure that unknown + values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the + Route to `status: False`, with a Reason + of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified + when type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? + has(self.replaceFullPath) : true' + - message: type must be 'ReplaceFullPath' when + replaceFullPath is set + rule: 'has(self.replaceFullPath) ? self.type + == ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified + when type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' + ? has(self.replacePrefixMatch) : true' + - message: type must be 'ReplacePrefixMatch' + when replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + port: + description: "Port is the port to be used in + the value of the `Location` header in the + response. \n If no port is specified, the + redirect port MUST be derived using the following + rules: \n * If redirect scheme is not-empty, + the redirect port MUST be the well-known port + associated with the redirect scheme. Specifically + \"http\" to port 80 and \"https\" to port + 443. If the redirect scheme does not have + a well-known port, the listener port of the + Gateway SHOULD be used. * If redirect scheme + is empty, the redirect port MUST be the Gateway + Listener port. \n Implementations SHOULD NOT + add the port number in the 'Location' header + in the following cases: \n * A Location header + that will use HTTP (whether that is determined + via the Listener protocol or the Scheme field) + _and_ use port 80. * A Location header that + will use HTTPS (whether that is determined + via the Listener protocol or the Scheme field) + _and_ use port 443. \n Support: Extended" + format: int32 + maximum: 65535 + minimum: 1 + type: integer + scheme: + description: "Scheme is the scheme to be used + in the value of the `Location` header in the + response. When empty, the scheme of the request + is used. \n Scheme redirects can affect the + port of the redirect, for more information, + refer to the documentation for the port field + of this filter. \n Note that values may be + added to this enum, implementations must ensure + that unknown values will not cause a crash. + \n Unknown values here must result in the + implementation setting the Accepted Condition + for the Route to `status: False`, with a Reason + of `UnsupportedValue`. \n Support: Extended" + enum: + - http + - https + type: string + statusCode: + default: 302 + description: "StatusCode is the HTTP status + code to be used in response. \n Note that + values may be added to this enum, implementations + must ensure that unknown values will not cause + a crash. \n Unknown values here must result + in the implementation setting the Accepted + Condition for the Route to `status: False`, + with a Reason of `UnsupportedValue`. \n Support: + Core" + enum: + - 301 + - 302 + type: integer + type: object + responseHeaderModifier: + description: "ResponseHeaderModifier defines a schema + for a filter that modifies response headers. \n + Support: Extended" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It + appends to any existing values associated + with the header name. \n Input: GET /foo HTTP/1.1 + my-header: foo \n Config: add: - name: \"my-header\" + value: \"bar,baz\" \n Output: GET /foo HTTP/1.1 + my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from + the HTTP request before the action. The value + of Remove is a list of HTTP header names. + Note that the header names are case-insensitive + (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo + my-header2: bar my-header3: baz \n Config: + remove: [\"my-header1\", \"my-header3\"] \n + Output: GET /foo HTTP/1.1 my-header2: bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with + the given header (name, value) before the + action. \n Input: GET /foo HTTP/1.1 my-header: + foo \n Config: set: - name: \"my-header\" + value: \"bar\" \n Output: GET /foo HTTP/1.1 + my-header: bar" + items: + description: HTTPHeader represents an HTTP + Header name and value as defined by RFC + 7230. + properties: + name: + description: "Name is the name of the + HTTP Header to be matched. Name matching + MUST be case insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an + equivalent name MUST be considered for + a match. Subsequent entries with an + equivalent header name MUST be ignored. + Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP + Header to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + type: + description: "Type identifies the type of filter + to apply. As with other API fields, types are + classified into three conformance levels: \n - + Core: Filter types and their corresponding configuration + defined by \"Support: Core\" in this package, + e.g. \"RequestHeaderModifier\". All implementations + must support core filters. \n - Extended: Filter + types and their corresponding configuration defined + by \"Support: Extended\" in this package, e.g. + \"RequestMirror\". Implementers are encouraged + to support extended filters. \n - Implementation-specific: + Filters that are defined and supported by specific + vendors. In the future, filters showing convergence + in behavior across multiple implementations will + be considered for inclusion in extended or core + conformance levels. Filter-specific configuration + for such filters is specified using the ExtensionRef + field. `Type` should be set to \"ExtensionRef\" + for custom filters. \n Implementers are encouraged + to define custom implementation types to extend + the core API with implementation-specific behavior. + \n If a reference to a custom filter type cannot + be resolved, the filter MUST NOT be skipped. Instead, + requests that would have been processed by that + filter MUST receive a HTTP error response. \n + Note that values may be added to this enum, implementations + must ensure that unknown values will not cause + a crash. \n Unknown values here must result in + the implementation setting the Accepted Condition + for the Route to `status: False`, with a Reason + of `UnsupportedValue`." + enum: + - RequestHeaderModifier + - ResponseHeaderModifier + - RequestMirror + - RequestRedirect + - URLRewrite + - ExtensionRef + type: string + urlRewrite: + description: "URLRewrite defines a schema for a + filter that modifies a request during forwarding. + \n Support: Extended" + properties: + hostname: + description: "Hostname is the value to be used + to replace the Host header value during forwarding. + \n Support: Extended" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines a path rewrite. \n + Support: Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the + value with which to replace the full path + of a request during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies + the value with which to replace the prefix + match of a request during a rewrite or + redirect. For example, a request to \"/foo/bar\" + with a prefix match of \"/foo\" and a + ReplacePrefixMatch of \"/xyz\" would be + modified to \"/xyz/bar\". \n Note that + this matches the behavior of the PathPrefix + match type. This matches full path elements. + A path element refers to the list of labels + in the path split by the `/` separator. + When specified, a trailing `/` is ignored. + For example, the paths `/abc`, `/abc/`, + and `/abc/def` would all match the prefix + `/abc`, but the path `/abcd` would not. + \n ReplacePrefixMatch is only compatible + with a `PathPrefix` HTTPRouteMatch. Using + any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the + Route to `status: False`. \n Request Path + | Prefix Match | Replace Prefix | Modified + Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | + /xyz/ | /xyz/bar /foo/bar | + /foo/ | /xyz | /xyz/bar + /foo/bar | /foo/ | /xyz/ | + /xyz/bar /foo | /foo | + /xyz | /xyz /foo/ | /foo + \ | /xyz | /xyz/ /foo/bar + \ | /foo | | + /bar /foo/ | /foo | | / /foo | /foo | + | / /foo/ | /foo + \ | / | / /foo | + /foo | / | /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path + modifier. Additional types may be added + in a future release of the API. \n Note + that values may be added to this enum, + implementations must ensure that unknown + values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the + Route to `status: False`, with a Reason + of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified + when type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? + has(self.replaceFullPath) : true' + - message: type must be 'ReplaceFullPath' when + replaceFullPath is set + rule: 'has(self.replaceFullPath) ? self.type + == ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified + when type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' + ? has(self.replacePrefixMatch) : true' + - message: type must be 'ReplacePrefixMatch' + when replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + type: object + required: + - type + type: object + x-kubernetes-validations: + - message: filter.requestHeaderModifier must be nil + if the filter.type is not RequestHeaderModifier + rule: '!(has(self.requestHeaderModifier) && self.type + != ''RequestHeaderModifier'')' + - message: filter.requestHeaderModifier must be specified + for RequestHeaderModifier filter.type + rule: '!(!has(self.requestHeaderModifier) && self.type + == ''RequestHeaderModifier'')' + - message: filter.responseHeaderModifier must be nil + if the filter.type is not ResponseHeaderModifier + rule: '!(has(self.responseHeaderModifier) && self.type + != ''ResponseHeaderModifier'')' + - message: filter.responseHeaderModifier must be specified + for ResponseHeaderModifier filter.type + rule: '!(!has(self.responseHeaderModifier) && self.type + == ''ResponseHeaderModifier'')' + - message: filter.requestMirror must be nil if the filter.type + is not RequestMirror + rule: '!(has(self.requestMirror) && self.type != ''RequestMirror'')' + - message: filter.requestMirror must be specified for + RequestMirror filter.type + rule: '!(!has(self.requestMirror) && self.type == + ''RequestMirror'')' + - message: filter.requestRedirect must be nil if the + filter.type is not RequestRedirect + rule: '!(has(self.requestRedirect) && self.type != + ''RequestRedirect'')' + - message: filter.requestRedirect must be specified + for RequestRedirect filter.type + rule: '!(!has(self.requestRedirect) && self.type == + ''RequestRedirect'')' + - message: filter.urlRewrite must be nil if the filter.type + is not URLRewrite + rule: '!(has(self.urlRewrite) && self.type != ''URLRewrite'')' + - message: filter.urlRewrite must be specified for URLRewrite + filter.type + rule: '!(!has(self.urlRewrite) && self.type == ''URLRewrite'')' + - message: filter.extensionRef must be nil if the filter.type + is not ExtensionRef + rule: '!(has(self.extensionRef) && self.type != ''ExtensionRef'')' + - message: filter.extensionRef must be specified for + ExtensionRef filter.type + rule: '!(!has(self.extensionRef) && self.type == ''ExtensionRef'')' + maxItems: 16 + type: array + x-kubernetes-validations: + - message: May specify either httpRouteFilterRequestRedirect + or httpRouteFilterRequestRewrite, but not both + rule: '!(self.exists(f, f.type == ''RequestRedirect'') + && self.exists(f, f.type == ''URLRewrite''))' + - message: May specify either httpRouteFilterRequestRedirect + or httpRouteFilterRequestRewrite, but not both + rule: '!(self.exists(f, f.type == ''RequestRedirect'') + && self.exists(f, f.type == ''URLRewrite''))' + - message: RequestHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'RequestHeaderModifier').size() + <= 1 + - message: ResponseHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'ResponseHeaderModifier').size() + <= 1 + - message: RequestRedirect filter cannot be repeated + rule: self.filter(f, f.type == 'RequestRedirect').size() + <= 1 + - message: URLRewrite filter cannot be repeated + rule: self.filter(f, f.type == 'URLRewrite').size() + <= 1 + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty + string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource kind of + the referent. For example \"Service\". \n Defaults to + \"Service\" when not specified. \n ExternalName services + can refer to CNAME DNS records that may live outside + of the cluster and as such are difficult to reason about + in terms of conformance. They also may not be safe to + forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName Services. + \n Support: Core (Services with a type other than ExternalName) + \n Support: Implementation-specific (Services with type + ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the backend. + When unspecified, the local namespace is inferred. \n + Note that when a namespace different than the local + namespace is specified, a ReferenceGrant object is required + in the referent namespace to allow that namespace's + owner to accept the reference. See the ReferenceGrant + documentation for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port number + to use for this resource. Port is required when the + referent is a Kubernetes Service. In this case, the + port number is the service port number, not the target + port. For other resources, destination port might be + derived from the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + weight: + default: 1 + description: "Weight specifies the proportion of requests + forwarded to the referenced backend. This is computed + as weight/(sum of all weights in this BackendRefs list). + For non-zero values, there may be some epsilon from + the exact proportion defined here depending on the precision + an implementation supports. Weight is not a percentage + and the sum of weights does not need to equal 100. \n + If only one backend is specified and it has a weight + greater than 0, 100% of the traffic is forwarded to + that backend. If weight is set to 0, no traffic should + be forwarded for this entry. If unspecified, weight + defaults to 1. \n Support for this field varies based + on the context where used." + format: int32 + maximum: 1000000 + minimum: 0 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + maxItems: 16 + type: array + filters: + description: "Filters define the filters that are applied to + requests that match this rule. \n The effects of ordering + of multiple behaviors are currently unspecified. This can + change in the future based on feedback during the alpha stage. + \n Conformance-levels at this level are defined based on the + type of filter: \n - ALL core filters MUST be supported by + all implementations. - Implementers are encouraged to support + extended filters. - Implementation-specific custom filters + have no API guarantees across implementations. \n Specifying + the same filter multiple times is not supported unless explicitly + indicated in the filter. \n All filters are expected to be + compatible with each other except for the URLRewrite and RequestRedirect + filters, which may not be combined. If an implementation can + not support other combinations of filters, they must clearly + document that limitation. In cases where incompatible or unsupported + filters are specified and cause the `Accepted` condition to + be set to status `False`, implementations may use the `IncompatibleFilters` + reason to specify this configuration error. \n Support: Core" + items: + description: HTTPRouteFilter defines processing steps that + must be completed during the request or response lifecycle. + HTTPRouteFilters are meant as an extension point to express + processing that may be done in Gateway implementations. + Some examples include request or response modification, + implementing authentication strategies, rate-limiting, and + traffic shaping. API guarantee/conformance is defined based + on the type of the filter. + properties: + extensionRef: + description: "ExtensionRef is an optional, implementation-specific + extension to the \"filter\" behavior. For example, + resource \"myroutefilter\" in group \"networking.example.net\"). + ExtensionRef MUST NOT be used for core and extended + filters. \n This filter can be used multiple times within + the same rule. \n Support: Implementation-specific" + properties: + group: + description: Group is the group of the referent. For + example, "gateway.networking.k8s.io". When unspecified + or empty string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: Kind is kind of the referent. For example + "HTTPRoute" or "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + - name + type: object + requestHeaderModifier: + description: "RequestHeaderModifier defines a schema for + a filter that modifies request headers. \n Support: + Core" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It appends + to any existing values associated with the header + name. \n Input: GET /foo HTTP/1.1 my-header: foo + \n Config: add: - name: \"my-header\" value: \"bar,baz\" + \n Output: GET /foo HTTP/1.1 my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from the + HTTP request before the action. The value of Remove + is a list of HTTP header names. Note that the header + names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo my-header2: + bar my-header3: baz \n Config: remove: [\"my-header1\", + \"my-header3\"] \n Output: GET /foo HTTP/1.1 my-header2: + bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with the + given header (name, value) before the action. \n + Input: GET /foo HTTP/1.1 my-header: foo \n Config: + set: - name: \"my-header\" value: \"bar\" \n Output: + GET /foo HTTP/1.1 my-header: bar" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + requestMirror: + description: "RequestMirror defines a schema for a filter + that mirrors requests. Requests are sent to the specified + destination, but responses from that destination are + ignored. \n This filter can be used multiple times within + the same rule. Note that not all implementations will + be able to support mirroring to multiple backends. \n + Support: Extended" + properties: + backendRef: + description: "BackendRef references a resource where + mirrored requests are sent. \n Mirrored requests + must be sent only to a single destination endpoint + within this BackendRef, irrespective of how many + endpoints are present within this BackendRef. \n + If the referent cannot be found, this BackendRef + is invalid and must be dropped from the Gateway. + The controller must ensure the \"ResolvedRefs\" + condition on the Route status is set to `status: + False` and not configure this backend in the underlying + implementation. \n If there is a cross-namespace + reference to an *existing* object that is not allowed + by a ReferenceGrant, the controller must ensure + the \"ResolvedRefs\" condition on the Route is + set to `status: False`, with the \"RefNotPermitted\" + reason and not configure this backend in the underlying + implementation. \n In either error case, the Message + of the `ResolvedRefs` Condition should be used to + provide more detail about the problem. \n Support: + Extended for Kubernetes Service \n Support: Implementation-specific + for any other resource" + properties: + group: + default: "" + description: Group is the group of the referent. + For example, "gateway.networking.k8s.io". When + unspecified or empty string, core API group + is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource + kind of the referent. For example \"Service\". + \n Defaults to \"Service\" when not specified. + \n ExternalName services can refer to CNAME + DNS records that may live outside of the cluster + and as such are difficult to reason about in + terms of conformance. They also may not be safe + to forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName + Services. \n Support: Core (Services with a + type other than ExternalName) \n Support: Implementation-specific + (Services with type ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the + backend. When unspecified, the local namespace + is inferred. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace + to allow that namespace's owner to accept the + reference. See the ReferenceGrant documentation + for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port + number to use for this resource. Port is required + when the referent is a Kubernetes Service. In + this case, the port number is the service port + number, not the target port. For other resources, + destination port might be derived from the referent + resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + required: + - backendRef + type: object + requestRedirect: + description: "RequestRedirect defines a schema for a filter + that responds to the request with an HTTP redirection. + \n Support: Core" + properties: + hostname: + description: "Hostname is the hostname to be used + in the value of the `Location` header in the response. + When empty, the hostname in the `Host` header of + the request is used. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines parameters used to modify + the path of the incoming request. The modified path + is then used to construct the `Location` header. + When empty, the request path is used as-is. \n Support: + Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the value + with which to replace the full path of a request + during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies the + value with which to replace the prefix match + of a request during a rewrite or redirect. For + example, a request to \"/foo/bar\" with a prefix + match of \"/foo\" and a ReplacePrefixMatch of + \"/xyz\" would be modified to \"/xyz/bar\". + \n Note that this matches the behavior of the + PathPrefix match type. This matches full path + elements. A path element refers to the list + of labels in the path split by the `/` separator. + When specified, a trailing `/` is ignored. For + example, the paths `/abc`, `/abc/`, and `/abc/def` + would all match the prefix `/abc`, but the path + `/abcd` would not. \n ReplacePrefixMatch is + only compatible with a `PathPrefix` HTTPRouteMatch. + Using any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the Route + to `status: False`. \n Request Path | Prefix + Match | Replace Prefix | Modified Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | /xyz/ + \ | /xyz/bar /foo/bar | /foo/ | + /xyz | /xyz/bar /foo/bar | /foo/ + \ | /xyz/ | /xyz/bar /foo | + /foo | /xyz | /xyz /foo/ | + /foo | /xyz | /xyz/ /foo/bar + \ | /foo | | /bar + /foo/ | /foo | + | / /foo | /foo | + | / /foo/ | /foo | / | + / /foo | /foo | / | + /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path modifier. + Additional types may be added in a future release + of the API. \n Note that values may be added + to this enum, implementations must ensure that + unknown values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the Route + to `status: False`, with a Reason of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified when + type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? has(self.replaceFullPath) + : true' + - message: type must be 'ReplaceFullPath' when replaceFullPath + is set + rule: 'has(self.replaceFullPath) ? self.type == + ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified when + type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' ? has(self.replacePrefixMatch) + : true' + - message: type must be 'ReplacePrefixMatch' when + replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + port: + description: "Port is the port to be used in the value + of the `Location` header in the response. \n If + no port is specified, the redirect port MUST be + derived using the following rules: \n * If redirect + scheme is not-empty, the redirect port MUST be the + well-known port associated with the redirect scheme. + Specifically \"http\" to port 80 and \"https\" to + port 443. If the redirect scheme does not have a + well-known port, the listener port of the Gateway + SHOULD be used. * If redirect scheme is empty, the + redirect port MUST be the Gateway Listener port. + \n Implementations SHOULD NOT add the port number + in the 'Location' header in the following cases: + \n * A Location header that will use HTTP (whether + that is determined via the Listener protocol or + the Scheme field) _and_ use port 80. * A Location + header that will use HTTPS (whether that is determined + via the Listener protocol or the Scheme field) _and_ + use port 443. \n Support: Extended" + format: int32 + maximum: 65535 + minimum: 1 + type: integer + scheme: + description: "Scheme is the scheme to be used in the + value of the `Location` header in the response. + When empty, the scheme of the request is used. \n + Scheme redirects can affect the port of the redirect, + for more information, refer to the documentation + for the port field of this filter. \n Note that + values may be added to this enum, implementations + must ensure that unknown values will not cause a + crash. \n Unknown values here must result in the + implementation setting the Accepted Condition for + the Route to `status: False`, with a Reason of `UnsupportedValue`. + \n Support: Extended" + enum: + - http + - https + type: string + statusCode: + default: 302 + description: "StatusCode is the HTTP status code to + be used in response. \n Note that values may be + added to this enum, implementations must ensure + that unknown values will not cause a crash. \n Unknown + values here must result in the implementation setting + the Accepted Condition for the Route to `status: + False`, with a Reason of `UnsupportedValue`. \n + Support: Core" + enum: + - 301 + - 302 + type: integer + type: object + responseHeaderModifier: + description: "ResponseHeaderModifier defines a schema + for a filter that modifies response headers. \n Support: + Extended" + properties: + add: + description: "Add adds the given header(s) (name, + value) to the request before the action. It appends + to any existing values associated with the header + name. \n Input: GET /foo HTTP/1.1 my-header: foo + \n Config: add: - name: \"my-header\" value: \"bar,baz\" + \n Output: GET /foo HTTP/1.1 my-header: foo,bar,baz" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + remove: + description: "Remove the given header(s) from the + HTTP request before the action. The value of Remove + is a list of HTTP header names. Note that the header + names are case-insensitive (see https://datatracker.ietf.org/doc/html/rfc2616#section-4.2). + \n Input: GET /foo HTTP/1.1 my-header1: foo my-header2: + bar my-header3: baz \n Config: remove: [\"my-header1\", + \"my-header3\"] \n Output: GET /foo HTTP/1.1 my-header2: + bar" + items: + type: string + maxItems: 16 + type: array + x-kubernetes-list-type: set + set: + description: "Set overwrites the request with the + given header (name, value) before the action. \n + Input: GET /foo HTTP/1.1 my-header: foo \n Config: + set: - name: \"my-header\" value: \"bar\" \n Output: + GET /foo HTTP/1.1 my-header: bar" + items: + description: HTTPHeader represents an HTTP Header + name and value as defined by RFC 7230. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case + insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent + header names, the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST + be ignored. Due to the case-insensitivity + of header names, \"foo\" and \"Foo\" are considered + equivalent." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + value: + description: Value is the value of HTTP Header + to be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + type: + description: "Type identifies the type of filter to apply. + As with other API fields, types are classified into + three conformance levels: \n - Core: Filter types and + their corresponding configuration defined by \"Support: + Core\" in this package, e.g. \"RequestHeaderModifier\". + All implementations must support core filters. \n - + Extended: Filter types and their corresponding configuration + defined by \"Support: Extended\" in this package, e.g. + \"RequestMirror\". Implementers are encouraged to support + extended filters. \n - Implementation-specific: Filters + that are defined and supported by specific vendors. + In the future, filters showing convergence in behavior + across multiple implementations will be considered for + inclusion in extended or core conformance levels. Filter-specific + configuration for such filters is specified using the + ExtensionRef field. `Type` should be set to \"ExtensionRef\" + for custom filters. \n Implementers are encouraged to + define custom implementation types to extend the core + API with implementation-specific behavior. \n If a reference + to a custom filter type cannot be resolved, the filter + MUST NOT be skipped. Instead, requests that would have + been processed by that filter MUST receive a HTTP error + response. \n Note that values may be added to this enum, + implementations must ensure that unknown values will + not cause a crash. \n Unknown values here must result + in the implementation setting the Accepted Condition + for the Route to `status: False`, with a Reason of `UnsupportedValue`." + enum: + - RequestHeaderModifier + - ResponseHeaderModifier + - RequestMirror + - RequestRedirect + - URLRewrite + - ExtensionRef + type: string + urlRewrite: + description: "URLRewrite defines a schema for a filter + that modifies a request during forwarding. \n Support: + Extended" + properties: + hostname: + description: "Hostname is the value to be used to + replace the Host header value during forwarding. + \n Support: Extended" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + path: + description: "Path defines a path rewrite. \n Support: + Extended" + properties: + replaceFullPath: + description: ReplaceFullPath specifies the value + with which to replace the full path of a request + during a rewrite or redirect. + maxLength: 1024 + type: string + replacePrefixMatch: + description: "ReplacePrefixMatch specifies the + value with which to replace the prefix match + of a request during a rewrite or redirect. For + example, a request to \"/foo/bar\" with a prefix + match of \"/foo\" and a ReplacePrefixMatch of + \"/xyz\" would be modified to \"/xyz/bar\". + \n Note that this matches the behavior of the + PathPrefix match type. This matches full path + elements. A path element refers to the list + of labels in the path split by the `/` separator. + When specified, a trailing `/` is ignored. For + example, the paths `/abc`, `/abc/`, and `/abc/def` + would all match the prefix `/abc`, but the path + `/abcd` would not. \n ReplacePrefixMatch is + only compatible with a `PathPrefix` HTTPRouteMatch. + Using any other HTTPRouteMatch type on the same + HTTPRouteRule will result in the implementation + setting the Accepted Condition for the Route + to `status: False`. \n Request Path | Prefix + Match | Replace Prefix | Modified Path -------------|--------------|----------------|---------- + /foo/bar | /foo | /xyz | + /xyz/bar /foo/bar | /foo | /xyz/ + \ | /xyz/bar /foo/bar | /foo/ | + /xyz | /xyz/bar /foo/bar | /foo/ + \ | /xyz/ | /xyz/bar /foo | + /foo | /xyz | /xyz /foo/ | + /foo | /xyz | /xyz/ /foo/bar + \ | /foo | | /bar + /foo/ | /foo | + | / /foo | /foo | + | / /foo/ | /foo | / | + / /foo | /foo | / | + /" + maxLength: 1024 + type: string + type: + description: "Type defines the type of path modifier. + Additional types may be added in a future release + of the API. \n Note that values may be added + to this enum, implementations must ensure that + unknown values will not cause a crash. \n Unknown + values here must result in the implementation + setting the Accepted Condition for the Route + to `status: False`, with a Reason of `UnsupportedValue`." + enum: + - ReplaceFullPath + - ReplacePrefixMatch + type: string + required: + - type + type: object + x-kubernetes-validations: + - message: replaceFullPath must be specified when + type is set to 'ReplaceFullPath' + rule: 'self.type == ''ReplaceFullPath'' ? has(self.replaceFullPath) + : true' + - message: type must be 'ReplaceFullPath' when replaceFullPath + is set + rule: 'has(self.replaceFullPath) ? self.type == + ''ReplaceFullPath'' : true' + - message: replacePrefixMatch must be specified when + type is set to 'ReplacePrefixMatch' + rule: 'self.type == ''ReplacePrefixMatch'' ? has(self.replacePrefixMatch) + : true' + - message: type must be 'ReplacePrefixMatch' when + replacePrefixMatch is set + rule: 'has(self.replacePrefixMatch) ? self.type + == ''ReplacePrefixMatch'' : true' + type: object + required: + - type + type: object + x-kubernetes-validations: + - message: filter.requestHeaderModifier must be nil if the + filter.type is not RequestHeaderModifier + rule: '!(has(self.requestHeaderModifier) && self.type != + ''RequestHeaderModifier'')' + - message: filter.requestHeaderModifier must be specified + for RequestHeaderModifier filter.type + rule: '!(!has(self.requestHeaderModifier) && self.type == + ''RequestHeaderModifier'')' + - message: filter.responseHeaderModifier must be nil if the + filter.type is not ResponseHeaderModifier + rule: '!(has(self.responseHeaderModifier) && self.type != + ''ResponseHeaderModifier'')' + - message: filter.responseHeaderModifier must be specified + for ResponseHeaderModifier filter.type + rule: '!(!has(self.responseHeaderModifier) && self.type + == ''ResponseHeaderModifier'')' + - message: filter.requestMirror must be nil if the filter.type + is not RequestMirror + rule: '!(has(self.requestMirror) && self.type != ''RequestMirror'')' + - message: filter.requestMirror must be specified for RequestMirror + filter.type + rule: '!(!has(self.requestMirror) && self.type == ''RequestMirror'')' + - message: filter.requestRedirect must be nil if the filter.type + is not RequestRedirect + rule: '!(has(self.requestRedirect) && self.type != ''RequestRedirect'')' + - message: filter.requestRedirect must be specified for RequestRedirect + filter.type + rule: '!(!has(self.requestRedirect) && self.type == ''RequestRedirect'')' + - message: filter.urlRewrite must be nil if the filter.type + is not URLRewrite + rule: '!(has(self.urlRewrite) && self.type != ''URLRewrite'')' + - message: filter.urlRewrite must be specified for URLRewrite + filter.type + rule: '!(!has(self.urlRewrite) && self.type == ''URLRewrite'')' + - message: filter.extensionRef must be nil if the filter.type + is not ExtensionRef + rule: '!(has(self.extensionRef) && self.type != ''ExtensionRef'')' + - message: filter.extensionRef must be specified for ExtensionRef + filter.type + rule: '!(!has(self.extensionRef) && self.type == ''ExtensionRef'')' + maxItems: 16 + type: array + x-kubernetes-validations: + - message: May specify either httpRouteFilterRequestRedirect + or httpRouteFilterRequestRewrite, but not both + rule: '!(self.exists(f, f.type == ''RequestRedirect'') && + self.exists(f, f.type == ''URLRewrite''))' + - message: RequestHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'RequestHeaderModifier').size() + <= 1 + - message: ResponseHeaderModifier filter cannot be repeated + rule: self.filter(f, f.type == 'ResponseHeaderModifier').size() + <= 1 + - message: RequestRedirect filter cannot be repeated + rule: self.filter(f, f.type == 'RequestRedirect').size() <= + 1 + - message: URLRewrite filter cannot be repeated + rule: self.filter(f, f.type == 'URLRewrite').size() <= 1 + matches: + default: + - path: + type: PathPrefix + value: / + description: "Matches define conditions used for matching the + rule against incoming HTTP requests. Each match is independent, + i.e. this rule will be matched if **any** one of the matches + is satisfied. \n For example, take the following matches configuration: + \n ``` matches: - path: value: \"/foo\" headers: - name: \"version\" + value: \"v2\" - path: value: \"/v2/foo\" ``` \n For a request + to match against this rule, a request must satisfy EITHER + of the two conditions: \n - path prefixed with `/foo` AND + contains the header `version: v2` - path prefix of `/v2/foo` + \n See the documentation for HTTPRouteMatch on how to specify + multiple match conditions that should be ANDed together. \n + If no matches are specified, the default is a prefix path + match on \"/\", which has the effect of matching every HTTP + request. \n Proxy or Load Balancer routing configuration generated + from HTTPRoutes MUST prioritize matches based on the following + criteria, continuing on ties. Across all rules specified on + applicable Routes, precedence must be given to the match having: + \n * \"Exact\" path match. * \"Prefix\" path match with largest + number of characters. * Method match. * Largest number of + header matches. * Largest number of query param matches. \n + Note: The precedence of RegularExpression path matches are + implementation-specific. \n If ties still exist across multiple + Routes, matching precedence MUST be determined in order of + the following criteria, continuing on ties: \n * The oldest + Route based on creation timestamp. * The Route appearing first + in alphabetical order by \"{namespace}/{name}\". \n If ties + still exist within an HTTPRoute, matching precedence MUST + be granted to the FIRST matching rule (in list order) with + a match meeting the above criteria. \n When no rules matching + a request have been successfully attached to the parent a + request is coming from, a HTTP 404 status code MUST be returned." + items: + description: "HTTPRouteMatch defines the predicate used to + match requests to a given action. Multiple match types are + ANDed together, i.e. the match will evaluate to true only + if all conditions are satisfied. \n For example, the match + below will match a HTTP request only if its path starts + with `/foo` AND it contains the `version: v1` header: \n + ``` match: \n path: value: \"/foo\" headers: - name: \"version\" + value \"v1\" \n ```" + properties: + headers: + description: Headers specifies HTTP request header matchers. + Multiple match values are ANDed together, meaning, a + request must match all the specified headers to select + the route. + items: + description: HTTPHeaderMatch describes how to select + a HTTP route by matching HTTP request headers. + properties: + name: + description: "Name is the name of the HTTP Header + to be matched. Name matching MUST be case insensitive. + (See https://tools.ietf.org/html/rfc7230#section-3.2). + \n If multiple entries specify equivalent header + names, only the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent header name MUST be + ignored. Due to the case-insensitivity of header + names, \"foo\" and \"Foo\" are considered equivalent. + \n When a header is repeated in an HTTP request, + it is implementation-specific behavior as to how + this is represented. Generally, proxies should + follow the guidance from the RFC: https://www.rfc-editor.org/rfc/rfc7230.html#section-3.2.2 + regarding processing a repeated header, with special + handling for \"Set-Cookie\"." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + type: + default: Exact + description: "Type specifies how to match against + the value of the header. \n Support: Core (Exact) + \n Support: Implementation-specific (RegularExpression) + \n Since RegularExpression HeaderMatchType has + implementation-specific conformance, implementations + can support POSIX, PCRE or any other dialects + of regular expressions. Please read the implementation's + documentation to determine the supported dialect." + enum: + - Exact + - RegularExpression + type: string + value: + description: Value is the value of HTTP Header to + be matched. + maxLength: 4096 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + method: + description: "Method specifies HTTP method matcher. When + specified, this route will be matched only if the request + has the specified method. \n Support: Extended" + enum: + - GET + - HEAD + - POST + - PUT + - DELETE + - CONNECT + - OPTIONS + - TRACE + - PATCH + type: string + path: + default: + type: PathPrefix + value: / + description: Path specifies a HTTP request path matcher. + If this field is not specified, a default prefix match + on the "/" path is provided. + properties: + type: + default: PathPrefix + description: "Type specifies how to match against + the path Value. \n Support: Core (Exact, PathPrefix) + \n Support: Implementation-specific (RegularExpression)" + enum: + - Exact + - PathPrefix + - RegularExpression + type: string + value: + default: / + description: Value of the HTTP path to match against. + maxLength: 1024 + type: string + type: object + x-kubernetes-validations: + - message: value must be an absolute path and start with + '/' when type one of ['Exact', 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? self.value.startsWith(''/'') + : true' + - message: must not contain '//' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''//'') + : true' + - message: must not contain '/./' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''/./'') + : true' + - message: must not contain '/../' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''/../'') + : true' + - message: must not contain '%2f' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''%2f'') + : true' + - message: must not contain '%2F' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''%2F'') + : true' + - message: must not contain '#' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.contains(''#'') + : true' + - message: must not end with '/..' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.endsWith(''/..'') + : true' + - message: must not end with '/.' when type one of ['Exact', + 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? !self.value.endsWith(''/.'') + : true' + - message: type must be one of ['Exact', 'PathPrefix', + 'RegularExpression'] + rule: self.type in ['Exact','PathPrefix'] || self.type + == 'RegularExpression' + - message: must only contain valid characters (matching + ^(?:[-A-Za-z0-9/._~!$&'()*+,;=:@]|[%][0-9a-fA-F]{2})+$) + for types ['Exact', 'PathPrefix'] + rule: '(self.type in [''Exact'',''PathPrefix'']) ? self.value.matches(r"""^(?:[-A-Za-z0-9/._~!$&''()*+,;=:@]|[%][0-9a-fA-F]{2})+$""") + : true' + queryParams: + description: "QueryParams specifies HTTP query parameter + matchers. Multiple match values are ANDed together, + meaning, a request must match all the specified query + parameters to select the route. \n Support: Extended" + items: + description: HTTPQueryParamMatch describes how to select + a HTTP route by matching HTTP query parameters. + properties: + name: + description: "Name is the name of the HTTP query + param to be matched. This must be an exact string + match. (See https://tools.ietf.org/html/rfc7230#section-2.7.3). + \n If multiple entries specify equivalent query + param names, only the first entry with an equivalent + name MUST be considered for a match. Subsequent + entries with an equivalent query param name MUST + be ignored. \n If a query param is repeated in + an HTTP request, the behavior is purposely left + undefined, since different data planes have different + capabilities. However, it is *recommended* that + implementations should match against the first + value of the param if the data plane supports + it, as this behavior is expected in other load + balancing contexts outside of the Gateway API. + \n Users SHOULD NOT route traffic based on repeated + query params to guard themselves against potential + differences in the implementations." + maxLength: 256 + minLength: 1 + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + type: + default: Exact + description: "Type specifies how to match against + the value of the query parameter. \n Support: + Extended (Exact) \n Support: Implementation-specific + (RegularExpression) \n Since RegularExpression + QueryParamMatchType has Implementation-specific + conformance, implementations can support POSIX, + PCRE or any other dialects of regular expressions. + Please read the implementation's documentation + to determine the supported dialect." + enum: + - Exact + - RegularExpression + type: string + value: + description: Value is the value of HTTP query param + to be matched. + maxLength: 1024 + minLength: 1 + type: string + required: + - name + - value + type: object + maxItems: 16 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + type: object + maxItems: 8 + type: array + type: object + x-kubernetes-validations: + - message: RequestRedirect filter must not be used together with + backendRefs + rule: '(has(self.backendRefs) && size(self.backendRefs) > 0) ? + (!has(self.filters) || self.filters.all(f, !has(f.requestRedirect))): + true' + - message: When using RequestRedirect filter with path.replacePrefixMatch, + exactly one PathPrefix match must be specified + rule: '(has(self.filters) && self.filters.exists_one(f, has(f.requestRedirect) + && has(f.requestRedirect.path) && f.requestRedirect.path.type + == ''ReplacePrefixMatch'' && has(f.requestRedirect.path.replacePrefixMatch))) + ? ((size(self.matches) != 1 || !has(self.matches[0].path) || + self.matches[0].path.type != ''PathPrefix'') ? false : true) + : true' + - message: When using URLRewrite filter with path.replacePrefixMatch, + exactly one PathPrefix match must be specified + rule: '(has(self.filters) && self.filters.exists_one(f, has(f.urlRewrite) + && has(f.urlRewrite.path) && f.urlRewrite.path.type == ''ReplacePrefixMatch'' + && has(f.urlRewrite.path.replacePrefixMatch))) ? ((size(self.matches) + != 1 || !has(self.matches[0].path) || self.matches[0].path.type + != ''PathPrefix'') ? false : true) : true' + - message: Within backendRefs, when using RequestRedirect filter + with path.replacePrefixMatch, exactly one PathPrefix match must + be specified + rule: '(has(self.backendRefs) && self.backendRefs.exists_one(b, + (has(b.filters) && b.filters.exists_one(f, has(f.requestRedirect) + && has(f.requestRedirect.path) && f.requestRedirect.path.type + == ''ReplacePrefixMatch'' && has(f.requestRedirect.path.replacePrefixMatch))) + )) ? ((size(self.matches) != 1 || !has(self.matches[0].path) + || self.matches[0].path.type != ''PathPrefix'') ? false : true) + : true' + - message: Within backendRefs, When using URLRewrite filter with + path.replacePrefixMatch, exactly one PathPrefix match must be + specified + rule: '(has(self.backendRefs) && self.backendRefs.exists_one(b, + (has(b.filters) && b.filters.exists_one(f, has(f.urlRewrite) + && has(f.urlRewrite.path) && f.urlRewrite.path.type == ''ReplacePrefixMatch'' + && has(f.urlRewrite.path.replacePrefixMatch))) )) ? ((size(self.matches) + != 1 || !has(self.matches[0].path) || self.matches[0].path.type + != ''PathPrefix'') ? false : true) : true' + maxItems: 16 + type: array + type: object + status: + description: Status defines the current state of HTTPRoute. + properties: + parents: + description: "Parents is a list of parent resources (usually Gateways) + that are associated with the route, and the status of the route + with respect to each parent. When this route attaches to a parent, + the controller that manages the parent must add an entry to this + list when the controller first sees the route and should update + the entry as appropriate when the route or gateway is modified. + \n Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this + API can only populate Route status for the Gateways/parent resources + they are responsible for. \n A maximum of 32 Gateways will be represented + in this list. An empty list means the route has not been attached + to any Gateway." + items: + description: RouteParentStatus describes the status of a route with + respect to an associated Parent. + properties: + conditions: + description: "Conditions describes the status of the route with + respect to the Gateway. Note that the route's availability + is also subject to the Gateway's own status conditions and + listener status. \n If the Route's ParentRef specifies an + existing Gateway that supports Routes of this kind AND that + Gateway's controller has sufficient access, then that Gateway's + controller MUST set the \"Accepted\" condition on the Route, + to indicate whether the route has been accepted or rejected + by the Gateway, and why. \n A Route MUST be considered \"Accepted\" + if at least one of the Route's rules is implemented by the + Gateway. \n There are a number of cases where the \"Accepted\" + condition may not be set due to lack of controller visibility, + that includes when: \n * The Route refers to a non-existent + parent. * The Route is of a type that the controller does + not support. * The Route is in a namespace the controller + does not have access to." + items: + description: "Condition contains details for one aspect of + the current state of this API Resource. --- This struct + is intended for direct use as an array at the field path + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition + `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should + be when the underlying condition changed. If that is + not known, then using the time when the API field changed + is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation + that the condition was set based upon. For instance, + if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration + is 9, the condition is out of date with respect to the + current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: reason contains a programmatic identifier + indicating the reason for the condition's last transition. + Producers of specific condition types may define expected + values and meanings for this field, and whether the + values are considered a guaranteed API. The value should + be a CamelCase string. This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + --- Many .condition.type values are consistent across + resources like Available, but because arbitrary conditions + can be useful (see .node.status.conditions), the ability + to deconflict is important. The regex it matches is + (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + controllerName: + description: "ControllerName is a domain/path string that indicates + the name of the controller that wrote this status. This corresponds + with the controllerName field on GatewayClass. \n Example: + \"example.net/gateway-controller\". \n The format of this + field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid + Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + \n Controllers MUST populate this field when writing status. + Controllers should ensure that entries to status populated + with their ControllerName are cleaned up when they are no + longer necessary." + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ + type: string + parentRef: + description: ParentRef corresponds with a ParentRef in the spec + that this RouteParentStatus struct describes the status of. + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the + core API group (such as for a \"Service\" kind referent), + Group must be explicitly set to \"\" (empty string). \n + Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. + When unspecified, this refers to the local namespace of + the Route. \n Note that there are specific rules for ParentRefs + which cross namespace boundaries. Cross-namespace references + are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides + a generic way to enable any other kind of cross-namespace + reference. \n ParentRefs from a Route to a Service in + the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any + namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for + which the intended destination of the connections are + a Service targeted as a ParentRef of the Route. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. + It can be interpreted differently based on the type of + parent resource. \n When the parent resource is a Gateway, + this targets all listeners listening on the specified + port that also support this kind of Route(and select this + Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to + a specific port as opposed to a listener(s) whose port(s) + may be changed. When both Port and SectionName are specified, + the name and port of the selected listener must match + both specified values. \n When the parent resource is + a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are + specified, the name and port of the selected port must + match both specified values. \n Implementations MAY choose + to support other parent resources. Implementations supporting + other types of parent resources MUST clearly document + how/if Port is interpreted. \n For the purpose of status, + an attachment is considered successful as long as the + parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them + by Route kind, namespace, or hostname. If 1 of 2 Gateway + listeners accept attachment from the referencing Route, + the Route MUST be considered successfully attached. If + no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within + the target resource. In the following resources, SectionName + is interpreted as the following: \n * Gateway: Listener + Name. When both Port (experimental) and SectionName are + specified, the name and port of the selected listener + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this + will reference the entire resource. For the purpose of + status, an attachment is considered successful if at least + one section in the parent resource accepts it. For example, + Gateway listeners can restrict which Routes can attach + to them by Route kind, namespace, or hostname. If 1 of + 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + required: + - controllerName + - parentRef + type: object + maxItems: 32 + type: array + required: + - parents + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} +status: + acceptedNames: + kind: "" + plural: "" + conditions: null + storedVersions: null +--- +# +# config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml +# +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 + gateway.networking.k8s.io/channel: experimental + creationTimestamp: null + name: referencegrants.gateway.networking.k8s.io +spec: + group: gateway.networking.k8s.io + names: + categories: + - gateway-api + kind: ReferenceGrant + listKind: ReferenceGrantList + plural: referencegrants + shortNames: + - refgrant + singular: referencegrant + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + deprecated: true + deprecationWarning: The v1alpha2 version of ReferenceGrant has been deprecated + and will be removed in a future release of the API. Please upgrade to v1beta1. + name: v1alpha2 + schema: + openAPIV3Schema: + description: "ReferenceGrant identifies kinds of resources in other namespaces + that are trusted to reference the specified kinds of resources in the same + namespace as the policy. \n Each ReferenceGrant can be used to represent + a unique trust relationship. Additional Reference Grants can be used to + add to the set of trusted sources of inbound references for the namespace + they are defined within. \n A ReferenceGrant is required for all cross-namespace + references in Gateway API (with the exception of cross-namespace Route-Gateway + attachment, which is governed by the AllowedRoutes configuration on the + Gateway, and cross-namespace Service ParentRefs on a \"consumer\" mesh Route, + which defines routing rules applicable only to workloads in the Route namespace). + ReferenceGrants allowing a reference from a Route to a Service are only + applicable to BackendRefs. \n ReferenceGrant is a form of runtime verification + allowing users to assert which cross-namespace object references are permitted. + Implementations that support ReferenceGrant MUST NOT permit cross-namespace + references which have no grant, and MUST respond to the removal of a grant + by revoking the access that the grant allowed." + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of ReferenceGrant. + properties: + from: + description: "From describes the trusted namespaces and kinds that + can reference the resources described in \"To\". Each entry in this + list MUST be considered to be an additional place that references + can be valid from, or to put this another way, entries MUST be combined + using OR. \n Support: Core" + items: + description: ReferenceGrantFrom describes trusted namespaces and + kinds. + properties: + group: + description: "Group is the group of the referent. When empty, + the Kubernetes core API group is inferred. \n Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: "Kind is the kind of the referent. Although implementations + may support additional resources, the following types are + part of the \"Core\" support level for this field. \n When + used to permit a SecretObjectReference: \n * Gateway \n When + used to permit a BackendObjectReference: \n * GRPCRoute * + HTTPRoute * TCPRoute * TLSRoute * UDPRoute" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + namespace: + description: "Namespace is the namespace of the referent. \n + Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + required: + - group + - kind + - namespace + type: object + maxItems: 16 + minItems: 1 + type: array + to: + description: "To describes the resources that may be referenced by + the resources described in \"From\". Each entry in this list MUST + be considered to be an additional place that references can be valid + to, or to put this another way, entries MUST be combined using OR. + \n Support: Core" + items: + description: ReferenceGrantTo describes what Kinds are allowed as + targets of the references. + properties: + group: + description: "Group is the group of the referent. When empty, + the Kubernetes core API group is inferred. \n Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: "Kind is the kind of the referent. Although implementations + may support additional resources, the following types are + part of the \"Core\" support level for this field: \n * Secret + when used to permit a SecretObjectReference * Service when + used to permit a BackendObjectReference" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. When unspecified, + this policy refers to all resources of the specified Group + and Kind in the local namespace. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + type: object + maxItems: 16 + minItems: 1 + type: array + required: + - from + - to + type: object + type: object + served: true + storage: false + subresources: {} + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: "ReferenceGrant identifies kinds of resources in other namespaces + that are trusted to reference the specified kinds of resources in the same + namespace as the policy. \n Each ReferenceGrant can be used to represent + a unique trust relationship. Additional Reference Grants can be used to + add to the set of trusted sources of inbound references for the namespace + they are defined within. \n All cross-namespace references in Gateway API + (with the exception of cross-namespace Gateway-route attachment) require + a ReferenceGrant. \n ReferenceGrant is a form of runtime verification allowing + users to assert which cross-namespace object references are permitted. Implementations + that support ReferenceGrant MUST NOT permit cross-namespace references which + have no grant, and MUST respond to the removal of a grant by revoking the + access that the grant allowed." + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of ReferenceGrant. + properties: + from: + description: "From describes the trusted namespaces and kinds that + can reference the resources described in \"To\". Each entry in this + list MUST be considered to be an additional place that references + can be valid from, or to put this another way, entries MUST be combined + using OR. \n Support: Core" + items: + description: ReferenceGrantFrom describes trusted namespaces and + kinds. + properties: + group: + description: "Group is the group of the referent. When empty, + the Kubernetes core API group is inferred. \n Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: "Kind is the kind of the referent. Although implementations + may support additional resources, the following types are + part of the \"Core\" support level for this field. \n When + used to permit a SecretObjectReference: \n * Gateway \n When + used to permit a BackendObjectReference: \n * GRPCRoute * + HTTPRoute * TCPRoute * TLSRoute * UDPRoute" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + namespace: + description: "Namespace is the namespace of the referent. \n + Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + required: + - group + - kind + - namespace + type: object + maxItems: 16 + minItems: 1 + type: array + to: + description: "To describes the resources that may be referenced by + the resources described in \"From\". Each entry in this list MUST + be considered to be an additional place that references can be valid + to, or to put this another way, entries MUST be combined using OR. + \n Support: Core" + items: + description: ReferenceGrantTo describes what Kinds are allowed as + targets of the references. + properties: + group: + description: "Group is the group of the referent. When empty, + the Kubernetes core API group is inferred. \n Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + description: "Kind is the kind of the referent. Although implementations + may support additional resources, the following types are + part of the \"Core\" support level for this field: \n * Secret + when used to permit a SecretObjectReference * Service when + used to permit a BackendObjectReference" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. When unspecified, + this policy refers to all resources of the specified Group + and Kind in the local namespace. + maxLength: 253 + minLength: 1 + type: string + required: + - group + - kind + type: object + maxItems: 16 + minItems: 1 + type: array + required: + - from + - to + type: object + type: object + served: true + storage: true + subresources: {} +status: + acceptedNames: + kind: "" + plural: "" + conditions: null + storedVersions: null +--- +# +# config/crd/experimental/gateway.networking.k8s.io_tcproutes.yaml +# +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 + gateway.networking.k8s.io/channel: experimental + creationTimestamp: null + name: tcproutes.gateway.networking.k8s.io +spec: + group: gateway.networking.k8s.io + names: + categories: + - gateway-api + kind: TCPRoute + listKind: TCPRouteList + plural: tcproutes + singular: tcproute + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha2 + schema: + openAPIV3Schema: + description: TCPRoute provides a way to route TCP requests. When combined + with a Gateway listener, it can be used to forward connections on the port + specified by the listener to a set of backends specified by the TCPRoute. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of TCPRoute. + properties: + parentRefs: + description: "ParentRefs references the resources (usually Gateways) + that a Route wants to be attached to. Note that the referenced parent + resource needs to allow this for the attachment to be complete. + For Gateways, that means the Gateway needs to allow attachment from + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) \n This + API may be extended in the future to support additional kinds of + parent resources. \n It is invalid to reference an identical parent + more than once. It is valid to reference multiple distinct sections + within the same parent resource, such as two separate Listeners + on the same Gateway or two separate ports on the same Service. \n + It is possible to separately reference multiple distinct objects + that may be collapsed by an implementation. For example, some implementations + may choose to merge compatible Gateway Listeners together. If that + is the case, the list of routes attached to those resources should + also be merged. \n Note that for ParentRefs that cross namespace + boundaries, there are specific rules. Cross-namespace references + are only valid if they are explicitly allowed by something in the + namespace they are referring to. For example, Gateway has the AllowedRoutes + field, and ReferenceGrant provides a generic way to enable other + kinds of cross-namespace reference. \n ParentRefs from a Route to + a Service in the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any namespace + to the Service. \n ParentRefs from a Route to a Service in a different + namespace are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same namespace + as the Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. \n " + items: + description: "ParentReference identifies an API object (usually + a Gateway) that can be considered a parent of this resource (usually + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the core + API group (such as for a \"Service\" kind referent), Group + must be explicitly set to \"\" (empty string). \n Support: + Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. When + unspecified, this refers to the local namespace of the Route. + \n Note that there are specific rules for ParentRefs which + cross namespace boundaries. Cross-namespace references are + only valid if they are explicitly allowed by something in + the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + \n ParentRefs from a Route to a Service in the same namespace + are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. + \n ParentRefs from a Route to a Service in a different namespace + are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same + namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of + the Route. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. It + can be interpreted differently based on the type of parent + resource. \n When the parent resource is a Gateway, this targets + all listeners listening on the specified port that also support + this kind of Route(and select this Route). It's not recommended + to set `Port` unless the networking behaviors specified in + a Route must apply to a specific port as opposed to a listener(s) + whose port(s) may be changed. When both Port and SectionName + are specified, the name and port of the selected listener + must match both specified values. \n When the parent resource + is a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified + values. \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes can + attach to them by Route kind, namespace, or hostname. If 1 + of 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. \n + Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within the + target resource. In the following resources, SectionName is + interpreted as the following: \n * Gateway: Listener Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match both + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + maxItems: 32 + type: array + x-kubernetes-validations: + - message: sectionName or port must be specified when parentRefs includes + 2 or more references to the same parent + rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ + == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) + && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) + || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName + != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) + && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) + ) ) ): true ))' + - message: sectionName or port must be unique when parentRefs includes + 2 or more references to the same parent + rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) + || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ + == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) + || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName + == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName + == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port + == p2.port)))) + rules: + description: Rules are a list of TCP matchers and actions. + items: + description: TCPRouteRule is the configuration for a given rule. + properties: + backendRefs: + description: "BackendRefs defines the backend(s) where matching + requests should be sent. If unspecified or invalid (refers + to a non-existent resource or a Service with no endpoints), + the underlying implementation MUST actively reject connection + attempts to this backend. Connection rejections must respect + weight; if an invalid backend is requested to have 80% of + connections, then 80% of connections must be rejected instead. + \n Support: Core for Kubernetes Service \n Support: Extended + for Kubernetes ServiceImport \n Support: Implementation-specific + for any other resource \n Support for weight: Extended" + items: + description: "BackendRef defines how a Route should forward + a request to a Kubernetes resource. \n Note that when a + namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace + to allow that namespace's owner to accept the reference. + See the ReferenceGrant documentation for details." + properties: + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty + string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource kind of + the referent. For example \"Service\". \n Defaults to + \"Service\" when not specified. \n ExternalName services + can refer to CNAME DNS records that may live outside + of the cluster and as such are difficult to reason about + in terms of conformance. They also may not be safe to + forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName Services. + \n Support: Core (Services with a type other than ExternalName) + \n Support: Implementation-specific (Services with type + ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the backend. + When unspecified, the local namespace is inferred. \n + Note that when a namespace different than the local + namespace is specified, a ReferenceGrant object is required + in the referent namespace to allow that namespace's + owner to accept the reference. See the ReferenceGrant + documentation for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port number + to use for this resource. Port is required when the + referent is a Kubernetes Service. In this case, the + port number is the service port number, not the target + port. For other resources, destination port might be + derived from the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + weight: + default: 1 + description: "Weight specifies the proportion of requests + forwarded to the referenced backend. This is computed + as weight/(sum of all weights in this BackendRefs list). + For non-zero values, there may be some epsilon from + the exact proportion defined here depending on the precision + an implementation supports. Weight is not a percentage + and the sum of weights does not need to equal 100. \n + If only one backend is specified and it has a weight + greater than 0, 100% of the traffic is forwarded to + that backend. If weight is set to 0, no traffic should + be forwarded for this entry. If unspecified, weight + defaults to 1. \n Support for this field varies based + on the context where used." + format: int32 + maximum: 1000000 + minimum: 0 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + maxItems: 16 + minItems: 1 + type: array + type: object + maxItems: 16 + minItems: 1 + type: array + required: + - rules + type: object + status: + description: Status defines the current state of TCPRoute. + properties: + parents: + description: "Parents is a list of parent resources (usually Gateways) + that are associated with the route, and the status of the route + with respect to each parent. When this route attaches to a parent, + the controller that manages the parent must add an entry to this + list when the controller first sees the route and should update + the entry as appropriate when the route or gateway is modified. + \n Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this + API can only populate Route status for the Gateways/parent resources + they are responsible for. \n A maximum of 32 Gateways will be represented + in this list. An empty list means the route has not been attached + to any Gateway." + items: + description: RouteParentStatus describes the status of a route with + respect to an associated Parent. + properties: + conditions: + description: "Conditions describes the status of the route with + respect to the Gateway. Note that the route's availability + is also subject to the Gateway's own status conditions and + listener status. \n If the Route's ParentRef specifies an + existing Gateway that supports Routes of this kind AND that + Gateway's controller has sufficient access, then that Gateway's + controller MUST set the \"Accepted\" condition on the Route, + to indicate whether the route has been accepted or rejected + by the Gateway, and why. \n A Route MUST be considered \"Accepted\" + if at least one of the Route's rules is implemented by the + Gateway. \n There are a number of cases where the \"Accepted\" + condition may not be set due to lack of controller visibility, + that includes when: \n * The Route refers to a non-existent + parent. * The Route is of a type that the controller does + not support. * The Route is in a namespace the controller + does not have access to." + items: + description: "Condition contains details for one aspect of + the current state of this API Resource. --- This struct + is intended for direct use as an array at the field path + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition + `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should + be when the underlying condition changed. If that is + not known, then using the time when the API field changed + is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation + that the condition was set based upon. For instance, + if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration + is 9, the condition is out of date with respect to the + current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: reason contains a programmatic identifier + indicating the reason for the condition's last transition. + Producers of specific condition types may define expected + values and meanings for this field, and whether the + values are considered a guaranteed API. The value should + be a CamelCase string. This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + --- Many .condition.type values are consistent across + resources like Available, but because arbitrary conditions + can be useful (see .node.status.conditions), the ability + to deconflict is important. The regex it matches is + (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + controllerName: + description: "ControllerName is a domain/path string that indicates + the name of the controller that wrote this status. This corresponds + with the controllerName field on GatewayClass. \n Example: + \"example.net/gateway-controller\". \n The format of this + field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid + Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + \n Controllers MUST populate this field when writing status. + Controllers should ensure that entries to status populated + with their ControllerName are cleaned up when they are no + longer necessary." + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ + type: string + parentRef: + description: ParentRef corresponds with a ParentRef in the spec + that this RouteParentStatus struct describes the status of. + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the + core API group (such as for a \"Service\" kind referent), + Group must be explicitly set to \"\" (empty string). \n + Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. + When unspecified, this refers to the local namespace of + the Route. \n Note that there are specific rules for ParentRefs + which cross namespace boundaries. Cross-namespace references + are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides + a generic way to enable any other kind of cross-namespace + reference. \n ParentRefs from a Route to a Service in + the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any + namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for + which the intended destination of the connections are + a Service targeted as a ParentRef of the Route. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. + It can be interpreted differently based on the type of + parent resource. \n When the parent resource is a Gateway, + this targets all listeners listening on the specified + port that also support this kind of Route(and select this + Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to + a specific port as opposed to a listener(s) whose port(s) + may be changed. When both Port and SectionName are specified, + the name and port of the selected listener must match + both specified values. \n When the parent resource is + a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are + specified, the name and port of the selected port must + match both specified values. \n Implementations MAY choose + to support other parent resources. Implementations supporting + other types of parent resources MUST clearly document + how/if Port is interpreted. \n For the purpose of status, + an attachment is considered successful as long as the + parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them + by Route kind, namespace, or hostname. If 1 of 2 Gateway + listeners accept attachment from the referencing Route, + the Route MUST be considered successfully attached. If + no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within + the target resource. In the following resources, SectionName + is interpreted as the following: \n * Gateway: Listener + Name. When both Port (experimental) and SectionName are + specified, the name and port of the selected listener + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this + will reference the entire resource. For the purpose of + status, an attachment is considered successful if at least + one section in the parent resource accepts it. For example, + Gateway listeners can restrict which Routes can attach + to them by Route kind, namespace, or hostname. If 1 of + 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + required: + - controllerName + - parentRef + type: object + maxItems: 32 + type: array + required: + - parents + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} +status: + acceptedNames: + kind: "" + plural: "" + conditions: null + storedVersions: null +--- +# +# config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml +# +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 + gateway.networking.k8s.io/channel: experimental + creationTimestamp: null + name: tlsroutes.gateway.networking.k8s.io +spec: + group: gateway.networking.k8s.io + names: + categories: + - gateway-api + kind: TLSRoute + listKind: TLSRouteList + plural: tlsroutes + singular: tlsroute + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha2 + schema: + openAPIV3Schema: + description: "The TLSRoute resource is similar to TCPRoute, but can be configured + to match against TLS-specific metadata. This allows more flexibility in + matching streams for a given TLS listener. \n If you need to forward traffic + to a single target for a TLS listener, you could choose to use a TCPRoute + with a TLS listener." + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of TLSRoute. + properties: + hostnames: + description: "Hostnames defines a set of SNI names that should match + against the SNI attribute of TLS ClientHello message in TLS handshake. + This matches the RFC 1123 definition of a hostname with 2 notable + exceptions: \n 1. IPs are not allowed in SNI names per RFC 6066. + 2. A hostname may be prefixed with a wildcard label (`*.`). The + wildcard label must appear by itself as the first label. \n If a + hostname is specified by both the Listener and TLSRoute, there must + be at least one intersecting hostname for the TLSRoute to be attached + to the Listener. For example: \n * A Listener with `test.example.com` + as the hostname matches TLSRoutes that have either not specified + any hostnames, or have specified at least one of `test.example.com` + or `*.example.com`. * A Listener with `*.example.com` as the hostname + matches TLSRoutes that have either not specified any hostnames or + have specified at least one hostname that matches the Listener hostname. + For example, `test.example.com` and `*.example.com` would both match. + On the other hand, `example.com` and `test.example.net` would not + match. \n If both the Listener and TLSRoute have specified hostnames, + any TLSRoute hostnames that do not match the Listener hostname MUST + be ignored. For example, if a Listener specified `*.example.com`, + and the TLSRoute specified `test.example.com` and `test.example.net`, + `test.example.net` must not be considered for a match. \n If both + the Listener and TLSRoute have specified hostnames, and none match + with the criteria above, then the TLSRoute is not accepted. The + implementation must raise an 'Accepted' Condition with a status + of `False` in the corresponding RouteParentStatus. \n Support: Core" + items: + description: "Hostname is the fully qualified domain name of a network + host. This matches the RFC 1123 definition of a hostname with + 2 notable exceptions: \n 1. IPs are not allowed. 2. A hostname + may be prefixed with a wildcard label (`*.`). The wildcard label + must appear by itself as the first label. \n Hostname can be \"precise\" + which is a domain name without the terminating dot of a network + host (e.g. \"foo.example.com\") or \"wildcard\", which is a domain + name prefixed with a single wildcard label (e.g. `*.example.com`). + \n Note that as per RFC1035 and RFC1123, a *label* must consist + of lower case alphanumeric characters or '-', and must start and + end with an alphanumeric character. No other punctuation is allowed." + maxLength: 253 + minLength: 1 + pattern: ^(\*\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + maxItems: 16 + type: array + parentRefs: + description: "ParentRefs references the resources (usually Gateways) + that a Route wants to be attached to. Note that the referenced parent + resource needs to allow this for the attachment to be complete. + For Gateways, that means the Gateway needs to allow attachment from + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) \n This + API may be extended in the future to support additional kinds of + parent resources. \n It is invalid to reference an identical parent + more than once. It is valid to reference multiple distinct sections + within the same parent resource, such as two separate Listeners + on the same Gateway or two separate ports on the same Service. \n + It is possible to separately reference multiple distinct objects + that may be collapsed by an implementation. For example, some implementations + may choose to merge compatible Gateway Listeners together. If that + is the case, the list of routes attached to those resources should + also be merged. \n Note that for ParentRefs that cross namespace + boundaries, there are specific rules. Cross-namespace references + are only valid if they are explicitly allowed by something in the + namespace they are referring to. For example, Gateway has the AllowedRoutes + field, and ReferenceGrant provides a generic way to enable other + kinds of cross-namespace reference. \n ParentRefs from a Route to + a Service in the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any namespace + to the Service. \n ParentRefs from a Route to a Service in a different + namespace are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same namespace + as the Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. \n " + items: + description: "ParentReference identifies an API object (usually + a Gateway) that can be considered a parent of this resource (usually + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the core + API group (such as for a \"Service\" kind referent), Group + must be explicitly set to \"\" (empty string). \n Support: + Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. When + unspecified, this refers to the local namespace of the Route. + \n Note that there are specific rules for ParentRefs which + cross namespace boundaries. Cross-namespace references are + only valid if they are explicitly allowed by something in + the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + \n ParentRefs from a Route to a Service in the same namespace + are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. + \n ParentRefs from a Route to a Service in a different namespace + are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same + namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of + the Route. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. It + can be interpreted differently based on the type of parent + resource. \n When the parent resource is a Gateway, this targets + all listeners listening on the specified port that also support + this kind of Route(and select this Route). It's not recommended + to set `Port` unless the networking behaviors specified in + a Route must apply to a specific port as opposed to a listener(s) + whose port(s) may be changed. When both Port and SectionName + are specified, the name and port of the selected listener + must match both specified values. \n When the parent resource + is a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified + values. \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes can + attach to them by Route kind, namespace, or hostname. If 1 + of 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. \n + Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within the + target resource. In the following resources, SectionName is + interpreted as the following: \n * Gateway: Listener Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match both + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + maxItems: 32 + type: array + x-kubernetes-validations: + - message: sectionName or port must be specified when parentRefs includes + 2 or more references to the same parent + rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ + == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) + && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) + || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName + != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) + && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) + ) ) ): true ))' + - message: sectionName or port must be unique when parentRefs includes + 2 or more references to the same parent + rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) + || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ + == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) + || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName + == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName + == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port + == p2.port)))) + rules: + description: Rules are a list of TLS matchers and actions. + items: + description: TLSRouteRule is the configuration for a given rule. + properties: + backendRefs: + description: "BackendRefs defines the backend(s) where matching + requests should be sent. If unspecified or invalid (refers + to a non-existent resource or a Service with no endpoints), + the rule performs no forwarding; if no filters are specified + that would result in a response being sent, the underlying + implementation must actively reject request attempts to this + backend, by rejecting the connection or returning a 500 status + code. Request rejections must respect weight; if an invalid + backend is requested to have 80% of requests, then 80% of + requests must be rejected instead. \n Support: Core for Kubernetes + Service \n Support: Extended for Kubernetes ServiceImport + \n Support: Implementation-specific for any other resource + \n Support for weight: Extended" + items: + description: "BackendRef defines how a Route should forward + a request to a Kubernetes resource. \n Note that when a + namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace + to allow that namespace's owner to accept the reference. + See the ReferenceGrant documentation for details." + properties: + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty + string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource kind of + the referent. For example \"Service\". \n Defaults to + \"Service\" when not specified. \n ExternalName services + can refer to CNAME DNS records that may live outside + of the cluster and as such are difficult to reason about + in terms of conformance. They also may not be safe to + forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName Services. + \n Support: Core (Services with a type other than ExternalName) + \n Support: Implementation-specific (Services with type + ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the backend. + When unspecified, the local namespace is inferred. \n + Note that when a namespace different than the local + namespace is specified, a ReferenceGrant object is required + in the referent namespace to allow that namespace's + owner to accept the reference. See the ReferenceGrant + documentation for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port number + to use for this resource. Port is required when the + referent is a Kubernetes Service. In this case, the + port number is the service port number, not the target + port. For other resources, destination port might be + derived from the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + weight: + default: 1 + description: "Weight specifies the proportion of requests + forwarded to the referenced backend. This is computed + as weight/(sum of all weights in this BackendRefs list). + For non-zero values, there may be some epsilon from + the exact proportion defined here depending on the precision + an implementation supports. Weight is not a percentage + and the sum of weights does not need to equal 100. \n + If only one backend is specified and it has a weight + greater than 0, 100% of the traffic is forwarded to + that backend. If weight is set to 0, no traffic should + be forwarded for this entry. If unspecified, weight + defaults to 1. \n Support for this field varies based + on the context where used." + format: int32 + maximum: 1000000 + minimum: 0 + type: integer + required: + - name + type: object + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' + maxItems: 16 + minItems: 1 + type: array + type: object + maxItems: 16 + minItems: 1 + type: array + required: + - rules + type: object + status: + description: Status defines the current state of TLSRoute. + properties: + parents: + description: "Parents is a list of parent resources (usually Gateways) + that are associated with the route, and the status of the route + with respect to each parent. When this route attaches to a parent, + the controller that manages the parent must add an entry to this + list when the controller first sees the route and should update + the entry as appropriate when the route or gateway is modified. + \n Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this + API can only populate Route status for the Gateways/parent resources + they are responsible for. \n A maximum of 32 Gateways will be represented + in this list. An empty list means the route has not been attached + to any Gateway." + items: + description: RouteParentStatus describes the status of a route with + respect to an associated Parent. + properties: + conditions: + description: "Conditions describes the status of the route with + respect to the Gateway. Note that the route's availability + is also subject to the Gateway's own status conditions and + listener status. \n If the Route's ParentRef specifies an + existing Gateway that supports Routes of this kind AND that + Gateway's controller has sufficient access, then that Gateway's + controller MUST set the \"Accepted\" condition on the Route, + to indicate whether the route has been accepted or rejected + by the Gateway, and why. \n A Route MUST be considered \"Accepted\" + if at least one of the Route's rules is implemented by the + Gateway. \n There are a number of cases where the \"Accepted\" + condition may not be set due to lack of controller visibility, + that includes when: \n * The Route refers to a non-existent + parent. * The Route is of a type that the controller does + not support. * The Route is in a namespace the controller + does not have access to." + items: + description: "Condition contains details for one aspect of + the current state of this API Resource. --- This struct + is intended for direct use as an array at the field path + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition + `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should + be when the underlying condition changed. If that is + not known, then using the time when the API field changed + is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation + that the condition was set based upon. For instance, + if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration + is 9, the condition is out of date with respect to the + current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: reason contains a programmatic identifier + indicating the reason for the condition's last transition. + Producers of specific condition types may define expected + values and meanings for this field, and whether the + values are considered a guaranteed API. The value should + be a CamelCase string. This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + --- Many .condition.type values are consistent across + resources like Available, but because arbitrary conditions + can be useful (see .node.status.conditions), the ability + to deconflict is important. The regex it matches is + (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + controllerName: + description: "ControllerName is a domain/path string that indicates + the name of the controller that wrote this status. This corresponds + with the controllerName field on GatewayClass. \n Example: + \"example.net/gateway-controller\". \n The format of this + field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid + Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + \n Controllers MUST populate this field when writing status. + Controllers should ensure that entries to status populated + with their ControllerName are cleaned up when they are no + longer necessary." + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ + type: string + parentRef: + description: ParentRef corresponds with a ParentRef in the spec + that this RouteParentStatus struct describes the status of. + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the + core API group (such as for a \"Service\" kind referent), + Group must be explicitly set to \"\" (empty string). \n + Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. + When unspecified, this refers to the local namespace of + the Route. \n Note that there are specific rules for ParentRefs + which cross namespace boundaries. Cross-namespace references + are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides + a generic way to enable any other kind of cross-namespace + reference. \n ParentRefs from a Route to a Service in + the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any + namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for + which the intended destination of the connections are + a Service targeted as a ParentRef of the Route. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. + It can be interpreted differently based on the type of + parent resource. \n When the parent resource is a Gateway, + this targets all listeners listening on the specified + port that also support this kind of Route(and select this + Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to + a specific port as opposed to a listener(s) whose port(s) + may be changed. When both Port and SectionName are specified, + the name and port of the selected listener must match + both specified values. \n When the parent resource is + a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are + specified, the name and port of the selected port must + match both specified values. \n Implementations MAY choose + to support other parent resources. Implementations supporting + other types of parent resources MUST clearly document + how/if Port is interpreted. \n For the purpose of status, + an attachment is considered successful as long as the + parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them + by Route kind, namespace, or hostname. If 1 of 2 Gateway + listeners accept attachment from the referencing Route, + the Route MUST be considered successfully attached. If + no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within + the target resource. In the following resources, SectionName + is interpreted as the following: \n * Gateway: Listener + Name. When both Port (experimental) and SectionName are + specified, the name and port of the selected listener + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this + will reference the entire resource. For the purpose of + status, an attachment is considered successful if at least + one section in the parent resource accepts it. For example, + Gateway listeners can restrict which Routes can attach + to them by Route kind, namespace, or hostname. If 1 of + 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + required: + - controllerName + - parentRef + type: object + maxItems: 32 + type: array + required: + - parents + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} +status: + acceptedNames: + kind: "" + plural: "" + conditions: null + storedVersions: null +--- +# +# config/crd/experimental/gateway.networking.k8s.io_udproutes.yaml +# +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 + gateway.networking.k8s.io/bundle-version: v0.8.0 + gateway.networking.k8s.io/channel: experimental + creationTimestamp: null + name: udproutes.gateway.networking.k8s.io +spec: + group: gateway.networking.k8s.io + names: + categories: + - gateway-api + kind: UDPRoute + listKind: UDPRouteList + plural: udproutes + singular: udproute + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha2 + schema: + openAPIV3Schema: + description: UDPRoute provides a way to route UDP traffic. When combined with + a Gateway listener, it can be used to forward traffic on the port specified + by the listener to a set of backends specified by the UDPRoute. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of UDPRoute. + properties: + parentRefs: + description: "ParentRefs references the resources (usually Gateways) + that a Route wants to be attached to. Note that the referenced parent + resource needs to allow this for the attachment to be complete. + For Gateways, that means the Gateway needs to allow attachment from + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) \n This + API may be extended in the future to support additional kinds of + parent resources. \n It is invalid to reference an identical parent + more than once. It is valid to reference multiple distinct sections + within the same parent resource, such as two separate Listeners + on the same Gateway or two separate ports on the same Service. \n + It is possible to separately reference multiple distinct objects + that may be collapsed by an implementation. For example, some implementations + may choose to merge compatible Gateway Listeners together. If that + is the case, the list of routes attached to those resources should + also be merged. \n Note that for ParentRefs that cross namespace + boundaries, there are specific rules. Cross-namespace references + are only valid if they are explicitly allowed by something in the + namespace they are referring to. For example, Gateway has the AllowedRoutes + field, and ReferenceGrant provides a generic way to enable other + kinds of cross-namespace reference. \n ParentRefs from a Route to + a Service in the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any namespace + to the Service. \n ParentRefs from a Route to a Service in a different + namespace are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same namespace + as the Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. \n " + items: + description: "ParentReference identifies an API object (usually + a Gateway) that can be considered a parent of this resource (usually + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the core + API group (such as for a \"Service\" kind referent), Group + must be explicitly set to \"\" (empty string). \n Support: + Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. When + unspecified, this refers to the local namespace of the Route. + \n Note that there are specific rules for ParentRefs which + cross namespace boundaries. Cross-namespace references are + only valid if they are explicitly allowed by something in + the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + \n ParentRefs from a Route to a Service in the same namespace + are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. + \n ParentRefs from a Route to a Service in a different namespace + are \"consumer\" routes, and these routing rules are only + applied to outbound connections originating from the same + namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of + the Route. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. It + can be interpreted differently based on the type of parent + resource. \n When the parent resource is a Gateway, this targets + all listeners listening on the specified port that also support + this kind of Route(and select this Route). It's not recommended + to set `Port` unless the networking behaviors specified in + a Route must apply to a specific port as opposed to a listener(s) + whose port(s) may be changed. When both Port and SectionName + are specified, the name and port of the selected listener + must match both specified values. \n When the parent resource + is a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified + values. \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes can + attach to them by Route kind, namespace, or hostname. If 1 + of 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. \n + Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within the + target resource. In the following resources, SectionName is + interpreted as the following: \n * Gateway: Listener Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match both + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + maxItems: 32 + type: array + x-kubernetes-validations: + - message: sectionName or port must be specified when parentRefs includes + 2 or more references to the same parent + rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ + == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) + && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) + || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName + != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) + && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) + ) ) ): true ))' + - message: sectionName or port must be unique when parentRefs includes + 2 or more references to the same parent + rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) + || p1.__namespace__ == '') && (!has(p2.__namespace__) || p2.__namespace__ + == '')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__ )) && (((!has(p1.sectionName) + || p1.sectionName == '') && (!has(p2.sectionName) || p2.sectionName + == '')) || ( has(p1.sectionName) && has(p2.sectionName) && p1.sectionName + == p2.sectionName)) && (((!has(p1.port) || p1.port == 0) && (!has(p2.port) + || p2.port == 0)) || (has(p1.port) && has(p2.port) && p1.port + == p2.port)))) + rules: + description: Rules are a list of UDP matchers and actions. + items: + description: UDPRouteRule is the configuration for a given rule. + properties: + backendRefs: + description: "BackendRefs defines the backend(s) where matching + requests should be sent. If unspecified or invalid (refers + to a non-existent resource or a Service with no endpoints), + the underlying implementation MUST actively reject connection + attempts to this backend. Packet drops must respect weight; + if an invalid backend is requested to have 80% of the packets, + then 80% of packets must be dropped instead. \n Support: Core + for Kubernetes Service \n Support: Extended for Kubernetes + ServiceImport \n Support: Implementation-specific for any + other resource \n Support for weight: Extended" + items: + description: "BackendRef defines how a Route should forward + a request to a Kubernetes resource. \n Note that when a + namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace + to allow that namespace's owner to accept the reference. + See the ReferenceGrant documentation for details." + properties: + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty + string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: "Kind is the Kubernetes resource kind of + the referent. For example \"Service\". \n Defaults to + \"Service\" when not specified. \n ExternalName services + can refer to CNAME DNS records that may live outside + of the cluster and as such are difficult to reason about + in terms of conformance. They also may not be safe to + forward to (see CVE-2021-25740 for more information). + Implementations SHOULD NOT support ExternalName Services. + \n Support: Core (Services with a type other than ExternalName) + \n Support: Implementation-specific (Services with type + ExternalName)" + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the backend. + When unspecified, the local namespace is inferred. \n + Note that when a namespace different than the local + namespace is specified, a ReferenceGrant object is required + in the referent namespace to allow that namespace's + owner to accept the reference. See the ReferenceGrant + documentation for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port number + to use for this resource. Port is required when the + referent is a Kubernetes Service. In this case, the + port number is the service port number, not the target + port. For other resources, destination port might be + derived from the referent resource or this field. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + weight: + default: 1 + description: "Weight specifies the proportion of requests + forwarded to the referenced backend. This is computed + as weight/(sum of all weights in this BackendRefs list). + For non-zero values, there may be some epsilon from + the exact proportion defined here depending on the precision + an implementation supports. Weight is not a percentage + and the sum of weights does not need to equal 100. \n + If only one backend is specified and it has a weight + greater than 0, 100% of the traffic is forwarded to + that backend. If weight is set to 0, no traffic should + be forwarded for this entry. If unspecified, weight + defaults to 1. \n Support for this field varies based + on the context where used." + format: int32 + maximum: 1000000 + minimum: 0 + type: integer + required: + - name + type: object + # x-kubernetes-validations: + # - message: Must have port for Service reference + # rule: '(size(self.group) == 0 && self.kind == ''Service'') + # ? has(self.port) : true' + maxItems: 16 + minItems: 1 + type: array + type: object + maxItems: 16 + minItems: 1 + type: array + required: + - rules + type: object + status: + description: Status defines the current state of UDPRoute. + properties: + parents: + description: "Parents is a list of parent resources (usually Gateways) + that are associated with the route, and the status of the route + with respect to each parent. When this route attaches to a parent, + the controller that manages the parent must add an entry to this + list when the controller first sees the route and should update + the entry as appropriate when the route or gateway is modified. + \n Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this + API can only populate Route status for the Gateways/parent resources + they are responsible for. \n A maximum of 32 Gateways will be represented + in this list. An empty list means the route has not been attached + to any Gateway." + items: + description: RouteParentStatus describes the status of a route with + respect to an associated Parent. + properties: + conditions: + description: "Conditions describes the status of the route with + respect to the Gateway. Note that the route's availability + is also subject to the Gateway's own status conditions and + listener status. \n If the Route's ParentRef specifies an + existing Gateway that supports Routes of this kind AND that + Gateway's controller has sufficient access, then that Gateway's + controller MUST set the \"Accepted\" condition on the Route, + to indicate whether the route has been accepted or rejected + by the Gateway, and why. \n A Route MUST be considered \"Accepted\" + if at least one of the Route's rules is implemented by the + Gateway. \n There are a number of cases where the \"Accepted\" + condition may not be set due to lack of controller visibility, + that includes when: \n * The Route refers to a non-existent + parent. * The Route is of a type that the controller does + not support. * The Route is in a namespace the controller + does not have access to." + items: + description: "Condition contains details for one aspect of + the current state of this API Resource. --- This struct + is intended for direct use as an array at the field path + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition + `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should + be when the underlying condition changed. If that is + not known, then using the time when the API field changed + is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation + that the condition was set based upon. For instance, + if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration + is 9, the condition is out of date with respect to the + current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: reason contains a programmatic identifier + indicating the reason for the condition's last transition. + Producers of specific condition types may define expected + values and meanings for this field, and whether the + values are considered a guaranteed API. The value should + be a CamelCase string. This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + --- Many .condition.type values are consistent across + resources like Available, but because arbitrary conditions + can be useful (see .node.status.conditions), the ability + to deconflict is important. The regex it matches is + (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + controllerName: + description: "ControllerName is a domain/path string that indicates + the name of the controller that wrote this status. This corresponds + with the controllerName field on GatewayClass. \n Example: + \"example.net/gateway-controller\". \n The format of this + field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid + Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + \n Controllers MUST populate this field when writing status. + Controllers should ensure that entries to status populated + with their ControllerName are cleaned up when they are no + longer necessary." + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ + type: string + parentRef: + description: ParentRef corresponds with a ParentRef in the spec + that this RouteParentStatus struct describes the status of. + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the + core API group (such as for a \"Service\" kind referent), + Group must be explicitly set to \"\" (empty string). \n + Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. + When unspecified, this refers to the local namespace of + the Route. \n Note that there are specific rules for ParentRefs + which cross namespace boundaries. Cross-namespace references + are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides + a generic way to enable any other kind of cross-namespace + reference. \n ParentRefs from a Route to a Service in + the same namespace are \"producer\" routes, which apply + default routing rules to inbound connections from any + namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for + which the intended destination of the connections are + a Service targeted as a ParentRef of the Route. \n Support: + Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. + It can be interpreted differently based on the type of + parent resource. \n When the parent resource is a Gateway, + this targets all listeners listening on the specified + port that also support this kind of Route(and select this + Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to + a specific port as opposed to a listener(s) whose port(s) + may be changed. When both Port and SectionName are specified, + the name and port of the selected listener must match + both specified values. \n When the parent resource is + a Service, this targets a specific port in the Service + spec. When both Port (experimental) and SectionName are + specified, the name and port of the selected port must + match both specified values. \n Implementations MAY choose + to support other parent resources. Implementations supporting + other types of parent resources MUST clearly document + how/if Port is interpreted. \n For the purpose of status, + an attachment is considered successful as long as the + parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them + by Route kind, namespace, or hostname. If 1 of 2 Gateway + listeners accept attachment from the referencing Route, + the Route MUST be considered successfully attached. If + no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within + the target resource. In the following resources, SectionName + is interpreted as the following: \n * Gateway: Listener + Name. When both Port (experimental) and SectionName are + specified, the name and port of the selected listener + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. If that is the case, they MUST clearly document how SectionName is interpreted. \n When unspecified (empty string), this will reference the entire resource. For the purpose of @@ -2385,5 +10748,5 @@ status: acceptedNames: kind: "" plural: "" - conditions: [] - storedVersions: [] \ No newline at end of file + conditions: null + storedVersions: null diff --git a/deploy/manifests/static/stunner-crd.yaml b/deploy/manifests/static/stunner-crd.yaml index 04457aed..77b5eac5 100644 --- a/deploy/manifests/static/stunner-crd.yaml +++ b/deploy/manifests/static/stunner-crd.yaml @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.8.0 + controller-gen.kubebuilder.io/version: v0.11.1 creationTimestamp: null name: gatewayconfigs.stunner.l7mp.io spec: @@ -72,8 +72,7 @@ spec: type: string kind: default: Secret - description: Kind is kind of the referent. For example "HTTPRoute" - or "Service". + description: Kind is kind of the referent. For example "Secret". maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -86,10 +85,10 @@ spec: namespace: description: "Namespace is the namespace of the backend. When unspecified, the local namespace is inferred. \n Note that when - a namespace is specified, a ReferenceGrant object is required - in the referent namespace to allow that namespace's owner to - accept the reference. See the ReferenceGrant documentation for - details. \n Support: Core" + a namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace + to allow that namespace's owner to accept the reference. See + the ReferenceGrant documentation for details. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -103,6 +102,12 @@ spec: mechanism. pattern: ^plaintext|static|longterm|ephemeral|timewindowed$ type: string + dataplane: + default: default + description: Dataplane defines the TURN server to set up for the STUNner + Gateways using this GatewayConfig. Can be used to select the stunnerd + image repo and version or deploy into the host-network namespace. + type: string healthCheckEndpoint: description: HealthCheckEndpoint is the URI of the form `http://address:port` exposed for external HTTP health-checking. A liveness probe responder @@ -179,9 +184,1396 @@ spec: served: true storage: true subresources: {} -status: - acceptedNames: - kind: "" - plural: "" - conditions: [] - storedVersions: [] \ No newline at end of file +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.11.1 + creationTimestamp: null + name: staticservices.stunner.l7mp.io +spec: + group: stunner.l7mp.io + names: + categories: + - stunner + kind: StaticService + listKind: StaticServiceList + plural: staticservices + shortNames: + - ssvc + singular: staticservice + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: StaticService is a set of static IP address prefixes STUNner + allows access to via a Route. The purpose is to allow a Service-like CRD + containing a set of static IP address prefixes to be set as the backend + of a UDPRoute (or TCPRoute). + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the behavior of a service. + properties: + ports: + description: The list of ports reachable via this service (currently + omitted). + items: + description: ServicePort contains information on service's port. + properties: + appProtocol: + description: The application protocol for this port. This field + follows standard Kubernetes label syntax. Un-prefixed names + are reserved for IANA standard service names (as per RFC-6335 + and https://www.iana.org/assignments/service-names). Non-standard + protocols should use prefixed names such as mycompany.com/my-custom-protocol. + type: string + name: + description: The name of this port within the service. This + must be a DNS_LABEL. All ports within a ServiceSpec must have + unique names. When considering the endpoints for a Service, + this must match the 'name' field in the EndpointPort. Optional + if only one ServicePort is defined on this service. + type: string + nodePort: + description: 'The port on each node on which this service is + exposed when type is NodePort or LoadBalancer. Usually assigned + by the system. If a value is specified, in-range, and not + in use it will be used, otherwise the operation will fail. If + not specified, a port will be allocated if this Service requires + one. If this field is specified when creating a Service which + does not need it, creation will fail. This field will be wiped + when updating a Service to no longer need it (e.g. changing + type from NodePort to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport' + format: int32 + type: integer + port: + description: The port that will be exposed by this service. + format: int32 + type: integer + protocol: + default: TCP + description: The IP protocol for this port. Supports "TCP", + "UDP", and "SCTP". Default is TCP. + type: string + targetPort: + anyOf: + - type: integer + - type: string + description: 'Number or name of the port to access on the pods + targeted by the service. Number must be in the range 1 to + 65535. Name must be an IANA_SVC_NAME. If this is a string, + it will be looked up as a named port in the target Pod''s + container ports. If this is not specified, the value of the + ''port'' field is used (an identity map). This field is ignored + for services with clusterIP=None, and should be omitted or + set equal to the ''port'' field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service' + x-kubernetes-int-or-string: true + required: + - port + type: object + type: array + x-kubernetes-list-map-keys: + - port + - protocol + x-kubernetes-list-type: map + prefixes: + description: Prefixes is a list of IP address prefixes reachable via + this route. + items: + type: string + type: array + required: + - prefixes + type: object + type: object + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.11.1 + creationTimestamp: null + name: dataplanes.stunner.l7mp.io +spec: + group: stunner.l7mp.io + names: + categories: + - stunner + kind: Dataplane + listKind: DataplaneList + plural: dataplanes + shortNames: + - dps + singular: dataplane + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: Dataplane is a collection of configuration parameters that can + be used for spawning a `stunnerd` instance for a Gateway. Labels and annotations + on the Dataplane object will be copied verbatim into the target Deployment. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the behavior of a Dataplane resource. + properties: + affinity: + description: Scheduling constraints. + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for the + pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node matches + the corresponding matchExpressions; the node(s) with the + highest sum are the most preferred. + items: + description: An empty preferred scheduling term matches + all objects with implicit weight 0 (i.e. it's a no-op). + A null preferred scheduling term matches no objects (i.e. + is also a no-op). + properties: + preference: + description: A node selector term, associated with the + corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + weight: + description: Weight associated with matching the corresponding + nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to an update), the system may or may not try to + eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. + The terms are ORed. + items: + description: A null or empty node selector term matches + no objects. The requirements of them are ANDed. The + TopologySelectorTerm type implements a subset of the + NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + type: array + required: + - nodeSelectorTerms + type: object + x-kubernetes-map-type: atomic + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. co-locate + this pod in the same node, zone, etc. as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to a pod label update), the system may or may + not try to eventually evict the pod from its node. When + there are multiple elements, the lists of nodes corresponding + to each podAffinityTerm are intersected, i.e. all terms + must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules (e.g. + avoid putting this pod in the same node, zone, etc. as some + other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the anti-affinity expressions specified + by this field, but it may choose a node that violates one + or more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the anti-affinity requirements specified by + this field are not met at scheduling time, the pod will + not be scheduled onto the node. If the anti-affinity requirements + specified by this field cease to be met at some point during + pod execution (e.g. due to a pod label update), the system + may or may not try to eventually evict the pod from its + node. When there are multiple elements, the lists of nodes + corresponding to each podAffinityTerm are intersected, i.e. + all terms must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + type: object + args: + description: Arguments to the entrypoint. + items: + type: string + type: array + command: + description: 'Entrypoint array. Defaults: "stunnerd".' + items: + type: string + type: array + env: + description: List of environment variables to set in the stunnerd + container. + items: + description: EnvVar represents an environment variable present in + a Container. + properties: + name: + description: Name of the environment variable. Must be a C_IDENTIFIER. + type: string + value: + description: 'Variable references $(VAR_NAME) are expanded using + the previously defined environment variables in the container + and any service environment variables. If a variable cannot + be resolved, the reference in the input string will be unchanged. + Double $$ are reduced to a single $, which allows for escaping + the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the + string literal "$(VAR_NAME)". Escaped references will never + be expanded, regardless of whether the variable exists or + not. Defaults to "".' + type: string + valueFrom: + description: Source for the environment variable's value. Cannot + be used if value is not empty. + properties: + configMapKeyRef: + description: Selects a key of a ConfigMap. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + fieldRef: + description: 'Selects a field of the pod: supports metadata.name, + metadata.namespace, `metadata.labels['''']`, `metadata.annotations['''']`, + spec.nodeName, spec.serviceAccountName, status.hostIP, + status.podIP, status.podIPs.' + properties: + apiVersion: + description: Version of the schema the FieldPath is + written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the specified + API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + resourceFieldRef: + description: 'Selects a resource of the container: only + resources limits and requests (limits.cpu, limits.memory, + limits.ephemeral-storage, requests.cpu, requests.memory + and requests.ephemeral-storage) are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the exposed + resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + secretKeyRef: + description: Selects a key of a secret in the pod's namespace + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + required: + - name + type: object + type: array + healthCheckPort: + description: If specified, the health-check port. + type: integer + hostNetwork: + description: Host networking requested for the stunnerd pod to use + the host's network namespace. Can be used to implement public TURN + servers with Kubernetes. Defaults to false. + type: boolean + image: + description: Container image name. + type: string + imagePullPolicy: + description: Image pull policy. One of Always, Never, IfNotPresent. + type: string + replicas: + description: Number of desired pods. This is a pointer to distinguish + between explicit zero and not specified. Defaults to 1. + format: int32 + type: integer + resources: + description: Resources required by stunnerd. + properties: + claims: + description: "Claims lists the names of resources, defined in + spec.resourceClaims, that are used by this container. \n This + is an alpha field and requires enabling the DynamicResourceAllocation + feature gate. \n This field is immutable. It can only be set + for containers." + items: + description: ResourceClaim references one entry in PodSpec.ResourceClaims. + properties: + name: + description: Name must match the name of one entry in pod.spec.resourceClaims + of the Pod where this field is used. It makes that resource + available inside a container. + type: string + required: + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute resources + allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, otherwise + to an implementation-defined value. Requests cannot exceed Limits. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + securityContext: + description: SecurityContext holds pod-level security attributes and + common container settings. + properties: + fsGroup: + description: "A special supplemental group that applies to all + containers in a pod. Some volume types allow the Kubelet to + change the ownership of that volume to be owned by the pod: + \n 1. The owning GID will be the FSGroup 2. The setgid bit is + set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- \n If unset, + the Kubelet will not modify the ownership and permissions of + any volume. Note that this field cannot be set when spec.os.name + is windows." + format: int64 + type: integer + fsGroupChangePolicy: + description: 'fsGroupChangePolicy defines behavior of changing + ownership and permission of the volume before being exposed + inside Pod. This field will only apply to volume types which + support fsGroup based ownership(and permissions). It will have + no effect on ephemeral volume types such as: secret, configmaps + and emptydir. Valid values are "OnRootMismatch" and "Always". + If not specified, "Always" is used. Note that this field cannot + be set when spec.os.name is windows.' + type: string + runAsGroup: + description: The GID to run the entrypoint of the container process. + Uses runtime default if unset. May also be set in SecurityContext. If + set in both SecurityContext and PodSecurityContext, the value + specified in SecurityContext takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a non-root + user. If true, the Kubelet will validate the image at runtime + to ensure that it does not run as UID 0 (root) and fail to start + the container if it does. If unset or false, no such validation + will be performed. May also be set in SecurityContext. If set + in both SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in SecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. Note that this field cannot + be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random + SELinux context for each container. May also be set in SecurityContext. If + set in both SecurityContext and PodSecurityContext, the value + specified in SecurityContext takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies to + the container. + type: string + role: + description: Role is a SELinux role label that applies to + the container. + type: string + type: + description: Type is a SELinux type label that applies to + the container. + type: string + user: + description: User is a SELinux user label that applies to + the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by the containers in this + pod. Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile must be + preconfigured on the node to work. Must be a descending + path, relative to the kubelet's configured seccomp profile + location. Must be set if type is "Localhost". Must NOT be + set for any other type. + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - a profile + defined in a file on the node should be used. RuntimeDefault + - the container runtime default profile should be used. + Unconfined - no profile should be applied." + type: string + required: + - type + type: object + supplementalGroups: + description: A list of groups applied to the first process run + in each container, in addition to the container's primary GID, + the fsGroup (if specified), and group memberships defined in + the container image for the uid of the container process. If + unspecified, no additional groups are added to any container. + Note that group memberships defined in the container image for + the uid of the container process are still effective, even if + they are not included in this list. Note that this field cannot + be set when spec.os.name is windows. + items: + format: int64 + type: integer + type: array + sysctls: + description: Sysctls hold a list of namespaced sysctls used for + the pod. Pods with unsupported sysctls (by the container runtime) + might fail to launch. Note that this field cannot be set when + spec.os.name is windows. + items: + description: Sysctl defines a kernel parameter to be set + properties: + name: + description: Name of a property to set + type: string + value: + description: Value of a property to set + type: string + required: + - name + - value + type: object + type: array + windowsOptions: + description: The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named by + the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the GMSA + credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. All of a Pod's containers + must have the same effective HostProcess value (it is not + allowed to have a mix of HostProcess containers and non-HostProcess + containers). In addition, if HostProcess is true then HostNetwork + must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set in PodSecurityContext. + If set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. + type: string + type: object + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the stunnerd needs to terminate + gracefully. Defaults to 3600 seconds. + format: int64 + type: integer + tolerations: + description: If specified, the pod's tolerations. + items: + description: The pod this Toleration is attached to tolerates any + taint that matches the triple using the matching + operator . + properties: + effect: + description: Effect indicates the taint effect to match. Empty + means match all taint effects. When specified, allowed values + are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: Key is the taint key that the toleration applies + to. Empty means match all taint keys. If the key is empty, + operator must be Exists; this combination means to match all + values and all keys. + type: string + operator: + description: Operator represents a key's relationship to the + value. Valid operators are Exists and Equal. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod + can tolerate all taints of a particular category. + type: string + tolerationSeconds: + description: TolerationSeconds represents the period of time + the toleration (which must be of effect NoExecute, otherwise + this field is ignored) tolerates the taint. By default, it + is not set, which means tolerate the taint forever (do not + evict). Zero and negative values will be treated as 0 (evict + immediately) by the system. + format: int64 + type: integer + value: + description: Value is the taint value the toleration matches + to. If the operator is Exists, the value should be empty, + otherwise just a regular string. + type: string + type: object + type: array + type: object + type: object + served: true + storage: true +--- diff --git a/deploy/manifests/static/stunner-gateway-operator-manifests.yaml b/deploy/manifests/static/stunner-gateway-operator-manifests.yaml index 0fc55ce9..0d4968d6 100644 --- a/deploy/manifests/static/stunner-gateway-operator-manifests.yaml +++ b/deploy/manifests/static/stunner-gateway-operator-manifests.yaml @@ -49,6 +49,18 @@ kind: ClusterRole metadata: name: stunner-gateway-operator-manager-role rules: +- apiGroups: + - apps + resources: + - deployments + verbs: + - create + - delete + - get + - list + - patch + - update + - watch - apiGroups: - "" resources: @@ -70,9 +82,11 @@ rules: - apiGroups: - "" resources: - - endpoints - - nodes - - secrets + - deployments/finalizers + - deployments/status + - endpoints/status + - nodes/status + - services/status verbs: - get - list @@ -80,11 +94,14 @@ rules: - apiGroups: - "" resources: - - endpoints/status - - nodes/status - - services/status + - endpoints + - namespaces + - nodes + - secrets verbs: - get + - list + - watch - apiGroups: - "" resources: @@ -121,7 +138,9 @@ rules: - apiGroups: - stunner.l7mp.io resources: + - dataplanes - gatewayconfigs + - staticservices verbs: - get - list @@ -131,7 +150,9 @@ rules: - apiGroups: - stunner.l7mp.io resources: + - dataplanes/finalizers - gatewayconfigs/finalizers + - staticservices/finalizers verbs: - update --- @@ -306,10 +327,12 @@ spec: terminationGracePeriodSeconds: 10 containers: - name: stunner-auth-server - image: l7mp/stunner-auth-server:dev + image: "l7mp/stunner-auth-server:0.16.0" imagePullPolicy: Always command: [ "./manager" ] - args: ["-zap-log-level","10", "-port", "8088"] + args: + - --zap-log-level=10 + - --port=8088 securityContext: allowPrivilegeEscalation: false capabilities: @@ -334,6 +357,10 @@ spec: requests: cpu: 10m memory: 64Mi + nodeSelector: + kubernetes.io/os: linux + tolerations: + [] --- apiVersion: apps/v1 kind: Deployment @@ -366,34 +393,53 @@ spec: - containerPort: 8443 name: https protocol: TCP + - containerPort: 13478 + name: cds + protocol: TCP resources: limits: - cpu: 500m - memory: 128Mi + cpu: 1000m + memory: 256Mi requests: - cpu: 5m - memory: 64Mi + cpu: 250m + memory: 128Mi - args: - - --health-probe-bind-address=:8081 - - --metrics-bind-address=127.0.0.1:8080 - - --leader-elect + - --health-probe-bind-address=:8081 + - --metrics-bind-address=127.0.0.1:8080 + - --leader-elect + - --zap-log-level=info + - --dataplane-mode=legacy command: - /manager - image: "l7mp/stunner-gateway-operator:0.15.0" + env: + - name: STUNNER_GATEWAY_OPERATOR_ADDRESS + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: status.podIP + image: "l7mp/stunner-gateway-operator:0.16.0" imagePullPolicy: IfNotPresent livenessProbe: + failureThreshold: 3 httpGet: path: /healthz port: 8081 + scheme: HTTP initialDelaySeconds: 15 periodSeconds: 20 + successThreshold: 1 + timeoutSeconds: 1 name: manager readinessProbe: + failureThreshold: 3 httpGet: path: /readyz port: 8081 + scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 1 resources: limits: cpu: 1000m @@ -406,7 +452,8 @@ spec: securityContext: runAsNonRoot: true serviceAccountName: stunner-gateway-operator-controller-manager + serviceAccount: stunner-gateway-operator-controller-manager terminationGracePeriodSeconds: 10 nodeSelector: kubernetes.io/os: linux ---- +--- \ No newline at end of file diff --git a/deploy/manifests/static/stunner-manifests.yaml b/deploy/manifests/static/stunner-manifests.yaml index 09a62ad6..b363f0e1 100644 --- a/deploy/manifests/static/stunner-manifests.yaml +++ b/deploy/manifests/static/stunner-manifests.yaml @@ -6,21 +6,23 @@ metadata: namespace: stunner --- apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole +kind: Role metadata: - name: stunner-config-watcher-clusterrole + name: stunner-config-watcher-role + namespace: stunner rules: - apiGroups: [""] resources: ["configmaps", "secrets"] verbs: ["get", "watch", "list"] --- -kind: ClusterRoleBinding +kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: - name: stunner-config-watcher-clusterrolebind + name: stunner-config-watcher-rolebind + namespace: stunner roleRef: - kind: ClusterRole - name: stunner-config-watcher-clusterrole + kind: Role + name: stunner-config-watcher-role apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount @@ -50,8 +52,8 @@ metadata: namespace: stunner annotations: app: stunner - helm.sh/chart: stunner-0.15.0 - app.kubernetes.io/version: "0.15.0" + helm.sh/chart: stunner-0.16.0 + app.kubernetes.io/version: "0.16.0" spec: selector: matchLabels: @@ -75,7 +77,7 @@ spec: hostNetwork: false containers: - name: stunnerd - image: "l7mp/stunnerd:0.15.0" + image: "l7mp/stunnerd:0.16.0" imagePullPolicy: IfNotPresent command: ["stunnerd"] args: ["-w", "-c", "/etc/stunnerd/stunnerd.conf", "--udp-thread-num=16"] From 75954b2b07f53f3e41bbf9eb9523c95ff5455a55 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 2 Nov 2023 20:49:50 +0100 Subject: [PATCH 055/243] chore(CI/CD): Build image with go 1.21 --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index e907a606..b524bac0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ ########### # BUILD -FROM golang:1.20-alpine as builder +FROM golang:1.21-alpine as builder WORKDIR /app From 23a804b1ad013522748158e9f1627ba0beda385e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 9 Oct 2023 12:27:35 +0200 Subject: [PATCH 056/243] doc: Update stable docs URL --- README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 978a3fac..46e0b565 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@

-*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io).* +*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* # STUNner: A Kubernetes media gateway for WebRTC @@ -58,8 +58,6 @@ modification to your existing WebRTC codebase. STUNner implements the standard API](https://gateway-api.sigs.k8s.io) so you can configure it in the familiar YAML-engineering style via Kubernetes manifests. -See the full documentation [here](https://docs.l7mp.io). - ## Table of Contents 1. [Description](#description) 1. [Features](#features) @@ -601,7 +599,7 @@ applications into Kubernetes. ## Documentation -The documentation of the stable release can be found [here](https://docs.l7mp.io). The +The documentation of the stable release can be found [here](https://docs.l7mp.io/en/stable). The documentation for the latest development release can be found [here](/docs/README.md). ## Caveats From 632ae531dac918f0f9880a9e02178ff2928ce7a7 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sat, 25 Nov 2023 09:06:40 +0100 Subject: [PATCH 057/243] chore: Fix misleading error message --- cmd/stunnerd/main.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 792d204c..a36275f6 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -168,8 +168,8 @@ func main() { if e, ok := err.(v1alpha1.ErrRestarted); ok { log.Debugf("reconciliation ready: %s", e.Error()) } else { - log.Errorf("could not reconcile new configuration: %s, "+ - "rolling back to last running config", err.Error()) + log.Errorf("could not reconcile new configuration "+ + "(running configuration unchanged): %s", err.Error()) } } } From 932943a966d5ba66d03af9384bcdde4b7d821fa0 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sat, 25 Nov 2023 17:30:36 +0100 Subject: [PATCH 058/243] chore: Introduce a rate-limited logger The logger's output is controlled via the token bucket in x/time/rate. Also added logger tests. --- go.mod | 2 +- go.sum | 3 +- pkg/logger/logger.go | 92 +++++- pkg/logger/logger_test.go | 658 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 741 insertions(+), 14 deletions(-) create mode 100644 pkg/logger/logger_test.go diff --git a/go.mod b/go.mod index e349b737..5fac9ce1 100644 --- a/go.mod +++ b/go.mod @@ -16,6 +16,7 @@ require ( github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 golang.org/x/sys v0.12.0 + golang.org/x/time v0.4.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 sigs.k8s.io/controller-runtime v0.12.3 @@ -58,7 +59,6 @@ require ( golang.org/x/oauth2 v0.8.0 // indirect golang.org/x/term v0.12.0 // indirect golang.org/x/text v0.13.0 // indirect - golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/protobuf v1.31.0 // indirect gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect diff --git a/go.sum b/go.sum index 4a0c650d..8a8fe316 100644 --- a/go.sum +++ b/go.sum @@ -505,8 +505,9 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 h1:vVKdlvoWBphwdxWKrFZEuM0kGgGLxUOYcY4U/2Vjg44= golang.org/x/time v0.0.0-20220210224613-90d013bbcef8/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.4.0 h1:Z81tqI5ddIoXDPvVQ7/7CC9TnLM7ubaFG2qXYd5BbYY= +golang.org/x/time v0.4.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= diff --git a/pkg/logger/logger.go b/pkg/logger/logger.go index 575f6fe1..1f326b7b 100644 --- a/pkg/logger/logger.go +++ b/pkg/logger/logger.go @@ -6,12 +6,23 @@ import ( "log" "os" "strings" + "time" "github.com/pion/logging" + "golang.org/x/time/rate" ) const defaultFlags = log.Lmicroseconds | log.Lshortfile | log.Lmsgprefix +var logLevels = map[string]logging.LogLevel{ + "DISABLE": logging.LogLevelDisabled, + "ERROR": logging.LogLevelError, + "WARN": logging.LogLevelWarn, + "INFO": logging.LogLevelInfo, + "DEBUG": logging.LogLevelDebug, + "TRACE": logging.LogLevelTrace, +} + // LoggerFactory defines levels by scopes and creates new LeveledLogger. type LoggerFactory struct { Writer io.Writer @@ -36,17 +47,15 @@ func NewLoggerFactory(levelSpec string) *LoggerFactory { return &logger } -// NewLogger either returns the existing LeveledLoogger (if it exists) for the given scope or creates a new one. +// NewLogger either returns the existing LeveledLogger (if it exists) for the given scope or creates a new one. func (f *LoggerFactory) NewLogger(scope string) logging.LeveledLogger { logger, found := f.Loggers[scope] if found { return logger } - // create a new one logLevel := f.DefaultLogLevel scopeLevel, found := f.ScopeLevels[scope] - if found { logLevel = scopeLevel } @@ -63,17 +72,65 @@ func (f *LoggerFactory) NewLogger(scope string) logging.LeveledLogger { return l } -// Setlevel sets the loglevel. -func (f *LoggerFactory) SetLevel(levelSpec string) { - logLevels := map[string]logging.LogLevel{ - "DISABLE": logging.LogLevelDisabled, - "ERROR": logging.LogLevelError, - "WARN": logging.LogLevelWarn, - "INFO": logging.LogLevelInfo, - "DEBUG": logging.LogLevelDebug, - "TRACE": logging.LogLevelTrace, +// NewRateLimitedLogger creates a new rate-limited logger from a factory. Every loglevel is independently rate-limited with a token bucket ofthe given period and burst size. If addSuppressed is true and then the number of suppressed events is added to the output (provided that there were suppressed events). +func (f *LoggerFactory) NewRateLimitedLogger(scope string, period time.Duration, burst int, addSuppressed bool) logging.LeveledLogger { + logger, found := f.Loggers[scope] + if found { + return logger + } + + logLevel := f.DefaultLogLevel + scopeLevel, found := f.ScopeLevels[scope] + if found { + logLevel = scopeLevel } + l := logging.NewDefaultLeveledLoggerForScope(scope, logLevel, f.Writer). + WithTraceLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s TRACE: ", scope), defaultFlags)). + WithDebugLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s DEBUG: ", scope), defaultFlags)). + WithInfoLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s INFO: ", scope), defaultFlags)). + WithWarnLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s WARNING: ", scope), defaultFlags)). + WithErrorLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s ERROR: ", scope), defaultFlags)) + + f.Loggers[scope] = l + + return l +} + +type rateLimitedWriter struct { + io.Writer + addSuppressed bool + counter int + limiter *rate.Limiter +} + +// NewRateLimitedWriter is a writer limited by a token bucket rate-limiter to at most burst events per period. If addSuppressed is true then the number of suppressed events is appended to the output. +func NewRateLimitedWriter(writer io.Writer, period time.Duration, burst int, addSuppressed bool) io.Writer { + return &rateLimitedWriter{ + Writer: writer, + addSuppressed: addSuppressed, + counter: 0, + limiter: rate.NewLimiter(rate.Every(period), burst), + } +} + +func (w *rateLimitedWriter) Write(p []byte) (int, error) { + if !w.limiter.Allow() { + w.counter++ + return 0, nil + } + if w.addSuppressed && w.counter > 0 { + s := []byte(fmt.Sprintf(" (suppressed %d log events)", w.counter)) + p = append(p, s...) + } + n, err := w.Writer.Write(p) + w.counter = 0 + + return n, err +} + +// SetLevel sets the loglevel. +func (f *LoggerFactory) SetLevel(levelSpec string) { levels := strings.Split(levelSpec, ",") for _, s := range levels { scopedLevel := strings.SplitN(s, ":", 2) @@ -106,3 +163,14 @@ func (f *LoggerFactory) SetLevel(levelSpec string) { logger.SetLevel(l) } } + +// GetLevel gets the loglevel for the given scope. +func (f *LoggerFactory) GetLevel(scope string) string { + logLevel := f.DefaultLogLevel + scopeLevel, found := f.ScopeLevels[scope] + if found { + logLevel = scopeLevel + } + + return logLevel.String() +} diff --git a/pkg/logger/logger_test.go b/pkg/logger/logger_test.go new file mode 100644 index 00000000..ead27bd8 --- /dev/null +++ b/pkg/logger/logger_test.go @@ -0,0 +1,658 @@ +package logger + +import ( + "bytes" + "fmt" + "testing" + "time" + + "github.com/pion/transport/v3/test" + "github.com/stretchr/testify/assert" +) + +const testScope = "dummy-scope" + +type loggerTestCase struct { + name, defaultLogLevel, scopeLogLevel string + prep func(lf *LoggerFactory) + tester func(t *testing.T, lf *LoggerFactory) +} + +var loggerTests = []loggerTestCase{ + { + name: "default-loglevel", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "dummy scope: level") + + // resuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "default-loglevel-disable-scope", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "DISABLE", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Disabled", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "default-loglevel-error-scope", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "ERROR", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "default-loglevel-warn-scope", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "WARN", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Warn", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "default-loglevel-info-scope", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "INFO", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Info", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + + ld.Info("dummy") + assert.Containsf(t, lf.readr(), "dummy", "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "default-loglevel-debug-scope", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "DEBUG", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Debug", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + + ld.Info("dummy") + assert.Containsf(t, lf.readr(), "dummy", "INFO for level %s", level) + + ld.Debug("dummy") + assert.Containsf(t, lf.readr(), "dummy", "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "default-loglevel-trace-scope", + defaultLogLevel: "", // default is ERROR + scopeLogLevel: "TRACE", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Trace", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + + ld.Info("dummy") + assert.Containsf(t, lf.readr(), "dummy", "INFO for level %s", level) + + ld.Debug("dummy") + assert.Containsf(t, lf.readr(), "dummy", "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Containsf(t, lf.readr(), "dummy", "TRACE for level %s", level) + }, + }, + { + name: "override-loglevel-trace-scope", + defaultLogLevel: "all:TRACE", + scopeLogLevel: "ERROR", + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Trace", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "complex-loglevel-1", + defaultLogLevel: "all:TRACE", + scopeLogLevel: "TRACE", + prep: func(lf *LoggerFactory) { + lf.SetLevel("all:TRACE,dummy-scope:ERROR") + }, + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Trace", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, + { + name: "complex-loglevel-2", + defaultLogLevel: "all:TRACE", + scopeLogLevel: "TRACE", + prep: func(lf *LoggerFactory) { + lf.SetLevel("dummy-scope:DEBUG,nonExistentScope:TRACE,dummy-scope:ERROR,all:TRACE") + }, + tester: func(t *testing.T, lf *LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Trace", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "dummy scope: level") + + // reuse logger + ld := lf.NewLogger(testScope) + + ld.Error("dummy") + assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + + ld.Warn("dummy") + assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + + ld.Debug("dummy") + assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + }, + }, +} + +func (lf *LoggerFactory) len() int { + b, ok := lf.Writer.(*bytes.Buffer) + if !ok { + panic("not a test logger factory") + } + return b.Len() +} + +func (lf *LoggerFactory) lenr() int { + b, ok := lf.Writer.(*bytes.Buffer) + if !ok { + panic("not a test logger factory") + } + l := b.Len() + b.Reset() + return l +} + +func (lf *LoggerFactory) reset() { + b, ok := lf.Writer.(*bytes.Buffer) + if !ok { + panic("not a test logger factory") + } + b.Reset() +} + +func (lf *LoggerFactory) read() string { + b, ok := lf.Writer.(*bytes.Buffer) + if !ok { + panic("not a test logger factory") + } + return b.String() +} + +func (lf *LoggerFactory) readr() string { + b, ok := lf.Writer.(*bytes.Buffer) + if !ok { + panic("not a test logger factory") + } + ret := b.String() + b.Reset() + return ret +} + +func TestLogger(t *testing.T) { + lim := test.TimeOut(time.Second * 60) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + for _, c := range loggerTests { + t.Run(c.name, func(t *testing.T) { + // t.Logf("-------------- Running test: %s -------------", c.name) + + // create + loggerFactory := NewLoggerFactory(c.defaultLogLevel) + loggerFactory.Writer = &bytes.Buffer{} + + // create logger + _ = loggerFactory.NewLogger(testScope) + loggerFactory.SetLevel(fmt.Sprintf("%s:%s", testScope, c.scopeLogLevel)) + + // prepare + if c.prep != nil { + c.prep(loggerFactory) + } + + // t.Logf("%#v", loggerFactory) + // t.Logf("%#v", loggerFactory.ScopeLevels) + // t.Logf("%#v", logger) + + // test + c.tester(t, loggerFactory) + }) + } +} + +// rate-limiter tests + +type rateLimiterLoggerTestCase struct { + name string + addSuppressed bool + period time.Duration + burst int + prep func(lf *LoggerFactory) + tester func(t *testing.T, lf *LoggerFactory) +} + +var rateLimitedLoggerTests = []rateLimiterLoggerTestCase{ + { + name: "rate-limited-logger-default", + addSuppressed: true, + period: 10 * time.Millisecond, + burst: 1, + tester: func(t *testing.T, lf *LoggerFactory) { + // reuse logger + ld := lf.NewLogger(testScope) + + // only first call should succeed + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // wait until we get another token + time.Sleep(15 * time.Millisecond) + + ld.Info("dummy") + assert.Contains(t, lf.read(), "dummy") + assert.Contains(t, lf.readr(), "suppressed 2 log") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + }, + }, + { + name: "rate-limited-logger-suppressed-shown", + addSuppressed: true, + period: 10 * time.Millisecond, + burst: 1, + tester: func(t *testing.T, lf *LoggerFactory) { + // reuse logger + ld := lf.NewLogger(testScope) + + // first call should succeed + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + + // wait until we get another token + time.Sleep(15 * time.Millisecond) + + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + // no "suppressed" message should appear + assert.NotContains(t, lf.readr(), "suppressed 2 log") + + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + }, + }, + { + name: "rate-limited-logger-suppressed-supressed", + addSuppressed: false, + period: 10 * time.Millisecond, + burst: 1, + tester: func(t *testing.T, lf *LoggerFactory) { + // reuse logger + ld := lf.NewLogger(testScope) + + // only first call should succeed + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // wait until we get another token + time.Sleep(15 * time.Millisecond) + + ld.Info("dummy") + assert.Contains(t, lf.read(), "dummy") + assert.NotContains(t, lf.readr(), "suppressed 2 log") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + }, + }, + { + name: "rate-limited-logger-burst-3", + addSuppressed: true, + period: 10 * time.Millisecond, + burst: 3, + tester: func(t *testing.T, lf *LoggerFactory) { + // reuse logger + ld := lf.NewLogger(testScope) + + // only first 3 calls should succeed + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // wait until we get another token + time.Sleep(13 * time.Millisecond) + + ld.Info("dummy") + assert.Contains(t, lf.read(), "dummy") + assert.Contains(t, lf.readr(), "suppressed 2 log") + // consumed all tokens: these should be suppressed + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + }, + }, + { + name: "rate-limited-logger-independent", + addSuppressed: true, + period: 10 * time.Millisecond, + burst: 1, + tester: func(t *testing.T, lf *LoggerFactory) { + // reuse logger + ld := lf.NewLogger(testScope) + + // only first call should succeed + ld.Info("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // rate-limiters should be independent + ld.Error("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // wait until we get another token + time.Sleep(15 * time.Millisecond) + + ld.Info("dummy") + assert.Contains(t, lf.read(), "dummy") + assert.Contains(t, lf.readr(), "suppressed 2 log") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Info("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + ld.Error("dummy") + assert.Contains(t, lf.read(), "dummy") + assert.Contains(t, lf.readr(), "suppressed 2 log") + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + }, + }, + { + name: "rate-limited-logger-inactive-loggers-not-counted", + addSuppressed: true, + period: 10 * time.Millisecond, + burst: 1, + tester: func(t *testing.T, lf *LoggerFactory) { + // reuse logger + ld := lf.NewLogger(testScope) + + // Trace is inactive: no log + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "inactive") + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // wait until we would get another token + time.Sleep(15 * time.Millisecond) + + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "inactive") + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Trace("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + + // increase loglevel + lf.SetLevel(fmt.Sprintf("%s:TRACE", testScope)) + ld.Error("dummy") + assert.Contains(t, lf.readr(), "dummy") + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + ld.Error("dummy") + assert.Zerof(t, lf.lenr(), "suppressed") + }, + }, +} + +func TestRateLimitedLogger(t *testing.T) { + lim := test.TimeOut(time.Second * 60) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + for _, c := range rateLimitedLoggerTests { + t.Run(c.name, func(t *testing.T) { + // t.Logf("-------------- Running test: %s -------------", c.name) + + // create + loggerFactory := NewLoggerFactory("all:INFO") + loggerFactory.Writer = &bytes.Buffer{} + + // create ratee-limiter logger + _ = loggerFactory.NewRateLimitedLogger(testScope, c.period, c.burst, c.addSuppressed) + + // prepare + if c.prep != nil { + c.prep(loggerFactory) + } + + // t.Logf("%#v", loggerFactory) + // t.Logf("%#v", loggerFactory.ScopeLevels) + // t.Logf("%#v", logger) + + // test + c.tester(t, loggerFactory) + }) + } +} From d411f92b12d48f033c4fc20d868421f4d48888ee Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sat, 25 Nov 2023 18:47:20 +0100 Subject: [PATCH 059/243] chore: Fix linter error --- pkg/logger/logger_test.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pkg/logger/logger_test.go b/pkg/logger/logger_test.go index ead27bd8..5389fe27 100644 --- a/pkg/logger/logger_test.go +++ b/pkg/logger/logger_test.go @@ -327,6 +327,7 @@ var loggerTests = []loggerTestCase{ }, } +//nolint:golint,unused func (lf *LoggerFactory) len() int { b, ok := lf.Writer.(*bytes.Buffer) if !ok { @@ -345,6 +346,7 @@ func (lf *LoggerFactory) lenr() int { return l } +//nolint:golint,unused func (lf *LoggerFactory) reset() { b, ok := lf.Writer.(*bytes.Buffer) if !ok { From 1bed0fc336ada7a92d14578dde9dd2ea1170e9f3 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sun, 26 Nov 2023 19:16:53 +0100 Subject: [PATCH 060/243] chore: Introduce STUNner dataplane config API v1 The default config file version is now v1. The old v1alpha1 format is still accepted, but internally stunnerd keeps all config in v1 format. Old v1alpha1 configs are converted to v1 automatically in the config file parser. Also made the config file input pipeline more robust and added new tests for the config discovery client. --- cmd/stunnerd/main.go | 9 +- cmd/turncat/main.go | 24 +- config.go | 46 +- config_test.go | 264 ++++++++- deploy/manifests/stunner-test.yaml | 63 +++ go.mod | 10 +- go.sum | 20 +- handlers.go | 6 +- handlers_test.go | 64 +-- internal/manager/manager.go | 4 +- internal/manager/reconcile.go | 8 +- internal/object/admin.go | 33 +- internal/object/auth.go | 36 +- internal/object/cluster.go | 48 +- internal/object/listener.go | 42 +- internal/object/object.go | 12 +- pkg/apis/{v1alpha1 => v1}/admin.go | 2 +- pkg/apis/v1/auth.go | 130 +++++ pkg/apis/{v1alpha1 => v1}/cluster.go | 2 +- pkg/apis/{v1alpha1 => v1}/config.go | 4 +- pkg/apis/v1/default.go | 19 + pkg/apis/v1/errors.go | 26 + pkg/apis/{v1alpha1 => v1}/listener.go | 2 +- pkg/apis/v1/stunner.go | 172 ++++++ pkg/apis/v1/util.go | 198 +++++++ pkg/apis/v1alpha1/default.go | 5 - pkg/apis/v1alpha1/errors.go | 25 +- pkg/apis/v1alpha1/stunner.go | 56 +- pkg/apis/v1alpha1/util.go | 158 +----- pkg/config/client/client.go | 164 +++--- pkg/config/client/config.go | 70 ++- reconcile.go | 19 +- reconcile_test.go | 775 +++++++++++++------------- server.go | 10 +- server_unix_test.go | 27 +- stunner.go | 8 +- stunner_test.go | 436 ++++++++++----- turncat.go | 4 +- turncat_test.go | 27 +- uri.go | 10 +- 40 files changed, 1981 insertions(+), 1057 deletions(-) create mode 100644 deploy/manifests/stunner-test.yaml rename pkg/apis/{v1alpha1 => v1}/admin.go (99%) create mode 100644 pkg/apis/v1/auth.go rename pkg/apis/{v1alpha1 => v1}/cluster.go (99%) rename pkg/apis/{v1alpha1 => v1}/config.go (84%) create mode 100644 pkg/apis/v1/default.go create mode 100644 pkg/apis/v1/errors.go rename pkg/apis/{v1alpha1 => v1}/listener.go (99%) create mode 100644 pkg/apis/v1/stunner.go create mode 100644 pkg/apis/v1/util.go diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index a36275f6..5f047dce 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -11,7 +11,7 @@ import ( flag "github.com/spf13/pflag" "github.com/l7mp/stunner" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // usage: stunnerd -v turn://user1:passwd1@127.0.0.1:3478?transport=udp @@ -74,7 +74,7 @@ func main() { log.Infof("starting stunnerd instance %q", *id) - conf := make(chan v1alpha1.StunnerConfig, 1) + conf := make(chan stnrv1.StunnerConfig, 1) defer close(conf) var cancelConfigLoader context.CancelFunc @@ -108,6 +108,7 @@ func main() { defer cancel() cancelConfigLoader = cancel + // Watch closes the channel if err := st.WatchConfig(ctx, *config, conf); err != nil { log.Errorf("could not run config watcher: %s", err.Error()) os.Exit(1) @@ -131,7 +132,7 @@ func main() { os.Exit(0) case <-sigterm: - log.Infof("performing a graceful shutdown with %d active connections", + log.Infof("performing a graceful shutdown with %d active connection(s)", st.AllocationCount()) st.Shutdown() @@ -165,7 +166,7 @@ func main() { err := st.Reconcile(c) log.Trace("reconciliation ready") if err != nil { - if e, ok := err.(v1alpha1.ErrRestarted); ok { + if e, ok := err.(stnrv1.ErrRestarted); ok { log.Debugf("reconciliation ready: %s", e.Error()) } else { log.Errorf("could not reconcile new configuration "+ diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 155057eb..899b7e87 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -20,7 +20,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/config" "github.com/l7mp/stunner" - stunnerv1alpha1 "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/logger" ) @@ -103,7 +103,7 @@ func main() { t.Close() } -func getStunnerConf(uri string) (*stunnerv1alpha1.StunnerConfig, error) { +func getStunnerConf(uri string) (*stnrv1.StunnerConfig, error) { s := strings.Split(uri, "://") if len(s) < 2 { return nil, fmt.Errorf("cannot parse server URI") @@ -131,7 +131,7 @@ func getStunnerConf(uri string) (*stunnerv1alpha1.StunnerConfig, error) { } } -func getStunnerConfFromK8s(def string) (*stunnerv1alpha1.StunnerConfig, error) { +func getStunnerConfFromK8s(def string) (*stnrv1.StunnerConfig, error) { namespace, name, listener, err := parseK8sDef(def) if err != nil { return nil, err @@ -164,13 +164,13 @@ func getStunnerConfFromK8s(def string) (*stunnerv1alpha1.StunnerConfig, error) { defaultStunnerdConfigfileName) } - conf := stunnerv1alpha1.StunnerConfig{} + conf := stnrv1.StunnerConfig{} if err := json.Unmarshal([]byte(jsonConf), &conf); err != nil { return nil, err } // remove all but the named listener - ls := []stunnerv1alpha1.ListenerConfig{} + ls := []stnrv1.ListenerConfig{} for _, l := range conf.Listeners { // parse out the listener name (as per the Gateway API) from the TURN listener-name // (this is in the form: // @@ -196,13 +196,13 @@ func getStunnerConfFromK8s(def string) (*stunnerv1alpha1.StunnerConfig, error) { "specified TURN server URI", listener) } - conf.Listeners = []stunnerv1alpha1.ListenerConfig{{}} + conf.Listeners = []stnrv1.ListenerConfig{{}} copy(conf.Listeners, ls) return &conf, nil } -func getStunnerConfFromCLI(def string) (*stunnerv1alpha1.StunnerConfig, error) { +func getStunnerConfFromCLI(def string) (*stnrv1.StunnerConfig, error) { uri := fmt.Sprintf("turn://%s", def) conf, err := stunner.NewDefaultConfig(uri) @@ -225,15 +225,15 @@ func getStunnerConfFromCLI(def string) (*stunnerv1alpha1.StunnerConfig, error) { return conf, nil } -func getAuth(config *stunnerv1alpha1.StunnerConfig) (stunner.AuthGen, error) { +func getAuth(config *stnrv1.StunnerConfig) (stunner.AuthGen, error) { auth := config.Auth - atype, err := stunnerv1alpha1.NewAuthType(auth.Type) + atype, err := stnrv1.NewAuthType(auth.Type) if err != nil { return nil, err } switch atype { - case stunnerv1alpha1.AuthTypeLongTerm: + case stnrv1.AuthTypeEphemeral: s, found := auth.Credentials["secret"] if !found { return nil, fmt.Errorf("cannot find shared secret for %s authentication", @@ -243,7 +243,7 @@ func getAuth(config *stunnerv1alpha1.StunnerConfig) (stunner.AuthGen, error) { return turn.GenerateLongTermCredentials(s, defaultDuration) }, nil - case stunnerv1alpha1.AuthTypePlainText: + case stnrv1.AuthTypeStatic: u, found := auth.Credentials["username"] if !found { return nil, fmt.Errorf("cannot find username for %s authentication", @@ -264,7 +264,7 @@ func getAuth(config *stunnerv1alpha1.StunnerConfig) (stunner.AuthGen, error) { } } -func getStunnerURI(config *stunnerv1alpha1.StunnerConfig) (string, error) { +func getStunnerURI(config *stnrv1.StunnerConfig) (string, error) { // we should have only a single listener at this point if len(config.Listeners) != 1 { return "", fmt.Errorf("cannot find listener in STUNner configuration: %s", diff --git a/config.go b/config.go index b7076c40..476b520f 100644 --- a/config.go +++ b/config.go @@ -9,7 +9,7 @@ import ( "github.com/pion/transport/v3" "github.com/l7mp/stunner/internal/resolver" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" cds "github.com/l7mp/stunner/pkg/config/client" ) @@ -48,7 +48,7 @@ type Options struct { // `turn://user:pass@127.0.0.1:3478?transport=udp` will be parsed into a STUNner configuration with // a server running on the localhost at UDP port 3478, with plain-text authentication using the // username/password pair `user:pass`. Health-checks and metric scarping are disabled. -func NewDefaultConfig(uri string) (*v1alpha1.StunnerConfig, error) { +func NewDefaultConfig(uri string) (*stnrv1.StunnerConfig, error) { u, err := ParseUri(uri) if err != nil { return nil, fmt.Errorf("Invalid URI '%s': %s", uri, err) @@ -59,29 +59,29 @@ func NewDefaultConfig(uri string) (*v1alpha1.StunnerConfig, error) { } h := "" - c := &v1alpha1.StunnerConfig{ - ApiVersion: v1alpha1.ApiVersion, - Admin: v1alpha1.AdminConfig{ - LogLevel: v1alpha1.DefaultLogLevel, + c := &stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stnrv1.DefaultLogLevel, // MetricsEndpoint: "http://:8088", HealthCheckEndpoint: &h, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", - Realm: v1alpha1.DefaultRealm, + Realm: stnrv1.DefaultRealm, Credentials: map[string]string{ "username": u.Username, "password": u.Password, }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Protocol: u.Protocol, Addr: u.Address, Port: u.Port, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Type: "STATIC", Endpoints: []string{"0.0.0.0/0"}, @@ -106,45 +106,45 @@ func NewDefaultConfig(uri string) (*v1alpha1.StunnerConfig, error) { } // GetConfig returns the configuration of the running STUNner daemon. -func (s *Stunner) GetConfig() *v1alpha1.StunnerConfig { +func (s *Stunner) GetConfig() *stnrv1.StunnerConfig { s.log.Tracef("GetConfig") // singletons, but we want to avoid panics when GetConfig is called on an uninitialized // STUNner object - adminConf := v1alpha1.AdminConfig{} + adminConf := stnrv1.AdminConfig{} if len(s.adminManager.Keys()) > 0 { - adminConf = *s.GetAdmin().GetConfig().(*v1alpha1.AdminConfig) + adminConf = *s.GetAdmin().GetConfig().(*stnrv1.AdminConfig) } - authConf := v1alpha1.AuthConfig{} + authConf := stnrv1.AuthConfig{} if len(s.authManager.Keys()) > 0 { - authConf = *s.GetAuth().GetConfig().(*v1alpha1.AuthConfig) + authConf = *s.GetAuth().GetConfig().(*stnrv1.AuthConfig) } listeners := s.listenerManager.Keys() clusters := s.clusterManager.Keys() - c := v1alpha1.StunnerConfig{ + c := stnrv1.StunnerConfig{ ApiVersion: s.version, Admin: adminConf, Auth: authConf, - Listeners: make([]v1alpha1.ListenerConfig, len(listeners)), - Clusters: make([]v1alpha1.ClusterConfig, len(clusters)), + Listeners: make([]stnrv1.ListenerConfig, len(listeners)), + Clusters: make([]stnrv1.ClusterConfig, len(clusters)), } for i, name := range listeners { - c.Listeners[i] = *s.GetListener(name).GetConfig().(*v1alpha1.ListenerConfig) + c.Listeners[i] = *s.GetListener(name).GetConfig().(*stnrv1.ListenerConfig) } for i, name := range clusters { - c.Clusters[i] = *s.GetCluster(name).GetConfig().(*v1alpha1.ClusterConfig) + c.Clusters[i] = *s.GetCluster(name).GetConfig().(*stnrv1.ClusterConfig) } return &c } // LoadConfig loads a configuration from an origin. This is a shim wrapper around ConfigOrigin.Load. -func (s *Stunner) LoadConfig(config string) (*v1alpha1.StunnerConfig, error) { +func (s *Stunner) LoadConfig(config string) (*stnrv1.StunnerConfig, error) { client, err := cds.NewClient(config, s.id, s.logger) if err != nil { return nil, err @@ -154,8 +154,8 @@ func (s *Stunner) LoadConfig(config string) (*v1alpha1.StunnerConfig, error) { } // WatchConfig watches a configuration from an origin. This is a shim wrapper around ConfigOrigin.Watch. -func (s *Stunner) WatchConfig(ctx context.Context, config string, ch chan<- v1alpha1.StunnerConfig) error { - client, err := cds.NewClient(config, s.id, s.logger) +func (s *Stunner) WatchConfig(ctx context.Context, origin string, ch chan<- stnrv1.StunnerConfig) error { + client, err := cds.NewClient(origin, s.id, s.logger) if err != nil { return err } diff --git a/config_test.go b/config_test.go index 0161ae9f..e47b14a7 100644 --- a/config_test.go +++ b/config_test.go @@ -2,18 +2,21 @@ package stunner import ( "context" + "encoding/json" "fmt" "net" + "net/http" "os" "strings" "testing" "time" + "github.com/gorilla/websocket" "github.com/pion/transport/v3/test" "github.com/stretchr/testify/assert" "sigs.k8s.io/yaml" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) @@ -110,7 +113,7 @@ func TestStunnerConfigFileRoundTrip(t *testing.T) { file, err2 := yaml.Marshal(c) assert.NoError(t, err2, "marschal config fike") - newConf := &v1alpha1.StunnerConfig{} + newConf := &stnrv1.StunnerConfig{} err = yaml.Unmarshal(file, newConf) assert.NoError(t, err, "unmarschal config from file") @@ -136,7 +139,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { log.Debug("creating a temp file for config") f, err := os.CreateTemp("", "stunner_conf_*.yaml") assert.NoError(t, err, "creating temp config file") - // we just need the filename for now so we remove the fle first + // we just need the filename for now so we remove the file first file := f.Name() assert.NoError(t, os.Remove(file), "removing temp config file") @@ -144,12 +147,13 @@ func TestStunnerConfigFileWatcher(t *testing.T) { stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel}) log.Debug("starting watcher") - conf := make(chan v1alpha1.StunnerConfig, 1) + conf := make(chan stnrv1.StunnerConfig, 1) defer close(conf) log.Debug("init watcher with nonexistent config file") ctx, cancel := context.WithCancel(context.Background()) defer cancel() + err = stunner.WatchConfig(ctx, file, conf) assert.NoError(t, err, "creating config watcher") @@ -180,8 +184,8 @@ func TestStunnerConfigFileWatcher(t *testing.T) { _, err = f.Write(y) assert.NoError(t, err, "write config to temp file") - // wait a bit so that the watcher has time to react - time.Sleep(50 * time.Millisecond) + // // wait a bit so that the watcher has time to react + // time.Sleep(50 * time.Millisecond) // first read should yield a zeroconfig c2, ok := <-conf @@ -194,6 +198,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { checkDefaultConfig(t, &c2, "TURN-UDP") log.Debug("write a wrong config file (WatchConfig does not validate)") + c2.Listeners[0].Protocol = "dummy" y, err = yaml.Marshal(c2) assert.NoError(t, err, "marshal config file") @@ -207,9 +212,6 @@ func TestStunnerConfigFileWatcher(t *testing.T) { // this makes sure that we do not share anything with ConfigWatch c2.Listeners[0].PublicAddr = "AAAAAAAAAAAAAa" - // wait a bit so that the watcher has time to react - time.Sleep(50 * time.Millisecond) - c3 := <-conf checkDefaultConfig(t, &c3, "dummy") @@ -224,16 +226,246 @@ func TestStunnerConfigFileWatcher(t *testing.T) { _, err = f.Write(y) assert.NoError(t, err, "write config to temp file") - // wait a bit so that the watcher has time to react - time.Sleep(50 * time.Millisecond) - - // read back result c4 := <-conf checkDefaultConfig(t, &c4, "TURN-TCP") stunner.Close() } +const ( + testConfigV1 = `{"version":"v1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"static","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` + testConfigV1A1 = `{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"longterm","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` +) + +// test with v1alpha1 and v1 +func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { + lim := test.TimeOut(time.Second * 10) + defer lim.Stop() + + loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) + log := loggerFactory.NewLogger("test-watcher") + + testName := "TestStunnerConfigFileWatcher" + log.Debugf("-------------- Running test: %s -------------", testName) + + log.Debug("creating a temp file for config") + f, err := os.CreateTemp("", "stunner_conf_*.yaml") + assert.NoError(t, err, "creating temp config file") + // we just need the filename for now so we remove the file first + file := f.Name() + assert.NoError(t, os.Remove(file), "removing temp config file") + + log.Debug("creating a stunnerd") + stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel}) + + log.Debug("starting watcher") + conf := make(chan stnrv1.StunnerConfig, 1) + defer close(conf) + + log.Debug("init watcher with nonexistent config file") + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + err = stunner.WatchConfig(ctx, file, conf) + assert.NoError(t, err, "creating config watcher") + + // nothing should happen here: wait a bit so that the watcher has comfortable time to start + time.Sleep(50 * time.Millisecond) + + log.Debug("write v1 config and check") + + // recreate the temp file and write config + f, err = os.OpenFile(file, os.O_RDWR|os.O_CREATE, 0644) + assert.NoError(t, err, "recreate temp config file") + defer os.Remove(file) + + err = f.Truncate(0) + assert.NoError(t, err, "truncate temp file") + _, err = f.Seek(0, 0) + assert.NoError(t, err, "seek temp file") + _, err = f.WriteString(testConfigV1) + assert.NoError(t, err, "write config to temp file") + + // first read should yield a zeroconfig + c2, ok := <-conf + assert.True(t, ok, "zeroconfig emitted") + checkZeroConfig(t, &c2, stunner.GetId()) + + // second read yields the real config + c2, ok = <-conf + assert.True(t, ok, "config emitted") + + assert.Equal(t, stnrv1.ApiVersion, c2.ApiVersion, "version") + assert.Equal(t, "all:ERROR", c2.Admin.LogLevel, "loglevel") + assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") + assert.Len(t, c2.Listeners, 1, "listeners len") + assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") + assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") + assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") + assert.Len(t, c2.Clusters, 1, "clusters len") + assert.Equal(t, "echo-server-cluster", c2.Clusters[0].Name, "cluster name") + assert.Equal(t, "STATIC", c2.Clusters[0].Type, "cluster proto") + assert.Len(t, c2.Clusters[0].Endpoints, 1, "endpoints len") + assert.Equal(t, "1.2.3.5", c2.Clusters[0].Endpoints[0], "cluster port") + + err = f.Truncate(0) + assert.NoError(t, err, "truncate temp file") + _, err = f.Seek(0, 0) + assert.NoError(t, err, "seek temp file") + _, err = f.WriteString(testConfigV1A1) + assert.NoError(t, err, "write config to temp file") + + c2, ok = <-conf + assert.True(t, ok, "config emitted") + + assert.Equal(t, stnrv1.ApiVersion, c2.ApiVersion, "version") + assert.Equal(t, "all:ERROR", c2.Admin.LogLevel, "loglevel") + assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") + assert.Len(t, c2.Listeners, 1, "listeners len") + assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") + assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") + assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") + assert.Len(t, c2.Clusters, 1, "clusters len") + assert.Equal(t, "echo-server-cluster", c2.Clusters[0].Name, "cluster name") + assert.Equal(t, "STATIC", c2.Clusters[0].Type, "cluster proto") + assert.Len(t, c2.Clusters[0].Endpoints, 1, "endpoints len") + assert.Equal(t, "1.2.3.5", c2.Clusters[0].Endpoints[0], "cluster port") + + stunner.Close() +} + +func TestStunnerConfigPollerMultiVersion(t *testing.T) { + lim := test.TimeOut(time.Second * 10) + defer lim.Stop() + + loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) + log := loggerFactory.NewLogger("test-poller") + + testName := "TestStunnerConfigPoller" + log.Debugf("-------------- Running test: %s -------------", testName) + + log.Debug("creating a mock CDS server") + addr := "localhost:13478" + origin := "ws://" + addr + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + s := &http.Server{Addr: addr} + defer s.Close() + + http.HandleFunc("/api/v1/config/watch", + func(w http.ResponseWriter, req *http.Request) { + upgrader := websocket.Upgrader{ + ReadBufferSize: 1024, + WriteBufferSize: 1024, + } + + conn, err := upgrader.Upgrade(w, req, nil) + assert.NoError(t, err, "upgrade HTTP connection") + defer func() { _ = conn.Close() }() + + // for the pong handler: conn.Close() will kill this + go func() { + for { + _, _, err := conn.ReadMessage() + if err != nil { + return + } + } + }() + + conn.SetPingHandler(func(string) error { + return conn.WriteMessage(websocket.PongMessage, []byte("keepalive")) + }) + + // send initial config + z := cdsclient.ZeroConfig("tester") + j, err := json.Marshal(z) + assert.NoError(t, err, "json marshal zeroconfig") + assert.NoError(t, conn.WriteMessage(websocket.TextMessage, j), "write zeroconfig") + + // send v1config + assert.NoError(t, conn.WriteMessage(websocket.TextMessage, []byte(testConfigV1)), "write config v1") + + // send v1config + assert.NoError(t, conn.WriteMessage(websocket.TextMessage, []byte(testConfigV1A1)), "write config v1alpha1") + + select { + case <-ctx.Done(): + case <-req.Context().Done(): + } + + conn.Close() + }) + + // serve + go func() { + _ = s.ListenAndServe() + }() + + // wait a bit so that the server has time to setup + time.Sleep(50 * time.Millisecond) + + log.Debug("creating a stunnerd") + stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel}) + + log.Debug("starting watcher") + conf := make(chan stnrv1.StunnerConfig, 1) + defer close(conf) + + log.Debug("init config poller") + assert.NoError(t, stunner.WatchConfig(ctx, origin, conf), "creating config poller") + + // first read should yield a zeroconfig + c2, ok := <-conf + assert.True(t, ok, "zeroconfig emitted") + checkZeroConfig(t, &c2, "tester") + + // second read yields a v1 config + c2, ok = <-conf + assert.True(t, ok, "config emitted") + + assert.Equal(t, stnrv1.ApiVersion, c2.ApiVersion, "version") + assert.Equal(t, "all:ERROR", c2.Admin.LogLevel, "loglevel") + assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") + assert.Len(t, c2.Listeners, 1, "listeners len") + assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") + assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") + assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") + assert.Len(t, c2.Clusters, 1, "clusters len") + assert.Equal(t, "echo-server-cluster", c2.Clusters[0].Name, "cluster name") + assert.Equal(t, "STATIC", c2.Clusters[0].Type, "cluster proto") + assert.Len(t, c2.Clusters[0].Endpoints, 1, "endpoints len") + assert.Equal(t, "1.2.3.5", c2.Clusters[0].Endpoints[0], "cluster port") + + // thrird read yields a v1alpha1 config + c2, ok = <-conf + assert.True(t, ok, "config emitted") + + assert.Equal(t, stnrv1.ApiVersion, c2.ApiVersion, "version") + assert.Equal(t, "all:ERROR", c2.Admin.LogLevel, "loglevel") + assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") + assert.Len(t, c2.Listeners, 1, "listeners len") + assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") + assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") + assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") + assert.Len(t, c2.Clusters, 1, "clusters len") + assert.Equal(t, "echo-server-cluster", c2.Clusters[0].Name, "cluster name") + assert.Equal(t, "STATIC", c2.Clusters[0].Type, "cluster proto") + assert.Len(t, c2.Clusters[0].Endpoints, 1, "endpoints len") + assert.Equal(t, "1.2.3.5", c2.Clusters[0].Endpoints[0], "cluster port") + + stunner.Close() +} + func TestStunnerURIParser(t *testing.T) { lim := test.TimeOut(time.Second * 30) defer lim.Stop() @@ -287,8 +519,8 @@ func TestStunnerURIParser(t *testing.T) { } } -func checkDefaultConfig(t *testing.T, c *v1alpha1.StunnerConfig, proto string) { - assert.Equal(t, "plaintext", c.Auth.Type, "auth-type") +func checkDefaultConfig(t *testing.T, c *stnrv1.StunnerConfig, proto string) { + assert.Equal(t, "static", c.Auth.Type, "auth-type") assert.Equal(t, "user1", c.Auth.Credentials["username"], "username") assert.Equal(t, "passwd1", c.Auth.Credentials["password"], "passwd") assert.Len(t, c.Listeners, 1, "listeners len") @@ -301,6 +533,6 @@ func checkDefaultConfig(t *testing.T, c *v1alpha1.StunnerConfig, proto string) { assert.Equal(t, "0.0.0.0/0", c.Clusters[0].Endpoints[0], "endpoint") } -func checkZeroConfig(t *testing.T, c *v1alpha1.StunnerConfig, id string) { +func checkZeroConfig(t *testing.T, c *stnrv1.StunnerConfig, id string) { assert.True(t, c.DeepEqual(cdsclient.ZeroConfig(id)), "zeroconfig ok") } diff --git a/deploy/manifests/stunner-test.yaml b/deploy/manifests/stunner-test.yaml new file mode 100644 index 00000000..1c4ce998 --- /dev/null +++ b/deploy/manifests/stunner-test.yaml @@ -0,0 +1,63 @@ +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: GatewayClass +metadata: + name: stunner-gatewayclass +spec: + controllerName: "stunner.l7mp.io/gateway-operator" + parametersRef: + group: "stunner.l7mp.io" + kind: GatewayConfig + name: stunner-gatewayconfig + namespace: stunner + description: "STUNner is a WebRTC ingress gateway for Kubernetes" +--- + +apiVersion: stunner.l7mp.io/v1alpha1 +kind: GatewayConfig +metadata: + name: stunner-gatewayconfig + namespace: stunner +spec: + realm: stunner.l7mp.io + authType: plaintext + userName: "user-1" + password: "pass-1" + +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: TURN-UDP +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: tcp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: tcp-listener + port: 3478 + protocol: TURN-TCP +--- +apiVersion: gateway.networking.k8s.io/v1alpha2 +kind: UDPRoute +metadata: + name: media-plane + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + - name: tcp-gateway + rules: + - backendRefs: + - name: media-plane + namespace: default diff --git a/go.mod b/go.mod index 5fac9ce1..67e91b8e 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/fsnotify/fsnotify v1.6.0 github.com/google/uuid v1.3.1 - github.com/gorilla/websocket v1.5.0 + github.com/gorilla/websocket v1.5.1 github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb github.com/pion/dtls/v2 v2.2.7 github.com/pion/logging v0.2.2 @@ -15,7 +15,7 @@ require ( github.com/prometheus/client_golang v1.16.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 - golang.org/x/sys v0.12.0 + golang.org/x/sys v0.13.0 golang.org/x/time v0.4.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 @@ -54,10 +54,10 @@ require ( github.com/prometheus/client_model v0.4.0 // indirect github.com/prometheus/common v0.44.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect - golang.org/x/crypto v0.13.0 // indirect - golang.org/x/net v0.15.0 // indirect + golang.org/x/crypto v0.14.0 // indirect + golang.org/x/net v0.17.0 // indirect golang.org/x/oauth2 v0.8.0 // indirect - golang.org/x/term v0.12.0 // indirect + golang.org/x/term v0.13.0 // indirect golang.org/x/text v0.13.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/protobuf v1.31.0 // indirect diff --git a/go.sum b/go.sum index 8a8fe316..5ff91d3f 100644 --- a/go.sum +++ b/go.sum @@ -182,8 +182,8 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+ github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= -github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc= -github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY= +github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= @@ -314,8 +314,8 @@ golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5y golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/crypto v0.13.0 h1:mvySKfSWJ+UKUii46M40LOvyWfN0s2U+46/jDd0e6Ck= -golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc= +golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc= +golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -396,8 +396,8 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/net v0.15.0 h1:ugBLEUaxABaB5AJqW9enI0ACdci2RUd4eP51NTBvuJ8= -golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk= +golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM= +golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -478,16 +478,16 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.12.0 h1:CM0HF96J0hcLAwsHPJZjfdNzs0gftsLfgKt57wWHJ0o= -golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE= +golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= -golang.org/x/term v0.12.0 h1:/ZfYdc3zq+q02Rv9vGqTeSItdzZTSNDmfTi0mBAuidU= -golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU= +golang.org/x/term v0.13.0 h1:bb+I9cTfFazGW51MZqBVmZy7+JEJMouUHTUSKVQLBek= +golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= diff --git a/handlers.go b/handlers.go index 28eb2138..2d7953dc 100644 --- a/handlers.go +++ b/handlers.go @@ -7,7 +7,7 @@ import ( "github.com/l7mp/stunner/internal/object" "github.com/l7mp/stunner/internal/util" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" a12n "github.com/l7mp/stunner/pkg/authentication" ) @@ -21,7 +21,7 @@ func (s *Stunner) NewAuthHandler() a12n.AuthHandler { auth := s.GetAuth() switch auth.Type { - case v1alpha1.AuthTypePlainText: + case stnrv1.AuthTypeStatic: auth.Log.Infof("plaintext auth request: username=%q realm=%q srcAddr=%v\n", username, realm, srcAddr) @@ -34,7 +34,7 @@ func (s *Stunner) NewAuthHandler() a12n.AuthHandler { auth.Log.Info("plaintext auth request: failed: invalid username") return nil, false - case v1alpha1.AuthTypeLongTerm: + case stnrv1.AuthTypeEphemeral: auth.Log.Infof("longterm auth request: username=%q realm=%q srcAddr=%v", username, realm, srcAddr) diff --git a/handlers_test.go b/handlers_test.go index 0e2edc42..9911fe05 100644 --- a/handlers_test.go +++ b/handlers_test.go @@ -14,7 +14,7 @@ import ( "github.com/pion/turn/v3" "github.com/stretchr/testify/assert" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/logger" ) @@ -39,7 +39,7 @@ func longTermCredentials(username string, sharedSecret string) (string, error) { type StunnerTestAuthWithVnet struct { testName string - conf v1alpha1.StunnerConfig + conf stnrv1.StunnerConfig authCred func() (string, string) clientAddr string } @@ -48,26 +48,26 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ { testName: "plaintext", clientAddr: "1.1.1.1", - conf: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + conf: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -76,25 +76,25 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, { testName: "longterm - plain timestamp in username", - conf: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + conf: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -106,25 +106,25 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, { testName: "longterm - timestamp:userid in username", - conf: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + conf: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -138,25 +138,25 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, { testName: "longterm - userid:timestamp in username", - conf: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + conf: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -170,25 +170,25 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, { testName: "longterm - userid:timestamp:ramdom-crap in username", - conf: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + conf: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, diff --git a/internal/manager/manager.go b/internal/manager/manager.go index 328cc5a7..d4a791e0 100644 --- a/internal/manager/manager.go +++ b/internal/manager/manager.go @@ -8,7 +8,7 @@ import ( "github.com/pion/logging" "github.com/l7mp/stunner/internal/object" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // Manager stores STUNner objects @@ -20,7 +20,7 @@ type Manager interface { // Delete deletes the object from the store, may return ErrReturnRequired Delete(o object.Object) error // PrepareReconciliation prepares the reconciliation of the manager - PrepareReconciliation(confs []v1alpha1.Config, stunenerConf v1alpha1.Config) (*ReconciliationState, error) + PrepareReconciliation(confs []stnrv1.Config, stunenerConf stnrv1.Config) (*ReconciliationState, error) // FinishReconciliation finishes the reconciliation from the specified state FinishReconciliation(state *ReconciliationState) error // Keys returns the names iof all objects in the store in alphabetical order, suitable for iteration diff --git a/internal/manager/reconcile.go b/internal/manager/reconcile.go index 37bd3f27..7c8a30b5 100644 --- a/internal/manager/reconcile.go +++ b/internal/manager/reconcile.go @@ -4,12 +4,12 @@ import ( "fmt" "github.com/l7mp/stunner/internal/object" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) type ReconcileJob struct { Object object.Object - NewConfig, OldConfig v1alpha1.Config + NewConfig, OldConfig stnrv1.Config } type ReconciliationState struct { @@ -20,7 +20,7 @@ type ReconciliationState struct { // PrepareReconciliation prepares the reconciliation of the objects handled by the manager and returns a // set of reconciliation jobs to be performed, ErrRestartRequired if the server needs to be // restarted, and an error if the config was not accepted. Configuration must be validated. -func (m *managerImpl) PrepareReconciliation(confs []v1alpha1.Config, stunnerConf v1alpha1.Config) (*ReconciliationState, error) { +func (m *managerImpl) PrepareReconciliation(confs []stnrv1.Config, stunnerConf stnrv1.Config) (*ReconciliationState, error) { m.log.Tracef("preparing reconciliation") state := ReconciliationState{ @@ -134,7 +134,7 @@ func (m *managerImpl) FinishReconciliation(state *ReconciliationState) error { return nil } -func findConfByName(confs []v1alpha1.Config, name string) bool { +func findConfByName(confs []stnrv1.Config, name string) bool { for _, c := range confs { if c.ConfigName() == name { return true diff --git a/internal/object/admin.go b/internal/object/admin.go index 103a2eb8..49a4c980 100644 --- a/internal/object/admin.go +++ b/internal/object/admin.go @@ -8,14 +8,13 @@ import ( "net/http" "net/url" "strconv" - // "time" "github.com/pion/logging" "github.com/prometheus/client_golang/prometheus/promhttp" health "github.com/heptiolabs/healthcheck" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) const DefaultAdminObjectName = "DefaultAdmin" @@ -31,10 +30,10 @@ type Admin struct { } // NewAdmin creates a new Admin object. -func NewAdmin(conf v1alpha1.Config, dryRun bool, rc health.Check, logger logging.LoggerFactory) (Object, error) { - req, ok := conf.(*v1alpha1.AdminConfig) +func NewAdmin(conf stnrv1.Config, dryRun bool, rc health.Check, logger logging.LoggerFactory) (Object, error) { + req, ok := conf.(*stnrv1.AdminConfig) if !ok { - return nil, v1alpha1.ErrInvalidConf + return nil, stnrv1.ErrInvalidConf } admin := Admin{ @@ -58,16 +57,16 @@ func NewAdmin(conf v1alpha1.Config, dryRun bool, rc health.Check, logger logging // Inspect examines whether a configuration change requires a reconciliation (returns true if it // does) or restart (returns ErrRestartRequired). -func (a *Admin) Inspect(old, new, full v1alpha1.Config) (bool, error) { +func (a *Admin) Inspect(old, new, full stnrv1.Config) (bool, error) { return !old.DeepEqual(new), nil } // Reconcile updates the authenticator for a new configuration. Requires a valid reconciliation // request. -func (a *Admin) Reconcile(conf v1alpha1.Config) error { - req, ok := conf.(*v1alpha1.AdminConfig) +func (a *Admin) Reconcile(conf stnrv1.Config) error { + req, ok := conf.(*stnrv1.AdminConfig) if !ok { - return v1alpha1.ErrInvalidConf + return stnrv1.ErrInvalidConf } if err := req.Validate(); err != nil { @@ -96,7 +95,7 @@ func (a *Admin) Reconcile(conf v1alpha1.Config) error { // ObjectName returns the name of the object. func (a *Admin) ObjectName() string { - return v1alpha1.DefaultAdminName + return stnrv1.DefaultAdminName } // ObjectType returns the type of the object. @@ -105,14 +104,14 @@ func (a *Admin) ObjectType() string { } // GetConfig returns the configuration of the running object. -func (a *Admin) GetConfig() v1alpha1.Config { +func (a *Admin) GetConfig() stnrv1.Config { a.log.Tracef("GetConfig") // use a copy when taking the pointer: we don't want anyone downstream messing with our own // copies h := a.HealthCheckEndpoint - return &v1alpha1.AdminConfig{ + return &stnrv1.AdminConfig{ Name: a.Name, LogLevel: a.LogLevel, MetricsEndpoint: a.MetricsEndpoint, @@ -143,7 +142,7 @@ func (a *Admin) Close() error { return nil } -func (a *Admin) reconcileMetrics(req *v1alpha1.AdminConfig) error { +func (a *Admin) reconcileMetrics(req *stnrv1.AdminConfig) error { a.log.Trace("reconcileMetrics") if a.DryRun { @@ -205,7 +204,7 @@ end: } // req MUST be validated! -func (a *Admin) reconcileHealthCheck(req *v1alpha1.AdminConfig) error { +func (a *Admin) reconcileHealthCheck(req *stnrv1.AdminConfig) error { a.log.Trace("reconcileHealthCheck") // if req is validated then either @@ -287,7 +286,7 @@ func NewAdminFactory(dryRun bool, rc health.Check, logger logging.LoggerFactory) // New can produce a new Admin object from the given configuration. A nil config will create an // empty admin object (useful for creating throwaway objects for, e.g., calling Inpect) -func (f *AdminFactory) New(conf v1alpha1.Config) (Object, error) { +func (f *AdminFactory) New(conf stnrv1.Config) (Object, error) { if conf == nil { return &Admin{}, nil } @@ -315,7 +314,7 @@ func getHealthAddr(e string) string { port := u.Port() if port == "" { - port = fmt.Sprintf("%d", v1alpha1.DefaultHealthCheckPort) + port = fmt.Sprintf("%d", stnrv1.DefaultHealthCheckPort) } return addr + ":" + port @@ -341,7 +340,7 @@ func getMetricsAddr(e string) (string, string) { port := u.Port() if port == "" { - port = strconv.Itoa(v1alpha1.DefaultMetricsPort) + port = strconv.Itoa(stnrv1.DefaultMetricsPort) } addr = addr + ":" + port diff --git a/internal/object/auth.go b/internal/object/auth.go index cc770bea..da400d1c 100644 --- a/internal/object/auth.go +++ b/internal/object/auth.go @@ -6,21 +6,21 @@ import ( "github.com/pion/logging" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // Auth is the STUNner authenticator type Auth struct { - Type v1alpha1.AuthType + Type stnrv1.AuthType Realm, Username, Password, Secret string Log logging.LeveledLogger } // NewAuth creates a new authenticator. -func NewAuth(conf v1alpha1.Config, logger logging.LoggerFactory) (Object, error) { - req, ok := conf.(*v1alpha1.AuthConfig) +func NewAuth(conf stnrv1.Config, logger logging.LoggerFactory) (Object, error) { + req, ok := conf.(*stnrv1.AuthConfig) if !ok { - return nil, v1alpha1.ErrInvalidConf + return nil, stnrv1.ErrInvalidConf } auth := Auth{Log: logger.NewLogger("stunner-auth")} @@ -35,15 +35,15 @@ func NewAuth(conf v1alpha1.Config, logger logging.LoggerFactory) (Object, error) // Inspect examines whether a configuration change requires a reconciliation (returns true if it // does) or restart (returns ErrRestartRequired). -func (auth *Auth) Inspect(old, new, full v1alpha1.Config) (bool, error) { +func (auth *Auth) Inspect(old, new, full stnrv1.Config) (bool, error) { return !old.DeepEqual(new), nil } // Reconcile updates the authenticator for a new configuration. -func (auth *Auth) Reconcile(conf v1alpha1.Config) error { - req, ok := conf.(*v1alpha1.AuthConfig) +func (auth *Auth) Reconcile(conf stnrv1.Config) error { + req, ok := conf.(*stnrv1.AuthConfig) if !ok { - return v1alpha1.ErrInvalidConf + return stnrv1.ErrInvalidConf } if err := req.Validate(); err != nil { @@ -51,7 +51,7 @@ func (auth *Auth) Reconcile(conf v1alpha1.Config) error { } // type already validated - atype, _ := v1alpha1.NewAuthType(req.Type) + atype, _ := stnrv1.NewAuthType(req.Type) auth.Log.Debugf("using authentication: %s", atype.String()) @@ -59,10 +59,10 @@ func (auth *Auth) Reconcile(conf v1alpha1.Config) error { auth.Type = atype auth.Realm = req.Realm switch atype { - case v1alpha1.AuthTypePlainText: + case stnrv1.AuthTypeStatic: auth.Username = req.Credentials["username"] auth.Password = req.Credentials["password"] - case v1alpha1.AuthTypeLongTerm: + case stnrv1.AuthTypeEphemeral: auth.Secret = req.Credentials["secret"] } @@ -72,7 +72,7 @@ func (auth *Auth) Reconcile(conf v1alpha1.Config) error { // ObjectName returns the name of the object func (auth *Auth) ObjectName() string { // singleton! - return v1alpha1.DefaultAuthName + return stnrv1.DefaultAuthName } // ObjectType returns the type of the object @@ -81,18 +81,18 @@ func (a *Auth) ObjectType() string { } // GetConfig returns the configuration of the running authenticator -func (auth *Auth) GetConfig() v1alpha1.Config { +func (auth *Auth) GetConfig() stnrv1.Config { auth.Log.Tracef("GetConfig") - r := v1alpha1.AuthConfig{ + r := stnrv1.AuthConfig{ Type: auth.Type.String(), Realm: auth.Realm, Credentials: make(map[string]string), } switch auth.Type { - case v1alpha1.AuthTypePlainText: + case stnrv1.AuthTypeStatic: r.Credentials["username"] = auth.Username r.Credentials["password"] = auth.Password - case v1alpha1.AuthTypeLongTerm: + case stnrv1.AuthTypeEphemeral: r.Credentials["secret"] = auth.Secret } @@ -117,7 +117,7 @@ func NewAuthFactory(logger logging.LoggerFactory) Factory { // New can produce a new Auth object from the given configuration. A nil config will create an // empty auth object (useful for creating throwaway objects for, e.g., calling Inpect) -func (f *AuthFactory) New(conf v1alpha1.Config) (Object, error) { +func (f *AuthFactory) New(conf stnrv1.Config) (Object, error) { if conf == nil { return &Auth{}, nil } diff --git a/internal/object/cluster.go b/internal/object/cluster.go index a2dfb3c7..0fc621dd 100644 --- a/internal/object/cluster.go +++ b/internal/object/cluster.go @@ -10,14 +10,14 @@ import ( "github.com/l7mp/stunner/internal/resolver" "github.com/l7mp/stunner/internal/util" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // Listener implements a STUNner cluster type Cluster struct { Name string - Type v1alpha1.ClusterType - Protocol v1alpha1.ClusterProtocol + Type stnrv1.ClusterType + Protocol stnrv1.ClusterProtocol Endpoints []net.IPNet Domains []string Resolver resolver.DnsResolver // for strict DNS @@ -26,10 +26,10 @@ type Cluster struct { } // NewCluster creates a new cluster. -func NewCluster(conf v1alpha1.Config, resolver resolver.DnsResolver, logger logging.LoggerFactory) (Object, error) { - req, ok := conf.(*v1alpha1.ClusterConfig) +func NewCluster(conf stnrv1.Config, resolver resolver.DnsResolver, logger logging.LoggerFactory) (Object, error) { + req, ok := conf.(*stnrv1.ClusterConfig) if !ok { - return nil, v1alpha1.ErrInvalidConf + return nil, stnrv1.ErrInvalidConf } // make sure req.Name is correct @@ -46,7 +46,7 @@ func NewCluster(conf v1alpha1.Config, resolver resolver.DnsResolver, logger logg log: logger.NewLogger(fmt.Sprintf("stunner-cluster-%s", req.Name)), } - c.log.Tracef("NewCluster: %sv", req.String()) + c.log.Tracef("NewCluster: %s", req.String()) if err := c.Reconcile(req); err != nil && err != ErrRestartRequired { return nil, err @@ -57,15 +57,15 @@ func NewCluster(conf v1alpha1.Config, resolver resolver.DnsResolver, logger logg // Inspect examines whether a configuration change requires a reconciliation (returns true if it // does) or restart (returns ErrRestartRequired). -func (c *Cluster) Inspect(old, new, full v1alpha1.Config) (bool, error) { +func (c *Cluster) Inspect(old, new, full stnrv1.Config) (bool, error) { return !old.DeepEqual(new), nil } // Reconcile updates the authenticator for a new configuration. -func (c *Cluster) Reconcile(conf v1alpha1.Config) error { - req, ok := conf.(*v1alpha1.ClusterConfig) +func (c *Cluster) Reconcile(conf stnrv1.Config) error { + req, ok := conf.(*stnrv1.ClusterConfig) if !ok { - return v1alpha1.ErrInvalidConf + return stnrv1.ErrInvalidConf } if err := req.Validate(); err != nil { @@ -73,11 +73,11 @@ func (c *Cluster) Reconcile(conf v1alpha1.Config) error { } c.log.Tracef("Reconcile: %s", req.String()) - c.Type, _ = v1alpha1.NewClusterType(req.Type) - c.Protocol, _ = v1alpha1.NewClusterProtocol(req.Protocol) + c.Type, _ = stnrv1.NewClusterType(req.Type) + c.Protocol, _ = stnrv1.NewClusterProtocol(req.Protocol) switch c.Type { - case v1alpha1.ClusterTypeStatic: + case stnrv1.ClusterTypeStatic: // remove existing endpoints and start anew c.Endpoints = c.Endpoints[:0] for _, e := range req.Endpoints { @@ -111,7 +111,7 @@ func (c *Cluster) Reconcile(conf v1alpha1.Config) error { c.Endpoints = append(c.Endpoints, *n2) } - case v1alpha1.ClusterTypeStrictDNS: + case stnrv1.ClusterTypeStrictDNS: if c.Resolver == nil { return fmt.Errorf("STRICT_DNS cluster %q initialized with no DNS resolver", c.Name) } @@ -145,21 +145,21 @@ func (c *Cluster) ObjectType() string { } // GetConfig returns the configuration of the running cluster. -func (c *Cluster) GetConfig() v1alpha1.Config { - conf := v1alpha1.ClusterConfig{ +func (c *Cluster) GetConfig() stnrv1.Config { + conf := stnrv1.ClusterConfig{ Name: c.Name, Protocol: c.Protocol.String(), Type: c.Type.String(), } switch c.Type { - case v1alpha1.ClusterTypeStatic: + case stnrv1.ClusterTypeStatic: conf.Endpoints = make([]string, len(c.Endpoints)) for i, e := range c.Endpoints { // e.String() adds a /32 at the end of IPs, remove conf.Endpoints[i] = strings.TrimRight(e.String(), "/32") } - case v1alpha1.ClusterTypeStrictDNS: + case stnrv1.ClusterTypeStrictDNS: conf.Endpoints = make([]string, len(c.Domains)) copy(conf.Endpoints, c.Domains) conf.Endpoints = sort.StringSlice(conf.Endpoints) @@ -173,9 +173,9 @@ func (c *Cluster) Close() error { c.log.Trace("closing cluster") switch c.Type { - case v1alpha1.ClusterTypeStatic: + case stnrv1.ClusterTypeStatic: // do nothing - case v1alpha1.ClusterTypeStrictDNS: + case stnrv1.ClusterTypeStrictDNS: for _, d := range c.Domains { c.Resolver.Unregister(d) } @@ -190,7 +190,7 @@ func (c *Cluster) Route(peer net.IP) bool { peer.String()) switch c.Type { - case v1alpha1.ClusterTypeStatic: + case stnrv1.ClusterTypeStatic: // endpoints are IPNets for _, e := range c.Endpoints { c.log.Tracef("considering endpoint %q", e) @@ -199,7 +199,7 @@ func (c *Cluster) Route(peer net.IP) bool { } } - case v1alpha1.ClusterTypeStrictDNS: + case stnrv1.ClusterTypeStrictDNS: // endpoints are obtained from the DNS c.log.Tracef("running STRICT_DNS cluster with domains: [%s]", strings.Join(c.Domains, ", ")) @@ -237,7 +237,7 @@ func NewClusterFactory(resolver resolver.DnsResolver, logger logging.LoggerFacto // New can produce a new Cluster object from the given configuration. A nil config will create an // empty cluster object (useful for creating throwaway objects for, e.g., calling Inpect) -func (f *ClusterFactory) New(conf v1alpha1.Config) (Object, error) { +func (f *ClusterFactory) New(conf stnrv1.Config) (Object, error) { if conf == nil { return &Cluster{}, nil } diff --git a/internal/object/listener.go b/internal/object/listener.go index f46183ee..9a226566 100644 --- a/internal/object/listener.go +++ b/internal/object/listener.go @@ -13,13 +13,13 @@ import ( "github.com/pion/turn/v3" "github.com/l7mp/stunner/internal/util" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // Listener implements a STUNner listener. type Listener struct { Name, Realm string - Proto v1alpha1.ListenerProtocol + Proto stnrv1.ListenerProtocol Addr net.IP Port, MinPort, MaxPort int rawAddr string // net.IP.String() may rewrite the string representation @@ -34,10 +34,10 @@ type Listener struct { } // NewListener creates a new listener. Requires a server restart (returns ErrRestartRequired) -func NewListener(conf v1alpha1.Config, net transport.Net, realmHandler RealmHandler, logger logging.LoggerFactory) (Object, error) { - req, ok := conf.(*v1alpha1.ListenerConfig) +func NewListener(conf stnrv1.Config, net transport.Net, realmHandler RealmHandler, logger logging.LoggerFactory) (Object, error) { + req, ok := conf.(*stnrv1.ListenerConfig) if !ok { - return nil, v1alpha1.ErrInvalidConf + return nil, stnrv1.ErrInvalidConf } // make sure req.Name is correct @@ -65,20 +65,20 @@ func NewListener(conf v1alpha1.Config, net transport.Net, realmHandler RealmHand // Inspect examines whether a configuration change requires a reconciliation (returns true if it // does) or restart (returns ErrRestartRequired). -func (l *Listener) Inspect(old, new, full v1alpha1.Config) (bool, error) { - req, ok := new.(*v1alpha1.ListenerConfig) +func (l *Listener) Inspect(old, new, full stnrv1.Config) (bool, error) { + req, ok := new.(*stnrv1.ListenerConfig) if !ok { - return false, v1alpha1.ErrInvalidConf + return false, stnrv1.ErrInvalidConf } - stunnerConf, ok := full.(*v1alpha1.StunnerConfig) + stunnerConf, ok := full.(*stnrv1.StunnerConfig) if !ok { - return false, v1alpha1.ErrInvalidConf + return false, stnrv1.ErrInvalidConf } changed := !old.DeepEqual(req) - proto, _ := v1alpha1.NewListenerProtocol(req.Protocol) + proto, _ := stnrv1.NewListenerProtocol(req.Protocol) cert, err := base64.StdEncoding.DecodeString(req.Cert) if err != nil { return false, fmt.Errorf("invalid TLS certificate: base64-decode error: %w", err) @@ -112,10 +112,10 @@ func (l *Listener) Inspect(old, new, full v1alpha1.Config) (bool, error) { } // Reconcile updates a listener. -func (l *Listener) Reconcile(conf v1alpha1.Config) error { - req, ok := conf.(*v1alpha1.ListenerConfig) +func (l *Listener) Reconcile(conf stnrv1.Config) error { + req, ok := conf.(*stnrv1.ListenerConfig) if !ok { - return v1alpha1.ErrInvalidConf + return stnrv1.ErrInvalidConf } l.log.Tracef("Reconcile: %s", req.String()) @@ -124,7 +124,7 @@ func (l *Listener) Reconcile(conf v1alpha1.Config) error { return err } - proto, _ := v1alpha1.NewListenerProtocol(req.Protocol) + proto, _ := stnrv1.NewListenerProtocol(req.Protocol) ipAddr := net.ParseIP(req.Addr) // special-case "localhost" if ipAddr == nil && req.Addr == "localhost" { @@ -138,7 +138,7 @@ func (l *Listener) Reconcile(conf v1alpha1.Config) error { l.Addr = ipAddr l.rawAddr = req.Addr l.Port, l.MinPort, l.MaxPort = req.Port, req.MinRelayPort, req.MaxRelayPort - if proto == v1alpha1.ListenerProtocolTURNTLS || proto == v1alpha1.ListenerProtocolTURNDTLS { + if proto == stnrv1.ListenerProtocolTURNTLS || proto == stnrv1.ListenerProtocolTURNDTLS { cert, err := base64.StdEncoding.DecodeString(req.Cert) if err != nil { return fmt.Errorf("invalid TLS certificate: base64-decode error: %w", err) @@ -176,11 +176,11 @@ func (l *Listener) ObjectType() string { } // GetConfig returns the configuration of the running listener. -func (l *Listener) GetConfig() v1alpha1.Config { +func (l *Listener) GetConfig() stnrv1.Config { // must be sorted! sort.Strings(l.Routes) - c := &v1alpha1.ListenerConfig{ + c := &stnrv1.ListenerConfig{ Name: l.Name, Protocol: l.Proto.String(), Addr: l.rawAddr, @@ -204,7 +204,7 @@ func (l *Listener) Close() error { for _, c := range l.Conns { switch l.Proto { - case v1alpha1.ListenerProtocolTURNUDP: + case stnrv1.ListenerProtocolTURNUDP: l.log.Tracef("closing %s packet socket at %s", l.Proto.String(), l.Addr) conn, ok := c.(turn.PacketConnConfig) @@ -216,7 +216,7 @@ func (l *Listener) Close() error { if err := conn.PacketConn.Close(); err != nil && !util.IsClosedErr(err) { return err } - case v1alpha1.ListenerProtocolTURNTCP, v1alpha1.ListenerProtocolTURNTLS, v1alpha1.ListenerProtocolTURNDTLS: + case stnrv1.ListenerProtocolTURNTCP, stnrv1.ListenerProtocolTURNTLS, stnrv1.ListenerProtocolTURNDTLS: l.log.Tracef("closing %s listener socket at %s", l.Proto.String(), l.Addr) conn, ok := c.(turn.ListenerConfig) @@ -263,7 +263,7 @@ func NewListenerFactory(net transport.Net, realmHandler RealmHandler, logger log // New can produce a new Listener object from the given configuration. A nil config will create an // empty listener object (useful for creating throwaway objects for, e.g., calling Inpect) -func (f *ListenerFactory) New(conf v1alpha1.Config) (Object, error) { +func (f *ListenerFactory) New(conf stnrv1.Config) (Object, error) { if conf == nil { return &Listener{}, nil } diff --git a/internal/object/object.go b/internal/object/object.go index 5a2a1ec5..4a917bec 100644 --- a/internal/object/object.go +++ b/internal/object/object.go @@ -1,8 +1,6 @@ package object -import ( - "github.com/l7mp/stunner/pkg/apis/v1alpha1" -) +import stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" // Object is the high-level interface for all STUNner objects like listeners, clusters, etc. type Object interface { @@ -11,11 +9,11 @@ type Object interface { // ObjectType returns the type of the object. ObjectType() string // Inspect examines whether a configuration change requires a reconciliation or restart. - Inspect(old, new, full v1alpha1.Config) (bool, error) + Inspect(old, new, full stnrv1.Config) (bool, error) // Reconcile updates the object for a new configuration. - Reconcile(conf v1alpha1.Config) error + Reconcile(conf stnrv1.Config) error // GetConfig returns the configuration of the running authenticator. - GetConfig() v1alpha1.Config + GetConfig() stnrv1.Config // Close closes the object, may return ErrRestartRequired. Close() error } @@ -23,7 +21,7 @@ type Object interface { // Factory can create new objects type Factory interface { // New will spawn a new object from the factory - New(conf v1alpha1.Config) (Object, error) + New(conf stnrv1.Config) (Object, error) } // ReadinessHandler is a callback that allows an object to check the readiness of STUNner. diff --git a/pkg/apis/v1alpha1/admin.go b/pkg/apis/v1/admin.go similarity index 99% rename from pkg/apis/v1alpha1/admin.go rename to pkg/apis/v1/admin.go index 336ee463..0ee9d626 100644 --- a/pkg/apis/v1alpha1/admin.go +++ b/pkg/apis/v1/admin.go @@ -1,4 +1,4 @@ -package v1alpha1 +package v1 import ( "fmt" diff --git a/pkg/apis/v1/auth.go b/pkg/apis/v1/auth.go new file mode 100644 index 00000000..fcab3ee6 --- /dev/null +++ b/pkg/apis/v1/auth.go @@ -0,0 +1,130 @@ +package v1 + +import ( + "fmt" + "reflect" + "strings" +) + +// Auth defines the specification of the STUN/TURN authentication mechanism used by STUNner. +type AuthConfig struct { + // Type is the type of the STUN/TURN authentication mechanism ("plaintext" or "longterm"). + Type string `json:"type,omitempty"` + // Realm defines the STUN/TURN authentication realm. + Realm string `json:"realm,omitempty"` + // Credentials specifies the authententication credentials: for "plaintext" at least the + // keys "username" and "password" must be set, for "longterm" the key "secret" will hold + // the shared authentication secret. + Credentials map[string]string `json:"credentials"` +} + +// Validate checks a configuration and injects defaults. +func (req *AuthConfig) Validate() error { + if req.Type == "" { + req.Type = DefaultAuthType + } + + atype, err := NewAuthType(req.Type) + if err != nil { + return err + } + req.Type = atype.String() // normalize + + switch atype { + case AuthTypeStatic: + _, userFound := req.Credentials["username"] + _, passFound := req.Credentials["password"] + if !userFound || !passFound { + return fmt.Errorf("%s: empty username or password", atype.String()) + } + + case AuthTypeEphemeral: + _, secretFound := req.Credentials["secret"] + if !secretFound { + return fmt.Errorf("cannot handle auth config for type %s: invalid secret", + atype.String()) + } + default: + return fmt.Errorf("invalid authentication type %q", req.Type) + } + + if req.Realm == "" { + req.Realm = DefaultRealm + } + + return nil +} + +// Name returns the name of the object to be configured. +func (req *AuthConfig) ConfigName() string { + // singleton! + return DefaultAuthName +} + +// DeepEqual compares two configurations. +func (req *AuthConfig) DeepEqual(other Config) bool { + return reflect.DeepEqual(req, other) +} + +// DeepCopyInto copies a configuration. +func (req *AuthConfig) DeepCopyInto(dst Config) { + ret := dst.(*AuthConfig) + *ret = *req + ret.Credentials = make(map[string]string, len(req.Credentials)) + for k, v := range req.Credentials { + ret.Credentials[k] = v + } +} + +// String stringifies the configuration. +func (req *AuthConfig) String() string { + status := []string{} + if req.Realm != "" { + status = append(status, fmt.Sprintf("realm=%q", req.Realm)) + } + + if atype, err := NewAuthType(req.Type); err == nil { + switch atype { + case AuthTypeStatic: + u, userFound := req.Credentials["username"] + if userFound { + if u == "" { + u = "" + } else { + u = "" + } + } else { + u = "-" + } + p, passFound := req.Credentials["password"] + if passFound { + if p == "" { + p = "" + } else { + p = "" + } + } else { + p = "-" + } + status = append(status, fmt.Sprintf("type=%q,username=%q,password=%q", + atype.String(), u, p)) + + case AuthTypeEphemeral: + s, secretFound := req.Credentials["secret"] + if secretFound { + if s == "" { + s = "" + } else { + s = "" + } + } else { + s = "-" + } + + status = append(status, fmt.Sprintf("type=%q,shared-secret=%q", + atype.String(), s)) + } + } + + return fmt.Sprintf("auth:{%s}", strings.Join(status, ",")) +} diff --git a/pkg/apis/v1alpha1/cluster.go b/pkg/apis/v1/cluster.go similarity index 99% rename from pkg/apis/v1alpha1/cluster.go rename to pkg/apis/v1/cluster.go index 315f76b3..3c4ff9d5 100644 --- a/pkg/apis/v1alpha1/cluster.go +++ b/pkg/apis/v1/cluster.go @@ -1,4 +1,4 @@ -package v1alpha1 +package v1 import ( "fmt" diff --git a/pkg/apis/v1alpha1/config.go b/pkg/apis/v1/config.go similarity index 84% rename from pkg/apis/v1alpha1/config.go rename to pkg/apis/v1/config.go index a8f5a649..3c335c45 100644 --- a/pkg/apis/v1alpha1/config.go +++ b/pkg/apis/v1/config.go @@ -1,5 +1,5 @@ -// Package v1alpha1 is the v1alpha1 version of the STUNner API. -package v1alpha1 +// Package v1 is the v1 version of the STUNner API. +package v1 // Config is the main interface for STUNner configuration objects type Config interface { diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go new file mode 100644 index 00000000..7ecefcea --- /dev/null +++ b/pkg/apis/v1/default.go @@ -0,0 +1,19 @@ +package v1 + +const ApiVersion string = "v1" +const DefaultStunnerName = "default-stunnerd" +const DefaultProtocol = "turn-udp" +const DefaultClusterProtocol = "udp" +const DefaultPort int = 3478 +const DefaultLogLevel = "all:INFO" +const DefaultRealm = "stunner.l7mp.io" +const DefaultAuthType = "static" +const DefaultMinRelayPort int = 1 +const DefaultMaxRelayPort int = 1<<16 - 1 +const DefaultClusterType = "STATIC" + +const DefaultAdminName = "default-admin-config" +const DefaultAuthName = "default-auth-config" + +const DefaultMetricsPort int = 8080 +const DefaultHealthCheckPort int = 8086 diff --git a/pkg/apis/v1/errors.go b/pkg/apis/v1/errors.go new file mode 100644 index 00000000..2b5862d3 --- /dev/null +++ b/pkg/apis/v1/errors.go @@ -0,0 +1,26 @@ +package v1 + +import ( + "errors" + "fmt" + "strings" +) + +var ( + ErrInvalidConf = errors.New("invalid configuration") + ErrNoSuchListener = errors.New("no such listener") + ErrNoSuchCluster = errors.New("no such cluster") + // ErrInvalidRoute = errors.New("invalid route") +) + +type ErrRestarted struct { + Objects []string +} + +func (e ErrRestarted) Error() string { + s := []string{} + for _, o := range e.Objects { + s = append(s, fmt.Sprintf("[%s]", o)) + } + return fmt.Sprintf("restarted: %s", strings.Join(s, ", ")) +} diff --git a/pkg/apis/v1alpha1/listener.go b/pkg/apis/v1/listener.go similarity index 99% rename from pkg/apis/v1alpha1/listener.go rename to pkg/apis/v1/listener.go index e4c74ef2..c4d19a4f 100644 --- a/pkg/apis/v1alpha1/listener.go +++ b/pkg/apis/v1/listener.go @@ -1,4 +1,4 @@ -package v1alpha1 +package v1 import ( "fmt" diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go new file mode 100644 index 00000000..31959b81 --- /dev/null +++ b/pkg/apis/v1/stunner.go @@ -0,0 +1,172 @@ +package v1 + +import ( + "fmt" + // "sort" + "strings" +) + +// StunnerConfig specifies the configuration of the the STUnner daemon. +type StunnerConfig struct { + // ApiVersion is the version of the STUNner API implemented. + ApiVersion string `json:"version"` + // AdminConfig holds administrative configuration. + Admin AdminConfig `json:"admin,omitempty"` + // Auth defines the STUN/TURN authentication mechanism. + Auth AuthConfig `json:"auth"` + // Listeners defines the server sockets exposed to clients. + Listeners []ListenerConfig `json:"listeners,omitempty"` + // Clusters defines the upstream endpoints to which relay transport connections can be made + // by clients. + Clusters []ClusterConfig `json:"clusters,omitempty"` +} + +// Validate checks if a listener configuration is correct. +func (req *StunnerConfig) Validate() error { + // ApiVersion + if req.ApiVersion != ApiVersion { + return fmt.Errorf("unsupported API version: %q", req.ApiVersion) + } + + // validate admin + if err := req.Admin.Validate(); err != nil { + return err + } + + // validate auth + if err := req.Auth.Validate(); err != nil { + return err + } + + // validate listeners + for i, l := range req.Listeners { + if err := l.Validate(); err != nil { + return err + } + req.Listeners[i] = l + } + // // listeners are sorted by name + // sort.Slice(req.Listeners, func(i, j int) bool { + // return req.Listeners[i].Name < req.Listeners[j].Name + // }) + + // validate clusters + for i, c := range req.Clusters { + if err := c.Validate(); err != nil { + return err + } + req.Clusters[i] = c + } + + // // clusters are sorted by name + // sort.Slice(req.Clusters, func(i, j int) bool { + // return req.Clusters[i].Name < req.Clusters[j].Name + // }) + + return nil +} + +// Name returns the name of the object to be configured. +func (req *StunnerConfig) ConfigName() string { + return req.Admin.Name +} + +// DeepEqual compares two configurations. +func (req *StunnerConfig) DeepEqual(conf Config) bool { + other, ok := conf.(*StunnerConfig) + if !ok { + return false + } + + if req.ApiVersion != other.ApiVersion { + return false + } + if !req.Admin.DeepEqual(&other.Admin) { + return false + } + if !req.Auth.DeepEqual(&other.Auth) { + return false + } + + for i := range req.Listeners { + if i >= len(other.Listeners) { + return false + } + if !req.Listeners[i].DeepEqual(&other.Listeners[i]) { + return false + } + } + + for i := range req.Clusters { + if i >= len(other.Clusters) { + return false + } + if !req.Clusters[i].DeepEqual(&other.Clusters[i]) { + return false + } + } + + return true +} + +// DeepCopyInto copies a configuration. +func (req *StunnerConfig) DeepCopyInto(dst Config) { + ret := dst.(*StunnerConfig) + ret.ApiVersion = req.ApiVersion + req.Admin.DeepCopyInto(&ret.Admin) + req.Auth.DeepCopyInto(&ret.Auth) + + ret.Listeners = make([]ListenerConfig, len(req.Listeners)) + for i := range req.Listeners { + req.Listeners[i].DeepCopyInto(&ret.Listeners[i]) + } + + ret.Clusters = make([]ClusterConfig, len(req.Clusters)) + for i := range req.Clusters { + req.Clusters[i].DeepCopyInto(&ret.Clusters[i]) + } +} + +// String stringifies the configuration. +func (req *StunnerConfig) String() string { + status := []string{} + status = append(status, fmt.Sprintf("version=%q", req.ApiVersion)) + status = append(status, req.Admin.String()) + status = append(status, req.Auth.String()) + + ls := []string{} + for _, l := range req.Listeners { + ls = append(ls, l.String()) + } + status = append(status, fmt.Sprintf("listeners=[%s]", strings.Join(ls, ","))) + + cs := []string{} + for _, c := range req.Clusters { + cs = append(cs, c.String()) + } + status = append(status, fmt.Sprintf("clusters=[%s]", strings.Join(cs, ","))) + + return fmt.Sprintf("{%s}", strings.Join(status, ",")) +} + +// GetListenerConfig finds a Listener by name in a StunnerConfig or returns an error. +func (req *StunnerConfig) GetListenerConfig(name string) (ListenerConfig, error) { + for _, l := range req.Listeners { + if l.Name == name { + return l, nil + } + } + + return ListenerConfig{}, ErrNoSuchListener +} + +// GetClusterConfig finds a Cluster by name in a StunnerConfig or returns an error. +func (req *StunnerConfig) GetClusterConfig(name string) (ClusterConfig, error) { + for _, c := range req.Clusters { + if c.Name == name { + return c, nil + } + } + + return ClusterConfig{}, ErrNoSuchCluster +} diff --git a/pkg/apis/v1/util.go b/pkg/apis/v1/util.go new file mode 100644 index 00000000..cd10cae1 --- /dev/null +++ b/pkg/apis/v1/util.go @@ -0,0 +1,198 @@ +package v1 + +import ( + "fmt" + "strings" +) + +// AuthType species the type of the STUN/TURN authentication mechanism used by STUNner +type AuthType int + +const ( + AuthTypeStatic AuthType = iota + 1 + AuthTypeEphemeral + AuthTypeUnknown +) + +const ( + authTypeStaticStr = "static" + authTypeEphemeralStr = "ephemeral" + AuthTypePlainText = AuthTypeStatic + AuthTypeLongTerm = AuthTypeEphemeral + authTypePlainTextStr = "plaintext" + authTypeLongTermStr = "longterm" +) + +// NewAuthType parses the authentication mechanism specification +func NewAuthType(raw string) (AuthType, error) { + switch raw { + case authTypeStaticStr, authTypePlainTextStr: + return AuthTypeStatic, nil + case authTypeEphemeralStr, authTypeLongTermStr: + return AuthTypeEphemeral, nil + default: + return AuthTypeUnknown, fmt.Errorf("unknown authentication type: \"%s\"", raw) + } +} + +// String returns a string representation for the authentication mechanism +func (a AuthType) String() string { + switch a { + case AuthTypeStatic: + return authTypeStaticStr + case AuthTypeEphemeral: + return authTypeEphemeralStr + default: + return "" + } +} + +// ListenerProtocol specifies the network protocol for a listener +type ListenerProtocol int + +const ( + ListenerProtocolUnknown ListenerProtocol = iota + ListenerProtocolUDP + ListenerProtocolTCP + ListenerProtocolTLS + ListenerProtocolDTLS + ListenerProtocolTURNUDP + ListenerProtocolTURNTCP + ListenerProtocolTURNTLS + ListenerProtocolTURNDTLS +) + +const ( + listenerProtocolUDPStr = "UDP" + listenerProtocolTCPStr = "TCP" + listenerProtocolTLSStr = "TLS" + listenerProtocolDTLSStr = "DTLS" + listenerProtocolTURNUDPStr = "TURN-UDP" + listenerProtocolTURNTCPStr = "TURN-TCP" + listenerProtocolTURNTLSStr = "TURN-TLS" + listenerProtocolTURNDTLSStr = "TURN-DTLS" +) + +// NewListenerProtocol parses the protocol specification +func NewListenerProtocol(raw string) (ListenerProtocol, error) { + switch strings.ToUpper(raw) { + case listenerProtocolUDPStr: + return ListenerProtocolUDP, nil + case listenerProtocolTCPStr: + return ListenerProtocolTCP, nil + case listenerProtocolTLSStr: + return ListenerProtocolTLS, nil + case listenerProtocolDTLSStr: + return ListenerProtocolDTLS, nil + case listenerProtocolTURNUDPStr: + return ListenerProtocolTURNUDP, nil + case listenerProtocolTURNTCPStr: + return ListenerProtocolTURNTCP, nil + case listenerProtocolTURNTLSStr: + return ListenerProtocolTURNTLS, nil + case listenerProtocolTURNDTLSStr: + return ListenerProtocolTURNDTLS, nil + default: + return ListenerProtocol(ListenerProtocolUnknown), + fmt.Errorf("unknown listener protocol: \"%s\"", raw) + } +} + +// String returns a string representation of a listener protocol +func (l ListenerProtocol) String() string { + switch l { + case ListenerProtocolUDP: + return listenerProtocolUDPStr + case ListenerProtocolTCP: + return listenerProtocolTCPStr + case ListenerProtocolTLS: + return listenerProtocolTLSStr + case ListenerProtocolDTLS: + return listenerProtocolDTLSStr + case ListenerProtocolTURNUDP: + return listenerProtocolTURNUDPStr + case ListenerProtocolTURNTCP: + return listenerProtocolTURNTCPStr + case ListenerProtocolTURNTLS: + return listenerProtocolTURNTLSStr + case ListenerProtocolTURNDTLS: + return listenerProtocolTURNDTLSStr + default: + return "" + } +} + +// ClusterType specifies the cluster address resolution policy +type ClusterType int + +const ( + ClusterTypeStatic ClusterType = iota + 1 + ClusterTypeStrictDNS + ClusterTypeUnknown +) + +const ( + clusterTypeStaticStr = "STATIC" + clusterTypeStrictDNSStr = "STRICT_DNS" +) + +func NewClusterType(raw string) (ClusterType, error) { + switch strings.ToUpper(raw) { + case clusterTypeStaticStr: + return ClusterTypeStatic, nil + case clusterTypeStrictDNSStr: + return ClusterTypeStrictDNS, nil + default: + return ClusterType(ClusterTypeUnknown), fmt.Errorf("unknown cluster type: \"%s\"", raw) + } +} + +func (l ClusterType) String() string { + switch l { + case ClusterTypeStatic: + return clusterTypeStaticStr + case ClusterTypeStrictDNS: + return clusterTypeStrictDNSStr + default: + return "" + } +} + +// ClusterProtocol specifies the network protocol for a cluster +type ClusterProtocol int + +const ( + ClusterProtocolUDP ClusterProtocol = iota + 1 + ClusterProtocolTCP + ClusterProtocolUnknown +) + +const ( + clusterProtocolUDPStr = "UDP" + clusterProtocolTCPStr = "TCP" +) + +// NewClusterProtocol parses the protocol specification +func NewClusterProtocol(raw string) (ClusterProtocol, error) { + switch strings.ToUpper(raw) { + case clusterProtocolUDPStr: + return ClusterProtocolUDP, nil + case clusterProtocolTCPStr: + return ClusterProtocolTCP, nil + default: + return ClusterProtocol(ClusterProtocolUnknown), + fmt.Errorf("unknown cluster protocol: \"%s\"", raw) + } +} + +// String returns a string representation of a cluster protocol +func (p ClusterProtocol) String() string { + switch p { + case ClusterProtocolUDP: + return clusterProtocolUDPStr + case ClusterProtocolTCP: + return clusterProtocolTCPStr + default: + return "" + } +} diff --git a/pkg/apis/v1alpha1/default.go b/pkg/apis/v1alpha1/default.go index 49ee6927..17106798 100644 --- a/pkg/apis/v1alpha1/default.go +++ b/pkg/apis/v1alpha1/default.go @@ -8,11 +8,6 @@ const DefaultPort int = 3478 const DefaultLogLevel = "all:INFO" const DefaultRealm = "stunner.l7mp.io" const DefaultAuthType = "plaintext" - -// no more default user/pass pairs -// const DefaultUsername = "user1" -// const DefaultPassword = "passwd1" - const DefaultMinRelayPort int = 1 << 15 const DefaultMaxRelayPort int = 1<<16 - 1 const DefaultClusterType = "STATIC" diff --git a/pkg/apis/v1alpha1/errors.go b/pkg/apis/v1alpha1/errors.go index 786651df..5adba2e1 100644 --- a/pkg/apis/v1alpha1/errors.go +++ b/pkg/apis/v1alpha1/errors.go @@ -1,26 +1,11 @@ package v1alpha1 -import ( - "errors" - "fmt" - "strings" -) +import stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" var ( - ErrInvalidConf = errors.New("invalid configuration") - ErrNoSuchListener = errors.New("no such listener") - ErrNoSuchCluster = errors.New("no such cluster") - // ErrInvalidRoute = errors.New("invalid route") + ErrInvalidConf = stnrv1.ErrInvalidConf + ErrNoSuchListener = stnrv1.ErrNoSuchListener + ErrNoSuchCluster = stnrv1.ErrNoSuchCluster ) -type ErrRestarted struct { - Objects []string -} - -func (e ErrRestarted) Error() string { - s := []string{} - for _, o := range e.Objects { - s = append(s, fmt.Sprintf("[%s]", o)) - } - return fmt.Sprintf("restarted: %s", strings.Join(s, ", ")) -} +type ErrRestarted = stnrv1.ErrRestarted diff --git a/pkg/apis/v1alpha1/stunner.go b/pkg/apis/v1alpha1/stunner.go index 0ae9b8f1..a0e1df31 100644 --- a/pkg/apis/v1alpha1/stunner.go +++ b/pkg/apis/v1alpha1/stunner.go @@ -2,10 +2,30 @@ package v1alpha1 import ( "fmt" - // "sort" + "maps" "strings" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) +// Config is the main interface for STUNner configuration objects +type Config = stnrv1.Config + +// AdminConfig holds the administrative configuration. +type AdminConfig = stnrv1.AdminConfig + +// ClusterConfig specifies a set of upstream peers STUNner can open transport relay connections +// to. There are two address resolution policies. In STATIC clusters the allowed peer IP addresses +// are explicitly listed in the endpoint list. In STRICT_DNS clusters the endpoints are assumed to +// be proper DNS domain names. STUNner will resolve each domain name in the background and admits a +// new connection only if the peer address matches one of the IP addresses returned by the DNS +// resolver for one of the endpoints. STRICT_DNS clusters are best used with headless Kubernetes +// services. +type ClusterConfig = stnrv1.ClusterConfig + +// ListenerConfig specifies a server socket on which STUN/TURN connections will be served. +type ListenerConfig = stnrv1.ListenerConfig + // StunnerConfig specifies the configuration of the the STUnner daemon. type StunnerConfig struct { // ApiVersion is the version of the STUNner API implemented. @@ -170,3 +190,37 @@ func (req *StunnerConfig) GetClusterConfig(name string) (ClusterConfig, error) { return ClusterConfig{}, ErrNoSuchCluster } + +// GetClusterConfig finds a Cluster by name in a StunnerConfig or returns an error. +func ConvertToV1(sv1a1 *StunnerConfig) (*stnrv1.StunnerConfig, error) { + sv1 := stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + } + + (*stnrv1.AdminConfig)(&sv1a1.Admin).DeepCopyInto(&sv1.Admin) + + // auth needs to be converted + at, err := stnrv1.NewAuthType(sv1a1.Auth.Type) + if err != nil { + return nil, err + } + + sv1.Auth = stnrv1.AuthConfig{ + Type: at.String(), + Realm: sv1a1.Auth.Realm, + Credentials: make(map[string]string), + } + maps.Copy(sv1.Auth.Credentials, sv1a1.Auth.Credentials) + + sv1.Listeners = make([]stnrv1.ListenerConfig, len(sv1a1.Listeners)) + for i := range sv1a1.Listeners { + (*stnrv1.ListenerConfig)(&sv1a1.Listeners[i]).DeepCopyInto(&sv1.Listeners[i]) + } + + sv1.Clusters = make([]stnrv1.ClusterConfig, len(sv1a1.Clusters)) + for i := range sv1a1.Clusters { + (*stnrv1.ClusterConfig)(&sv1a1.Clusters[i]).DeepCopyInto(&sv1.Clusters[i]) + } + + return &sv1, nil +} diff --git a/pkg/apis/v1alpha1/util.go b/pkg/apis/v1alpha1/util.go index 95865b6d..db3be683 100644 --- a/pkg/apis/v1alpha1/util.go +++ b/pkg/apis/v1alpha1/util.go @@ -2,16 +2,17 @@ package v1alpha1 import ( "fmt" - "strings" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // AuthType species the type of the STUN/TURN authentication mechanism used by STUNner -type AuthType int +type AuthType stnrv1.AuthType const ( - AuthTypePlainText AuthType = iota + 1 - AuthTypeLongTerm - AuthTypeUnknown + AuthTypePlainText AuthType = AuthType(stnrv1.AuthTypeStatic) + AuthTypeLongTerm AuthType = AuthType(stnrv1.AuthTypeLongTerm) + AuthTypeUnknown AuthType = AuthType(stnrv1.AuthTypeUnknown) ) const ( @@ -44,151 +45,10 @@ func (a AuthType) String() string { } // ListenerProtocol specifies the network protocol for a listener -type ListenerProtocol int - -const ( - ListenerProtocolUnknown ListenerProtocol = iota - ListenerProtocolUDP - ListenerProtocolTCP - ListenerProtocolTLS - ListenerProtocolDTLS - ListenerProtocolTURNUDP - ListenerProtocolTURNTCP - ListenerProtocolTURNTLS - ListenerProtocolTURNDTLS -) - -const ( - listenerProtocolUDPStr = "UDP" - listenerProtocolTCPStr = "TCP" - listenerProtocolTLSStr = "TLS" - listenerProtocolDTLSStr = "DTLS" - listenerProtocolTURNUDPStr = "TURN-UDP" - listenerProtocolTURNTCPStr = "TURN-TCP" - listenerProtocolTURNTLSStr = "TURN-TLS" - listenerProtocolTURNDTLSStr = "TURN-DTLS" -) - -// NewListenerProtocol parses the protocol specification -func NewListenerProtocol(raw string) (ListenerProtocol, error) { - switch strings.ToUpper(raw) { - case listenerProtocolUDPStr: - return ListenerProtocolUDP, nil - case listenerProtocolTCPStr: - return ListenerProtocolTCP, nil - case listenerProtocolTLSStr: - return ListenerProtocolTLS, nil - case listenerProtocolDTLSStr: - return ListenerProtocolDTLS, nil - case listenerProtocolTURNUDPStr: - return ListenerProtocolTURNUDP, nil - case listenerProtocolTURNTCPStr: - return ListenerProtocolTURNTCP, nil - case listenerProtocolTURNTLSStr: - return ListenerProtocolTURNTLS, nil - case listenerProtocolTURNDTLSStr: - return ListenerProtocolTURNDTLS, nil - default: - return ListenerProtocol(ListenerProtocolUnknown), - fmt.Errorf("unknown listener protocol: \"%s\"", raw) - } -} - -// String returns a string representation of a listener protocol -func (l ListenerProtocol) String() string { - switch l { - case ListenerProtocolUDP: - return listenerProtocolUDPStr - case ListenerProtocolTCP: - return listenerProtocolTCPStr - case ListenerProtocolTLS: - return listenerProtocolTLSStr - case ListenerProtocolDTLS: - return listenerProtocolDTLSStr - case ListenerProtocolTURNUDP: - return listenerProtocolTURNUDPStr - case ListenerProtocolTURNTCP: - return listenerProtocolTURNTCPStr - case ListenerProtocolTURNTLS: - return listenerProtocolTURNTLSStr - case ListenerProtocolTURNDTLS: - return listenerProtocolTURNDTLSStr - default: - return "" - } -} +type ListenerProtocol = stnrv1.ListenerProtocol // ClusterType specifies the cluster address resolution policy -type ClusterType int - -const ( - ClusterTypeStatic ClusterType = iota + 1 - ClusterTypeStrictDNS - ClusterTypeUnknown -) - -const ( - clusterTypeStaticStr = "STATIC" - clusterTypeStrictDNSStr = "STRICT_DNS" -) - -func NewClusterType(raw string) (ClusterType, error) { - switch strings.ToUpper(raw) { - case clusterTypeStaticStr: - return ClusterTypeStatic, nil - case clusterTypeStrictDNSStr: - return ClusterTypeStrictDNS, nil - default: - return ClusterType(ClusterTypeUnknown), fmt.Errorf("unknown cluster type: \"%s\"", raw) - } -} - -func (l ClusterType) String() string { - switch l { - case ClusterTypeStatic: - return clusterTypeStaticStr - case ClusterTypeStrictDNS: - return clusterTypeStrictDNSStr - default: - return "" - } -} +type ClusterType = stnrv1.ClusterType // ClusterProtocol specifies the network protocol for a cluster -type ClusterProtocol int - -const ( - ClusterProtocolUDP ClusterProtocol = iota + 1 - ClusterProtocolTCP - ClusterProtocolUnknown -) - -const ( - clusterProtocolUDPStr = "UDP" - clusterProtocolTCPStr = "TCP" -) - -// NewClusterProtocol parses the protocol specification -func NewClusterProtocol(raw string) (ClusterProtocol, error) { - switch strings.ToUpper(raw) { - case clusterProtocolUDPStr: - return ClusterProtocolUDP, nil - case clusterProtocolTCPStr: - return ClusterProtocolTCP, nil - default: - return ClusterProtocol(ClusterProtocolUnknown), - fmt.Errorf("unknown cluster protocol: \"%s\"", raw) - } -} - -// String returns a string representation of a cluster protocol -func (p ClusterProtocol) String() string { - switch p { - case ClusterProtocolUDP: - return clusterProtocolUDPStr - case ClusterProtocolTCP: - return clusterProtocolTCPStr - default: - return "" - } -} +type ClusterProtocol = stnrv1.ClusterProtocol diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index 45409e10..167767fd 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -1,4 +1,4 @@ -package configdiscoveryclient +package client import ( "context" @@ -13,11 +13,12 @@ import ( "github.com/fsnotify/fsnotify" "github.com/gorilla/websocket" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/pion/logging" - - "github.com/l7mp/stunner/pkg/apis/v1alpha1" ) +var errFileTruncated = errors.New("zero-length config file") + var ( // Time RetryPeriod = 1 * time.Second @@ -37,10 +38,10 @@ var ( // receiver side. type Client interface { // Load grabs a new configuration from the config client. - Load() (*v1alpha1.StunnerConfig, error) - // Watch listens to new configurations and returns them on the channel ch. ctx cancels the - // watcher. - Watch(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error + Load() (*stnrv1.StunnerConfig, error) + // Watch listens to new configurations and returns them on the channel ch. The context ctx + // cancels the watcher. + Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error fmt.Stringer } @@ -84,18 +85,22 @@ func (w *configFileClient) String() string { return fmt.Sprintf("config client using file %q", w.configFile) } -func (w *configFileClient) Load() (*v1alpha1.StunnerConfig, error) { +func (w *configFileClient) Load() (*stnrv1.StunnerConfig, error) { b, err := os.ReadFile(w.configFile) if err != nil { return nil, fmt.Errorf("could not read config file %q: %s", w.configFile, err.Error()) } + if len(b) == 0 { + return nil, errFileTruncated + } + return ParseConfig(b) } // WatchConfig watches a configuration file for changes. If no file exists at the given path, // WatchConfig will periodically retry until the file appears. -func (w *configFileClient) Watch(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error { +func (w *configFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { if w.configFile == "" { return errors.New("uninitialized config file path") } @@ -117,51 +122,14 @@ func (w *configFileClient) Watch(ctx context.Context, ch chan<- v1alpha1.Stunner return } } - }() return nil } -// tryWatchConfig runs a timer to look for the config file at the given path and returns it -// immediately once found. Returns true if further action is needed (configWatcher has to be -// started) or false on normal exit. -func (w *configFileClient) tryWatchConfig(ctx context.Context) bool { - w.log.Tracef("tryWatchConfig") - config := w.configFile - - ticker := time.NewTicker(RetryPeriod) - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return false - - case <-ticker.C: - w.log.Debugf("trying to read config file %q from periodic timer", - config) - - // check if config file exists and it is readable - if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { - w.log.Debugf("config file %q does not exist", config) - - // report status in every 10th second - if time.Now().Second()%10 == 0 { - w.log.Warnf("waiting for config file %q", config) - } - - continue - } - - return true - } - } -} - // configWatcher watches the config file and emits new configs on the specified channel. Returns // true if further action is needed (tryWatchConfig is to be started) or false on normal exit. -func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) bool { +func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.StunnerConfig) bool { w.log.Tracef("configWatcher") // create a new watcher @@ -184,15 +152,16 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- v1alpha1 return true } - w.log.Debugf("config file successfully loaded from %q", config) - // send a deepcopy over the channel - copy := v1alpha1.StunnerConfig{} - c.DeepCopyInto(©) - ch <- copy + confCopy := stnrv1.StunnerConfig{} + c.DeepCopyInto(&confCopy) + + w.log.Debugf("initial config file successfully loaded from %q: %s", config, confCopy.String()) + + ch <- confCopy // save deepcopy so that we can filter repeated events - prev := v1alpha1.StunnerConfig{} + prev := stnrv1.StunnerConfig{} c.DeepCopyInto(&prev) for { @@ -228,6 +197,10 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- v1alpha1 w.log.Debugf("loading configuration file: %s", config) c, err := w.Load() if err != nil { + if errors.Is(err, errFileTruncated) { + w.log.Debugf("ignoring: %s", err.Error()) + continue + } w.log.Warnf("error loading config file: %s", err.Error()) return true } @@ -238,11 +211,12 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- v1alpha1 continue } - w.log.Debugf("config file successfully loaded from %q", config) + confCopy := stnrv1.StunnerConfig{} + c.DeepCopyInto(&confCopy) - copy := v1alpha1.StunnerConfig{} - c.DeepCopyInto(©) - ch <- copy + w.log.Debugf("config file successfully loaded from %q: %s", config, confCopy.String()) + + ch <- confCopy // save deepcopy so that we can filter repeated events c.DeepCopyInto(&prev) @@ -265,6 +239,42 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- v1alpha1 } } +// tryWatchConfig runs a timer to look for the config file at the given path and returns it +// immediately once found. Returns true if further action is needed (configWatcher has to be +// started) or false on normal exit. +func (w *configFileClient) tryWatchConfig(ctx context.Context) bool { + w.log.Tracef("tryWatchConfig") + config := w.configFile + + ticker := time.NewTicker(RetryPeriod) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return false + + case <-ticker.C: + w.log.Debugf("trying to read config file %q from periodic timer", + config) + + // check if config file exists and it is readable + if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { + w.log.Debugf("config file %q does not exist", config) + + // report status in every 10th second + if time.Now().Second()%10 == 0 { + w.log.Warnf("waiting for config file %q", config) + } + + continue + } + + return true + } + } +} + // configDiscoveryClient is the the implementation of the config discovery service. type configDiscoveryClient struct { // serverAddress is the URL of the config discovery server. @@ -281,7 +291,7 @@ func (p *configDiscoveryClient) String() string { return fmt.Sprintf("config discovery service using server %q", p.serverAddress) } -func (p *configDiscoveryClient) Load() (*v1alpha1.StunnerConfig, error) { +func (p *configDiscoveryClient) Load() (*stnrv1.StunnerConfig, error) { location, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, false) if err != nil { return nil, err @@ -304,8 +314,7 @@ func (p *configDiscoveryClient) Load() (*v1alpha1.StunnerConfig, error) { } if len(body) == 0 { - // empty config: this is an error for the config loader - return nil, errors.New("empty config received") + return nil, errFileTruncated } // fmt.Println("++++++++++++++++++++") @@ -314,10 +323,10 @@ func (p *configDiscoveryClient) Load() (*v1alpha1.StunnerConfig, error) { return ParseConfig(body) } -// Watch polls a config discovery server for a configuration file by sending a config request -// and then waits for the server to push a valid `StunnerConfig`. Use the `context` to cancel the +// Watch polls a config discovery server for a configuration file by sending a config request and +// then waits for the server to push a valid `StunnerConfig`. Use the `context` to cancel the // watcher. -func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error { +func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { _, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, true) if err != nil { return err @@ -345,7 +354,7 @@ func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- v1alpha1.St return nil } -func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- v1alpha1.StunnerConfig) error { +func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { p.log.Tracef("configPoller: trying to open connection to config discovery server at %q", p.serverAddress) location, origin, _ := getConfigDiscoveryLocation(p.serverAddress, p.id, true) @@ -362,8 +371,8 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- v1al defer conn.Close() // pinger - resCh := make(chan v1alpha1.StunnerConfig) - errCh := make(chan error) + resCh := make(chan stnrv1.StunnerConfig, 16) + errCh := make(chan error, 1) pingTicker := time.NewTicker(PingPeriod) closePinger := make(chan any) @@ -418,14 +427,15 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- v1al return } + if len(msg) == 0 { + p.log.Warn("ignoring zero-length config config fil") + continue + } + // fmt.Println("++++++++++++++++++++") - // fmt.Println(n) - // fmt.Println("++++++++++++++++++++") - // fmt.Println(msg) + // fmt.Println(string(msg)) // fmt.Println("++++++++++++++++++++") - // msg = bytes.TrimRight(msg, "\x00") - c, err := ParseConfig(msg) if err != nil { // assume it is a YAML/JSON syntax error: report and ignore @@ -433,12 +443,12 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- v1al continue } - p.log.Debugf("new config received: %s", c.String()) + confCopy := stnrv1.StunnerConfig{} + c.DeepCopyInto(&confCopy) - copy := v1alpha1.StunnerConfig{} - c.DeepCopyInto(©) + p.log.Debugf("new config received from %q: %s", p.serverAddress, confCopy.String()) - resCh <- copy + resCh <- confCopy } }() @@ -448,14 +458,18 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- v1al case <-ctx.Done(): // cancel: normal return closePinger <- struct{}{} + return nil case err := <-errCh: // error: return it closePinger <- struct{}{} + return err case conf := <-resCh: // new config: pass it along and move on + p.log.Debugf("new config available: %s", conf.String()) ch <- conf + continue } } diff --git a/pkg/config/client/config.go b/pkg/config/client/config.go index e3538f06..b8ec1d80 100644 --- a/pkg/config/client/config.go +++ b/pkg/config/client/config.go @@ -1,4 +1,4 @@ -package configdiscoveryclient +package client import ( "encoding/json" @@ -7,39 +7,43 @@ import ( "regexp" "strconv" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + stnrv1a1 "github.com/l7mp/stunner/pkg/apis/v1alpha1" "sigs.k8s.io/yaml" - - "github.com/l7mp/stunner/pkg/apis/v1alpha1" ) +type ConfigSkeleton struct { + ApiVersion string `json:"version"` +} + // ZeroConfig builds a zero configuration useful for bootstrapping STUNner. It starts with // plaintext authentication and opens no listeners and clusters. -func ZeroConfig(id string) *v1alpha1.StunnerConfig { - return &v1alpha1.StunnerConfig{ - ApiVersion: v1alpha1.ApiVersion, - Admin: v1alpha1.AdminConfig{Name: id}, - Auth: v1alpha1.AuthConfig{ +func ZeroConfig(id string) *stnrv1.StunnerConfig { + return &stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{Name: id}, + Auth: stnrv1.AuthConfig{ Type: "plaintext", - Realm: v1alpha1.DefaultRealm, + Realm: stnrv1.DefaultRealm, Credentials: map[string]string{ "username": "dummy-username", "password": "dummy-password", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{}, + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{}, } } // ParseConfig parses a raw buffer holding a configuration, substituting environment variables for // placeholders in the configuration. Returns the new configuration or error if parsing fails. -func ParseConfig(c []byte) (*v1alpha1.StunnerConfig, error) { +func ParseConfig(c []byte) (*stnrv1.StunnerConfig, error) { // substitute environtment variables // default port: STUNNER_PUBLIC_PORT -> STUNNER_PORT re := regexp.MustCompile(`^[0-9]+$`) port, ok := os.LookupEnv("STUNNER_PORT") if !ok || port == "" || !re.Match([]byte(port)) { - publicPort := v1alpha1.DefaultPort + publicPort := stnrv1.DefaultPort publicPortStr, ok := os.LookupEnv("STUNNER_PUBLIC_PORT") if ok { if p, err := strconv.Atoi(publicPortStr); err == nil { @@ -51,16 +55,44 @@ func ParseConfig(c []byte) (*v1alpha1.StunnerConfig, error) { e := os.ExpandEnv(string(c)) - s := v1alpha1.StunnerConfig{} - // try YAML first - if err := yaml.Unmarshal([]byte(e), &s); err != nil { - // if it fails, try to json - if errJ := json.Unmarshal([]byte(e), &s); err != nil { - return nil, fmt.Errorf("could not parse config file: "+ + // try to parse only the config version first + k := ConfigSkeleton{} + if err := yaml.Unmarshal([]byte(e), &k); err != nil { + if errJ := json.Unmarshal([]byte(e), &k); err != nil { + return nil, fmt.Errorf("could not parse config file API version: "+ "YAML parse error: %s, JSON parse error: %s\n", err.Error(), errJ.Error()) } } + s := stnrv1.StunnerConfig{} + + switch k.ApiVersion { + case stnrv1.ApiVersion: + if err := yaml.Unmarshal([]byte(e), &s); err != nil { + if errJ := json.Unmarshal([]byte(e), &s); err != nil { + return nil, fmt.Errorf("could not parse config file: "+ + "YAML parse error: %s, JSON parse error: %s\n", + err.Error(), errJ.Error()) + } + } + case stnrv1a1.ApiVersion: + a := stnrv1a1.StunnerConfig{} + if err := yaml.Unmarshal([]byte(e), &a); err != nil { + if errJ := json.Unmarshal([]byte(e), &a); err != nil { + return nil, fmt.Errorf("could not parse config file: "+ + "YAML parse error: %s, JSON parse error: %s\n", + err.Error(), errJ.Error()) + } + } + + sv1, err := stnrv1a1.ConvertToV1(&a) + if err != nil { + return nil, fmt.Errorf("could not convert config to API V1: %s", err) + } + + sv1.DeepCopyInto(&s) + } + return &s, nil } diff --git a/reconcile.go b/reconcile.go index 6a41143b..cce6d19c 100644 --- a/reconcile.go +++ b/reconcile.go @@ -4,23 +4,22 @@ import ( "fmt" "github.com/l7mp/stunner/internal/object" - - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // Reconcile handles updates to the STUNner configuration. Some updates are destructive: in this // case the returned error contains the names of the objects (usually, listeners) that were // restarted during reconciliation (see the documentation of the corresponding STUNner objects for // when STUNner may restart after a reconciliation). Reconcile returns nil no objects were -// restarted, v1alpha1.ErrRestarted to indicate that a shutdown-restart cycle was performed for at +// restarted, v1.ErrRestarted to indicate that a shutdown-restart cycle was performed for at // least one internal object (usually, a listener) for the new config (unless DryRun is enabled), // and an error if an error has occurred during reconciliation, in which case it will rollback the // last working configuration (unless SuppressRollback is on). -func (s *Stunner) Reconcile(req v1alpha1.StunnerConfig) error { +func (s *Stunner) Reconcile(req stnrv1.StunnerConfig) error { return s.reconcileWithRollback(req, false) } -func (s *Stunner) reconcileWithRollback(req v1alpha1.StunnerConfig, inRollback bool) error { +func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback bool) error { var errFinal error new, deleted, changed := 0, 0, 0 @@ -34,7 +33,7 @@ func (s *Stunner) reconcileWithRollback(req v1alpha1.StunnerConfig, inRollback b toBeStarted, toBeRestarted := []object.Object{}, []object.Object{} // admin - adminState, err := s.adminManager.PrepareReconciliation([]v1alpha1.Config{&req.Admin}, &req) + adminState, err := s.adminManager.PrepareReconciliation([]stnrv1.Config{&req.Admin}, &req) if err != nil { return fmt.Errorf("error preparing reconciliation for admin config: %s", err.Error()) @@ -45,7 +44,7 @@ func (s *Stunner) reconcileWithRollback(req v1alpha1.StunnerConfig, inRollback b deleted += len(adminState.DeletedJobQueue) // auth - authState, err := s.authManager.PrepareReconciliation([]v1alpha1.Config{&req.Auth}, &req) + authState, err := s.authManager.PrepareReconciliation([]stnrv1.Config{&req.Auth}, &req) if err != nil { return fmt.Errorf("error preparing reconciliation for auth config: %s", err.Error()) @@ -56,7 +55,7 @@ func (s *Stunner) reconcileWithRollback(req v1alpha1.StunnerConfig, inRollback b deleted += len(authState.DeletedJobQueue) // listener - lconf := make([]v1alpha1.Config, len(req.Listeners)) + lconf := make([]stnrv1.Config, len(req.Listeners)) for i := range req.Listeners { lconf[i] = &(req.Listeners[i]) } @@ -70,7 +69,7 @@ func (s *Stunner) reconcileWithRollback(req v1alpha1.StunnerConfig, inRollback b deleted += len(listenerState.DeletedJobQueue) // cluster - cconf := make([]v1alpha1.Config, len(req.Clusters)) + cconf := make([]stnrv1.Config, len(req.Clusters)) for i := range req.Clusters { cconf[i] = &(req.Clusters[i]) } @@ -186,7 +185,7 @@ func (s *Stunner) reconcileWithRollback(req v1alpha1.StunnerConfig, inRollback b names[i] = fmt.Sprintf("%s: %s", n.ObjectType(), n.ObjectName()) } - return v1alpha1.ErrRestarted{Objects: names} + return stnrv1.ErrRestarted{Objects: names} } return nil diff --git a/reconcile_test.go b/reconcile_test.go index b7a57b1c..1c32c564 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -4,6 +4,7 @@ import ( "bytes" "fmt" "net" + // "strconv" "testing" "time" @@ -13,7 +14,7 @@ import ( "github.com/l7mp/stunner/internal/object" "github.com/l7mp/stunner/internal/resolver" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" a12n "github.com/l7mp/stunner/pkg/authentication" "github.com/l7mp/stunner/pkg/logger" ) @@ -30,30 +31,30 @@ const ( // ***************** type StunnerReconcileTestConfig struct { name string - config v1alpha1.StunnerConfig + config stnrv1.StunnerConfig tester func(t *testing.T, s *Stunner, err error) } var testReconcileDefault = []StunnerReconcileTestConfig{ { name: "reconcile-test: default admin", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -63,23 +64,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() - assert.Equal(t, admin.Name, v1alpha1.DefaultStunnerName, "stunner name") + assert.Equal(t, admin.Name, stnrv1.DefaultStunnerName, "stunner name") // make sure we get the right loglevel, we may override this for debugging the tests - // assert.Equal(t, admin.LogLevel, v1alpha1.DefaultLogLevel, "stunner loglevel") + // assert.Equal(t, admin.LogLevel, stnrv1.DefaultLogLevel, "stunner loglevel") assert.Len(t, s.authManager.Keys(), 1, "authManager keys") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypePlainText, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeStatic, "auth type ok") assert.Equal(t, auth.Username, "user", "username ok") assert.Equal(t, auth.Password, "pass", "password ok") handler := s.NewAuthHandler() - key, ok := handler("user", v1alpha1.DefaultRealm, + key, ok := handler("user", stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") assert.Equal(t, key, a12n.GenerateAuthKey("user", - v1alpha1.DefaultRealm, "pass"), "auth handler ok") + stnrv1.DefaultRealm, "pass"), "auth handler ok") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -87,11 +88,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -100,7 +101,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -124,22 +125,22 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty credentials errs: user", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -150,22 +151,22 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty credentials errs: passwd", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -176,19 +177,19 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty listener is fine", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{{ + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -200,22 +201,22 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty listener name errs", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -226,23 +227,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty cluster is fine", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{}, + Clusters: []stnrv1.ClusterConfig{}, }, tester: func(t *testing.T, s *Stunner, err error) { assert.NoError(t, err, "no restart needed") @@ -250,23 +251,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty cluster name errs", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Endpoints: []string{"0.0.0.0/0"}, }}, }, @@ -278,24 +279,24 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ /// admin { name: "reconcile-test: reconcile name", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ Name: "new-name", LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -308,21 +309,21 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() assert.Equal(t, admin.Name, "new-name", "stunner name") - // assert.Equal(t, admin.LogLevel, v1alpha1.DefaultLogLevel, "stunner loglevel") + // assert.Equal(t, admin.LogLevel, stnrv1.DefaultLogLevel, "stunner loglevel") assert.Len(t, s.authManager.Keys(), 1, "authManager keys") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypePlainText, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeStatic, "auth type ok") assert.Equal(t, auth.Username, "user", "username ok") assert.Equal(t, auth.Password, "pass", "password ok") handler := s.NewAuthHandler() - key, ok := handler("user", v1alpha1.DefaultRealm, + key, ok := handler("user", stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") assert.Equal(t, key, a12n.GenerateAuthKey("user", - v1alpha1.DefaultRealm, "pass"), "auth handler ok") + stnrv1.DefaultRealm, "pass"), "auth handler ok") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -330,11 +331,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -343,7 +344,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -366,23 +367,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile loglevel", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: "anything", }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -398,17 +399,17 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Len(t, s.authManager.Keys(), 1, "authManager keys") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypePlainText, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeStatic, "auth type ok") assert.Equal(t, auth.Username, "user", "username ok") assert.Equal(t, auth.Password, "pass", "password ok") handler := s.NewAuthHandler() - key, ok := handler("user", v1alpha1.DefaultRealm, + key, ok := handler("user", stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") assert.Equal(t, key, a12n.GenerateAuthKey("user", - v1alpha1.DefaultRealm, "pass"), "auth handler ok") + stnrv1.DefaultRealm, "pass"), "auth handler ok") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -416,11 +417,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -429,7 +430,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -452,24 +453,24 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile metrics_endpoint", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: "anything", MetricsEndpoint: "http://0.0.0.0:8080/metrics", }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -482,23 +483,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() assert.Equal(t, admin.Name, "default-stunnerd", "stunner name") - // assert.Equal(t, admin.LogLevel, v1alpha1.DefaultLogLevel, "stunner loglevel") + // assert.Equal(t, admin.LogLevel, stnrv1.DefaultLogLevel, "stunner loglevel") assert.Equal(t, admin.MetricsEndpoint, "http://0.0.0.0:8080/metrics", "stunner metrics endpoint") assert.Len(t, s.authManager.Keys(), 1, "authManager keys") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypePlainText, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeStatic, "auth type ok") assert.Equal(t, auth.Username, "user", "username ok") assert.Equal(t, auth.Password, "pass", "password ok") handler := s.NewAuthHandler() - key, ok := handler("user", v1alpha1.DefaultRealm, + key, ok := handler("user", stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") assert.Equal(t, key, a12n.GenerateAuthKey("user", - v1alpha1.DefaultRealm, "pass"), "auth handler ok") + stnrv1.DefaultRealm, "pass"), "auth handler ok") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -506,11 +507,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -519,7 +520,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -543,23 +544,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ /// auth { name: "reconcile-test: reconcile plaintextauth name", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "newuser", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -569,21 +570,21 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NoError(t, err, "no restart needed") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypePlainText, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeStatic, "auth type ok") assert.Equal(t, auth.Username, "newuser", "username ok") assert.Equal(t, auth.Password, "pass", "password ok") handler := s.NewAuthHandler() - key, ok := handler("newuser", v1alpha1.DefaultRealm, + key, ok := handler("newuser", stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") assert.Equal(t, key, a12n.GenerateAuthKey("newuser", - v1alpha1.DefaultRealm, "pass"), "auth handler ok") + stnrv1.DefaultRealm, "pass"), "auth handler ok") assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() - assert.Equal(t, admin.Name, v1alpha1.DefaultStunnerName, "stunner name") + assert.Equal(t, admin.Name, stnrv1.DefaultStunnerName, "stunner name") // assert.Equal(t, admin.LogLevel, "anything", "stunner loglevel") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -592,11 +593,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -605,7 +606,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -628,23 +629,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile plaintext auth passwd", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "newpass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -654,21 +655,21 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NoError(t, err, "no restart needed") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypePlainText, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeStatic, "auth type ok") assert.Equal(t, auth.Username, "user", "username ok") assert.Equal(t, auth.Password, "newpass", "password ok") handler := s.NewAuthHandler() - key, ok := handler("user", v1alpha1.DefaultRealm, + key, ok := handler("user", stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") assert.Equal(t, key, a12n.GenerateAuthKey("user", - v1alpha1.DefaultRealm, "newpass"), "auth handler ok") + stnrv1.DefaultRealm, "newpass"), "auth handler ok") assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() - assert.Equal(t, admin.Name, v1alpha1.DefaultStunnerName, "stunner name") + assert.Equal(t, admin.Name, stnrv1.DefaultStunnerName, "stunner name") // assert.Equal(t, admin.LogLevel, "anything", "stunner loglevel") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -677,11 +678,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -690,7 +691,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -713,23 +714,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile longterm auth", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "newsecret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -739,7 +740,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NoError(t, err, "no restart needed") auth := s.GetAuth() - assert.Equal(t, auth.Type, v1alpha1.AuthTypeLongTerm, "auth type ok") + assert.Equal(t, auth.Type, stnrv1.AuthTypeEphemeral, "auth type ok") assert.Equal(t, auth.Secret, "newsecret") duration, _ := time.ParseDuration("10h") @@ -748,16 +749,16 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NoError(t, err, "GetLongTermCredential") handler := s.NewAuthHandler() - key, ok := handler(username, v1alpha1.DefaultRealm, + key, ok := handler(username, stnrv1.DefaultRealm, &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}) assert.True(t, ok, "authHandler key ok") - key2 := a12n.GenerateAuthKey(username, v1alpha1.DefaultRealm, passwd) + key2 := a12n.GenerateAuthKey(username, stnrv1.DefaultRealm, passwd) assert.Equal(t, key, key2, "authHandler key matches") assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() - assert.Equal(t, admin.Name, v1alpha1.DefaultStunnerName, "stunner name") + assert.Equal(t, admin.Name, stnrv1.DefaultStunnerName, "stunner name") // assert.Equal(t, admin.LogLevel, "anything", "stunner loglevel") assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") @@ -766,11 +767,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -779,7 +780,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -803,18 +804,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ /// listener { name: "reconcile-test: reconcile existing listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Protocol: "turn-tcp", Addr: "127.0.0.2", @@ -823,7 +824,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ MaxRelayPort: 100, Routes: []string{"none", "dummy"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -831,7 +832,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ tester: func(t *testing.T, s *Stunner, err error) { // requires a restart! assert.Error(t, err, "restarted") - e, ok := err.(v1alpha1.ErrRestarted) + e, ok := err.(stnrv1.ErrRestarted) assert.True(t, ok, "restarted status") assert.Len(t, e.Objects, 1, "restarted object") assert.Contains(t, e.Objects, "listener: default-listener") @@ -842,7 +843,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 12345, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -854,7 +855,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Len(t, s.adminManager.Keys(), 1, "adminManager keys") admin := s.GetAdmin() - assert.Equal(t, admin.Name, v1alpha1.DefaultStunnerName, "stunner name") + assert.Equal(t, admin.Name, stnrv1.DefaultStunnerName, "stunner name") // assert.Equal(t, admin.LogLevel, "anything", "stunner loglevel") assert.Len(t, s.clusterManager.Keys(), 1, "clusterManager keys") @@ -862,7 +863,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -885,18 +886,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile new listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "newlistener", Protocol: "turn-tcp", Addr: "127.0.0.2", @@ -905,7 +906,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ MaxRelayPort: 100, Routes: []string{"none", "dummy"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -923,7 +924,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -936,7 +937,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -959,18 +960,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: empty TLS credentials errs", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "newlistener", Protocol: "turn-tls", Addr: "127.0.0.2", @@ -979,7 +980,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ MaxRelayPort: 100, Routes: []string{"none", "dummy"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -990,18 +991,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile additional listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, @@ -1014,7 +1015,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ MaxRelayPort: 100, Routes: []string{"none", "dummy"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -1028,18 +1029,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ l := s.GetListener("default-listener") assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1063,7 +1064,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -1090,18 +1091,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile existing listener with TLS cert and add a new one", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Protocol: "TURN-DTLS", @@ -1117,7 +1118,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ MaxRelayPort: 100, Routes: []string{"none", "dummy"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -1125,7 +1126,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ tester: func(t *testing.T, s *Stunner, err error) { // default-listener restarts assert.Error(t, err, "restarted") - e, ok := err.(v1alpha1.ErrRestarted) + e, ok := err.(stnrv1.ErrRestarted) assert.True(t, ok, "restarted status") assert.Len(t, e.Objects, 1, "restarted object") assert.Contains(t, e.Objects, "listener: default-listener") @@ -1135,20 +1136,20 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ l := s.GetListener("default-listener") assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNDTLS, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNDTLS, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, bytes.Compare(l.Cert, []byte("dummy-cert")), 0, "listener cert ok") assert.Equal(t, bytes.Compare(l.Key, []byte("dummy-key")), 0, "listener key ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1172,7 +1173,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -1199,18 +1200,18 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile existing listener with TLS cert and add a new one", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Protocol: "TURN-TLS", @@ -1226,7 +1227,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ MaxRelayPort: 100, Routes: []string{"none", "dummy"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -1234,7 +1235,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ tester: func(t *testing.T, s *Stunner, err error) { // default-listener restarts assert.Error(t, err, "restarted") - e, ok := err.(v1alpha1.ErrRestarted) + e, ok := err.(stnrv1.ErrRestarted) assert.True(t, ok, "restarted status") assert.Len(t, e.Objects, 1, "restarted object") assert.Contains(t, e.Objects, "listener: default-listener") @@ -1244,20 +1245,20 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ l := s.GetListener("default-listener") assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTLS, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTLS, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, bytes.Compare(l.Cert, []byte("dummy-cert")), 0, "listener cert ok") assert.Equal(t, bytes.Compare(l.Key, []byte("dummy-key")), 0, "listener key ok") - assert.Equal(t, l.Port, v1alpha1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, v1alpha1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, v1alpha1.DefaultMaxRelayPort, "listener maxport ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") + assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1281,7 +1282,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, l, "listener found") assert.IsType(t, l, &object.Listener{}, "listener type ok") - assert.Equal(t, l.Proto, v1alpha1.ListenerProtocolTURNTCP, "listener proto ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") assert.Equal(t, l.MinPort, 10, "listener minport ok") @@ -1308,19 +1309,19 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile deleted listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{{ + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -1342,23 +1343,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ /// cluster { name: "reconcile-test: reconcile existing cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, }}, @@ -1371,7 +1372,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1398,23 +1399,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile new cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "newcluster", Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, }}, @@ -1430,7 +1431,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c = s.GetCluster("newcluster") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1458,23 +1459,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile additional cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "newcluster", Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, }, { @@ -1490,7 +1491,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1503,7 +1504,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c = s.GetCluster("newcluster") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ = net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1527,23 +1528,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile additional cluster and reroute", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"newcluster"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "newcluster", Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, }, { @@ -1560,7 +1561,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1573,7 +1574,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c = s.GetCluster("newcluster") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.Type, v1alpha1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ = net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1596,23 +1597,23 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, { name: "reconcile-test: reconcile deleted cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Addr: "127.0.0.1", Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{}, + Clusters: []stnrv1.ClusterConfig{}, }, tester: func(t *testing.T, s *Stunner, err error) { assert.NoError(t, err, err) @@ -1705,7 +1706,7 @@ func TestStunnerReconcile(t *testing.T) { type StunnerTestReconcileE2EConfig struct { testName string - config v1alpha1.StunnerConfig + config stnrv1.StunnerConfig echoServerAddr string bindSuccess, allocateSuccess, echoResult, restart bool } @@ -1790,19 +1791,19 @@ func testStunnerReconcileWithVNet(t *testing.T, testcases []StunnerTestReconcile var testReconcileE2E = []StunnerTestReconcileE2EConfig{ { testName: "initial E2E reconcile test: empty server", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{}, + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{}, }, echoServerAddr: "1.2.3.5:5678", restart: false, @@ -1812,18 +1813,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "adding a listener at the wrong port", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1832,7 +1833,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{}, + Clusters: []stnrv1.ClusterConfig{}, }, echoServerAddr: "1.2.3.5:5678", restart: false, @@ -1842,18 +1843,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "adding a cluster to a listener at the wrong port", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1862,7 +1863,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -1877,18 +1878,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "adding a listener at the right port", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1905,7 +1906,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -1920,18 +1921,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "changing the port in the wrong listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1948,7 +1949,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -1963,18 +1964,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "changing plaintext credentials to a wrong passwd", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "dummy", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1991,7 +1992,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2006,18 +2007,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "changing auth to longterm credentials errs", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ - Type: "longterm", + Auth: stnrv1.AuthConfig{ + Type: "ephemeral", Credentials: map[string]string{ "secret": "dummy", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2034,7 +2035,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2049,19 +2050,19 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "reverting good plaintext credentials ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Realm: "stunner.l7mp.io", Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2078,7 +2079,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2093,19 +2094,19 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "realm reset induces a server restart", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Realm: "dummy", Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2122,7 +2123,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2137,19 +2138,19 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "reverting the realm induces another server restart", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Realm: "stunner.l7mp.io", Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2166,7 +2167,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2181,18 +2182,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "adding a cluster to the wrong IP", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2211,7 +2212,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2229,18 +2230,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "removing working cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2259,7 +2260,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "dummy-cluster", Endpoints: []string{}, }}, @@ -2272,18 +2273,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "reintroducing good cluster to the wrong IP", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2302,7 +2303,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2320,18 +2321,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "removing wrong listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2341,7 +2342,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2359,18 +2360,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "correct the wrong cluster and remove the good one", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2380,7 +2381,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.10", @@ -2400,18 +2401,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "removing wrong cluster and reverting the working one", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2421,7 +2422,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2436,18 +2437,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "removing dangling cluster ref", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2456,7 +2457,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2471,18 +2472,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "converting cluster to strict dns", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2492,7 +2493,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "dummy-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STRICT_DNS", Endpoints: []string{ @@ -2508,18 +2509,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "rewiring to an open cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2528,7 +2529,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "open-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "open-cluster", Endpoints: []string{ "0.0.0.0/0", @@ -2543,18 +2544,18 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "closing open cluster", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-ok", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2563,7 +2564,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ "open-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{}, + Clusters: []stnrv1.ClusterConfig{}, }, echoServerAddr: "1.2.3.5:5678", restart: false, @@ -2573,19 +2574,19 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ }, { testName: "closing listener", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{}, + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{}, }, echoServerAddr: "1.2.3.5:5678", restart: false, @@ -2609,18 +2610,18 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ "reconcile protocol": { { testName: "base config", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -2629,7 +2630,7 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -2646,18 +2647,18 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ // this will trigger an error at a later stage of reconciliation that the // validation phase cannot catch and cause a rollback testName: "reconcile listener with an invalid TLS cert/key", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user", "password": "pass", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Protocol: "turn-tls", Addr: "1.2.3.4", @@ -2668,7 +2669,7 @@ var testReconcileRollback = map[string][]StunnerTestReconcileE2EConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", diff --git a/server.go b/server.go index 15c60a98..cb06f0dd 100644 --- a/server.go +++ b/server.go @@ -11,7 +11,7 @@ import ( "github.com/l7mp/stunner/internal/object" "github.com/l7mp/stunner/internal/telemetry" "github.com/l7mp/stunner/internal/util" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // Start will start the TURN server that belongs to a listener. @@ -37,7 +37,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { addr := fmt.Sprintf("0.0.0.0:%d", l.Port) switch l.Proto { - case v1alpha1.ListenerProtocolTURNUDP: + case stnrv1.ListenerProtocolTURNUDP: socketPool := util.NewPacketConnPool(l.Net, s.udpThreadNum) s.log.Infof("setting up UDP listener socket pool at %s with %d readloop threads", @@ -59,7 +59,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { pConns = append(pConns, conn) } - case v1alpha1.ListenerProtocolTURNTCP: + case stnrv1.ListenerProtocolTURNTCP: s.log.Debugf("setting up TCP listener at %s", addr) tcpListener, err := net.Listen("tcp", addr) @@ -79,7 +79,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { l.Conns = append(l.Conns, conn) // cannot test this on vnet, no TLS in vnet.Net - case v1alpha1.ListenerProtocolTURNTLS: + case stnrv1.ListenerProtocolTURNTLS: s.log.Debugf("setting up TLS/TCP listener at %s", addr) cer, err := tls.X509KeyPair(l.Cert, l.Key) @@ -106,7 +106,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { lConns = append(lConns, conn) l.Conns = append(l.Conns, conn) - case v1alpha1.ListenerProtocolTURNDTLS: + case stnrv1.ListenerProtocolTURNDTLS: s.log.Debugf("setting up DTLS/UDP listener at %s", addr) cer, err := tls.X509KeyPair(l.Cert, l.Key) diff --git a/server_unix_test.go b/server_unix_test.go index 2515d4d5..1188307d 100644 --- a/server_unix_test.go +++ b/server_unix_test.go @@ -8,8 +8,7 @@ import ( "testing" "time" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" - + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/logger" ) @@ -18,20 +17,20 @@ const clientNum = 20 // multithreaded UDP tests var TestStunnerConfigsMultithreadedUDP = []TestStunnerConfigCase{ { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // udp, plaintext - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "127.0.0.1", @@ -40,7 +39,7 @@ var TestStunnerConfigsMultithreadedUDP = []TestStunnerConfigCase{ PublicPort: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -69,19 +68,19 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { }) log.Debug("starting stunnerd") - err := stunner.Reconcile(v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + err := stunner.Reconcile(stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "default-listener", Protocol: proto, Addr: "127.0.0.1", @@ -90,7 +89,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { Key: keyPem64, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, diff --git a/stunner.go b/stunner.go index 681ae439..6d3f7464 100644 --- a/stunner.go +++ b/stunner.go @@ -15,7 +15,7 @@ import ( "github.com/l7mp/stunner/internal/object" "github.com/l7mp/stunner/internal/resolver" "github.com/l7mp/stunner/internal/telemetry" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/logger" ) @@ -84,7 +84,7 @@ func NewStunner(options Options) *Stunner { s := &Stunner{ id: id, - version: v1alpha1.ApiVersion, + version: stnrv1.ApiVersion, logger: logger, log: log, suppressRollback: options.SuppressRollback, @@ -139,7 +139,7 @@ func (s *Stunner) Shutdown() { // GetAdmin returns the admin object underlying STUNner. func (s *Stunner) GetAdmin() *object.Admin { - a, found := s.adminManager.Get(v1alpha1.DefaultAdminName) + a, found := s.adminManager.Get(stnrv1.DefaultAdminName) if !found { panic("internal error: no Admin found") } @@ -148,7 +148,7 @@ func (s *Stunner) GetAdmin() *object.Admin { // GetAuth returns the authenitation object underlying STUNner. func (s *Stunner) GetAuth() *object.Auth { - a, found := s.authManager.Get(v1alpha1.DefaultAuthName) + a, found := s.authManager.Get(stnrv1.DefaultAuthName) if !found { panic("internal error: no Auth found") } diff --git a/stunner_test.go b/stunner_test.go index a81147a8..4acc9830 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -3,6 +3,7 @@ package stunner import ( "crypto/tls" "encoding/base64" + "encoding/json" "fmt" "net" "net/http" @@ -23,13 +24,15 @@ import ( "github.com/l7mp/stunner/internal/resolver" "github.com/l7mp/stunner/pkg/logger" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + stnrv1a1 "github.com/l7mp/stunner/pkg/apis/v1alpha1" a12n "github.com/l7mp/stunner/pkg/authentication" + cfgclient "github.com/l7mp/stunner/pkg/config/client" ) var stunnerTestLoglevel string = "all:ERROR" -// var stunnerTestLoglevel string = v1alpha1.DefaultLogLevel +// var stunnerTestLoglevel string = stnrv1.DefaultLogLevel // var stunnerTestLoglevel string = "all:INFO" // var stunnerTestLoglevel string = "all:TRACE" // var stunnerTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" @@ -261,26 +264,26 @@ func buildVNet(logger logging.LoggerFactory) (*VNet, error) { *********************************************/ type TestStunnerConfigCase struct { - config v1alpha1.StunnerConfig + config stnrv1.StunnerConfig uri string } var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // udp, plaintext - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "127.0.0.1", @@ -289,7 +292,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ PublicPort: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -297,19 +300,19 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ uri: "turn:1.2.3.4:3478?transport=udp", }, { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // udp, longterm - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "127.0.0.1", @@ -318,7 +321,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ PublicPort: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -326,20 +329,20 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ uri: "turn:1.2.3.4:3478?transport=udp", }, { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // tcp, plaintext - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ - Type: "plaintext", + Auth: stnrv1.AuthConfig{ + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "tcp", Protocol: "turn-tcp", Addr: "127.0.0.1", @@ -348,7 +351,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ PublicPort: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -356,19 +359,19 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ uri: "turn:1.2.3.4:3478?transport=tcp", }, { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // tcp, longterm - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "tcp", Protocol: "turn-tcp", Addr: "127.0.0.1", @@ -377,7 +380,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ PublicPort: 3478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -385,20 +388,20 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ uri: "turn:1.2.3.4:3478?transport=tcp", }, { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // tls, plaintext - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ - Type: "plaintext", + Auth: stnrv1.AuthConfig{ + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "tls", Protocol: "turn-tls", Addr: "127.0.0.1", @@ -409,7 +412,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ Key: keyPem64, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -417,19 +420,19 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ uri: "turns:1.2.3.4:3478?transport=tcp", }, { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // tls, longterm - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": "my-secret", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "tls", Protocol: "turn-tls", Addr: "127.0.0.1", @@ -440,7 +443,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ Key: keyPem64, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -448,20 +451,20 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ uri: "turns:1.2.3.4:3478?transport=tcp", }, { - config: v1alpha1.StunnerConfig{ + config: stnrv1.StunnerConfig{ // dtls, plaintext - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "dtls", Protocol: "turn-dtls", Addr: "127.0.0.1", @@ -472,7 +475,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ Key: keyPem64, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -481,24 +484,24 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, // // dtls, longterm // { - // ApiVersion: "v1alpha1", - // Admin: v1alpha1.AdminConfig{ + // ApiVersion: stnrv1.ApiVersion, + // Admin: stnrv1.AdminConfig{ // LogLevel: stunnerTestLoglevel, // }, - // Auth: v1alpha1.AuthConfig{ + // Auth: stnrv1.AuthConfig{ // Type: "longterm", // Credentials: map[string]string{ // "secret": "my-secret", // }, // }, - // Listeners: []v1alpha1.ListenerConfig{{ + // Listeners: []stnrv1.ListenerConfig{{ // Name: "dtls", // Protocol: "turn-dtls", // Addr: "127.0.0.1", // Port: 23478, // Routes: []string{"allow-any"}, // }}, - // Clusters: []v1alpha1.ClusterConfig{{ + // Clusters: []stnrv1.ClusterConfig{{ // Name: "allow-any", // Endpoints: []string{"0.0.0.0/0"}, // }}, @@ -559,10 +562,10 @@ func testStunnerLocalhost(t *testing.T, udpThreadNum int, tests []TestStunnerCon var u, p string switch auth { - case "plaintext": + case "plaintext", "static": u = "user1" p = "passwd1" - case "longterm": + case "longterm", "ephemeral": u = a12n.GenerateTimeWindowedUsername(time.Now(), time.Minute, "") p2, err := a12n.GetLongTermCredential(u, "my-secret") assert.NoError(t, err, err) @@ -640,13 +643,13 @@ func testStunnerLocalhost(t *testing.T, udpThreadNum int, tests []TestStunnerCon // ***************** // // type StunnerClusterConfig struct { -// config v1alpha1.StunnerConfig +// config stnrv1.StunnerConfig // echoServerAddr string // result bool // } type StunnerTestClusterConfig struct { testName string - config v1alpha1.StunnerConfig + config stnrv1.StunnerConfig echoServerAddr string result bool } @@ -654,26 +657,26 @@ type StunnerTestClusterConfig struct { var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ { testName: "open ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", Port: 3478, Routes: []string{"echo-server-cluster"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -686,19 +689,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "default cluster type static ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -707,7 +710,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Endpoints: []string{ "1.2.3.5", @@ -719,19 +722,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "static endpoint ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -740,7 +743,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -753,19 +756,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "static endpoint with wrong peer addr: fail", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -774,7 +777,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -787,19 +790,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "static endpoint with multiple routes ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -809,7 +812,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "dummy_cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -828,19 +831,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "static endpoint with multiple routes and wrong peer addr fail", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -850,7 +853,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -869,19 +872,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "static endpoint with multiple ips ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -890,7 +893,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -907,19 +910,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "static endpoint with multiple ips with wrong peer addr fail", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -928,7 +931,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STATIC", Endpoints: []string{ @@ -944,19 +947,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "strict_dns ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -965,7 +968,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STRICT_DNS", Endpoints: []string{ @@ -978,19 +981,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "strict_dns cluster and wrong peer addr fail", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -999,7 +1002,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STRICT_DNS", Endpoints: []string{ @@ -1012,19 +1015,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "strict_dns cluster with multiple domains ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1033,7 +1036,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "echo-server-cluster", Type: "STRICT_DNS", Endpoints: []string{ @@ -1047,19 +1050,19 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ }, { testName: "multiple strict_dns clusters ok", - config: v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp", Protocol: "turn-udp", Addr: "1.2.3.4", @@ -1069,7 +1072,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "stunner-cluster", Type: "STRICT_DNS", Endpoints: []string{ @@ -1128,10 +1131,10 @@ func TestStunnerClusterWithVNet(t *testing.T) { var u, p string auth := c.config.Auth.Type switch auth { - case "plaintext": + case "plaintext", "static": u = "user1" p = "passwd1" - case "longterm": + case "longterm", "ephemeral": u, p, err = turn.GenerateLongTermCredentials("my-secret", time.Minute) assert.NoError(t, err, err) default: @@ -1279,17 +1282,17 @@ func TestStunnerLifecycle(t *testing.T) { assert.Error(t, err, "no default readiness check for empty server") log.Debug("starting stunnerd with an empty stunner config") - conf := v1alpha1.StunnerConfig{ - ApiVersion: v1alpha1.ApiVersion, - Admin: v1alpha1.AdminConfig{LogLevel: stunnerTestLoglevel}, - Auth: v1alpha1.AuthConfig{ + conf := stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{LogLevel: stunnerTestLoglevel}, + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user-1", "password": "pass-1", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{}, + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{}, } log.Debug("reconciling empty server") @@ -1328,7 +1331,7 @@ func TestStunnerLifecycle(t *testing.T) { port := u.Port() if port == "" { - port = strconv.Itoa(v1alpha1.DefaultHealthCheckPort) + port = strconv.Itoa(stnrv1.DefaultHealthCheckPort) } hc := fmt.Sprintf("http://%s:%s", addr, port) @@ -1450,17 +1453,17 @@ func TestStunnerMetrics(t *testing.T) { // assert.False(t, s.IsReady(), "empty server not ready") log.Debug("starting stunnerd with an empty stunner config") - conf := v1alpha1.StunnerConfig{ - ApiVersion: v1alpha1.ApiVersion, - Admin: v1alpha1.AdminConfig{LogLevel: stunnerTestLoglevel}, - Auth: v1alpha1.AuthConfig{ + conf := stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{LogLevel: stunnerTestLoglevel}, + Auth: stnrv1.AuthConfig{ Credentials: map[string]string{ "username": "user-1", "password": "pass-1", }, }, - Listeners: []v1alpha1.ListenerConfig{}, - Clusters: []v1alpha1.ClusterConfig{}, + Listeners: []stnrv1.ListenerConfig{}, + Clusters: []stnrv1.ClusterConfig{}, } log.Debug("reconciling empty server") @@ -1489,7 +1492,7 @@ func TestStunnerMetrics(t *testing.T) { port := u.Port() if port == "" { - port = strconv.Itoa(v1alpha1.DefaultMetricsPort) + port = strconv.Itoa(stnrv1.DefaultMetricsPort) } path := u.EscapedPath() @@ -1530,3 +1533,148 @@ func doLivenessCheck(uri string) (bool, error) { func doReadinessCheck(uri string) (bool, error) { return doHttp(uri + "/ready") } + +// ***************** +// v1alpha1 API compatibility tests +// ***************** +type TestConfigV1Alpha1 struct { + testName string + config []byte + echoServerAddr string + result bool +} + +var testConfigsV1Alpha1 = []TestConfigV1Alpha1{ + { + testName: "open ok", + config: []byte(`{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"plaintext","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}`), + echoServerAddr: "1.2.3.5:5678", + result: true, + }, + { + testName: "default cluster type static ok", + config: []byte(`{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"plaintext","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","endpoints":["1.2.3.5"]}]}`), + echoServerAddr: "1.2.3.5:5678", + result: true, + }, + { + testName: "static endpoint ok", + config: []byte(`{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"plaintext","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}`), + echoServerAddr: "1.2.3.5:5678", + result: true, + }, + { + testName: "static endpoint with multiple routes ok", + config: []byte(`{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"plaintext","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster","dummy_cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]},{"name":"dummy_cluster","type":"STATIC","endpoints":["9.8.7.6"]}]}`), + echoServerAddr: "1.2.3.5:5678", + result: true, + }, + { + testName: "longterm endpoint with multiple routes ok", + config: []byte(`{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"longterm","credentials":{"secret":"my-secret"}},"listeners":[{"name":"udp","protocol":"turn-udp","public_address":"1.2.3.4","public_port":3478,"address":"127.0.0.1","port":3478,"routes":["allow-any"]}],"clusters":[{"name":"allow-any","endpoints":["0.0.0.0/0"]}]}`), + echoServerAddr: "1.2.3.5:5678", + result: true, + }, +} + +func TestStunnerConfigV1Alpha1(t *testing.T) { + lim := test.TimeOut(time.Second * 60) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) + log := loggerFactory.NewLogger("test") + + for _, c := range testConfigsV1Alpha1 { + t.Run(c.testName, func(t *testing.T) { + log.Debugf("-------------- Running test: %s -------------", c.testName) + + // patch in the vnet + log.Debug("building virtual network") + v, err := buildVNet(loggerFactory) + assert.NoError(t, err, err) + + log.Debug("creating a stunnerd") + stunner := NewStunner(Options{ + LogLevel: stunnerTestLoglevel, + SuppressRollback: true, + Net: v.podnet, + }) + + log.Debug("parsing config to v1alpha1 format") + a := stnrv1a1.StunnerConfig{} + assert.NoError(t, json.Unmarshal(c.config, &a), "parsing config file to v1alpha1 format") + + assert.Equal(t, stnrv1a1.ApiVersion, a.ApiVersion, "version") + assert.Equal(t, "all:ERROR", a.Admin.LogLevel, "loglevel") + // expect the old names + assert.True(t, a.Auth.Type == "plaintext" || a.Auth.Type == "longterm", "loglevel") + assert.Len(t, a.Listeners, 1, "listeners len") + assert.Equal(t, "udp", a.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", a.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, a.Listeners[0].Port, "listener port") + assert.True(t, len(a.Clusters) > 0, "clusters len") + + log.Debug("conveting config to v1 format") + a = stnrv1a1.StunnerConfig{} + assert.NoError(t, json.Unmarshal(c.config, &a), "parsing config file to v1alpha1 format") + config, err := stnrv1a1.ConvertToV1(&a) + assert.NoError(t, err, "convert load v1alpha1 config to v1") + + assert.Equal(t, stnrv1.ApiVersion, config.ApiVersion, "version") + assert.Equal(t, "all:ERROR", config.Admin.LogLevel, "loglevel") + // expect the new names + assert.True(t, config.Auth.Type == "static" || config.Auth.Type == "ephemeral", "loglevel") + assert.Len(t, config.Listeners, 1, "listeners len") + assert.Equal(t, "udp", config.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", config.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, config.Listeners[0].Port, "listener port") + assert.True(t, len(config.Clusters) > 0, "clusters len") + + log.Debug("parsing config directly to v1 format") + config, err = cfgclient.ParseConfig(c.config) + assert.NoError(t, err, "load v1alpha1 config ") + + assert.Equal(t, stnrv1.ApiVersion, config.ApiVersion, "version") + assert.Equal(t, "all:ERROR", config.Admin.LogLevel, "loglevel") + // expect the new names + assert.True(t, config.Auth.Type == "static" || config.Auth.Type == "ephemeral", "loglevel") + assert.Len(t, config.Listeners, 1, "listeners len") + assert.Equal(t, "udp", config.Listeners[0].Name, "listener name") + assert.Equal(t, "turn-udp", config.Listeners[0].Protocol, "listener proto") + assert.Equal(t, 3478, config.Listeners[0].Port, "listener port") + assert.True(t, len(config.Clusters) > 0, "clusters len") + + log.Debug("starting stunnerd") + assert.NoError(t, stunner.Reconcile(*config), "starting server") + + var u, p string + auth := config.Auth.Type + switch auth { + case "plaintext", "static": + u = "user1" + p = "passwd1" + case "longterm", "ephemeral": + u, p, err = turn.GenerateLongTermCredentials("my-secret", time.Minute) + assert.NoError(t, err, err) + default: + assert.NoError(t, fmt.Errorf("internal error: unknown auth type in test")) + } + + log.Debug("creating a client") + lconn, err := v.wan.ListenPacket("udp4", "0.0.0.0:0") + assert.NoError(t, err, "cannot create client listening socket") + + testConfig := echoTestConfig{t, v.podnet, v.wan, stunner, + "stunner.l7mp.io:3478", lconn, u, p, net.IPv4(5, 6, 7, 8), + c.echoServerAddr, true, true, c.result, loggerFactory} + stunnerEchoTest(testConfig) + + assert.NoError(t, lconn.Close(), "cannot close TURN client connection") + stunner.Close() + assert.NoError(t, v.Close(), "cannot close VNet") + }) + } +} diff --git a/turncat.go b/turncat.go index d65d8200..f5223b71 100644 --- a/turncat.go +++ b/turncat.go @@ -15,7 +15,7 @@ import ( "github.com/pion/turn/v3" "github.com/l7mp/stunner/internal/util" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) const UDP_PACKET_SIZE = 1500 @@ -109,7 +109,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { } if config.Realm == "" { - config.Realm = v1alpha1.DefaultRealm + config.Realm = stnrv1.DefaultRealm } // a global listener connection for the local tunnel endpoint diff --git a/turncat_test.go b/turncat_test.go index 7f2d81bd..a8264d89 100644 --- a/turncat_test.go +++ b/turncat_test.go @@ -13,9 +13,8 @@ import ( "github.com/pion/turn/v3" "github.com/stretchr/testify/assert" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/logger" - - "github.com/l7mp/stunner/pkg/apis/v1alpha1" ) var turncatTestLoglevel string = "all:ERROR" @@ -112,20 +111,20 @@ func TestTurncatPlaintext(t *testing.T) { SuppressRollback: true, }) - err := stunner.Reconcile(v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + err := stunner.Reconcile(stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: turncatTestLoglevel, MetricsEndpoint: "", }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "plaintext", Credentials: map[string]string{ "username": "user1", "password": "passwd1", }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-listener-23478", Protocol: "turn-udp", Addr: "127.0.0.1", @@ -138,7 +137,7 @@ func TestTurncatPlaintext(t *testing.T) { Port: 23478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, @@ -228,18 +227,18 @@ func TestTurncatLongterm(t *testing.T) { LogLevel: turncatTestLoglevel, SuppressRollback: true, }) - err := stunner.Reconcile(v1alpha1.StunnerConfig{ - ApiVersion: "v1alpha1", - Admin: v1alpha1.AdminConfig{ + err := stunner.Reconcile(stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ LogLevel: turncatTestLoglevel, }, - Auth: v1alpha1.AuthConfig{ + Auth: stnrv1.AuthConfig{ Type: "longterm", Credentials: map[string]string{ "secret": sharedSecret, }, }, - Listeners: []v1alpha1.ListenerConfig{{ + Listeners: []stnrv1.ListenerConfig{{ Name: "udp-listener-23478", Protocol: "turn-udp", Addr: "127.0.0.1", @@ -252,7 +251,7 @@ func TestTurncatLongterm(t *testing.T) { Port: 23478, Routes: []string{"allow-any"}, }}, - Clusters: []v1alpha1.ClusterConfig{{ + Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, }}, diff --git a/uri.go b/uri.go index be9bafc8..ec4cf88f 100644 --- a/uri.go +++ b/uri.go @@ -10,7 +10,7 @@ import ( "syscall" "github.com/l7mp/stunner/internal/util" - "github.com/l7mp/stunner/pkg/apis/v1alpha1" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) // StunnerUri is the specification of a STUNner listener URI @@ -95,7 +95,7 @@ func ParseUri(uri string) (*StunnerUri, error) { } func (u *StunnerUri) String() string { - req := v1alpha1.ListenerConfig{ + req := stnrv1.ListenerConfig{ Protocol: u.Protocol, PublicAddr: u.Address, PublicPort: u.Port, @@ -110,17 +110,17 @@ func (u *StunnerUri) String() string { } // GetUriFromListener returns a standard TURN URI as per RFC7065from a listener config. -func GetUriFromListener(req *v1alpha1.ListenerConfig) (string, error) { +func GetUriFromListener(req *stnrv1.ListenerConfig) (string, error) { return req.GetListenerURI(true) } // GetStandardURLFromListener returns a standard URL (that can be parsed using net/url) from a listener config. -func GetStandardURLFromListener(req *v1alpha1.ListenerConfig) (string, error) { +func GetStandardURLFromListener(req *stnrv1.ListenerConfig) (string, error) { return req.GetListenerURI(false) } // GetUriFromListener returns a standard TURN URI from a listener config -func GetTurnUris(req *v1alpha1.StunnerConfig) ([]string, error) { +func GetTurnUris(req *stnrv1.StunnerConfig) ([]string, error) { ret := []string{} for i := range req.Listeners { uri, err := GetUriFromListener(&req.Listeners[i]) From e61cfb2df1309148adc7c4eb42b108018e0a134b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 1 Dec 2023 23:49:58 +0100 Subject: [PATCH 061/243] feat: Implement port range filtering on peer connections It is now possible to limit the port range available on the peers via STUNner. If your media server allocates client connections in the [10000:20000] port range, now it is possible to configure stunnerd to admit peer connections only to that target port range. Denied packets are reported with an "administratively prohibited error" by the TURN server, and logs are rate-limited to at most 1 report per sec to throttle excessive log message flood. --- go.mod | 2 +- internal/telemetry/relay.go | 132 ------- internal/telemetry/statsconn.go | 2 +- internal/util/conn.go | 90 +++++ internal/util/relay.go | 111 ++++++ pkg/apis/v1/admin.go | 18 +- pkg/apis/v1/auth.go | 17 +- pkg/apis/v1/cluster.go | 25 +- pkg/apis/v1/config.go | 2 +- pkg/apis/v1/listener.go | 41 +- pkg/apis/v1/stunner.go | 17 +- pkg/apis/v1/util.go | 20 +- pkg/logger/logger.go | 266 ++++++++----- pkg/logger/logger_test.go | 652 ++++++++++++++------------------ server.go | 20 +- stunner.go | 2 +- stunner_test.go | 333 +++++++++++++++- 17 files changed, 1073 insertions(+), 677 deletions(-) delete mode 100644 internal/telemetry/relay.go create mode 100644 internal/util/relay.go diff --git a/go.mod b/go.mod index 67e91b8e..1f07145d 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,6 @@ require ( github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb github.com/pion/dtls/v2 v2.2.7 github.com/pion/logging v0.2.2 - github.com/pion/randutil v0.1.0 github.com/pion/transport/v3 v3.0.1 github.com/pion/turn/v3 v3.0.1 github.com/prometheus/client_golang v1.16.0 @@ -47,6 +46,7 @@ require ( github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/pion/randutil v0.1.0 // indirect github.com/pion/stun/v2 v2.0.0 // indirect github.com/pion/transport/v2 v2.2.1 // indirect github.com/pkg/errors v0.9.1 // indirect diff --git a/internal/telemetry/relay.go b/internal/telemetry/relay.go deleted file mode 100644 index a18c4e50..00000000 --- a/internal/telemetry/relay.go +++ /dev/null @@ -1,132 +0,0 @@ -package telemetry - -// code adopted from github.com/livekit/pkg/telemetry - -import ( - "errors" - "fmt" - "net" - - "github.com/pion/randutil" - "github.com/pion/transport/v3" - "github.com/pion/transport/v3/stdnet" -) - -var ( - errInvalidName = errors.New("RelayAddressGenerator: Name must be set") - errRelayAddressInvalid = errors.New("RelayAddressGenerator: invalid RelayAddress") - errMinPortNotZero = errors.New("RelayAddressGenerator: MinPort must be not 0") - errMaxPortNotZero = errors.New("RelayAddressGenerator: MaxPort must be not 0") - errListeningAddressInvalid = errors.New("RelayAddressGenerator: invalid ListeningAddress") - errNilConn = errors.New("cannot allocate relay connection") - errMaxRetriesExceeded = errors.New("max retries exceeded when trying to generate new relay connection: MinPort:MaxPort range too small?") - errTodo = errors.New("relay to Net.Conn not implemented") -) - -// RelayAddressGenerator can be used to only allocate connections inside a defined port range. A -// static ip address can be set. -type RelayAddressGenerator struct { - // Name is the name of the listener this relay address generator belongs to. Note that - // packets sent to/received from upstream cluster are reported with the name of the - // *listener* that the packet belongs to, and not the cluster. - Name string - - // RelayAddress is the IP returned to the user when the relay is created. - RelayAddress net.IP - - // MinPort the minimum port to allocate. - MinPort uint16 - // MaxPort the maximum (inclusive) port to allocate. - MaxPort uint16 - - // MaxRetries the amount of tries to allocate a random port in the defined range. - MaxRetries int - - // Rand the random source of numbers. - Rand randutil.MathRandomGenerator - - // Address is passed to Listen/ListenPacket when creating the Relay. - Address string - - // Net is a pion/transport VNet, used for testing. - Net transport.Net -} - -// Validate is called on server startup and confirms the RelayAddressGenerator is properly configured -func (r *RelayAddressGenerator) Validate() error { - if r.Name == "" { - return errInvalidName - } - - if r.Net == nil { - r.Net, _ = stdnet.NewNet() - } - - if r.Rand == nil { - r.Rand = randutil.NewMathRandomGenerator() - } - - if r.MaxRetries == 0 { - r.MaxRetries = 10 - } - - switch { - case r.MinPort == 0: - return errMinPortNotZero - case r.MaxPort == 0: - return errMaxPortNotZero - case r.RelayAddress == nil: - return errRelayAddressInvalid - case r.Address == "": - return errListeningAddressInvalid - default: - return nil - } -} - -// AllocatePacketConn generates a new PacketConn to receive traffic on and the IP/Port to populate -// the allocation response with -func (r *RelayAddressGenerator) AllocatePacketConn(network string, requestedPort int) (net.PacketConn, net.Addr, error) { - if requestedPort != 0 { - conn, err := r.Net.ListenPacket(network, fmt.Sprintf("%s:%d", r.Address, requestedPort)) - if err != nil { - return nil, nil, err - } - - conn = NewPacketConn(conn, r.Name, ClusterType) - - relayAddr, ok := conn.LocalAddr().(*net.UDPAddr) - if !ok { - return nil, nil, errNilConn - } - - relayAddr.IP = r.RelayAddress - return conn, relayAddr, nil - } - - for try := 0; try < r.MaxRetries; try++ { - port := r.MinPort + uint16(r.Rand.Intn(int((r.MaxPort+1)-r.MinPort))) - conn, err := r.Net.ListenPacket(network, fmt.Sprintf("%s:%d", r.Address, port)) - if err != nil { - continue - } - - conn = NewPacketConn(conn, r.Name, ClusterType) - - relayAddr, ok := conn.LocalAddr().(*net.UDPAddr) - if !ok { - return nil, nil, errNilConn - } - - relayAddr.IP = r.RelayAddress - return conn, relayAddr, nil - } - - return nil, nil, errMaxRetriesExceeded -} - -// AllocateConn generates a new Conn to receive traffic on and the IP/Port to populate the -// allocation response with -func (g *RelayAddressGenerator) AllocateConn(network string, requestedPort int) (net.Conn, net.Addr, error) { - return nil, nil, errTodo -} diff --git a/internal/telemetry/statsconn.go b/internal/telemetry/statsconn.go index 4b0cad9a..a83565ed 100644 --- a/internal/telemetry/statsconn.go +++ b/internal/telemetry/statsconn.go @@ -74,7 +74,7 @@ type PacketConn struct { connType ConnType } -// NewPacketConn allocates a stats conn that knows its name and type. +// NewPacketConn decorates a PacketConnn with metric reporting. func NewPacketConn(c net.PacketConn, n string, t ConnType) *PacketConn { AddConnection(n, t) return &PacketConn{PacketConn: c, name: n, connType: t} diff --git a/internal/util/conn.go b/internal/util/conn.go index 3e3e50dd..ebdfc5d3 100644 --- a/internal/util/conn.go +++ b/internal/util/conn.go @@ -1,14 +1,21 @@ package util import ( + "errors" "fmt" "net" "os" "time" + "github.com/pion/logging" "github.com/pion/transport/v3" ) +var ( + ErrPortProhibited = errors.New("peer port administratively prohibited") + ErrInvalidPeerProtocol = errors.New("unknown peer transport protocol") +) + type FileConnAddr struct { File *os.File } @@ -88,3 +95,86 @@ func (p *defaultPacketConnPool) Make(network, address string) ([]net.PacketConn, } func (p *defaultPacketConnPool) Size() int { return 1 } + +// PortRangePacketConn is a net.PacketConn that filters on the target port range. +type PortRangePacketConn struct { + net.PacketConn + name string + minPort, maxPort int + log logging.LeveledLogger + readDeadline time.Time +} + +// NewPortRangePacketConn decorates a PacketConn with filtering on a target port range. Errors are reported per listener name. +func NewPortRangePacketConn(c net.PacketConn, listenerName string, minPort, maxPort int, log logging.LeveledLogger) net.PacketConn { + return &PortRangePacketConn{ + PacketConn: c, + name: listenerName, + minPort: minPort, + maxPort: maxPort, + log: log, + } +} + +// WriteTo writes to the PacketConn. +func (c *PortRangePacketConn) WriteTo(p []byte, peerAddr net.Addr) (int, error) { + switch addr := peerAddr.(type) { + case *net.UDPAddr: + if addr.Port < c.minPort || addr.Port > c.maxPort { + // c.log.Infof("sending UDP packet with invalid peer port %d rejected on listener %q (must be in [%d:%d])", + // addr.Port, c.name, c.minPort, c.maxPort) + return 0, ErrPortProhibited + } + case *net.TCPAddr: + if addr.Port < c.minPort || addr.Port > c.maxPort { + // c.log.Infof("sending TCP packet with invalid peer port %d rejected on listener %q (must be in [%d:%d])", + // addr.Port, c.name, c.minPort, c.maxPort) + return 0, ErrPortProhibited + } + default: + return 0, ErrInvalidPeerProtocol + } + + return c.PacketConn.WriteTo(p, peerAddr) +} + +// ReadFrom reads from the PortRangePacketConn. Blocks until a packet from the speciifed port range +// is received and drops all other packets. +func (c *PortRangePacketConn) ReadFrom(p []byte) (int, net.Addr, error) { + for { + var peerAddr net.Addr + + err := c.PacketConn.SetReadDeadline(c.readDeadline) + if err != nil { + return 0, peerAddr, err + } + + n, peerAddr, err := c.PacketConn.ReadFrom(p) + + // Return errors unconditionally: peerAddr will most probably not be valid anyway + // so it is not worth checking + if err != nil { + return n, peerAddr, err + } + + switch addr := peerAddr.(type) { + case *net.UDPAddr: + if addr.Port >= c.minPort && addr.Port <= c.maxPort { + return n, peerAddr, err + } + // c.log.Infof("received UDP packet with invalid peer port %d dropped on listener %q (must be in [%d:%d])", + // addr.Port, c.name, c.minPort, c.maxPort) + case *net.TCPAddr: + if addr.Port >= c.minPort && addr.Port <= c.maxPort { + return n, peerAddr, err + } + // c.log.Infof("received TCP packet with invalid peer port %d dropped on listener %q (must be in [%d:%d])", + // addr.Port, c.name, c.minPort, c.maxPort) + } + } +} + +func (c *PortRangePacketConn) SetReadDeadline(t time.Time) error { + c.readDeadline = t + return nil +} diff --git a/internal/util/relay.go b/internal/util/relay.go new file mode 100644 index 00000000..bd55f817 --- /dev/null +++ b/internal/util/relay.go @@ -0,0 +1,111 @@ +package util + +// code adopted from github.com/livekit/pkg/telemetry + +import ( + "errors" + "fmt" + "net" + + "github.com/pion/transport/v3" + "github.com/pion/transport/v3/stdnet" + + "github.com/l7mp/stunner/internal/telemetry" + "github.com/l7mp/stunner/pkg/logger" +) + +var ( + errInvalidName = errors.New("RelayAddressGenerator: Name must be set") + errRelayAddressInvalid = errors.New("RelayAddressGenerator: invalid RelayAddress") + errInvalidRelayPortRange = errors.New("RelayAddressGenerator: invalid target port range [MinRelayPort:MaxRelayPort]") + errListeningAddressInvalid = errors.New("RelayAddressGenerator: invalid ListeningAddress") + errNilConn = errors.New("cannot allocate relay connection") + errTodo = errors.New("relay to Net.Conn not implemented") +) + +// RelayAddressGenerator can be used to only allocate connections inside a defined target port +// range. A static ip address can be set. +type RelayAddressGenerator struct { + // ListenerName is the name of the listener this relay address generator belongs to. Note that + // packets sent to/received from upstream cluster are reported with the name of the + // *listener* that the packet belongs to, and not the cluster. + ListenerName string + + // RelayAddress is the IP returned to the user when the relay is created. + RelayAddress net.IP + + // MinRelayPort the minimum target port (inclusive). + MinRelayPort int + + // MaxRelayPort the maximum target port (inclusive). + MaxRelayPort int + + // Address is passed to Listen/ListenPacket when creating the Relay. + Address string + + // Net is a pion/transport VNet, used for testing. + Net transport.Net + + // Logger is a logger factory we can use to generate per-listener relay loggers. + Logger *logger.LeveledLoggerFactory +} + +// Validate is called on server startup and confirms the RelayAddressGenerator is properly configured. +func (r *RelayAddressGenerator) Validate() error { + if r.ListenerName == "" { + return errInvalidName + } + + if r.Net == nil { + r.Net, _ = stdnet.NewNet() + } + + switch { + case r.MinRelayPort == 0: + return errInvalidRelayPortRange + case r.MaxRelayPort == 0: + return errInvalidRelayPortRange + case r.MinRelayPort > r.MaxRelayPort: + return errInvalidRelayPortRange + case r.RelayAddress == nil: + return errRelayAddressInvalid + case r.Address == "": + return errListeningAddressInvalid + default: + return nil + } +} + +// AllocatePacketConn generates a new transport relay connection and returns the IP/Port to be +// returned to the client in the allocation response. +func (r *RelayAddressGenerator) AllocatePacketConn(network string, requestedPort int) (net.PacketConn, net.Addr, error) { + if requestedPort <= 1 || requestedPort > 2<<16-1 { + // Let the OS to automatically assign a port + requestedPort = 0 + } + + conn, err := r.Net.ListenPacket(network, fmt.Sprintf("%s:%d", r.Address, requestedPort)) + if err != nil { + return nil, nil, err + } + + conn = NewPortRangePacketConn(conn, r.ListenerName, r.MinRelayPort, r.MaxRelayPort, + r.Logger.NewLogger(fmt.Sprintf("relay-%s", r.ListenerName))) + + // Decorate with a telemetry reporter. + conn = telemetry.NewPacketConn(conn, r.ListenerName, telemetry.ClusterType) + + relayAddr, ok := conn.LocalAddr().(*net.UDPAddr) + if !ok { + return nil, nil, errNilConn + } + + relayAddr.IP = r.RelayAddress + return conn, relayAddr, nil +} + +// AllocateConn generates a new Conn to receive traffic on and the IP/Port to populate the +// allocation response with +func (g *RelayAddressGenerator) AllocateConn(network string, requestedPort int) (net.Conn, net.Addr, error) { + return nil, nil, errTodo +} diff --git a/pkg/apis/v1/admin.go b/pkg/apis/v1/admin.go index 0ee9d626..808b99cc 100644 --- a/pkg/apis/v1/admin.go +++ b/pkg/apis/v1/admin.go @@ -9,7 +9,7 @@ import ( // AdminConfig holds the administrative configuration. type AdminConfig struct { - // Name is the name of the server, optional. + // Name of the server. Default is "default-stunnerd". Name string `json:"name,omitempty"` // LogLevel is the desired log verbosity, e.g.: "stunner:TRACE,all:INFO". Default is // "all:INFO". @@ -21,15 +21,14 @@ type AdminConfig struct { // HealthCheckEndpoint is the URI of the form `http://address:port` exposed for external // HTTP health-checking. A liveness probe responder will be exposed on path `/live` and // readiness probe on path `/ready`. The scheme (`http://`) is mandatory, and if no port is - // specified then the default port is 8086. If pointer value nil then the default is to - // enable health-checking at `http://0.0.0.0:8086`, set to a pointer to an enpty string if - // you want to disable health-checking. + // specified then the default port is 8086. If ignored, then the default is to enable + // health-checking at `http://0.0.0.0:8086`. Set to a pointer to an empty string to disable + // health-checking. HealthCheckEndpoint *string `json:"healthcheck_endpoint,omitempty"` } // Validate checks a configuration and injects defaults. func (req *AdminConfig) Validate() error { - //FIXME: no validation for loglevel (we'd need to create a new logger and it's not worth) if req.LogLevel == "" { req.LogLevel = DefaultLogLevel } @@ -39,7 +38,7 @@ func (req *AdminConfig) Validate() error { } if req.MetricsEndpoint != "" { - //metrics endpoint set: validate. empty string is valid + //Metrics endpoint set: validate. The empty string is valid if _, err := url.Parse(req.MetricsEndpoint); err != nil { return fmt.Errorf("invalid metric server endpoint URL %s: %s", req.MetricsEndpoint, err.Error()) @@ -47,11 +46,11 @@ func (req *AdminConfig) Validate() error { } if req.HealthCheckEndpoint == nil { - // no healtchcheck endpoint given: use default URL + // No healtchcheck endpoint given: use default URL e := fmt.Sprintf("http://:%d", DefaultHealthCheckPort) req.HealthCheckEndpoint = &e } else { - //healtcheck endpoint set: validate. empty string is valid + // Healtcheck endpoint set: validate. Empty string is valid if _, err := url.Parse(*req.HealthCheckEndpoint); err != nil { return fmt.Errorf("invalid health-check server endpoint URL %s: %s", *req.HealthCheckEndpoint, err.Error()) @@ -63,7 +62,7 @@ func (req *AdminConfig) Validate() error { // Name returns the name of the object to be configured. func (req *AdminConfig) ConfigName() string { - // singleton! + // Singleton! return DefaultAdminName } @@ -75,7 +74,6 @@ func (req *AdminConfig) DeepEqual(other Config) bool { // DeepCopyInto copies a configuration. func (req *AdminConfig) DeepCopyInto(dst Config) { ret := dst.(*AdminConfig) - // admin conf contians primitive types only so this is safe *ret = *req } diff --git a/pkg/apis/v1/auth.go b/pkg/apis/v1/auth.go index fcab3ee6..c3173098 100644 --- a/pkg/apis/v1/auth.go +++ b/pkg/apis/v1/auth.go @@ -6,15 +6,17 @@ import ( "strings" ) -// Auth defines the specification of the STUN/TURN authentication mechanism used by STUNner. +// Auth specifies the STUN/TURN authentication mechanism used by STUNner. type AuthConfig struct { - // Type is the type of the STUN/TURN authentication mechanism ("plaintext" or "longterm"). + // Type of the STUN/TURN authentication mechanism ("static" or "ephemeral"). The deprecated + // type name "plaintext" is accepted for "static" and the deprecated type name "longterm" + // is accepted for "ephemeral" for compatibility with older versions. Type string `json:"type,omitempty"` // Realm defines the STUN/TURN authentication realm. Realm string `json:"realm,omitempty"` - // Credentials specifies the authententication credentials: for "plaintext" at least the - // keys "username" and "password" must be set, for "longterm" the key "secret" will hold - // the shared authentication secret. + // Credentials specifies the authententication credentials: for "static" at least the keys + // "username" and "password" must be set, for "ephemeral" the key "secret" specifying the + // shared authentication secret must be set. Credentials map[string]string `json:"credentials"` } @@ -24,11 +26,12 @@ func (req *AuthConfig) Validate() error { req.Type = DefaultAuthType } + // Normalize atype, err := NewAuthType(req.Type) if err != nil { return err } - req.Type = atype.String() // normalize + req.Type = atype.String() switch atype { case AuthTypeStatic: @@ -57,7 +60,7 @@ func (req *AuthConfig) Validate() error { // Name returns the name of the object to be configured. func (req *AuthConfig) ConfigName() string { - // singleton! + // Singleton! return DefaultAuthName } diff --git a/pkg/apis/v1/cluster.go b/pkg/apis/v1/cluster.go index 3c4ff9d5..b7c4d08e 100644 --- a/pkg/apis/v1/cluster.go +++ b/pkg/apis/v1/cluster.go @@ -7,17 +7,18 @@ import ( "strings" ) -// ClusterConfig specifies a set of upstream peers STUNner can open transport relay connections -// to. There are two address resolution policies. In STATIC clusters the allowed peer IP addresses -// are explicitly listed in the endpoint list. In STRICT_DNS clusters the endpoints are assumed to -// be proper DNS domain names. STUNner will resolve each domain name in the background and admits a -// new connection only if the peer address matches one of the IP addresses returned by the DNS -// resolver for one of the endpoints. STRICT_DNS clusters are best used with headless Kubernetes -// services. +// ClusterConfig specifies a set of upstream peers to which STUNner can open transport relay +// connections. There are two address resolution policies. In STATIC clusters the allowed peer IP +// addresses are explicitly listed in the endpoint list. In STRICT_DNS clusters the endpoints are +// assumed to be proper DNS domain names: STUNner will resolve each domain name in the background +// and admit a new connection only if the peer address matches one of the IP addresses returned by +// the DNS resolver for one of the endpoints. STRICT_DNS clusters are best used with headless +// Kubernetes services. type ClusterConfig struct { - // Name is the name of the cluster. + // Name of the cluster. Name is mandatory. Name string `json:"name"` - // Type specifies the cluster address resolution policy, either STATIC or STRICT_DNS. + // Type specifies the cluster address resolution policy, either STATIC or + // STRICT_DNS. Default is "STATIC". Type string `json:"type,omitempty"` // Protocol specifies the protocol to be used with the cluster, either UDP (default) or TCP // (not implemented yet). @@ -32,6 +33,7 @@ func (req *ClusterConfig) Validate() error { return fmt.Errorf("missing name in cluster configuration: %s", req.String()) } + // Normalize if req.Type == "" { req.Type = DefaultClusterType } @@ -39,8 +41,9 @@ func (req *ClusterConfig) Validate() error { if err != nil { return err } - req.Type = t.String() // normalize + req.Type = t.String() + // Normalize if req.Protocol == "" { req.Protocol = DefaultClusterProtocol } @@ -48,7 +51,7 @@ func (req *ClusterConfig) Validate() error { if err != nil { return err } - req.Protocol = p.String() // normalize + req.Protocol = p.String() sort.Strings(req.Endpoints) return nil diff --git a/pkg/apis/v1/config.go b/pkg/apis/v1/config.go index 3c335c45..1d614949 100644 --- a/pkg/apis/v1/config.go +++ b/pkg/apis/v1/config.go @@ -1,7 +1,7 @@ // Package v1 is the v1 version of the STUNner API. package v1 -// Config is the main interface for STUNner configuration objects +// Config is the main interface for STUNner configuration objects. type Config interface { // Validate checks a configuration and injects defaults. Validate() error diff --git a/pkg/apis/v1/listener.go b/pkg/apis/v1/listener.go index c4d19a4f..d39fbc83 100644 --- a/pkg/apis/v1/listener.go +++ b/pkg/apis/v1/listener.go @@ -9,26 +9,30 @@ import ( // ListenerConfig specifies a server socket on which STUN/TURN connections will be served. type ListenerConfig struct { - // Name is the name of the listener. + // Name of the listener. Name string `json:"name,omitempty"` - // Protocol is the transport protocol used by the listener ("UDP", "TCP", "TLS", - // "DTLS"). The application-layer protocol on top of the transport protocol is always - // STUN/TURN. + // Protocol is the transport protocol ("UDP", "TCP", "TLS", "DTLS") or the complete L4/L7 + // protocol stack ("TURN-UDP", "TURN-TCP", "TURN-TLS", "TURN-DTLS") used by the listener. + // The application-layer protocol on top of the transport protocol is always TURN, so "UDP" + // and "TURN-UDP" are equivalent (and so on for the other protocols). Default is + // "TURN-UDP". Protocol string `json:"protocol,omitempty"` // PublicAddr is the Internet-facing public IP address for the listener (ignored by // STUNner). PublicAddr string `json:"public_address,omitempty"` // PublicPort is the Internet-facing public port for the listener (ignored by STUNner). PublicPort int `json:"public_port,omitempty"` - // Addr is the IP address for the listener. + // Addr is the IP address for the listener. Default is localhost. Addr string `json:"address,omitempty"` - // Port is the port for the listener. + // Port is the port for the listener. Default is the standard TURN port (3478). Port int `json:"port,omitempty"` - // MinRelayPort is the smallest relay port assigned for the relay connections spawned by - // the listener. + // MinRelayPort is the lowest peer target port admitted on the listener (inclusive). The + // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport + // relay connections created via the listener. Default is 1. MinRelayPort int `json:"min_relay_port,omitempty"` - // MaxRelayPort is the highest relay port assigned for the relay connections spawned by the - // listener. + // MaxRelayPort is the highest peer target port admitted on the listener (inclusive). The + // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport + // relay connections created via the listener. Default is 65535. MaxRelayPort int `json:"max_relay_port,omitempty"` // Cert is the base64-encoded TLS cert. Cert string `json:"cert,omitempty"` @@ -44,6 +48,7 @@ func (req *ListenerConfig) Validate() error { return fmt.Errorf("missing name in listener configuration: %s", req.String()) } + // Normalize if req.Protocol == "" { req.Protocol = DefaultProtocol } @@ -51,7 +56,7 @@ func (req *ListenerConfig) Validate() error { if err != nil { return err } - req.Protocol = proto.String() // normalize + req.Protocol = proto.String() if req.Addr == "" { req.Addr = "0.0.0.0" @@ -71,6 +76,10 @@ func (req *ListenerConfig) Validate() error { return fmt.Errorf("invalid port: %d", p) } } + if req.MinRelayPort > req.MaxRelayPort { + return fmt.Errorf("invalid relay port range: min port (%d) <= max port (%d) must hold", + req.MinRelayPort, req.MaxRelayPort) + } if proto == ListenerProtocolTURNTLS || proto == ListenerProtocolTURNDTLS || proto == ListenerProtocolTLS || proto == ListenerProtocolDTLS { @@ -91,9 +100,8 @@ func (req *ListenerConfig) ConfigName() string { return req.Name } -// DeepEqual compares two configurations. +// DeepEqual compares two configurations. Routes must be sorted in both configs! func (req *ListenerConfig) DeepEqual(other Config) bool { - // routes must be sorted in both configs! return reflect.DeepEqual(req, other) } @@ -145,7 +153,8 @@ func (req *ListenerConfig) String() string { return fmt.Sprintf("%q:{%s}", n, strings.Join(status, ",")) } -// GetListenerURI is a helper that can output two types of Listener URIs: one with :// after the scheme or one with only : (as per RFC7065). +// GetListenerURI is a helper that can output two types of Listener URIs: one with "://" after the +// scheme or one with only ":" (as per RFC7065). func (req *ListenerConfig) GetListenerURI(rfc7065 bool) (string, error) { proto, err := NewListenerProtocol(req.Protocol) if err != nil { @@ -170,13 +179,13 @@ func (req *ListenerConfig) GetListenerURI(rfc7065 bool) (string, error) { addr := req.PublicAddr if addr == "" { - // fallback to server addr + // Fallback to server addr addr = req.Addr } port := req.PublicPort if port == 0 { - // fallback to server addr + // Fallback to server addr port = req.Port } diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go index 31959b81..84414b35 100644 --- a/pkg/apis/v1/stunner.go +++ b/pkg/apis/v1/stunner.go @@ -6,9 +6,9 @@ import ( "strings" ) -// StunnerConfig specifies the configuration of the the STUnner daemon. +// StunnerConfig specifies the configuration for the STUnner daemon. type StunnerConfig struct { - // ApiVersion is the version of the STUNner API implemented. + // ApiVersion is the version of the STUNner API implemented. Must be set to "v1". ApiVersion string `json:"version"` // AdminConfig holds administrative configuration. Admin AdminConfig `json:"admin,omitempty"` @@ -28,29 +28,21 @@ func (req *StunnerConfig) Validate() error { return fmt.Errorf("unsupported API version: %q", req.ApiVersion) } - // validate admin if err := req.Admin.Validate(); err != nil { return err } - // validate auth if err := req.Auth.Validate(); err != nil { return err } - // validate listeners for i, l := range req.Listeners { if err := l.Validate(); err != nil { return err } req.Listeners[i] = l } - // // listeners are sorted by name - // sort.Slice(req.Listeners, func(i, j int) bool { - // return req.Listeners[i].Name < req.Listeners[j].Name - // }) - // validate clusters for i, c := range req.Clusters { if err := c.Validate(); err != nil { return err @@ -58,11 +50,6 @@ func (req *StunnerConfig) Validate() error { req.Clusters[i] = c } - // // clusters are sorted by name - // sort.Slice(req.Clusters, func(i, j int) bool { - // return req.Clusters[i].Name < req.Clusters[j].Name - // }) - return nil } diff --git a/pkg/apis/v1/util.go b/pkg/apis/v1/util.go index cd10cae1..0cafe3b3 100644 --- a/pkg/apis/v1/util.go +++ b/pkg/apis/v1/util.go @@ -5,7 +5,7 @@ import ( "strings" ) -// AuthType species the type of the STUN/TURN authentication mechanism used by STUNner +// AuthType species the type of the STUN/TURN authentication mechanism used by STUNner. type AuthType int const ( @@ -23,7 +23,7 @@ const ( authTypeLongTermStr = "longterm" ) -// NewAuthType parses the authentication mechanism specification +// NewAuthType parses the authentication mechanism specification. func NewAuthType(raw string) (AuthType, error) { switch raw { case authTypeStaticStr, authTypePlainTextStr: @@ -35,7 +35,7 @@ func NewAuthType(raw string) (AuthType, error) { } } -// String returns a string representation for the authentication mechanism +// String returns a string representation for the authentication mechanism. func (a AuthType) String() string { switch a { case AuthTypeStatic: @@ -47,7 +47,7 @@ func (a AuthType) String() string { } } -// ListenerProtocol specifies the network protocol for a listener +// ListenerProtocol specifies the network protocol for a listener. type ListenerProtocol int const ( @@ -73,7 +73,7 @@ const ( listenerProtocolTURNDTLSStr = "TURN-DTLS" ) -// NewListenerProtocol parses the protocol specification +// NewListenerProtocol parses the protocol specification. func NewListenerProtocol(raw string) (ListenerProtocol, error) { switch strings.ToUpper(raw) { case listenerProtocolUDPStr: @@ -98,7 +98,7 @@ func NewListenerProtocol(raw string) (ListenerProtocol, error) { } } -// String returns a string representation of a listener protocol +// String returns a string representation of a listener protocol. func (l ListenerProtocol) String() string { switch l { case ListenerProtocolUDP: @@ -122,7 +122,7 @@ func (l ListenerProtocol) String() string { } } -// ClusterType specifies the cluster address resolution policy +// ClusterType specifies the cluster address resolution policy. type ClusterType int const ( @@ -158,7 +158,7 @@ func (l ClusterType) String() string { } } -// ClusterProtocol specifies the network protocol for a cluster +// ClusterProtocol specifies the network protocol for a cluster. type ClusterProtocol int const ( @@ -172,7 +172,7 @@ const ( clusterProtocolTCPStr = "TCP" ) -// NewClusterProtocol parses the protocol specification +// NewClusterProtocol parses the protocol specification. func NewClusterProtocol(raw string) (ClusterProtocol, error) { switch strings.ToUpper(raw) { case clusterProtocolUDPStr: @@ -185,7 +185,7 @@ func NewClusterProtocol(raw string) (ClusterProtocol, error) { } } -// String returns a string representation of a cluster protocol +// String returns a string representation of a cluster protocol. func (p ClusterProtocol) String() string { switch p { case ClusterProtocolUDP: diff --git a/pkg/logger/logger.go b/pkg/logger/logger.go index 1f326b7b..418c6297 100644 --- a/pkg/logger/logger.go +++ b/pkg/logger/logger.go @@ -1,18 +1,22 @@ package logger import ( + "bytes" "fmt" "io" "log" "os" "strings" - "time" "github.com/pion/logging" "golang.org/x/time/rate" ) -const defaultFlags = log.Lmicroseconds | log.Lshortfile | log.Lmsgprefix +const ( + defaultFlags = log.Lmicroseconds | log.Lshortfile | log.Lmsgprefix + DefaultRateLimit = rate.Limit(.25) + DefaultBurstSize = 1 +) var logLevels = map[string]logging.LogLevel{ "DISABLE": logging.LogLevelDisabled, @@ -23,24 +27,31 @@ var logLevels = map[string]logging.LogLevel{ "TRACE": logging.LogLevelTrace, } -// LoggerFactory defines levels by scopes and creates new LeveledLogger. -type LoggerFactory struct { +// LoggerFactory is the basic pion LoggerFactory interface extended with functions for setting and querying the loglevel per scope. +type LoggerFactory interface { + logging.LoggerFactory + // SetLevel sets the global loglevel. + SetLevel(levelSpec string) + // GetLevel gets the loglevel for the given scope. + GetLevel(scope string) string +} + +// LeveledLoggerFactory defines levels by scopes and creates new LeveledLoggers that can dynamically change their own loglevels. +type LeveledLoggerFactory struct { Writer io.Writer DefaultLogLevel logging.LogLevel ScopeLevels map[string]logging.LogLevel - Loggers map[string]*logging.DefaultLeveledLogger + Loggers map[string]*RateLimitedLogger } // NewLoggerFactory sets up a scoped logger for STUNner. -func NewLoggerFactory(levelSpec string) *LoggerFactory { - logger := LoggerFactory{} +func NewLoggerFactory(levelSpec string) *LeveledLoggerFactory { + logger := LeveledLoggerFactory{} logger.DefaultLogLevel = logging.LogLevelError logger.ScopeLevels = make(map[string]logging.LogLevel) + logger.Loggers = make(map[string]*RateLimitedLogger) logger.Writer = os.Stdout - logger.ScopeLevels = make(map[string]logging.LogLevel) - logger.Loggers = make(map[string]*logging.DefaultLeveledLogger) - // resets all child loggers logger.SetLevel(levelSpec) @@ -48,129 +59,206 @@ func NewLoggerFactory(levelSpec string) *LoggerFactory { } // NewLogger either returns the existing LeveledLogger (if it exists) for the given scope or creates a new one. -func (f *LoggerFactory) NewLogger(scope string) logging.LeveledLogger { - logger, found := f.Loggers[scope] - if found { - return logger +func (f *LeveledLoggerFactory) NewLogger(scope string) logging.LeveledLogger { + logger := f.newLogger(scope, DefaultRateLimit, DefaultBurstSize) + logger.DisableRateLimiter() + return logger +} + +// SetLevel sets the loglevel. +func (f *LeveledLoggerFactory) SetLevel(levelSpec string) { + levels := strings.Split(levelSpec, ",") + for _, s := range levels { + scopedLevel := strings.SplitN(s, ":", 2) + if len(scopedLevel) != 2 { + continue + } + scope := scopedLevel[0] + level := scopedLevel[1] + + l, ok := logLevels[strings.ToUpper(level)] + if !ok { + continue + } + + if strings.ToLower(scope) == "all" { + for c := range f.Loggers { + f.ScopeLevels[c] = l + } + f.DefaultLogLevel = l + continue + } + + f.ScopeLevels[scope] = l + } + + for scope, logger := range f.Loggers { + l, found := f.ScopeLevels[scope] + if !found { + l = f.DefaultLogLevel + } + + logger.SetLevel(l) + + // disable rate-limiting at DEBUG and TRACE level + if l == logging.LogLevelDebug || l == logging.LogLevelTrace { + logger.DisableRateLimiter() + } } +} +// GetLevel gets the loglevel for the given scope. +func (f *LeveledLoggerFactory) GetLevel(scope string) string { logLevel := f.DefaultLogLevel scopeLevel, found := f.ScopeLevels[scope] if found { logLevel = scopeLevel } - l := logging.NewDefaultLeveledLoggerForScope(scope, logLevel, f.Writer). - WithTraceLogger(log.New(f.Writer, fmt.Sprintf("%s TRACE: ", scope), defaultFlags)). - WithDebugLogger(log.New(f.Writer, fmt.Sprintf("%s DEBUG: ", scope), defaultFlags)). - WithInfoLogger(log.New(f.Writer, fmt.Sprintf("%s INFO: ", scope), defaultFlags)). - WithWarnLogger(log.New(f.Writer, fmt.Sprintf("%s WARNING: ", scope), defaultFlags)). - WithErrorLogger(log.New(f.Writer, fmt.Sprintf("%s ERROR: ", scope), defaultFlags)) + return logLevel.String() +} - f.Loggers[scope] = l +// RateLimitedLoggerFactory is a logger factory that can emit rate-limited loggers. Note that all +// loglevels are rate-limited via single token bucket. Rate-limiting only applies at high loglevels +// (ERROR, WARN and INFO), a logger set to alower loglevel (DEBUG and TRACE) is never rate-limited +// to ease debugging. +type RateLimitedLoggerFactory struct { + *LeveledLoggerFactory + Limit rate.Limit + Burst int +} - return l +// WithRateLimiter decorates a logger factory with a rate-limiter. All loggers emitted by the +// factory will be automatically rate-limited. +func (f *LeveledLoggerFactory) WithRateLimiter(limit rate.Limit, burst int) *RateLimitedLoggerFactory { + return &RateLimitedLoggerFactory{ + LeveledLoggerFactory: f, + Limit: limit, + Burst: burst, + } } -// NewRateLimitedLogger creates a new rate-limited logger from a factory. Every loglevel is independently rate-limited with a token bucket ofthe given period and burst size. If addSuppressed is true and then the number of suppressed events is added to the output (provided that there were suppressed events). -func (f *LoggerFactory) NewRateLimitedLogger(scope string, period time.Duration, burst int, addSuppressed bool) logging.LeveledLogger { - logger, found := f.Loggers[scope] +// NewLogger either returns the existing LeveledLogger (if it exists) for the given scope or creates a new one. +func (f *RateLimitedLoggerFactory) NewLogger(scope string) logging.LeveledLogger { + logger := f.LeveledLoggerFactory.newLogger(scope, f.Limit, f.Burst) + + // disable rate-limiting logging at lower loglevels + l := f.DefaultLogLevel + + // this will never happen: loggers are always created with the default loglevel + scopeLevel, found := f.ScopeLevels[scope] if found { - return logger + l = scopeLevel } - logLevel := f.DefaultLogLevel - scopeLevel, found := f.ScopeLevels[scope] + // disable rate-limiting at DEBUG and TRACE level + if l == logging.LogLevelDebug || l == logging.LogLevelTrace { + logger.DisableRateLimiter() + } else { + logger.EnableRateLimiter() + } + + return logger +} + +// RateLimitedLogger is a rate-limiter logger for a specific scope. +type RateLimitedLogger struct { + *logging.DefaultLeveledLogger + *RateLimitedWriter +} + +// NewRateLimitedLoggerForScope returns a LeveledLogger configured with a default rate limiter. +func NewRateLimitedLoggerForScope(scope string, level logging.LogLevel, writer io.Writer, limit rate.Limit, burst int) *RateLimitedLogger { + // NewLogger will set the limit and burst + w := NewRateLimitedWriter(writer, limit, burst, true) + return &RateLimitedLogger{ + DefaultLeveledLogger: logging.NewDefaultLeveledLoggerForScope(scope, level, writer), + RateLimitedWriter: w, + } +} + +// newLogger knows how to emit rate-limited loggers. +func (f *LeveledLoggerFactory) newLogger(scope string, limit rate.Limit, burst int) *RateLimitedLogger { + logger, found := f.Loggers[scope] if found { - logLevel = scopeLevel + return logger } - l := logging.NewDefaultLeveledLoggerForScope(scope, logLevel, f.Writer). - WithTraceLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s TRACE: ", scope), defaultFlags)). - WithDebugLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s DEBUG: ", scope), defaultFlags)). - WithInfoLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s INFO: ", scope), defaultFlags)). - WithWarnLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s WARNING: ", scope), defaultFlags)). - WithErrorLogger(log.New(NewRateLimitedWriter(f.Writer, period, burst, addSuppressed), fmt.Sprintf("%s ERROR: ", scope), defaultFlags)) + l := NewRateLimitedLoggerForScope(scope, f.DefaultLogLevel, f.Writer, limit, burst) + + l.DefaultLeveledLogger. + WithTraceLogger(log.New(l.RateLimitedWriter, fmt.Sprintf("%s TRACE: ", scope), defaultFlags)). + WithDebugLogger(log.New(l.RateLimitedWriter, fmt.Sprintf("%s DEBUG: ", scope), defaultFlags)). + WithInfoLogger(log.New(l.RateLimitedWriter, fmt.Sprintf("%s INFO: ", scope), defaultFlags)). + WithWarnLogger(log.New(l.RateLimitedWriter, fmt.Sprintf("%s WARNING: ", scope), defaultFlags)). + WithErrorLogger(log.New(l.RateLimitedWriter, fmt.Sprintf("%s ERROR: ", scope), defaultFlags)) f.Loggers[scope] = l return l } -type rateLimitedWriter struct { +// RateLimitedWriter is a writer limited by a token bucket. +type RateLimitedWriter struct { io.Writer - addSuppressed bool - counter int - limiter *rate.Limiter + *RateLimiter + Counter int + AddSuppressed bool } -// NewRateLimitedWriter is a writer limited by a token bucket rate-limiter to at most burst events per period. If addSuppressed is true then the number of suppressed events is appended to the output. -func NewRateLimitedWriter(writer io.Writer, period time.Duration, burst int, addSuppressed bool) io.Writer { - return &rateLimitedWriter{ +// NewRateLimitedWriter creates a writer rate-limited by a token bucket to at most limit events per +// second with the given burst size. If addSuppressed is true then the number of events suppressed +// between logged events is appended to the output. +func NewRateLimitedWriter(writer io.Writer, limit rate.Limit, burst int, addSuppressed bool) *RateLimitedWriter { + return &RateLimitedWriter{ Writer: writer, - addSuppressed: addSuppressed, - counter: 0, - limiter: rate.NewLimiter(rate.Every(period), burst), + RateLimiter: NewRateLimiter(limit, burst), + Counter: 0, // no need to lock: we are being called under a lock from DefaultLeveledLogger + AddSuppressed: addSuppressed, } } -func (w *rateLimitedWriter) Write(p []byte) (int, error) { - if !w.limiter.Allow() { - w.counter++ +// Write fulfills io.Writer. +func (w *RateLimitedWriter) Write(p []byte) (int, error) { + if !w.RateLimiter.Allow() { + w.Counter++ return 0, nil } - if w.addSuppressed && w.counter > 0 { - s := []byte(fmt.Sprintf(" (suppressed %d log events)", w.counter)) - p = append(p, s...) + + if w.AddSuppressed && w.Counter > 0 { + suffix := fmt.Sprintf(" (suppressed %d log events)\n", w.Counter) + p = append(bytes.TrimRight(p, "\r\n"), suffix...) } n, err := w.Writer.Write(p) - w.counter = 0 + w.Counter = 0 return n, err } -// SetLevel sets the loglevel. -func (f *LoggerFactory) SetLevel(levelSpec string) { - levels := strings.Split(levelSpec, ",") - for _, s := range levels { - scopedLevel := strings.SplitN(s, ":", 2) - if len(scopedLevel) != 2 { - continue - } - scope := scopedLevel[0] - level := scopedLevel[1] - - // set log-level - l, found := logLevels[strings.ToUpper(level)] - if !found { - continue - } - - if strings.ToLower(scope) == "all" { - f.DefaultLogLevel = l - continue - } +// RateLimiter is a token bucket that can be disabled. +type RateLimiter struct { + *rate.Limiter + EnableRateLimiterd bool +} - f.ScopeLevels[scope] = l +func NewRateLimiter(r rate.Limit, b int) *RateLimiter { + return &RateLimiter{ + Limiter: rate.NewLimiter(r, b), + EnableRateLimiterd: false, } +} - for scope, logger := range f.Loggers { - l, found := f.ScopeLevels[scope] - if !found { - l = f.DefaultLogLevel - } +func (l *RateLimiter) EnableRateLimiter() { + l.EnableRateLimiterd = true +} - logger.SetLevel(l) - } +func (l *RateLimiter) DisableRateLimiter() { + l.EnableRateLimiterd = false } -// GetLevel gets the loglevel for the given scope. -func (f *LoggerFactory) GetLevel(scope string) string { - logLevel := f.DefaultLogLevel - scopeLevel, found := f.ScopeLevels[scope] - if found { - logLevel = scopeLevel +func (l *RateLimiter) Allow() bool { + if !l.EnableRateLimiterd { + return true } - - return logLevel.String() + return l.Limiter.Allow() } diff --git a/pkg/logger/logger_test.go b/pkg/logger/logger_test.go index 5389fe27..12282f86 100644 --- a/pkg/logger/logger_test.go +++ b/pkg/logger/logger_test.go @@ -8,14 +8,17 @@ import ( "github.com/pion/transport/v3/test" "github.com/stretchr/testify/assert" + "golang.org/x/time/rate" ) const testScope = "dummy-scope" +var logBuffer = &bytes.Buffer{} + type loggerTestCase struct { name, defaultLogLevel, scopeLogLevel string - prep func(lf *LoggerFactory) - tester func(t *testing.T, lf *LoggerFactory) + prep func(lf LoggerFactory) + tester func(t *testing.T, lf LoggerFactory) } var loggerTests = []loggerTestCase{ @@ -23,356 +26,303 @@ var loggerTests = []loggerTestCase{ name: "default-loglevel", defaultLogLevel: "", // default is ERROR scopeLogLevel: "", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") + log := lf.NewLogger(testScope) + level = lf.GetLevel(testScope) assert.Equal(t, "Error", level, "dummy scope: level") - // resuse logger - ld := lf.NewLogger(testScope) - - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + log.Warn("dummy") + assert.Zerof(t, loglenr(), "WARN for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "default-loglevel-disable-scope", defaultLogLevel: "", // default is ERROR scopeLogLevel: "DISABLE", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Disabled", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "ERROR for level %s", level) + log.Error("dummy") + assert.Zerof(t, loglenr(), "ERROR for level %s", level) - ld.Warn("dummy") - assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + log.Warn("dummy") + assert.Zerof(t, loglenr(), "WARN for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "default-loglevel-error-scope", defaultLogLevel: "", // default is ERROR scopeLogLevel: "ERROR", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Error", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + log.Warn("dummy") + assert.Zerof(t, loglenr(), "WARN for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "default-loglevel-warn-scope", defaultLogLevel: "", // default is ERROR scopeLogLevel: "WARN", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Warn", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + log.Warn("dummy") + assert.Containsf(t, logreadr(), "dummy", "WARN for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "default-loglevel-info-scope", defaultLogLevel: "", // default is ERROR scopeLogLevel: "INFO", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Info", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + log.Warn("dummy") + assert.Containsf(t, logreadr(), "dummy", "WARN for level %s", level) - ld.Info("dummy") - assert.Containsf(t, lf.readr(), "dummy", "INFO for level %s", level) + log.Info("dummy") + assert.Containsf(t, logreadr(), "dummy", "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "default-loglevel-debug-scope", defaultLogLevel: "", // default is ERROR scopeLogLevel: "DEBUG", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Debug", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + log.Warn("dummy") + assert.Containsf(t, logreadr(), "dummy", "WARN for level %s", level) - ld.Info("dummy") - assert.Containsf(t, lf.readr(), "dummy", "INFO for level %s", level) + log.Info("dummy") + assert.Containsf(t, logreadr(), "dummy", "INFO for level %s", level) - ld.Debug("dummy") - assert.Containsf(t, lf.readr(), "dummy", "DEBUG for level %s", level) + log.Debug("dummy") + assert.Containsf(t, logreadr(), "dummy", "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "default-loglevel-trace-scope", defaultLogLevel: "", // default is ERROR scopeLogLevel: "TRACE", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Trace", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Containsf(t, lf.readr(), "dummy", "WARN for level %s", level) + log.Warn("dummy") + assert.Containsf(t, logreadr(), "dummy", "WARN for level %s", level) - ld.Info("dummy") - assert.Containsf(t, lf.readr(), "dummy", "INFO for level %s", level) + log.Info("dummy") + assert.Containsf(t, logreadr(), "dummy", "INFO for level %s", level) - ld.Debug("dummy") - assert.Containsf(t, lf.readr(), "dummy", "DEBUG for level %s", level) + log.Debug("dummy") + assert.Containsf(t, logreadr(), "dummy", "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Containsf(t, lf.readr(), "dummy", "TRACE for level %s", level) + log.Trace("dummy") + assert.Containsf(t, logreadr(), "dummy", "TRACE for level %s", level) }, }, { name: "override-loglevel-trace-scope", defaultLogLevel: "all:TRACE", scopeLogLevel: "ERROR", - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Trace", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Error", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + log.Warn("dummy") + assert.Zerof(t, loglenr(), "WARN for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "complex-loglevel-1", defaultLogLevel: "all:TRACE", scopeLogLevel: "TRACE", - prep: func(lf *LoggerFactory) { + prep: func(lf LoggerFactory) { lf.SetLevel("all:TRACE,dummy-scope:ERROR") }, - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") assert.Equal(t, "Trace", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Error", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + log := lf.NewLogger(testScope) - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Warn("dummy") - assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + log.Warn("dummy") + assert.Zerof(t, loglenr(), "WARN for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, { name: "complex-loglevel-2", defaultLogLevel: "all:TRACE", scopeLogLevel: "TRACE", - prep: func(lf *LoggerFactory) { - lf.SetLevel("dummy-scope:DEBUG,nonExistentScope:TRACE,dummy-scope:ERROR,all:TRACE") + prep: func(lf LoggerFactory) { + lf.SetLevel("dummy-scope:DEBUG,nonExistentScope:TRACE,dummy-scope:ERROR,all:error,some-other-scope:TRACE") }, - tester: func(t *testing.T, lf *LoggerFactory) { + tester: func(t *testing.T, lf LoggerFactory) { level := lf.GetLevel("all") - assert.Equal(t, "Trace", level, "default scope: level") + assert.Equal(t, "Error", level, "default scope: level") level = lf.GetLevel(testScope) assert.Equal(t, "Error", level, "dummy scope: level") - // reuse logger - ld := lf.NewLogger(testScope) + level = lf.GetLevel("some-other-scope") + assert.Equal(t, "Trace", level, "other scope: level") - ld.Error("dummy") - assert.Containsf(t, lf.readr(), "dummy", "ERROR for level %s", level) + log := lf.NewLogger(testScope) - ld.Warn("dummy") - assert.Zerof(t, lf.lenr(), "WARN for level %s", level) + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "INFO for level %s", level) + log.Warn("dummy") + assert.Zerof(t, loglenr(), "WARN for level %s", level) - ld.Debug("dummy") - assert.Zerof(t, lf.lenr(), "DEBUG for level %s", level) + log.Info("dummy") + assert.Zerof(t, loglenr(), "INFO for level %s", level) - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "TRACE for level %s", level) + log.Debug("dummy") + assert.Zerof(t, loglenr(), "DEBUG for level %s", level) + + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, }, } -//nolint:golint,unused -func (lf *LoggerFactory) len() int { - b, ok := lf.Writer.(*bytes.Buffer) - if !ok { - panic("not a test logger factory") - } - return b.Len() -} - -func (lf *LoggerFactory) lenr() int { - b, ok := lf.Writer.(*bytes.Buffer) - if !ok { - panic("not a test logger factory") - } - l := b.Len() - b.Reset() - return l -} - -//nolint:golint,unused -func (lf *LoggerFactory) reset() { - b, ok := lf.Writer.(*bytes.Buffer) - if !ok { - panic("not a test logger factory") - } - b.Reset() -} - -func (lf *LoggerFactory) read() string { - b, ok := lf.Writer.(*bytes.Buffer) - if !ok { - panic("not a test logger factory") - } - return b.String() -} - -func (lf *LoggerFactory) readr() string { - b, ok := lf.Writer.(*bytes.Buffer) - if !ok { - panic("not a test logger factory") - } - ret := b.String() - b.Reset() - return ret -} - func TestLogger(t *testing.T) { lim := test.TimeOut(time.Second * 60) defer lim.Stop() @@ -386,7 +336,8 @@ func TestLogger(t *testing.T) { // create loggerFactory := NewLoggerFactory(c.defaultLogLevel) - loggerFactory.Writer = &bytes.Buffer{} + loggerFactory.Writer = logBuffer + logreset() // create logger _ = loggerFactory.NewLogger(testScope) @@ -397,10 +348,6 @@ func TestLogger(t *testing.T) { c.prep(loggerFactory) } - // t.Logf("%#v", loggerFactory) - // t.Logf("%#v", loggerFactory.ScopeLevels) - // t.Logf("%#v", logger) - // test c.tester(t, loggerFactory) }) @@ -410,218 +357,149 @@ func TestLogger(t *testing.T) { // rate-limiter tests type rateLimiterLoggerTestCase struct { - name string - addSuppressed bool - period time.Duration - burst int - prep func(lf *LoggerFactory) - tester func(t *testing.T, lf *LoggerFactory) + name, level string + limit rate.Limit + burst int + prep func(lf LoggerFactory) + tester func(t *testing.T, lf LoggerFactory) } var rateLimitedLoggerTests = []rateLimiterLoggerTestCase{ { - name: "rate-limited-logger-default", - addSuppressed: true, - period: 10 * time.Millisecond, - burst: 1, - tester: func(t *testing.T, lf *LoggerFactory) { - // reuse logger - ld := lf.NewLogger(testScope) - - // only first call should succeed - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + name: "rate-limited-logger-default", + limit: 1.0, + burst: 1, + tester: func(t *testing.T, lf LoggerFactory) { + level := lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "other scope: level") - // wait until we get another token - time.Sleep(15 * time.Millisecond) + log := lf.NewLogger(testScope) - ld.Info("dummy") - assert.Contains(t, lf.read(), "dummy") - assert.Contains(t, lf.readr(), "suppressed 2 log") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + // only first call should succeed + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Error("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Error("dummy") + assert.Zerof(t, loglenr(), "suppressed") }, }, { - name: "rate-limited-logger-suppressed-shown", - addSuppressed: true, - period: 10 * time.Millisecond, - burst: 1, - tester: func(t *testing.T, lf *LoggerFactory) { - // reuse logger - ld := lf.NewLogger(testScope) + name: "rate-limited-logger-burst-2", + level: "all:INFO", + limit: 100.0, + burst: 2, + tester: func(t *testing.T, lf LoggerFactory) { + level := lf.GetLevel(testScope) + assert.Equal(t, "Info", level, "scope: level") + + log := lf.NewLogger(testScope) // first call should succeed - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") // wait until we get another token - time.Sleep(15 * time.Millisecond) - - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - // no "suppressed" message should appear - assert.NotContains(t, lf.readr(), "suppressed 2 log") - - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + time.Sleep(25 * time.Millisecond) + + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Error("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Error("dummy") + assert.Zerof(t, loglenr(), "suppressed") }, }, { - name: "rate-limited-logger-suppressed-supressed", - addSuppressed: false, - period: 10 * time.Millisecond, - burst: 1, - tester: func(t *testing.T, lf *LoggerFactory) { - // reuse logger - ld := lf.NewLogger(testScope) - - // only first call should succeed - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + name: "rate-limited-logger-burst-4", + level: "all:INFO", + limit: 100.0, + burst: 4, + tester: func(t *testing.T, lf LoggerFactory) { + level := lf.GetLevel(testScope) + assert.Equal(t, "Info", level, "scope: level") + + log := lf.NewLogger(testScope) + + // only first 4 calls should succeed + log.Info("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Info("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Info("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Info("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") // wait until we get another token time.Sleep(15 * time.Millisecond) - ld.Info("dummy") - assert.Contains(t, lf.read(), "dummy") - assert.NotContains(t, lf.readr(), "suppressed 2 log") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - }, - }, - { - name: "rate-limited-logger-burst-3", - addSuppressed: true, - period: 10 * time.Millisecond, - burst: 3, - tester: func(t *testing.T, lf *LoggerFactory) { - // reuse logger - ld := lf.NewLogger(testScope) - - // only first 3 calls should succeed - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - - // wait until we get another token - time.Sleep(13 * time.Millisecond) - - ld.Info("dummy") - assert.Contains(t, lf.read(), "dummy") - assert.Contains(t, lf.readr(), "suppressed 2 log") + log.Info("dummy") + assert.Contains(t, logread(), "dummy") + assert.Contains(t, logreadr(), "suppressed 2 log") // consumed all tokens: these should be suppressed - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") }, }, { - name: "rate-limited-logger-independent", - addSuppressed: true, - period: 10 * time.Millisecond, - burst: 1, - tester: func(t *testing.T, lf *LoggerFactory) { - // reuse logger - ld := lf.NewLogger(testScope) + name: "rate-limited-logger-global-rate-limit", + level: "all:INFO", + limit: 100.0, + burst: 1, + tester: func(t *testing.T, lf LoggerFactory) { + level := lf.GetLevel(testScope) + assert.Equal(t, "Info", level, "scope: level") + + log := lf.NewLogger(testScope) // only first call should succeed - ld.Info("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - - // rate-limiters should be independent - ld.Error("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Error("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Warn("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Debug("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Trace("dummy") + assert.Zerof(t, loglenr(), "suppressed") // wait until we get another token time.Sleep(15 * time.Millisecond) - ld.Info("dummy") - assert.Contains(t, lf.read(), "dummy") - assert.Contains(t, lf.readr(), "suppressed 2 log") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Info("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - - ld.Error("dummy") - assert.Contains(t, lf.read(), "dummy") - assert.Contains(t, lf.readr(), "suppressed 2 log") - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - }, - }, - { - name: "rate-limited-logger-inactive-loggers-not-counted", - addSuppressed: true, - period: 10 * time.Millisecond, - burst: 1, - tester: func(t *testing.T, lf *LoggerFactory) { - // reuse logger - ld := lf.NewLogger(testScope) - - // Trace is inactive: no log - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "inactive") - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - - // wait until we would get another token - time.Sleep(15 * time.Millisecond) - - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "inactive") - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Trace("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - - // increase loglevel - lf.SetLevel(fmt.Sprintf("%s:TRACE", testScope)) - ld.Error("dummy") - assert.Contains(t, lf.readr(), "dummy") - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") - ld.Error("dummy") - assert.Zerof(t, lf.lenr(), "suppressed") + log.Error("dummy") + assert.Contains(t, logreadr(), "dummy") + log.Error("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Warn("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Info("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Debug("dummy") + assert.Zerof(t, loglenr(), "suppressed") + log.Trace("dummy") + assert.Zerof(t, loglenr(), "suppressed") }, }, } @@ -638,11 +516,9 @@ func TestRateLimitedLogger(t *testing.T) { // t.Logf("-------------- Running test: %s -------------", c.name) // create - loggerFactory := NewLoggerFactory("all:INFO") - loggerFactory.Writer = &bytes.Buffer{} - - // create ratee-limiter logger - _ = loggerFactory.NewRateLimitedLogger(testScope, c.period, c.burst, c.addSuppressed) + loggerFactory := NewLoggerFactory(c.level).WithRateLimiter(c.limit, c.burst) + loggerFactory.Writer = logBuffer + logreset() // prepare if c.prep != nil { @@ -658,3 +534,29 @@ func TestRateLimitedLogger(t *testing.T) { }) } } + +//nolint:golint,unused +func loglen() int { + return logBuffer.Len() +} + +func loglenr() int { + l := logBuffer.Len() + logBuffer.Reset() + return l +} + +//nolint:golint,unused +func logreset() { + logBuffer.Reset() +} + +func logread() string { + return logBuffer.String() +} + +func logreadr() string { + ret := logBuffer.String() + logBuffer.Reset() + return ret +} diff --git a/server.go b/server.go index cb06f0dd..0cd38966 100644 --- a/server.go +++ b/server.go @@ -7,6 +7,7 @@ import ( "github.com/pion/dtls/v2" "github.com/pion/turn/v3" + "golang.org/x/time/rate" "github.com/l7mp/stunner/internal/object" "github.com/l7mp/stunner/internal/telemetry" @@ -14,6 +15,14 @@ import ( stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) +// Number of log events per second reported at ERROR, WARN and INFO loglevel (logging at DEBUG and +// TRACE levels is not rate-limited). +var LogRateLimit rate.Limit = 1.0 + +// Burst size for rate-limited logging at ERROR, WARN and INFO loglevel (logging at DEBUG and TRACE +// levels is not rate-limited). +var LogBurst = 3 + // Start will start the TURN server that belongs to a listener. func (s *Stunner) StartServer(l *object.Listener) error { s.log.Infof("listener %s (re)starting", l.String()) @@ -23,13 +32,14 @@ func (s *Stunner) StartServer(l *object.Listener) error { var lConns []turn.ListenerConfig // listen on all IPs, relay to the listener address - relay := &telemetry.RelayAddressGenerator{ - Name: l.Name, + relay := &util.RelayAddressGenerator{ + ListenerName: l.Name, RelayAddress: l.Addr, Address: "0.0.0.0", - MinPort: uint16(l.MinPort), - MaxPort: uint16(l.MaxPort), + MinRelayPort: l.MinPort, + MaxRelayPort: l.MaxPort, Net: l.Net, + Logger: s.logger, } permissionHandler := s.NewPermissionHandler(l) @@ -152,9 +162,9 @@ func (s *Stunner) StartServer(l *object.Listener) error { t, err := turn.NewServer(turn.ServerConfig{ Realm: s.GetRealm(), AuthHandler: s.NewAuthHandler(), - LoggerFactory: s.logger, PacketConnConfigs: pConns, ListenerConfigs: lConns, + LoggerFactory: s.logger.WithRateLimiter(LogRateLimit, LogBurst), }) if err != nil { return fmt.Errorf("cannot set up TURN server for listener %s: %w", diff --git a/stunner.go b/stunner.go index 6d3f7464..335f8464 100644 --- a/stunner.go +++ b/stunner.go @@ -31,7 +31,7 @@ type Stunner struct { suppressRollback, dryRun bool resolver resolver.DnsResolver udpThreadNum int - logger *logger.LoggerFactory + logger *logger.LeveledLoggerFactory log logging.LeveledLogger net transport.Net ready, shutdown bool diff --git a/stunner_test.go b/stunner_test.go index 4acc9830..181e03c6 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -154,8 +154,10 @@ func stunnerEchoTest(conf echoTestConfig) { _, err = conn.WriteTo([]byte("Hello"), echoConn.LocalAddr()) assert.NoError(t, err, err) - _, from, err2 := conn.ReadFrom(buf) + n, from, err2 := conn.ReadFrom(buf) assert.NoError(t, err2, err2) + assert.Equal(t, n, len("Hello"), "message OK") + assert.Equal(t, []byte("Hello"), buf[:n], "message OK") // verify the message was received from the relay address assert.Equal(t, echoConn.LocalAddr().String(), from.String(), @@ -166,7 +168,7 @@ func stunnerEchoTest(conf echoTestConfig) { } else { // should fail _, err = conn.WriteTo([]byte("Hello"), echoConn.LocalAddr()) - assert.Errorf(t, err, "got error message %s", err.Error()) + assert.Errorf(t, err, "got error message %s", err) } assert.NoError(t, conn.Close(), "cannot close relay connection") assert.NoError(t, echoConn.Close(), "cannot close echo server connection") @@ -174,7 +176,6 @@ func stunnerEchoTest(conf echoTestConfig) { } time.Sleep(150 * time.Millisecond) client.Close() - } // ***************** @@ -1157,6 +1158,332 @@ func TestStunnerClusterWithVNet(t *testing.T) { } } +// ***************** +// Port range filtering tests with VNet +// ***************** +var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ + // port range filtering + { + testName: "static endpoint with peer address in the admitted port range ok", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Type: "plaintext", + Credentials: map[string]string{ + "username": "user1", + "password": "passwd1", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + MinRelayPort: 5670, + MaxRelayPort: 5680, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Type: "STATIC", + Endpoints: []string{ + "1.2.3.5", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + result: true, + }, + { + testName: "static endpoint with peer address matching singleton admitted port ok", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Type: "plaintext", + Credentials: map[string]string{ + "username": "user1", + "password": "passwd1", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + MinRelayPort: 5678, + MaxRelayPort: 5678, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Type: "STATIC", + Endpoints: []string{ + "1.2.3.5", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + result: true, + }, + { + testName: "static endpoint with peer address in rejected port range fails", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Type: "plaintext", + Credentials: map[string]string{ + "username": "user1", + "password": "passwd1", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + MinRelayPort: 1, + MaxRelayPort: 5677, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Type: "STATIC", + Endpoints: []string{ + "1.2.3.5", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + result: false, + }, + { + testName: "static endpoint with peer address in rejected singleton port fails", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Type: "plaintext", + Credentials: map[string]string{ + "username": "user1", + "password": "passwd1", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + MinRelayPort: 5677, + MaxRelayPort: 5677, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Type: "STATIC", + Endpoints: []string{ + "1.2.3.5", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + result: false, + }, +} + +func TestStunnerPortRangeWithVNet(t *testing.T) { + lim := test.TimeOut(time.Second * 60) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) + log := loggerFactory.NewLogger("test") + + // log rate-limiter settings + LogRateLimit = 2 + LogBurst = 1 + + for _, c := range testPortRangeConfigsWithVNet { + t.Run(c.testName, func(t *testing.T) { + log.Debugf("-------------- Running test: %s -------------", c.testName) + + // patch in the vnet + log.Debug("building virtual network") + v, err := buildVNet(loggerFactory) + assert.NoError(t, err, err) + + log.Debug("setting up the mock DNS") + mockDns := resolver.NewMockResolver(map[string]([]string){ + "stunner.l7mp.io": []string{"1.2.3.4"}, + "echo-server.l7mp.io": []string{"1.2.3.5"}, + "dummy.l7mp.io": []string{"1.2.3.10"}, + }, loggerFactory) + + log.Debug("creating a stunnerd") + stunner := NewStunner(Options{ + LogLevel: stunnerTestLoglevel, + SuppressRollback: true, + Resolver: mockDns, + Net: v.podnet, + }) + + log.Debug("starting stunnerd") + assert.NoError(t, stunner.Reconcile(c.config), "starting server") + + var u, p string + auth := c.config.Auth.Type + switch auth { + case "plaintext", "static": + u = "user1" + p = "passwd1" + case "longterm", "ephemeral": + u, p, err = turn.GenerateLongTermCredentials("my-secret", time.Minute) + assert.NoError(t, err, err) + default: + assert.NoError(t, fmt.Errorf("internal error: unknown auth type in test")) + } + + log.Debug("creating a client") + lconn, err := v.wan.ListenPacket("udp4", "0.0.0.0:0") + assert.NoError(t, err, "cannot create client listening socket") + + testConfig := echoTestConfig{t, v.podnet, v.wan, stunner, + "stunner.l7mp.io:3478", lconn, u, p, net.IPv4(5, 6, 7, 8), + c.echoServerAddr, true, true, c.result, loggerFactory} + stunnerEchoFloodTest(testConfig) + + assert.NoError(t, lconn.Close(), "cannot close TURN client connection") + stunner.Close() + assert.NoError(t, v.Close(), "cannot close VNet") + }) + } +} + +func stunnerEchoFloodTest(conf echoTestConfig) { + t := conf.t + log := conf.loggerFactory.NewLogger("test") + + client, err := turn.NewClient(&turn.ClientConfig{ + STUNServerAddr: conf.stunnerAddr, + TURNServerAddr: conf.stunnerAddr, + Username: conf.user, + Password: conf.pass, + Conn: conf.lconn, + Net: conf.wan, + LoggerFactory: conf.loggerFactory, + }) + + assert.NoError(t, err, "cannot create TURN client") + assert.NoError(t, client.Listen(), "cannot listen on TURN client") + defer client.Close() + + log.Debug("sending a binding request") + // reflAddr, err := bindingRequestWithTimeout(client, 10000 * time.Millisecond) + reflAddr, err := client.SendBindingRequest() + if conf.bindSuccess == false { + assert.Error(t, err, "binding request failed") + } else { + assert.NoError(t, err, "binding request ok") + log.Debugf("mapped-address: %v", reflAddr.String()) + udpAddr := reflAddr.(*net.UDPAddr) + + // The mapped-address should have IP address that was assigned to the LAN router. + assert.True(t, udpAddr.IP.Equal(conf.natAddr), "wrong srfx address") + + log.Debug("sending an allocate request") + conn, err := client.Allocate() + if conf.allocateSuccess == false { + assert.Error(t, err, err) + } else { + assert.NoError(t, err, err) + + // log.Debugf("laddr: %s", conn.LocalAddr().String()) + + log.Debugf("creating echo-server listener socket at: %s", conn.LocalAddr().String()) + echoConn, err := conf.podnet.ListenPacket("udp4", conf.echoServerAddr) + assert.NoError(t, err, "creating echo socket") + + // assert.NotNil(t, err, "echo socket not nil") + + go func() { + buf := make([]byte, 1600) + for { + n, from, err2 := echoConn.ReadFrom(buf) + if err2 != nil { + break + } + + // verify the message was received from the relay address + assert.Equal(t, conn.LocalAddr().String(), from.String(), + "message should be received from the relay address") + assert.Equal(t, "Hello", string(buf[:n]), "wrong message payload") + + // echo the data + _, err2 = echoConn.WriteTo(buf[:n], from) + assert.NoError(t, err2, err2) + } + }() + + buf := make([]byte, 1600) + if conf.echoSuccess == true { + for i := 0; i < 500; i++ { + log.Debug("sending \"Hello\"") + _, err = conn.WriteTo([]byte("Hello"), echoConn.LocalAddr()) + assert.NoError(t, err, err) + + n, from, err2 := conn.ReadFrom(buf) + assert.NoError(t, err2, err2) + assert.Equal(t, n, len("Hello"), "message OK") + assert.Equal(t, []byte("Hello"), buf[:n], "message OK") + + // verify the message was received from the relay address + assert.Equal(t, echoConn.LocalAddr().String(), from.String(), + "message should be received from the relay address") + + time.Sleep(2 * time.Millisecond) + } + } else { + // should fail but it does not: client does not get feedback on + // server-side port filtering + for i := 0; i < 500; i++ { + log.Debug("sending \"Hello\"") + _, err = conn.WriteTo([]byte("Hello"), echoConn.LocalAddr()) + assert.NoError(t, err, err) + + // read should time out + assert.NoError(t, conn.SetReadDeadline(time.Now().Add(2*time.Millisecond)), "read deafline") + _, _, err2 := conn.ReadFrom(buf) + assert.Error(t, err2, "deadline exceeded") + } + } + assert.NoError(t, conn.Close(), "cannot close relay connection") + assert.NoError(t, echoConn.Close(), "cannot close echo server connection") + } + } + time.Sleep(150 * time.Millisecond) + client.Close() +} + /******************************************** * * lifecycle + health check tests From a92d3f7a52d7507e119dbb3237db79da5871385e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 4 Dec 2023 13:05:09 +0100 Subject: [PATCH 062/243] fix: LoggerFactory now emits loggers at the correct loglevel --- pkg/logger/logger.go | 9 +++++++-- pkg/logger/logger_test.go | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/pkg/logger/logger.go b/pkg/logger/logger.go index 418c6297..9f5d694a 100644 --- a/pkg/logger/logger.go +++ b/pkg/logger/logger.go @@ -145,7 +145,6 @@ func (f *RateLimitedLoggerFactory) NewLogger(scope string) logging.LeveledLogger // disable rate-limiting logging at lower loglevels l := f.DefaultLogLevel - // this will never happen: loggers are always created with the default loglevel scopeLevel, found := f.ScopeLevels[scope] if found { l = scopeLevel @@ -184,7 +183,13 @@ func (f *LeveledLoggerFactory) newLogger(scope string, limit rate.Limit, burst i return logger } - l := NewRateLimitedLoggerForScope(scope, f.DefaultLogLevel, f.Writer, limit, burst) + logLevel := f.DefaultLogLevel + scopeLevel, found := f.ScopeLevels[scope] + if found { + logLevel = scopeLevel + } + + l := NewRateLimitedLoggerForScope(scope, logLevel, f.Writer, limit, burst) l.DefaultLeveledLogger. WithTraceLogger(log.New(l.RateLimitedWriter, fmt.Sprintf("%s TRACE: ", scope), defaultFlags)). diff --git a/pkg/logger/logger_test.go b/pkg/logger/logger_test.go index 12282f86..f7c74c4f 100644 --- a/pkg/logger/logger_test.go +++ b/pkg/logger/logger_test.go @@ -317,6 +317,44 @@ var loggerTests = []loggerTestCase{ log.Debug("dummy") assert.Zerof(t, loglenr(), "DEBUG for level %s", level) + log.Trace("dummy") + assert.Zerof(t, loglenr(), "TRACE for level %s", level) + }, + }, + { + name: "set-loglevel-for-newly-created-logger", + defaultLogLevel: "all:TRACE", + scopeLogLevel: "TRACE", + prep: func(lf LoggerFactory) { + lf.SetLevel("all:error,new-scope:DEBUG") + }, + tester: func(t *testing.T, lf LoggerFactory) { + level := lf.GetLevel("all") + assert.Equal(t, "Error", level, "default scope: level") + + level = lf.GetLevel(testScope) + assert.Equal(t, "Error", level, "dummy scope: level") + + level = lf.GetLevel("new-scope") + assert.Equal(t, "Debug", level, "new scope: level") + + log := lf.NewLogger("new-scope") + + level = lf.GetLevel("new-scope") + assert.Equal(t, "Debug", level, "new scope: level") + + log.Error("dummy") + assert.Containsf(t, logreadr(), "dummy", "ERROR for level %s", level) + + log.Warn("dummy") + assert.Containsf(t, logreadr(), "dummy", "WARN for level %s", level) + + log.Info("dummy") + assert.Containsf(t, logreadr(), "dummy", "INFO for level %s", level) + + log.Debug("dummy") + assert.Containsf(t, logreadr(), "dummy", "DEBUG for level %s", level) + log.Trace("dummy") assert.Zerof(t, loglenr(), "TRACE for level %s", level) }, From f6723bb734bde7c4b70dd97459051833d5d7c47c Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 4 Dec 2023 13:14:41 +0100 Subject: [PATCH 063/243] refactor: Move CDS clients into separate files --- pkg/config/client/cds_client.go | 252 +++++++++++++++++ pkg/config/client/client.go | 447 ------------------------------- pkg/config/client/file_client.go | 217 +++++++++++++++ 3 files changed, 469 insertions(+), 447 deletions(-) create mode 100644 pkg/config/client/cds_client.go create mode 100644 pkg/config/client/file_client.go diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go new file mode 100644 index 00000000..7c869e95 --- /dev/null +++ b/pkg/config/client/cds_client.go @@ -0,0 +1,252 @@ +package client + +import ( + "context" + "fmt" + "io" + "net/http" + "net/url" + "time" + + "github.com/gorilla/websocket" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/pion/logging" +) + +// configDiscoveryClient is the the implementation of the config discovery service. +type configDiscoveryClient struct { + // serverAddress is the URL of the config discovery server. + serverAddress string + // Id is the name of the stunnerd instance that is used to bootstrap the connection + // poller. Set to namespace/name of the pod when using the stunner gateway operator for + // config discovery. + id string + // Log is a leveled logger used to report progress. + log logging.LeveledLogger +} + +func (p *configDiscoveryClient) String() string { + return fmt.Sprintf("config discovery service using server %q", p.serverAddress) +} + +func (p *configDiscoveryClient) Load() (*stnrv1.StunnerConfig, error) { + location, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, false) + if err != nil { + return nil, err + } + + resp, err := http.Get(location) + if err != nil { + return nil, err + } + + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("invalid HTTP response status: %s", resp.Status) + } + + defer resp.Body.Close() + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, err + } + + if len(body) == 0 { + return nil, errFileTruncated + } + + // fmt.Println("++++++++++++++++++++") + // fmt.Println(string(body)) + + return ParseConfig(body) +} + +// Watch polls a config discovery server for a configuration file by sending a config request and +// then waits for the server to push a valid `StunnerConfig`. Use the `context` to cancel the +// watcher. +func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + _, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, true) + if err != nil { + return err + } + + // Note: we do not emit an initial config but rather wait for the CDS server to send one, + // so that pod will not be able to bootstrap the healthchecker and keep on restarting until + // it finds the CDS server + + go func() { + for { + // try to watch + if err := p.configPoller(ctx, ch); err != nil { + p.log.Errorf("config file discovery service: %s", err.Error()) + } else { + // context got cancelled + return + } + + // wait between each attempt + time.Sleep(RetryPeriod) + } + }() + + return nil +} + +func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + p.log.Tracef("configPoller: trying to open connection to config discovery server at %q", p.serverAddress) + + location, origin, _ := getConfigDiscoveryLocation(p.serverAddress, p.id, true) + header := http.Header{} + header.Set("origin", origin) + conn, _, err := websocket.DefaultDialer.DialContext(ctx, location, header) + if err != nil { + return err + } + + p.log.Infof("connection sucessfully opened to config discovery server at %q", location) + + // this will close the poller goroutine + defer conn.Close() + + // pinger + resCh := make(chan stnrv1.StunnerConfig, 16) + errCh := make(chan error, 1) + + pingTicker := time.NewTicker(PingPeriod) + closePinger := make(chan any) + defer close(closePinger) + + go func() { + defer pingTicker.Stop() + for { + select { + case <-pingTicker.C: + // p.log.Tracef("++++ PING ++++ for CDS server %q at client %q", location, p.id) + conn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck + if err := conn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { + errCh <- fmt.Errorf("could not ping CDS server at %q: %w", + conn.RemoteAddr(), err) + return + } + case <-closePinger: + p.log.Tracef("closing ping handler to config discovery server at %q at client %q", + location, p.id) + return + } + } + }() + + // poller + go func() { + defer close(resCh) + defer close(errCh) + + // the next pong must arrive within the PongWait period + conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + // reinit the deadline when we get a pong + conn.SetPongHandler(func(string) error { + // p.log.Tracef("++++ PONG ++++ from CDS server %q at client %q", location, p.id) + conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + return nil + }) + + for { + // ping-pong deadline misses will end up being caught here as a read beyond + // the deadline + msgType, msg, err := conn.ReadMessage() + if err != nil { + errCh <- err + return + } + + if msgType != websocket.TextMessage { + errCh <- fmt.Errorf("unexpected message type (code: %d) from client %q", + msgType, conn.RemoteAddr().String()) + return + } + + if len(msg) == 0 { + p.log.Warn("ignoring zero-length config config fil") + continue + } + + // fmt.Println("++++++++++++++++++++") + // fmt.Println(string(msg)) + // fmt.Println("++++++++++++++++++++") + + c, err := ParseConfig(msg) + if err != nil { + // assume it is a YAML/JSON syntax error: report and ignore + p.log.Warnf("could not parse config: %s", err.Error()) + continue + } + + confCopy := stnrv1.StunnerConfig{} + c.DeepCopyInto(&confCopy) + + p.log.Debugf("new config received from %q: %s", p.serverAddress, confCopy.String()) + + resCh <- confCopy + } + }() + + // wait fo cancel + for { + select { + case <-ctx.Done(): + // cancel: normal return + closePinger <- struct{}{} + + return nil + case err := <-errCh: + // error: return it + closePinger <- struct{}{} + + return err + case conf := <-resCh: + // new config: pass it along and move on + p.log.Debugf("new config available: %s", conf.String()) + ch <- conf + + continue + } + } +} + +// getConfigLocation returns a valid URL from config server address, either for a single HTTP GET +// to query the config discovery server for a single config file or a websocket URL and client for +// polling config file updates +func getConfigDiscoveryLocation(addr, id string, ws bool) (string, string, error) { + u, err := url.Parse(addr) + if err != nil { + return "", "", fmt.Errorf("invalid config discovery server URL %q: %w", addr, err) + } + + q, err := url.ParseQuery(u.RawQuery) + if err != nil { + return "", "", fmt.Errorf("invalid config discovery query server URL %q: %w", addr, err) + } + + // add our id as a query parameter + q.Set("id", id) + u.RawQuery = q.Encode() + + // TODO: share between server and client + u.Path = "/api/v1/config" + if ws { + u.Scheme = "ws" + u.Path = u.Path + "/watch" + } else { + u.Scheme = "http" + } + + // target URL + location := u.String() + + // client + u.Scheme = "http" + u.RawQuery = "" + client := u.String() + + return location, client, nil +} diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index 167767fd..b583a1f0 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -4,15 +4,10 @@ import ( "context" "errors" "fmt" - "io" - "net/http" "net/url" - "os" "strings" "time" - "github.com/fsnotify/fsnotify" - "github.com/gorilla/websocket" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/pion/logging" ) @@ -70,445 +65,3 @@ func NewClient(origin string, id string, logger logging.LoggerFactory) (Client, return client, nil } - -// configFileClient is the implementation of the Client interface for config files. -type configFileClient struct { - // configFile specifies the config file name to watch. - configFile string - // id is the name of the stunnerd instance. - id string - // log is a leveled logger used to report progress. Either Logger or Log must be specified. - log logging.LeveledLogger -} - -func (w *configFileClient) String() string { - return fmt.Sprintf("config client using file %q", w.configFile) -} - -func (w *configFileClient) Load() (*stnrv1.StunnerConfig, error) { - b, err := os.ReadFile(w.configFile) - if err != nil { - return nil, fmt.Errorf("could not read config file %q: %s", w.configFile, err.Error()) - } - - if len(b) == 0 { - return nil, errFileTruncated - } - - return ParseConfig(b) -} - -// WatchConfig watches a configuration file for changes. If no file exists at the given path, -// WatchConfig will periodically retry until the file appears. -func (w *configFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { - if w.configFile == "" { - return errors.New("uninitialized config file path") - } - - // emit an empty config: this bootstraps stunner the default resources (above all, starts - // the health-checker) - w.log.Debug("bootstrapping with zero configuration") - initConf := ZeroConfig(w.id) - ch <- *initConf - - go func() { - for { - // try to watch - if !w.configWatcher(ctx, ch) { - return - } - - if !w.tryWatchConfig(ctx) { - return - } - } - }() - - return nil -} - -// configWatcher watches the config file and emits new configs on the specified channel. Returns -// true if further action is needed (tryWatchConfig is to be started) or false on normal exit. -func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.StunnerConfig) bool { - w.log.Tracef("configWatcher") - - // create a new watcher - watcher, err := fsnotify.NewWatcher() - if err != nil { - return true - } - defer watcher.Close() - - config := w.configFile - if err := watcher.Add(config); err != nil { - w.log.Debugf("could not add watcher for config file %q: %s", config, err.Error()) - return true - } - - // emit an initial config - c, err := w.Load() - if err != nil { - w.log.Warnf("cannot load config file: %s", err.Error()) - return true - } - - // send a deepcopy over the channel - confCopy := stnrv1.StunnerConfig{} - c.DeepCopyInto(&confCopy) - - w.log.Debugf("initial config file successfully loaded from %q: %s", config, confCopy.String()) - - ch <- confCopy - - // save deepcopy so that we can filter repeated events - prev := stnrv1.StunnerConfig{} - c.DeepCopyInto(&prev) - - for { - select { - case <-ctx.Done(): - return false - - case e, ok := <-watcher.Events: - if !ok { - w.log.Debug("config watcher event handler received invalid event") - return true - } - - w.log.Debugf("received watcher event: %s", e.String()) - - if e.Has(fsnotify.Remove) { - w.log.Warnf("config file deleted %q, disabling watcher", e.Op.String()) - - if err := watcher.Remove(config); err != nil { - w.log.Debugf("could not remove config file %q watcher: %s", - config, err.Error()) - } - - return true - } - - if !e.Has(fsnotify.Write) { - w.log.Debugf("unhandled notify op on config file %q (ignoring): %s", - e.Name, e.Op.String()) - continue - } - - w.log.Debugf("loading configuration file: %s", config) - c, err := w.Load() - if err != nil { - if errors.Is(err, errFileTruncated) { - w.log.Debugf("ignoring: %s", err.Error()) - continue - } - w.log.Warnf("error loading config file: %s", err.Error()) - return true - } - - // suppress repeated events - if c.DeepEqual(&prev) { - w.log.Debugf("ignoring recurrent notify event for the same config file") - continue - } - - confCopy := stnrv1.StunnerConfig{} - c.DeepCopyInto(&confCopy) - - w.log.Debugf("config file successfully loaded from %q: %s", config, confCopy.String()) - - ch <- confCopy - - // save deepcopy so that we can filter repeated events - c.DeepCopyInto(&prev) - - case err, ok := <-watcher.Errors: - if !ok { - w.log.Debugf("config watcher error handler received invalid error") - return true - } - - w.log.Debugf("watcher error, deactivating watcher: %s", err.Error()) - - if err := watcher.Remove(config); err != nil { - w.log.Debugf("could not remove config file %q watcher: %s", - config, err.Error()) - } - - return true - } - } -} - -// tryWatchConfig runs a timer to look for the config file at the given path and returns it -// immediately once found. Returns true if further action is needed (configWatcher has to be -// started) or false on normal exit. -func (w *configFileClient) tryWatchConfig(ctx context.Context) bool { - w.log.Tracef("tryWatchConfig") - config := w.configFile - - ticker := time.NewTicker(RetryPeriod) - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return false - - case <-ticker.C: - w.log.Debugf("trying to read config file %q from periodic timer", - config) - - // check if config file exists and it is readable - if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { - w.log.Debugf("config file %q does not exist", config) - - // report status in every 10th second - if time.Now().Second()%10 == 0 { - w.log.Warnf("waiting for config file %q", config) - } - - continue - } - - return true - } - } -} - -// configDiscoveryClient is the the implementation of the config discovery service. -type configDiscoveryClient struct { - // serverAddress is the URL of the config discovery server. - serverAddress string - // Id is the name of the stunnerd instance that is used to bootstrap the connection - // poller. Set to namespace/name of the pod when using the stunner gateway operator for - // config discovery. - id string - // Log is a leveled logger used to report progress. - log logging.LeveledLogger -} - -func (p *configDiscoveryClient) String() string { - return fmt.Sprintf("config discovery service using server %q", p.serverAddress) -} - -func (p *configDiscoveryClient) Load() (*stnrv1.StunnerConfig, error) { - location, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, false) - if err != nil { - return nil, err - } - - resp, err := http.Get(location) - if err != nil { - return nil, err - } - - if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("invalid HTTP response status: %s", resp.Status) - } - - defer resp.Body.Close() - - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, err - } - - if len(body) == 0 { - return nil, errFileTruncated - } - - // fmt.Println("++++++++++++++++++++") - // fmt.Println(string(body)) - - return ParseConfig(body) -} - -// Watch polls a config discovery server for a configuration file by sending a config request and -// then waits for the server to push a valid `StunnerConfig`. Use the `context` to cancel the -// watcher. -func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { - _, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, true) - if err != nil { - return err - } - - // Note: we do not emit an initial config but rather wait for the CDS server to send one, - // so that pod will not be able to bootstrap the healthchecker and keep on restarting until - // it finds the CDS server - - go func() { - for { - // try to watch - if err := p.configPoller(ctx, ch); err != nil { - p.log.Errorf("config file discovery service: %s", err.Error()) - } else { - // context got cancelled - return - } - - // wait between each attempt - time.Sleep(RetryPeriod) - } - }() - - return nil -} - -func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { - p.log.Tracef("configPoller: trying to open connection to config discovery server at %q", p.serverAddress) - - location, origin, _ := getConfigDiscoveryLocation(p.serverAddress, p.id, true) - header := http.Header{} - header.Set("origin", origin) - conn, _, err := websocket.DefaultDialer.DialContext(ctx, location, header) - if err != nil { - return err - } - - p.log.Infof("connection sucessfully opened to config discovery server at %q", location) - - // this will close the poller goroutine - defer conn.Close() - - // pinger - resCh := make(chan stnrv1.StunnerConfig, 16) - errCh := make(chan error, 1) - - pingTicker := time.NewTicker(PingPeriod) - closePinger := make(chan any) - defer close(closePinger) - - go func() { - defer pingTicker.Stop() - for { - select { - case <-pingTicker.C: - // p.log.Tracef("++++ PING ++++ for CDS server %q at client %q", location, p.id) - conn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck - if err := conn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { - errCh <- fmt.Errorf("could not ping CDS server at %q: %w", - conn.RemoteAddr(), err) - return - } - case <-closePinger: - p.log.Tracef("closing ping handler to config discovery server at %q at client %q", - location, p.id) - return - } - } - }() - - // poller - go func() { - defer close(resCh) - defer close(errCh) - - // the next pong must arrive within the PongWait period - conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck - // reinit the deadline when we get a pong - conn.SetPongHandler(func(string) error { - // p.log.Tracef("++++ PONG ++++ from CDS server %q at client %q", location, p.id) - conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck - return nil - }) - - for { - // ping-pong deadline misses will end up being caught here as a read beyond - // the deadline - msgType, msg, err := conn.ReadMessage() - if err != nil { - errCh <- err - return - } - - if msgType != websocket.TextMessage { - errCh <- fmt.Errorf("unexpected message type (code: %d) from client %q", - msgType, conn.RemoteAddr().String()) - return - } - - if len(msg) == 0 { - p.log.Warn("ignoring zero-length config config fil") - continue - } - - // fmt.Println("++++++++++++++++++++") - // fmt.Println(string(msg)) - // fmt.Println("++++++++++++++++++++") - - c, err := ParseConfig(msg) - if err != nil { - // assume it is a YAML/JSON syntax error: report and ignore - p.log.Warnf("could not parse config: %s", err.Error()) - continue - } - - confCopy := stnrv1.StunnerConfig{} - c.DeepCopyInto(&confCopy) - - p.log.Debugf("new config received from %q: %s", p.serverAddress, confCopy.String()) - - resCh <- confCopy - } - }() - - // wait fo cancel - for { - select { - case <-ctx.Done(): - // cancel: normal return - closePinger <- struct{}{} - - return nil - case err := <-errCh: - // error: return it - closePinger <- struct{}{} - - return err - case conf := <-resCh: - // new config: pass it along and move on - p.log.Debugf("new config available: %s", conf.String()) - ch <- conf - - continue - } - } -} - -// getConfigLocation returns a valid URL from config server address, either for a single HTTP GET -// to query the config discovery server for a single config file or a websocket URL and client for -// polling config file updates -func getConfigDiscoveryLocation(addr, id string, ws bool) (string, string, error) { - u, err := url.Parse(addr) - if err != nil { - return "", "", fmt.Errorf("invalid config discovery server URL %q: %w", addr, err) - } - - q, err := url.ParseQuery(u.RawQuery) - if err != nil { - return "", "", fmt.Errorf("invalid config discovery query server URL %q: %w", addr, err) - } - - // add our id as a query parameter - q.Set("id", id) - u.RawQuery = q.Encode() - - // TODO: share between server and client - u.Path = "/api/v1/config" - if ws { - u.Scheme = "ws" - u.Path = u.Path + "/watch" - } else { - u.Scheme = "http" - } - - // target URL - location := u.String() - - // client - u.Scheme = "http" - u.RawQuery = "" - client := u.String() - - return location, client, nil -} diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go new file mode 100644 index 00000000..176194c2 --- /dev/null +++ b/pkg/config/client/file_client.go @@ -0,0 +1,217 @@ +package client + +import ( + "context" + "errors" + "fmt" + "os" + "time" + + "github.com/fsnotify/fsnotify" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/pion/logging" +) + +// configFileClient is the implementation of the Client interface for config files. +type configFileClient struct { + // configFile specifies the config file name to watch. + configFile string + // id is the name of the stunnerd instance. + id string + // log is a leveled logger used to report progress. Either Logger or Log must be specified. + log logging.LeveledLogger +} + +func (w *configFileClient) String() string { + return fmt.Sprintf("config client using file %q", w.configFile) +} + +func (w *configFileClient) Load() (*stnrv1.StunnerConfig, error) { + b, err := os.ReadFile(w.configFile) + if err != nil { + return nil, fmt.Errorf("could not read config file %q: %s", w.configFile, err.Error()) + } + + if len(b) == 0 { + return nil, errFileTruncated + } + + return ParseConfig(b) +} + +// WatchConfig watches a configuration file for changes. If no file exists at the given path, +// WatchConfig will periodically retry until the file appears. +func (w *configFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + if w.configFile == "" { + return errors.New("uninitialized config file path") + } + + // emit an empty config: this bootstraps stunner the default resources (above all, starts + // the health-checker) + w.log.Debug("bootstrapping with zero configuration") + initConf := ZeroConfig(w.id) + ch <- *initConf + + go func() { + for { + // try to watch + if !w.configWatcher(ctx, ch) { + return + } + + if !w.tryWatchConfig(ctx) { + return + } + } + }() + + return nil +} + +// configWatcher watches the config file and emits new configs on the specified channel. Returns +// true if further action is needed (tryWatchConfig is to be started) or false on normal exit. +func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.StunnerConfig) bool { + w.log.Tracef("configWatcher") + + // create a new watcher + watcher, err := fsnotify.NewWatcher() + if err != nil { + return true + } + defer watcher.Close() + + config := w.configFile + if err := watcher.Add(config); err != nil { + w.log.Debugf("could not add watcher for config file %q: %s", config, err.Error()) + return true + } + + // emit an initial config + c, err := w.Load() + if err != nil { + w.log.Warnf("cannot load config file: %s", err.Error()) + return true + } + + // send a deepcopy over the channel + confCopy := stnrv1.StunnerConfig{} + c.DeepCopyInto(&confCopy) + + w.log.Debugf("initial config file successfully loaded from %q: %s", config, confCopy.String()) + + ch <- confCopy + + // save deepcopy so that we can filter repeated events + prev := stnrv1.StunnerConfig{} + c.DeepCopyInto(&prev) + + for { + select { + case <-ctx.Done(): + return false + + case e, ok := <-watcher.Events: + if !ok { + w.log.Debug("config watcher event handler received invalid event") + return true + } + + w.log.Debugf("received watcher event: %s", e.String()) + + if e.Has(fsnotify.Remove) { + w.log.Warnf("config file deleted %q, disabling watcher", e.Op.String()) + + if err := watcher.Remove(config); err != nil { + w.log.Debugf("could not remove config file %q watcher: %s", + config, err.Error()) + } + + return true + } + + if !e.Has(fsnotify.Write) { + w.log.Debugf("unhandled notify op on config file %q (ignoring): %s", + e.Name, e.Op.String()) + continue + } + + w.log.Debugf("loading configuration file: %s", config) + c, err := w.Load() + if err != nil { + if errors.Is(err, errFileTruncated) { + w.log.Debugf("ignoring: %s", err.Error()) + continue + } + w.log.Warnf("error loading config file: %s", err.Error()) + return true + } + + // suppress repeated events + if c.DeepEqual(&prev) { + w.log.Debugf("ignoring recurrent notify event for the same config file") + continue + } + + confCopy := stnrv1.StunnerConfig{} + c.DeepCopyInto(&confCopy) + + w.log.Debugf("config file successfully loaded from %q: %s", config, confCopy.String()) + + ch <- confCopy + + // save deepcopy so that we can filter repeated events + c.DeepCopyInto(&prev) + + case err, ok := <-watcher.Errors: + if !ok { + w.log.Debugf("config watcher error handler received invalid error") + return true + } + + w.log.Debugf("watcher error, deactivating watcher: %s", err.Error()) + + if err := watcher.Remove(config); err != nil { + w.log.Debugf("could not remove config file %q watcher: %s", + config, err.Error()) + } + + return true + } + } +} + +// tryWatchConfig runs a timer to look for the config file at the given path and returns it +// immediately once found. Returns true if further action is needed (configWatcher has to be +// started) or false on normal exit. +func (w *configFileClient) tryWatchConfig(ctx context.Context) bool { + w.log.Tracef("tryWatchConfig") + config := w.configFile + + ticker := time.NewTicker(RetryPeriod) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return false + + case <-ticker.C: + w.log.Debugf("trying to read config file %q from periodic timer", + config) + + // check if config file exists and it is readable + if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { + w.log.Debugf("config file %q does not exist", config) + + // report status in every 10th second + if time.Now().Second()%10 == 0 { + w.log.Warnf("waiting for config file %q", config) + } + + continue + } + + return true + } + } +} From 058d400a9c6857f2ea7f7e5f73a1bcc84659b6f5 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 7 Dec 2023 10:29:30 +0100 Subject: [PATCH 064/243] chore(CI/CD): Bump actions/setup-go to v5 --- .github/workflows/e2e-test-dev-legacy.yml | 2 +- .github/workflows/e2e-test-dev-managed.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- .github/workflows/lint.yml | 2 +- .github/workflows/test.yml | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index 1d5f5eac..8883038e 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -12,7 +12,7 @@ jobs: - uses: actions/checkout@v4 - name: Set up Go - uses: actions/setup-go@v4 + uses: actions/setup-go@v5 with: go-version: '1.21' diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index 414d6a21..091b8a06 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -12,7 +12,7 @@ jobs: - uses: actions/checkout@v4 - name: Set up Go - uses: actions/setup-go@v4 + uses: actions/setup-go@v5 with: go-version: '1.21' diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index e289a1f0..48ac7997 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -12,7 +12,7 @@ jobs: - uses: actions/checkout@v4 - name: Set up Go - uses: actions/setup-go@v4 + uses: actions/setup-go@v5 with: go-version: '1.21' diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 26b35311..e3ba6e9e 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -18,7 +18,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Install Go - uses: actions/setup-go@v4 + uses: actions/setup-go@v5 with: go-version: '1.21' - name: Checkout code diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 796cb920..4197274d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -21,7 +21,7 @@ jobs: steps: - uses: actions/checkout@v4 - name: Set up Go - uses: actions/setup-go@v4 + uses: actions/setup-go@v5 with: go-version: '1.21' - name: Download modules @@ -35,7 +35,7 @@ jobs: steps: - name: Install Go if: success() - uses: actions/setup-go@v4 + uses: actions/setup-go@v5 with: go-version: '1.21' - name: Checkout code From 2b36ec4106459771ad34baced74ab9bbc33c0417 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 7 Dec 2023 21:58:09 +0100 Subject: [PATCH 065/243] refactor: Re-implement CDS API over OpenAPI Since the CDS service becomes critical for the push/pull of STUNner configs between different components, gateway-operator, stunnerd, stunnerctl (TODO), and the auth-service (TODO), it has become imperative to make the code reusable, extensible and thoroughly tested. This refactor implements this change: - create OpenAPI def for the STUNner dataplane API (see pkg/config/api) - implement a general purpose CDS server - add 3 APIs: ListV1Configs to get/watch for *all* configs in the cluster, ListV1ConfigsNamespace to get/watch all configs in a given namespace, and GetV1ConfigNamespaceName to get/watch the config of a specific gateway (namespace/name) - implement a general purpose client to use all 3 APIs - reimplement the CDS client/server over the new code - add lots of tests --- Makefile | 35 ++ bin/.gitkeep | 0 cmd/stunnerd/README.md | 7 +- cmd/stunnerd/main.go | 1 - config.go | 6 +- config_test.go | 12 +- docs/cmd/stunnerctl.md | 4 +- docs/cmd/stunnerd.md | 16 +- docs/cmd/turncat.md | 4 +- go.mod | 19 +- go.sum | 53 +- handlers.go | 14 +- handlers_test.go | 20 +- internal/util/conn_test.go | 228 ++++++++ pkg/apis/v1/default.go | 3 + pkg/apis/v1/stunner.go | 24 +- pkg/config/api/stunner_openapi.yaml | 118 ++++ pkg/config/client/api/client.gen.go | 530 +++++++++++++++++ pkg/config/client/cds_api.go | 280 +++++++++ pkg/config/client/cds_client.go | 160 ++---- pkg/config/client/cfg.yaml | 5 + pkg/config/client/client.go | 48 +- pkg/config/client/client_test.go | 44 ++ pkg/config/client/config.go | 7 +- pkg/config/client/file_client.go | 59 +- pkg/config/server/api/server.gen.go | 386 +++++++++++++ pkg/config/server/cfg.yaml | 8 + pkg/config/server/config.go | 122 ++++ pkg/config/server/conn.go | 84 +++ pkg/config/server/error.go | 34 ++ pkg/config/server/handler.go | 123 ++++ pkg/config/server/server.go | 207 +++++++ pkg/config/server/server_test.go | 842 ++++++++++++++++++++++++++++ reconcile_test.go | 14 +- stunner.go | 2 +- stunner_test.go | 60 +- tools.go | 7 + turncat_test.go | 4 +- 38 files changed, 3320 insertions(+), 270 deletions(-) create mode 100644 Makefile create mode 100644 bin/.gitkeep create mode 100644 internal/util/conn_test.go create mode 100644 pkg/config/api/stunner_openapi.yaml create mode 100644 pkg/config/client/api/client.gen.go create mode 100644 pkg/config/client/cds_api.go create mode 100644 pkg/config/client/cfg.yaml create mode 100644 pkg/config/client/client_test.go create mode 100644 pkg/config/server/api/server.gen.go create mode 100644 pkg/config/server/cfg.yaml create mode 100644 pkg/config/server/config.go create mode 100644 pkg/config/server/conn.go create mode 100644 pkg/config/server/error.go create mode 100644 pkg/config/server/handler.go create mode 100644 pkg/config/server/server.go create mode 100644 pkg/config/server/server_test.go create mode 100644 tools.go diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..c0598e7b --- /dev/null +++ b/Makefile @@ -0,0 +1,35 @@ +.PHONY: all +all: build + +.PHONY: generate +generate: ## OpenAPI codegen + go generate ./pkg/config/... + +.PHONY: fmt +fmt: ## Run go fmt against code. + go fmt ./... + +.PHONY: vet +vet: ## Run go vet against code. + go vet ./... + +.PHONY: test +test: manifests generate fmt vet test ## Run tests. + +##@ Build + +.PHONY: build +build: generate fmt vet ## Build binary. + go build -o bin/stunnerd cmd/stunnerd/main.go + go build -o bin/turncat cmd/turncat/main.go + +.PHONY: run +run: manifests generate fmt vet ## Run a controller from your host. + go run cmd/stunnerd/main.go + +# clean up generated files +.PHONY: clean +clean: + echo 'Use "make generate` to autogenerate server code' > pkg/server/server.go + echo 'Use "make generate` to autogenerate client code' > pkg/client/client.go + echo 'Use "make generate` to autogenerate client code' > pkg/types/types.go diff --git a/bin/.gitkeep b/bin/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/cmd/stunnerd/README.md b/cmd/stunnerd/README.md index 41eb67db..b28382aa 100644 --- a/cmd/stunnerd/README.md +++ b/cmd/stunnerd/README.md @@ -23,8 +23,9 @@ the daemon can pick up the latest configuration using the watch mode. Extensions for TCP Allocations * TURN transport over UDP, TCP, TLS/TCP and DTLS/UDP. * TURN/UDP listener CPU scaling. -* Two authentication modes via the long-term STUN/TURN credential mechanism: `plaintext` using a - static username/password pair, and `longterm` with dynamically generated time-scoped credentials. +* Two authentication modes via the long-term STUN/TURN credential mechanism: `static` using a + static username/password pair, and `ephemeral` with dynamically generated time-scoped + credentials. ## Getting Started @@ -77,7 +78,7 @@ admin: realm: "my-realm.example.com" static: auth: - type: longterm + type: ephemeral credentials: secret: $STUNNER_SHARED_SECRET listeners: diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 5f047dce..0081df6b 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -22,7 +22,6 @@ const ( // defaultDiscoveryAddress = "ws://localhost:13478/api/v1/config/watch" envVarName = "STUNNER_NAME" envVarNamespace = "STUNNER_NAMESPACE" - envVarId = "STUNNER_ID" envVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" ) diff --git a/config.go b/config.go index 476b520f..d46ea99c 100644 --- a/config.go +++ b/config.go @@ -10,7 +10,7 @@ import ( "github.com/l7mp/stunner/internal/resolver" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" - cds "github.com/l7mp/stunner/pkg/config/client" + "github.com/l7mp/stunner/pkg/config/client" ) // Options defines various options for the STUNner server. @@ -145,7 +145,7 @@ func (s *Stunner) GetConfig() *stnrv1.StunnerConfig { // LoadConfig loads a configuration from an origin. This is a shim wrapper around ConfigOrigin.Load. func (s *Stunner) LoadConfig(config string) (*stnrv1.StunnerConfig, error) { - client, err := cds.NewClient(config, s.id, s.logger) + client, err := client.New(config, s.id, s.logger) if err != nil { return nil, err } @@ -155,7 +155,7 @@ func (s *Stunner) LoadConfig(config string) (*stnrv1.StunnerConfig, error) { // WatchConfig watches a configuration from an origin. This is a shim wrapper around ConfigOrigin.Watch. func (s *Stunner) WatchConfig(ctx context.Context, origin string, ch chan<- stnrv1.StunnerConfig) error { - client, err := cds.NewClient(origin, s.id, s.logger) + client, err := client.New(origin, s.id, s.logger) if err != nil { return err } diff --git a/config_test.go b/config_test.go index e47b14a7..251fc980 100644 --- a/config_test.go +++ b/config_test.go @@ -233,8 +233,8 @@ func TestStunnerConfigFileWatcher(t *testing.T) { } const ( - testConfigV1 = `{"version":"v1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"static","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` - testConfigV1A1 = `{"version":"v1alpha1","admin":{"loglevel":"all:ERROR"},"auth":{"type":"longterm","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` + testConfigV1 = `{"version":"v1","admin":{"name":"ns1/tester", "loglevel":"all:ERROR"},"auth":{"type":"static","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` + testConfigV1A1 = `{"version":"v1alpha1","admin":{"name":"ns1/tester", "loglevel":"all:ERROR"},"auth":{"type":"longterm","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` ) // test with v1alpha1 and v1 @@ -357,7 +357,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { s := &http.Server{Addr: addr} defer s.Close() - http.HandleFunc("/api/v1/config/watch", + http.HandleFunc("/api/v1/configs/ns1/tester", func(w http.ResponseWriter, req *http.Request) { upgrader := websocket.Upgrader{ ReadBufferSize: 1024, @@ -383,7 +383,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { }) // send initial config - z := cdsclient.ZeroConfig("tester") + z := cdsclient.ZeroConfig("ns1/tester") j, err := json.Marshal(z) assert.NoError(t, err, "json marshal zeroconfig") assert.NoError(t, conn.WriteMessage(websocket.TextMessage, j), "write zeroconfig") @@ -411,7 +411,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { time.Sleep(50 * time.Millisecond) log.Debug("creating a stunnerd") - stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel}) + stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel, Id: "ns1/tester"}) log.Debug("starting watcher") conf := make(chan stnrv1.StunnerConfig, 1) @@ -423,7 +423,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { // first read should yield a zeroconfig c2, ok := <-conf assert.True(t, ok, "zeroconfig emitted") - checkZeroConfig(t, &c2, "tester") + checkZeroConfig(t, &c2, "ns1/tester") // second read yields a v1 config c2, ok = <-conf diff --git a/docs/cmd/stunnerctl.md b/docs/cmd/stunnerctl.md index b0231788..9c4debcf 100644 --- a/docs/cmd/stunnerctl.md +++ b/docs/cmd/stunnerctl.md @@ -7,11 +7,11 @@ A CLI tool to simplify the interaction with STUNner. Dump the running config from a live STUNner deployment in human-readable format. ```console cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext +STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 Listener: udp-listener Protocol: UDP Public address: 34.118.36.108 Public port: 3478 -``` \ No newline at end of file +``` diff --git a/docs/cmd/stunnerd.md b/docs/cmd/stunnerd.md index 92396348..e801e660 100644 --- a/docs/cmd/stunnerd.md +++ b/docs/cmd/stunnerd.md @@ -62,21 +62,21 @@ to go. Using the below configuration, `stunnerd` will open 4 STUNner listeners: two for accepting unencrypted connections at UDP/3478 and TCP/3478, and two for encrypted connections at TLS/TCP/3479 -and DTLS/UDP/3479. For easier debugging, the port for the transport relay connections opened by -`stunnerd` will be taken from [10000:19999] for the UDP listener, [20000:29999] for the TCP -listener, etc. The daemon will use `longterm` authentication, with the shared secret read from the -environment variable `$STUNNER_SHARED_SECRET` during initialization. The relay address is taken -from the `$STUNNER_ADDR` environment variable. +and DTLS/UDP/3479. Transport relay connections opened by `stunnerd` will be restricted to peer +ports in the range [10000:19999] for the UDP listener, [20000:29999] for the TCP listener, etc. +The daemon will use `ephemeral` authentication, with the shared secret read from the environment +variable `$STUNNER_SHARED_SECRET` during initialization. The relay address is taken from the +`$STUNNER_ADDR` environment variable. ``` yaml -version: v1alpha1 +version: v1 admin: name: my-stunnerd logLevel: all:DEBUG realm: "my-realm.example.com" static: auth: - type: longterm + type: ephemeral credentials: secret: $STUNNER_SHARED_SECRET listeners: @@ -106,4 +106,4 @@ static: key: "my-key.key" minPort: 40000 maxPort: 49999 -``` \ No newline at end of file +``` diff --git a/docs/cmd/turncat.md b/docs/cmd/turncat.md index 6be4dfcd..83ea9355 100644 --- a/docs/cmd/turncat.md +++ b/docs/cmd/turncat.md @@ -20,7 +20,7 @@ go build -o turncat cmd/turncat/main.go Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP server located at -`192.0.2.2:53`. Use the longterm STUN/TURN credential mechanism to authenticate with the TURN +`192.0.2.2:53`. Use the `static` STUN/TURN credential mechanism to authenticate with the TURN server and set the user/passwd to `test/test`: ```console @@ -45,4 +45,4 @@ enable verbose logging. ```console ./turncat -v - k8s://stunner/stunnerd-config:udp-listener udp://10.0.0.1:9001 -``` \ No newline at end of file +``` diff --git a/go.mod b/go.mod index 1f07145d..b2facf0f 100644 --- a/go.mod +++ b/go.mod @@ -3,10 +3,16 @@ module github.com/l7mp/stunner go 1.19 require ( + github.com/deepmap/oapi-codegen/v2 v2.0.0 github.com/fsnotify/fsnotify v1.6.0 - github.com/google/uuid v1.3.1 + github.com/getkin/kin-openapi v0.118.0 + github.com/go-logr/logr v1.3.0 + github.com/go-logr/zapr v1.3.0 + github.com/google/uuid v1.4.0 + github.com/gorilla/mux v1.8.0 github.com/gorilla/websocket v1.5.1 github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb + github.com/oapi-codegen/runtime v1.1.0 github.com/pion/dtls/v2 v2.2.7 github.com/pion/logging v0.2.2 github.com/pion/transport/v3 v3.0.1 @@ -14,6 +20,7 @@ require ( github.com/prometheus/client_golang v1.16.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 + go.uber.org/zap v1.26.0 golang.org/x/sys v0.13.0 golang.org/x/time v0.4.0 k8s.io/api v0.24.3 @@ -25,12 +32,12 @@ require ( require ( github.com/PuerkitoBio/purell v1.1.1 // indirect github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect + github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/emicklei/go-restful v2.9.5+incompatible // indirect github.com/evanphx/json-patch v4.12.0+incompatible // indirect - github.com/go-logr/logr v1.2.0 // indirect github.com/go-openapi/jsonpointer v0.19.5 // indirect github.com/go-openapi/jsonreference v0.19.5 // indirect github.com/go-openapi/swag v0.19.14 // indirect @@ -39,13 +46,16 @@ require ( github.com/google/gnostic v0.5.7-v3refs // indirect github.com/google/gofuzz v1.1.0 // indirect github.com/imdario/mergo v0.3.12 // indirect + github.com/invopop/yaml v0.1.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/mailru/easyjson v0.7.6 // indirect + github.com/mailru/easyjson v0.7.7 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/perimeterx/marshmallow v1.1.4 // indirect github.com/pion/randutil v0.1.0 // indirect github.com/pion/stun/v2 v2.0.0 // indirect github.com/pion/transport/v2 v2.2.1 // indirect @@ -54,11 +64,14 @@ require ( github.com/prometheus/client_model v0.4.0 // indirect github.com/prometheus/common v0.44.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect + go.uber.org/multierr v1.10.0 // indirect golang.org/x/crypto v0.14.0 // indirect + golang.org/x/mod v0.12.0 // indirect golang.org/x/net v0.17.0 // indirect golang.org/x/oauth2 v0.8.0 // indirect golang.org/x/term v0.13.0 // indirect golang.org/x/text v0.13.0 // indirect + golang.org/x/tools v0.12.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/protobuf v1.31.0 // indirect gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect diff --git a/go.sum b/go.sum index 5ff91d3f..7dc510b4 100644 --- a/go.sum +++ b/go.sum @@ -50,10 +50,14 @@ github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tN github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= +github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= +github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/bmatcuk/doublestar v1.1.1/go.mod h1:UD6OnuiIn0yFxxA2le/rnRU1G4RaI4UvFv1sNto9p6w= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= @@ -68,6 +72,8 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/deepmap/oapi-codegen/v2 v2.0.0 h1:3TS7w3r+XnjKFXcbFbc16pTWzfTy0OLPkCsutEHjWDA= +github.com/deepmap/oapi-codegen/v2 v2.0.0/go.mod h1:7zR+ZL3WzLeCkr2k8oWTxEa0v8y/F25ane0l6A5UjLA= github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= @@ -88,15 +94,19 @@ github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4 github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY= github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw= github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg= +github.com/getkin/kin-openapi v0.118.0 h1:z43njxPmJ7TaPpMSCQb7PN0dEYno4tyBPQcrFdHoLuM= +github.com/getkin/kin-openapi v0.118.0/go.mod h1:l5e9PaFUo9fyLJCPGQeXI2ML8c3P8BHOEV2VaAVf/pc= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= -github.com/go-logr/logr v1.2.0 h1:QK40JKJyMdUDz+h+xvCsru/bJhvG0UxvePV0ufL/AcE= github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= -github.com/go-logr/zapr v1.2.0 h1:n4JnPI1T3Qq1SFEi/F8rwLrZERp2bso19PJZDB9dayk= +github.com/go-logr/logr v1.3.0 h1:2y3SDp0ZXuc6/cjLSZ+Q3ir+QB9T/iG5yYRXqsagWSY= +github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= +github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= @@ -106,6 +116,8 @@ github.com/go-openapi/jsonreference v0.19.5/go.mod h1:RdybgQwPxbL4UEjuAruzK1x3nE github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/swag v0.19.14 h1:gm3vOOXfiuw5i9p5N9xJvfjvuofpyvLA9Wr6QfK5Fng= github.com/go-openapi/swag v0.19.14/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= +github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= +github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= @@ -176,10 +188,11 @@ github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4= -github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4= +github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= +github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI= github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY= @@ -195,6 +208,8 @@ github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1: github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= +github.com/invopop/yaml v0.1.0 h1:YW3WGUoJEXYfzWBjn00zIlrw7brGVD0fUKRYDPAPhrc= +github.com/invopop/yaml v0.1.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= @@ -202,6 +217,7 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= +github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= @@ -213,8 +229,9 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA= github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= +github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo= github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= @@ -225,6 +242,8 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJ github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 h1:RWengNIwukTxcDr9M+97sNutRR1RKhG96O6jWumTTnw= +github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8= github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= @@ -232,6 +251,8 @@ github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= +github.com/oapi-codegen/runtime v1.1.0 h1:rJpoNUawn5XTvekgfkvSZr0RqEnoYpFkyvrzfWeFKWM= +github.com/oapi-codegen/runtime v1.1.0/go.mod h1:BeSfBkWWWnAnGdyS+S/GnlbmHKzf8/hwkvelJZDeKA8= github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= @@ -241,6 +262,8 @@ github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGV github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE= +github.com/perimeterx/marshmallow v1.1.4 h1:pZLDH9RjlLGGorbXhcaQLhfuV0pFMNfPO55FuFkxqLw= +github.com/perimeterx/marshmallow v1.1.4/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/pion/dtls/v2 v2.2.7 h1:cSUBsETxepsCSFSxC3mc/aDo14qQLMSL+O6IjG28yV8= github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= @@ -274,6 +297,7 @@ github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjR github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= @@ -285,9 +309,14 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/ugorji/go v1.2.7 h1:qYhyWUUd6WbiM+C6JZAUkIJt/1WrjzNHY9+KCIjVqTo= +github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M= +github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY= +github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -301,9 +330,11 @@ go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E= -go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw= -go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4= -go.uber.org/zap v1.19.1 h1:ue41HOKd1vGURxrmeKIgELGb3jPW9DMUDGtsinblHwI= +go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= +go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= +go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= +go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -339,6 +370,7 @@ golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRu golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= +golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= @@ -352,6 +384,8 @@ golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.12.0 h1:rmsUpXtvNzj340zd98LZ4KntptpfRHwpFOHG188oHXc= +golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -560,6 +594,8 @@ golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= +golang.org/x/tools v0.12.0 h1:YW6HUoUmYBpwSgyaGaZq1fHjrBjX1rlpZ54T6mu2kss= +golang.org/x/tools v0.12.0/go.mod h1:Sc0INKfu04TlqNoRA1hgpFZbhYXHPr4V5DzpSBTPqQM= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -690,6 +726,7 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/handlers.go b/handlers.go index 2d7953dc..42d0a32f 100644 --- a/handlers.go +++ b/handlers.go @@ -22,35 +22,35 @@ func (s *Stunner) NewAuthHandler() a12n.AuthHandler { switch auth.Type { case stnrv1.AuthTypeStatic: - auth.Log.Infof("plaintext auth request: username=%q realm=%q srcAddr=%v\n", + auth.Log.Infof("static auth request: username=%q realm=%q srcAddr=%v\n", username, realm, srcAddr) key := a12n.GenerateAuthKey(auth.Username, auth.Realm, auth.Password) if username == auth.Username { - auth.Log.Debug("plaintext auth request: valid username") + auth.Log.Debug("static auth request: valid username") return key, true } - auth.Log.Info("plaintext auth request: failed: invalid username") + auth.Log.Info("static auth request: failed: invalid username") return nil, false case stnrv1.AuthTypeEphemeral: - auth.Log.Infof("longterm auth request: username=%q realm=%q srcAddr=%v", + auth.Log.Infof("ephemeral auth request: username=%q realm=%q srcAddr=%v", username, realm, srcAddr) if err := a12n.CheckTimeWindowedUsername(username); err != nil { - auth.Log.Infof("longterm auth request: failed: %s", err) + auth.Log.Infof("ephemeral auth request: failed: %s", err) return nil, false } password, err := a12n.GetLongTermCredential(username, auth.Secret) if err != nil { - auth.Log.Infof("longterm auth request: error generating password: %s", + auth.Log.Infof("ephemeral auth request: error generating password: %s", err) return nil, false } - auth.Log.Info("longterm auth request: success") + auth.Log.Info("ephemeral auth request: success") return a12n.GenerateAuthKey(username, auth.Realm, password), true default: diff --git a/handlers_test.go b/handlers_test.go index 9911fe05..a35ff00d 100644 --- a/handlers_test.go +++ b/handlers_test.go @@ -46,7 +46,7 @@ type StunnerTestAuthWithVnet struct { var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ { - testName: "plaintext", + testName: "static", clientAddr: "1.1.1.1", conf: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, @@ -54,7 +54,7 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -75,14 +75,14 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ authCred: func() (string, string) { return "user1", "passwd1" }, }, { - testName: "longterm - plain timestamp in username", + testName: "ephemeral - plain timestamp in username", conf: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, @@ -105,14 +105,14 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, }, { - testName: "longterm - timestamp:userid in username", + testName: "ephemeral - timestamp:userid in username", conf: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, @@ -137,14 +137,14 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, }, { - testName: "longterm - userid:timestamp in username", + testName: "ephemeral - userid:timestamp in username", conf: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, @@ -169,14 +169,14 @@ var testStunnerAuthWithVnet = []StunnerTestAuthWithVnet{ }, }, { - testName: "longterm - userid:timestamp:ramdom-crap in username", + testName: "ephemeral - userid:timestamp:ramdom-crap in username", conf: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, diff --git a/internal/util/conn_test.go b/internal/util/conn_test.go new file mode 100644 index 00000000..90e9a491 --- /dev/null +++ b/internal/util/conn_test.go @@ -0,0 +1,228 @@ +package util + +import ( + "net" + "testing" + "time" + + "github.com/l7mp/stunner/pkg/logger" + "github.com/pion/transport/v3/test" + "github.com/pion/transport/v3/vnet" + "github.com/stretchr/testify/assert" +) + +var connTestLoglevel string = "all:ERROR" + +// var connTestLoglevel string = stnrv1.DefaultLogLevel +// var connTestLoglevel string = "all:INFO" +// var connTestLoglevel string = "all:TRACE" +// var connTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" + +func TestPortRangePacketConn(t *testing.T) { + lim := test.TimeOut(time.Second * 30) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + loggerFactory := logger.NewLoggerFactory(connTestLoglevel) + log := loggerFactory.NewLogger("test") + + log.Debug("Creating vnet") + nw, err := vnet.NewNet(&vnet.NetConfig{}) + if !assert.NoError(t, err, "should succeed") { + return + } + + t.Run("LoopbackOnValidPort", func(t *testing.T) { + log.Debug("Creating base socket") + addr := "127.0.0.1:15000" + baseConn, err := nw.ListenPacket("udp", addr) + assert.NoError(t, err, "should succeed") + msg := "PING!" + + log.Debug("Creating filtered packet conn wrappeer socket") + conn := NewPortRangePacketConn(baseConn, "test", 10000, 20000, log) + assert.NoError(t, err, "should create port-range filtered packetconn") + + log.Debug("Sending packet") + udpAddr, err := net.ResolveUDPAddr("udp", addr) + assert.NoError(t, err, "should resolve UDP address") + n, err := conn.WriteTo([]byte(msg), udpAddr) + assert.NoError(t, err, "should succeed") + assert.Equal(t, len(msg), n, "should match") + + log.Debug("Receiving packet") + buf := make([]byte, 1000) + n, remoteAddr, err := conn.ReadFrom(buf) + assert.NoError(t, err, "should succeed") + assert.Equal(t, len(msg), n, "should match") + assert.Equal(t, msg, string(buf[:n]), "should match") + assert.Equal(t, udpAddr.String(), remoteAddr.String(), "should match") //nolint:forcetypeassert + + log.Debug("Closing connection") + assert.NoError(t, conn.Close(), "should succeed") // should close baseConn + }) + + t.Run("LoopbackOnInvalidPort", func(t *testing.T) { + log.Debug("Creating base socket") + addr := "127.0.0.1:25000" + baseConn, err := nw.ListenPacket("udp", addr) + assert.NoError(t, err, "should succeed") + msg := "PING!" + + log.Debug("Creating filtered packet conn wrappeer socket") + conn := NewPortRangePacketConn(baseConn, "test", 10000, 20000, log) + assert.NoError(t, err, "should create port-range filtered packetconn") + + log.Debug("Sending packet") + udpAddr, err := net.ResolveUDPAddr("udp", addr) + assert.NoError(t, err, "should resolve UDP address") + n, err := conn.WriteTo([]byte(msg), udpAddr) + assert.Error(t, err, "should reject") + assert.Equal(t, 0, n, "should match") + + log.Debug("Receiving packet") + buf := make([]byte, 1000) + // this would hang otherwise + assert.NoError(t, conn.SetReadDeadline(time.Now().Add(10*time.Millisecond)), "read deadline") + _, _, err = conn.ReadFrom(buf) + assert.Error(t, err, "should be rejected") + + log.Debug("Closing connection") + assert.NoError(t, conn.Close(), "should succeed") + }) + + t.Run("LoopbackOnSinglePort", func(t *testing.T) { + log.Debug("Creating base socket") + addr := "127.0.0.1:15000" + baseConn, err := nw.ListenPacket("udp", addr) + assert.NoError(t, err, "should succeed") + msg := "PING!" + + log.Debug("Creating filtered packet conn wrappeer socket") + conn := NewPortRangePacketConn(baseConn, "test", 15000, 15000, log) + assert.NoError(t, err, "should create port-range filtered packetconn") + + log.Debug("Sending packet") + udpAddr, err := net.ResolveUDPAddr("udp", addr) + assert.NoError(t, err, "should resolve UDP address") + n, err := conn.WriteTo([]byte(msg), udpAddr) + assert.NoError(t, err, "should succeed") + assert.Equal(t, len(msg), n, "should match") + + log.Debug("Receiving packet") + buf := make([]byte, 1000) + n, remoteAddr, err := conn.ReadFrom(buf) + assert.NoError(t, err, "should succeed") + assert.Equal(t, len(msg), n, "should match") + assert.Equal(t, msg, string(buf[:n]), "should match") + assert.Equal(t, udpAddr.String(), remoteAddr.String(), "should match") //nolint:forcetypeassert + + log.Debug("Closing connection") + assert.NoError(t, conn.Close(), "should succeed") // should close baseConn + }) +} + +// BenchmarkPortRangePacketConn sends lots of invalid packets: this is mostly for testing the logger +func BenchmarkPortRangePacketConn(b *testing.B) { + loggerFactory := logger.NewLoggerFactory(connTestLoglevel) + log := loggerFactory.NewLogger("test") + // relayLog := loggerFactory.WithRateLimiter(.25, 1).NewLogger("relay") + relayLog := log + + log.Debug("Creating vnet") + nw, err := vnet.NewNet(&vnet.NetConfig{}) + if err != nil { + b.Fatalf("Cannot allocate vnet: %s", err.Error()) + } + + log.Debug("Creating base socket") + addr := "127.0.0.1:25000" + baseConn, err := nw.ListenPacket("udp", addr) + if err != nil { + b.Fatalf("Cannot listen on vnet: %s", err.Error()) + } + msg := "PING!" + + log.Debug("Creating filtered packet conn wrappeer socket") + conn := WithCounter(NewPortRangePacketConn(baseConn, "test", 15000, 15000, relayLog)) + if err != nil { + b.Fatalf("Cannot create port-range packetconn: %s", err.Error()) + } + + udpAddr, err := net.ResolveUDPAddr("udp", addr) + if err != nil { + b.Fatalf("Cannot resove UDP addess: %s", err.Error()) + } + + // Run benchmark + buffer := make([]byte, 1024) + for j := 0; j < b.N; j++ { + _, err := conn.WriteTo([]byte(msg), udpAddr) + if err == nil { + b.Fatal("Conn should reject write to invalid port") + } + + // should never receive: we drop everything + err = conn.SetReadDeadline(time.Now().Add(50 * time.Millisecond)) + if err == nil { + b.Fatal("Could not set read deadline") + } + + _, _, err = conn.ReadFrom(buffer) + if err == nil { + b.Fatal("Conn should never read") + } + } + + readCounter := conn.(*CounterPacketConn).ReadCounter() + if readCounter != 0 { + b.Fatalf("Read counter (%d) should be 0", readCounter) + } + + writeCounter := conn.(*CounterPacketConn).WriteCounter() + if writeCounter != 0 { + b.Fatalf("Write counter (%d) should be %d", writeCounter, b.N) + } + + log.Debug("Closing connection") + err = conn.Close() + if err != nil { + b.Fatalf("Cannot close connection: %s", err.Error()) + } +} + +// CounterPacketConn is a net.PacketConn that filters on the target port range. +type CounterPacketConn struct { + net.PacketConn + readCounter, writeCounter int +} + +// WithCounter decorates a PacketConn with a counter. +func WithCounter(c net.PacketConn) net.PacketConn { + return &CounterPacketConn{ + PacketConn: c, + } +} + +// WriteTo writes to the PacketConn. +func (c *CounterPacketConn) WriteTo(p []byte, peerAddr net.Addr) (int, error) { + n, err := c.PacketConn.WriteTo(p, peerAddr) + if err == nil { + c.writeCounter++ + } + return n, err +} + +// ReadFrom reads from the CounterPacketConn. +func (c *CounterPacketConn) ReadFrom(p []byte) (int, net.Addr, error) { + n, addr, err := c.PacketConn.ReadFrom(p) + if err == nil { + c.readCounter++ + } + return n, addr, err +} + +func (c *CounterPacketConn) ReadCounter() int { return c.readCounter } +func (c *CounterPacketConn) WriteCounter() int { return c.writeCounter } diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 7ecefcea..e77f5fb7 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -17,3 +17,6 @@ const DefaultAuthName = "default-auth-config" const DefaultMetricsPort int = 8080 const DefaultHealthCheckPort int = 8086 + +// DefaultConfigDiscoveryAddress is the default URI at which config discovery requests are served. +const DefaultConfigDiscoveryAddress = "0.0.0.0:13478" diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go index 84414b35..6b143e32 100644 --- a/pkg/apis/v1/stunner.go +++ b/pkg/apis/v1/stunner.go @@ -36,18 +36,26 @@ func (req *StunnerConfig) Validate() error { return err } - for i, l := range req.Listeners { - if err := l.Validate(); err != nil { - return err + if req.Listeners == nil { + req.Listeners = []ListenerConfig{} + } else { + for i, l := range req.Listeners { + if err := l.Validate(); err != nil { + return err + } + req.Listeners[i] = l } - req.Listeners[i] = l } - for i, c := range req.Clusters { - if err := c.Validate(); err != nil { - return err + if req.Clusters == nil { + req.Clusters = []ClusterConfig{} + } else { + for i, c := range req.Clusters { + if err := c.Validate(); err != nil { + return err + } + req.Clusters[i] = c } - req.Clusters[i] = c } return nil diff --git a/pkg/config/api/stunner_openapi.yaml b/pkg/config/api/stunner_openapi.yaml new file mode 100644 index 00000000..d2932f21 --- /dev/null +++ b/pkg/config/api/stunner_openapi.yaml @@ -0,0 +1,118 @@ +openapi: 3.0.3 +info: + title: REST API for STUNner dataplane config + description: The STUNner dataplane API exposes a reconfigurable multi-protocol multi-listener TURN service for WebRTC media ingestion. + version: 1.0.0 +servers: + - url: "" +tags: + - name: STUNner + description: 'A Kubernetes media gateway for WebRTC. Contact: info@l7mp.io' +paths: + /api/v1/configs: + get: + description: API to list or watch config objects in all namespaces. + operationId: listV1Configs + parameters: + - name: watch + in: query + description: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + schema: + type: boolean + responses: + "501": + description: Unimplemented + # "200": + # description: OK + # content: + # application/json: + # schema: + # $ref: '#/components/schemas/v1.ConfigList' + # application/json;stream=watch: + # schema: + # $ref: '#/components/schemas/v1.ConfigList' + /api/v1/configs/{namespace}: + get: + description: API to list or watch config objects in a specific namespace. + operationId: listV1ConfigsNamespace + parameters: + - name: namespace + in: path + description: Namespace of the gateway for which config is requested. + required: true + schema: + type: string + - name: watch + in: query + description: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + schema: + type: boolean + responses: + "501": + description: Unimplemented + # "200": + # description: OK + # content: + # application/json: + # schema: + # $ref: '#/components/schemas/v1.ConfigList' + # application/json;stream=watch: + # schema: + # $ref: '#/components/schemas/v1.ConfigList' + /api/v1/configs/{namespace}/{name}: + get: + description: API to list or watch config objects for a specific gateway in a specific namespace. + operationId: getV1ConfigNamespaceName + parameters: + - name: namespace + in: path + description: Namespace of the gateway for which config is requested. + required: true + schema: + type: string + - name: name + in: path + description: Name of the gateway for which config is requested. + required: true + schema: + type: string + - name: watch + in: query + description: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + schema: + type: boolean + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: '#/components/schemas/v1.Config' + application/json;stream=watch: + schema: + $ref: '#/components/schemas/v1.Config' +components: + schemas: + v1.ConfigList: + description: ConfigList is a list of Configs. + type: object + required: + - items + properties: + version: + description: version defines the versioned schema of this object. + type: string + items: + description: Items is the list of Config objects in the list. + type: array + items: + $ref: '#/components/schemas/v1.Config' + kind: + description: Kind is a string value representing the REST resource this object represents. + type: string + v1.Config: + description: 'Config provides a STUNner config. Schema is defined in https://github.com/l7mp/stunner/tree/main/pkg/apis/v1' + x-go-type: "stunnerv1.StunnerConfig" + x-go-type-import: + name: stunnerv1 + path: "github.com/l7mp/stunner/pkg/apis/v1" diff --git a/pkg/config/client/api/client.gen.go b/pkg/config/client/api/client.gen.go new file mode 100644 index 00000000..d7cd5cdd --- /dev/null +++ b/pkg/config/client/api/client.gen.go @@ -0,0 +1,530 @@ +// Package api provides primitives to interact with the openapi HTTP API. +// +// Code generated by github.com/deepmap/oapi-codegen/v2 version v2.0.0 DO NOT EDIT. +package api + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "strings" + + stunnerv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/oapi-codegen/runtime" +) + +// V1Config Config provides a STUNner config. Schema is defined in https://github.com/l7mp/stunner/tree/main/pkg/apis/v1 +type V1Config = stunnerv1.StunnerConfig + +// ListV1ConfigsParams defines parameters for ListV1Configs. +type ListV1ConfigsParams struct { + // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` +} + +// ListV1ConfigsNamespaceParams defines parameters for ListV1ConfigsNamespace. +type ListV1ConfigsNamespaceParams struct { + // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` +} + +// GetV1ConfigNamespaceNameParams defines parameters for GetV1ConfigNamespaceName. +type GetV1ConfigNamespaceNameParams struct { + // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` +} + +// RequestEditorFn is the function signature for the RequestEditor callback function +type RequestEditorFn func(ctx context.Context, req *http.Request) error + +// Doer performs HTTP requests. +// +// The standard http.Client implements this interface. +type HttpRequestDoer interface { + Do(req *http.Request) (*http.Response, error) +} + +// Client which conforms to the OpenAPI3 specification for this service. +type Client struct { + // The endpoint of the server conforming to this interface, with scheme, + // https://api.deepmap.com for example. This can contain a path relative + // to the server, such as https://api.deepmap.com/dev-test, and all the + // paths in the swagger spec will be appended to the server. + Server string + + // Doer for performing requests, typically a *http.Client with any + // customized settings, such as certificate chains. + Client HttpRequestDoer + + // A list of callbacks for modifying requests which are generated before sending over + // the network. + RequestEditors []RequestEditorFn +} + +// ClientOption allows setting custom parameters during construction +type ClientOption func(*Client) error + +// Creates a new Client, with reasonable defaults +func NewClient(server string, opts ...ClientOption) (*Client, error) { + // create a client with sane default values + client := Client{ + Server: server, + } + // mutate client and add all optional params + for _, o := range opts { + if err := o(&client); err != nil { + return nil, err + } + } + // ensure the server URL always has a trailing slash + if !strings.HasSuffix(client.Server, "/") { + client.Server += "/" + } + // create httpClient, if not already present + if client.Client == nil { + client.Client = &http.Client{} + } + return &client, nil +} + +// WithHTTPClient allows overriding the default Doer, which is +// automatically created using http.Client. This is useful for tests. +func WithHTTPClient(doer HttpRequestDoer) ClientOption { + return func(c *Client) error { + c.Client = doer + return nil + } +} + +// WithRequestEditorFn allows setting up a callback function, which will be +// called right before sending the request. This can be used to mutate the request. +func WithRequestEditorFn(fn RequestEditorFn) ClientOption { + return func(c *Client) error { + c.RequestEditors = append(c.RequestEditors, fn) + return nil + } +} + +// The interface specification for the client above. +type ClientInterface interface { + // ListV1Configs request + ListV1Configs(ctx context.Context, params *ListV1ConfigsParams, reqEditors ...RequestEditorFn) (*http.Response, error) + + // ListV1ConfigsNamespace request + ListV1ConfigsNamespace(ctx context.Context, namespace string, params *ListV1ConfigsNamespaceParams, reqEditors ...RequestEditorFn) (*http.Response, error) + + // GetV1ConfigNamespaceName request + GetV1ConfigNamespaceName(ctx context.Context, namespace string, name string, params *GetV1ConfigNamespaceNameParams, reqEditors ...RequestEditorFn) (*http.Response, error) +} + +func (c *Client) ListV1Configs(ctx context.Context, params *ListV1ConfigsParams, reqEditors ...RequestEditorFn) (*http.Response, error) { + req, err := NewListV1ConfigsRequest(c.Server, params) + if err != nil { + return nil, err + } + req = req.WithContext(ctx) + if err := c.applyEditors(ctx, req, reqEditors); err != nil { + return nil, err + } + return c.Client.Do(req) +} + +func (c *Client) ListV1ConfigsNamespace(ctx context.Context, namespace string, params *ListV1ConfigsNamespaceParams, reqEditors ...RequestEditorFn) (*http.Response, error) { + req, err := NewListV1ConfigsNamespaceRequest(c.Server, namespace, params) + if err != nil { + return nil, err + } + req = req.WithContext(ctx) + if err := c.applyEditors(ctx, req, reqEditors); err != nil { + return nil, err + } + return c.Client.Do(req) +} + +func (c *Client) GetV1ConfigNamespaceName(ctx context.Context, namespace string, name string, params *GetV1ConfigNamespaceNameParams, reqEditors ...RequestEditorFn) (*http.Response, error) { + req, err := NewGetV1ConfigNamespaceNameRequest(c.Server, namespace, name, params) + if err != nil { + return nil, err + } + req = req.WithContext(ctx) + if err := c.applyEditors(ctx, req, reqEditors); err != nil { + return nil, err + } + return c.Client.Do(req) +} + +// NewListV1ConfigsRequest generates requests for ListV1Configs +func NewListV1ConfigsRequest(server string, params *ListV1ConfigsParams) (*http.Request, error) { + var err error + + serverURL, err := url.Parse(server) + if err != nil { + return nil, err + } + + operationPath := fmt.Sprintf("/api/v1/configs") + if operationPath[0] == '/' { + operationPath = "." + operationPath + } + + queryURL, err := serverURL.Parse(operationPath) + if err != nil { + return nil, err + } + + if params != nil { + queryValues := queryURL.Query() + + if params.Watch != nil { + + if queryFrag, err := runtime.StyleParamWithLocation("form", true, "watch", runtime.ParamLocationQuery, *params.Watch); err != nil { + return nil, err + } else if parsed, err := url.ParseQuery(queryFrag); err != nil { + return nil, err + } else { + for k, v := range parsed { + for _, v2 := range v { + queryValues.Add(k, v2) + } + } + } + + } + + queryURL.RawQuery = queryValues.Encode() + } + + req, err := http.NewRequest("GET", queryURL.String(), nil) + if err != nil { + return nil, err + } + + return req, nil +} + +// NewListV1ConfigsNamespaceRequest generates requests for ListV1ConfigsNamespace +func NewListV1ConfigsNamespaceRequest(server string, namespace string, params *ListV1ConfigsNamespaceParams) (*http.Request, error) { + var err error + + var pathParam0 string + + pathParam0, err = runtime.StyleParamWithLocation("simple", false, "namespace", runtime.ParamLocationPath, namespace) + if err != nil { + return nil, err + } + + serverURL, err := url.Parse(server) + if err != nil { + return nil, err + } + + operationPath := fmt.Sprintf("/api/v1/configs/%s", pathParam0) + if operationPath[0] == '/' { + operationPath = "." + operationPath + } + + queryURL, err := serverURL.Parse(operationPath) + if err != nil { + return nil, err + } + + if params != nil { + queryValues := queryURL.Query() + + if params.Watch != nil { + + if queryFrag, err := runtime.StyleParamWithLocation("form", true, "watch", runtime.ParamLocationQuery, *params.Watch); err != nil { + return nil, err + } else if parsed, err := url.ParseQuery(queryFrag); err != nil { + return nil, err + } else { + for k, v := range parsed { + for _, v2 := range v { + queryValues.Add(k, v2) + } + } + } + + } + + queryURL.RawQuery = queryValues.Encode() + } + + req, err := http.NewRequest("GET", queryURL.String(), nil) + if err != nil { + return nil, err + } + + return req, nil +} + +// NewGetV1ConfigNamespaceNameRequest generates requests for GetV1ConfigNamespaceName +func NewGetV1ConfigNamespaceNameRequest(server string, namespace string, name string, params *GetV1ConfigNamespaceNameParams) (*http.Request, error) { + var err error + + var pathParam0 string + + pathParam0, err = runtime.StyleParamWithLocation("simple", false, "namespace", runtime.ParamLocationPath, namespace) + if err != nil { + return nil, err + } + + var pathParam1 string + + pathParam1, err = runtime.StyleParamWithLocation("simple", false, "name", runtime.ParamLocationPath, name) + if err != nil { + return nil, err + } + + serverURL, err := url.Parse(server) + if err != nil { + return nil, err + } + + operationPath := fmt.Sprintf("/api/v1/configs/%s/%s", pathParam0, pathParam1) + if operationPath[0] == '/' { + operationPath = "." + operationPath + } + + queryURL, err := serverURL.Parse(operationPath) + if err != nil { + return nil, err + } + + if params != nil { + queryValues := queryURL.Query() + + if params.Watch != nil { + + if queryFrag, err := runtime.StyleParamWithLocation("form", true, "watch", runtime.ParamLocationQuery, *params.Watch); err != nil { + return nil, err + } else if parsed, err := url.ParseQuery(queryFrag); err != nil { + return nil, err + } else { + for k, v := range parsed { + for _, v2 := range v { + queryValues.Add(k, v2) + } + } + } + + } + + queryURL.RawQuery = queryValues.Encode() + } + + req, err := http.NewRequest("GET", queryURL.String(), nil) + if err != nil { + return nil, err + } + + return req, nil +} + +func (c *Client) applyEditors(ctx context.Context, req *http.Request, additionalEditors []RequestEditorFn) error { + for _, r := range c.RequestEditors { + if err := r(ctx, req); err != nil { + return err + } + } + for _, r := range additionalEditors { + if err := r(ctx, req); err != nil { + return err + } + } + return nil +} + +// ClientWithResponses builds on ClientInterface to offer response payloads +type ClientWithResponses struct { + ClientInterface +} + +// NewClientWithResponses creates a new ClientWithResponses, which wraps +// Client with return type handling +func NewClientWithResponses(server string, opts ...ClientOption) (*ClientWithResponses, error) { + client, err := NewClient(server, opts...) + if err != nil { + return nil, err + } + return &ClientWithResponses{client}, nil +} + +// WithBaseURL overrides the baseURL. +func WithBaseURL(baseURL string) ClientOption { + return func(c *Client) error { + newBaseURL, err := url.Parse(baseURL) + if err != nil { + return err + } + c.Server = newBaseURL.String() + return nil + } +} + +// ClientWithResponsesInterface is the interface specification for the client with responses above. +type ClientWithResponsesInterface interface { + // ListV1ConfigsWithResponse request + ListV1ConfigsWithResponse(ctx context.Context, params *ListV1ConfigsParams, reqEditors ...RequestEditorFn) (*ListV1ConfigsResponse, error) + + // ListV1ConfigsNamespaceWithResponse request + ListV1ConfigsNamespaceWithResponse(ctx context.Context, namespace string, params *ListV1ConfigsNamespaceParams, reqEditors ...RequestEditorFn) (*ListV1ConfigsNamespaceResponse, error) + + // GetV1ConfigNamespaceNameWithResponse request + GetV1ConfigNamespaceNameWithResponse(ctx context.Context, namespace string, name string, params *GetV1ConfigNamespaceNameParams, reqEditors ...RequestEditorFn) (*GetV1ConfigNamespaceNameResponse, error) +} + +type ListV1ConfigsResponse struct { + Body []byte + HTTPResponse *http.Response +} + +// Status returns HTTPResponse.Status +func (r ListV1ConfigsResponse) Status() string { + if r.HTTPResponse != nil { + return r.HTTPResponse.Status + } + return http.StatusText(0) +} + +// StatusCode returns HTTPResponse.StatusCode +func (r ListV1ConfigsResponse) StatusCode() int { + if r.HTTPResponse != nil { + return r.HTTPResponse.StatusCode + } + return 0 +} + +type ListV1ConfigsNamespaceResponse struct { + Body []byte + HTTPResponse *http.Response +} + +// Status returns HTTPResponse.Status +func (r ListV1ConfigsNamespaceResponse) Status() string { + if r.HTTPResponse != nil { + return r.HTTPResponse.Status + } + return http.StatusText(0) +} + +// StatusCode returns HTTPResponse.StatusCode +func (r ListV1ConfigsNamespaceResponse) StatusCode() int { + if r.HTTPResponse != nil { + return r.HTTPResponse.StatusCode + } + return 0 +} + +type GetV1ConfigNamespaceNameResponse struct { + Body []byte + HTTPResponse *http.Response + JSON200 *V1Config +} + +// Status returns HTTPResponse.Status +func (r GetV1ConfigNamespaceNameResponse) Status() string { + if r.HTTPResponse != nil { + return r.HTTPResponse.Status + } + return http.StatusText(0) +} + +// StatusCode returns HTTPResponse.StatusCode +func (r GetV1ConfigNamespaceNameResponse) StatusCode() int { + if r.HTTPResponse != nil { + return r.HTTPResponse.StatusCode + } + return 0 +} + +// ListV1ConfigsWithResponse request returning *ListV1ConfigsResponse +func (c *ClientWithResponses) ListV1ConfigsWithResponse(ctx context.Context, params *ListV1ConfigsParams, reqEditors ...RequestEditorFn) (*ListV1ConfigsResponse, error) { + rsp, err := c.ListV1Configs(ctx, params, reqEditors...) + if err != nil { + return nil, err + } + return ParseListV1ConfigsResponse(rsp) +} + +// ListV1ConfigsNamespaceWithResponse request returning *ListV1ConfigsNamespaceResponse +func (c *ClientWithResponses) ListV1ConfigsNamespaceWithResponse(ctx context.Context, namespace string, params *ListV1ConfigsNamespaceParams, reqEditors ...RequestEditorFn) (*ListV1ConfigsNamespaceResponse, error) { + rsp, err := c.ListV1ConfigsNamespace(ctx, namespace, params, reqEditors...) + if err != nil { + return nil, err + } + return ParseListV1ConfigsNamespaceResponse(rsp) +} + +// GetV1ConfigNamespaceNameWithResponse request returning *GetV1ConfigNamespaceNameResponse +func (c *ClientWithResponses) GetV1ConfigNamespaceNameWithResponse(ctx context.Context, namespace string, name string, params *GetV1ConfigNamespaceNameParams, reqEditors ...RequestEditorFn) (*GetV1ConfigNamespaceNameResponse, error) { + rsp, err := c.GetV1ConfigNamespaceName(ctx, namespace, name, params, reqEditors...) + if err != nil { + return nil, err + } + return ParseGetV1ConfigNamespaceNameResponse(rsp) +} + +// ParseListV1ConfigsResponse parses an HTTP response from a ListV1ConfigsWithResponse call +func ParseListV1ConfigsResponse(rsp *http.Response) (*ListV1ConfigsResponse, error) { + bodyBytes, err := io.ReadAll(rsp.Body) + defer func() { _ = rsp.Body.Close() }() + if err != nil { + return nil, err + } + + response := &ListV1ConfigsResponse{ + Body: bodyBytes, + HTTPResponse: rsp, + } + + return response, nil +} + +// ParseListV1ConfigsNamespaceResponse parses an HTTP response from a ListV1ConfigsNamespaceWithResponse call +func ParseListV1ConfigsNamespaceResponse(rsp *http.Response) (*ListV1ConfigsNamespaceResponse, error) { + bodyBytes, err := io.ReadAll(rsp.Body) + defer func() { _ = rsp.Body.Close() }() + if err != nil { + return nil, err + } + + response := &ListV1ConfigsNamespaceResponse{ + Body: bodyBytes, + HTTPResponse: rsp, + } + + return response, nil +} + +// ParseGetV1ConfigNamespaceNameResponse parses an HTTP response from a GetV1ConfigNamespaceNameWithResponse call +func ParseGetV1ConfigNamespaceNameResponse(rsp *http.Response) (*GetV1ConfigNamespaceNameResponse, error) { + bodyBytes, err := io.ReadAll(rsp.Body) + defer func() { _ = rsp.Body.Close() }() + if err != nil { + return nil, err + } + + response := &GetV1ConfigNamespaceNameResponse{ + Body: bodyBytes, + HTTPResponse: rsp, + } + + switch { + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 200: + var dest V1Config + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON200 = &dest + + case rsp.StatusCode == 200: + // Content-type (application/json;stream=watch) unsupported + + } + + return response, nil +} diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go new file mode 100644 index 00000000..a61a775c --- /dev/null +++ b/pkg/config/client/cds_api.go @@ -0,0 +1,280 @@ +//go:generate go run github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen --config=cfg.yaml ../api/stunner_openapi.yaml +package client + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/config/client/api" + "github.com/pion/logging" +) + +const ( + ConfigNamespaceNameAPIEndpoint = "/api/v1/configs/%s/%s" + ConfigsNamespaceAPIEndpoint = "/api/v1/configs/%s" + AllConfigsAPIEndpoint = "/api/v1/configs" +) + +type ConfigList struct { + Version string `json:"version"` + Items []*stnrv1.StunnerConfig `json:"items"` +} + +type CDSAPI interface { + // Endpoint returns the address of the server plus the WebSocket API endpoint. + Endpoint() (string, string) + // Get loads the config(s) from the API endpoint. + Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error) + // Watch watches config(s) from the API endpoint of a CDS server. If the server is not + // available watch will retry, and if the connection goes away it will create a new one. + Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + // Poll creates a one-shot config watcher without the retry mechanincs of Watch. + Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + logging.LeveledLogger +} + +// AllConfigsAPI is the API for listing all configs in a namespace. +type AllConfigsAPI struct { + addr, httpURI, wsURI string + client *api.ClientWithResponses + logging.LeveledLogger +} + +func NewAllConfigsAPI(addr string, logger logging.LeveledLogger) (CDSAPI, error) { + httpuri, err := getURI(addr) + if err != nil { + return nil, err + } + + wsuri, err := wsURI(addr, AllConfigsAPIEndpoint) + if err != nil { + return nil, err + } + + client, err := api.NewClientWithResponses(httpuri.String()) + if err != nil { + return nil, err + } + + return &AllConfigsAPI{ + addr: addr, + httpURI: httpuri.String(), + wsURI: wsuri, + client: client, + LeveledLogger: logger, + }, nil +} + +func (a *AllConfigsAPI) Endpoint() (string, string) { + return a.addr, a.wsURI +} + +func (a *AllConfigsAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error) { + a.Debugf("GET: loading all configs from CDS server %s", a.addr) + + r, err := a.client.ListV1ConfigsWithResponse(ctx, nil) + if err != nil { + return []*stnrv1.StunnerConfig{}, err + } + + if r.HTTPResponse.StatusCode != http.StatusOK { + return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error: %s", r.HTTPResponse.Status) + } + + return decodeConfigList(r.Body) +} + +func (a *AllConfigsAPI) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + a.Debugf("WATCH: watching all configs from CDS server %s", a.wsURI) + return watch(ctx, a, ch) +} + +func (a *AllConfigsAPI) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + a.Debugf("POLL: polling all configs from CDS server %s", a.wsURI) + return poll(ctx, a, ch) +} + +// ConfigsNamespaceAPI is the API for listing all configs in a namespace. +type ConfigsNamespaceAPI struct { + addr, namespace, httpURI, wsURI string + client *api.ClientWithResponses + logging.LeveledLogger +} + +func NewConfigsNamespaceAPI(addr, namespace string, logger logging.LeveledLogger) (CDSAPI, error) { + httpuri, err := getURI(addr) + if err != nil { + return nil, err + } + + wsuri, err := wsURI(addr, fmt.Sprintf(ConfigsNamespaceAPIEndpoint, namespace)) + if err != nil { + return nil, err + } + + client, err := api.NewClientWithResponses(httpuri.String()) + if err != nil { + return nil, err + } + + return &ConfigsNamespaceAPI{ + addr: addr, + namespace: namespace, + httpURI: httpuri.String(), + wsURI: wsuri, + client: client, + LeveledLogger: logger, + }, nil +} + +func (a *ConfigsNamespaceAPI) Endpoint() (string, string) { + return a.addr, a.wsURI +} + +func (a *ConfigsNamespaceAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error) { + a.Debugf("GET: loading all configs in namespace %s from CDS server %s", + a.namespace, a.addr) + + r, err := a.client.ListV1ConfigsNamespaceWithResponse(ctx, a.namespace, nil) + if err != nil { + return []*stnrv1.StunnerConfig{}, err + } + + if r.HTTPResponse.StatusCode != http.StatusOK { + return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error: %s", + r.HTTPResponse.Status) + } + + return decodeConfigList(r.Body) +} + +func (a *ConfigsNamespaceAPI) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + a.Debugf("WATCH: watching all configs in namespace %s from CDS server %s", + a.namespace, a.wsURI) + return watch(ctx, a, ch) +} + +func (a *ConfigsNamespaceAPI) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + a.Debugf("POLL: polling all configs in namespace %s from CDS server %s", + a.namespace, a.wsURI) + return poll(ctx, a, ch) +} + +type ConfigNamespaceNameAPI struct { + addr, namespace, name, httpURI, wsURI string + client *api.ClientWithResponses + logging.LeveledLogger +} + +func NewConfigNamespaceNameAPI(addr, namespace, name string, logger logging.LeveledLogger) (CDSAPI, error) { + httpuri, err := getURI(addr) + if err != nil { + return nil, err + } + + wsuri, err := wsURI(addr, fmt.Sprintf(ConfigNamespaceNameAPIEndpoint, namespace, name)) + if err != nil { + return nil, err + } + + client, err := api.NewClientWithResponses(httpuri.String()) + if err != nil { + return nil, err + } + + return &ConfigNamespaceNameAPI{ + addr: addr, + namespace: namespace, + name: name, + httpURI: httpuri.String(), + wsURI: wsuri, + client: client, + LeveledLogger: logger, + }, nil +} + +func (a *ConfigNamespaceNameAPI) Endpoint() (string, string) { + return a.addr, a.wsURI +} + +func (a *ConfigNamespaceNameAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error) { + a.Debugf("GET: loading config for gateway %s/%s from CDS server %s", + a.namespace, a.name, a.addr) + + r, err := a.client.GetV1ConfigNamespaceNameWithResponse(ctx, a.namespace, a.name, nil) + if err != nil { + return []*stnrv1.StunnerConfig{}, err + } + + if r.HTTPResponse.StatusCode != http.StatusOK { + return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error: %s", + r.HTTPResponse.Status) + } + + return decodeConfig(r.Body) +} + +func (a *ConfigNamespaceNameAPI) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + a.Debugf("WATCH: watching config for gateway %s/%s from CDS server %s", + a.namespace, a.name, a.wsURI) + return watch(ctx, a, ch) +} + +func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { + a.Debugf("POLL: polling config for gateway %s/%s from CDS server %s", + a.namespace, a.name, a.wsURI) + return poll(ctx, a, ch) +} + +func decodeConfig(r []byte) ([]*stnrv1.StunnerConfig, error) { + c := stnrv1.StunnerConfig{} + if err := json.Unmarshal(r, &c); err != nil { + return nil, err + } + + // copy + + return []*stnrv1.StunnerConfig{&c}, nil +} + +func decodeConfigList(r []byte) ([]*stnrv1.StunnerConfig, error) { + l := ConfigList{} + if err := json.Unmarshal(r, &l); err != nil { + return nil, err + } + return l.Items, nil +} + +// getURI tries to parse an address or an URL or a file name into an URL. +func getURI(addr string) (*url.URL, error) { + url, err := url.Parse(addr) + if err != nil { + // try to parse with a http scheme as a last resort + u, err2 := url.Parse("http://" + addr) + if err2 != nil { + return nil, err + } + url = u + } + return url, nil +} + +// wsURI returns a websocket url from a HTTP URI. +func wsURI(addr, endpoint string) (string, error) { + uri, err := getURI(addr) + if err != nil { + return "", err + } + + uri.Scheme = "ws" + uri.Path = endpoint + v := url.Values{} + v.Set("watch", "true") + uri.RawQuery = v.Encode() + + return uri.String(), nil +} diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go index 7c869e95..6eb69364 100644 --- a/pkg/config/client/cds_client.go +++ b/pkg/config/client/cds_client.go @@ -1,11 +1,11 @@ +//go:generate go run github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen --config=cfg.yaml ../api/stunner_openapi.yaml package client import ( "context" "fmt" - "io" "net/http" - "net/url" + "strings" "time" "github.com/gorilla/websocket" @@ -13,72 +13,50 @@ import ( "github.com/pion/logging" ) -// configDiscoveryClient is the the implementation of the config discovery service. -type configDiscoveryClient struct { - // serverAddress is the URL of the config discovery server. - serverAddress string - // Id is the name of the stunnerd instance that is used to bootstrap the connection - // poller. Set to namespace/name of the pod when using the stunner gateway operator for - // config discovery. - id string - // Log is a leveled logger used to report progress. - log logging.LeveledLogger +// CDSClient is a client for the config discovery service that knows how to poll configs for a +// specific gateway. Use the CDSAPI to access the general CDS client set. +type CDSClient struct { + CDSAPI + addr, id string } -func (p *configDiscoveryClient) String() string { - return fmt.Sprintf("config discovery service using server %q", p.serverAddress) -} - -func (p *configDiscoveryClient) Load() (*stnrv1.StunnerConfig, error) { - location, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, false) - if err != nil { - return nil, err +func NewCDSClient(addr, id string, logger logging.LeveledLogger) (Client, error) { + ps := strings.Split(id, "/") + if len(ps) != 2 { + return nil, fmt.Errorf("invalid id: %q", id) } - resp, err := http.Get(location) + client, err := NewConfigNamespaceNameAPI(addr, ps[0], ps[1], logger) if err != nil { return nil, err } - if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("invalid HTTP response status: %s", resp.Status) - } + return &CDSClient{CDSAPI: client, addr: addr, id: id}, nil +} - defer resp.Body.Close() +func (p *CDSClient) String() string { + return fmt.Sprintf("config discovery client %q: using server %s", p.id, p.addr) +} - body, err := io.ReadAll(resp.Body) +func (p *CDSClient) Load() (*stnrv1.StunnerConfig, error) { + configs, err := p.CDSAPI.Get(context.Background()) if err != nil { return nil, err } - - if len(body) == 0 { - return nil, errFileTruncated + if len(configs) != 1 { + return nil, fmt.Errorf("expected exactly one config, got %d", len(configs)) } - // fmt.Println("++++++++++++++++++++") - // fmt.Println(string(body)) - - return ParseConfig(body) + return configs[0], nil } -// Watch polls a config discovery server for a configuration file by sending a config request and -// then waits for the server to push a valid `StunnerConfig`. Use the `context` to cancel the -// watcher. -func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { - _, _, err := getConfigDiscoveryLocation(p.serverAddress, p.id, true) - if err != nil { - return err - } - - // Note: we do not emit an initial config but rather wait for the CDS server to send one, - // so that pod will not be able to bootstrap the healthchecker and keep on restarting until - // it finds the CDS server - +func watch(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error { go func() { for { // try to watch - if err := p.configPoller(ctx, ch); err != nil { - p.log.Errorf("config file discovery service: %s", err.Error()) + if err := poll(ctx, a, ch); err != nil { + _, wsuri := a.Endpoint() + a.Errorf("failed to init CDS watcher (url: %s): %s", wsuri, err.Error()) } else { // context got cancelled return @@ -92,26 +70,19 @@ func (p *configDiscoveryClient) Watch(ctx context.Context, ch chan<- stnrv1.Stun return nil } -func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { - p.log.Tracef("configPoller: trying to open connection to config discovery server at %q", p.serverAddress) +func poll(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error { + _, url := a.Endpoint() + a.Tracef("poll: trying to open connection to CDS server at %s", url) - location, origin, _ := getConfigDiscoveryLocation(p.serverAddress, p.id, true) - header := http.Header{} - header.Set("origin", origin) - conn, _, err := websocket.DefaultDialer.DialContext(ctx, location, header) + conn, _, err := websocket.DefaultDialer.DialContext(ctx, url, makeHeader(url)) if err != nil { return err } + defer conn.Close() // this will close the poller goroutine - p.log.Infof("connection sucessfully opened to config discovery server at %q", location) - - // this will close the poller goroutine - defer conn.Close() + a.Infof("connection successfully opened to config discovery server at %s", url) - // pinger - resCh := make(chan stnrv1.StunnerConfig, 16) errCh := make(chan error, 1) - pingTicker := time.NewTicker(PingPeriod) closePinger := make(chan any) defer close(closePinger) @@ -129,8 +100,7 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnr return } case <-closePinger: - p.log.Tracef("closing ping handler to config discovery server at %q at client %q", - location, p.id) + a.Tracef("closing ping handler to config discovery server at %q", url) return } } @@ -138,14 +108,13 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnr // poller go func() { - defer close(resCh) defer close(errCh) // the next pong must arrive within the PongWait period conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck // reinit the deadline when we get a pong conn.SetPongHandler(func(string) error { - // p.log.Tracef("++++ PONG ++++ from CDS server %q at client %q", location, p.id) + // a.Tracef("Got PONG from server %q", url) conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck return nil }) @@ -166,7 +135,7 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnr } if len(msg) == 0 { - p.log.Warn("ignoring zero-length config config fil") + a.Warn("ignoring zero-length config") continue } @@ -177,16 +146,16 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnr c, err := ParseConfig(msg) if err != nil { // assume it is a YAML/JSON syntax error: report and ignore - p.log.Warnf("could not parse config: %s", err.Error()) + a.Warnf("could not parse config: %s", err.Error()) continue } confCopy := stnrv1.StunnerConfig{} c.DeepCopyInto(&confCopy) - p.log.Debugf("new config received from %q: %s", p.serverAddress, confCopy.String()) + a.Debugf("new config received from %q: %q", url, confCopy.String()) - resCh <- confCopy + ch <- confCopy } }() @@ -196,57 +165,22 @@ func (p *configDiscoveryClient) configPoller(ctx context.Context, ch chan<- stnr case <-ctx.Done(): // cancel: normal return closePinger <- struct{}{} - return nil case err := <-errCh: // error: return it closePinger <- struct{}{} - return err - case conf := <-resCh: - // new config: pass it along and move on - p.log.Debugf("new config available: %s", conf.String()) - ch <- conf - - continue } } } -// getConfigLocation returns a valid URL from config server address, either for a single HTTP GET -// to query the config discovery server for a single config file or a websocket URL and client for -// polling config file updates -func getConfigDiscoveryLocation(addr, id string, ws bool) (string, string, error) { - u, err := url.Parse(addr) - if err != nil { - return "", "", fmt.Errorf("invalid config discovery server URL %q: %w", addr, err) - } - - q, err := url.ParseQuery(u.RawQuery) - if err != nil { - return "", "", fmt.Errorf("invalid config discovery query server URL %q: %w", addr, err) - } - - // add our id as a query parameter - q.Set("id", id) - u.RawQuery = q.Encode() - - // TODO: share between server and client - u.Path = "/api/v1/config" - if ws { - u.Scheme = "ws" - u.Path = u.Path + "/watch" - } else { - u.Scheme = "http" - } - - // target URL - location := u.String() - - // client - u.Scheme = "http" - u.RawQuery = "" - client := u.String() - - return location, client, nil +// creates an origin header +func makeHeader(uri string) http.Header { + header := http.Header{} + url, _ := getURI(uri) //nolint:errcheck + origin := *url + origin.Scheme = "http" + origin.Path = "" + header.Set("origin", origin.String()) + return header } diff --git a/pkg/config/client/cfg.yaml b/pkg/config/client/cfg.yaml new file mode 100644 index 00000000..1a22cccd --- /dev/null +++ b/pkg/config/client/cfg.yaml @@ -0,0 +1,5 @@ +package: api +generate: + client: true + models: true +output: api/client.gen.go diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index b583a1f0..7d054ff5 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -4,7 +4,6 @@ import ( "context" "errors" "fmt" - "net/url" "strings" "time" @@ -15,17 +14,17 @@ import ( var errFileTruncated = errors.New("zero-length config file") var ( - // Time - RetryPeriod = 1 * time.Second - - // Time allowed to write a message to the CDS server. - WriteWait = 2 * time.Second + // Send pings to the CDS server with this period. Must be less than PongWait. + PingPeriod = 5 * time.Second // Time allowed to read the next pong message from the CDS server. PongWait = 8 * time.Second - // Send pings to the CDS server with this period. Must be less than PongWait. - PingPeriod = 5 * time.Second + // Time allowed to write a message to the CDS server. + WriteWait = 2 * time.Second + + // Period for retrying failed CDS connections. + RetryPeriod = 1 * time.Second ) // Client represents a generic config client. Currently supported config providers: http, ws, or @@ -34,34 +33,33 @@ var ( type Client interface { // Load grabs a new configuration from the config client. Load() (*stnrv1.StunnerConfig, error) - // Watch listens to new configurations and returns them on the channel ch. The context ctx - // cancels the watcher. + // Watch grabs new configs from a config origin (config file or CDS server) and returns + // them on the channel. The context cancels the watcher. If the origin is not available + // watch will retry. Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + // Poll creates a one-shot config watcher without the retry mechanincs of Watch. + Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error fmt.Stringer } -// GetConfig returns the configuration of the running STUNner daemon. -func NewClient(origin string, id string, logger logging.LoggerFactory) (Client, error) { - u, err := url.Parse(origin) +func New(origin string, id string, logger logging.LoggerFactory) (Client, error) { + u, err := getURI(origin) if err != nil { - return nil, fmt.Errorf("could not parse config origin address %q: %w", origin, err) + return nil, fmt.Errorf("could not parse config origin URI %q: %w", origin, err) } - var client Client switch strings.ToLower(u.Scheme) { case "http", "ws", "https", "wss": - client = &configDiscoveryClient{ - serverAddress: origin, - id: id, - log: logger.NewLogger("config-poller"), + client, err := NewCDSClient(u.String(), id, logger.NewLogger("cds-client")) + if err != nil { + return nil, err } + return client, nil default: - client = &configFileClient{ - configFile: origin, - id: id, - log: logger.NewLogger("config-watcher"), + client, err := NewConfigFileClient(origin, id, logger.NewLogger("config-file-client")) + if err != nil { + return nil, err } + return client, nil } - - return client, nil } diff --git a/pkg/config/client/client_test.go b/pkg/config/client/client_test.go new file mode 100644 index 00000000..97765a17 --- /dev/null +++ b/pkg/config/client/client_test.go @@ -0,0 +1,44 @@ +package client + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestParseURI(t *testing.T) { + // file + u, err := getURI("/tmp/a") + assert.NoError(t, err, "file URI parse") + assert.Equal(t, "", u.Scheme, "file URI scheme") + assert.Equal(t, "", u.Host, "file URI host") + assert.Equal(t, "/tmp/a", u.Path, "file URI path") + + // addr + u, err = getURI("1.2.3.4:12345") + assert.NoError(t, err, "file URI parse") + assert.Equal(t, "http", u.Scheme, "file URI scheme") + assert.Equal(t, "1.2.3.4:12345", u.Host, "file URI host") + assert.Equal(t, "", u.Path, "file URI path") + + // addr+path + u, err = getURI("1.2.3.4:12345/a/b") + assert.NoError(t, err, "file URI parse") + assert.Equal(t, "http", u.Scheme, "file URI scheme") + assert.Equal(t, "1.2.3.4:12345", u.Host, "file URI host") + assert.Equal(t, "/a/b", u.Path, "file URI path") + + // scheme+addr+path + u, err = getURI("http://1.2.3.4:12345/a/b") + assert.NoError(t, err, "file URI parse") + assert.Equal(t, "http", u.Scheme, "file URI scheme") + assert.Equal(t, "1.2.3.4:12345", u.Host, "file URI host") + assert.Equal(t, "/a/b", u.Path, "file URI path") + + // ws+addr+path + u, err = getURI("ws://1.2.3.4:12345/a/b") + assert.NoError(t, err, "file URI parse") + assert.Equal(t, "ws", u.Scheme, "file URI scheme") + assert.Equal(t, "1.2.3.4:12345", u.Host, "file URI host") + assert.Equal(t, "/a/b", u.Path, "file URI path") +} diff --git a/pkg/config/client/config.go b/pkg/config/client/config.go index b8ec1d80..a05f9b61 100644 --- a/pkg/config/client/config.go +++ b/pkg/config/client/config.go @@ -16,14 +16,15 @@ type ConfigSkeleton struct { ApiVersion string `json:"version"` } -// ZeroConfig builds a zero configuration useful for bootstrapping STUNner. It starts with -// plaintext authentication and opens no listeners and clusters. +// ZeroConfig builds a zero configuration useful for bootstrapping STUNner. The minimal config +// defaults to static authentication with a dummy username and password and opens no listeners or +// clusters. func ZeroConfig(id string) *stnrv1.StunnerConfig { return &stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{Name: id}, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Realm: stnrv1.DefaultRealm, Credentials: map[string]string{ "username": "dummy-username", diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index 176194c2..9144b1aa 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -12,8 +12,8 @@ import ( "github.com/pion/logging" ) -// configFileClient is the implementation of the Client interface for config files. -type configFileClient struct { +// ConfigFileClient is the implementation of the Client interface for config files. +type ConfigFileClient struct { // configFile specifies the config file name to watch. configFile string // id is the name of the stunnerd instance. @@ -22,11 +22,20 @@ type configFileClient struct { log logging.LeveledLogger } -func (w *configFileClient) String() string { +func NewConfigFileClient(origin, id string, logger logging.LeveledLogger) (Client, error) { + return &ConfigFileClient{ + configFile: origin, + id: id, + log: logger, + }, nil + +} + +func (w *ConfigFileClient) String() string { return fmt.Sprintf("config client using file %q", w.configFile) } -func (w *configFileClient) Load() (*stnrv1.StunnerConfig, error) { +func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { b, err := os.ReadFile(w.configFile) if err != nil { return nil, fmt.Errorf("could not read config file %q: %s", w.configFile, err.Error()) @@ -41,7 +50,7 @@ func (w *configFileClient) Load() (*stnrv1.StunnerConfig, error) { // WatchConfig watches a configuration file for changes. If no file exists at the given path, // WatchConfig will periodically retry until the file appears. -func (w *configFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { if w.configFile == "" { return errors.New("uninitialized config file path") } @@ -55,7 +64,10 @@ func (w *configFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerCo go func() { for { // try to watch - if !w.configWatcher(ctx, ch) { + if err := w.Poll(ctx, ch); err != nil { + w.log.Warnf("error loading config file %q: %s", + w.configFile, err.Error()) + } else { return } @@ -68,29 +80,27 @@ func (w *configFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerCo return nil } -// configWatcher watches the config file and emits new configs on the specified channel. Returns -// true if further action is needed (tryWatchConfig is to be started) or false on normal exit. -func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.StunnerConfig) bool { +// Poll watches the config file and emits new configs on the specified channel. Returns an error if +// further action is needed (tryWatchConfig is to be started) or nil on normal exit. +func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { w.log.Tracef("configWatcher") // create a new watcher watcher, err := fsnotify.NewWatcher() if err != nil { - return true + return err } defer watcher.Close() config := w.configFile if err := watcher.Add(config); err != nil { - w.log.Debugf("could not add watcher for config file %q: %s", config, err.Error()) - return true + return err } // emit an initial config c, err := w.Load() if err != nil { - w.log.Warnf("cannot load config file: %s", err.Error()) - return true + return err } // send a deepcopy over the channel @@ -108,25 +118,22 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.S for { select { case <-ctx.Done(): - return false + return nil case e, ok := <-watcher.Events: if !ok { - w.log.Debug("config watcher event handler received invalid event") - return true + return errors.New("config watcher event handler received invalid event") } w.log.Debugf("received watcher event: %s", e.String()) if e.Has(fsnotify.Remove) { - w.log.Warnf("config file deleted %q, disabling watcher", e.Op.String()) - if err := watcher.Remove(config); err != nil { w.log.Debugf("could not remove config file %q watcher: %s", config, err.Error()) } - return true + return fmt.Errorf("config file deleted %q, disabling watcher", e.Op.String()) } if !e.Has(fsnotify.Write) { @@ -142,8 +149,7 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.S w.log.Debugf("ignoring: %s", err.Error()) continue } - w.log.Warnf("error loading config file: %s", err.Error()) - return true + return err } // suppress repeated events @@ -164,18 +170,15 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.S case err, ok := <-watcher.Errors: if !ok { - w.log.Debugf("config watcher error handler received invalid error") - return true + return errors.New("config watcher error handler received invalid error") } - w.log.Debugf("watcher error, deactivating watcher: %s", err.Error()) - if err := watcher.Remove(config); err != nil { w.log.Debugf("could not remove config file %q watcher: %s", config, err.Error()) } - return true + return fmt.Errorf("watcher error, deactivating watcher: %w", err) } } } @@ -183,7 +186,7 @@ func (w *configFileClient) configWatcher(ctx context.Context, ch chan<- stnrv1.S // tryWatchConfig runs a timer to look for the config file at the given path and returns it // immediately once found. Returns true if further action is needed (configWatcher has to be // started) or false on normal exit. -func (w *configFileClient) tryWatchConfig(ctx context.Context) bool { +func (w *ConfigFileClient) tryWatchConfig(ctx context.Context) bool { w.log.Tracef("tryWatchConfig") config := w.configFile diff --git a/pkg/config/server/api/server.gen.go b/pkg/config/server/api/server.gen.go new file mode 100644 index 00000000..ab4afe57 --- /dev/null +++ b/pkg/config/server/api/server.gen.go @@ -0,0 +1,386 @@ +// Package api provides primitives to interact with the openapi HTTP API. +// +// Code generated by github.com/deepmap/oapi-codegen/v2 version v2.0.0 DO NOT EDIT. +package api + +import ( + "bytes" + "compress/gzip" + "encoding/base64" + "fmt" + "net/http" + "net/url" + "path" + "strings" + + "github.com/getkin/kin-openapi/openapi3" + "github.com/gorilla/mux" + stunnerv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/oapi-codegen/runtime" +) + +// V1Config Config provides a STUNner config. Schema is defined in https://github.com/l7mp/stunner/tree/main/pkg/apis/v1 +type V1Config = stunnerv1.StunnerConfig + +// ListV1ConfigsParams defines parameters for ListV1Configs. +type ListV1ConfigsParams struct { + // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` +} + +// ListV1ConfigsNamespaceParams defines parameters for ListV1ConfigsNamespace. +type ListV1ConfigsNamespaceParams struct { + // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` +} + +// GetV1ConfigNamespaceNameParams defines parameters for GetV1ConfigNamespaceName. +type GetV1ConfigNamespaceNameParams struct { + // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. + Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` +} + +// ServerInterface represents all server handlers. +type ServerInterface interface { + + // (GET /api/v1/configs) + ListV1Configs(w http.ResponseWriter, r *http.Request, params ListV1ConfigsParams) + + // (GET /api/v1/configs/{namespace}) + ListV1ConfigsNamespace(w http.ResponseWriter, r *http.Request, namespace string, params ListV1ConfigsNamespaceParams) + + // (GET /api/v1/configs/{namespace}/{name}) + GetV1ConfigNamespaceName(w http.ResponseWriter, r *http.Request, namespace string, name string, params GetV1ConfigNamespaceNameParams) +} + +// ServerInterfaceWrapper converts contexts to parameters. +type ServerInterfaceWrapper struct { + Handler ServerInterface + HandlerMiddlewares []MiddlewareFunc + ErrorHandlerFunc func(w http.ResponseWriter, r *http.Request, err error) +} + +type MiddlewareFunc func(http.Handler) http.Handler + +// ListV1Configs operation middleware +func (siw *ServerInterfaceWrapper) ListV1Configs(w http.ResponseWriter, r *http.Request) { + ctx := r.Context() + + var err error + + // Parameter object where we will unmarshal all parameters from the context + var params ListV1ConfigsParams + + // ------------- Optional query parameter "watch" ------------- + + err = runtime.BindQueryParameter("form", true, false, "watch", r.URL.Query(), ¶ms.Watch) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "watch", Err: err}) + return + } + + handler := http.Handler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + siw.Handler.ListV1Configs(w, r, params) + })) + + for i := len(siw.HandlerMiddlewares) - 1; i >= 0; i-- { + handler = siw.HandlerMiddlewares[i](handler) + } + + handler.ServeHTTP(w, r.WithContext(ctx)) +} + +// ListV1ConfigsNamespace operation middleware +func (siw *ServerInterfaceWrapper) ListV1ConfigsNamespace(w http.ResponseWriter, r *http.Request) { + ctx := r.Context() + + var err error + + // ------------- Path parameter "namespace" ------------- + var namespace string + + err = runtime.BindStyledParameter("simple", false, "namespace", mux.Vars(r)["namespace"], &namespace) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "namespace", Err: err}) + return + } + + // Parameter object where we will unmarshal all parameters from the context + var params ListV1ConfigsNamespaceParams + + // ------------- Optional query parameter "watch" ------------- + + err = runtime.BindQueryParameter("form", true, false, "watch", r.URL.Query(), ¶ms.Watch) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "watch", Err: err}) + return + } + + handler := http.Handler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + siw.Handler.ListV1ConfigsNamespace(w, r, namespace, params) + })) + + for i := len(siw.HandlerMiddlewares) - 1; i >= 0; i-- { + handler = siw.HandlerMiddlewares[i](handler) + } + + handler.ServeHTTP(w, r.WithContext(ctx)) +} + +// GetV1ConfigNamespaceName operation middleware +func (siw *ServerInterfaceWrapper) GetV1ConfigNamespaceName(w http.ResponseWriter, r *http.Request) { + ctx := r.Context() + + var err error + + // ------------- Path parameter "namespace" ------------- + var namespace string + + err = runtime.BindStyledParameter("simple", false, "namespace", mux.Vars(r)["namespace"], &namespace) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "namespace", Err: err}) + return + } + + // ------------- Path parameter "name" ------------- + var name string + + err = runtime.BindStyledParameter("simple", false, "name", mux.Vars(r)["name"], &name) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "name", Err: err}) + return + } + + // Parameter object where we will unmarshal all parameters from the context + var params GetV1ConfigNamespaceNameParams + + // ------------- Optional query parameter "watch" ------------- + + err = runtime.BindQueryParameter("form", true, false, "watch", r.URL.Query(), ¶ms.Watch) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "watch", Err: err}) + return + } + + handler := http.Handler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + siw.Handler.GetV1ConfigNamespaceName(w, r, namespace, name, params) + })) + + for i := len(siw.HandlerMiddlewares) - 1; i >= 0; i-- { + handler = siw.HandlerMiddlewares[i](handler) + } + + handler.ServeHTTP(w, r.WithContext(ctx)) +} + +type UnescapedCookieParamError struct { + ParamName string + Err error +} + +func (e *UnescapedCookieParamError) Error() string { + return fmt.Sprintf("error unescaping cookie parameter '%s'", e.ParamName) +} + +func (e *UnescapedCookieParamError) Unwrap() error { + return e.Err +} + +type UnmarshalingParamError struct { + ParamName string + Err error +} + +func (e *UnmarshalingParamError) Error() string { + return fmt.Sprintf("Error unmarshaling parameter %s as JSON: %s", e.ParamName, e.Err.Error()) +} + +func (e *UnmarshalingParamError) Unwrap() error { + return e.Err +} + +type RequiredParamError struct { + ParamName string +} + +func (e *RequiredParamError) Error() string { + return fmt.Sprintf("Query argument %s is required, but not found", e.ParamName) +} + +type RequiredHeaderError struct { + ParamName string + Err error +} + +func (e *RequiredHeaderError) Error() string { + return fmt.Sprintf("Header parameter %s is required, but not found", e.ParamName) +} + +func (e *RequiredHeaderError) Unwrap() error { + return e.Err +} + +type InvalidParamFormatError struct { + ParamName string + Err error +} + +func (e *InvalidParamFormatError) Error() string { + return fmt.Sprintf("Invalid format for parameter %s: %s", e.ParamName, e.Err.Error()) +} + +func (e *InvalidParamFormatError) Unwrap() error { + return e.Err +} + +type TooManyValuesForParamError struct { + ParamName string + Count int +} + +func (e *TooManyValuesForParamError) Error() string { + return fmt.Sprintf("Expected one value for %s, got %d", e.ParamName, e.Count) +} + +// Handler creates http.Handler with routing matching OpenAPI spec. +func Handler(si ServerInterface) http.Handler { + return HandlerWithOptions(si, GorillaServerOptions{}) +} + +type GorillaServerOptions struct { + BaseURL string + BaseRouter *mux.Router + Middlewares []MiddlewareFunc + ErrorHandlerFunc func(w http.ResponseWriter, r *http.Request, err error) +} + +// HandlerFromMux creates http.Handler with routing matching OpenAPI spec based on the provided mux. +func HandlerFromMux(si ServerInterface, r *mux.Router) http.Handler { + return HandlerWithOptions(si, GorillaServerOptions{ + BaseRouter: r, + }) +} + +func HandlerFromMuxWithBaseURL(si ServerInterface, r *mux.Router, baseURL string) http.Handler { + return HandlerWithOptions(si, GorillaServerOptions{ + BaseURL: baseURL, + BaseRouter: r, + }) +} + +// HandlerWithOptions creates http.Handler with additional options +func HandlerWithOptions(si ServerInterface, options GorillaServerOptions) http.Handler { + r := options.BaseRouter + + if r == nil { + r = mux.NewRouter() + } + if options.ErrorHandlerFunc == nil { + options.ErrorHandlerFunc = func(w http.ResponseWriter, r *http.Request, err error) { + http.Error(w, err.Error(), http.StatusBadRequest) + } + } + wrapper := ServerInterfaceWrapper{ + Handler: si, + HandlerMiddlewares: options.Middlewares, + ErrorHandlerFunc: options.ErrorHandlerFunc, + } + + r.HandleFunc(options.BaseURL+"/api/v1/configs", wrapper.ListV1Configs).Methods("GET") + + r.HandleFunc(options.BaseURL+"/api/v1/configs/{namespace}", wrapper.ListV1ConfigsNamespace).Methods("GET") + + r.HandleFunc(options.BaseURL+"/api/v1/configs/{namespace}/{name}", wrapper.GetV1ConfigNamespaceName).Methods("GET") + + return r +} + +// Base64 encoded, gzipped, json marshaled Swagger object +var swaggerSpec = []string{ + + "H4sIAAAAAAAC/+xWTY/kNBD9K6WCYybuZoWQgpBYjRBaLRrQTA97QHtwO9WJl8T22pXMjkb576ic9Md2", + "L1rEcEFwaiddqffqvWfZT2h8H7wjxwmrJ0ympV7n5bgur73b2UYeakom2sDWO6xwfg8h+tHWlEDD3eb+", + "xlEEk/8p4S73AZugpp11VIN10DKHVCnVWG6HbWl8r7pv+qASD85RVByJVK+tU+H3RulgkxrXWOCHq8Zf", + "8WMgrHCpHdfl3bxaOJ5UXdk++MhC2+n+o4+wwKC5xQr/jMMp8jRNU4HW7fylBJuWDkPXmnXotCN4+csr", + "oA/BpyxKpFmOIeptR9APHdurED1747vlsbOJSZps7m9vIFEcrSHY+QhvaHu7uYaeaqvBuoaSQJdYIFvu", + "ZKzbH+42GVLKL8mYvTIjxTSzXpercoVTgT6Q08FihS/KVfli0SX7LuOrca3mz/OrhvhSAQFmDzIA+AgP", + "mk27YILfviPDSUzXXQdiQwraUBL6PlDU0uRVjRX+ZBP/ur5ewIRH1D0xxYTVb+eYbzKIjGtaLZIIA24J", + "5rIt1RAp+SEaMcDJEw/RSUkPWjxJHEn34Heg67qAIdSaqVhqez8SOM92Z01mmPlaQX4/UHzEYp+oPC0W", + "y34RcZZ8br3vSDucprcFRkrBu0RZxK9X60sR753tQ0c9OaZ6CdyZAerpIN/0bDMgBTIy3tGTz1hys6/7", + "nDeHQhFXPGk004N+zHY9tPZIyCaI9H6gxFQfBM778qCvO0GVWhupxorjQJ/QPHG0rsFpKv7Py8d5mdfP", + "i42od5Kbvat/OU0/0iFMh4jI4t+YJ+H0T9H57yb7q9VKfox3TC6nUofQLUDqXZIBn04afhlphxV+oY53", + "FbVcVNTxliJb4bzPt/OQ380k/1bTqThT/ufXed9NBcppvc/uEDusEGVW1s2n8vwSXg9bio6Y0nKqn0Zo", + "Pu5LuPaOteEK5N7xvdxNSuuPci/HPE5vpz8CAAD//xcDuTG8CQAA", +} + +// GetSwagger returns the content of the embedded swagger specification file +// or error if failed to decode +func decodeSpec() ([]byte, error) { + zipped, err := base64.StdEncoding.DecodeString(strings.Join(swaggerSpec, "")) + if err != nil { + return nil, fmt.Errorf("error base64 decoding spec: %w", err) + } + zr, err := gzip.NewReader(bytes.NewReader(zipped)) + if err != nil { + return nil, fmt.Errorf("error decompressing spec: %w", err) + } + var buf bytes.Buffer + _, err = buf.ReadFrom(zr) + if err != nil { + return nil, fmt.Errorf("error decompressing spec: %w", err) + } + + return buf.Bytes(), nil +} + +var rawSpec = decodeSpecCached() + +// a naive cached of a decoded swagger spec +func decodeSpecCached() func() ([]byte, error) { + data, err := decodeSpec() + return func() ([]byte, error) { + return data, err + } +} + +// Constructs a synthetic filesystem for resolving external references when loading openapi specifications. +func PathToRawSpec(pathToFile string) map[string]func() ([]byte, error) { + res := make(map[string]func() ([]byte, error)) + if len(pathToFile) > 0 { + res[pathToFile] = rawSpec + } + + return res +} + +// GetSwagger returns the Swagger specification corresponding to the generated code +// in this file. The external references of Swagger specification are resolved. +// The logic of resolving external references is tightly connected to "import-mapping" feature. +// Externally referenced files must be embedded in the corresponding golang packages. +// Urls can be supported but this task was out of the scope. +func GetSwagger() (swagger *openapi3.T, err error) { + resolvePath := PathToRawSpec("") + + loader := openapi3.NewLoader() + loader.IsExternalRefsAllowed = true + loader.ReadFromURIFunc = func(loader *openapi3.Loader, url *url.URL) ([]byte, error) { + pathToFile := url.String() + pathToFile = path.Clean(pathToFile) + getSpec, ok := resolvePath[pathToFile] + if !ok { + err1 := fmt.Errorf("path not found: %s", pathToFile) + return nil, err1 + } + return getSpec() + } + var specData []byte + specData, err = rawSpec() + if err != nil { + return + } + swagger, err = loader.LoadFromData(specData) + if err != nil { + return + } + return +} diff --git a/pkg/config/server/cfg.yaml b/pkg/config/server/cfg.yaml new file mode 100644 index 00000000..d98f1968 --- /dev/null +++ b/pkg/config/server/cfg.yaml @@ -0,0 +1,8 @@ +package: api +generate: + gorilla-server: true + models: true + embedded-spec: true +output: api/server.gen.go +compatibility: + apply-gorilla-middleware-first-to-last: true diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go new file mode 100644 index 00000000..da12134a --- /dev/null +++ b/pkg/config/server/config.go @@ -0,0 +1,122 @@ +package server + +import ( + "fmt" + "sync" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +type ConfigList struct { + Version string `json:"version"` + Items []*stnrv1.StunnerConfig `json:"items"` +} + +type Config struct { + Id string + Config *stnrv1.StunnerConfig +} + +func (e Config) String() string { + return fmt.Sprintf("id=%s: %s", e.Id, e.Config.String()) +} + +func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { + s.configCh <- Config{Id: id, Config: c} +} + +// DeleteConfig should remove a config from the client. Theoretically, this would be done by +// sending the client a zero-config. However, in order to avoid that a client being removed and +// entering the graceful shutdown cycle receive a zeroconfig and abruprly kill all listeners with +// all active connections allocated to it, currently we suppress the config update. +func (s *Server) DeleteConfig(id string) { + s.log.Info("suppressing config update for terminating client", "client", id) + // s.configCh <- Config{Id: id, Config: client.ZeroConfig(id)} +} + +// UpdateConfig receives a set of ids and newConfigs that represent the state-of-the-world at a +// particular instance of time and generates an update per each change. +func (s *Server) UpdateConfig(newConfigs []Config) error { + oldNewConfigs := s.configs.Snapshot() + + for _, oldC := range oldNewConfigs { + found := false + for _, newC := range newConfigs { + if oldC.Id == newC.Id { + if !oldC.Config.DeepEqual(newC.Config) { + s.log.V(2).Info("updating config", "client", newC.Id, "config", + newC.Config.String()) + + s.configs.Upsert(newC.Id, newC.Config) + s.UpsertConfig(newC.Id, newC.Config) + } + found = true + break + } + } + + if !found { + s.log.V(2).Info("removing config", "client", oldC.Id) + + s.configs.Delete(oldC.Id) + s.DeleteConfig(oldC.Id) + } + } + + for _, newC := range newConfigs { + found := false + for _, oldC := range oldNewConfigs { + if oldC.Id == newC.Id { + found = true + break + } + } + + if !found { + s.log.V(2).Info("adding config", "client", newC.Id, "config", newC.Config) + + s.configs.Upsert(newC.Id, newC.Config) + s.UpsertConfig(newC.Id, newC.Config) + } + } + + return nil +} + +type ConfigStore struct { + configs map[string]*stnrv1.StunnerConfig + lock sync.RWMutex +} + +func NewConfigStore() *ConfigStore { + return &ConfigStore{ + configs: make(map[string]*stnrv1.StunnerConfig), + } +} + +func (t *ConfigStore) Get(id string) *stnrv1.StunnerConfig { + t.lock.RLock() + defer t.lock.RUnlock() + return t.configs[id] +} + +func (t *ConfigStore) Snapshot() []Config { + t.lock.RLock() + defer t.lock.RUnlock() + ret := []Config{} + for id, c := range t.configs { + ret = append(ret, Config{Id: id, Config: c}) + } + return ret +} + +func (t *ConfigStore) Upsert(id string, c *stnrv1.StunnerConfig) { + t.lock.Lock() + defer t.lock.Unlock() + t.configs[id] = c +} +func (t *ConfigStore) Delete(id string) { + t.lock.Lock() + defer t.lock.Unlock() + delete(t.configs, id) +} diff --git a/pkg/config/server/conn.go b/pkg/config/server/conn.go new file mode 100644 index 00000000..14c39588 --- /dev/null +++ b/pkg/config/server/conn.go @@ -0,0 +1,84 @@ +package server + +import ( + "fmt" + "sync" + + "github.com/gorilla/websocket" +) + +type Conn struct { + *websocket.Conn + Filter FilterConfig + readLock, writeLock sync.Mutex // for writemessage +} + +func NewConn(conn *websocket.Conn, filter FilterConfig) *Conn { + return &Conn{ + Conn: conn, + Filter: filter, + } +} + +func (c *Conn) Id() string { + return fmt.Sprintf("%s:%s", c.RemoteAddr().Network(), c.RemoteAddr().String()) +} + +func (c *Conn) WriteMessage(messageType int, data []byte) error { + c.writeLock.Lock() + defer c.writeLock.Unlock() + return c.Conn.WriteMessage(messageType, data) +} + +func (c *Conn) ReadMessage() (int, []byte, error) { + c.readLock.Lock() + defer c.readLock.Unlock() + return c.Conn.ReadMessage() +} + +type ConnTrack struct { + conns []*Conn + lock sync.RWMutex +} + +func NewConnTrack() *ConnTrack { + return &ConnTrack{ + conns: []*Conn{}, + } +} + +func (t *ConnTrack) Get(cid string) *Conn { + t.lock.RLock() + defer t.lock.RUnlock() + for _, c := range t.conns { + if c.Id() == cid { + return c + } + } + return nil +} + +func (t *ConnTrack) Upsert(c *Conn) { + t.lock.Lock() + defer t.lock.Unlock() + t.conns = append(t.conns, c) +} + +func (t *ConnTrack) Delete(conn *Conn) { + id := conn.Id() + t.lock.Lock() + defer t.lock.Unlock() + for i, c := range t.conns { + if c.Id() == id { + t.conns = append(t.conns[:i], t.conns[i+1:]...) + } + } +} + +func (t *ConnTrack) Snapshot() []*Conn { + t.lock.RLock() + defer t.lock.RUnlock() + ret := make([]*Conn, len(t.conns)) + copy(ret, t.conns) + return ret +} diff --git a/pkg/config/server/error.go b/pkg/config/server/error.go new file mode 100644 index 00000000..bf0dae65 --- /dev/null +++ b/pkg/config/server/error.go @@ -0,0 +1,34 @@ +package server + +import ( + "encoding/json" + "net/http" +) + +// Error defines model for Error. +type Error struct { + // Code Error code + Code int32 `json:"code"` + + // Message Error message + Message string `json:"message"` +} + +func (e Error) Error() string { return e.Message } + +// sendServerError wraps sending of an error in the Error format, and handling the failure to +// marshal that. +func sendServerError(w http.ResponseWriter, message string, code int) { + err := &Error{ + Code: int32(code), + Message: message, + } + sendServerErrorRaw(w, err) +} + +// sendServerError wraps sending of an error in the Error format, and handling the failure to +// marshal that. +func sendServerErrorRaw(w http.ResponseWriter, err *Error) { + w.WriteHeader(int(err.Code)) + _ = json.NewEncoder(w).Encode(err) +} diff --git a/pkg/config/server/handler.go b/pkg/config/server/handler.go new file mode 100644 index 00000000..9ac4829c --- /dev/null +++ b/pkg/config/server/handler.go @@ -0,0 +1,123 @@ +package server + +import ( + "encoding/json" + "fmt" + "net/http" + "strings" + + "github.com/l7mp/stunner/pkg/config/server/api" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +// make sure the server satisfies the generate OpenAPI server interface +var _ api.ServerInterface = &Server{} + +// ResponseGen is a callback to generate a response to a request (also used for sending the initial config dump for watches). +type ResponseGen func() ([]byte, *Error) + +// FilterConfig is a callback to filter config updates for a client. +type FilterConfig func(confId string) bool + +// (GET /api/v1/configs) +func (s *Server) ListV1Configs(w http.ResponseWriter, r *http.Request, params api.ListV1ConfigsParams) { + endpoint := "/api/v1/configs" + + responder := func() ([]byte, *Error) { + configs := s.configs.Snapshot() + response := ConfigList{Version: "v1", Items: []*stnrv1.StunnerConfig{}} + for _, c := range configs { + response.Items = append(response.Items, c.Config) + } + + json, err := json.Marshal(response) + if err != nil { + return []byte{}, &Error{ + Code: http.StatusInternalServerError, + Message: "Could not JSON marshal config list", + } + } + + return json, nil + } + + filter := func(confId string) bool { + return true + } + + if params.Watch != nil && *params.Watch { + s.handleConn(w, r, endpoint, responder, filter) + } else { + s.handleReq(w, r, endpoint, responder) + } +} + +// (GET /api/v1/configs/{namespace}) +func (s *Server) ListV1ConfigsNamespace(w http.ResponseWriter, r *http.Request, namespace string, params api.ListV1ConfigsNamespaceParams) { + endpoint := fmt.Sprintf("/api/v1/configs/%s", namespace) + + responder := func() ([]byte, *Error) { + configs := s.configs.Snapshot() + response := ConfigList{Version: "v1", Items: []*stnrv1.StunnerConfig{}} + for _, c := range configs { + ps := strings.Split(c.Id, "/") + if len(ps) == 2 && ps[0] == namespace { + response.Items = append(response.Items, c.Config) + } + } + + json, err := json.Marshal(response) + if err != nil { + return []byte{}, &Error{ + Code: http.StatusInternalServerError, + Message: "Could not JSON marshal config list", + } + } + + return json, nil + } + filter := func(confId string) bool { + ps := strings.Split(confId, "/") + return len(ps) == 2 && ps[0] == namespace + } + + if params.Watch != nil && *params.Watch { + s.handleConn(w, r, endpoint, responder, filter) + } else { + s.handleReq(w, r, endpoint, responder) + } +} + +// (GET /api/v1/configs/{namespace}/{name}) +func (s *Server) GetV1ConfigNamespaceName(w http.ResponseWriter, r *http.Request, namespace string, name string, params api.GetV1ConfigNamespaceNameParams) { + id := fmt.Sprintf("%s/%s", namespace, name) + endpoint := fmt.Sprintf("/api/v1/configs/%s/%s", namespace, name) + + responder := func() ([]byte, *Error) { + c := s.configs.Get(id) + if c == nil { + return []byte{}, &Error{ + Code: http.StatusBadRequest, + Message: "Config not found", + } + } + + json, err := json.Marshal(c) + if err != nil { + return []byte{}, &Error{ + Code: http.StatusInternalServerError, + Message: "Config not found", + } + } + + return json, nil + } + filter := func(confId string) bool { return confId == id } + + if params.Watch != nil && *params.Watch { + s.handleConn(w, r, endpoint, responder, filter) + } else { + s.handleReq(w, r, endpoint, responder) + } +} diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go new file mode 100644 index 00000000..4c2594e2 --- /dev/null +++ b/pkg/config/server/server.go @@ -0,0 +1,207 @@ +//go:generate go run github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen --config=cfg.yaml ../api/stunner_openapi.yaml +package server + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net" + "net/http" + + "github.com/go-logr/logr" + "github.com/gorilla/mux" + "github.com/gorilla/websocket" + "github.com/l7mp/stunner/pkg/config/server/api" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +type Server struct { + *http.Server + addr string + conns *ConnTrack + configs *ConfigStore + configCh chan Config + log logr.Logger +} + +func New(addr string, logger logr.Logger) *Server { + if addr == "" { + addr = stnrv1.DefaultConfigDiscoveryAddress + } + + cds := &Server{ + conns: NewConnTrack(), + configs: NewConfigStore(), + configCh: make(chan Config, 8), + addr: addr, + log: logger, + } + return cds +} + +func (s *Server) Start(ctx context.Context) error { + r := mux.NewRouter() + api.HandlerFromMux(s, r) + s.Server = &http.Server{Addr: s.addr, Handler: r} + + // serve + go func() { + s.log.Info("Starting CDS server", "address", s.addr) + + err := s.ListenAndServe() + if err != nil { + if errors.Is(err, net.ErrClosed) || errors.Is(err, http.ErrServerClosed) { + s.log.Info("closing config discovery server") + } else { + s.log.Error(err, "error closing config discovery server", "error", err.Error()) + } + return + } + }() + + // listen to config update events and cancel requests + go func() { + defer close(s.configCh) + defer s.Close() + + for { + select { + case e := <-s.configCh: + s.broadcastConfig(e) + case <-ctx.Done(): + return + } + } + }() + + return nil +} + +// Close closes the server and drops all active connections. +func (s *Server) Close() { + // first close the underlying HTTP server so that we do not get any new connnections + s.Server.Close() + // then kill all active connections + for _, conn := range s.conns.Snapshot() { + s.closeConn(conn) + } +} + +func (s *Server) GetConfigChannel() chan Config { + return s.configCh +} + +func (s *Server) handleReq(w http.ResponseWriter, r *http.Request, endpoint string, responder ResponseGen) { + s.log.V(1).Info("received new request", "api", endpoint, "client", r.RemoteAddr) + + response, err := responder() + if err != nil { + s.log.Error(err, "error generating client response", "api", endpoint, + "client", r.RemoteAddr, "code", err.Code) + sendServerErrorRaw(w, err) + } + + s.log.V(2).Info("sending response to client", "api", endpoint, "client", r.RemoteAddr, + "response", string(response)) + + if _, err := w.Write(response); err != nil { + s.log.Error(err, "could not write config", "api", endpoint, "client", r.RemoteAddr) + http.Error(w, "Could not write config", http.StatusInternalServerError) + return + } +} + +func (s *Server) handleConn(w http.ResponseWriter, r *http.Request, endpoint string, responder ResponseGen, filter FilterConfig) { + // upgrade to webSocket + upgrader := websocket.Upgrader{ + ReadBufferSize: 1024, + WriteBufferSize: 1024, + } + + wsConn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + msg := fmt.Sprintf("could not upgrade HTTP connection for client %s: %s", + r.RemoteAddr, err.Error()) + sendServerError(w, msg, http.StatusInternalServerError) + return + } + conn := NewConn(wsConn, filter) + s.conns.Upsert(conn) + + // a dummy reader that drops everything it receives: this must be there for the + // WebSocket server to call our pong-handler: conn.Close() will kill this goroutine + go func() { + for { + // drop anything we receive + _, _, err := conn.ReadMessage() + if err != nil { + return + } + } + }() + + conn.SetPingHandler(func(string) error { + return conn.WriteMessage(websocket.PongMessage, []byte("keepalive")) + }) + + s.log.V(1).Info("new config stream connection", "api", endpoint, "client", conn.Id()) + + // send initial config(list) + for _, conf := range s.configs.Snapshot() { + if filter(conf.Id) { + s.sendConfig(conn, conf) + } + } + + // wait until client closes the connection or the server is cancelled (which will kill all + // the running connections) + <-r.Context().Done() + + s.log.V(1).Info("client connection closed", "api", endpoint, + "client", r.RemoteAddr) + + conn.Close() +} + +// iterate through all connections and send response if needed +func (s *Server) broadcastConfig(e Config) { + json, err := json.Marshal(e.Config) + if err != nil { + s.log.Error(err, "error JSON marshaling config", "event", e.String()) + return + } + + for _, conn := range s.conns.Snapshot() { + if conn.Filter(e.Id) { + if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { + s.log.Error(err, "error sending config update", + "client", conn.Id()) + s.closeConn(conn) + } + } + } +} + +// iterate through all connections and send response if needed +func (s *Server) sendConfig(conn *Conn, e Config) { + json, err := json.Marshal(e.Config) + if err != nil { + s.log.Error(err, "error JSON marshaling config", "event", e.String()) + return + } + if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { + s.log.Error(err, "error sending config update", + "client", conn.Id()) + s.closeConn(conn) + } +} + +func (s *Server) closeConn(conn *Conn) { + s.log.V(1).Info("closing client connection", "client", conn.Id()) + + conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck + s.conns.Delete(conn) + conn.Close() +} diff --git a/pkg/config/server/server_test.go b/pkg/config/server/server_test.go new file mode 100644 index 00000000..5ca08ccf --- /dev/null +++ b/pkg/config/server/server_test.go @@ -0,0 +1,842 @@ +package server + +import ( + "context" + "testing" + "time" + + "github.com/go-logr/zapr" + "github.com/stretchr/testify/assert" + "go.uber.org/zap" + "go.uber.org/zap/zapcore" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/config/client" + "github.com/l7mp/stunner/pkg/logger" +) + +var testerLogLevel = zapcore.Level(-4) + +//var testerLogLevel = zapcore.DebugLevel +//var testerLogLevel = zapcore.ErrorLevel + +const stunnerLogLevel = "all:TRACE" + +func init() { + // setup a fast pinger so that we get a timely error notification + client.PingPeriod = 500 * time.Millisecond + client.PongWait = 800 * time.Millisecond + client.WriteWait = 200 * time.Millisecond + client.RetryPeriod = 250 * time.Millisecond + +} + +func TestServerLoad(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + testLog.Debug("create server") + server := New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, server, "server") + err = server.Start(ctx) + assert.NoError(t, err, "start") + + time.Sleep(20 * time.Millisecond) + + testLog.Debug("create client") + client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + assert.NoError(t, err, "client 1") + client2, err := client.New("127.0.0.1:13478", "ns1/gw2", logger) + assert.NoError(t, err, "client 2") + // nonexistent + client3, err := client.New("127.0.0.1:13478", "ns1/gw3", logger) + assert.NoError(t, err, "client 3") + + testLog.Debug("load: error") + c, err := client1.Load() + assert.Error(t, err, "load") + assert.Nil(t, c, "conf") + c, err = client2.Load() + assert.Error(t, err, "load") + assert.Nil(t, c, "conf") + c, err = client3.Load() + assert.Error(t, err, "load") + assert.Nil(t, c, "conf") + + c1 := testConfig("ns1/gw1", "realm1") + c2 := testConfig("ns1/gw2", "realm1") + err = server.UpdateConfig([]Config{c1, c2}) + assert.NoError(t, err, "update") + + cs := server.configs.Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 := server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 := server.configs.Get("ns1/gw2") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") + sc3 := server.configs.Get("ns1/gw3") + assert.Nil(t, sc3, "get 3") + + testLog.Debug("load: config ok") + c, err = client1.Load() + assert.NoError(t, err, "load") + assert.True(t, c.DeepEqual(sc1), "deepeq") + c, err = client2.Load() + assert.NoError(t, err, "load") + assert.True(t, c.DeepEqual(sc2), "deepeq") + c, err = client3.Load() + assert.Error(t, err, "load") + assert.Nil(t, c, "conf") + + testLog.Debug("remove 2 configs") + err = server.UpdateConfig([]Config{}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 0, "snapshot len") + + testLog.Debug("load: no result") + _, err = client1.Load() + assert.Error(t, err, "load") + _, err = client2.Load() + assert.Error(t, err, "load") + _, err = client3.Load() + assert.Error(t, err, "load") + assert.Nil(t, c, "conf") +} + +func TestServerPoll(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + testLog.Debug("create server") + server := New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, server, "server") + err = server.Start(ctx) + assert.NoError(t, err, "start") + + time.Sleep(20 * time.Millisecond) + + testLog.Debug("create client") + client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + assert.NoError(t, err, "client 1") + client2, err := client.New("127.0.0.1:13478", "ns1/gw2", logger) + assert.NoError(t, err, "client 2") + client3, err := client.New("127.0.0.1:13478", "ns1/gw3", logger) + assert.NoError(t, err, "client 3") + + testLog.Debug("poll: no result") + ch1 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch1) + ch2 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch2) + ch3 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch3) + + go func() { + err = client1.Poll(ctx, ch1) + assert.NoError(t, err, "client 1 cancelled") + }() + go func() { + err = client2.Poll(ctx, ch2) + assert.NoError(t, err, "client 2 cancelled") + }() + go func() { + err = client3.Poll(ctx, ch2) + assert.NoError(t, err, "client 3 cancelled") + }() + + s := watchConfig(ch1, 10*time.Millisecond) + assert.Nil(t, s, "config 1") + s = watchConfig(ch2, 10*time.Millisecond) + assert.Nil(t, s, "config 2") + s = watchConfig(ch3, 10*time.Millisecond) + assert.Nil(t, s, "config 3") + + testLog.Debug("poll: one result") + c1 := testConfig("ns1/gw1", "realm1") + c2 := testConfig("ns1/gw2", "realm1") + err = server.UpdateConfig([]Config{c1, c2}) + assert.NoError(t, err, "update") + + cs := server.configs.Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 := server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 := server.configs.Get("ns1/gw2") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") + sc3 := server.configs.Get("ns1/gw3") + assert.Nil(t, sc3, "get 3") + + // poll should have fed the configs to the channels + s = watchConfig(ch1, 500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + s = watchConfig(ch2, 500*time.Millisecond) + assert.NotNil(t, s, "config 2") + assert.True(t, s.DeepEqual(sc2), "deepeq 2") + s = watchConfig(ch3, 500*time.Millisecond) + assert.Nil(t, s, "config 3") + + testLog.Debug("remove 2 configs") + err = server.UpdateConfig([]Config{}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 0, "snapshot len") + + testLog.Debug("poll: zeroconfig") + s = watchConfig(ch1, 10*time.Millisecond) + assert.Nil(t, s, "config") + s = watchConfig(ch2, 10*time.Millisecond) + assert.Nil(t, s, "config") + s = watchConfig(ch3, 10*time.Millisecond) + assert.Nil(t, s, "config") +} + +func TestServerWatch(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + serverCtx, serverCancel := context.WithCancel(context.Background()) + + testLog.Debug("create server") + server := New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, server, "server") + err = server.Start(serverCtx) + assert.NoError(t, err, "start") + + testLog.Debug("create client") + client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + assert.NoError(t, err, "client 1") + client2, err := client.New("127.0.0.1:13478", "ns1/gw2", logger) + assert.NoError(t, err, "client 2") + client3, err := client.New("127.0.0.1:13478", "ns1/gw3", logger) + assert.NoError(t, err, "client 3") + + testLog.Debug("watch: no result") + ch1 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch1) + ch2 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch2) + ch3 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch3) + + clientCtx, clientCancel := context.WithCancel(context.Background()) + defer clientCancel() + err = client1.Watch(clientCtx, ch1) + assert.NoError(t, err, "client 1 watch") + err = client2.Watch(clientCtx, ch2) + assert.NoError(t, err, "client 2 watch") + err = client3.Watch(clientCtx, ch3) + assert.NoError(t, err, "client 3 watch") + + s := watchConfig(ch1, 150*time.Millisecond) + assert.Nil(t, s, "config 1") + s = watchConfig(ch2, 150*time.Millisecond) + assert.Nil(t, s, "config 2") + s = watchConfig(ch3, 150*time.Millisecond) + assert.Nil(t, s, "config 3") + + testLog.Debug("poll: one result") + c1 := testConfig("ns1/gw1", "realm1") + c2 := testConfig("ns1/gw2", "realm1") + err = server.UpdateConfig([]Config{c1, c2}) + assert.NoError(t, err, "update") + + cs := server.configs.Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 := server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 := server.configs.Get("ns1/gw2") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc3 := server.configs.Get("ns1/gw3") + assert.Nil(t, sc3, "get 3") + + // poll should have fed the configs to the channels + s = watchConfig(ch1, 500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + s = watchConfig(ch2, 500*time.Millisecond) + assert.NotNil(t, s, "config 2") + assert.True(t, s.DeepEqual(sc2), "deepeq 2") + s = watchConfig(ch3, 500*time.Millisecond) + assert.Nil(t, s, "config 3") + + testLog.Debug("update: conf 1 and conf 3") + c1 = testConfig("ns1/gw1", "realm-new") + c3 := testConfig("ns1/gw3", "realm3") + err = server.UpdateConfig([]Config{c1, c2, c3}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 3, "snapshot len") + sc1 = server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") + sc2 = server.configs.Get("ns1/gw2") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") + sc3 = server.configs.Get("ns1/gw3") + assert.NotNil(t, sc3, "get 3") + assert.True(t, c3.Config.DeepEqual(sc3), "deepeq 3") + + // poll should have fed the configs to the channels + s = watchConfig(ch1, 500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + s = watchConfig(ch2, 500*time.Millisecond) + assert.Nil(t, s, "config 2") + s = watchConfig(ch3, 500*time.Millisecond) + assert.NotNil(t, s, "config 3") + assert.True(t, s.DeepEqual(sc3), "deepeq 3") + + testLog.Debug("restarting server") + serverCancel() + // let the server shut down and restart + time.Sleep(50 * time.Millisecond) + serverCtx, serverCancel = context.WithCancel(context.Background()) + defer serverCancel() + server = New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, server, "server") + err = server.Start(serverCtx) + assert.NoError(t, err, "start") + err = server.UpdateConfig([]Config{c1, c2, c3}) + assert.NoError(t, err, "update") + + // obtain the initial configs: this may take a while + s = watchConfig(ch1, 5000*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + s = watchConfig(ch2, 500*time.Millisecond) + assert.NotNil(t, s, "config 2") + assert.True(t, s.DeepEqual(sc2), "deepeq 2") + s = watchConfig(ch3, 500*time.Millisecond) + assert.NotNil(t, s, "config 3") + assert.True(t, s.DeepEqual(sc3), "deepeq 3") + + testLog.Debug("remove 1 config (the 2nd)") + err = server.UpdateConfig([]Config{c1, c3}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 = server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") + sc2 = server.configs.Get("ns1/gw2") + assert.Nil(t, sc2, "get 2") + sc3 = server.configs.Get("ns1/gw3") + assert.NotNil(t, sc3, "get 3") + assert.True(t, c3.Config.DeepEqual(sc3), "deepeq 3") + + s = watchConfig(ch1, 50*time.Millisecond) + assert.Nil(t, s, "config 1") + s = watchConfig(ch2, 50*time.Millisecond) + assert.Nil(t, s, "config 2") + s = watchConfig(ch3, 50*time.Millisecond) + assert.Nil(t, s, "config 3") + + testLog.Debug("remove remaining 2 configs") + err = server.UpdateConfig([]Config{}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 0, "snapshot len") + + testLog.Debug("poll: no config") + s = watchConfig(ch1, 10*time.Millisecond) + assert.Nil(t, s, "config") + s = watchConfig(ch2, 10*time.Millisecond) + assert.Nil(t, s, "config") + s = watchConfig(ch3, 10*time.Millisecond) + assert.Nil(t, s, "config") +} + +// test APIs +func TestServerAPI(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + serverCtx, serverCancel := context.WithCancel(context.Background()) + + testLog.Debug("create server") + server := New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, server, "server") + err = server.Start(serverCtx) + assert.NoError(t, err, "start") + + testLog.Debug("create client") + client1, err := client.NewAllConfigsAPI("127.0.0.1:13478", logger.NewLogger("all-config-client")) + assert.NoError(t, err, "client 1") + client2, err := client.NewConfigsNamespaceAPI("127.0.0.1:13478", "ns1", logger.NewLogger("ns-config-client-ns1")) + assert.NoError(t, err, "client 2") + client3, err := client.NewConfigsNamespaceAPI("127.0.0.1:13478", "ns2", logger.NewLogger("ns-config-client-ns2")) + assert.NoError(t, err, "client 3") + client4, err := client.NewConfigNamespaceNameAPI("127.0.0.1:13478", "ns1", "gw1", logger.NewLogger("gw-config-client")) + assert.NoError(t, err, "client 4") + + testLog.Debug("watch: no result") + ch1 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch1) + ch2 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch2) + ch3 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch3) + ch4 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch4) + + clientCtx, clientCancel := context.WithCancel(context.Background()) + defer clientCancel() + err = client1.Watch(clientCtx, ch1) + assert.NoError(t, err, "client 1 watch") + err = client2.Watch(clientCtx, ch2) + assert.NoError(t, err, "client 2 watch") + err = client3.Watch(clientCtx, ch3) + assert.NoError(t, err, "client 3 watch") + err = client4.Watch(clientCtx, ch4) + assert.NoError(t, err, "client 4 watch") + + s := watchConfig(ch1, 50*time.Millisecond) + assert.Nil(t, s, "config 1") + s = watchConfig(ch2, 50*time.Millisecond) + assert.Nil(t, s, "config 2") + s = watchConfig(ch3, 50*time.Millisecond) + assert.Nil(t, s, "config 3") + s = watchConfig(ch4, 50*time.Millisecond) + assert.Nil(t, s, "config 4") + + testLog.Debug("--------------------------------") + testLog.Debug("Update1: ns1/gw1 + ns2/gw1 ") + testLog.Debug("--------------------------------") + testLog.Debug("poll: one result") + c1 := testConfig("ns1/gw1", "realm1") + c2 := testConfig("ns2/gw1", "realm1") + err = server.UpdateConfig([]Config{c1, c2}) + assert.NoError(t, err, "update") + + cs := server.configs.Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 := server.configs.Get("ns1/gw1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") + assert.NotNil(t, sc1, "get 1") + sc2 := server.configs.Get("ns2/gw1") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") + + testLog.Debug("load") + + // all-configs should result sc1 and sc2 + scs, err := client1.Get(clientCtx) + assert.NoError(t, err, "load 1") + assert.Len(t, scs, 2, "load 1") + co := findConfById(scs, "ns1/gw1") + assert.NotNil(t, co, "c1") + assert.True(t, co.DeepEqual(sc1), "deepeq") + co = findConfById(scs, "ns2/gw1") + assert.NotNil(t, co, "c2") + assert.True(t, co.DeepEqual(sc2), "deepeq") + + // ns1 client should yield 1 config + scs, err = client2.Get(clientCtx) + assert.NoError(t, err, "load 2") + assert.Len(t, scs, 1, "load 2") + assert.True(t, scs[0].DeepEqual(sc1), "deepeq") + + // ns2 client should yield 1 config + scs, err = client3.Get(clientCtx) + assert.NoError(t, err, "load 3") + assert.Len(t, scs, 1, "load 3") + assert.True(t, scs[0].DeepEqual(sc2), "deepeq") + + // ns1/gw1 client should yield 1 config + scs, err = client4.Get(clientCtx) + assert.NoError(t, err, "load 4") + assert.Len(t, scs, 1, "load 4") + assert.True(t, scs[0].DeepEqual(sc1), "deepeq") + + // two configs from client1 watch + s1 := watchConfig(ch1, 50*time.Millisecond) + assert.NotNil(t, s1) + s2 := watchConfig(ch1, 50*time.Millisecond) + assert.NotNil(t, s2) + s3 := watchConfig(ch1, 50*time.Millisecond) + assert.Nil(t, s3) + lst := []*stnrv1.StunnerConfig{s1, s2} + assert.NotNil(t, findConfById(lst, "ns1/gw1")) + assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq 1") + assert.NotNil(t, findConfById(lst, "ns2/gw1")) + assert.True(t, findConfById(lst, "ns2/gw1").DeepEqual(sc2), "deepeq 1") + + // 1 config from client2 watch + s = watchConfig(ch2, 50*time.Millisecond) + assert.NotNil(t, s) + assert.True(t, s.DeepEqual(sc1)) + s = watchConfig(ch2, 50*time.Millisecond) + assert.Nil(t, s) + + // 1 config from client3 watch + s = watchConfig(ch3, 50*time.Millisecond) + assert.NotNil(t, s, "config 3") + assert.True(t, s.DeepEqual(sc2)) + s = watchConfig(ch3, 50*time.Millisecond) + assert.Nil(t, s) + + // 1 config from client4 watch + s = watchConfig(ch4, 50*time.Millisecond) + assert.NotNil(t, s) + assert.True(t, s.DeepEqual(sc1)) + s = watchConfig(ch4, 50*time.Millisecond) + assert.Nil(t, s) + + testLog.Debug("--------------------------------") + testLog.Debug("Update1: ns1/gw1 + ns1/gw2 ") + testLog.Debug("--------------------------------") + testLog.Debug("update: conf 1 and conf 3") + c1 = testConfig("ns1/gw1", "realm-new") + c3 := testConfig("ns1/gw2", "realm3") + err = server.UpdateConfig([]Config{c1, c2, c3}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 3, "snapshot len") + sc1 = server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 = server.configs.Get("ns2/gw1") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") + sc3 := server.configs.Get("ns1/gw2") + assert.NotNil(t, sc3, "get 3") + assert.True(t, c3.Config.DeepEqual(sc3), "deepeq") + + // all-configs should result sc1 and sc2 and sc3 + scs, err = client1.Get(clientCtx) + assert.NoError(t, err, "load 1") + assert.Len(t, scs, 3, "load 1") + co = findConfById(scs, "ns1/gw1") + assert.NotNil(t, co, "c1") + assert.True(t, co.DeepEqual(sc1), "deepeq") + co = findConfById(scs, "ns2/gw1") + assert.NotNil(t, co, "c2") + assert.True(t, co.DeepEqual(sc2), "deepeq") + co = findConfById(scs, "ns1/gw2") + assert.NotNil(t, co, "c3") + assert.True(t, co.DeepEqual(sc3), "deepeq") + + // ns1 client should yield 2 configs + scs, err = client2.Get(clientCtx) + assert.NoError(t, err, "load 2") + assert.Len(t, scs, 2, "load 2") + assert.NotNil(t, findConfById(scs, "ns1/gw1")) + assert.True(t, findConfById(scs, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(scs, "ns1/gw2")) + assert.True(t, findConfById(scs, "ns1/gw2").DeepEqual(sc3), "deepeq") + + // ns2 client should yield 1 config + scs, err = client3.Get(clientCtx) + assert.NoError(t, err, "load 3") + assert.Len(t, scs, 1, "load 3") + assert.True(t, scs[0].DeepEqual(sc2), "deepeq") + + // ns1/gw1 client should yield 1 config + scs, err = client4.Get(clientCtx) + assert.NoError(t, err, "load 4") + assert.Len(t, scs, 1, "load 4") + assert.True(t, scs[0].DeepEqual(sc1), "deepeq") + + // 2 configs from client1 watch + s1 = watchConfig(ch1, 1500*time.Millisecond) + assert.NotNil(t, s1) + s2 = watchConfig(ch1, 150*time.Millisecond) + assert.NotNil(t, s2) + s3 = watchConfig(ch1, 150*time.Millisecond) + assert.Nil(t, s3) + lst = []*stnrv1.StunnerConfig{s1, s2} + assert.NotNil(t, findConfById(lst, "ns1/gw1")) + assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(lst, "ns1/gw2")) + assert.True(t, findConfById(lst, "ns1/gw2").DeepEqual(sc3), "deepeq") + + // 2 configs from client2 watch + s1 = watchConfig(ch2, 1500*time.Millisecond) + assert.NotNil(t, s1) + s2 = watchConfig(ch2, 150*time.Millisecond) + assert.NotNil(t, s2) + s3 = watchConfig(ch2, 50*time.Millisecond) + assert.Nil(t, s3) + lst = []*stnrv1.StunnerConfig{s1, s2} + assert.NotNil(t, findConfById(lst, "ns1/gw1")) + assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(lst, "ns1/gw2")) + assert.True(t, findConfById(lst, "ns1/gw2").DeepEqual(sc3), "deepeq") + + // 0 config from client3 watch + s = watchConfig(ch3, 50*time.Millisecond) + assert.Nil(t, s, "config 3") + + // 1 config from client4 watch + s = watchConfig(ch4, 50*time.Millisecond) + assert.NotNil(t, s) + assert.True(t, s.DeepEqual(sc1), "deepeq") + + testLog.Debug("--------------------------------") + testLog.Debug("Restart + Update1: ns1/gw1 + ns2/gw1 + ns1/gw2") + testLog.Debug("--------------------------------") + testLog.Debug("restarting server") + serverCancel() + // let the server shut down and restart + time.Sleep(50 * time.Millisecond) + serverCtx, serverCancel = context.WithCancel(context.Background()) + defer serverCancel() + server = New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, server, "server") + err = server.Start(serverCtx) + assert.NoError(t, err, "start") + err = server.UpdateConfig([]Config{c1, c2, c3}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 3, "snapshot len") + sc1 = server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 = server.configs.Get("ns2/gw1") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") + sc3 = server.configs.Get("ns1/gw2") + assert.NotNil(t, sc3, "get 3") + assert.True(t, c3.Config.DeepEqual(sc3), "deepeq") + + // all-configs should result sc1 and sc2 and sc3 + scs, err = client1.Get(clientCtx) + assert.NoError(t, err, "load 1") + assert.Len(t, scs, 3, "load 1") + co = findConfById(scs, "ns1/gw1") + assert.NotNil(t, co, "c1") + assert.True(t, co.DeepEqual(sc1), "deepeq") + co = findConfById(scs, "ns2/gw1") + assert.NotNil(t, co, "c2") + assert.True(t, co.DeepEqual(sc2), "deepeq") + co = findConfById(scs, "ns1/gw2") + assert.NotNil(t, co, "c3") + assert.True(t, co.DeepEqual(sc3), "deepeq") + + // ns1 client should yield 2 configs + scs, err = client2.Get(clientCtx) + assert.NoError(t, err, "load 2") + assert.Len(t, scs, 2, "load 2") + assert.NotNil(t, findConfById(scs, "ns1/gw1")) + assert.True(t, findConfById(scs, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(scs, "ns1/gw2")) + assert.True(t, findConfById(scs, "ns1/gw2").DeepEqual(sc3), "deepeq") + + // ns2 client should yield 1 config + scs, err = client3.Get(clientCtx) + assert.NoError(t, err, "load 3") + assert.Len(t, scs, 1, "load 3") + assert.True(t, scs[0].DeepEqual(sc2), "deepeq") + + // ns1/gw1 client should yield 1 config + scs, err = client4.Get(clientCtx) + assert.NoError(t, err, "load 4") + assert.Len(t, scs, 1, "load 4") + assert.True(t, scs[0].DeepEqual(sc1), "deepeq") + + // 3 configs from client1 watch + s1 = watchConfig(ch1, 5000*time.Millisecond) + assert.NotNil(t, s1) + s2 = watchConfig(ch1, 100*time.Millisecond) + assert.NotNil(t, s2) + s3 = watchConfig(ch1, 100*time.Millisecond) + assert.NotNil(t, s2) + s4 := watchConfig(ch1, 100*time.Millisecond) + assert.Nil(t, s4) + lst = []*stnrv1.StunnerConfig{s1, s2, s3} + assert.NotNil(t, findConfById(lst, "ns1/gw1")) + assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(lst, "ns1/gw2")) + assert.True(t, findConfById(lst, "ns2/gw1").DeepEqual(sc2), "deepeq") + assert.NotNil(t, findConfById(lst, "ns2/gw1")) + assert.True(t, findConfById(lst, "ns1/gw2").DeepEqual(sc3), "deepeq") + + // 2 configs from client2 watch + s1 = watchConfig(ch2, 50*time.Millisecond) + assert.NotNil(t, s1) + s2 = watchConfig(ch2, 50*time.Millisecond) + assert.NotNil(t, s2) + s3 = watchConfig(ch2, 50*time.Millisecond) + assert.Nil(t, s3) + lst = []*stnrv1.StunnerConfig{s1, s2} + assert.NotNil(t, findConfById(lst, "ns1/gw1")) + assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(lst, "ns1/gw2")) + assert.True(t, findConfById(lst, "ns1/gw2").DeepEqual(sc3), "deepeq") + + // 1 config from client3 watch + s = watchConfig(ch3, 50*time.Millisecond) + assert.NotNil(t, s, "config 3") + assert.True(t, s.DeepEqual(sc2)) + s = watchConfig(ch3, 50*time.Millisecond) + assert.Nil(t, s) + + // 1 config from client4 watch + s = watchConfig(ch4, 50*time.Millisecond) + assert.NotNil(t, s) + assert.True(t, s.DeepEqual(sc1)) + s = watchConfig(ch4, 50*time.Millisecond) + assert.Nil(t, s) + + testLog.Debug("--------------------------------") + testLog.Debug("Update1: ns1/gw1 + ns3/gw1 ") + testLog.Debug("--------------------------------") + testLog.Debug("update: conf 1, remove conf 3, and add conf 4") + c1 = testConfig("ns1/gw1", "realm-newer") + c4 := testConfig("ns3/gw1", "realm4") + err = server.UpdateConfig([]Config{c1, c2, c4}) + assert.NoError(t, err, "update") + + cs = server.configs.Snapshot() + assert.Len(t, cs, 3, "snapshot len") + sc1 = server.configs.Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 = server.configs.Get("ns2/gw1") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") + sc4 := server.configs.Get("ns3/gw1") + assert.NotNil(t, sc3, "get 3") + assert.True(t, c4.Config.DeepEqual(sc4), "deepeq") + + // all-configs should result sc1 and sc2 and sc4 + scs, err = client1.Get(clientCtx) + assert.NoError(t, err, "load 1") + assert.Len(t, scs, 3, "load 1") + co = findConfById(scs, "ns1/gw1") + assert.NotNil(t, co, "c1") + assert.True(t, co.DeepEqual(sc1), "deepeq") + co = findConfById(scs, "ns2/gw1") + assert.NotNil(t, co, "c2") + assert.True(t, co.DeepEqual(sc2), "deepeq") + co = findConfById(scs, "ns3/gw1") + assert.NotNil(t, co, "c4") + assert.True(t, co.DeepEqual(sc4), "deepeq") + + // ns1 client should yield 1 config + scs, err = client2.Get(clientCtx) + assert.NoError(t, err, "load 2") + assert.Len(t, scs, 1, "load 2") + assert.True(t, scs[0].DeepEqual(sc1), "deepeq") + + // ns2 client should yield 1 config + scs, err = client3.Get(clientCtx) + assert.NoError(t, err, "load 3") + assert.Len(t, scs, 1, "load 3") + assert.True(t, scs[0].DeepEqual(sc2), "deepeq") + + // ns1/gw1 client should yield 1 config + scs, err = client4.Get(clientCtx) + assert.NoError(t, err, "load 4") + assert.Len(t, scs, 1, "load 4") + assert.True(t, scs[0].DeepEqual(sc1), "deepeq") + + // 2 configs from client1 watch + s1 = watchConfig(ch1, 5000*time.Millisecond) + assert.NotNil(t, s1) + s2 = watchConfig(ch1, 500*time.Millisecond) + assert.NotNil(t, s2) + s3 = watchConfig(ch1, 500*time.Millisecond) + assert.Nil(t, s3) + lst = []*stnrv1.StunnerConfig{s1, s2} + assert.NotNil(t, findConfById(lst, "ns1/gw1")) + assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq") + assert.NotNil(t, findConfById(lst, "ns3/gw1")) + assert.True(t, findConfById(lst, "ns3/gw1").DeepEqual(sc4), "deepeq") + + // 1 config from client2 watch (removed config never updated) + s1 = watchConfig(ch2, 50*time.Millisecond) + assert.NotNil(t, s1) + s2 = watchConfig(ch2, 50*time.Millisecond) + assert.Nil(t, s2) + assert.True(t, s1.DeepEqual(sc1), "deepeq") + + // no config from client3 watch + s = watchConfig(ch3, 50*time.Millisecond) + assert.Nil(t, s, "config 3") + + // 1 config from client4 watch + s = watchConfig(ch4, 50*time.Millisecond) + assert.NotNil(t, s) + assert.True(t, s.DeepEqual(sc1), "deepeq") +} + +// only differ in id and realm +func testConfig(id, realm string) Config { + c := client.ZeroConfig(id) + c.Auth.Realm = realm + + return Config{id, c} +} + +// wait for some configurable time for a watch element +func watchConfig(ch chan stnrv1.StunnerConfig, d time.Duration) *stnrv1.StunnerConfig { + select { + case c := <-ch: + // fmt.Println("++++++++++++ got config ++++++++++++: ", c.String()) + return &c + case <-time.After(d): + // fmt.Println("++++++++++++ timeout ++++++++++++") + return nil + } +} + +func findConfById(cs []*stnrv1.StunnerConfig, id string) *stnrv1.StunnerConfig { + for _, c := range cs { + if c != nil && c.Admin.Name == id { + return c + } + + } + + return nil +} diff --git a/reconcile_test.go b/reconcile_test.go index 1c32c564..d4bd3466 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -543,7 +543,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, /// auth { - name: "reconcile-test: reconcile plaintextauth name", + name: "reconcile-test: reconcile staticauth name", config: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ @@ -628,7 +628,7 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, }, { - name: "reconcile-test: reconcile plaintext auth passwd", + name: "reconcile-test: reconcile static auth passwd", config: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ @@ -713,14 +713,14 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, }, { - name: "reconcile-test: reconcile longterm auth", + name: "reconcile-test: reconcile ephemeral auth", config: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "newsecret", }, @@ -1963,7 +1963,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ echoResult: true, }, { - testName: "changing plaintext credentials to a wrong passwd", + testName: "changing static credentials to a wrong passwd", config: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ @@ -2006,7 +2006,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ echoResult: false, }, { - testName: "changing auth to longterm credentials errs", + testName: "changing auth to ephemeral credentials errs", config: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ @@ -2049,7 +2049,7 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ echoResult: false, }, { - testName: "reverting good plaintext credentials ok", + testName: "reverting good static credentials ok", config: stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ diff --git a/stunner.go b/stunner.go index 335f8464..0ad49a73 100644 --- a/stunner.go +++ b/stunner.go @@ -21,7 +21,7 @@ import ( const DefaultLogLevel = "all:WARN" -var DefaultInstanceId = fmt.Sprintf("stunnerd-%s", uuid.New().String()) +var DefaultInstanceId = fmt.Sprintf("default/stunnerd-%s", uuid.New().String()) // Stunner is an instance of the STUNner deamon. type Stunner struct { diff --git a/stunner_test.go b/stunner_test.go index 181e03c6..bf6bf548 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -272,13 +272,13 @@ type TestStunnerConfigCase struct { var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ { config: stnrv1.StunnerConfig{ - // udp, plaintext + // udp, static ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -302,13 +302,13 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, { config: stnrv1.StunnerConfig{ - // udp, longterm + // udp, ephemeral ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, @@ -331,7 +331,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, { config: stnrv1.StunnerConfig{ - // tcp, plaintext + // tcp, static ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, @@ -361,13 +361,13 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, { config: stnrv1.StunnerConfig{ - // tcp, longterm + // tcp, ephemeral ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, @@ -390,7 +390,7 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, { config: stnrv1.StunnerConfig{ - // tls, plaintext + // tls, static ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, @@ -422,13 +422,13 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, { config: stnrv1.StunnerConfig{ - // tls, longterm + // tls, ephemeral ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": "my-secret", }, @@ -453,13 +453,13 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, { config: stnrv1.StunnerConfig{ - // dtls, plaintext + // dtls, static ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -483,14 +483,14 @@ var TestStunnerConfigsWithLocalhost = []TestStunnerConfigCase{ }, uri: "turns:1.2.3.4:3478?transport=udp", }, - // // dtls, longterm + // // dtls, ephemeral // { // ApiVersion: stnrv1.ApiVersion, // Admin: stnrv1.AdminConfig{ // LogLevel: stunnerTestLoglevel, // }, // Auth: stnrv1.AuthConfig{ - // Type: "longterm", + // Type: "ephemeral", // Credentials: map[string]string{ // "secret": "my-secret", // }, @@ -664,7 +664,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -696,7 +696,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -729,7 +729,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -763,7 +763,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -797,7 +797,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -838,7 +838,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -879,7 +879,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -917,7 +917,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -954,7 +954,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -988,7 +988,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -1022,7 +1022,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -1057,7 +1057,7 @@ var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -1171,7 +1171,7 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -1207,7 +1207,7 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -1243,7 +1243,7 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", @@ -1279,7 +1279,7 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ LogLevel: stunnerTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "plaintext", + Type: "static", Credentials: map[string]string{ "username": "user1", "password": "passwd1", diff --git a/tools.go b/tools.go new file mode 100644 index 00000000..2940fbb0 --- /dev/null +++ b/tools.go @@ -0,0 +1,7 @@ +//go:build tools + +package stunner + +import ( + _ "github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen" +) diff --git a/turncat_test.go b/turncat_test.go index a8264d89..ec82d8f4 100644 --- a/turncat_test.go +++ b/turncat_test.go @@ -233,7 +233,7 @@ func TestTurncatLongterm(t *testing.T) { LogLevel: turncatTestLoglevel, }, Auth: stnrv1.AuthConfig{ - Type: "longterm", + Type: "ephemeral", Credentials: map[string]string{ "secret": sharedSecret, }, @@ -298,7 +298,7 @@ func TestTurncatLongterm(t *testing.T) { server, err := ParseUri(c.ServerAddr) assert.NoError(t, err, "cannot parse server URI") - testName := fmt.Sprintf("TestTurncat_NewTurncat_Longterm_client:%s_server:%s", + testName := fmt.Sprintf("TestTurncat_NewTurncat_Ephemeral_client:%s_server:%s", listener.Scheme, server.Protocol) t.Run(testName, func(t *testing.T) { From 211bf344d472d0f6855c87fc8883ed7b9b987262 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 8 Dec 2023 14:44:10 +0100 Subject: [PATCH 066/243] test: Add CDS client reconnect tests --- cmd/stunnerd/main.go | 6 +- .../{server/server_test.go => cds_test.go} | 223 ++++++++++++------ pkg/config/server/conn.go | 11 + pkg/config/server/server.go | 29 ++- 4 files changed, 188 insertions(+), 81 deletions(-) rename pkg/config/{server/server_test.go => cds_test.go} (82%) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 0081df6b..2e2aae92 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -12,6 +12,7 @@ import ( "github.com/l7mp/stunner" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" ) // usage: stunnerd -v turn://user1:passwd1@127.0.0.1:3478?transport=udp @@ -101,8 +102,11 @@ func main() { conf <- *c } else if *config != "" && *watch { - log.Infof("watching configuration at origin %q", *config) + log.Info("bootstrapping with minimal config") + z := cdsclient.ZeroConfig(st.GetId()) + conf <- *z + log.Infof("watching configuration at origin %q", *config) ctx, cancel := context.WithCancel(context.Background()) defer cancel() cancelConfigLoader = cancel diff --git a/pkg/config/server/server_test.go b/pkg/config/cds_test.go similarity index 82% rename from pkg/config/server/server_test.go rename to pkg/config/cds_test.go index 5ca08ccf..031394c6 100644 --- a/pkg/config/server/server_test.go +++ b/pkg/config/cds_test.go @@ -12,15 +12,16 @@ import ( stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/config/client" + "github.com/l7mp/stunner/pkg/config/server" "github.com/l7mp/stunner/pkg/logger" ) -var testerLogLevel = zapcore.Level(-4) +// var testerLogLevel = zapcore.Level(-4) +// var testerLogLevel = zapcore.DebugLevel +var testerLogLevel = zapcore.ErrorLevel -//var testerLogLevel = zapcore.DebugLevel -//var testerLogLevel = zapcore.ErrorLevel - -const stunnerLogLevel = "all:TRACE" +// const stunnerLogLevel = "all:TRACE" +const stunnerLogLevel = "all:ERROR" func init() { // setup a fast pinger so that we get a timely error notification @@ -46,9 +47,9 @@ func TestServerLoad(t *testing.T) { defer cancel() testLog.Debug("create server") - server := New(stnrv1.DefaultConfigDiscoveryAddress, log) - assert.NotNil(t, server, "server") - err = server.Start(ctx) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(ctx) assert.NoError(t, err, "start") time.Sleep(20 * time.Millisecond) @@ -75,18 +76,18 @@ func TestServerLoad(t *testing.T) { c1 := testConfig("ns1/gw1", "realm1") c2 := testConfig("ns1/gw2", "realm1") - err = server.UpdateConfig([]Config{c1, c2}) + err = srv.UpdateConfig([]server.Config{c1, c2}) assert.NoError(t, err, "update") - cs := server.configs.Snapshot() + cs := srv.GetConfigStore().Snapshot() assert.Len(t, cs, 2, "snapshot len") - sc1 := server.configs.Get("ns1/gw1") + sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc2 := server.configs.Get("ns1/gw2") + sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") - sc3 := server.configs.Get("ns1/gw3") + sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") testLog.Debug("load: config ok") @@ -101,10 +102,10 @@ func TestServerLoad(t *testing.T) { assert.Nil(t, c, "conf") testLog.Debug("remove 2 configs") - err = server.UpdateConfig([]Config{}) + err = srv.UpdateConfig([]server.Config{}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 0, "snapshot len") testLog.Debug("load: no result") @@ -132,9 +133,9 @@ func TestServerPoll(t *testing.T) { defer cancel() testLog.Debug("create server") - server := New(stnrv1.DefaultConfigDiscoveryAddress, log) - assert.NotNil(t, server, "server") - err = server.Start(ctx) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(ctx) assert.NoError(t, err, "start") time.Sleep(20 * time.Millisecond) @@ -178,18 +179,18 @@ func TestServerPoll(t *testing.T) { testLog.Debug("poll: one result") c1 := testConfig("ns1/gw1", "realm1") c2 := testConfig("ns1/gw2", "realm1") - err = server.UpdateConfig([]Config{c1, c2}) + err = srv.UpdateConfig([]server.Config{c1, c2}) assert.NoError(t, err, "update") - cs := server.configs.Snapshot() + cs := srv.GetConfigStore().Snapshot() assert.Len(t, cs, 2, "snapshot len") - sc1 := server.configs.Get("ns1/gw1") + sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc2 := server.configs.Get("ns1/gw2") + sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") - sc3 := server.configs.Get("ns1/gw3") + sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") // poll should have fed the configs to the channels @@ -203,10 +204,10 @@ func TestServerPoll(t *testing.T) { assert.Nil(t, s, "config 3") testLog.Debug("remove 2 configs") - err = server.UpdateConfig([]Config{}) + err = srv.UpdateConfig([]server.Config{}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 0, "snapshot len") testLog.Debug("poll: zeroconfig") @@ -232,9 +233,9 @@ func TestServerWatch(t *testing.T) { serverCtx, serverCancel := context.WithCancel(context.Background()) testLog.Debug("create server") - server := New(stnrv1.DefaultConfigDiscoveryAddress, log) - assert.NotNil(t, server, "server") - err = server.Start(serverCtx) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) assert.NoError(t, err, "start") testLog.Debug("create client") @@ -272,18 +273,18 @@ func TestServerWatch(t *testing.T) { testLog.Debug("poll: one result") c1 := testConfig("ns1/gw1", "realm1") c2 := testConfig("ns1/gw2", "realm1") - err = server.UpdateConfig([]Config{c1, c2}) + err = srv.UpdateConfig([]server.Config{c1, c2}) assert.NoError(t, err, "update") - cs := server.configs.Snapshot() + cs := srv.GetConfigStore().Snapshot() assert.Len(t, cs, 2, "snapshot len") - sc1 := server.configs.Get("ns1/gw1") + sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc2 := server.configs.Get("ns1/gw2") + sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc3 := server.configs.Get("ns1/gw3") + sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") // poll should have fed the configs to the channels @@ -299,18 +300,18 @@ func TestServerWatch(t *testing.T) { testLog.Debug("update: conf 1 and conf 3") c1 = testConfig("ns1/gw1", "realm-new") c3 := testConfig("ns1/gw3", "realm3") - err = server.UpdateConfig([]Config{c1, c2, c3}) + err = srv.UpdateConfig([]server.Config{c1, c2, c3}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 3, "snapshot len") - sc1 = server.configs.Get("ns1/gw1") + sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") - sc2 = server.configs.Get("ns1/gw2") + sc2 = srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") - sc3 = server.configs.Get("ns1/gw3") + sc3 = srv.GetConfigStore().Get("ns1/gw3") assert.NotNil(t, sc3, "get 3") assert.True(t, c3.Config.DeepEqual(sc3), "deepeq 3") @@ -330,11 +331,11 @@ func TestServerWatch(t *testing.T) { time.Sleep(50 * time.Millisecond) serverCtx, serverCancel = context.WithCancel(context.Background()) defer serverCancel() - server = New(stnrv1.DefaultConfigDiscoveryAddress, log) - assert.NotNil(t, server, "server") - err = server.Start(serverCtx) + srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) assert.NoError(t, err, "start") - err = server.UpdateConfig([]Config{c1, c2, c3}) + err = srv.UpdateConfig([]server.Config{c1, c2, c3}) assert.NoError(t, err, "update") // obtain the initial configs: this may take a while @@ -349,17 +350,17 @@ func TestServerWatch(t *testing.T) { assert.True(t, s.DeepEqual(sc3), "deepeq 3") testLog.Debug("remove 1 config (the 2nd)") - err = server.UpdateConfig([]Config{c1, c3}) + err = srv.UpdateConfig([]server.Config{c1, c3}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 2, "snapshot len") - sc1 = server.configs.Get("ns1/gw1") + sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") - sc2 = server.configs.Get("ns1/gw2") + sc2 = srv.GetConfigStore().Get("ns1/gw2") assert.Nil(t, sc2, "get 2") - sc3 = server.configs.Get("ns1/gw3") + sc3 = srv.GetConfigStore().Get("ns1/gw3") assert.NotNil(t, sc3, "get 3") assert.True(t, c3.Config.DeepEqual(sc3), "deepeq 3") @@ -371,10 +372,10 @@ func TestServerWatch(t *testing.T) { assert.Nil(t, s, "config 3") testLog.Debug("remove remaining 2 configs") - err = server.UpdateConfig([]Config{}) + err = srv.UpdateConfig([]server.Config{}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 0, "snapshot len") testLog.Debug("poll: no config") @@ -401,9 +402,9 @@ func TestServerAPI(t *testing.T) { serverCtx, serverCancel := context.WithCancel(context.Background()) testLog.Debug("create server") - server := New(stnrv1.DefaultConfigDiscoveryAddress, log) - assert.NotNil(t, server, "server") - err = server.Start(serverCtx) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) assert.NoError(t, err, "start") testLog.Debug("create client") @@ -452,15 +453,15 @@ func TestServerAPI(t *testing.T) { testLog.Debug("poll: one result") c1 := testConfig("ns1/gw1", "realm1") c2 := testConfig("ns2/gw1", "realm1") - err = server.UpdateConfig([]Config{c1, c2}) + err = srv.UpdateConfig([]server.Config{c1, c2}) assert.NoError(t, err, "update") - cs := server.configs.Snapshot() + cs := srv.GetConfigStore().Snapshot() assert.Len(t, cs, 2, "snapshot len") - sc1 := server.configs.Get("ns1/gw1") + sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") assert.NotNil(t, sc1, "get 1") - sc2 := server.configs.Get("ns2/gw1") + sc2 := srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") @@ -535,18 +536,18 @@ func TestServerAPI(t *testing.T) { testLog.Debug("update: conf 1 and conf 3") c1 = testConfig("ns1/gw1", "realm-new") c3 := testConfig("ns1/gw2", "realm3") - err = server.UpdateConfig([]Config{c1, c2, c3}) + err = srv.UpdateConfig([]server.Config{c1, c2, c3}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 3, "snapshot len") - sc1 = server.configs.Get("ns1/gw1") + sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc2 = server.configs.Get("ns2/gw1") + sc2 = srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") - sc3 := server.configs.Get("ns1/gw2") + sc3 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc3, "get 3") assert.True(t, c3.Config.DeepEqual(sc3), "deepeq") @@ -629,22 +630,22 @@ func TestServerAPI(t *testing.T) { time.Sleep(50 * time.Millisecond) serverCtx, serverCancel = context.WithCancel(context.Background()) defer serverCancel() - server = New(stnrv1.DefaultConfigDiscoveryAddress, log) - assert.NotNil(t, server, "server") - err = server.Start(serverCtx) + srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) assert.NoError(t, err, "start") - err = server.UpdateConfig([]Config{c1, c2, c3}) + err = srv.UpdateConfig([]server.Config{c1, c2, c3}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 3, "snapshot len") - sc1 = server.configs.Get("ns1/gw1") + sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc2 = server.configs.Get("ns2/gw1") + sc2 = srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") - sc3 = server.configs.Get("ns1/gw2") + sc3 = srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc3, "get 3") assert.True(t, c3.Config.DeepEqual(sc3), "deepeq") @@ -733,18 +734,18 @@ func TestServerAPI(t *testing.T) { testLog.Debug("update: conf 1, remove conf 3, and add conf 4") c1 = testConfig("ns1/gw1", "realm-newer") c4 := testConfig("ns3/gw1", "realm4") - err = server.UpdateConfig([]Config{c1, c2, c4}) + err = srv.UpdateConfig([]server.Config{c1, c2, c4}) assert.NoError(t, err, "update") - cs = server.configs.Snapshot() + cs = srv.GetConfigStore().Snapshot() assert.Len(t, cs, 3, "snapshot len") - sc1 = server.configs.Get("ns1/gw1") + sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") - sc2 = server.configs.Get("ns2/gw1") + sc2 = srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") - sc4 := server.configs.Get("ns3/gw1") + sc4 := srv.GetConfigStore().Get("ns3/gw1") assert.NotNil(t, sc3, "get 3") assert.True(t, c4.Config.DeepEqual(sc4), "deepeq") @@ -810,12 +811,82 @@ func TestServerAPI(t *testing.T) { assert.True(t, s.DeepEqual(sc1), "deepeq") } +func TestClientReconnect(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + serverCtx, serverCancel := context.WithCancel(context.Background()) + defer serverCancel() + + testLog.Debug("create server") + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) + assert.NoError(t, err, "start") + + testLog.Debug("create client") + client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + assert.NoError(t, err, "client 1") + + testLog.Debug("watch: no result") + ch1 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch1) + + clientCtx, clientCancel := context.WithCancel(context.Background()) + defer clientCancel() + err = client1.Watch(clientCtx, ch1) + assert.NoError(t, err, "client 1 watch") + + s := watchConfig(ch1, 150*time.Millisecond) + assert.Nil(t, s, "config 1") + + testLog.Debug("update") + c1 := testConfig("ns1/gw1", "realm1") + err = srv.UpdateConfig([]server.Config{c1}) + assert.NoError(t, err, "update") + + cs := srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 1, "snapshot len") + sc1 := srv.GetConfigStore().Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + + // poll should have fed the config to the channels + s = watchConfig(ch1, 500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + + log.Info("killing the connection of the watcher", "id", "ns1/gw1") + conns := srv.GetConnTrack() + assert.NotNil(t, conns) + snapshot := conns.Snapshot() + assert.Len(t, snapshot, 1) + connId := snapshot[0].Id() + srv.RemoveClient(connId) + + // after 2 pong-waits, client should have reconnected + time.Sleep(client.RetryPeriod) + time.Sleep(client.RetryPeriod) + + // watcher should receive its config + s = watchConfig(ch1, 1500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") +} + // only differ in id and realm -func testConfig(id, realm string) Config { +func testConfig(id, realm string) server.Config { c := client.ZeroConfig(id) c.Auth.Realm = realm - return Config{id, c} + return server.Config{Id: id, Config: c} } // wait for some configurable time for a watch element diff --git a/pkg/config/server/conn.go b/pkg/config/server/conn.go index 14c39588..49feaf8a 100644 --- a/pkg/config/server/conn.go +++ b/pkg/config/server/conn.go @@ -7,12 +7,14 @@ import ( "github.com/gorilla/websocket" ) +// Conn represents a client WebSocket connection. type Conn struct { *websocket.Conn Filter FilterConfig readLock, writeLock sync.Mutex // for writemessage } +// NewConn wraps a WebSocket connection. func NewConn(conn *websocket.Conn, filter FilterConfig) *Conn { return &Conn{ Conn: conn, @@ -20,33 +22,39 @@ func NewConn(conn *websocket.Conn, filter FilterConfig) *Conn { } } +// Id returns the IP 5-tuple for a client connection. func (c *Conn) Id() string { return fmt.Sprintf("%s:%s", c.RemoteAddr().Network(), c.RemoteAddr().String()) } +// WriteMessage writes a message to the client connection with proper locking. func (c *Conn) WriteMessage(messageType int, data []byte) error { c.writeLock.Lock() defer c.writeLock.Unlock() return c.Conn.WriteMessage(messageType, data) } +// ReadMessage reads a message from the client connection with proper locking. func (c *Conn) ReadMessage() (int, []byte, error) { c.readLock.Lock() defer c.readLock.Unlock() return c.Conn.ReadMessage() } +// ConnTrack represents the server's connection tracking table. type ConnTrack struct { conns []*Conn lock sync.RWMutex } +// NewConnTrack creates a new connection tracking table. func NewConnTrack() *ConnTrack { return &ConnTrack{ conns: []*Conn{}, } } +// Get returns a client connection by the IP 5-tuple. func (t *ConnTrack) Get(cid string) *Conn { t.lock.RLock() defer t.lock.RUnlock() @@ -58,12 +66,14 @@ func (t *ConnTrack) Get(cid string) *Conn { return nil } +// Upsert insert a new client connection. func (t *ConnTrack) Upsert(c *Conn) { t.lock.Lock() defer t.lock.Unlock() t.conns = append(t.conns, c) } +// Delete removes a client connection. func (t *ConnTrack) Delete(conn *Conn) { id := conn.Id() t.lock.Lock() @@ -75,6 +85,7 @@ func (t *ConnTrack) Delete(conn *Conn) { } } +// Snapshot creates a snapshot of the connection tracking table. func (t *ConnTrack) Snapshot() []*Conn { t.lock.RLock() defer t.lock.RUnlock() diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index 4c2594e2..cf1dcb9b 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -17,6 +17,7 @@ import ( stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) +// Server is a generic config discovery server implementation. type Server struct { *http.Server addr string @@ -26,6 +27,7 @@ type Server struct { log logr.Logger } +// New creates a new config discovery server instance for the specified address. func New(addr string, logger logr.Logger) *Server { if addr == "" { addr = stnrv1.DefaultConfigDiscoveryAddress @@ -41,12 +43,12 @@ func New(addr string, logger logr.Logger) *Server { return cds } +// Start let the config discovery server listen to new client connections. func (s *Server) Start(ctx context.Context) error { r := mux.NewRouter() api.HandlerFromMux(s, r) s.Server = &http.Server{Addr: s.addr, Handler: r} - // serve go func() { s.log.Info("Starting CDS server", "address", s.addr) @@ -61,7 +63,6 @@ func (s *Server) Start(ctx context.Context) error { } }() - // listen to config update events and cancel requests go func() { defer close(s.configCh) defer s.Close() @@ -89,17 +90,37 @@ func (s *Server) Close() { } } +// GetConfigChannel returns the channel that can be used to add configs to the server's config +// store. Use Update to specify more configs at once. func (s *Server) GetConfigChannel() chan Config { return s.configCh } +// GetConfigStore returns the dataplane configs stores in the server. +func (s *Server) GetConfigStore() *ConfigStore { + return s.configs +} + +// GetConnTrack returns the client connection tracking table of the server. +func (s *Server) GetConnTrack() *ConnTrack { + return s.conns +} + +// RemoveClient forcefully closes a client connection. This is used mainly for testing. +func (s *Server) RemoveClient(id string) { + if c := s.conns.Get(id); c != nil { + s.log.V(1).Info("forcefully removing client connection", "client", id) + s.closeConn(c) + } +} + func (s *Server) handleReq(w http.ResponseWriter, r *http.Request, endpoint string, responder ResponseGen) { s.log.V(1).Info("received new request", "api", endpoint, "client", r.RemoteAddr) response, err := responder() if err != nil { - s.log.Error(err, "error generating client response", "api", endpoint, - "client", r.RemoteAddr, "code", err.Code) + s.log.Info("client config not found", "api", endpoint, "client", r.RemoteAddr, + "code", err.Code, "message", err.Message) sendServerErrorRaw(w, err) } From 3fd76bf4316b588ed00aa0a501cbc190167bf6fb Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 11 Dec 2023 20:23:35 +0100 Subject: [PATCH 067/243] refactor: Steamline command line argument parsing --- cmd/stunnerd/main.go | 38 +++++++++++++++++++------------------- pkg/apis/v1/default.go | 3 +-- 2 files changed, 20 insertions(+), 21 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 2e2aae92..c65140e1 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -18,17 +18,16 @@ import ( // usage: stunnerd -v turn://user1:passwd1@127.0.0.1:3478?transport=udp const ( - defaultLoglevel = "all:INFO" - // environment for the config poller - // defaultDiscoveryAddress = "ws://localhost:13478/api/v1/config/watch" - envVarName = "STUNNER_NAME" - envVarNamespace = "STUNNER_NAMESPACE" - envVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" + defaultLoglevel = "all:INFO" + defaultConfigDiscoveryAddress = "http://localhost:13478" + envVarName = "STUNNER_NAME" + envVarNamespace = "STUNNER_NAMESPACE" + envVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" ) func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", "Config origin, either a valid URL to the CDS server or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: none).") + var config = flag.StringP("config", "c", "", fmt.Sprintf("Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: %s).", defaultConfigDiscoveryAddress)) var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO).") var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: ).") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false).") @@ -47,11 +46,12 @@ func main() { logLevel = *level } - if *config == "" { - origin, ok := os.LookupEnv(envVarConfigOrigin) - if ok { - *config = origin - } + configOrigin := defaultConfigDiscoveryAddress + if origin, ok := os.LookupEnv(envVarConfigOrigin); ok { + configOrigin = origin + } + if *config != "" { + configOrigin = *config } if *id == "" { @@ -78,7 +78,7 @@ func main() { defer close(conf) var cancelConfigLoader context.CancelFunc - if *config == "" && flag.NArg() == 1 { + if configOrigin == "" && flag.NArg() == 1 { log.Infof("starting %s with default configuration at TURN URI: %s", os.Args[0], flag.Arg(0)) @@ -90,10 +90,10 @@ func main() { conf <- *c - } else if *config != "" && !*watch { - log.Infof("loading configuration from origin %q", *config) + } else if configOrigin != "" && !*watch { + log.Infof("loading configuration from origin %q", configOrigin) - c, err := st.LoadConfig(*config) + c, err := st.LoadConfig(configOrigin) if err != nil { log.Error(err.Error()) os.Exit(1) @@ -101,18 +101,18 @@ func main() { conf <- *c - } else if *config != "" && *watch { + } else if configOrigin != "" && *watch { log.Info("bootstrapping with minimal config") z := cdsclient.ZeroConfig(st.GetId()) conf <- *z - log.Infof("watching configuration at origin %q", *config) + log.Infof("watching configuration at origin %q", configOrigin) ctx, cancel := context.WithCancel(context.Background()) defer cancel() cancelConfigLoader = cancel // Watch closes the channel - if err := st.WatchConfig(ctx, *config, conf); err != nil { + if err := st.WatchConfig(ctx, configOrigin, conf); err != nil { log.Errorf("could not run config watcher: %s", err.Error()) os.Exit(1) } diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index e77f5fb7..51b5bd8f 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -18,5 +18,4 @@ const DefaultAuthName = "default-auth-config" const DefaultMetricsPort int = 8080 const DefaultHealthCheckPort int = 8086 -// DefaultConfigDiscoveryAddress is the default URI at which config discovery requests are served. -const DefaultConfigDiscoveryAddress = "0.0.0.0:13478" +const DefaultConfigDiscoveryAddress = ":13478" From 793e2ad1bf65b98934fe4f95795c9d25f391fe0e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 11 Dec 2023 20:24:08 +0100 Subject: [PATCH 068/243] test: Make CDS tests more robust --- internal/util/relay.go | 2 +- pkg/apis/v1alpha1/stunner.go | 2 +- pkg/config/cds_test.go | 88 +++++++++++++++++++++++++++++++++ pkg/config/client/cds_client.go | 9 +++- pkg/config/server/server.go | 18 ++++--- 5 files changed, 107 insertions(+), 12 deletions(-) diff --git a/internal/util/relay.go b/internal/util/relay.go index bd55f817..ccbd50b1 100644 --- a/internal/util/relay.go +++ b/internal/util/relay.go @@ -80,7 +80,7 @@ func (r *RelayAddressGenerator) Validate() error { // returned to the client in the allocation response. func (r *RelayAddressGenerator) AllocatePacketConn(network string, requestedPort int) (net.PacketConn, net.Addr, error) { if requestedPort <= 1 || requestedPort > 2<<16-1 { - // Let the OS to automatically assign a port + // let the OS automatically assign a port requestedPort = 0 } diff --git a/pkg/apis/v1alpha1/stunner.go b/pkg/apis/v1alpha1/stunner.go index a0e1df31..17662d04 100644 --- a/pkg/apis/v1alpha1/stunner.go +++ b/pkg/apis/v1alpha1/stunner.go @@ -191,7 +191,7 @@ func (req *StunnerConfig) GetClusterConfig(name string) (ClusterConfig, error) { return ClusterConfig{}, ErrNoSuchCluster } -// GetClusterConfig finds a Cluster by name in a StunnerConfig or returns an error. +// ConvertToV1 upgrades a v1alpha1 StunnerConfig to a v1. func ConvertToV1(sv1a1 *StunnerConfig) (*stnrv1.StunnerConfig, error) { sv1 := stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index 031394c6..aea4b64d 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -387,6 +387,94 @@ func TestServerWatch(t *testing.T) { assert.Nil(t, s, "config") } +// config already available when watcher joins +func TestServerWatchBootstrap(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + serverCtx, serverCancel := context.WithCancel(context.Background()) + defer serverCancel() + + testLog.Debug("create server") + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) + assert.NoError(t, err, "start") + + testLog.Debug("create client") + client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + assert.NoError(t, err, "client 1") + + testLog.Debug("bootstrap") + c1 := testConfig("ns1/gw1", "realm1") + c2 := testConfig("ns1/gw2", "realm1") + err = srv.UpdateConfig([]server.Config{c1, c2}) + assert.NoError(t, err, "update") + + cs := srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 := srv.GetConfigStore().Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc2 := srv.GetConfigStore().Get("ns1/gw2") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") + sc3 := srv.GetConfigStore().Get("ns1/gw3") + assert.Nil(t, sc3, "get 3") + + testLog.Debug("watch: 1 result") + ch1 := make(chan stnrv1.StunnerConfig, 8) + defer close(ch1) + + clientCtx, clientCancel := context.WithCancel(context.Background()) + defer clientCancel() + err = client1.Watch(clientCtx, ch1) + assert.NoError(t, err, "client 1 watch") + + s := watchConfig(ch1, 1500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + // only 1 config + s = watchConfig(ch1, 150*time.Millisecond) + assert.Nil(t, s, "config 1") + + testLog.Debug("update: conf 1 and conf 2") + c1 = testConfig("ns1/gw1", "realm-new") + c2 = testConfig("ns1/gw2", "realm3") + err = srv.UpdateConfig([]server.Config{c1, c2}) + assert.NoError(t, err, "update") + + cs = srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 = srv.GetConfigStore().Get("ns1/gw1") + assert.NotNil(t, sc1, "get 1") + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") + sc2 = srv.GetConfigStore().Get("ns1/gw2") + assert.NotNil(t, sc2, "get 2") + assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") + + s = watchConfig(ch1, 500*time.Millisecond) + assert.NotNil(t, s, "config 1") + assert.True(t, s.DeepEqual(sc1), "deepeq 1") + + testLog.Debug("remove 2 configs") + err = srv.UpdateConfig([]server.Config{}) + assert.NoError(t, err, "update") + + cs = srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 0, "snapshot len") + + testLog.Debug("poll: no config") + s = watchConfig(ch1, 10*time.Millisecond) +} + // test APIs func TestServerAPI(t *testing.T) { zc := zap.NewProductionConfig() diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go index 6eb69364..ee4cffaa 100644 --- a/pkg/config/client/cds_client.go +++ b/pkg/config/client/cds_client.go @@ -161,14 +161,19 @@ func poll(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error { // wait fo cancel for { + defer func() { + a.Infof("closing connection for client %s", conn.RemoteAddr().String()) + conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck + conn.Close() + closePinger <- struct{}{} + }() + select { case <-ctx.Done(): // cancel: normal return - closePinger <- struct{}{} return nil case err := <-errCh: // error: return it - closePinger <- struct{}{} return err } } diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index cf1dcb9b..16221cba 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -196,25 +196,27 @@ func (s *Server) broadcastConfig(e Config) { for _, conn := range s.conns.Snapshot() { if conn.Filter(e.Id) { - if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { - s.log.Error(err, "error sending config update", - "client", conn.Id()) - s.closeConn(conn) - } + s.sendJSONConfig(conn, json) } } } -// iterate through all connections and send response if needed func (s *Server) sendConfig(conn *Conn, e Config) { json, err := json.Marshal(e.Config) if err != nil { s.log.Error(err, "error JSON marshaling config", "event", e.String()) return } + + s.sendJSONConfig(conn, json) +} + +func (s *Server) sendJSONConfig(conn *Conn, json []byte) { + s.log.V(2).Info("sending configuration to client", "client", conn.Id(), + "config", string(json)) + if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { - s.log.Error(err, "error sending config update", - "client", conn.Id()) + s.log.Error(err, "error sending config update", "client", conn.Id()) s.closeConn(conn) } } From db7bdc7994b4ed483434fb4b52d437581114d33a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 12 Dec 2023 00:30:12 +0100 Subject: [PATCH 069/243] refactor: MinPort/MaxPort are now a property of clusters This commit refactors the API and the relay code to consider port-ranges per cluster, not per listener. Relays now dynamically look up the cluster to/from which packets are send/received, correctly enforce port ranges and report cluster telemetry per cluster, and not per listener. --- go.mod | 1 + go.sum | 2 + internal/object/cluster.go | 30 ++- internal/object/listener.go | 14 +- internal/telemetry/metrics.go | 42 ++-- internal/util/conn.go | 95 +------- internal/util/relay.go | 111 ---------- internal/util/socketpool.go | 4 +- internal/util/socketpool_unix.go | 12 +- pkg/apis/v1/cluster.go | 33 ++- pkg/apis/v1/listener.go | 33 +-- pkg/config/cds_test.go | 1 + reconcile_test.go | 138 ++++++------ relay.go | 226 ++++++++++++++++++++ internal/util/conn_test.go => relay_test.go | 34 ++- server.go | 17 +- stunner_test.go | 64 +++--- 17 files changed, 452 insertions(+), 405 deletions(-) delete mode 100644 internal/util/relay.go create mode 100644 relay.go rename internal/util/conn_test.go => relay_test.go (89%) diff --git a/go.mod b/go.mod index b2facf0f..3897a2ec 100644 --- a/go.mod +++ b/go.mod @@ -45,6 +45,7 @@ require ( github.com/golang/protobuf v1.5.3 // indirect github.com/google/gnostic v0.5.7-v3refs // indirect github.com/google/gofuzz v1.1.0 // indirect + github.com/hashicorp/golang-lru v1.0.2 // indirect github.com/imdario/mergo v0.3.12 // indirect github.com/invopop/yaml v0.1.0 // indirect github.com/josharian/intern v1.0.0 // indirect diff --git a/go.sum b/go.sum index 7dc510b4..5dbd6e72 100644 --- a/go.sum +++ b/go.sum @@ -200,6 +200,8 @@ github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZH github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c= +github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb h1:tsEKRC3PU9rMw18w/uAptoijhgG4EvlA5kfJPtwrMDk= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb/go.mod h1:NtmN9h8vrTveVQRLHcX2HQ5wIPBDCsZ351TGbZWgg38= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= diff --git a/internal/object/cluster.go b/internal/object/cluster.go index 0fc621dd..743220c1 100644 --- a/internal/object/cluster.go +++ b/internal/object/cluster.go @@ -15,14 +15,16 @@ import ( // Listener implements a STUNner cluster type Cluster struct { - Name string - Type stnrv1.ClusterType - Protocol stnrv1.ClusterProtocol - Endpoints []net.IPNet - Domains []string - Resolver resolver.DnsResolver // for strict DNS - logger logging.LoggerFactory - log logging.LeveledLogger + Name string + Type stnrv1.ClusterType + Protocol stnrv1.ClusterProtocol + MinPort, MaxPort int + Endpoints []net.IPNet + Domains []string + Resolver resolver.DnsResolver // for strict DNS + + logger logging.LoggerFactory + log logging.LeveledLogger } // NewCluster creates a new cluster. @@ -40,6 +42,8 @@ func NewCluster(conf stnrv1.Config, resolver resolver.DnsResolver, logger loggin c := Cluster{ Name: req.Name, Endpoints: []net.IPNet{}, + MinPort: req.MinRelayPort, + MaxPort: req.MaxRelayPort, Domains: []string{}, Resolver: resolver, logger: logger, @@ -75,6 +79,8 @@ func (c *Cluster) Reconcile(conf stnrv1.Config) error { c.log.Tracef("Reconcile: %s", req.String()) c.Type, _ = stnrv1.NewClusterType(req.Type) c.Protocol, _ = stnrv1.NewClusterProtocol(req.Protocol) + c.MinPort = req.MinRelayPort + c.MaxPort = req.MaxRelayPort switch c.Type { case stnrv1.ClusterTypeStatic: @@ -147,9 +153,11 @@ func (c *Cluster) ObjectType() string { // GetConfig returns the configuration of the running cluster. func (c *Cluster) GetConfig() stnrv1.Config { conf := stnrv1.ClusterConfig{ - Name: c.Name, - Protocol: c.Protocol.String(), - Type: c.Type.String(), + Name: c.Name, + Protocol: c.Protocol.String(), + Type: c.Type.String(), + MinRelayPort: c.MinPort, + MaxRelayPort: c.MaxPort, } switch c.Type { diff --git a/internal/object/listener.go b/internal/object/listener.go index 9a226566..3274a9ef 100644 --- a/internal/object/listener.go +++ b/internal/object/listener.go @@ -94,8 +94,6 @@ func (l *Listener) Inspect(old, new, full stnrv1.Config) (bool, error) { l.Proto == proto && // protocol unchanged l.rawAddr == req.Addr && // address unchanged l.Port == req.Port && // ports unchanged - l.MinPort == req.MinRelayPort && - l.MaxPort == req.MaxRelayPort && bytes.Equal(l.Cert, cert) && // TLS creds unchanged bytes.Equal(l.Key, key) { restart = nil @@ -137,7 +135,7 @@ func (l *Listener) Reconcile(conf stnrv1.Config) error { l.Proto = proto l.Addr = ipAddr l.rawAddr = req.Addr - l.Port, l.MinPort, l.MaxPort = req.Port, req.MinRelayPort, req.MaxRelayPort + l.Port = req.Port if proto == stnrv1.ListenerProtocolTURNTLS || proto == stnrv1.ListenerProtocolTURNDTLS { cert, err := base64.StdEncoding.DecodeString(req.Cert) if err != nil { @@ -181,12 +179,10 @@ func (l *Listener) GetConfig() stnrv1.Config { sort.Strings(l.Routes) c := &stnrv1.ListenerConfig{ - Name: l.Name, - Protocol: l.Proto.String(), - Addr: l.rawAddr, - Port: l.Port, - MinRelayPort: l.MinPort, - MaxRelayPort: l.MaxPort, + Name: l.Name, + Protocol: l.Proto.String(), + Addr: l.rawAddr, + Port: l.Port, } c.Cert = string(l.Cert) diff --git a/internal/telemetry/metrics.go b/internal/telemetry/metrics.go index 815917e8..3f107064 100644 --- a/internal/telemetry/metrics.go +++ b/internal/telemetry/metrics.go @@ -18,8 +18,8 @@ var ( promListenerConnsActive *prometheus.GaugeVec promClusterPacketsTotal *prometheus.CounterVec promClusterBytesTotal *prometheus.CounterVec - promClusterConnsTotal *prometheus.CounterVec - promClusterConnsActive *prometheus.GaugeVec + // promClusterConnsTotal *prometheus.CounterVec + // promClusterConnsActive *prometheus.GaugeVec ) func Init() { @@ -55,18 +55,18 @@ func Init() { prometheus.MustRegister(promListenerConnsActive) // cluster stats - promClusterConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ - Namespace: stunnerNamespace, - Subsystem: "cluster", - Name: "connections", - Help: "Number of active upstream connections on behalf of a listener", - }, promConnLabels) - promClusterConnsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: stunnerNamespace, - Subsystem: "cluster", - Name: "connections_total", - Help: "Number of upstream connections on behalf of a listener.", - }, promConnLabels) + // promClusterConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + // Namespace: stunnerNamespace, + // Subsystem: "cluster", + // Name: "connections", + // Help: "Number of active upstream connections on behalf of a listener", + // }, promConnLabels) + // promClusterConnsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ + // Namespace: stunnerNamespace, + // Subsystem: "cluster", + // Name: "connections_total", + // Help: "Number of upstream connections on behalf of a listener.", + // }, promConnLabels) promClusterPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: stunnerNamespace, Subsystem: "cluster", @@ -82,8 +82,8 @@ func Init() { prometheus.MustRegister(promClusterPacketsTotal) prometheus.MustRegister(promClusterBytesTotal) - prometheus.MustRegister(promClusterConnsTotal) - prometheus.MustRegister(promClusterConnsActive) + // prometheus.MustRegister(promClusterConnsTotal) + // prometheus.MustRegister(promClusterConnsActive) } func Close() { @@ -93,8 +93,8 @@ func Close() { _ = prometheus.Unregister(promListenerConnsActive) _ = prometheus.Unregister(promClusterPacketsTotal) _ = prometheus.Unregister(promClusterBytesTotal) - _ = prometheus.Unregister(promClusterConnsTotal) - _ = prometheus.Unregister(promClusterConnsActive) + // _ = prometheus.Unregister(promClusterConnsTotal) + // _ = prometheus.Unregister(promClusterConnsActive) } func IncrementPackets(n string, c ConnType, d Direction, count uint64) { @@ -121,8 +121,8 @@ func AddConnection(n string, c ConnType) { promListenerConnsActive.WithLabelValues(n).Add(1) promListenerConnsTotal.WithLabelValues(n).Add(1) case ClusterType: - promClusterConnsActive.WithLabelValues(n).Add(1) - promClusterConnsTotal.WithLabelValues(n).Add(1) + // promClusterConnsActive.WithLabelValues(n).Add(1) + // promClusterConnsTotal.WithLabelValues(n).Add(1) } } @@ -131,7 +131,7 @@ func SubConnection(n string, c ConnType) { case ListenerType: promListenerConnsActive.WithLabelValues(n).Sub(1) case ClusterType: - promClusterConnsActive.WithLabelValues(n).Sub(1) + // promClusterConnsActive.WithLabelValues(n).Sub(1) } } diff --git a/internal/util/conn.go b/internal/util/conn.go index ebdfc5d3..459d9bd8 100644 --- a/internal/util/conn.go +++ b/internal/util/conn.go @@ -1,21 +1,15 @@ package util import ( - "errors" "fmt" "net" "os" "time" - "github.com/pion/logging" + "github.com/l7mp/stunner/internal/telemetry" "github.com/pion/transport/v3" ) -var ( - ErrPortProhibited = errors.New("peer port administratively prohibited") - ErrInvalidPeerProtocol = errors.New("unknown peer transport protocol") -) - type FileConnAddr struct { File *os.File } @@ -78,6 +72,7 @@ type PacketConnPool interface { // defaultPacketConPool implements a socketpool that consists of only a single socket, used as a fallback for architectures that do not support SO_REUSEPORT or when socket pooling is disabled. type defaultPacketConnPool struct { transport.Net + listenerName string } // Make creates a PacketConnPool, caller must make sure to close the sockets. @@ -89,92 +84,10 @@ func (p *defaultPacketConnPool) Make(network, address string) ([]net.PacketConn, return []net.PacketConn{}, fmt.Errorf("failed to create PacketConn at %s "+ "(REUSEPORT: false): %s", address, err) } - conns = append(conns, conn) + conn = telemetry.NewPacketConn(conn, p.listenerName, telemetry.ListenerType) + conns = append(conns, conn) return conns, nil } func (p *defaultPacketConnPool) Size() int { return 1 } - -// PortRangePacketConn is a net.PacketConn that filters on the target port range. -type PortRangePacketConn struct { - net.PacketConn - name string - minPort, maxPort int - log logging.LeveledLogger - readDeadline time.Time -} - -// NewPortRangePacketConn decorates a PacketConn with filtering on a target port range. Errors are reported per listener name. -func NewPortRangePacketConn(c net.PacketConn, listenerName string, minPort, maxPort int, log logging.LeveledLogger) net.PacketConn { - return &PortRangePacketConn{ - PacketConn: c, - name: listenerName, - minPort: minPort, - maxPort: maxPort, - log: log, - } -} - -// WriteTo writes to the PacketConn. -func (c *PortRangePacketConn) WriteTo(p []byte, peerAddr net.Addr) (int, error) { - switch addr := peerAddr.(type) { - case *net.UDPAddr: - if addr.Port < c.minPort || addr.Port > c.maxPort { - // c.log.Infof("sending UDP packet with invalid peer port %d rejected on listener %q (must be in [%d:%d])", - // addr.Port, c.name, c.minPort, c.maxPort) - return 0, ErrPortProhibited - } - case *net.TCPAddr: - if addr.Port < c.minPort || addr.Port > c.maxPort { - // c.log.Infof("sending TCP packet with invalid peer port %d rejected on listener %q (must be in [%d:%d])", - // addr.Port, c.name, c.minPort, c.maxPort) - return 0, ErrPortProhibited - } - default: - return 0, ErrInvalidPeerProtocol - } - - return c.PacketConn.WriteTo(p, peerAddr) -} - -// ReadFrom reads from the PortRangePacketConn. Blocks until a packet from the speciifed port range -// is received and drops all other packets. -func (c *PortRangePacketConn) ReadFrom(p []byte) (int, net.Addr, error) { - for { - var peerAddr net.Addr - - err := c.PacketConn.SetReadDeadline(c.readDeadline) - if err != nil { - return 0, peerAddr, err - } - - n, peerAddr, err := c.PacketConn.ReadFrom(p) - - // Return errors unconditionally: peerAddr will most probably not be valid anyway - // so it is not worth checking - if err != nil { - return n, peerAddr, err - } - - switch addr := peerAddr.(type) { - case *net.UDPAddr: - if addr.Port >= c.minPort && addr.Port <= c.maxPort { - return n, peerAddr, err - } - // c.log.Infof("received UDP packet with invalid peer port %d dropped on listener %q (must be in [%d:%d])", - // addr.Port, c.name, c.minPort, c.maxPort) - case *net.TCPAddr: - if addr.Port >= c.minPort && addr.Port <= c.maxPort { - return n, peerAddr, err - } - // c.log.Infof("received TCP packet with invalid peer port %d dropped on listener %q (must be in [%d:%d])", - // addr.Port, c.name, c.minPort, c.maxPort) - } - } -} - -func (c *PortRangePacketConn) SetReadDeadline(t time.Time) error { - c.readDeadline = t - return nil -} diff --git a/internal/util/relay.go b/internal/util/relay.go deleted file mode 100644 index ccbd50b1..00000000 --- a/internal/util/relay.go +++ /dev/null @@ -1,111 +0,0 @@ -package util - -// code adopted from github.com/livekit/pkg/telemetry - -import ( - "errors" - "fmt" - "net" - - "github.com/pion/transport/v3" - "github.com/pion/transport/v3/stdnet" - - "github.com/l7mp/stunner/internal/telemetry" - "github.com/l7mp/stunner/pkg/logger" -) - -var ( - errInvalidName = errors.New("RelayAddressGenerator: Name must be set") - errRelayAddressInvalid = errors.New("RelayAddressGenerator: invalid RelayAddress") - errInvalidRelayPortRange = errors.New("RelayAddressGenerator: invalid target port range [MinRelayPort:MaxRelayPort]") - errListeningAddressInvalid = errors.New("RelayAddressGenerator: invalid ListeningAddress") - errNilConn = errors.New("cannot allocate relay connection") - errTodo = errors.New("relay to Net.Conn not implemented") -) - -// RelayAddressGenerator can be used to only allocate connections inside a defined target port -// range. A static ip address can be set. -type RelayAddressGenerator struct { - // ListenerName is the name of the listener this relay address generator belongs to. Note that - // packets sent to/received from upstream cluster are reported with the name of the - // *listener* that the packet belongs to, and not the cluster. - ListenerName string - - // RelayAddress is the IP returned to the user when the relay is created. - RelayAddress net.IP - - // MinRelayPort the minimum target port (inclusive). - MinRelayPort int - - // MaxRelayPort the maximum target port (inclusive). - MaxRelayPort int - - // Address is passed to Listen/ListenPacket when creating the Relay. - Address string - - // Net is a pion/transport VNet, used for testing. - Net transport.Net - - // Logger is a logger factory we can use to generate per-listener relay loggers. - Logger *logger.LeveledLoggerFactory -} - -// Validate is called on server startup and confirms the RelayAddressGenerator is properly configured. -func (r *RelayAddressGenerator) Validate() error { - if r.ListenerName == "" { - return errInvalidName - } - - if r.Net == nil { - r.Net, _ = stdnet.NewNet() - } - - switch { - case r.MinRelayPort == 0: - return errInvalidRelayPortRange - case r.MaxRelayPort == 0: - return errInvalidRelayPortRange - case r.MinRelayPort > r.MaxRelayPort: - return errInvalidRelayPortRange - case r.RelayAddress == nil: - return errRelayAddressInvalid - case r.Address == "": - return errListeningAddressInvalid - default: - return nil - } -} - -// AllocatePacketConn generates a new transport relay connection and returns the IP/Port to be -// returned to the client in the allocation response. -func (r *RelayAddressGenerator) AllocatePacketConn(network string, requestedPort int) (net.PacketConn, net.Addr, error) { - if requestedPort <= 1 || requestedPort > 2<<16-1 { - // let the OS automatically assign a port - requestedPort = 0 - } - - conn, err := r.Net.ListenPacket(network, fmt.Sprintf("%s:%d", r.Address, requestedPort)) - if err != nil { - return nil, nil, err - } - - conn = NewPortRangePacketConn(conn, r.ListenerName, r.MinRelayPort, r.MaxRelayPort, - r.Logger.NewLogger(fmt.Sprintf("relay-%s", r.ListenerName))) - - // Decorate with a telemetry reporter. - conn = telemetry.NewPacketConn(conn, r.ListenerName, telemetry.ClusterType) - - relayAddr, ok := conn.LocalAddr().(*net.UDPAddr) - if !ok { - return nil, nil, errNilConn - } - - relayAddr.IP = r.RelayAddress - return conn, relayAddr, nil -} - -// AllocateConn generates a new Conn to receive traffic on and the IP/Port to populate the -// allocation response with -func (g *RelayAddressGenerator) AllocateConn(network string, requestedPort int) (net.Conn, net.Addr, error) { - return nil, nil, errTodo -} diff --git a/internal/util/socketpool.go b/internal/util/socketpool.go index b2b4e5e5..c02b9281 100644 --- a/internal/util/socketpool.go +++ b/internal/util/socketpool.go @@ -9,7 +9,7 @@ import ( // NewPacketConnPool creates a new packet connection pool which is fixed to a single connection, // used if threadNum is zero or if we are running on top of transport.VNet (which does not support // reuseport), or if we are on non-unix, see the fallback in socketpool.go. -func NewPacketConnPool(vnet transport.Net, threadNum int) PacketConnPool { +func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int) PacketConnPool { // default to a single socket for vnet or if udp multithreading is disabled - return &defaultPacketConnPool{Net: vnet} + return &defaultPacketConnPool{Net: vnet, listenerName: listenerName} } diff --git a/internal/util/socketpool_unix.go b/internal/util/socketpool_unix.go index d5c61765..d6f09764 100644 --- a/internal/util/socketpool_unix.go +++ b/internal/util/socketpool_unix.go @@ -10,6 +10,7 @@ import ( "golang.org/x/sys/unix" + "github.com/l7mp/stunner/internal/telemetry" "github.com/pion/transport/v3" "github.com/pion/transport/v3/stdnet" ) @@ -17,18 +18,18 @@ import ( // unixPacketConPool implements socketpools for unix with full support for SO_REUSEPORT type unixPacketConnPool struct { net.ListenConfig - size int + listenerName string + size int } // NewPacketConnPool creates a new packet connection pool. Pooling is disabled if threadNum is zero // or if we are running on top of transport.VNet (which does not support reuseport), or if we are // on non-unix, see the fallback in socketpool.go. -func NewPacketConnPool(vnet transport.Net, threadNum int) PacketConnPool { +func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int) PacketConnPool { // default to a single socket for vnet or if udp multithreading is disabled _, ok := vnet.(*stdnet.Net) if ok && threadNum > 0 { return &unixPacketConnPool{ - size: threadNum, ListenConfig: net.ListenConfig{ Control: func(network, address string, conn syscall.RawConn) error { var operr error @@ -42,9 +43,11 @@ func NewPacketConnPool(vnet transport.Net, threadNum int) PacketConnPool { return operr }, }, + size: threadNum, + listenerName: listenerName, } } else { - return &defaultPacketConnPool{Net: vnet} + return &defaultPacketConnPool{listenerName: listenerName, Net: vnet} } } @@ -59,6 +62,7 @@ func (p *unixPacketConnPool) Make(network, address string) ([]net.PacketConn, er return []net.PacketConn{}, fmt.Errorf("failed to create PacketConn "+ "%d at %s (REUSEPORT: %t): %s", i, address, (p.size > 0), err) } + conn = telemetry.NewPacketConn(conn, p.listenerName, telemetry.ListenerType) conns = append(conns, conn) } diff --git a/pkg/apis/v1/cluster.go b/pkg/apis/v1/cluster.go index b7c4d08e..5fc2b75d 100644 --- a/pkg/apis/v1/cluster.go +++ b/pkg/apis/v1/cluster.go @@ -23,6 +23,14 @@ type ClusterConfig struct { // Protocol specifies the protocol to be used with the cluster, either UDP (default) or TCP // (not implemented yet). Protocol string `json:"protocol,omitempty"` + // MinRelayPort is the lowest peer target port admitted on the cluster (inclusive). The + // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport + // relay connections created via the cluster. Default is 1. + MinRelayPort int `json:"min_relay_port,omitempty"` + // MaxRelayPort is the highest peer target port admitted on the cluster (inclusive). The + // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport + // relay connections created via the cluster. Default is 65535. + MaxRelayPort int `json:"max_relay_port,omitempty"` // Endpoints specifies the peers that can be reached via this cluster. Endpoints []string `json:"endpoints,omitempty"` } @@ -53,6 +61,22 @@ func (req *ClusterConfig) Validate() error { } req.Protocol = p.String() + if req.MinRelayPort == 0 { + req.MinRelayPort = DefaultMinRelayPort + } + if req.MaxRelayPort == 0 { + req.MaxRelayPort = DefaultMaxRelayPort + } + for _, p := range []int{req.MinRelayPort, req.MaxRelayPort} { + if p <= 0 || p > 65535 { + return fmt.Errorf("invalid port: %d", p) + } + } + if req.MinRelayPort > req.MaxRelayPort { + return fmt.Errorf("invalid relay port range: min port (%d) <= max port (%d) must hold", + req.MinRelayPort, req.MaxRelayPort) + } + sort.Strings(req.Endpoints) return nil } @@ -92,8 +116,15 @@ func (req *ClusterConfig) String() string { status = append(status, fmt.Sprintf("protocol=%q", req.Protocol)) } + min, max := 0, 65535 + if req.MinRelayPort != 0 { + min = req.MinRelayPort + } + if req.MaxRelayPort != 0 { + max = req.MaxRelayPort + } status = append(status, fmt.Sprintf("endpoints=[%s]", strings.Join(req.Endpoints, ","))) - return fmt.Sprintf("%q:{%s}", n, strings.Join(status, ",")) + return fmt.Sprintf("%q:{%s}<%d-%d>", n, strings.Join(status, ","), min, max) } diff --git a/pkg/apis/v1/listener.go b/pkg/apis/v1/listener.go index d39fbc83..c972ed47 100644 --- a/pkg/apis/v1/listener.go +++ b/pkg/apis/v1/listener.go @@ -26,14 +26,6 @@ type ListenerConfig struct { Addr string `json:"address,omitempty"` // Port is the port for the listener. Default is the standard TURN port (3478). Port int `json:"port,omitempty"` - // MinRelayPort is the lowest peer target port admitted on the listener (inclusive). The - // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport - // relay connections created via the listener. Default is 1. - MinRelayPort int `json:"min_relay_port,omitempty"` - // MaxRelayPort is the highest peer target port admitted on the listener (inclusive). The - // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport - // relay connections created via the listener. Default is 65535. - MaxRelayPort int `json:"max_relay_port,omitempty"` // Cert is the base64-encoded TLS cert. Cert string `json:"cert,omitempty"` // Key is the base64-encoded TLS key. @@ -65,20 +57,8 @@ func (req *ListenerConfig) Validate() error { if req.Port == 0 { req.Port = DefaultPort } - if req.MinRelayPort == 0 { - req.MinRelayPort = DefaultMinRelayPort - } - if req.MaxRelayPort == 0 { - req.MaxRelayPort = DefaultMaxRelayPort - } - for _, p := range []int{req.Port, req.MinRelayPort, req.MaxRelayPort} { - if p <= 0 || p > 65535 { - return fmt.Errorf("invalid port: %d", p) - } - } - if req.MinRelayPort > req.MaxRelayPort { - return fmt.Errorf("invalid relay port range: min port (%d) <= max port (%d) must hold", - req.MinRelayPort, req.MaxRelayPort) + if req.Port <= 0 || req.Port > 65535 { + return fmt.Errorf("invalid port: %d", req.Port) } if proto == ListenerProtocolTURNTLS || proto == ListenerProtocolTURNDTLS || @@ -122,14 +102,7 @@ func (req *ListenerConfig) String() string { n = req.Name } - min, max := 0, 65535 - if req.MinRelayPort != 0 { - min = req.MinRelayPort - } - if req.MaxRelayPort != 0 { - max = req.MaxRelayPort - } - status = append(status, fmt.Sprintf("turn://0.0.0.0:%d<%d-%d>", req.Port, min, max)) + status = append(status, fmt.Sprintf("turn://0.0.0.0:%d", req.Port)) a, p := "-", "-" if req.PublicAddr != "" { diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index aea4b64d..e576198a 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -473,6 +473,7 @@ func TestServerWatchBootstrap(t *testing.T) { testLog.Debug("poll: no config") s = watchConfig(ch1, 10*time.Millisecond) + assert.Nil(t, s, "config 1") } // test APIs diff --git a/reconcile_test.go b/reconcile_test.go index d4bd3466..fdea401b 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -91,8 +91,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -102,6 +100,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -334,8 +334,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -345,6 +343,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -420,8 +420,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -430,6 +428,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") @@ -510,8 +510,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -521,6 +519,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -596,8 +596,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -607,6 +605,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -681,8 +681,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -692,6 +690,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -770,8 +770,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -781,6 +779,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -816,13 +816,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "default-listener", - Protocol: "turn-tcp", - Addr: "127.0.0.2", - Port: 12345, - MinRelayPort: 10, - MaxRelayPort: 100, - Routes: []string{"none", "dummy"}, + Name: "default-listener", + Protocol: "turn-tcp", + Addr: "127.0.0.2", + Port: 12345, + Routes: []string{"none", "dummy"}, }}, Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", @@ -846,8 +844,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 12345, "listener port ok") - assert.Equal(t, l.MinPort, 10, "listener minport ok") - assert.Equal(t, l.MaxPort, 100, "listener maxport ok") assert.Len(t, l.Routes, 2, "listener route count ok") // sorted!!! assert.Equal(t, l.Routes[0], "dummy", "listener route name ok") @@ -864,6 +860,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -898,13 +896,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "newlistener", - Protocol: "turn-tcp", - Addr: "127.0.0.2", - Port: 1, - MinRelayPort: 10, - MaxRelayPort: 100, - Routes: []string{"none", "dummy"}, + Name: "newlistener", + Protocol: "turn-tcp", + Addr: "127.0.0.2", + Port: 1, + Routes: []string{"none", "dummy"}, }}, Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", @@ -927,8 +923,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") - assert.Equal(t, l.MinPort, 10, "listener minport ok") - assert.Equal(t, l.MaxPort, 100, "listener maxport ok") assert.Len(t, l.Routes, 2, "listener route count ok") // sorted! assert.Equal(t, l.Routes[0], "dummy", "listener route name ok") @@ -938,6 +932,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -972,13 +968,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "newlistener", - Protocol: "turn-tls", - Addr: "127.0.0.2", - Port: 1, - MinRelayPort: 10, - MaxRelayPort: 100, - Routes: []string{"none", "dummy"}, + Name: "newlistener", + Protocol: "turn-tls", + Addr: "127.0.0.2", + Port: 1, + Routes: []string{"none", "dummy"}, }}, Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", @@ -1007,13 +1001,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Addr: "127.0.0.1", Routes: []string{"allow-any"}, }, { - Name: "newlistener", - Protocol: "turn-tcp", - Addr: "127.0.0.2", - Port: 1, - MinRelayPort: 10, - MaxRelayPort: 100, - Routes: []string{"none", "dummy"}, + Name: "newlistener", + Protocol: "turn-tcp", + Addr: "127.0.0.2", + Port: 1, + Routes: []string{"none", "dummy"}, }}, Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", @@ -1032,8 +1024,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -1041,6 +1031,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1067,8 +1059,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") - assert.Equal(t, l.MinPort, 10, "listener minport ok") - assert.Equal(t, l.MaxPort, 100, "listener maxport ok") assert.Len(t, l.Routes, 2, "listener route count ok") // sorted! assert.Equal(t, l.Routes[0], "dummy", "listener route name ok") @@ -1110,13 +1100,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Key: dummyKey64, Routes: []string{"allow-any"}, }, { - Name: "newlistener", - Protocol: "turn-tcp", - Addr: "127.0.0.2", - Port: 1, - MinRelayPort: 10, - MaxRelayPort: 100, - Routes: []string{"none", "dummy"}, + Name: "newlistener", + Protocol: "turn-tcp", + Addr: "127.0.0.2", + Port: 1, + Routes: []string{"none", "dummy"}, }}, Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", @@ -1141,8 +1129,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, bytes.Compare(l.Cert, []byte("dummy-cert")), 0, "listener cert ok") assert.Equal(t, bytes.Compare(l.Key, []byte("dummy-key")), 0, "listener key ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -1150,6 +1136,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1176,8 +1164,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") - assert.Equal(t, l.MinPort, 10, "listener minport ok") - assert.Equal(t, l.MaxPort, 100, "listener maxport ok") assert.Len(t, l.Routes, 2, "listener route count ok") // sorted! assert.Equal(t, l.Routes[0], "dummy", "listener route name ok") @@ -1219,13 +1205,11 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Key: dummyKey64, Routes: []string{"allow-any"}, }, { - Name: "newlistener", - Protocol: "turn-tcp", - Addr: "127.0.0.2", - Port: 1, - MinRelayPort: 10, - MaxRelayPort: 100, - Routes: []string{"none", "dummy"}, + Name: "newlistener", + Protocol: "turn-tcp", + Addr: "127.0.0.2", + Port: 1, + Routes: []string{"none", "dummy"}, }}, Clusters: []stnrv1.ClusterConfig{{ Name: "allow-any", @@ -1250,8 +1234,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, bytes.Compare(l.Cert, []byte("dummy-cert")), 0, "listener cert ok") assert.Equal(t, bytes.Compare(l.Key, []byte("dummy-key")), 0, "listener key ok") assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") - assert.Equal(t, l.MinPort, stnrv1.DefaultMinRelayPort, "listener minport ok") - assert.Equal(t, l.MaxPort, stnrv1.DefaultMaxRelayPort, "listener maxport ok") assert.Len(t, l.Routes, 1, "listener route count ok") assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") @@ -1259,6 +1241,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1285,8 +1269,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNTCP, "listener proto ok") assert.Equal(t, l.Addr.String(), "127.0.0.2", "listener address ok") assert.Equal(t, l.Port, 1, "listener port ok") - assert.Equal(t, l.MinPort, 10, "listener minport ok") - assert.Equal(t, l.MaxPort, 100, "listener maxport ok") assert.Len(t, l.Routes, 2, "listener route count ok") // sorted! assert.Equal(t, l.Routes[0], "dummy", "listener route name ok") @@ -1432,6 +1414,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1476,8 +1460,10 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Routes: []string{"allow-any"}, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "newcluster", - Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, + Name: "newcluster", + Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, + MinRelayPort: 10, + MaxRelayPort: 100, }, { Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, @@ -1505,6 +1491,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, 10, "minport ok") + assert.Equal(t, c.MaxPort, 100, "maxport ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ = net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") @@ -1575,6 +1563,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") + assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") _, n, _ = net.ParseCIDR("1.1.1.1/32") assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") diff --git a/relay.go b/relay.go new file mode 100644 index 00000000..ff9001b5 --- /dev/null +++ b/relay.go @@ -0,0 +1,226 @@ +package stunner + +// code adopted from github.com/livekit/pkg/telemetry + +import ( + "errors" + "fmt" + "net" + "sync" + "time" + + "github.com/pion/logging" + "github.com/pion/transport/v3" + "k8s.io/utils/lru" + + "github.com/l7mp/stunner/internal/object" + "github.com/l7mp/stunner/internal/telemetry" + "github.com/l7mp/stunner/pkg/logger" +) + +const ClusterCacheSize = 128 + +var ( + errNilConn = errors.New("cannot allocate relay connection") + errTodo = errors.New("relay to Net.Conn not implemented") +) + +var ( + ErrPortProhibited = errors.New("peer port administratively prohibited") + ErrInvalidPeerProtocol = errors.New("unknown peer transport protocol") +) + +type PortRangeChecker = func(addr net.Addr) (*object.Cluster, bool) + +// RelayGen can be used to only allocate connections inside a defined target port +// range. A static ip address can be set. +type RelayGen struct { + // Listener is the listener on behalf of which the relay address generator is created. + Listener *object.Listener + + // RelayAddress is the IP returned to the user when the relay is created. + RelayAddress net.IP + + // Address is passed to Listen/ListenPacket when creating the Relay. + Address string + + // ClusterCache is a cache that is used to couple relayed packets to clusters. + ClusterCache *lru.Cache + + // PortRangeChecker is a callback to check whether a peer address is allowed by any of the + // clusters of the listener. + PortRangeChecker PortRangeChecker + + // Net is a pion/transport VNet, used for testing. + Net transport.Net + + // Logger is a logger factory we can use to generate per-listener relay loggers. + Logger *logger.LeveledLoggerFactory +} + +func NewRelayGen(l *object.Listener, logger *logger.LeveledLoggerFactory) *RelayGen { + return &RelayGen{ + Listener: l, + RelayAddress: l.Addr, + Address: "0.0.0.0", + ClusterCache: lru.New(ClusterCacheSize), + Net: l.Net, + Logger: logger, + } +} + +// Validate is called on server startup and confirms the RelayAddressGenerator is properly configured. +func (r *RelayGen) Validate() error { + return nil +} + +// AllocatePacketConn generates a new transport relay connection and returns the IP/Port to be +// returned to the client in the allocation response. +func (r *RelayGen) AllocatePacketConn(network string, requestedPort int) (net.PacketConn, net.Addr, error) { + if requestedPort <= 1 || requestedPort > 2<<16-1 { + requestedPort = 0 + } + + conn, err := r.Net.ListenPacket(network, fmt.Sprintf("%s:%d", r.Address, requestedPort)) + if err != nil { + return nil, nil, err + } + + conn = NewPortRangePacketConn(conn, r.PortRangeChecker, + r.Logger.NewLogger(fmt.Sprintf("relay-%s", r.Listener.Name))) + + relayAddr, ok := conn.LocalAddr().(*net.UDPAddr) + if !ok { + return nil, nil, errNilConn + } + + relayAddr.IP = r.RelayAddress + return conn, relayAddr, nil +} + +// AllocateConn generates a new Conn to receive traffic on and the IP/Port to populate the +// allocation response with +func (g *RelayGen) AllocateConn(network string, requestedPort int) (net.Conn, net.Addr, error) { + return nil, nil, errTodo +} + +// GenPortRangeChecker finds the cluster that is responsible for routing the packet and checks +// whether the peer address is in the port range specified for the cluster. The RelayGen caches +// recent hits for simplicity. +func (s *Stunner) GenPortRangeChecker(g *RelayGen) PortRangeChecker { + return func(addr net.Addr) (*object.Cluster, bool) { + u, ok := addr.(*net.UDPAddr) + if !ok { + return nil, false + } + + ip := u.IP.String() + c, ok := g.ClusterCache.Get(ip) + var cluster *object.Cluster + if ok { + // cache hit + cluster = c.(*object.Cluster) + } else { + // route + for _, r := range g.Listener.Routes { + c := s.GetCluster(r) + if c != nil && c.Route(u.IP) { + cluster = c + g.ClusterCache.Add(ip, c) + break + } + } + } + + if cluster != nil { + return cluster, u.Port >= cluster.MinPort && u.Port <= cluster.MaxPort + } + + return nil, false + } +} + +// PortRangePacketConn is a net.PacketConn that filters on the target port range and also handles +// telemetry. +type PortRangePacketConn struct { + net.PacketConn + checker PortRangeChecker + log logging.LeveledLogger + readDeadline time.Time + lock sync.Mutex +} + +// NewPortRangePacketConn decorates a PacketConn with filtering on a target port range. Errors are reported per listener name. +func NewPortRangePacketConn(c net.PacketConn, checker PortRangeChecker, log logging.LeveledLogger) net.PacketConn { + // cluster add/sub connection is not tracked + // AddConnection(n, t) + r := PortRangePacketConn{ + PacketConn: c, + checker: checker, + log: log, + } + + return &r +} + +// WriteTo writes to the PacketConn. +func (c *PortRangePacketConn) WriteTo(p []byte, peerAddr net.Addr) (int, error) { + cluster, ok := c.checker(peerAddr) + if !ok { + return 0, ErrPortProhibited + } + + n, err := c.PacketConn.WriteTo(p, peerAddr) + if n > 0 { + telemetry.IncrementBytes(cluster.Name, telemetry.ClusterType, telemetry.Outgoing, uint64(n)) + telemetry.IncrementPackets(cluster.Name, telemetry.ClusterType, telemetry.Outgoing, 1) + } + + return n, err +} + +// ReadFrom reads from the PortRangePacketConn. Blocks until a packet from the speciifed port range +// is received and drops all other packets. +func (c *PortRangePacketConn) ReadFrom(p []byte) (int, net.Addr, error) { + for { + var peerAddr net.Addr + + err := c.PacketConn.SetReadDeadline(c.readDeadline) + if err != nil { + return 0, peerAddr, err + } + + n, peerAddr, err := c.PacketConn.ReadFrom(p) + + // Return errors unconditionally: peerAddr will most probably not be valid anyway + // so it is not worth checking + if err != nil { + return n, peerAddr, err + } + + cluster, ok := c.checker(peerAddr) + if !ok { + continue + } + + if n > 0 { + telemetry.IncrementBytes(cluster.Name, telemetry.ClusterType, telemetry.Incoming, uint64(n)) + telemetry.IncrementPackets(cluster.Name, telemetry.ClusterType, telemetry.Incoming, 1) + } + + return n, peerAddr, nil + } +} + +func (c *PortRangePacketConn) SetReadDeadline(t time.Time) error { + c.lock.Lock() + defer c.lock.Unlock() + c.readDeadline = t + return nil +} + +func (c *PortRangePacketConn) Close() error { + // cluster add/sub connection is not tracked + // SubConnection(c.name, c.connType) + return c.PacketConn.Close() +} diff --git a/internal/util/conn_test.go b/relay_test.go similarity index 89% rename from internal/util/conn_test.go rename to relay_test.go index 90e9a491..c5598f09 100644 --- a/internal/util/conn_test.go +++ b/relay_test.go @@ -1,14 +1,17 @@ -package util +package stunner import ( "net" "testing" "time" - "github.com/l7mp/stunner/pkg/logger" "github.com/pion/transport/v3/test" "github.com/pion/transport/v3/vnet" "github.com/stretchr/testify/assert" + + "github.com/l7mp/stunner/internal/object" + "github.com/l7mp/stunner/internal/telemetry" + "github.com/l7mp/stunner/pkg/logger" ) var connTestLoglevel string = "all:ERROR" @@ -18,7 +21,23 @@ var connTestLoglevel string = "all:ERROR" // var connTestLoglevel string = "all:TRACE" // var connTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" +var testCluster = object.Cluster{Name: "test-cluster"} + +func getChecker(minPort, maxPort int) PortRangeChecker { + return func(addr net.Addr) (*object.Cluster, bool) { + u, ok := addr.(*net.UDPAddr) + if !ok { + return nil, false + } + + return &testCluster, u.Port >= minPort && u.Port <= maxPort + } +} + func TestPortRangePacketConn(t *testing.T) { + telemetry.Init() + defer telemetry.Close() + lim := test.TimeOut(time.Second * 30) defer lim.Stop() @@ -42,7 +61,7 @@ func TestPortRangePacketConn(t *testing.T) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := NewPortRangePacketConn(baseConn, "test", 10000, 20000, log) + conn := NewPortRangePacketConn(baseConn, getChecker(10000, 20000), log) assert.NoError(t, err, "should create port-range filtered packetconn") log.Debug("Sending packet") @@ -72,7 +91,7 @@ func TestPortRangePacketConn(t *testing.T) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := NewPortRangePacketConn(baseConn, "test", 10000, 20000, log) + conn := NewPortRangePacketConn(baseConn, getChecker(10000, 20000), log) assert.NoError(t, err, "should create port-range filtered packetconn") log.Debug("Sending packet") @@ -101,7 +120,7 @@ func TestPortRangePacketConn(t *testing.T) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := NewPortRangePacketConn(baseConn, "test", 15000, 15000, log) + conn := NewPortRangePacketConn(baseConn, getChecker(15000, 15000), log) assert.NoError(t, err, "should create port-range filtered packetconn") log.Debug("Sending packet") @@ -126,6 +145,9 @@ func TestPortRangePacketConn(t *testing.T) { // BenchmarkPortRangePacketConn sends lots of invalid packets: this is mostly for testing the logger func BenchmarkPortRangePacketConn(b *testing.B) { + telemetry.Init() + defer telemetry.Close() + loggerFactory := logger.NewLoggerFactory(connTestLoglevel) log := loggerFactory.NewLogger("test") // relayLog := loggerFactory.WithRateLimiter(.25, 1).NewLogger("relay") @@ -146,7 +168,7 @@ func BenchmarkPortRangePacketConn(b *testing.B) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := WithCounter(NewPortRangePacketConn(baseConn, "test", 15000, 15000, relayLog)) + conn := WithCounter(NewPortRangePacketConn(baseConn, getChecker(15000, 15000), relayLog)) if err != nil { b.Fatalf("Cannot create port-range packetconn: %s", err.Error()) } diff --git a/server.go b/server.go index 0cd38966..56c91061 100644 --- a/server.go +++ b/server.go @@ -31,16 +31,8 @@ func (s *Stunner) StartServer(l *object.Listener) error { var pConns []turn.PacketConnConfig var lConns []turn.ListenerConfig - // listen on all IPs, relay to the listener address - relay := &util.RelayAddressGenerator{ - ListenerName: l.Name, - RelayAddress: l.Addr, - Address: "0.0.0.0", - MinRelayPort: l.MinPort, - MaxRelayPort: l.MaxPort, - Net: l.Net, - Logger: s.logger, - } + relay := NewRelayGen(l, s.logger) + relay.PortRangeChecker = s.GenPortRangeChecker(relay) permissionHandler := s.NewPermissionHandler(l) @@ -48,7 +40,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { switch l.Proto { case stnrv1.ListenerProtocolTURNUDP: - socketPool := util.NewPacketConnPool(l.Net, s.udpThreadNum) + socketPool := util.NewPacketConnPool(l.Name, l.Net, s.udpThreadNum) s.log.Infof("setting up UDP listener socket pool at %s with %d readloop threads", addr, socketPool.Size()) @@ -58,9 +50,8 @@ func (s *Stunner) StartServer(l *object.Listener) error { } for _, c := range conns { - udpListener := telemetry.NewPacketConn(c, l.Name, telemetry.ListenerType) conn := turn.PacketConnConfig{ - PacketConn: udpListener, + PacketConn: c, RelayAddressGenerator: relay, PermissionHandler: permissionHandler, } diff --git a/stunner_test.go b/stunner_test.go index bf6bf548..aa669ff2 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -1178,19 +1178,19 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "udp", - Protocol: "turn-udp", - Addr: "1.2.3.4", - Port: 3478, - MinRelayPort: 5670, - MaxRelayPort: 5680, + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, Routes: []string{ "echo-server-cluster", }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", + Name: "echo-server-cluster", + Type: "STATIC", + MinRelayPort: 5670, + MaxRelayPort: 5680, Endpoints: []string{ "1.2.3.5", }, @@ -1214,19 +1214,19 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "udp", - Protocol: "turn-udp", - Addr: "1.2.3.4", - Port: 3478, - MinRelayPort: 5678, - MaxRelayPort: 5678, + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, Routes: []string{ "echo-server-cluster", }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", + Name: "echo-server-cluster", + Type: "STATIC", + MinRelayPort: 5678, + MaxRelayPort: 5678, Endpoints: []string{ "1.2.3.5", }, @@ -1250,19 +1250,19 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "udp", - Protocol: "turn-udp", - Addr: "1.2.3.4", - Port: 3478, - MinRelayPort: 1, - MaxRelayPort: 5677, + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, Routes: []string{ "echo-server-cluster", }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", + Name: "echo-server-cluster", + Type: "STATIC", + MinRelayPort: 1, + MaxRelayPort: 5677, Endpoints: []string{ "1.2.3.5", }, @@ -1286,19 +1286,19 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }, Listeners: []stnrv1.ListenerConfig{{ - Name: "udp", - Protocol: "turn-udp", - Addr: "1.2.3.4", - Port: 3478, - MinRelayPort: 5677, - MaxRelayPort: 5677, + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, Routes: []string{ "echo-server-cluster", }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", + Name: "echo-server-cluster", + Type: "STATIC", + MinRelayPort: 5677, + MaxRelayPort: 5677, Endpoints: []string{ "1.2.3.5", }, From 7111782f569ea57eb88aea7a4895694ccbc2ee5a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 12 Dec 2023 16:42:10 +0100 Subject: [PATCH 070/243] refactor: TURN relay rewrite TURN relays now correctly identify the cluster (UDPRoute) on behalf of which peers connect. This makes it possible to enforce per-cluster (per UDPRoute) port ranges and (at last) report peer rx/tx packets per cluster (and not per listener, as it used to be the fairly confusing state before). --- internal/telemetry/metrics.go | 86 ++++++----- relay.go | 2 +- stunner_test.go | 279 ++++++++++++++++++++++++++++++++-- 3 files changed, 317 insertions(+), 50 deletions(-) diff --git a/internal/telemetry/metrics.go b/internal/telemetry/metrics.go index 3f107064..88236fd6 100644 --- a/internal/telemetry/metrics.go +++ b/internal/telemetry/metrics.go @@ -1,7 +1,6 @@ package telemetry import ( - // "github.com/pion/logging" "github.com/prometheus/client_golang/prometheus" ) @@ -10,49 +9,49 @@ const ( ) var ( - promConnLabels = []string{"name"} - promCounterLabels = []string{"name", "direction"} - promListenerPacketsTotal *prometheus.CounterVec - promListenerBytesTotal *prometheus.CounterVec - promListenerConnsTotal *prometheus.CounterVec - promListenerConnsActive *prometheus.GaugeVec - promClusterPacketsTotal *prometheus.CounterVec - promClusterBytesTotal *prometheus.CounterVec + ConnLabels = []string{"name"} + CounterLabels = []string{"name", "direction"} + ListenerPacketsTotal *prometheus.CounterVec + ListenerBytesTotal *prometheus.CounterVec + ListenerConnsTotal *prometheus.CounterVec + ListenerConnsActive *prometheus.GaugeVec + ClusterPacketsTotal *prometheus.CounterVec + ClusterBytesTotal *prometheus.CounterVec // promClusterConnsTotal *prometheus.CounterVec // promClusterConnsActive *prometheus.GaugeVec ) func Init() { // listener stats - promListenerConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + ListenerConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Namespace: stunnerNamespace, Subsystem: "listener", Name: "connections", Help: "Number of active downstream connections at a listener.", - }, promConnLabels) - promListenerConnsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ + }, ConnLabels) + ListenerConnsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: stunnerNamespace, Subsystem: "listener", Name: "connections_total", Help: "Number of downstream connections at a listener.", - }, promConnLabels) - promListenerPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ + }, ConnLabels) + ListenerPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: stunnerNamespace, Subsystem: "listener", Name: "packets_total", Help: "Number of datagrams sent or received at a listener.", - }, promCounterLabels) - promListenerBytesTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ + }, CounterLabels) + ListenerBytesTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: stunnerNamespace, Subsystem: "listener", Name: "bytes_total", Help: "Number of bytes sent or received at a listener.", - }, promCounterLabels) + }, CounterLabels) - prometheus.MustRegister(promListenerPacketsTotal) - prometheus.MustRegister(promListenerBytesTotal) - prometheus.MustRegister(promListenerConnsTotal) - prometheus.MustRegister(promListenerConnsActive) + prometheus.MustRegister(ListenerPacketsTotal) + prometheus.MustRegister(ListenerBytesTotal) + prometheus.MustRegister(ListenerConnsTotal) + prometheus.MustRegister(ListenerConnsActive) // cluster stats // promClusterConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ @@ -67,32 +66,32 @@ func Init() { // Name: "connections_total", // Help: "Number of upstream connections on behalf of a listener.", // }, promConnLabels) - promClusterPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ + ClusterPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: stunnerNamespace, Subsystem: "cluster", Name: "packets_total", Help: "Number of datagrams sent to backends or received from backends on behalf of a listener", - }, promCounterLabels) - promClusterBytesTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ + }, CounterLabels) + ClusterBytesTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: stunnerNamespace, Subsystem: "cluster", Name: "bytes_total", Help: "Number of bytes sent to backends or received from backends on behalf of a listener.", - }, promCounterLabels) + }, CounterLabels) - prometheus.MustRegister(promClusterPacketsTotal) - prometheus.MustRegister(promClusterBytesTotal) + prometheus.MustRegister(ClusterPacketsTotal) + prometheus.MustRegister(ClusterBytesTotal) // prometheus.MustRegister(promClusterConnsTotal) // prometheus.MustRegister(promClusterConnsActive) } func Close() { - _ = prometheus.Unregister(promListenerPacketsTotal) - _ = prometheus.Unregister(promListenerBytesTotal) - _ = prometheus.Unregister(promListenerConnsTotal) - _ = prometheus.Unregister(promListenerConnsActive) - _ = prometheus.Unregister(promClusterPacketsTotal) - _ = prometheus.Unregister(promClusterBytesTotal) + _ = prometheus.Unregister(ListenerPacketsTotal) + _ = prometheus.Unregister(ListenerBytesTotal) + _ = prometheus.Unregister(ListenerConnsTotal) + _ = prometheus.Unregister(ListenerConnsActive) + _ = prometheus.Unregister(ClusterPacketsTotal) + _ = prometheus.Unregister(ClusterBytesTotal) // _ = prometheus.Unregister(promClusterConnsTotal) // _ = prometheus.Unregister(promClusterConnsActive) } @@ -100,26 +99,26 @@ func Close() { func IncrementPackets(n string, c ConnType, d Direction, count uint64) { switch c { case ListenerType: - promListenerPacketsTotal.WithLabelValues(n, d.String()).Add(float64(count)) + ListenerPacketsTotal.WithLabelValues(n, d.String()).Add(float64(count)) case ClusterType: - promClusterPacketsTotal.WithLabelValues(n, d.String()).Add(float64(count)) + ClusterPacketsTotal.WithLabelValues(n, d.String()).Add(float64(count)) } } func IncrementBytes(n string, c ConnType, d Direction, count uint64) { switch c { case ListenerType: - promListenerBytesTotal.WithLabelValues(n, d.String()).Add(float64(count)) + ListenerBytesTotal.WithLabelValues(n, d.String()).Add(float64(count)) case ClusterType: - promClusterBytesTotal.WithLabelValues(n, d.String()).Add(float64(count)) + ClusterBytesTotal.WithLabelValues(n, d.String()).Add(float64(count)) } } func AddConnection(n string, c ConnType) { switch c { case ListenerType: - promListenerConnsActive.WithLabelValues(n).Add(1) - promListenerConnsTotal.WithLabelValues(n).Add(1) + ListenerConnsActive.WithLabelValues(n).Add(1) + ListenerConnsTotal.WithLabelValues(n).Add(1) case ClusterType: // promClusterConnsActive.WithLabelValues(n).Add(1) // promClusterConnsTotal.WithLabelValues(n).Add(1) @@ -129,7 +128,7 @@ func AddConnection(n string, c ConnType) { func SubConnection(n string, c ConnType) { switch c { case ListenerType: - promListenerConnsActive.WithLabelValues(n).Sub(1) + ListenerConnsActive.WithLabelValues(n).Sub(1) case ClusterType: // promClusterConnsActive.WithLabelValues(n).Sub(1) } @@ -159,3 +158,10 @@ func SubConnection(n string, c ConnType) { // } // log.Warn("GaugeFunc 'stunner_allocations_active' cannot be unregistered.") // } + +// func GetListenerPacketsTotal(ch chan prometheus.Metric) { +// go func() { +// defer close(ch) +// promListenerPacketsTotal.Collect(ch) +// }() +// } diff --git a/relay.go b/relay.go index ff9001b5..037afc8f 100644 --- a/relay.go +++ b/relay.go @@ -18,7 +18,7 @@ import ( "github.com/l7mp/stunner/pkg/logger" ) -const ClusterCacheSize = 128 +const ClusterCacheSize = 512 var ( errNilConn = errors.New("cannot allocate relay connection") diff --git a/stunner_test.go b/stunner_test.go index aa669ff2..2225bd5f 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -19,9 +19,11 @@ import ( "github.com/pion/transport/v3/test" "github.com/pion/transport/v3/vnet" "github.com/pion/turn/v3" + "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/assert" "github.com/l7mp/stunner/internal/resolver" + "github.com/l7mp/stunner/internal/telemetry" "github.com/l7mp/stunner/pkg/logger" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" @@ -653,6 +655,7 @@ type StunnerTestClusterConfig struct { config stnrv1.StunnerConfig echoServerAddr string result bool + tester func(t *testing.T) } var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ @@ -1162,7 +1165,6 @@ func TestStunnerClusterWithVNet(t *testing.T) { // Port range filtering tests with VNet // ***************** var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ - // port range filtering { testName: "static endpoint with peer address in the admitted port range ok", config: stnrv1.StunnerConfig{ @@ -1198,6 +1200,35 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: true, + tester: func(t *testing.T) { + c := telemetry.ListenerConnsTotal + assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + g := telemetry.ListenerConnsActive + assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + c = telemetry.ListenerPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(500)) + + c = telemetry.ListenerBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(2000)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(2000)) + + c = telemetry.ClusterPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(500)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(500)) + + c = telemetry.ClusterBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(2000)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) + }, }, { testName: "static endpoint with peer address matching singleton admitted port ok", @@ -1222,18 +1253,53 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ "echo-server-cluster", }, }}, - Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", - MinRelayPort: 5678, - MaxRelayPort: 5678, - Endpoints: []string{ - "1.2.3.5", + Clusters: []stnrv1.ClusterConfig{ + { + Name: "dummy-cluster", + Type: "STATIC", + MinRelayPort: 5678, + MaxRelayPort: 5678, + Endpoints: []string{"1.2.3.6"}, + }, { + Name: "echo-server-cluster", + Type: "STATIC", + MinRelayPort: 5678, + MaxRelayPort: 5678, + Endpoints: []string{"1.2.3.5"}, }, - }}, + }, }, echoServerAddr: "1.2.3.5:5678", result: true, + tester: func(t *testing.T) { + c := telemetry.ListenerConnsTotal + assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + g := telemetry.ListenerConnsActive + assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + c = telemetry.ListenerPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(500)) + + c = telemetry.ListenerBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(2000)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(2000)) + + c = telemetry.ClusterPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(500)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(500)) + + c = telemetry.ClusterBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(2000)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) + }, }, { testName: "static endpoint with peer address in rejected port range fails", @@ -1270,6 +1336,35 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: false, + tester: func(t *testing.T) { + c := telemetry.ListenerConnsTotal + assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + g := telemetry.ListenerConnsActive + assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + c = telemetry.ListenerPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data + assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling + + c = telemetry.ListenerBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data + assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling + + c = telemetry.ClusterPacketsTotal + assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + + c = telemetry.ClusterBytesTotal + assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + }, }, { testName: "static endpoint with peer address in rejected singleton port fails", @@ -1306,6 +1401,167 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: false, + tester: func(t *testing.T) { + c := telemetry.ListenerConnsTotal + assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + g := telemetry.ListenerConnsActive + assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + c = telemetry.ListenerPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data + assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling + + c = telemetry.ListenerBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data + assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling + + c = telemetry.ClusterPacketsTotal + assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + + c = telemetry.ClusterBytesTotal + assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + }, + }, + { + testName: "strict_dns with default port range ok", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Type: "static", + Credentials: map[string]string{ + "username": "user1", + "password": "passwd1", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{ + { + Name: "dummy-cluster", + Type: "STATIC", + Endpoints: []string{"1.2.3.6"}, + }, { + Name: "echo-server-cluster", + Type: "STRICT_DNS", + Endpoints: []string{"echo-server.l7mp.io"}, + }, + }, + }, + echoServerAddr: "1.2.3.5:5678", + result: true, + tester: func(t *testing.T) { + c := telemetry.ListenerConnsTotal + assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + g := telemetry.ListenerConnsActive + assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + c = telemetry.ListenerPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(500)) + + c = telemetry.ListenerBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(2000)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(2000)) + + c = telemetry.ClusterPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(500)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(500)) + + c = telemetry.ClusterBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(2000)) + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) + }, + }, + { + testName: "strict_dns with prohibited port range fails", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Type: "static", + Credentials: map[string]string{ + "username": "user1", + "password": "passwd1", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Type: "STRICT_DNS", + MinRelayPort: 1, + MaxRelayPort: 1, + Endpoints: []string{ + "echo-server.l7mp.io", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + result: false, + tester: func(t *testing.T) { + c := telemetry.ListenerConnsTotal + assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + g := telemetry.ListenerConnsActive + assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + c = telemetry.ListenerPacketsTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data + assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling + + c = telemetry.ListenerBytesTotal + assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data + assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling + + c = telemetry.ClusterPacketsTotal + assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + + c = telemetry.ClusterBytesTotal + assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + }, }, } @@ -1372,6 +1628,10 @@ func TestStunnerPortRangeWithVNet(t *testing.T) { c.echoServerAddr, true, true, c.result, loggerFactory} stunnerEchoFloodTest(testConfig) + if c.tester != nil { + c.tester(t) + } + assert.NoError(t, lconn.Close(), "cannot close TURN client connection") stunner.Close() assert.NoError(t, v.Close(), "cannot close VNet") @@ -1480,6 +1740,7 @@ func stunnerEchoFloodTest(conf echoTestConfig) { assert.NoError(t, echoConn.Close(), "cannot close echo server connection") } } + time.Sleep(150 * time.Millisecond) client.Close() } From fb1b1127d9605017779d0c1025c014067eadd73d Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 13 Dec 2023 22:49:23 +0100 Subject: [PATCH 071/243] feature: Per-cluster port range filtering --- internal/object/cluster.go | 74 ++++----- internal/util/endpoint.go | 106 ++++++++++++ internal/util/endpoint_test.go | 159 ++++++++++++++++++ pkg/apis/v1/cluster.go | 38 ++--- reconcile_test.go | 286 +++++++++++++++++++++++++-------- relay.go | 2 +- stunner_test.go | 171 ++++++++++---------- 7 files changed, 603 insertions(+), 233 deletions(-) create mode 100644 internal/util/endpoint.go create mode 100644 internal/util/endpoint_test.go diff --git a/internal/object/cluster.go b/internal/object/cluster.go index 743220c1..67d4daa9 100644 --- a/internal/object/cluster.go +++ b/internal/object/cluster.go @@ -15,13 +15,12 @@ import ( // Listener implements a STUNner cluster type Cluster struct { - Name string - Type stnrv1.ClusterType - Protocol stnrv1.ClusterProtocol - MinPort, MaxPort int - Endpoints []net.IPNet - Domains []string - Resolver resolver.DnsResolver // for strict DNS + Name string + Type stnrv1.ClusterType + Protocol stnrv1.ClusterProtocol + Endpoints []*util.Endpoint + Domains []string + Resolver resolver.DnsResolver // for strict DNS logger logging.LoggerFactory log logging.LeveledLogger @@ -41,9 +40,7 @@ func NewCluster(conf stnrv1.Config, resolver resolver.DnsResolver, logger loggin c := Cluster{ Name: req.Name, - Endpoints: []net.IPNet{}, - MinPort: req.MinRelayPort, - MaxPort: req.MaxRelayPort, + Endpoints: []*util.Endpoint{}, Domains: []string{}, Resolver: resolver, logger: logger, @@ -79,8 +76,6 @@ func (c *Cluster) Reconcile(conf stnrv1.Config) error { c.log.Tracef("Reconcile: %s", req.String()) c.Type, _ = stnrv1.NewClusterType(req.Type) c.Protocol, _ = stnrv1.NewClusterProtocol(req.Protocol) - c.MinPort = req.MinRelayPort - c.MaxPort = req.MaxRelayPort switch c.Type { case stnrv1.ClusterTypeStatic: @@ -88,36 +83,16 @@ func (c *Cluster) Reconcile(conf stnrv1.Config) error { c.Endpoints = c.Endpoints[:0] for _, e := range req.Endpoints { // try to parse as a subnet - _, n, err := net.ParseCIDR(e) - if err == nil { - c.Endpoints = append(c.Endpoints, *n) - continue - } - - // try to parse as an IP address - a := net.ParseIP(e) - if a == nil { - c.log.Warnf("cluster %q: invalid endpoint IP: %q, ignoring", c.Name, e) - continue - } - - // add a prefix and reparse - if a.To4() == nil { - e = e + "/128" - } else { - e = e + "/32" - } - - _, n2, err := net.ParseCIDR(e) + ep, err := util.ParseEndpoint(e) if err != nil { - c.log.Warnf("cluster %q: could not convert endpoint %q to CIDR subnet ", + c.log.Warnf("cluster %q: could not parse endpoint %q ", "(ignoring): %s", c.Name, e, err.Error()) - continue } - c.Endpoints = append(c.Endpoints, *n2) + c.Endpoints = append(c.Endpoints, ep) } case stnrv1.ClusterTypeStrictDNS: + // TOD: port-range support for DNS clusters if c.Resolver == nil { return fmt.Errorf("STRICT_DNS cluster %q initialized with no DNS resolver", c.Name) } @@ -153,19 +128,16 @@ func (c *Cluster) ObjectType() string { // GetConfig returns the configuration of the running cluster. func (c *Cluster) GetConfig() stnrv1.Config { conf := stnrv1.ClusterConfig{ - Name: c.Name, - Protocol: c.Protocol.String(), - Type: c.Type.String(), - MinRelayPort: c.MinPort, - MaxRelayPort: c.MaxPort, + Name: c.Name, + Protocol: c.Protocol.String(), + Type: c.Type.String(), } switch c.Type { case stnrv1.ClusterTypeStatic: conf.Endpoints = make([]string, len(c.Endpoints)) for i, e := range c.Endpoints { - // e.String() adds a /32 at the end of IPs, remove - conf.Endpoints[i] = strings.TrimRight(e.String(), "/32") + conf.Endpoints[i] = e.String() } case stnrv1.ClusterTypeStrictDNS: conf.Endpoints = make([]string, len(c.Domains)) @@ -192,24 +164,33 @@ func (c *Cluster) Close() error { return nil } -// Route decides whwther a peer IP appears among the permitted endpoints of a cluster. +// Route decides whether a peer IP appears among the permitted endpoints of a cluster. func (c *Cluster) Route(peer net.IP) bool { + return c.Match(peer, 0) +} + +// Match decides whether a peer IP and port matches one of the permitted endpoints of a cluster. If +// port is zero then port-matching is disabled. +func (c *Cluster) Match(peer net.IP, port int) bool { + c.log.Tracef("Route: cluster %q of type %s, peer IP: %s", c.Name, c.Type.String(), peer.String()) switch c.Type { case stnrv1.ClusterTypeStatic: // endpoints are IPNets + c.log.Tracef("route: STATIC cluster with %d endpoints", len(c.Endpoints)) + for _, e := range c.Endpoints { c.log.Tracef("considering endpoint %q", e) - if e.Contains(peer) { + if e.Match(peer, port) { return true } } case stnrv1.ClusterTypeStrictDNS: // endpoints are obtained from the DNS - c.log.Tracef("running STRICT_DNS cluster with domains: [%s]", strings.Join(c.Domains, ", ")) + c.log.Tracef("route: STRICT_DNS cluster with domains: [%s]", strings.Join(c.Domains, ", ")) for _, d := range c.Domains { c.log.Tracef("considering domain %q", d) @@ -221,7 +202,6 @@ func (c *Cluster) Route(peer net.IP) bool { for _, n := range hs { c.log.Tracef("considering IP address %q", n) - if n.Equal(peer) { return true } diff --git a/internal/util/endpoint.go b/internal/util/endpoint.go new file mode 100644 index 00000000..f33b466e --- /dev/null +++ b/internal/util/endpoint.go @@ -0,0 +1,106 @@ +package util + +import ( + "fmt" + "net" + "regexp" + "strconv" +) + +var endPointMatcher = regexp.MustCompile("^(.*):<([0-9]+)-([0-9]+)>$") + +// Endpoint is a pair of an IP prefix and a port range. +type Endpoint struct { + prefix net.IPNet + port, endPort int + hasPrefixLen, hasPort bool +} + +// ParseEndpoint parses an endpoint from the canonical format: "[optional slash and prefix length]:= port + } else { + return true + } +} + +func (ep *Endpoint) Network() string { + return ep.prefix.Network() +} + +func (ep *Endpoint) String() string { + ip, portRange := "", "" + if ep.hasPrefixLen { + ip = ep.prefix.String() + } else { + ip = ep.prefix.IP.String() + } + if ep.hasPort { + portRange = fmt.Sprintf(":<%d-%d>", ep.port, ep.endPort) + } + + return ip + portRange +} diff --git a/internal/util/endpoint_test.go b/internal/util/endpoint_test.go new file mode 100644 index 00000000..6ed70274 --- /dev/null +++ b/internal/util/endpoint_test.go @@ -0,0 +1,159 @@ +package util + +import ( + "net" + "testing" + + "github.com/stretchr/testify/assert" +) + +type endpointTest struct { + name, input, output, ipnet string + port, endPort int + success bool +} + +var endpointTester = []endpointTest{ + { + name: "ipv4 - full", + input: "1.2.3.4/16:<1-2>", + output: "1.2.0.0/16:<1-2>", + ipnet: "1.2.0.0/16", + port: 1, + endPort: 2, + success: true, + }, + { + name: "ipv4 - no port", + input: "1.2.3.4/16", + output: "1.2.0.0/16", + ipnet: "1.2.0.0/16", + port: 1, + endPort: 65535, + success: true, + }, + { + name: "ipv4 - no prefix len", + input: "1.2.3.4:<1-2>", + output: "1.2.3.4:<1-2>", + ipnet: "1.2.3.4/32", + port: 1, + endPort: 2, + success: true, + }, + { + name: "ipv4 - no port, no prefix len ", + input: "1.2.3.4", + output: "1.2.3.4", + ipnet: "1.2.3.4/32", + port: 1, + endPort: 65535, + success: true, + }, + { + name: "ipv6 - full", + input: "2001:db8:3333:4444:5555:6666:7777:8888/32:<1-2>", + output: "2001:db8::/32:<1-2>", + ipnet: "2001:db8::/32", + port: 1, + endPort: 2, + success: true, + }, + { + name: "ipv6 - no port", + input: "2001:db8:3333:4444:5555:6666:7777:8888/32", + output: "2001:db8::/32", + ipnet: "2001:db8::/32", + port: 1, + endPort: 65535, + success: true, + }, + { + name: "ipv6 - no prefix len", + input: "2001:db8:3333:4444:5555:6666:7777:8888:<1-2>", + output: "2001:db8:3333:4444:5555:6666:7777:8888:<1-2>", + ipnet: "2001:db8:3333:4444:5555:6666:7777:8888/128", + port: 1, + endPort: 2, + success: true, + }, + { + name: "ipv6 - no port, no prefix len ", + input: "2001:db8:3333:4444:5555:6666:7777:8888", + output: "2001:db8:3333:4444:5555:6666:7777:8888", + ipnet: "2001:db8:3333:4444:5555:6666:7777:8888/128", + port: 1, + endPort: 65535, + success: true, + }, + { + name: "ipv4 - no addr fails ", + input: ":<1-65535>", + success: false, + }, + { + name: "ipv4 - random stuff fails ", + input: "dummy", + success: false, + }, +} + +func TestEndpointParse(t *testing.T) { + for _, c := range endpointTester { + t.Run(c.name, func(t *testing.T) { + ep, err := ParseEndpoint(c.input) + if c.success { + assert.NoError(t, err, "parse") + assert.Equal(t, c.ipnet, ep.prefix.String(), "ip equal") + assert.Equal(t, c.port, ep.port, "port equal") + assert.Equal(t, c.endPort, ep.endPort, "endport equal") + assert.Equal(t, c.output, ep.String(), "output") + } else { + assert.Error(t, err, "parse") + } + + }) + } +} + +type matchTest struct { + name, input, ip string + port int + match, route bool +} + +var matchTester = []matchTest{{ + name: "ipv4 - full - both", + input: "1.2.3.4/16:<1-2>", + ip: "1.2.3.5", + port: 1, + match: true, + route: true, +}, { + name: "ipv4 - full - route", + input: "1.2.3.4/16:<1-2>", + ip: "1.2.4.6", + port: 3, + match: false, + route: true, +}, { + name: "ipv4 - full - neither", + input: "1.2.3.4/16:<1-2>", + ip: "1.3.3.4", + port: 1, + match: false, + route: false, +}} + +func TestRouteMatch(t *testing.T) { + for _, c := range matchTester { + t.Run(c.name, func(t *testing.T) { + ep, err := ParseEndpoint(c.input) + assert.NoError(t, err, "endpoint parse") + ip := net.ParseIP(c.ip) + assert.NotNil(t, ip, "ip parse") + assert.True(t, ep.Contains(ip) == c.route, "route") + assert.True(t, ep.Match(ip, c.port) == c.match, "match") + }) + } +} diff --git a/pkg/apis/v1/cluster.go b/pkg/apis/v1/cluster.go index 5fc2b75d..29c3a789 100644 --- a/pkg/apis/v1/cluster.go +++ b/pkg/apis/v1/cluster.go @@ -5,6 +5,8 @@ import ( "reflect" "sort" "strings" + + "github.com/l7mp/stunner/internal/util" ) // ClusterConfig specifies a set of upstream peers to which STUNner can open transport relay @@ -23,14 +25,6 @@ type ClusterConfig struct { // Protocol specifies the protocol to be used with the cluster, either UDP (default) or TCP // (not implemented yet). Protocol string `json:"protocol,omitempty"` - // MinRelayPort is the lowest peer target port admitted on the cluster (inclusive). The - // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport - // relay connections created via the cluster. Default is 1. - MinRelayPort int `json:"min_relay_port,omitempty"` - // MaxRelayPort is the highest peer target port admitted on the cluster (inclusive). The - // interval [MinRelayPort:MaxRelayPort] specifies the port range reachable on the transport - // relay connections created via the cluster. Default is 65535. - MaxRelayPort int `json:"max_relay_port,omitempty"` // Endpoints specifies the peers that can be reached via this cluster. Endpoints []string `json:"endpoints,omitempty"` } @@ -61,21 +55,14 @@ func (req *ClusterConfig) Validate() error { } req.Protocol = p.String() - if req.MinRelayPort == 0 { - req.MinRelayPort = DefaultMinRelayPort - } - if req.MaxRelayPort == 0 { - req.MaxRelayPort = DefaultMaxRelayPort - } - for _, p := range []int{req.MinRelayPort, req.MaxRelayPort} { - if p <= 0 || p > 65535 { - return fmt.Errorf("invalid port: %d", p) + // Do endpoints parse? + if t == ClusterTypeStatic { + for _, ep := range req.Endpoints { + if _, err := util.ParseEndpoint(ep); err != nil { + return err + } } } - if req.MinRelayPort > req.MaxRelayPort { - return fmt.Errorf("invalid relay port range: min port (%d) <= max port (%d) must hold", - req.MinRelayPort, req.MaxRelayPort) - } sort.Strings(req.Endpoints) return nil @@ -116,15 +103,8 @@ func (req *ClusterConfig) String() string { status = append(status, fmt.Sprintf("protocol=%q", req.Protocol)) } - min, max := 0, 65535 - if req.MinRelayPort != 0 { - min = req.MinRelayPort - } - if req.MaxRelayPort != 0 { - max = req.MaxRelayPort - } status = append(status, fmt.Sprintf("endpoints=[%s]", strings.Join(req.Endpoints, ","))) - return fmt.Sprintf("%q:{%s}<%d-%d>", n, strings.Join(status, ","), min, max) + return fmt.Sprintf("%q:{%s}", n, strings.Join(status, ",")) } diff --git a/reconcile_test.go b/reconcile_test.go index fdea401b..25e8f1df 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -100,11 +100,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -343,11 +340,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -428,12 +422,9 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ c := s.GetCluster("allow-any") assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -519,11 +510,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -605,11 +593,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -690,11 +675,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -779,11 +761,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the open cluster for routing @@ -860,11 +839,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the old cluster for routing @@ -932,11 +908,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the old cluster for routing @@ -1031,11 +1004,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the old cluster for routing @@ -1136,11 +1106,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the old cluster for routing @@ -1241,11 +1208,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") // listener uses the old cluster for routing @@ -1356,11 +1320,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") - _, n, _ := net.ParseCIDR("1.1.1.1/32") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") - assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") - _, n, _ = net.ParseCIDR("2.2.2.2/8") - assert.IsType(t, c.Endpoints[1], *n, "cluster endpoint type ok") + assert.Equal(t, c.Endpoints[0].String(), "1.1.1.1", "cluster endpoint ok") + _, n, _ := net.ParseCIDR("2.2.2.2/8") assert.Equal(t, c.Endpoints[1].String(), n.String(), "cluster endpoint ok") l := s.GetListener("default-listener") @@ -1414,14 +1375,9 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") - _, n, _ := net.ParseCIDR("1.1.1.1/32") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") - assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") - _, n, _ = net.ParseCIDR("2.2.2.2/8") - assert.IsType(t, c.Endpoints[1], *n, "cluster endpoint type ok") + assert.Equal(t, c.Endpoints[0].String(), "1.1.1.1", "cluster endpoint ok") + _, n, _ := net.ParseCIDR("2.2.2.2/8") assert.Equal(t, c.Endpoints[1].String(), n.String(), "cluster endpoint ok") l := s.GetListener("default-listener") @@ -1441,6 +1397,62 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ net.ParseIP("3.0.0.0")), "route to 3.0.0.0 fails") }, }, + { + name: "reconcile-test: reconcile cluster with port range", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Credentials: map[string]string{ + "username": "user", + "password": "pass", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "default-listener", + Addr: "127.0.0.1", + Routes: []string{"allow-any"}, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "newcluster", + Endpoints: []string{"1.1.1.1:<1-2>", "2.2.2.2/8:<3-4>"}, + }}, + }, + tester: func(t *testing.T, s *Stunner, err error) { + assert.NoError(t, err, err) + + assert.Len(t, s.clusterManager.Keys(), 1, "clusterManager keys") + + c := s.GetCluster("allow-any") + assert.Nil(t, c, "cluster found") + + c = s.GetCluster("newcluster") + assert.NotNil(t, c, "cluster found") + assert.IsType(t, c, &object.Cluster{}, "cluster type ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") + assert.Equal(t, c.Endpoints[0].String(), "1.1.1.1:<1-2>", "cluster endpoint ok") + assert.Equal(t, c.Endpoints[1].String(), "2.0.0.0/8:<3-4>", "cluster endpoint ok") + + l := s.GetListener("default-listener") + p := s.NewPermissionHandler(l) + assert.NotNil(t, p, "permission handler exists") + + // listener still uses the old cluster for routing + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("1.1.1.1")), "route to 1.1.1.1 ok") + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("1.1.1.2")), "route to 1.1.1.2 fails") + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("2.2.2.2")), "route to 2.2.2.2 fails") + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("2.128.3.3")), "route to 2.128.3.3 fails") + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("3.0.0.0")), "route to 3.0.0.0 fails") + }, + }, { name: "reconcile-test: reconcile additional cluster", config: stnrv1.StunnerConfig{ @@ -1460,10 +1472,8 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ Routes: []string{"allow-any"}, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "newcluster", - Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, - MinRelayPort: 10, - MaxRelayPort: 100, + Name: "newcluster", + Endpoints: []string{"1.1.1.1", "2.2.2.2/8"}, }, { Name: "allow-any", Endpoints: []string{"0.0.0.0/0"}, @@ -1480,7 +1490,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") l := s.GetListener("default-listener") @@ -1491,14 +1500,9 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, 10, "minport ok") - assert.Equal(t, c.MaxPort, 100, "maxport ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") - _, n, _ = net.ParseCIDR("1.1.1.1/32") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") - assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") + assert.Equal(t, c.Endpoints[0].String(), "1.1.1.1", "cluster endpoint ok") _, n, _ = net.ParseCIDR("2.2.2.2/8") - assert.IsType(t, c.Endpoints[1], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[1].String(), n.String(), "cluster endpoint ok") // listener still uses the old open cluster for routing @@ -1552,7 +1556,6 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") _, n, _ := net.ParseCIDR("0.0.0.0/0") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") l := s.GetListener("default-listener") @@ -1563,14 +1566,9 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ assert.NotNil(t, c, "cluster found") assert.IsType(t, c, &object.Cluster{}, "cluster type ok") assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") - assert.Equal(t, c.MinPort, stnrv1.DefaultMinRelayPort, "minport ok") - assert.Equal(t, c.MaxPort, stnrv1.DefaultMaxRelayPort, "maxport ok") assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") - _, n, _ = net.ParseCIDR("1.1.1.1/32") - assert.IsType(t, c.Endpoints[0], *n, "cluster endpoint type ok") - assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") + assert.Equal(t, c.Endpoints[0].String(), "1.1.1.1", "cluster endpoint ok") _, n, _ = net.ParseCIDR("2.2.2.2/8") - assert.IsType(t, c.Endpoints[1], *n, "cluster endpoint type ok") assert.Equal(t, c.Endpoints[1].String(), n.String(), "cluster endpoint ok") assert.True(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, @@ -1585,6 +1583,92 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ net.ParseIP("3.0.0.0")), "route to 3.0.0.0 fails") }, }, + { + name: "reconcile-test: reconcile port-range", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Credentials: map[string]string{ + "username": "user", + "password": "pass", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "default-listener", + Addr: "127.0.0.1", + Routes: []string{"newcluster"}, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "newcluster", + Endpoints: []string{"1.1.1.1:<1-2>", "2.2.2.2/8:<3-4>"}, + }, { + Name: "allow-any", + Endpoints: []string{"0.0.0.0/0"}, + }}, + }, + tester: func(t *testing.T, s *Stunner, err error) { + // only routes have changed, we shouldn't need a restart + assert.NoError(t, err, err) + + assert.Len(t, s.clusterManager.Keys(), 2, "clusterManager keys") + + c := s.GetCluster("allow-any") + assert.NotNil(t, c, "cluster found") + assert.IsType(t, c, &object.Cluster{}, "cluster type ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") + _, n, _ := net.ParseCIDR("0.0.0.0/0") + assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") + + l := s.GetListener("default-listener") + p := s.NewPermissionHandler(l) + assert.NotNil(t, p, "permission handler exists") + + c = s.GetCluster("newcluster") + assert.NotNil(t, c, "cluster found") + assert.IsType(t, c, &object.Cluster{}, "cluster type ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Len(t, c.Endpoints, 2, "cluster endpoint count ok") + assert.Equal(t, c.Endpoints[0].String(), "1.1.1.1:<1-2>", "cluster endpoint ok") + assert.Equal(t, c.Endpoints[1].String(), "2.0.0.0/8:<3-4>", "cluster endpoint ok") + + assert.True(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("1.1.1.1")), "route to 1.1.1.1 ok") + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("1.1.1.2")), "route to 1.1.1.2 fails") + assert.True(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("2.2.2.2")), "route to 2.2.2.2 ok") + assert.True(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("2.128.3.3")), "route to 2.128.3.3 ok") + assert.False(t, p(&net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 1234}, + net.ParseIP("3.0.0.0")), "route to 3.0.0.0 fails") + + assert.True(t, c.Route(net.ParseIP("1.1.1.1")), "route to 1.1.1.1 ok") + assert.False(t, c.Route(net.ParseIP("1.1.1.2")), "route to 1.1.1.2 fails") + assert.True(t, c.Route(net.ParseIP("2.2.2.2")), "route to 2.2.2.2 ok") + assert.True(t, c.Route(net.ParseIP("2.128.3.3")), "route to 2.128.3.3 ok") + assert.False(t, c.Route(net.ParseIP("3.0.0.0")), "route to 3.0.0.0 fails") + + assert.True(t, c.Match(net.ParseIP("1.1.1.1"), 1), "match 1.1.1.1:1 ok") + assert.True(t, c.Match(net.ParseIP("1.1.1.1"), 2), "match 1.1.1.1:2 ok") + assert.False(t, c.Match(net.ParseIP("1.1.1.1"), 3), "match 1.1.1.1:3 fails") + + assert.False(t, c.Match(net.ParseIP("1.1.1.2"), 1), "match 1.1.1.2 fails") + + assert.True(t, c.Match(net.ParseIP("2.2.2.2"), 3), "match 2.2.2.2:3 ok") + assert.True(t, c.Match(net.ParseIP("2.2.2.2"), 4), "match 2.2.2.2:4 ok") + assert.False(t, c.Match(net.ParseIP("2.2.2.2"), 5), "match 2.2.2.2:4 fails") + + assert.True(t, c.Match(net.ParseIP("2.128.3.3"), 3), "match 2.128.3.3:3 ok") + assert.True(t, c.Match(net.ParseIP("2.128.3.3"), 4), "match 2.128.3.3:4 ok") + assert.False(t, c.Match(net.ParseIP("2.128.3.3"), 5), "match 2.128.3.3:5 ok") + + assert.False(t, c.Match(net.ParseIP("3.0.0.0"), 1), "match 3.0.0.0 fails") + }, + }, { name: "reconcile-test: reconcile deleted cluster", config: stnrv1.StunnerConfig{ @@ -2460,6 +2544,76 @@ var testReconcileE2E = []StunnerTestReconcileE2EConfig{ allocateSuccess: true, echoResult: true, }, + { + testName: "adding port range to cluster ok", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Credentials: map[string]string{ + "username": "user", + "password": "pass", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp-ok", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Endpoints: []string{ + "1.2.3.5:<5678-5678>", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + restart: false, + bindSuccess: true, + allocateSuccess: true, + echoResult: true, + }, + { + testName: "extensing port range still ok", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Credentials: map[string]string{ + "username": "user", + "password": "pass", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "udp-ok", + Protocol: "turn-udp", + Addr: "1.2.3.4", + Port: 3478, + Routes: []string{ + "echo-server-cluster", + }, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "echo-server-cluster", + Endpoints: []string{ + "1.2.3.5:<1-10000>", + }, + }}, + }, + echoServerAddr: "1.2.3.5:5678", + restart: false, + bindSuccess: true, + allocateSuccess: true, + echoResult: true, + }, { testName: "converting cluster to strict dns", config: stnrv1.StunnerConfig{ diff --git a/relay.go b/relay.go index 037afc8f..14ac6b45 100644 --- a/relay.go +++ b/relay.go @@ -133,7 +133,7 @@ func (s *Stunner) GenPortRangeChecker(g *RelayGen) PortRangeChecker { } if cluster != nil { - return cluster, u.Port >= cluster.MinPort && u.Port <= cluster.MaxPort + return cluster, cluster.Match(u.IP, u.Port) } return nil, false diff --git a/stunner_test.go b/stunner_test.go index 2225bd5f..09d4077e 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -1189,12 +1189,10 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", - MinRelayPort: 5670, - MaxRelayPort: 5680, + Name: "echo-server-cluster", + Type: "STATIC", Endpoints: []string{ - "1.2.3.5", + "1.2.3.5:<5670-5680>", }, }}, }, @@ -1255,17 +1253,13 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }}, Clusters: []stnrv1.ClusterConfig{ { - Name: "dummy-cluster", - Type: "STATIC", - MinRelayPort: 5678, - MaxRelayPort: 5678, - Endpoints: []string{"1.2.3.6"}, + Name: "dummy-cluster", + Type: "STATIC", + Endpoints: []string{"1.2.3.6:<5678-5678>"}, }, { - Name: "echo-server-cluster", - Type: "STATIC", - MinRelayPort: 5678, - MaxRelayPort: 5678, - Endpoints: []string{"1.2.3.5"}, + Name: "echo-server-cluster", + Type: "STATIC", + Endpoints: []string{"1.2.3.5:<5678-5678>"}, }, }, }, @@ -1325,12 +1319,10 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", - MinRelayPort: 1, - MaxRelayPort: 5677, + Name: "echo-server-cluster", + Type: "STATIC", Endpoints: []string{ - "1.2.3.5", + "1.2.3.5:<1-5677>", }, }}, }, @@ -1390,12 +1382,10 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, }}, Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STATIC", - MinRelayPort: 5677, - MaxRelayPort: 5677, + Name: "echo-server-cluster", + Type: "STATIC", Endpoints: []string{ - "1.2.3.5", + "1.2.3.5:<5677-5677>", }, }}, }, @@ -1498,71 +1488,72 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) }, }, - { - testName: "strict_dns with prohibited port range fails", - config: stnrv1.StunnerConfig{ - ApiVersion: stnrv1.ApiVersion, - Admin: stnrv1.AdminConfig{ - LogLevel: stunnerTestLoglevel, - }, - Auth: stnrv1.AuthConfig{ - Type: "static", - Credentials: map[string]string{ - "username": "user1", - "password": "passwd1", - }, - }, - Listeners: []stnrv1.ListenerConfig{{ - Name: "udp", - Protocol: "turn-udp", - Addr: "1.2.3.4", - Port: 3478, - Routes: []string{ - "echo-server-cluster", - }, - }}, - Clusters: []stnrv1.ClusterConfig{{ - Name: "echo-server-cluster", - Type: "STRICT_DNS", - MinRelayPort: 1, - MaxRelayPort: 1, - Endpoints: []string{ - "echo-server.l7mp.io", - }, - }}, - }, - echoServerAddr: "1.2.3.5:5678", - result: false, - tester: func(t *testing.T) { - c := telemetry.ListenerConnsTotal - assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) - - g := telemetry.ListenerConnsActive - assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) - - c = telemetry.ListenerPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data - assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling - - c = telemetry.ListenerBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data - assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling - - c = telemetry.ClusterPacketsTotal - assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) - - c = telemetry.ClusterBytesTotal - assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) - }, - }, + // TODO: implement port-range filtering for DNS clusters + // { + // testName: "strict_dns with prohibited port range fails", + // config: stnrv1.StunnerConfig{ + // ApiVersion: stnrv1.ApiVersion, + // Admin: stnrv1.AdminConfig{ + // LogLevel: stunnerTestLoglevel, + // }, + // Auth: stnrv1.AuthConfig{ + // Type: "static", + // Credentials: map[string]string{ + // "username": "user1", + // "password": "passwd1", + // }, + // }, + // Listeners: []stnrv1.ListenerConfig{{ + // Name: "udp", + // Protocol: "turn-udp", + // Addr: "1.2.3.4", + // Port: 3478, + // Routes: []string{ + // "echo-server-cluster", + // }, + // }}, + // Clusters: []stnrv1.ClusterConfig{{ + // Name: "echo-server-cluster", + // Type: "STRICT_DNS", + // MinRelayPort: 1, + // MaxRelayPort: 1, + // Endpoints: []string{ + // "echo-server.l7mp.io", + // }, + // }}, + // }, + // echoServerAddr: "1.2.3.5:5678", + // result: false, + // tester: func(t *testing.T) { + // c := telemetry.ListenerConnsTotal + // assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") + // assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) + + // g := telemetry.ListenerConnsActive + // assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") + // assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) + + // c = telemetry.ListenerPacketsTotal + // assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + // assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data + // assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling + + // c = telemetry.ListenerBytesTotal + // assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") + // assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data + // assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling + + // c = telemetry.ClusterPacketsTotal + // assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + // assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + // assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + + // c = telemetry.ClusterBytesTotal + // assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") + // assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) + // assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + // }, + // }, } func TestStunnerPortRangeWithVNet(t *testing.T) { From 323aed228944712fdf35941e3408dc1528c7e8dc Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Dec 2023 18:28:20 +0100 Subject: [PATCH 072/243] refactor: Simplify the CDS server API --- handlers.go | 6 +-- internal/object/cluster.go | 3 +- pkg/config/cds_test.go | 90 +++++++++++++++++++++++++++++++++++++ pkg/config/server/config.go | 15 +++---- turncat.go | 2 +- 5 files changed, 101 insertions(+), 15 deletions(-) diff --git a/handlers.go b/handlers.go index 42d0a32f..6bcab5ca 100644 --- a/handlers.go +++ b/handlers.go @@ -71,10 +71,10 @@ func (s *Stunner) NewPermissionHandler(l *object.Listener) a12n.PermissionHandle auth := s.GetAuth() peerIP := peer.String() - auth.Log.Debugf("permission handler for listener %q: client %q, peer %q", - l.Name, src.String(), peerIP) - clusters := s.clusterManager.Keys() + auth.Log.Debugf("permission handler for listener %q: client %q, peer %q", l.Name, + src.String(), peerIP) + clusters := s.clusterManager.Keys() for _, r := range l.Routes { auth.Log.Tracef("considering route to cluster %q", r) if util.Member(clusters, r) { diff --git a/internal/object/cluster.go b/internal/object/cluster.go index 67d4daa9..cc4654fc 100644 --- a/internal/object/cluster.go +++ b/internal/object/cluster.go @@ -172,8 +172,7 @@ func (c *Cluster) Route(peer net.IP) bool { // Match decides whether a peer IP and port matches one of the permitted endpoints of a cluster. If // port is zero then port-matching is disabled. func (c *Cluster) Match(peer net.IP, port int) bool { - - c.log.Tracef("Route: cluster %q of type %s, peer IP: %s", c.Name, c.Type.String(), + c.log.Tracef("Match: cluster %q of type %s, peer IP: %s", c.Name, c.Type.String(), peer.String()) switch c.Type { diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index e576198a..9603317f 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -970,6 +970,85 @@ func TestClientReconnect(t *testing.T) { assert.True(t, s.DeepEqual(sc1), "deepeq 1") } +// test server config update mechanism +func TestServerUpdate(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + serverCtx, serverCancel := context.WithCancel(context.Background()) + defer serverCancel() + + testLog.Debug("create server") + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) + assert.NoError(t, err, "start") + + oldC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/udp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","username":"a","password":"b"},"listeners":[{"name": "stunner/udp-gateway/udp-listener", "protocol":"turn-udp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[]}`)) + assert.NoError(t, err, "parse 1") + + srv.UpsertConfig("stunner/udp-gateway", oldC) + + cs := srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 1, "snapshot len") + sc1 := srv.GetConfigStore().Get("stunner/udp-gateway") + assert.NotNil(t, sc1, "get") + assert.True(t, sc1.DeepEqual(oldC), "deepeq") + + // reapply - no change + srv.UpsertConfig("stunner/udp-gateway", oldC) + time.Sleep(20 * time.Millisecond) // let the server process + + cs = srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 1, "snapshot len") + sc1 = srv.GetConfigStore().Get("stunner/udp-gateway") + assert.NotNil(t, sc1, "get") + assert.True(t, sc1.DeepEqual(oldC), "deepeq") + + // add another config + tcpC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/tcp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","username":"a","password":"b"},"listeners":[{"name": "stunner/tcp-gateway/tcp-listener", "protocol":"turn-tcp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[{"name":"stunner/media-plane", "type":"STATIC","protocol":"UDP","endpoints":["0.0.0.0/0"]}]}`)) + assert.NoError(t, err, "parse") + + srv.UpsertConfig("stunner/tcp-gateway", tcpC) + time.Sleep(20 * time.Millisecond) // let the server process + + cs = srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 = srv.GetConfigStore().Get("stunner/udp-gateway") + assert.NotNil(t, sc1, "get") + assert.True(t, sc1.DeepEqual(oldC), "deepeq") + sc2 := srv.GetConfigStore().Get("stunner/tcp-gateway") + assert.NotNil(t, sc2, "get") + assert.True(t, sc2.DeepEqual(tcpC), "deepeq") + + // add a cluster + newC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/udp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","username":"a","password":"b"},"listeners":[{"name": "stunner/udp-gateway/udp-listener", "protocol":"turn-udp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[{"name": "stunner/media-plane", "type":"STATIC","protocol":"UDP","endpoints":["0.0.0.0/0"]}]}`)) + assert.NoError(t, err, "parse 1") + assert.False(t, oldC.DeepEqual(newC), "deepeq") + + // process in a single go + err = srv.UpdateConfig([]server.Config{{Id: "stunner/udp-gateway", Config: newC}, {Id: "stunner/tcp-gateway", Config: tcpC}}) + assert.NoError(t, err, "parse 1") + + time.Sleep(20 * time.Millisecond) // let the server process + + cs = srv.GetConfigStore().Snapshot() + assert.Len(t, cs, 2, "snapshot len") + sc1 = srv.GetConfigStore().Get("stunner/udp-gateway") + assert.NotNil(t, sc1, "get") + assert.True(t, sc1.DeepEqual(newC), "deepeq") + sc2 = srv.GetConfigStore().Get("stunner/tcp-gateway") + assert.NotNil(t, sc2, "get") + assert.True(t, sc2.DeepEqual(tcpC), "deepeq") +} + // only differ in id and realm func testConfig(id, realm string) server.Config { c := client.ZeroConfig(id) @@ -990,6 +1069,17 @@ func watchConfig(ch chan stnrv1.StunnerConfig, d time.Duration) *stnrv1.StunnerC } } +// func watchServerConfig(ch chan server.Config, d time.Duration) *stnrv1.StunnerConfig { +// select { +// case c := <-ch: +// // fmt.Println("++++++++++++ got config ++++++++++++: ", c.String()) +// return c.Config +// case <-time.After(d): +// // fmt.Println("++++++++++++ timeout ++++++++++++") +// return nil +// } +// } + func findConfById(cs []*stnrv1.StunnerConfig, id string) *stnrv1.StunnerConfig { for _, c := range cs { if c != nil && c.Admin.Name == id { diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go index da12134a..35e18d1d 100644 --- a/pkg/config/server/config.go +++ b/pkg/config/server/config.go @@ -21,7 +21,9 @@ func (e Config) String() string { return fmt.Sprintf("id=%s: %s", e.Id, e.Config.String()) } +// UpsertConfig upserts a single config in the server. func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { + s.configs.Upsert(id, c) s.configCh <- Config{Id: id, Config: c} } @@ -30,6 +32,7 @@ func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { // entering the graceful shutdown cycle receive a zeroconfig and abruprly kill all listeners with // all active connections allocated to it, currently we suppress the config update. func (s *Server) DeleteConfig(id string) { + s.configs.Delete(id) s.log.Info("suppressing config update for terminating client", "client", id) // s.configCh <- Config{Id: id, Config: client.ZeroConfig(id)} } @@ -37,17 +40,15 @@ func (s *Server) DeleteConfig(id string) { // UpdateConfig receives a set of ids and newConfigs that represent the state-of-the-world at a // particular instance of time and generates an update per each change. func (s *Server) UpdateConfig(newConfigs []Config) error { - oldNewConfigs := s.configs.Snapshot() + oldConfigs := s.configs.Snapshot() - for _, oldC := range oldNewConfigs { + for _, oldC := range oldConfigs { found := false for _, newC := range newConfigs { if oldC.Id == newC.Id { if !oldC.Config.DeepEqual(newC.Config) { s.log.V(2).Info("updating config", "client", newC.Id, "config", newC.Config.String()) - - s.configs.Upsert(newC.Id, newC.Config) s.UpsertConfig(newC.Id, newC.Config) } found = true @@ -57,15 +58,13 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { if !found { s.log.V(2).Info("removing config", "client", oldC.Id) - - s.configs.Delete(oldC.Id) s.DeleteConfig(oldC.Id) } } for _, newC := range newConfigs { found := false - for _, oldC := range oldNewConfigs { + for _, oldC := range oldConfigs { if oldC.Id == newC.Id { found = true break @@ -74,8 +73,6 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { if !found { s.log.V(2).Info("adding config", "client", newC.Id, "config", newC.Config) - - s.configs.Upsert(newC.Id, newC.Config) s.UpsertConfig(newC.Id, newC.Config) } } diff --git a/turncat.go b/turncat.go index f5223b71..0e835462 100644 --- a/turncat.go +++ b/turncat.go @@ -412,7 +412,7 @@ func (t *Turncat) runConnection(conn *connection) { t.peerAddr.Network(), t.peerAddr.String()) if _, writeErr := conn.serverConn.WriteTo(buffer[0:n], t.peerAddr); writeErr != nil { - t.log.Debugf("cannot write to TURN relay connection for client %s (likely hamrless): %s", + t.log.Debugf("cannot write to TURN relay connection for client %s (likely harmless): %s", conn.clientAddr.String(), writeErr.Error()) t.deleteConnection(conn) return From 19df3f1c0b569443b21ea0917e20297278438676 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Dec 2023 18:28:27 +0100 Subject: [PATCH 073/243] fix: Fix error ib API.V1 DeepEqual --- pkg/apis/v1/stunner.go | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go index 6b143e32..23e5054b 100644 --- a/pkg/apis/v1/stunner.go +++ b/pkg/apis/v1/stunner.go @@ -83,19 +83,20 @@ func (req *StunnerConfig) DeepEqual(conf Config) bool { return false } + if len(req.Listeners) != len(other.Listeners) { + return false + } for i := range req.Listeners { - if i >= len(other.Listeners) { - return false - } if !req.Listeners[i].DeepEqual(&other.Listeners[i]) { return false } } + if len(req.Clusters) != len(other.Clusters) { + return false + } + for i := range req.Clusters { - if i >= len(other.Clusters) { - return false - } if !req.Clusters[i].DeepEqual(&other.Clusters[i]) { return false } From 8ad707a3bec2fa14cb4fff8dd72eda1336874866 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Dec 2023 19:26:35 +0100 Subject: [PATCH 074/243] chore: Update Milestones --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 46e0b565..11ba2bf7 100644 --- a/README.md +++ b/README.md @@ -627,9 +627,9 @@ notable limitations at this point are as follows. * v0.11: Control plane: Kubernetes gateway operator and dataplane reconciliation. * v0.12: Security: Expose TLS/DTLS settings via the Gateway API. * v0.13: Observability: Prometheus + Grafana dashboard. -* v0.15: Performance: per-allocation CPU load-balancing for UDP -* v0.16: Management: managed STUNner dataplane. -* v0.17: Performance: eBPF TURN acceleration. +* v0.15: Performance: Per-allocation CPU load-balancing for UDP +* v0.16: Management: Managed STUNner dataplane. +* v0.17: Stabilization: All Gateway and STUNner APIs move to V1. * v1.0: GA * v2.0: Service mesh: adaptive scaling & resiliency From 089b1a9c14ef26e098c17bbb522c6397f4ff0549 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Mon, 18 Dec 2023 13:26:31 +0100 Subject: [PATCH 075/243] chore(CI/CD): Set legacy mode in e2e legacy test --- .github/workflows/e2e-test-dev-legacy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index 8883038e..fd81c34b 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -44,7 +44,7 @@ jobs: run: | helm repo add stunner https://l7mp.io/stunner helm repo update - helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m + helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=legacy --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner --set stunner.deployment.container.stunnerd.resources.requests.cpu=200m - name: Deploy iperf server From 93da4ecb275aff5168139f4adc495ae141b22258 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 20 Dec 2023 13:43:18 +0100 Subject: [PATCH 076/243] docs: Add ref to mediasoup demo Spotted by @rg0now in #108 --- docs/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/README.md b/docs/README.md index dd02c537..33448b4b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -39,6 +39,7 @@ * [One to one video call with Kurento](examples/kurento-one2one-call) * [Magic mirror with Kurento](examples/kurento-magic-mirror/README.md) * [Video-conferencing with LiveKit](examples/livekit/README.md) +* [Video-conferencing with mediasoup](examples/mediasoup/README.md) * [Video-conferencing with Jitsi](examples/jitsi/README.md) * [Cloud-gaming with Cloudretro](examples/cloudretro/README.md) * [Remote desktop access with Neko](examples/neko/README.md) From e5d772757f693046976446ec18e31155638ec0f0 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 21 Dec 2023 16:21:56 +0100 Subject: [PATCH 077/243] chore: Update static manifests --- deploy/manifests/static/dataplane.yaml | 1 - deploy/manifests/static/gateway-api-crd.yaml | 1301 ++++++---- deploy/manifests/static/stunner-crd.yaml | 2103 ++++++++++++++++- .../stunner-gateway-operator-manifests.yaml | 5 +- 4 files changed, 2978 insertions(+), 432 deletions(-) diff --git a/deploy/manifests/static/dataplane.yaml b/deploy/manifests/static/dataplane.yaml index 3c8319f7..9f420f76 100644 --- a/deploy/manifests/static/dataplane.yaml +++ b/deploy/manifests/static/dataplane.yaml @@ -21,4 +21,3 @@ spec: memory: 128Mi terminationGracePeriodSeconds: 3600 hostNetwork: false ---- \ No newline at end of file diff --git a/deploy/manifests/static/gateway-api-crd.yaml b/deploy/manifests/static/gateway-api-crd.yaml index 646684fd..9a8c9e04 100644 --- a/deploy/manifests/static/gateway-api-crd.yaml +++ b/deploy/manifests/static/gateway-api-crd.yaml @@ -3,8 +3,8 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: gatewayclasses.gateway.networking.k8s.io @@ -35,10 +35,7 @@ spec: name: Description priority: 1 type: string - # deprecated: true - # deprecationWarning: The v1alpha2 version of GatewayClass has been deprecated and - # will be removed in a future release of the API. Please upgrade to v1beta1. - name: v1alpha2 + name: v1 schema: openAPIV3Schema: description: "GatewayClass describes a class of Gateways available to the @@ -221,6 +218,35 @@ spec: x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map + supportedFeatures: + description: 'SupportedFeatures is the set of features the GatewayClass + support. It MUST be sorted in ascending alphabetical order. ' + items: + description: SupportedFeature is used to describe distinct features + that are covered by conformance tests. + enum: + - Gateway + - GatewayPort8080 + - GatewayStaticAddresses + - HTTPRoute + - HTTPRouteDestinationPortMatching + - HTTPRouteHostRewrite + - HTTPRouteMethodMatching + - HTTPRoutePathRedirect + - HTTPRoutePathRewrite + - HTTPRoutePortRedirect + - HTTPRouteQueryParamMatching + - HTTPRouteRequestMirror + - HTTPRouteRequestMultipleMirrors + - HTTPRouteResponseHeaderModification + - HTTPRouteSchemeRedirect + - Mesh + - ReferenceGrant + - TLSRoute + type: string + maxItems: 64 + type: array + x-kubernetes-list-type: set type: object required: - spec @@ -426,6 +452,35 @@ spec: x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map + supportedFeatures: + description: 'SupportedFeatures is the set of features the GatewayClass + support. It MUST be sorted in ascending alphabetical order. ' + items: + description: SupportedFeature is used to describe distinct features + that are covered by conformance tests. + enum: + - Gateway + - GatewayPort8080 + - GatewayStaticAddresses + - HTTPRoute + - HTTPRouteDestinationPortMatching + - HTTPRouteHostRewrite + - HTTPRouteMethodMatching + - HTTPRoutePathRedirect + - HTTPRoutePathRewrite + - HTTPRoutePortRedirect + - HTTPRouteQueryParamMatching + - HTTPRouteRequestMirror + - HTTPRouteRequestMultipleMirrors + - HTTPRouteResponseHeaderModification + - HTTPRouteSchemeRedirect + - Mesh + - ReferenceGrant + - TLSRoute + type: string + maxItems: 64 + type: array + x-kubernetes-list-type: set type: object required: - spec @@ -445,8 +500,8 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: gateways.gateway.networking.k8s.io @@ -476,10 +531,7 @@ spec: - jsonPath: .metadata.creationTimestamp name: Age type: date - # deprecated: true - # deprecationWarning: The v1alpha2 version of Gateway has been deprecated and will - # be removed in a future release of the API. Please upgrade to v1beta1. - name: v1alpha2 + name: v1 schema: openAPIV3Schema: description: Gateway represents an instance of a service-traffic handling @@ -509,14 +561,12 @@ spec: for the address(es) on the \"outside of the Gateway\", that traffic bound for this Gateway will use. This could be the IP address or hostname of an external load balancer or other networking infrastructure, - or some other address that traffic will be sent to. \n The .listener.hostname - field is used to route traffic that has already arrived at the Gateway - to the correct in-cluster destination. \n If no Addresses are specified, - the implementation MAY schedule the Gateway in an implementation-specific - manner, assigning an appropriate set of Addresses. \n The implementation - MUST bind all Listeners to every GatewayAddress that it assigns - to the Gateway and add a corresponding entry in GatewayStatus.Addresses. - \n Support: Extended \n " + or some other address that traffic will be sent to. \n If no Addresses + are specified, the implementation MAY schedule the Gateway in an + implementation-specific manner, assigning an appropriate set of + Addresses. \n The implementation MUST bind all Listeners to every + GatewayAddress that it assigns to the Gateway and add a corresponding + entry in GatewayStatus.Addresses. \n Support: Extended \n " items: description: GatewayAddress describes an address that can be bound to a Gateway. @@ -572,39 +622,134 @@ spec: maxLength: 253 minLength: 1 type: string + infrastructure: + description: "Infrastructure defines infrastructure level attributes + about this Gateway instance. \n Support: Core \n " + properties: + annotations: + additionalProperties: + description: AnnotationValue is the value of an annotation in + Gateway API. This is used for validation of maps such as TLS + options. This roughly matches Kubernetes annotation validation, + although the length validation in that case is based on the + entire size of the annotations struct. + maxLength: 4096 + minLength: 0 + type: string + description: "Annotations that SHOULD be applied to any resources + created in response to this Gateway. \n For implementations + creating other Kubernetes objects, this should be the `metadata.annotations` + field on resources. For other implementations, this refers to + any relevant (implementation specific) \"annotations\" concepts. + \n An implementation may chose to add additional implementation-specific + annotations as they see fit. \n Support: Extended" + maxProperties: 8 + type: object + labels: + additionalProperties: + description: AnnotationValue is the value of an annotation in + Gateway API. This is used for validation of maps such as TLS + options. This roughly matches Kubernetes annotation validation, + although the length validation in that case is based on the + entire size of the annotations struct. + maxLength: 4096 + minLength: 0 + type: string + description: "Labels that SHOULD be applied to any resources created + in response to this Gateway. \n For implementations creating + other Kubernetes objects, this should be the `metadata.labels` + field on resources. For other implementations, this refers to + any relevant (implementation specific) \"labels\" concepts. + \n An implementation may chose to add additional implementation-specific + labels as they see fit. \n Support: Extended" + maxProperties: 8 + type: object + type: object listeners: description: "Listeners associated with this Gateway. Listeners define logical endpoints that are bound on this Gateway's addresses. At - least one Listener MUST be specified. \n Each listener in a Gateway - must have a unique combination of Hostname, Port, and Protocol. - \n Within the HTTP Conformance Profile, the below combinations of - port and protocol are considered Core and MUST be supported: \n - 1. Port: 80, Protocol: HTTP 2. Port: 443, Protocol: HTTPS \n Within - the TLS Conformance Profile, the below combinations of port and - protocol are considered Core and MUST be supported: \n 1. Port: - 443, Protocol: TLS \n Port and protocol combinations not listed - above are considered Extended. \n An implementation MAY group Listeners - by Port and then collapse each group of Listeners into a single - Listener if the implementation determines that the Listeners in - the group are \"compatible\". An implementation MAY also group together - and collapse compatible Listeners belonging to different Gateways. - \n For example, an implementation might consider Listeners to be - compatible with each other if all of the following conditions are - met: \n 1. Either each Listener within the group specifies the \"HTTP\" - Protocol or each Listener within the group specifies either the - \"HTTPS\" or \"TLS\" Protocol. \n 2. Each Listener within the group - specifies a Hostname that is unique within the group. \n 3. As a - special case, one Listener within a group may omit Hostname, in - which case this Listener matches when no other Listener matches. - \n If the implementation does collapse compatible Listeners, the - hostname provided in the incoming client request MUST be matched - to a Listener to find the correct set of Routes. The incoming hostname - MUST be matched using the Hostname field for each Listener in order - of most to least specific. That is, exact matches must be processed - before wildcard matches. \n If this field specifies multiple Listeners - that have the same Port value but are not compatible, the implementation - must raise a \"Conflicted\" condition in the Listener status. \n - Support: Core" + least one Listener MUST be specified. \n Each Listener in a set + of Listeners (for example, in a single Gateway) MUST be _distinct_, + in that a traffic flow MUST be able to be assigned to exactly one + listener. (This section uses \"set of Listeners\" rather than \"Listeners + in a single Gateway\" because implementations MAY merge configuration + from multiple Gateways onto a single data plane, and these rules + _also_ apply in that case). \n Practically, this means that each + listener in a set MUST have a unique combination of Port, Protocol, + and, if supported by the protocol, Hostname. \n Some combinations + of port, protocol, and TLS settings are considered Core support + and MUST be supported by implementations based on their targeted + conformance profile: \n HTTP Profile \n 1. HTTPRoute, Port: 80, + Protocol: HTTP 2. HTTPRoute, Port: 443, Protocol: HTTPS, TLS Mode: + Terminate, TLS keypair provided \n TLS Profile \n 1. TLSRoute, Port: + 443, Protocol: TLS, TLS Mode: Passthrough \n \"Distinct\" Listeners + have the following property: \n The implementation can match inbound + requests to a single distinct Listener. When multiple Listeners + share values for fields (for example, two Listeners with the same + Port value), the implementation can match requests to only one of + the Listeners using other Listener fields. \n For example, the following + Listener scenarios are distinct: \n 1. Multiple Listeners with the + same Port that all use the \"HTTP\" Protocol that all have unique + Hostname values. 2. Multiple Listeners with the same Port that use + either the \"HTTPS\" or \"TLS\" Protocol that all have unique Hostname + values. 3. A mixture of \"TCP\" and \"UDP\" Protocol Listeners, + where no Listener with the same Protocol has the same Port value. + \n Some fields in the Listener struct have possible values that + affect whether the Listener is distinct. Hostname is particularly + relevant for HTTP or HTTPS protocols. \n When using the Hostname + value to select between same-Port, same-Protocol Listeners, the + Hostname value must be different on each Listener for the Listener + to be distinct. \n When the Listeners are distinct based on Hostname, + inbound request hostnames MUST match from the most specific to least + specific Hostname values to choose the correct Listener and its + associated set of Routes. \n Exact matches must be processed before + wildcard matches, and wildcard matches must be processed before + fallback (empty Hostname value) matches. For example, `\"foo.example.com\"` + takes precedence over `\"*.example.com\"`, and `\"*.example.com\"` + takes precedence over `\"\"`. \n Additionally, if there are multiple + wildcard entries, more specific wildcard entries must be processed + before less specific wildcard entries. For example, `\"*.foo.example.com\"` + takes precedence over `\"*.example.com\"`. The precise definition + here is that the higher the number of dots in the hostname to the + right of the wildcard character, the higher the precedence. \n The + wildcard character will match any number of characters _and dots_ + to the left, however, so `\"*.example.com\"` will match both `\"foo.bar.example.com\"` + _and_ `\"bar.example.com\"`. \n If a set of Listeners contains Listeners + that are not distinct, then those Listeners are Conflicted, and + the implementation MUST set the \"Conflicted\" condition in the + Listener Status to \"True\". \n Implementations MAY choose to accept + a Gateway with some Conflicted Listeners only if they only accept + the partial Listener set that contains no Conflicted Listeners. + To put this another way, implementations may accept a partial Listener + set only if they throw out *all* the conflicting Listeners. No picking + one of the conflicting listeners as the winner. This also means + that the Gateway must have at least one non-conflicting Listener + in this case, otherwise it violates the requirement that at least + one Listener must be present. \n The implementation MUST set a \"ListenersNotValid\" + condition on the Gateway Status when the Gateway contains Conflicted + Listeners whether or not they accept the Gateway. That Condition + SHOULD clearly indicate in the Message which Listeners are conflicted, + and which are Accepted. Additionally, the Listener status for those + listeners SHOULD indicate which Listeners are conflicted and not + Accepted. \n A Gateway's Listeners are considered \"compatible\" + if: \n 1. They are distinct. 2. The implementation can serve them + in compliance with the Addresses requirement that all Listeners + are available on all assigned addresses. \n Compatible combinations + in Extended support are expected to vary across implementations. + A combination that is compatible for one implementation may not + be compatible for another. \n For example, an implementation that + cannot serve both TCP and UDP listeners on the same address, or + cannot mix HTTPS and generic TLS listens on the same port would + not consider those cases compatible, even though they are distinct. + \n Note that requests SHOULD match at most one Listener. For example, + if Listeners are defined for \"foo.example.com\" and \"*.example.com\", + a request to \"foo.example.com\" SHOULD only be routed using routes + attached to the \"foo.example.com\" Listener (and not the \"*.example.com\" + Listener). This concept is known as \"Listener Isolation\". Implementations + that do not support Listener Isolation MUST clearly document this. + \n Implementations MAY merge separate Gateways onto a single set + of Addresses if all Listeners across all Gateways are compatible. + \n Support: Core" items: description: Listener embodies the concept of a logical endpoint where a Gateway accepts network connections. @@ -853,14 +998,14 @@ spec: minLength: 1 type: string namespace: - description: "Namespace is the namespace of the backend. - When unspecified, the local namespace is inferred. - \n Note that when a namespace different than the - local namespace is specified, a ReferenceGrant object - is required in the referent namespace to allow that - namespace's owner to accept the reference. See the - ReferenceGrant documentation for details. \n Support: - Core" + description: "Namespace is the namespace of the referenced + object. When unspecified, the local namespace is + inferred. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace to + allow that namespace's owner to accept the reference. + See the ReferenceGrant documentation for details. + \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -941,7 +1086,7 @@ spec: for each listener rule: 'self.all(l1, self.exists_one(l2, l1.port == l2.port && l1.protocol == l2.protocol && (has(l1.hostname) && has(l2.hostname) ? l1.hostname - == l2.hostname : true)))' + == l2.hostname : !has(l1.hostname) && !has(l2.hostname))))' required: - gatewayClassName - listeners @@ -962,13 +1107,15 @@ spec: description: Status defines the current state of Gateway. properties: addresses: - description: "Addresses lists the IP addresses that have actually - been bound to the Gateway. These addresses may differ from the addresses - in the Spec, e.g. if the Gateway automatically assigns an address - from a reserved pool. \n " + description: "Addresses lists the network addresses that have been + bound to the Gateway. \n This list may differ from the addresses + provided in the spec under some conditions: \n * no addresses are + specified, all addresses are dynamically assigned * a combination + of specified and dynamic addresses are assigned * a specified address + was unusable (e.g. already in use) \n " items: - description: GatewayStatusAddress describes an address that is bound - to a Gateway. + description: GatewayStatusAddress describes a network address that + is bound to a Gateway. oneOf: - properties: type: @@ -1104,8 +1251,23 @@ spec: description: ListenerStatus is the status associated with a Listener. properties: attachedRoutes: - description: AttachedRoutes represents the total number of accepted + description: "AttachedRoutes represents the total number of Routes that have been successfully attached to this Listener. + \n Successful attachment of a Route to a Listener is based + solely on the combination of the AllowedRoutes field on the + corresponding Listener and the Route's ParentRefs field. A + Route is successfully attached to a Listener when it is selected + by the Listener's AllowedRoutes field AND the Route has a + valid ParentRef selecting the whole Gateway resource or a + specific Listener as a parent resource (more detail on attachment + semantics can be found in the documentation on the various + Route kinds ParentRefs fields). Listener or Route status does + not impact successful attachment, i.e. the AttachedRoutes + field count MUST be set for Listeners with condition Accepted: + false and MUST count successfully attached Routes that may + themselves have Accepted: false conditions. \n Uses for this + field include troubleshooting Route attachment and measuring + blast radius/impact of changes to a Listener." format: int32 type: integer conditions: @@ -1287,14 +1449,12 @@ spec: for the address(es) on the \"outside of the Gateway\", that traffic bound for this Gateway will use. This could be the IP address or hostname of an external load balancer or other networking infrastructure, - or some other address that traffic will be sent to. \n The .listener.hostname - field is used to route traffic that has already arrived at the Gateway - to the correct in-cluster destination. \n If no Addresses are specified, - the implementation MAY schedule the Gateway in an implementation-specific - manner, assigning an appropriate set of Addresses. \n The implementation - MUST bind all Listeners to every GatewayAddress that it assigns - to the Gateway and add a corresponding entry in GatewayStatus.Addresses. - \n Support: Extended \n " + or some other address that traffic will be sent to. \n If no Addresses + are specified, the implementation MAY schedule the Gateway in an + implementation-specific manner, assigning an appropriate set of + Addresses. \n The implementation MUST bind all Listeners to every + GatewayAddress that it assigns to the Gateway and add a corresponding + entry in GatewayStatus.Addresses. \n Support: Extended \n " items: description: GatewayAddress describes an address that can be bound to a Gateway. @@ -1350,39 +1510,134 @@ spec: maxLength: 253 minLength: 1 type: string + infrastructure: + description: "Infrastructure defines infrastructure level attributes + about this Gateway instance. \n Support: Core \n " + properties: + annotations: + additionalProperties: + description: AnnotationValue is the value of an annotation in + Gateway API. This is used for validation of maps such as TLS + options. This roughly matches Kubernetes annotation validation, + although the length validation in that case is based on the + entire size of the annotations struct. + maxLength: 4096 + minLength: 0 + type: string + description: "Annotations that SHOULD be applied to any resources + created in response to this Gateway. \n For implementations + creating other Kubernetes objects, this should be the `metadata.annotations` + field on resources. For other implementations, this refers to + any relevant (implementation specific) \"annotations\" concepts. + \n An implementation may chose to add additional implementation-specific + annotations as they see fit. \n Support: Extended" + maxProperties: 8 + type: object + labels: + additionalProperties: + description: AnnotationValue is the value of an annotation in + Gateway API. This is used for validation of maps such as TLS + options. This roughly matches Kubernetes annotation validation, + although the length validation in that case is based on the + entire size of the annotations struct. + maxLength: 4096 + minLength: 0 + type: string + description: "Labels that SHOULD be applied to any resources created + in response to this Gateway. \n For implementations creating + other Kubernetes objects, this should be the `metadata.labels` + field on resources. For other implementations, this refers to + any relevant (implementation specific) \"labels\" concepts. + \n An implementation may chose to add additional implementation-specific + labels as they see fit. \n Support: Extended" + maxProperties: 8 + type: object + type: object listeners: description: "Listeners associated with this Gateway. Listeners define logical endpoints that are bound on this Gateway's addresses. At - least one Listener MUST be specified. \n Each listener in a Gateway - must have a unique combination of Hostname, Port, and Protocol. - \n Within the HTTP Conformance Profile, the below combinations of - port and protocol are considered Core and MUST be supported: \n - 1. Port: 80, Protocol: HTTP 2. Port: 443, Protocol: HTTPS \n Within - the TLS Conformance Profile, the below combinations of port and - protocol are considered Core and MUST be supported: \n 1. Port: - 443, Protocol: TLS \n Port and protocol combinations not listed - above are considered Extended. \n An implementation MAY group Listeners - by Port and then collapse each group of Listeners into a single - Listener if the implementation determines that the Listeners in - the group are \"compatible\". An implementation MAY also group together - and collapse compatible Listeners belonging to different Gateways. - \n For example, an implementation might consider Listeners to be - compatible with each other if all of the following conditions are - met: \n 1. Either each Listener within the group specifies the \"HTTP\" - Protocol or each Listener within the group specifies either the - \"HTTPS\" or \"TLS\" Protocol. \n 2. Each Listener within the group - specifies a Hostname that is unique within the group. \n 3. As a - special case, one Listener within a group may omit Hostname, in - which case this Listener matches when no other Listener matches. - \n If the implementation does collapse compatible Listeners, the - hostname provided in the incoming client request MUST be matched - to a Listener to find the correct set of Routes. The incoming hostname - MUST be matched using the Hostname field for each Listener in order - of most to least specific. That is, exact matches must be processed - before wildcard matches. \n If this field specifies multiple Listeners - that have the same Port value but are not compatible, the implementation - must raise a \"Conflicted\" condition in the Listener status. \n - Support: Core" + least one Listener MUST be specified. \n Each Listener in a set + of Listeners (for example, in a single Gateway) MUST be _distinct_, + in that a traffic flow MUST be able to be assigned to exactly one + listener. (This section uses \"set of Listeners\" rather than \"Listeners + in a single Gateway\" because implementations MAY merge configuration + from multiple Gateways onto a single data plane, and these rules + _also_ apply in that case). \n Practically, this means that each + listener in a set MUST have a unique combination of Port, Protocol, + and, if supported by the protocol, Hostname. \n Some combinations + of port, protocol, and TLS settings are considered Core support + and MUST be supported by implementations based on their targeted + conformance profile: \n HTTP Profile \n 1. HTTPRoute, Port: 80, + Protocol: HTTP 2. HTTPRoute, Port: 443, Protocol: HTTPS, TLS Mode: + Terminate, TLS keypair provided \n TLS Profile \n 1. TLSRoute, Port: + 443, Protocol: TLS, TLS Mode: Passthrough \n \"Distinct\" Listeners + have the following property: \n The implementation can match inbound + requests to a single distinct Listener. When multiple Listeners + share values for fields (for example, two Listeners with the same + Port value), the implementation can match requests to only one of + the Listeners using other Listener fields. \n For example, the following + Listener scenarios are distinct: \n 1. Multiple Listeners with the + same Port that all use the \"HTTP\" Protocol that all have unique + Hostname values. 2. Multiple Listeners with the same Port that use + either the \"HTTPS\" or \"TLS\" Protocol that all have unique Hostname + values. 3. A mixture of \"TCP\" and \"UDP\" Protocol Listeners, + where no Listener with the same Protocol has the same Port value. + \n Some fields in the Listener struct have possible values that + affect whether the Listener is distinct. Hostname is particularly + relevant for HTTP or HTTPS protocols. \n When using the Hostname + value to select between same-Port, same-Protocol Listeners, the + Hostname value must be different on each Listener for the Listener + to be distinct. \n When the Listeners are distinct based on Hostname, + inbound request hostnames MUST match from the most specific to least + specific Hostname values to choose the correct Listener and its + associated set of Routes. \n Exact matches must be processed before + wildcard matches, and wildcard matches must be processed before + fallback (empty Hostname value) matches. For example, `\"foo.example.com\"` + takes precedence over `\"*.example.com\"`, and `\"*.example.com\"` + takes precedence over `\"\"`. \n Additionally, if there are multiple + wildcard entries, more specific wildcard entries must be processed + before less specific wildcard entries. For example, `\"*.foo.example.com\"` + takes precedence over `\"*.example.com\"`. The precise definition + here is that the higher the number of dots in the hostname to the + right of the wildcard character, the higher the precedence. \n The + wildcard character will match any number of characters _and dots_ + to the left, however, so `\"*.example.com\"` will match both `\"foo.bar.example.com\"` + _and_ `\"bar.example.com\"`. \n If a set of Listeners contains Listeners + that are not distinct, then those Listeners are Conflicted, and + the implementation MUST set the \"Conflicted\" condition in the + Listener Status to \"True\". \n Implementations MAY choose to accept + a Gateway with some Conflicted Listeners only if they only accept + the partial Listener set that contains no Conflicted Listeners. + To put this another way, implementations may accept a partial Listener + set only if they throw out *all* the conflicting Listeners. No picking + one of the conflicting listeners as the winner. This also means + that the Gateway must have at least one non-conflicting Listener + in this case, otherwise it violates the requirement that at least + one Listener must be present. \n The implementation MUST set a \"ListenersNotValid\" + condition on the Gateway Status when the Gateway contains Conflicted + Listeners whether or not they accept the Gateway. That Condition + SHOULD clearly indicate in the Message which Listeners are conflicted, + and which are Accepted. Additionally, the Listener status for those + listeners SHOULD indicate which Listeners are conflicted and not + Accepted. \n A Gateway's Listeners are considered \"compatible\" + if: \n 1. They are distinct. 2. The implementation can serve them + in compliance with the Addresses requirement that all Listeners + are available on all assigned addresses. \n Compatible combinations + in Extended support are expected to vary across implementations. + A combination that is compatible for one implementation may not + be compatible for another. \n For example, an implementation that + cannot serve both TCP and UDP listeners on the same address, or + cannot mix HTTPS and generic TLS listens on the same port would + not consider those cases compatible, even though they are distinct. + \n Note that requests SHOULD match at most one Listener. For example, + if Listeners are defined for \"foo.example.com\" and \"*.example.com\", + a request to \"foo.example.com\" SHOULD only be routed using routes + attached to the \"foo.example.com\" Listener (and not the \"*.example.com\" + Listener). This concept is known as \"Listener Isolation\". Implementations + that do not support Listener Isolation MUST clearly document this. + \n Implementations MAY merge separate Gateways onto a single set + of Addresses if all Listeners across all Gateways are compatible. + \n Support: Core" items: description: Listener embodies the concept of a logical endpoint where a Gateway accepts network connections. @@ -1631,14 +1886,14 @@ spec: minLength: 1 type: string namespace: - description: "Namespace is the namespace of the backend. - When unspecified, the local namespace is inferred. - \n Note that when a namespace different than the - local namespace is specified, a ReferenceGrant object - is required in the referent namespace to allow that - namespace's owner to accept the reference. See the - ReferenceGrant documentation for details. \n Support: - Core" + description: "Namespace is the namespace of the referenced + object. When unspecified, the local namespace is + inferred. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace to + allow that namespace's owner to accept the reference. + See the ReferenceGrant documentation for details. + \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -1719,7 +1974,7 @@ spec: for each listener rule: 'self.all(l1, self.exists_one(l2, l1.port == l2.port && l1.protocol == l2.protocol && (has(l1.hostname) && has(l2.hostname) ? l1.hostname - == l2.hostname : true)))' + == l2.hostname : !has(l1.hostname) && !has(l2.hostname))))' required: - gatewayClassName - listeners @@ -1740,13 +1995,15 @@ spec: description: Status defines the current state of Gateway. properties: addresses: - description: "Addresses lists the IP addresses that have actually - been bound to the Gateway. These addresses may differ from the addresses - in the Spec, e.g. if the Gateway automatically assigns an address - from a reserved pool. \n " + description: "Addresses lists the network addresses that have been + bound to the Gateway. \n This list may differ from the addresses + provided in the spec under some conditions: \n * no addresses are + specified, all addresses are dynamically assigned * a combination + of specified and dynamic addresses are assigned * a specified address + was unusable (e.g. already in use) \n " items: - description: GatewayStatusAddress describes an address that is bound - to a Gateway. + description: GatewayStatusAddress describes a network address that + is bound to a Gateway. oneOf: - properties: type: @@ -1882,8 +2139,23 @@ spec: description: ListenerStatus is the status associated with a Listener. properties: attachedRoutes: - description: AttachedRoutes represents the total number of accepted + description: "AttachedRoutes represents the total number of Routes that have been successfully attached to this Listener. + \n Successful attachment of a Route to a Listener is based + solely on the combination of the AllowedRoutes field on the + corresponding Listener and the Route's ParentRefs field. A + Route is successfully attached to a Listener when it is selected + by the Listener's AllowedRoutes field AND the Route has a + valid ParentRef selecting the whole Gateway resource or a + specific Listener as a parent resource (more detail on attachment + semantics can be found in the documentation on the various + Route kinds ParentRefs fields). Listener or Route status does + not impact successful attachment, i.e. the AttachedRoutes + field count MUST be set for Listeners with condition Accepted: + false and MUST count successfully attached Routes that may + themselves have Accepted: false conditions. \n Uses for this + field include troubleshooting Route attachment and measuring + blast radius/impact of changes to a Listener." format: int32 type: integer conditions: @@ -2029,15 +2301,12 @@ status: conditions: null storedVersions: null --- -# -# config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml -# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: grpcroutes.gateway.networking.k8s.io @@ -2168,30 +2437,41 @@ spec: for governing ParentRefs to Services - it is not possible to create a \"producer\" route for a Service in a different namespace from the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services only) \n This + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) This API may be extended in the future to support additional kinds of - parent resources. \n It is invalid to reference an identical parent - more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as two separate Listeners - on the same Gateway or two separate ports on the same Service. \n - It is possible to separately reference multiple distinct objects - that may be collapsed by an implementation. For example, some implementations - may choose to merge compatible Gateway Listeners together. If that - is the case, the list of routes attached to those resources should - also be merged. \n Note that for ParentRefs that cross namespace - boundaries, there are specific rules. Cross-namespace references - are only valid if they are explicitly allowed by something in the - namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable other - kinds of cross-namespace reference. \n ParentRefs from a Route to - a Service in the same namespace are \"producer\" routes, which apply - default routing rules to inbound connections from any namespace - to the Service. \n ParentRefs from a Route to a Service in a different - namespace are \"consumer\" routes, and these routing rules are only - applied to outbound connections originating from the same namespace - as the Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. \n " + parent resources. \n ParentRefs must be _distinct_. This means either + that: \n * They select different objects. If this is the case, + then parentRef entries are distinct. In terms of fields, this means + that the multi-part key defined by `group`, `kind`, `namespace`, + and `name` must be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field + used, each ParentRef that selects the same object must set the same + set of optional fields to different values. If one ParentRef sets + a combination of optional fields, all must set the same combination. + \n Some examples: \n * If one ParentRef sets `sectionName`, all + ParentRefs referencing the same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. * If one ParentRef sets `sectionName` + and `port`, all ParentRefs referencing the same object must also + set `sectionName` and `port`. \n It is possible to separately reference + multiple distinct objects that may be collapsed by an implementation. + For example, some implementations may choose to merge compatible + Gateway Listeners together. If that is the case, the list of routes + attached to those resources should also be merged. \n Note that + for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For + example, Gateway has the AllowedRoutes field, and ReferenceGrant + provides a generic way to enable other kinds of cross-namespace + reference. \n ParentRefs from a Route to a Service in the same + namespace are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. \n ParentRefs + from a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which the + intended destination of the connections are a Service targeted as + a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually @@ -2239,7 +2519,7 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route to a Service in the same namespace + \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to a Service in a different namespace @@ -2247,7 +2527,7 @@ spec: applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are a Service targeted as a ParentRef of - the Route. \n Support: Core" + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -2262,11 +2542,11 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n When the parent resource + must match both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must match both specified - values. \n Implementations MAY choose to support other parent + values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered @@ -2318,16 +2598,13 @@ spec: - message: sectionName or port must be specified when parentRefs includes 2 or more references to the same parent rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind - == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ - == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) - && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) - || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName - == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) - || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName - != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) - && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) - ) ) ): true ))' + == '''')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) + || p1.sectionName == '''') == (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) + || p2.port == 0)): true))' - message: sectionName or port must be unique when parentRefs includes 2 or more references to the same parent rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind @@ -2368,8 +2645,25 @@ spec: Service \n Support: Implementation-specific for any other resource \n Support for weight: Core" items: - description: GRPCBackendRef defines how a GRPCRoute forwards - a gRPC request. + description: "GRPCBackendRef defines how a GRPCRoute forwards + a gRPC request. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace to allow that + namespace's owner to accept the reference. See the ReferenceGrant + documentation for details. \n + \n When the BackendRef points to a Kubernetes Service, implementations + SHOULD honor the appProtocol field if it is set for the + target Service Port. \n Implementations supporting appProtocol + SHOULD recognize the Kubernetes Standard Application Protocols + defined in KEP-3726. \n If a Service appProtocol isn't specified, + an implementation MAY infer the backend protocol through + its own means. Implementations MAY infer the protocol from + the Route type referring to the backend Service. \n If a + Route is not able to send traffic to the backend using the + specified protocol then the backend is considered invalid. + Implementations MUST set the \"ResolvedRefs\" condition + to \"False\" with the \"UnsupportedProtocol\" reason. \n + " properties: filters: description: "Filters defined at this level MUST be executed @@ -3635,7 +3929,7 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs from a Route to a Service in + reference. \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to @@ -3643,7 +3937,7 @@ spec: and these routing rules are only applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are - a Service targeted as a ParentRef of the Route. \n Support: + a Service targeted as a ParentRef of the Route. \n Support: Core" maxLength: 63 minLength: 1 @@ -3660,11 +3954,11 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n When the parent resource is + both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must - match both specified values. \n Implementations MAY choose + match both specified values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, @@ -3734,15 +4028,12 @@ status: conditions: null storedVersions: null --- -# -# config/crd/experimental/gateway.networking.k8s.io_httproutes.yaml -# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: httproutes.gateway.networking.k8s.io @@ -3764,10 +4055,7 @@ spec: - jsonPath: .metadata.creationTimestamp name: Age type: date - deprecated: true - deprecationWarning: The v1alpha2 version of HTTPRoute has been deprecated and - will be removed in a future release of the API. Please upgrade to v1beta1. - name: v1alpha2 + name: v1 schema: openAPIV3Schema: description: HTTPRoute provides a way to route HTTP requests. This includes @@ -3860,30 +4148,41 @@ spec: for governing ParentRefs to Services - it is not possible to create a \"producer\" route for a Service in a different namespace from the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services only) \n This + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) This API may be extended in the future to support additional kinds of - parent resources. \n It is invalid to reference an identical parent - more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as two separate Listeners - on the same Gateway or two separate ports on the same Service. \n - It is possible to separately reference multiple distinct objects - that may be collapsed by an implementation. For example, some implementations - may choose to merge compatible Gateway Listeners together. If that - is the case, the list of routes attached to those resources should - also be merged. \n Note that for ParentRefs that cross namespace - boundaries, there are specific rules. Cross-namespace references - are only valid if they are explicitly allowed by something in the - namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable other - kinds of cross-namespace reference. \n ParentRefs from a Route to - a Service in the same namespace are \"producer\" routes, which apply - default routing rules to inbound connections from any namespace - to the Service. \n ParentRefs from a Route to a Service in a different - namespace are \"consumer\" routes, and these routing rules are only - applied to outbound connections originating from the same namespace - as the Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. \n " + parent resources. \n ParentRefs must be _distinct_. This means either + that: \n * They select different objects. If this is the case, + then parentRef entries are distinct. In terms of fields, this means + that the multi-part key defined by `group`, `kind`, `namespace`, + and `name` must be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field + used, each ParentRef that selects the same object must set the same + set of optional fields to different values. If one ParentRef sets + a combination of optional fields, all must set the same combination. + \n Some examples: \n * If one ParentRef sets `sectionName`, all + ParentRefs referencing the same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. * If one ParentRef sets `sectionName` + and `port`, all ParentRefs referencing the same object must also + set `sectionName` and `port`. \n It is possible to separately reference + multiple distinct objects that may be collapsed by an implementation. + For example, some implementations may choose to merge compatible + Gateway Listeners together. If that is the case, the list of routes + attached to those resources should also be merged. \n Note that + for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For + example, Gateway has the AllowedRoutes field, and ReferenceGrant + provides a generic way to enable other kinds of cross-namespace + reference. \n ParentRefs from a Route to a Service in the same + namespace are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. \n ParentRefs + from a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which the + intended destination of the connections are a Service targeted as + a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually @@ -3931,7 +4230,7 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route to a Service in the same namespace + \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to a Service in a different namespace @@ -3939,7 +4238,7 @@ spec: applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are a Service targeted as a ParentRef of - the Route. \n Support: Core" + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -3954,11 +4253,11 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n When the parent resource + must match both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must match both specified - values. \n Implementations MAY choose to support other parent + values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered @@ -4010,16 +4309,13 @@ spec: - message: sectionName or port must be specified when parentRefs includes 2 or more references to the same parent rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind - == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ - == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) - && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) - || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName - == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) - || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName - != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) - && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) - ) ) ): true ))' + == '''')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) + || p1.sectionName == '''') == (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) + || p2.port == 0)): true))' - message: sectionName or port must be unique when parentRefs includes 2 or more references to the same parent rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind @@ -4065,8 +4361,25 @@ spec: for Kubernetes ServiceImport \n Support: Implementation-specific for any other resource \n Support for weight: Core" items: - description: HTTPBackendRef defines how a HTTPRoute should - forward an HTTP request. + description: "HTTPBackendRef defines how a HTTPRoute forwards + a HTTP request. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace to allow that + namespace's owner to accept the reference. See the ReferenceGrant + documentation for details. \n + \n When the BackendRef points to a Kubernetes Service, implementations + SHOULD honor the appProtocol field if it is set for the + target Service Port. \n Implementations supporting appProtocol + SHOULD recognize the Kubernetes Standard Application Protocols + defined in KEP-3726. \n If a Service appProtocol isn't specified, + an implementation MAY infer the backend protocol through + its own means. Implementations MAY infer the protocol from + the Route type referring to the backend Service. \n If a + Route is not able to send traffic to the backend using the + specified protocol then the backend is considered invalid. + Implementations MUST set the \"ResolvedRefs\" condition + to \"False\" with the \"UnsupportedProtocol\" reason. \n + " properties: filters: description: "Filters defined at this level should be @@ -5852,6 +6165,48 @@ spec: type: object maxItems: 8 type: array + timeouts: + description: "Timeouts defines the timeouts that can be configured + for an HTTP request. \n Support: Extended \n " + properties: + backendRequest: + description: "BackendRequest specifies a timeout for an + individual request from the gateway to a backend. This + covers the time from when the request first starts being + sent from the gateway to when the full response has been + received from the backend. \n An entire client HTTP transaction + with a gateway, covered by the Request timeout, may result + in more than one call from the gateway to the destination + backend, for example, if automatic retries are supported. + \n Because the Request timeout encompasses the BackendRequest + timeout, the value of BackendRequest must be <= the value + of Request timeout. \n Support: Extended" + pattern: ^([0-9]{1,5}(h|m|s|ms)){1,4}$ + type: string + request: + description: "Request specifies the maximum duration for + a gateway to respond to an HTTP request. If the gateway + has not been able to respond before this deadline is met, + the gateway MUST return a timeout error. \n For example, + setting the `rules.timeouts.request` field to the value + `10s` in an `HTTPRoute` will cause a timeout if a client + request is taking longer than 10 seconds to complete. + \n This timeout is intended to cover as close to the whole + request-response transaction as possible although an implementation + MAY choose to start the timeout after the entire request + stream has been received instead of immediately after + the transaction is initiated by the client. \n When this + field is unspecified, request timeout behavior is implementation-specific. + \n Support: Extended" + pattern: ^([0-9]{1,5}(h|m|s|ms)){1,4}$ + type: string + type: object + x-kubernetes-validations: + - message: backendRequest timeout cannot be longer than request + timeout + rule: '!(has(self.request) && has(self.backendRequest) && + duration(self.request) != duration(''0s'') && duration(self.backendRequest) + > duration(self.request))' type: object x-kubernetes-validations: - message: RequestRedirect filter must not be used together with @@ -6065,7 +6420,7 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs from a Route to a Service in + reference. \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to @@ -6073,7 +6428,7 @@ spec: and these routing rules are only applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are - a Service targeted as a ParentRef of the Route. \n Support: + a Service targeted as a ParentRef of the Route. \n Support: Core" maxLength: 63 minLength: 1 @@ -6090,11 +6445,11 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n When the parent resource is + both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must - match both specified values. \n Implementations MAY choose + match both specified values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, @@ -6155,7 +6510,7 @@ spec: required: - spec type: object - served: false + served: true storage: false subresources: status: {} @@ -6259,30 +6614,41 @@ spec: for governing ParentRefs to Services - it is not possible to create a \"producer\" route for a Service in a different namespace from the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services only) \n This + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) This API may be extended in the future to support additional kinds of - parent resources. \n It is invalid to reference an identical parent - more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as two separate Listeners - on the same Gateway or two separate ports on the same Service. \n - It is possible to separately reference multiple distinct objects - that may be collapsed by an implementation. For example, some implementations - may choose to merge compatible Gateway Listeners together. If that - is the case, the list of routes attached to those resources should - also be merged. \n Note that for ParentRefs that cross namespace - boundaries, there are specific rules. Cross-namespace references - are only valid if they are explicitly allowed by something in the - namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable other - kinds of cross-namespace reference. \n ParentRefs from a Route to - a Service in the same namespace are \"producer\" routes, which apply - default routing rules to inbound connections from any namespace - to the Service. \n ParentRefs from a Route to a Service in a different - namespace are \"consumer\" routes, and these routing rules are only - applied to outbound connections originating from the same namespace - as the Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. \n " + parent resources. \n ParentRefs must be _distinct_. This means either + that: \n * They select different objects. If this is the case, + then parentRef entries are distinct. In terms of fields, this means + that the multi-part key defined by `group`, `kind`, `namespace`, + and `name` must be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field + used, each ParentRef that selects the same object must set the same + set of optional fields to different values. If one ParentRef sets + a combination of optional fields, all must set the same combination. + \n Some examples: \n * If one ParentRef sets `sectionName`, all + ParentRefs referencing the same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. * If one ParentRef sets `sectionName` + and `port`, all ParentRefs referencing the same object must also + set `sectionName` and `port`. \n It is possible to separately reference + multiple distinct objects that may be collapsed by an implementation. + For example, some implementations may choose to merge compatible + Gateway Listeners together. If that is the case, the list of routes + attached to those resources should also be merged. \n Note that + for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For + example, Gateway has the AllowedRoutes field, and ReferenceGrant + provides a generic way to enable other kinds of cross-namespace + reference. \n ParentRefs from a Route to a Service in the same + namespace are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. \n ParentRefs + from a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which the + intended destination of the connections are a Service targeted as + a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually @@ -6330,7 +6696,7 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route to a Service in the same namespace + \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to a Service in a different namespace @@ -6338,7 +6704,7 @@ spec: applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are a Service targeted as a ParentRef of - the Route. \n Support: Core" + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -6353,11 +6719,11 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n When the parent resource + must match both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must match both specified - values. \n Implementations MAY choose to support other parent + values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered @@ -6409,16 +6775,13 @@ spec: - message: sectionName or port must be specified when parentRefs includes 2 or more references to the same parent rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind - == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ - == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) - && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) - || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName - == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) - || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName - != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) - && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) - ) ) ): true ))' + == '''')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) + || p1.sectionName == '''') == (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) + || p2.port == 0)): true))' - message: sectionName or port must be unique when parentRefs includes 2 or more references to the same parent rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind @@ -6464,8 +6827,25 @@ spec: for Kubernetes ServiceImport \n Support: Implementation-specific for any other resource \n Support for weight: Core" items: - description: HTTPBackendRef defines how a HTTPRoute should - forward an HTTP request. + description: "HTTPBackendRef defines how a HTTPRoute forwards + a HTTP request. \n Note that when a namespace different + than the local namespace is specified, a ReferenceGrant + object is required in the referent namespace to allow that + namespace's owner to accept the reference. See the ReferenceGrant + documentation for details. \n + \n When the BackendRef points to a Kubernetes Service, implementations + SHOULD honor the appProtocol field if it is set for the + target Service Port. \n Implementations supporting appProtocol + SHOULD recognize the Kubernetes Standard Application Protocols + defined in KEP-3726. \n If a Service appProtocol isn't specified, + an implementation MAY infer the backend protocol through + its own means. Implementations MAY infer the protocol from + the Route type referring to the backend Service. \n If a + Route is not able to send traffic to the backend using the + specified protocol then the backend is considered invalid. + Implementations MUST set the \"ResolvedRefs\" condition + to \"False\" with the \"UnsupportedProtocol\" reason. \n + " properties: filters: description: "Filters defined at this level should be @@ -8251,6 +8631,48 @@ spec: type: object maxItems: 8 type: array + timeouts: + description: "Timeouts defines the timeouts that can be configured + for an HTTP request. \n Support: Extended \n " + properties: + backendRequest: + description: "BackendRequest specifies a timeout for an + individual request from the gateway to a backend. This + covers the time from when the request first starts being + sent from the gateway to when the full response has been + received from the backend. \n An entire client HTTP transaction + with a gateway, covered by the Request timeout, may result + in more than one call from the gateway to the destination + backend, for example, if automatic retries are supported. + \n Because the Request timeout encompasses the BackendRequest + timeout, the value of BackendRequest must be <= the value + of Request timeout. \n Support: Extended" + pattern: ^([0-9]{1,5}(h|m|s|ms)){1,4}$ + type: string + request: + description: "Request specifies the maximum duration for + a gateway to respond to an HTTP request. If the gateway + has not been able to respond before this deadline is met, + the gateway MUST return a timeout error. \n For example, + setting the `rules.timeouts.request` field to the value + `10s` in an `HTTPRoute` will cause a timeout if a client + request is taking longer than 10 seconds to complete. + \n This timeout is intended to cover as close to the whole + request-response transaction as possible although an implementation + MAY choose to start the timeout after the entire request + stream has been received instead of immediately after + the transaction is initiated by the client. \n When this + field is unspecified, request timeout behavior is implementation-specific. + \n Support: Extended" + pattern: ^([0-9]{1,5}(h|m|s|ms)){1,4}$ + type: string + type: object + x-kubernetes-validations: + - message: backendRequest timeout cannot be longer than request + timeout + rule: '!(has(self.request) && has(self.backendRequest) && + duration(self.request) != duration(''0s'') && duration(self.backendRequest) + > duration(self.request))' type: object x-kubernetes-validations: - message: RequestRedirect filter must not be used together with @@ -8464,7 +8886,7 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs from a Route to a Service in + reference. \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to @@ -8472,7 +8894,7 @@ spec: and these routing rules are only applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are - a Service targeted as a ParentRef of the Route. \n Support: + a Service targeted as a ParentRef of the Route. \n Support: Core" maxLength: 63 minLength: 1 @@ -8489,11 +8911,11 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n When the parent resource is + both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must - match both specified values. \n Implementations MAY choose + match both specified values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, @@ -8565,15 +8987,12 @@ status: conditions: null storedVersions: null --- -# -# config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml -# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: referencegrants.gateway.networking.k8s.io @@ -8854,15 +9273,12 @@ status: conditions: null storedVersions: null --- -# -# config/crd/experimental/gateway.networking.k8s.io_tcproutes.yaml -# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: tcproutes.gateway.networking.k8s.io @@ -8915,30 +9331,41 @@ spec: for governing ParentRefs to Services - it is not possible to create a \"producer\" route for a Service in a different namespace from the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services only) \n This + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) This API may be extended in the future to support additional kinds of - parent resources. \n It is invalid to reference an identical parent - more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as two separate Listeners - on the same Gateway or two separate ports on the same Service. \n - It is possible to separately reference multiple distinct objects - that may be collapsed by an implementation. For example, some implementations - may choose to merge compatible Gateway Listeners together. If that - is the case, the list of routes attached to those resources should - also be merged. \n Note that for ParentRefs that cross namespace - boundaries, there are specific rules. Cross-namespace references - are only valid if they are explicitly allowed by something in the - namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable other - kinds of cross-namespace reference. \n ParentRefs from a Route to - a Service in the same namespace are \"producer\" routes, which apply - default routing rules to inbound connections from any namespace - to the Service. \n ParentRefs from a Route to a Service in a different - namespace are \"consumer\" routes, and these routing rules are only - applied to outbound connections originating from the same namespace - as the Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. \n " + parent resources. \n ParentRefs must be _distinct_. This means either + that: \n * They select different objects. If this is the case, + then parentRef entries are distinct. In terms of fields, this means + that the multi-part key defined by `group`, `kind`, `namespace`, + and `name` must be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field + used, each ParentRef that selects the same object must set the same + set of optional fields to different values. If one ParentRef sets + a combination of optional fields, all must set the same combination. + \n Some examples: \n * If one ParentRef sets `sectionName`, all + ParentRefs referencing the same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. * If one ParentRef sets `sectionName` + and `port`, all ParentRefs referencing the same object must also + set `sectionName` and `port`. \n It is possible to separately reference + multiple distinct objects that may be collapsed by an implementation. + For example, some implementations may choose to merge compatible + Gateway Listeners together. If that is the case, the list of routes + attached to those resources should also be merged. \n Note that + for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For + example, Gateway has the AllowedRoutes field, and ReferenceGrant + provides a generic way to enable other kinds of cross-namespace + reference. \n ParentRefs from a Route to a Service in the same + namespace are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. \n ParentRefs + from a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which the + intended destination of the connections are a Service targeted as + a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually @@ -8986,7 +9413,7 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route to a Service in the same namespace + \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to a Service in a different namespace @@ -8994,7 +9421,7 @@ spec: applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are a Service targeted as a ParentRef of - the Route. \n Support: Core" + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -9009,11 +9436,11 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n When the parent resource + must match both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must match both specified - values. \n Implementations MAY choose to support other parent + values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered @@ -9065,16 +9492,13 @@ spec: - message: sectionName or port must be specified when parentRefs includes 2 or more references to the same parent rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind - == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ - == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) - && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) - || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName - == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) - || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName - != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) - && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) - ) ) ): true ))' + == '''')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) + || p1.sectionName == '''') == (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) + || p2.port == 0)): true))' - message: sectionName or port must be unique when parentRefs includes 2 or more references to the same parent rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind @@ -9109,7 +9533,24 @@ spec: namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. - See the ReferenceGrant documentation for details." + See the ReferenceGrant documentation for details. \n + \n When the BackendRef points to a Kubernetes Service, implementations + SHOULD honor the appProtocol field if it is set for the + target Service Port. \n Implementations supporting appProtocol + SHOULD recognize the Kubernetes Standard Application Protocols + defined in KEP-3726. \n If a Service appProtocol isn't specified, + an implementation MAY infer the backend protocol through + its own means. Implementations MAY infer the protocol from + the Route type referring to the backend Service. \n If a + Route is not able to send traffic to the backend using the + specified protocol then the backend is considered invalid. + Implementations MUST set the \"ResolvedRefs\" condition + to \"False\" with the \"UnsupportedProtocol\" reason. \n + \n Note that when the + BackendTLSPolicy object is enabled by the implementation, + there are some extra rules about validity to consider here. + See the fields where this struct is used for more information + about the exact behavior." properties: group: default: "" @@ -9369,7 +9810,7 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs from a Route to a Service in + reference. \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to @@ -9377,7 +9818,7 @@ spec: and these routing rules are only applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are - a Service targeted as a ParentRef of the Route. \n Support: + a Service targeted as a ParentRef of the Route. \n Support: Core" maxLength: 63 minLength: 1 @@ -9394,11 +9835,11 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n When the parent resource is + both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must - match both specified values. \n Implementations MAY choose + match both specified values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, @@ -9470,15 +9911,12 @@ status: conditions: null storedVersions: null --- -# -# config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml -# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: tlsroutes.gateway.networking.k8s.io @@ -9577,30 +10015,41 @@ spec: for governing ParentRefs to Services - it is not possible to create a \"producer\" route for a Service in a different namespace from the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services only) \n This + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) This API may be extended in the future to support additional kinds of - parent resources. \n It is invalid to reference an identical parent - more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as two separate Listeners - on the same Gateway or two separate ports on the same Service. \n - It is possible to separately reference multiple distinct objects - that may be collapsed by an implementation. For example, some implementations - may choose to merge compatible Gateway Listeners together. If that - is the case, the list of routes attached to those resources should - also be merged. \n Note that for ParentRefs that cross namespace - boundaries, there are specific rules. Cross-namespace references - are only valid if they are explicitly allowed by something in the - namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable other - kinds of cross-namespace reference. \n ParentRefs from a Route to - a Service in the same namespace are \"producer\" routes, which apply - default routing rules to inbound connections from any namespace - to the Service. \n ParentRefs from a Route to a Service in a different - namespace are \"consumer\" routes, and these routing rules are only - applied to outbound connections originating from the same namespace - as the Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. \n " + parent resources. \n ParentRefs must be _distinct_. This means either + that: \n * They select different objects. If this is the case, + then parentRef entries are distinct. In terms of fields, this means + that the multi-part key defined by `group`, `kind`, `namespace`, + and `name` must be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field + used, each ParentRef that selects the same object must set the same + set of optional fields to different values. If one ParentRef sets + a combination of optional fields, all must set the same combination. + \n Some examples: \n * If one ParentRef sets `sectionName`, all + ParentRefs referencing the same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. * If one ParentRef sets `sectionName` + and `port`, all ParentRefs referencing the same object must also + set `sectionName` and `port`. \n It is possible to separately reference + multiple distinct objects that may be collapsed by an implementation. + For example, some implementations may choose to merge compatible + Gateway Listeners together. If that is the case, the list of routes + attached to those resources should also be merged. \n Note that + for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For + example, Gateway has the AllowedRoutes field, and ReferenceGrant + provides a generic way to enable other kinds of cross-namespace + reference. \n ParentRefs from a Route to a Service in the same + namespace are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. \n ParentRefs + from a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which the + intended destination of the connections are a Service targeted as + a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually @@ -9648,7 +10097,7 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route to a Service in the same namespace + \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to a Service in a different namespace @@ -9656,7 +10105,7 @@ spec: applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are a Service targeted as a ParentRef of - the Route. \n Support: Core" + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -9671,11 +10120,11 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n When the parent resource + must match both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must match both specified - values. \n Implementations MAY choose to support other parent + values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered @@ -9727,16 +10176,13 @@ spec: - message: sectionName or port must be specified when parentRefs includes 2 or more references to the same parent rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind - == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ - == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) - && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) - || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName - == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) - || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName - != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) - && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) - ) ) ): true ))' + == '''')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) + || p1.sectionName == '''') == (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) + || p2.port == 0)): true))' - message: sectionName or port must be unique when parentRefs includes 2 or more references to the same parent rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind @@ -9774,7 +10220,24 @@ spec: namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. - See the ReferenceGrant documentation for details." + See the ReferenceGrant documentation for details. \n + \n When the BackendRef points to a Kubernetes Service, implementations + SHOULD honor the appProtocol field if it is set for the + target Service Port. \n Implementations supporting appProtocol + SHOULD recognize the Kubernetes Standard Application Protocols + defined in KEP-3726. \n If a Service appProtocol isn't specified, + an implementation MAY infer the backend protocol through + its own means. Implementations MAY infer the protocol from + the Route type referring to the backend Service. \n If a + Route is not able to send traffic to the backend using the + specified protocol then the backend is considered invalid. + Implementations MUST set the \"ResolvedRefs\" condition + to \"False\" with the \"UnsupportedProtocol\" reason. \n + \n Note that when the + BackendTLSPolicy object is enabled by the implementation, + there are some extra rules about validity to consider here. + See the fields where this struct is used for more information + about the exact behavior." properties: group: default: "" @@ -10034,7 +10497,7 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs from a Route to a Service in + reference. \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to @@ -10042,7 +10505,7 @@ spec: and these routing rules are only applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are - a Service targeted as a ParentRef of the Route. \n Support: + a Service targeted as a ParentRef of the Route. \n Support: Core" maxLength: 63 minLength: 1 @@ -10059,11 +10522,11 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n When the parent resource is + both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must - match both specified values. \n Implementations MAY choose + match both specified values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, @@ -10135,15 +10598,12 @@ status: conditions: null storedVersions: null --- -# -# config/crd/experimental/gateway.networking.k8s.io_udproutes.yaml -# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2245 - gateway.networking.k8s.io/bundle-version: v0.8.0 + api-approved.kubernetes.io: https://github.com/kubernetes-sigs/gateway-api/pull/2466 + gateway.networking.k8s.io/bundle-version: v1.0.0 gateway.networking.k8s.io/channel: experimental creationTimestamp: null name: udproutes.gateway.networking.k8s.io @@ -10196,30 +10656,41 @@ spec: for governing ParentRefs to Services - it is not possible to create a \"producer\" route for a Service in a different namespace from the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services only) \n This + support: \n * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services only) This API may be extended in the future to support additional kinds of - parent resources. \n It is invalid to reference an identical parent - more than once. It is valid to reference multiple distinct sections - within the same parent resource, such as two separate Listeners - on the same Gateway or two separate ports on the same Service. \n - It is possible to separately reference multiple distinct objects - that may be collapsed by an implementation. For example, some implementations - may choose to merge compatible Gateway Listeners together. If that - is the case, the list of routes attached to those resources should - also be merged. \n Note that for ParentRefs that cross namespace - boundaries, there are specific rules. Cross-namespace references - are only valid if they are explicitly allowed by something in the - namespace they are referring to. For example, Gateway has the AllowedRoutes - field, and ReferenceGrant provides a generic way to enable other - kinds of cross-namespace reference. \n ParentRefs from a Route to - a Service in the same namespace are \"producer\" routes, which apply - default routing rules to inbound connections from any namespace - to the Service. \n ParentRefs from a Route to a Service in a different - namespace are \"consumer\" routes, and these routing rules are only - applied to outbound connections originating from the same namespace - as the Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. \n " + parent resources. \n ParentRefs must be _distinct_. This means either + that: \n * They select different objects. If this is the case, + then parentRef entries are distinct. In terms of fields, this means + that the multi-part key defined by `group`, `kind`, `namespace`, + and `name` must be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field + used, each ParentRef that selects the same object must set the same + set of optional fields to different values. If one ParentRef sets + a combination of optional fields, all must set the same combination. + \n Some examples: \n * If one ParentRef sets `sectionName`, all + ParentRefs referencing the same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. * If one ParentRef sets `sectionName` + and `port`, all ParentRefs referencing the same object must also + set `sectionName` and `port`. \n It is possible to separately reference + multiple distinct objects that may be collapsed by an implementation. + For example, some implementations may choose to merge compatible + Gateway Listeners together. If that is the case, the list of routes + attached to those resources should also be merged. \n Note that + for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For + example, Gateway has the AllowedRoutes field, and ReferenceGrant + provides a generic way to enable other kinds of cross-namespace + reference. \n ParentRefs from a Route to a Service in the same + namespace are \"producer\" routes, which apply default routing rules + to inbound connections from any namespace to the Service. \n ParentRefs + from a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which the + intended destination of the connections are a Service targeted as + a ParentRef of the Route. \n " items: description: "ParentReference identifies an API object (usually a Gateway) that can be considered a parent of this resource (usually @@ -10267,7 +10738,7 @@ spec: the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route to a Service in the same namespace + \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to a Service in a different namespace @@ -10275,7 +10746,7 @@ spec: applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are a Service targeted as a ParentRef of - the Route. \n Support: Core" + the Route. \n Support: Core" maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -10290,11 +10761,11 @@ spec: a Route must apply to a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener - must match both specified values. \n When the parent resource + must match both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must match both specified - values. \n Implementations MAY choose to support other parent + values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, an attachment is considered @@ -10346,16 +10817,13 @@ spec: - message: sectionName or port must be specified when parentRefs includes 2 or more references to the same parent rule: 'self.all(p1, self.all(p2, p1.group == p2.group && p1.kind - == p2.kind && p1.name == p2.name && ( ( (!has(p1.__namespace__) + == p2.kind && p1.name == p2.name && (((!has(p1.__namespace__) || p1.__namespace__ == '''') && (!has(p2.__namespace__) || p2.__namespace__ - == '''') ) || ( has(p1.__namespace__) && has(p2.__namespace__) - && p1.__namespace__ == p2.__namespace__ ) ) ? ( ( ( (!has(p1.sectionName) - || p1.sectionName == '''') && (!has(p2.sectionName) || p2.sectionName - == '''') && (!has(p1.port) || p1.port == 0) && (!has(p2.port) - || p2.port == 0) ) || ( ( (has(p1.sectionName) && p1.sectionName - != '''') || (has(p1.port) && p1.port != 0) ) && ( (has(p2.sectionName) - && p2.sectionName != '''') || (has(p2.port) && p2.port != 0) ) - ) ) ): true ))' + == '''')) || (has(p1.__namespace__) && has(p2.__namespace__) && + p1.__namespace__ == p2.__namespace__)) ? ((!has(p1.sectionName) + || p1.sectionName == '''') == (!has(p2.sectionName) || p2.sectionName + == '''') && (!has(p1.port) || p1.port == 0) == (!has(p2.port) + || p2.port == 0)): true))' - message: sectionName or port must be unique when parentRefs includes 2 or more references to the same parent rule: self.all(p1, self.exists_one(p2, p1.group == p2.group && p1.kind @@ -10390,7 +10858,24 @@ spec: namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. - See the ReferenceGrant documentation for details." + See the ReferenceGrant documentation for details. \n + \n When the BackendRef points to a Kubernetes Service, implementations + SHOULD honor the appProtocol field if it is set for the + target Service Port. \n Implementations supporting appProtocol + SHOULD recognize the Kubernetes Standard Application Protocols + defined in KEP-3726. \n If a Service appProtocol isn't specified, + an implementation MAY infer the backend protocol through + its own means. Implementations MAY infer the protocol from + the Route type referring to the backend Service. \n If a + Route is not able to send traffic to the backend using the + specified protocol then the backend is considered invalid. + Implementations MUST set the \"ResolvedRefs\" condition + to \"False\" with the \"UnsupportedProtocol\" reason. \n + \n Note that when the + BackendTLSPolicy object is enabled by the implementation, + there are some extra rules about validity to consider here. + See the fields where this struct is used for more information + about the exact behavior." properties: group: default: "" @@ -10467,10 +10952,10 @@ spec: required: - name type: object - # x-kubernetes-validations: - # - message: Must have port for Service reference - # rule: '(size(self.group) == 0 && self.kind == ''Service'') - # ? has(self.port) : true' + x-kubernetes-validations: + - message: Must have port for Service reference + rule: '(size(self.group) == 0 && self.kind == ''Service'') + ? has(self.port) : true' maxItems: 16 minItems: 1 type: array @@ -10650,7 +11135,7 @@ spec: in the namespace they are referring to. For example: Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs from a Route to a Service in + reference. \n ParentRefs from a Route to a Service in the same namespace are \"producer\" routes, which apply default routing rules to inbound connections from any namespace to the Service. \n ParentRefs from a Route to @@ -10658,7 +11143,7 @@ spec: and these routing rules are only applied to outbound connections originating from the same namespace as the Route, for which the intended destination of the connections are - a Service targeted as a ParentRef of the Route. \n Support: + a Service targeted as a ParentRef of the Route. \n Support: Core" maxLength: 63 minLength: 1 @@ -10675,11 +11160,11 @@ spec: a specific port as opposed to a listener(s) whose port(s) may be changed. When both Port and SectionName are specified, the name and port of the selected listener must match - both specified values. \n When the parent resource is + both specified values. \n When the parent resource is a Service, this targets a specific port in the Service spec. When both Port (experimental) and SectionName are specified, the name and port of the selected port must - match both specified values. \n Implementations MAY choose + match both specified values. \n Implementations MAY choose to support other parent resources. Implementations supporting other types of parent resources MUST clearly document how/if Port is interpreted. \n For the purpose of status, diff --git a/deploy/manifests/static/stunner-crd.yaml b/deploy/manifests/static/stunner-crd.yaml index 77b5eac5..c7243451 100644 --- a/deploy/manifests/static/stunner-crd.yaml +++ b/deploy/manifests/static/stunner-crd.yaml @@ -3,8 +3,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.11.1 - creationTimestamp: null + controller-gen.kubebuilder.io/version: v0.13.0 name: gatewayconfigs.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -15,10 +14,145 @@ spec: listKind: GatewayConfigList plural: gatewayconfigs shortNames: - - gtwconf + - gwconf singular: gatewayconfig scope: Namespaced versions: + - additionalPrinterColumns: + - jsonPath: .spec.realm + name: Realm + type: string + - jsonPath: .spec.dataplane + name: Dataplane + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1 + schema: + openAPIV3Schema: + description: GatewayConfig is the Schema for the gatewayconfigs API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: GatewayConfigSpec defines the desired state of GatewayConfig + properties: + authLifetime: + description: AuthLifetime defines the lifetime of "longterm" authentication + credentials in seconds. + format: int32 + type: integer + authRef: + description: 'Note that externally set credentials override any inline + auth credentials (AuthType, AuthUsername, etc.): if AuthRef is nonempty + then it is expected that the referenced Secret exists and *all* + authentication credentials are correctly set in the referenced Secret + (username/password or shared secret). Mixing of credential sources + (inline/external) is not supported.' + properties: + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty string, + core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Secret + description: Kind is kind of the referent. For example "Secret". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referenced object. + When unspecified, the local namespace is inferred. \n Note that + when a namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace + to allow that namespace's owner to accept the reference. See + the ReferenceGrant documentation for details. \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + required: + - name + type: object + authType: + default: plaintext + description: AuthType is the type of the STUN/TURN authentication + mechanism. + pattern: ^plaintext|static|longterm|ephemeral|timewindowed$ + type: string + dataplane: + default: default + description: Dataplane defines the dataplane (stunnerd image, version, + etc) for STUNner gateways using this GatewayConfig. + type: string + loadBalancerServiceAnnotations: + additionalProperties: + type: string + description: "LoadBalancerServiceAnnotations is a list of annotations + that will go into the LoadBalancer services created automatically + by the operator to wrap Gateways. \n NOTE: removing annotations + from a GatewayConfig will not result in the removal of the corresponding + annotations from the LoadBalancer service, in order to prevent the + accidental removal of an annotation installed there by Kubernetes + or the cloud provider. If you really want to remove an annotation, + do this manually or simply remove all Gateways (which will remove + the corresponding LoadBalancer services), update the GatewayConfig + and then recreate the Gateways, so that the newly created LoadBalancer + services will contain the required annotations." + type: object + logLevel: + description: LogLevel specifies the default loglevel for the STUNner + daemon. + type: string + password: + description: Password defines the `password` credential for "plaintext" + authentication. + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + realm: + default: stunner.l7mp.io + description: "Realm defines the STUN/TURN authentication realm to + be used for clients toauthenticate with STUNner. \n The realm must + consist of lower case alphanumeric characters or '-', and must start + and end with an alphanumeric character. No other punctuation is + allowed." + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + sharedSecret: + description: SharedSecret defines the shared secret to be used for + "longterm" authentication. + type: string + userName: + description: Username defines the `username` credential for "plaintext" + authentication. + pattern: ^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$ + type: string + type: object + type: object + served: true + storage: true + subresources: {} - additionalPrinterColumns: - jsonPath: .spec.realm name: Realm @@ -83,9 +217,9 @@ spec: minLength: 1 type: string namespace: - description: "Namespace is the namespace of the backend. When - unspecified, the local namespace is inferred. \n Note that when - a namespace different than the local namespace is specified, + description: "Namespace is the namespace of the referenced object. + When unspecified, the local namespace is inferred. \n Note that + when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. \n Support: Core" @@ -182,15 +316,14 @@ spec: type: object type: object served: true - storage: true + storage: false subresources: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.11.1 - creationTimestamp: null + controller-gen.kubebuilder.io/version: v0.13.0 name: staticservices.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -205,6 +338,41 @@ spec: singular: staticservice scope: Namespaced versions: + - name: v1 + schema: + openAPIV3Schema: + description: StaticService is a set of static IP address prefixes STUNner + allows access to via a UDPRoute (or TCPRoute in the future). In contrast + to Kubernetes Services, StaticServices expose all ports on the given IPs. + See also https://github.com/kubernetes/enhancements/pull/2611. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the behavior of a service. + properties: + prefixes: + description: Prefixes is a list of IP address prefixes reachable via + this route. + items: + type: string + type: array + required: + - prefixes + type: object + type: object + served: true + storage: true - name: v1alpha1 schema: openAPIV3Schema: @@ -235,11 +403,19 @@ spec: description: ServicePort contains information on service's port. properties: appProtocol: - description: The application protocol for this port. This field - follows standard Kubernetes label syntax. Un-prefixed names - are reserved for IANA standard service names (as per RFC-6335 - and https://www.iana.org/assignments/service-names). Non-standard - protocols should use prefixed names such as mycompany.com/my-custom-protocol. + description: "The application protocol for this port. This is + used as a hint for implementations to offer richer behavior + for protocols that they understand. This field follows standard + Kubernetes label syntax. Valid values are either: \n * Un-prefixed + protocol names - reserved for IANA standard service names + (as per RFC-6335 and https://www.iana.org/assignments/service-names). + \n * Kubernetes-defined prefixed names: * 'kubernetes.io/h2c' + - HTTP/2 over cleartext as described in https://www.rfc-editor.org/rfc/rfc7540 + * 'kubernetes.io/ws' - WebSocket over cleartext as described + in https://www.rfc-editor.org/rfc/rfc6455 * 'kubernetes.io/wss' + - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455 + \n * Other protocols should use implementation-defined prefixed + names such as mycompany.com/my-custom-protocol." type: string name: description: The name of this port within the service. This @@ -301,14 +477,13 @@ spec: type: object type: object served: true - storage: true + storage: false --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.11.1 - creationTimestamp: null + controller-gen.kubebuilder.io/version: v0.13.0 name: dataplanes.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -323,7 +498,7 @@ spec: singular: dataplane scope: Cluster versions: - - name: v1alpha1 + - name: v1 schema: openAPIV3Schema: description: Dataplane is a collection of configuration parameters that can @@ -1183,6 +1358,16 @@ spec: items: type: string type: array + disableHealthCheck: + description: 'Disable health-checking. Default is to enable HTTP health-checks + on port 8086: a liveness probe responder will be exposed on path + `/live` and readiness probe on path `/ready`.' + type: boolean + enableMetricsEndpoint: + description: EnableMetricsEnpoint can be used to enable metrics scraping + (Prometheus). If enabled, a metrics endpoint will available at http://0.0.0.0:8080 + at all dataplane pods. Default is no metrics collection. + type: boolean env: description: List of environment variables to set in the stunnerd container. @@ -1293,9 +1478,42 @@ spec: - name type: object type: array - healthCheckPort: - description: If specified, the health-check port. - type: integer + envFrom: + description: List of sources to populate environment variables in + the stunnerd container. + items: + description: EnvFromSource represents the source of a set of ConfigMaps + properties: + configMapRef: + description: The ConfigMap to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + prefix: + description: An optional identifier to prepend to each key in + the ConfigMap. Must be a C_IDENTIFIER. + type: string + secretRef: + description: The Secret to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + type: object + type: array hostNetwork: description: Host networking requested for the stunnerd pod to use the host's network namespace. Can be used to implement public TURN @@ -1576,4 +1794,1845 @@ spec: type: object served: true storage: true ---- + - name: v1alpha1 + schema: + openAPIV3Schema: + description: Dataplane is a collection of configuration parameters that can + be used for spawning a `stunnerd` instance for a Gateway. Labels and annotations + on the Dataplane object will be copied verbatim into the target Deployment. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the behavior of a Dataplane resource. + properties: + affinity: + description: Scheduling constraints. + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for the + pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node matches + the corresponding matchExpressions; the node(s) with the + highest sum are the most preferred. + items: + description: An empty preferred scheduling term matches + all objects with implicit weight 0 (i.e. it's a no-op). + A null preferred scheduling term matches no objects (i.e. + is also a no-op). + properties: + preference: + description: A node selector term, associated with the + corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + weight: + description: Weight associated with matching the corresponding + nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to an update), the system may or may not try to + eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. + The terms are ORed. + items: + description: A null or empty node selector term matches + no objects. The requirements of them are ANDed. The + TopologySelectorTerm type implements a subset of the + NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + type: array + required: + - nodeSelectorTerms + type: object + x-kubernetes-map-type: atomic + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. co-locate + this pod in the same node, zone, etc. as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to a pod label update), the system may or may + not try to eventually evict the pod from its node. When + there are multiple elements, the lists of nodes corresponding + to each podAffinityTerm are intersected, i.e. all terms + must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules (e.g. + avoid putting this pod in the same node, zone, etc. as some + other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the anti-affinity expressions specified + by this field, but it may choose a node that violates one + or more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the anti-affinity requirements specified by + this field are not met at scheduling time, the pod will + not be scheduled onto the node. If the anti-affinity requirements + specified by this field cease to be met at some point during + pod execution (e.g. due to a pod label update), the system + may or may not try to eventually evict the pod from its + node. When there are multiple elements, the lists of nodes + corresponding to each podAffinityTerm are intersected, i.e. + all terms must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + type: object + args: + description: Arguments to the entrypoint. + items: + type: string + type: array + command: + description: 'Entrypoint array. Defaults: "stunnerd".' + items: + type: string + type: array + env: + description: List of environment variables to set in the stunnerd + container. + items: + description: EnvVar represents an environment variable present in + a Container. + properties: + name: + description: Name of the environment variable. Must be a C_IDENTIFIER. + type: string + value: + description: 'Variable references $(VAR_NAME) are expanded using + the previously defined environment variables in the container + and any service environment variables. If a variable cannot + be resolved, the reference in the input string will be unchanged. + Double $$ are reduced to a single $, which allows for escaping + the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the + string literal "$(VAR_NAME)". Escaped references will never + be expanded, regardless of whether the variable exists or + not. Defaults to "".' + type: string + valueFrom: + description: Source for the environment variable's value. Cannot + be used if value is not empty. + properties: + configMapKeyRef: + description: Selects a key of a ConfigMap. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + fieldRef: + description: 'Selects a field of the pod: supports metadata.name, + metadata.namespace, `metadata.labels['''']`, `metadata.annotations['''']`, + spec.nodeName, spec.serviceAccountName, status.hostIP, + status.podIP, status.podIPs.' + properties: + apiVersion: + description: Version of the schema the FieldPath is + written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the specified + API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + resourceFieldRef: + description: 'Selects a resource of the container: only + resources limits and requests (limits.cpu, limits.memory, + limits.ephemeral-storage, requests.cpu, requests.memory + and requests.ephemeral-storage) are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the exposed + resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + secretKeyRef: + description: Selects a key of a secret in the pod's namespace + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + required: + - name + type: object + type: array + healthCheckPort: + description: If specified, the health-check port. + type: integer + hostNetwork: + description: Host networking requested for the stunnerd pod to use + the host's network namespace. Can be used to implement public TURN + servers with Kubernetes. Defaults to false. + type: boolean + image: + description: Container image name. + type: string + imagePullPolicy: + description: Image pull policy. One of Always, Never, IfNotPresent. + type: string + replicas: + description: Number of desired pods. This is a pointer to distinguish + between explicit zero and not specified. Defaults to 1. + format: int32 + type: integer + resources: + description: Resources required by stunnerd. + properties: + claims: + description: "Claims lists the names of resources, defined in + spec.resourceClaims, that are used by this container. \n This + is an alpha field and requires enabling the DynamicResourceAllocation + feature gate. \n This field is immutable. It can only be set + for containers." + items: + description: ResourceClaim references one entry in PodSpec.ResourceClaims. + properties: + name: + description: Name must match the name of one entry in pod.spec.resourceClaims + of the Pod where this field is used. It makes that resource + available inside a container. + type: string + required: + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute resources + allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, otherwise + to an implementation-defined value. Requests cannot exceed Limits. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + securityContext: + description: SecurityContext holds pod-level security attributes and + common container settings. + properties: + fsGroup: + description: "A special supplemental group that applies to all + containers in a pod. Some volume types allow the Kubelet to + change the ownership of that volume to be owned by the pod: + \n 1. The owning GID will be the FSGroup 2. The setgid bit is + set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- \n If unset, + the Kubelet will not modify the ownership and permissions of + any volume. Note that this field cannot be set when spec.os.name + is windows." + format: int64 + type: integer + fsGroupChangePolicy: + description: 'fsGroupChangePolicy defines behavior of changing + ownership and permission of the volume before being exposed + inside Pod. This field will only apply to volume types which + support fsGroup based ownership(and permissions). It will have + no effect on ephemeral volume types such as: secret, configmaps + and emptydir. Valid values are "OnRootMismatch" and "Always". + If not specified, "Always" is used. Note that this field cannot + be set when spec.os.name is windows.' + type: string + runAsGroup: + description: The GID to run the entrypoint of the container process. + Uses runtime default if unset. May also be set in SecurityContext. If + set in both SecurityContext and PodSecurityContext, the value + specified in SecurityContext takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a non-root + user. If true, the Kubelet will validate the image at runtime + to ensure that it does not run as UID 0 (root) and fail to start + the container if it does. If unset or false, no such validation + will be performed. May also be set in SecurityContext. If set + in both SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in SecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. Note that this field cannot + be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random + SELinux context for each container. May also be set in SecurityContext. If + set in both SecurityContext and PodSecurityContext, the value + specified in SecurityContext takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies to + the container. + type: string + role: + description: Role is a SELinux role label that applies to + the container. + type: string + type: + description: Type is a SELinux type label that applies to + the container. + type: string + user: + description: User is a SELinux user label that applies to + the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by the containers in this + pod. Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile must be + preconfigured on the node to work. Must be a descending + path, relative to the kubelet's configured seccomp profile + location. Must be set if type is "Localhost". Must NOT be + set for any other type. + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - a profile + defined in a file on the node should be used. RuntimeDefault + - the container runtime default profile should be used. + Unconfined - no profile should be applied." + type: string + required: + - type + type: object + supplementalGroups: + description: A list of groups applied to the first process run + in each container, in addition to the container's primary GID, + the fsGroup (if specified), and group memberships defined in + the container image for the uid of the container process. If + unspecified, no additional groups are added to any container. + Note that group memberships defined in the container image for + the uid of the container process are still effective, even if + they are not included in this list. Note that this field cannot + be set when spec.os.name is windows. + items: + format: int64 + type: integer + type: array + sysctls: + description: Sysctls hold a list of namespaced sysctls used for + the pod. Pods with unsupported sysctls (by the container runtime) + might fail to launch. Note that this field cannot be set when + spec.os.name is windows. + items: + description: Sysctl defines a kernel parameter to be set + properties: + name: + description: Name of a property to set + type: string + value: + description: Value of a property to set + type: string + required: + - name + - value + type: object + type: array + windowsOptions: + description: The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named by + the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the GMSA + credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. All of a Pod's containers + must have the same effective HostProcess value (it is not + allowed to have a mix of HostProcess containers and non-HostProcess + containers). In addition, if HostProcess is true then HostNetwork + must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set in PodSecurityContext. + If set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. + type: string + type: object + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the stunnerd needs to terminate + gracefully. Defaults to 3600 seconds. + format: int64 + type: integer + tolerations: + description: If specified, the pod's tolerations. + items: + description: The pod this Toleration is attached to tolerates any + taint that matches the triple using the matching + operator . + properties: + effect: + description: Effect indicates the taint effect to match. Empty + means match all taint effects. When specified, allowed values + are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: Key is the taint key that the toleration applies + to. Empty means match all taint keys. If the key is empty, + operator must be Exists; this combination means to match all + values and all keys. + type: string + operator: + description: Operator represents a key's relationship to the + value. Valid operators are Exists and Equal. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod + can tolerate all taints of a particular category. + type: string + tolerationSeconds: + description: TolerationSeconds represents the period of time + the toleration (which must be of effect NoExecute, otherwise + this field is ignored) tolerates the taint. By default, it + is not set, which means tolerate the taint forever (do not + evict). Zero and negative values will be treated as 0 (evict + immediately) by the system. + format: int64 + type: integer + value: + description: Value is the taint value the toleration matches + to. If the operator is Exists, the value should be empty, + otherwise just a regular string. + type: string + type: object + type: array + type: object + type: object + served: true + storage: false +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.13.0 + name: udproutes.stunner.l7mp.io +spec: + group: stunner.l7mp.io + names: + categories: + - stunner + kind: UDPRoute + listKind: UDPRouteList + plural: udproutes + singular: udproute + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1 + schema: + openAPIV3Schema: + description: "UDPRoute provides a way to route UDP traffic. When combined + with a Gateway listener, it can be used to forward traffic on the port specified + by the listener to a set of backends specified by the UDPRoute. \n Differences + from Gateway API UDPRoutes - port-ranges are correctly handled ([port, endPort]) + - port is not mandatory - backend weight is not supported" + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Spec defines the desired state of UDPRoute. + properties: + parentRefs: + description: "ParentRefs references the resources (usually Gateways) + that a Route wants to be attached to. Note that the referenced parent + resource needs to allow this for the attachment to be complete. + For Gateways, that means the Gateway needs to allow attachment from + Routes of this kind and namespace. For Services, that means the + Service must either be in the same namespace for a \"producer\" + route, or the mesh implementation must support and allow \"consumer\" + routes for the referenced Service. ReferenceGrant is not applicable + for governing ParentRefs to Services - it is not possible to create + a \"producer\" route for a Service in a different namespace from + the Route. \n There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) + * Service (Mesh conformance profile, experimental, ClusterIP Services + only) This API may be extended + in the future to support additional kinds of parent resources. \n + ParentRefs must be _distinct_. This means either that: \n * They + select different objects. If this is the case, then parentRef entries + are distinct. In terms of fields, this means that the multi-part + key defined by `group`, `kind`, `namespace`, and `name` must be + unique across all parentRef entries in the Route. * They do not + select different objects, but for each optional field used, each + ParentRef that selects the same object must set the same set of + optional fields to different values. If one ParentRef sets a combination + of optional fields, all must set the same combination. \n Some examples: + \n * If one ParentRef sets `sectionName`, all ParentRefs referencing + the same object must also set `sectionName`. * If one ParentRef + sets `port`, all ParentRefs referencing the same object must also + set `port`. * If one ParentRef sets `sectionName` and `port`, all + ParentRefs referencing the same object must also set `sectionName` + and `port`. \n It is possible to separately reference multiple distinct + objects that may be collapsed by an implementation. For example, + some implementations may choose to merge compatible Gateway Listeners + together. If that is the case, the list of routes attached to those + resources should also be merged. \n Note that for ParentRefs that + cross namespace boundaries, there are specific rules. Cross-namespace + references are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example, Gateway has + the AllowedRoutes field, and ReferenceGrant provides a generic way + to enable other kinds of cross-namespace reference. \n + ParentRefs from a Route to a Service in the same namespace are \"producer\" + routes, which apply default routing rules to inbound connections + from any namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, and + these routing rules are only applied to outbound connections originating + from the same namespace as the Route, for which the intended destination + of the connections are a Service targeted as a ParentRef of the + Route. \n + + + " + items: + description: "ParentReference identifies an API object (usually + a Gateway) that can be considered a parent of this resource (usually + a route). There are two kinds of parent resources with \"Core\" + support: \n * Gateway (Gateway conformance profile) * Service + (Mesh conformance profile, experimental, ClusterIP Services only) + \n This API may be extended in the future to support additional + kinds of parent resources. \n The API object must be valid in + the cluster; the Group and Kind must be registered in the cluster + for this reference to be valid." + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the core + API group (such as for a \"Service\" kind referent), Group + must be explicitly set to \"\" (empty string). \n Support: + Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are two + kinds of parent resources with \"Core\" support: \n * Gateway + (Gateway conformance profile) * Service (Mesh conformance + profile, experimental, ClusterIP Services only) \n Support + for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. When + unspecified, this refers to the local namespace of the Route. + \n Note that there are specific rules for ParentRefs which + cross namespace boundaries. Cross-namespace references are + only valid if they are explicitly allowed by something in + the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + \n ParentRefs from a Route + to a Service in the same namespace are \"producer\" routes, + which apply default routing rules to inbound connections from + any namespace to the Service. \n ParentRefs from a Route to + a Service in a different namespace are \"consumer\" routes, + and these routing rules are only applied to outbound connections + originating from the same namespace as the Route, for which + the intended destination of the connections are a Service + targeted as a ParentRef of the Route. + \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. It + can be interpreted differently based on the type of parent + resource. \n When the parent resource is a Gateway, this targets + all listeners listening on the specified port that also support + this kind of Route(and select this Route). It's not recommended + to set `Port` unless the networking behaviors specified in + a Route must apply to a specific port as opposed to a listener(s) + whose port(s) may be changed. When both Port and SectionName + are specified, the name and port of the selected listener + must match both specified values. \n + When the parent resource is a Service, this targets a specific + port in the Service spec. When both Port (experimental) and + SectionName are specified, the name and port of the selected + port must match both specified values. + \n Implementations MAY choose to support other parent resources. + Implementations supporting other types of parent resources + MUST clearly document how/if Port is interpreted. \n For the + purpose of status, an attachment is considered successful + as long as the parent resource accepts it partially. For example, + Gateway listeners can restrict which Routes can attach to + them by Route kind, namespace, or hostname. If 1 of 2 Gateway + listeners accept attachment from the referencing Route, the + Route MUST be considered successfully attached. If no Gateway + listeners accept attachment from this Route, the Route MUST + be considered detached from the Gateway. \n Support: Extended + \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within the + target resource. In the following resources, SectionName is + interpreted as the following: \n * Gateway: Listener Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match both + specified values. * Service: Port Name. When both Port (experimental) + and SectionName are specified, the name and port of the selected + listener must match both specified values. Note that attaching + Routes to Services as Parents is part of experimental Mesh + support and is not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this will + reference the entire resource. For the purpose of status, + an attachment is considered successful if at least one section + in the parent resource accepts it. For example, Gateway listeners + can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept + attachment from the referencing Route, the Route MUST be considered + successfully attached. If no Gateway listeners accept attachment + from this Route, the Route MUST be considered detached from + the Gateway. \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + maxItems: 32 + type: array + rules: + description: Rules are a list of UDP matchers and actions. + items: + description: UDPRouteRule is the configuration for a given rule. + properties: + backendRefs: + description: BackendRefs defines the backend(s) where matching + requests should be sent. UDPRouteRules correctly handle port + ranges. + items: + description: BackendRef defines how a Route should forward + a request to a Kubernetes resource. + properties: + endPort: + description: EndPort specifies the upper threshold of + the port-range. Only considered of port is also specified. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + group: + default: "" + description: Group is the group of the referent. For example, + "gateway.networking.k8s.io". When unspecified or empty + string, core API group is inferred. + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Service + description: Kind is the Kubernetes resource kind of the + referent. For example "Service". + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: Name is the name of the referent. + maxLength: 253 + minLength: 1 + type: string + namespace: + description: Namespace is the namespace of the backend. + When unspecified, the local namespace is inferred. + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: Port specifies the destination port number + to use for this resource. If port is not specified, + all ports are allowed. If port is defined but endPort + is not, allow only access to the given port. If both + are specified, allows access in the port-range [port, + endPort] inclusive. + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - name + type: object + maxItems: 16 + minItems: 1 + type: array + type: object + maxItems: 16 + minItems: 1 + type: array + required: + - rules + type: object + status: + description: Status defines the current state of UDPRoute. + properties: + parents: + description: "Parents is a list of parent resources (usually Gateways) + that are associated with the route, and the status of the route + with respect to each parent. When this route attaches to a parent, + the controller that manages the parent must add an entry to this + list when the controller first sees the route and should update + the entry as appropriate when the route or gateway is modified. + \n Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this + API can only populate Route status for the Gateways/parent resources + they are responsible for. \n A maximum of 32 Gateways will be represented + in this list. An empty list means the route has not been attached + to any Gateway." + items: + description: RouteParentStatus describes the status of a route with + respect to an associated Parent. + properties: + conditions: + description: "Conditions describes the status of the route with + respect to the Gateway. Note that the route's availability + is also subject to the Gateway's own status conditions and + listener status. \n If the Route's ParentRef specifies an + existing Gateway that supports Routes of this kind AND that + Gateway's controller has sufficient access, then that Gateway's + controller MUST set the \"Accepted\" condition on the Route, + to indicate whether the route has been accepted or rejected + by the Gateway, and why. \n A Route MUST be considered \"Accepted\" + if at least one of the Route's rules is implemented by the + Gateway. \n There are a number of cases where the \"Accepted\" + condition may not be set due to lack of controller visibility, + that includes when: \n * The Route refers to a non-existent + parent. * The Route is of a type that the controller does + not support. * The Route is in a namespace the controller + does not have access to." + items: + description: "Condition contains details for one aspect of + the current state of this API Resource. --- This struct + is intended for direct use as an array at the field path + .status.conditions. For example, \n type FooStatus struct{ + // Represents the observations of a foo's current state. + // Known .status.conditions.type are: \"Available\", \"Progressing\", + and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge + // +listType=map // +listMapKey=type Conditions []metav1.Condition + `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" + protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields + }" + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should + be when the underlying condition changed. If that is + not known, then using the time when the API field changed + is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation + that the condition was set based upon. For instance, + if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration + is 9, the condition is out of date with respect to the + current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: reason contains a programmatic identifier + indicating the reason for the condition's last transition. + Producers of specific condition types may define expected + values and meanings for this field, and whether the + values are considered a guaranteed API. The value should + be a CamelCase string. This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + --- Many .condition.type values are consistent across + resources like Available, but because arbitrary conditions + can be useful (see .node.status.conditions), the ability + to deconflict is important. The regex it matches is + (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + maxItems: 8 + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + controllerName: + description: "ControllerName is a domain/path string that indicates + the name of the controller that wrote this status. This corresponds + with the controllerName field on GatewayClass. \n Example: + \"example.net/gateway-controller\". \n The format of this + field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid + Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + \n Controllers MUST populate this field when writing status. + Controllers should ensure that entries to status populated + with their ControllerName are cleaned up when they are no + longer necessary." + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ + type: string + parentRef: + description: ParentRef corresponds with a ParentRef in the spec + that this RouteParentStatus struct describes the status of. + properties: + group: + default: gateway.networking.k8s.io + description: "Group is the group of the referent. When unspecified, + \"gateway.networking.k8s.io\" is inferred. To set the + core API group (such as for a \"Service\" kind referent), + Group must be explicitly set to \"\" (empty string). \n + Support: Core" + maxLength: 253 + pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + kind: + default: Gateway + description: "Kind is kind of the referent. \n There are + two kinds of parent resources with \"Core\" support: \n + * Gateway (Gateway conformance profile) * Service (Mesh + conformance profile, experimental, ClusterIP Services + only) \n Support for other resources is Implementation-Specific." + maxLength: 63 + minLength: 1 + pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ + type: string + name: + description: "Name is the name of the referent. \n Support: + Core" + maxLength: 253 + minLength: 1 + type: string + namespace: + description: "Namespace is the namespace of the referent. + When unspecified, this refers to the local namespace of + the Route. \n Note that there are specific rules for ParentRefs + which cross namespace boundaries. Cross-namespace references + are only valid if they are explicitly allowed by something + in the namespace they are referring to. For example: Gateway + has the AllowedRoutes field, and ReferenceGrant provides + a generic way to enable any other kind of cross-namespace + reference. \n ParentRefs + from a Route to a Service in the same namespace are \"producer\" + routes, which apply default routing rules to inbound connections + from any namespace to the Service. \n ParentRefs from + a Route to a Service in a different namespace are \"consumer\" + routes, and these routing rules are only applied to outbound + connections originating from the same namespace as the + Route, for which the intended destination of the connections + are a Service targeted as a ParentRef of the Route. + \n Support: Core" + maxLength: 63 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ + type: string + port: + description: "Port is the network port this Route targets. + It can be interpreted differently based on the type of + parent resource. \n When the parent resource is a Gateway, + this targets all listeners listening on the specified + port that also support this kind of Route(and select this + Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to + a specific port as opposed to a listener(s) whose port(s) + may be changed. When both Port and SectionName are specified, + the name and port of the selected listener must match + both specified values. \n + When the parent resource is a Service, this targets a + specific port in the Service spec. When both Port (experimental) + and SectionName are specified, the name and port of the + selected port must match both specified values. + \n Implementations MAY choose to support other parent + resources. Implementations supporting other types of parent + resources MUST clearly document how/if Port is interpreted. + \n For the purpose of status, an attachment is considered + successful as long as the parent resource accepts it partially. + For example, Gateway listeners can restrict which Routes + can attach to them by Route kind, namespace, or hostname. + If 1 of 2 Gateway listeners accept attachment from the + referencing Route, the Route MUST be considered successfully + attached. If no Gateway listeners accept attachment from + this Route, the Route MUST be considered detached from + the Gateway. \n Support: Extended \n " + format: int32 + maximum: 65535 + minimum: 1 + type: integer + sectionName: + description: "SectionName is the name of a section within + the target resource. In the following resources, SectionName + is interpreted as the following: \n * Gateway: Listener + Name. When both Port (experimental) and SectionName are + specified, the name and port of the selected listener + must match both specified values. * Service: Port Name. + When both Port (experimental) and SectionName are specified, + the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services + as Parents is part of experimental Mesh support and is + not supported for any other purpose. \n Implementations + MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName + is interpreted. \n When unspecified (empty string), this + will reference the entire resource. For the purpose of + status, an attachment is considered successful if at least + one section in the parent resource accepts it. For example, + Gateway listeners can restrict which Routes can attach + to them by Route kind, namespace, or hostname. If 1 of + 2 Gateway listeners accept attachment from the referencing + Route, the Route MUST be considered successfully attached. + If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + \n Support: Core" + maxLength: 253 + minLength: 1 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + required: + - name + type: object + required: + - controllerName + - parentRef + type: object + maxItems: 32 + type: array + required: + - parents + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/deploy/manifests/static/stunner-gateway-operator-manifests.yaml b/deploy/manifests/static/stunner-gateway-operator-manifests.yaml index 0d4968d6..09b9b5c7 100644 --- a/deploy/manifests/static/stunner-gateway-operator-manifests.yaml +++ b/deploy/manifests/static/stunner-gateway-operator-manifests.yaml @@ -141,6 +141,7 @@ rules: - dataplanes - gatewayconfigs - staticservices + - udproutes verbs: - get - list @@ -153,6 +154,8 @@ rules: - dataplanes/finalizers - gatewayconfigs/finalizers - staticservices/finalizers + - udproutes/finalizers + - udproutes/status verbs: - update --- @@ -456,4 +459,4 @@ spec: terminationGracePeriodSeconds: 10 nodeSelector: kubernetes.io/os: linux ---- \ No newline at end of file +--- From d7e82653ec961ab5b9d0e322132389795b287719 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 21 Dec 2023 19:57:34 +0100 Subject: [PATCH 078/243] test: Update benchmarks to the STUNner TURN URI format --- relay_test.go | 9 +++------ server_unix_test.go | 15 +++++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/relay_test.go b/relay_test.go index c5598f09..b9845570 100644 --- a/relay_test.go +++ b/relay_test.go @@ -188,14 +188,11 @@ func BenchmarkPortRangePacketConn(b *testing.B) { // should never receive: we drop everything err = conn.SetReadDeadline(time.Now().Add(50 * time.Millisecond)) - if err == nil { - b.Fatal("Could not set read deadline") + if err != nil { + b.Fatalf("Could not set read deadline: %s", err) } - _, _, err = conn.ReadFrom(buffer) - if err == nil { - b.Fatal("Conn should never read") - } + conn.ReadFrom(buffer) //nolint:errcheck } readCounter := conn.(*CounterPacketConn).ReadCounter() diff --git a/server_unix_test.go b/server_unix_test.go index 1188307d..a2b16c96 100644 --- a/server_unix_test.go +++ b/server_unix_test.go @@ -54,7 +54,7 @@ func TestStunnerMultithreadedUDP(t *testing.T) { // Benchmark func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { - // loggerFactory := logger.NewLoggerFactory("all:TRACE") + //loggerFactory := logger.NewLoggerFactory("all:TRACE") loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) log := loggerFactory.NewLogger("test") initSeq := []byte("init-data") @@ -66,6 +66,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { SuppressRollback: true, UDPListenerThreadNum: udpThreadNum, // ignored for anything but UDP }) + defer stunner.Close() log.Debug("starting stunnerd") err := stunner.Reconcile(stnrv1.StunnerConfig{ @@ -109,6 +110,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { if err != nil { b.Fatalf("Failed to allocate sink: %s", err) } + defer sink.Close() //nolint:errcheck go func() { buf := make([]byte, 1600) @@ -124,11 +126,14 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { }() log.Debug("creating a turncat client") - stunnerURI := fmt.Sprintf("turn://127.0.0.1:23478?transport=%s", proto) - clientProto := "tcp" + clientProto, turnScheme := "tcp", "turn" if proto == "turn-udp" || proto == "turn-dtls" { clientProto = "udp" } + if proto == "turn-tls" || proto == "turn-dtls" { + turnScheme = "turns" + } + stunnerURI := fmt.Sprintf("%s://127.0.0.1:23478?transport=%s", turnScheme, clientProto) testTurncatConfig := TurncatConfig{ ListenerAddr: fmt.Sprintf("%s://127.0.0.1:25000", clientProto), ServerAddr: stunnerURI, @@ -142,6 +147,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { if err != nil { b.Fatalf("Failed to create turncat client: %s", err) } + defer turncat.Close() // test with 20 clients log.Debugf("creating %d senders", clientNum) @@ -182,12 +188,9 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { time.Sleep(750 * time.Millisecond) - turncat.Close() - stunner.Close() for i := 0; i < clientNum; i++ { clients[i].Close() } - sink.Close() //nolint:errcheck } // BenchmarkUDPServer will benchmark the STUNner UDP server with a different number of readloop From 1da6838422090a767f13bc61ae52fb2b8c7e54f4 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 21 Dec 2023 19:58:17 +0100 Subject: [PATCH 079/243] fix: Let the default stunnerd id pass our own validation check --- cmd/stunnerd/main.go | 10 +++++----- stunner.go | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index c65140e1..52bf399e 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -29,7 +29,7 @@ func main() { os.Args[0] = "stunnerd" var config = flag.StringP("config", "c", "", fmt.Sprintf("Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: %s).", defaultConfigDiscoveryAddress)) var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO).") - var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: ).") + var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: ).") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false).") var udpThreadNum = flag.IntP("udp-thread-num", "u", 0, "Number of readloop threads (CPU cores) per UDP listener. Zero disables UDP multithreading (default: 0).") @@ -72,13 +72,13 @@ func main() { log := st.GetLogger().NewLogger("stunnerd") - log.Infof("starting stunnerd instance %q", *id) + log.Infof("starting stunnerd instance %q", st.GetId()) conf := make(chan stnrv1.StunnerConfig, 1) defer close(conf) var cancelConfigLoader context.CancelFunc - if configOrigin == "" && flag.NArg() == 1 { + if flag.NArg() == 1 { log.Infof("starting %s with default configuration at TURN URI: %s", os.Args[0], flag.Arg(0)) @@ -90,7 +90,7 @@ func main() { conf <- *c - } else if configOrigin != "" && !*watch { + } else if !*watch { log.Infof("loading configuration from origin %q", configOrigin) c, err := st.LoadConfig(configOrigin) @@ -101,7 +101,7 @@ func main() { conf <- *c - } else if configOrigin != "" && *watch { + } else if *watch { log.Info("bootstrapping with minimal config") z := cdsclient.ZeroConfig(st.GetId()) conf <- *z diff --git a/stunner.go b/stunner.go index 0ad49a73..728261f3 100644 --- a/stunner.go +++ b/stunner.go @@ -78,7 +78,7 @@ func NewStunner(options Options) *Stunner { if h, err := os.Hostname(); err != nil { id = DefaultInstanceId } else { - id = h + id = fmt.Sprintf("default/stunnerd-%s", h) } } From 27026ab12a13651a0ebd5dc674e241a792d7b184 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 22 Dec 2023 22:20:48 +0100 Subject: [PATCH 080/243] doc: Update docs for v1, take 1 --- README.md | 138 +++-- deploy/manifests/default-dataplane.yaml | 2 +- deploy/manifests/static/dataplane.yaml | 2 +- deploy/manifests/stunner-expose-kube-dns.yaml | 13 +- deploy/manifests/stunner-test.yaml | 10 +- docs/AUTH.md | 125 +---- docs/CONCEPTS.md | 22 +- docs/DEPLOYMENT.md | 98 ++-- docs/GATEWAY.md | 203 ++++--- docs/INSTALL.md | 127 ++--- docs/MONITORING.md | 70 +-- docs/OBSOLETE.md | 501 ----------------- docs/README.md | 6 +- docs/SCALING.md | 99 +--- docs/SECURITY.md | 92 +--- docs/WHY.md | 39 +- .../benchmark/performance-stunner.yaml | 8 +- docs/examples/cloudretro/README.md | 4 +- .../cloudretro-stunner-cleanup.yaml | 6 +- docs/examples/cloudretro/stunner-gwcc.yaml | 6 +- docs/examples/direct-one2one-call/README.md | 4 +- .../direct-one2one-call-stunner.yaml | 10 +- docs/examples/jitsi/README.md | 6 +- docs/examples/jitsi/jitsi-call-stunner.yaml | 10 +- .../kurento-magic-mirror-stunner.yaml | 8 +- docs/examples/kurento-one2one-call/README.md | 6 +- .../kurento-one2one-call-stunner.yaml | 8 +- docs/examples/livekit/README.md | 6 +- .../livekit/livekit-call-stunner.yaml | 10 +- docs/examples/mediasoup/README.md | 4 +- .../mediasoup/mediasoup-call-stunner.yaml | 10 +- docs/examples/neko/README.md | 3 +- docs/examples/neko/stunner.yaml | 8 +- docs/examples/simple-tunnel/README.md | 6 +- .../examples/simple-tunnel/iperf-stunner.yaml | 10 +- docs/img/stunner_arch_big.svg | 511 +++++++++++------- 36 files changed, 822 insertions(+), 1369 deletions(-) delete mode 100644 docs/OBSOLETE.md diff --git a/README.md b/README.md index 11ba2bf7..c7b3f20e 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@

-*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* +*This is the documentation for the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* # STUNner: A Kubernetes media gateway for WebRTC @@ -54,7 +54,7 @@ Worry no more! STUNner allows you to deploy *any* WebRTC service into Kubernetes integrating it into the [cloud-native ecosystem](https://landscape.cncf.io). STUNner exposes a standards-compliant STUN/TURN gateway for clients to access your virtualized WebRTC infrastructure running in Kubernetes, maintaining full browser compatibility and requiring minimal or no -modification to your existing WebRTC codebase. STUNner implements the standard [Kubernetes Gateway +modification to your existing WebRTC codebase. STUNner supports the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) so you can configure it in the familiar YAML-engineering style via Kubernetes manifests. @@ -83,8 +83,8 @@ features we have come to expect from modern network services. Worse yet, the ent on a handful of [public](https://bloggeek.me/google-free-turn-server/) [STUN servers](https://www.npmjs.com/package/freeice) and [hosted TURN services](https://bloggeek.me/managed-webrtc-turn-speed) to connect clients behind a NAT/firewall, -which may create a useless dependency on externally operated services, introduce a bottleneck, -raise security concerns, and come with a non-trivial price tag. +which may create a useless dependency on externally operated services, introduce a performance +bottleneck, raise security concerns, and come with a non-trivial price tag. The main goal of STUNner is to allow *anyone* to deploy their own WebRTC infrastructure into Kubernetes, without relying on any external service other than the cloud-provider's standard hosted @@ -93,8 +93,8 @@ servers can use as a scalable NAT traversal facility (headless model), or it can for ingesting WebRTC media traffic into the Kubernetes cluster by exposing a public-facing STUN/TURN server that WebRTC clients can connect to (media-plane model). This makes it possible to deploy WebRTC application servers and media servers into ordinary Kubernetes pods, taking advantage -of Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC -infrastructure like any other cloud-bound workload. +of the full cloud native feature set to manage, scale, monitor and troubleshoot the WebRTC +infrastructure like any other Kubernetes workload. ![STUNner media-plane deployment architecture](./docs/img/stunner_arch.svg) @@ -146,9 +146,6 @@ way. potentially malicious access; with STUNner *all* media is received through a single ingress port that you can tightly monitor and control. - - - * **Simple code and extremely small size.** Written in pure Go using the battle-tested [pion/webrtc](https://github.com/pion/webrtc) framework, STUNner is just a couple of hundred lines of fully open-source code. The server is extremely lightweight: the typical STUNner @@ -177,15 +174,13 @@ minutes. The simplest way to deploy STUNner is through [Helm](https://helm.sh). STUNner configuration parameters are available for customization as [Helm -Values](https://helm.sh/docs/chart_template_guide/values_files). We recommend deploying STUNner -into a separate namespace and we usually name this namespace as `stunner`, so as to isolate it from -the rest of the workload. +Values](https://helm.sh/docs/chart_template_guide/values_files). ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system -helm install stunner stunner/stunner --create-namespace --namespace=stunner +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace \ + --namespace=stunner-system ``` Find out more about the charts in the [STUNner-helm repository](https://github.com/l7mp/stunner-helm). @@ -193,10 +188,15 @@ Find out more about the charts in the [STUNner-helm repository](https://github.c ### Configuration The standard way to interact with STUNner is via the standard Kubernetes [Gateway - API](https://gateway-api.sigs.k8s.io). This is much akin to the way you configure *all* - Kubernetes workloads: specify your intents in YAML files and issue a `kubectl apply`, and the - [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) will automatically - reconcile the STUNner dataplane for the new configuration. +API](https://gateway-api.sigs.k8s.io). This is much akin to the way you configure *all* Kubernetes +workloads: specify your intents in YAML files and issue a `kubectl apply`, and the [STUNner gateway +operator](https://github.com/l7mp/stunner-gateway-operator) will automatically create the STUNner +dataplane (that is, the `stunnerd` pods that implement the STUN/TURN service) and downloads the new +configuration to the dataplane pods. + +It is generally a good idea to maintain STUNner configuration into a separate Kubernetes +namespace. Below we will use the `stunner` namespace; create it with `kubectl create namespace +stunner` if it does not exist. 1. Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway API. This amounts to creating a @@ -212,7 +212,7 @@ The standard way to interact with STUNner is via the standard Kubernetes [Gatewa ``` console kubectl apply -f - < **Warning** -STUNner deviates somewhat from the standard rules Kubernetes uses to handle ports in Services. In +Note that STUNner deviates somewhat from the way Kubernetes handles ports in Services. In Kubernetes each Service is associated with one or more protocol-port pairs and connections via the Service can be made to only these specific protocol-port pairs. WebRTC media servers, however, usually open lots of different ports, typically one per each client connection, and it would be -cumbersome to create a separate backend Service and UDPRoute for each port. In order to simplify +cumbersome to create a separate backend Service and UDPRoute per each port. In order to simplify this, STUNner **ignores the protocol and port specified in the backend service** and allows connections to the backend pods via *any* protocol-port pair. STUNner can therefore use only a -*single* backend Service to reach any port exposed on a WebRTC media server. - - +*single* backend Service to reach any port exposed on a WebRTC media server. And that's all. You don't need to worry about client-side NAT traversal and WebRTC media routing because STUNner has you covered! Even better, every time you change a Gateway API resource in -Kubernetes, say, you update the GatewayConfig to reset your STUN/TURN credentials or change the -protocol or port in one of your Gateways, the [STUNner gateway +Kubernetes, say, you update the GatewayConfig to reset the STUN/TURN credentials or change the +protocol or port in a Gateway, the [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) will automatically pick up your modifications and update the underlying dataplane. Kubernetes is beautiful, isn't it? ### Check your config -The current STUNner dataplane configuration is always made available in a convenient ConfigMap -called `stunnerd-config` (you can choose the name in the GatewayConfig). The STUNner dataplane pods -themselves will use the very same ConfigMap to reconcile their internal state, so you can consider -the content to be the ground truth. +The current STUNner dataplane configuration is always made available in a convenient ConfigMap that +has the same name and namespace as the Gateway it belongs to (so this is supposed to be +`stunner/udp-gateway` as per our example). STUNner comes with a small utility to dump the running configuration in human readable format (you -must have [`jq`](https://stedolan.github.io/jq) installed in your PATH to be able to use it). Chdir -into the main STUNner directory and issue. +must have [`jq`](https://stedolan.github.io/jq) installed in your PATH to be able to use it). Issue +the below from the main STUNner directory. ```console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener: udp-listener -Protocol: TURN-UDP -Public address: 34.118.36.108 -Public port: 3478 +cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway +STUN/TURN authentication type: static +STUN/TURN username: user-1 +STUN/TURN password: pass-1 +Listener 1 + Name: stunner/udp-gateway/udp-listener + Listener: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address: 34.34.150.65 + Public port: 3478 ``` As it turns out, STUNner has successfully assigned a public IP and port to our Gateway and set the @@ -379,7 +371,7 @@ STUN/TURN credentials based on the GatewayConfig. You can use the below to dump configuration; `jq` is there just to pretty-print JSON. ```console -kubectl get cm -n stunner stunnerd-config -o jsonpath="{.data.stunnerd\.conf}" | jq . +kubectl get cm -n stunner udp-gateway -o jsonpath="{.data.stunnerd\.conf}" | jq . ``` ### Testing @@ -394,7 +386,7 @@ a heartwarming welcome message. The below manifest spawns the service in the `default` namespace and wraps it in a Kubernetes service called `media-plane`. Recall, this is the target service in our UDPRoute. Note that the type of the `media-plane` service is `ClusterIP`, which means that Kubernetes will *not* expose - it to the Internet: the only way for clients to obtain a response is via STUNner. + it to the outside world: the only way for clients to obtain a response is via STUNner. ```console kubectl apply -f deploy/manifests/udp-greeter.yaml @@ -413,16 +405,16 @@ a heartwarming welcome message. see a nice greeting from your cluster! ```console - ./turncat - k8s://stunner/stunnerd-config:udp-listener udp://${PEER_IP}:9001 + ./turncat - k8s://stunner/udp-gateway:udp-listener udp://${PEER_IP}:9001 Hello STUNner Greetings from STUNner! ``` -Observe that we haven't specified the public IP address and port: `turncat` is clever enough to -parse the running [STUNner configuration](#check-your-config) from Kubernetes directly. Just -specify the special STUNner URI `k8s://stunner/stunnerd-config:udp-listener`, identifying the -namespace (`stunner` here) and the name for the STUNner ConfigMap (`stunnerd-config`), plus the -listener to connect to (`udp-listener`), and `turncat` will do the heavy lifting. +Note that we haven't specified the public IP address and port: `turncat` is clever enough to parse +the running [STUNner configuration](#check-your-config) from Kubernetes directly. Just specify the +special STUNner URI `k8s://stunner/udp-gateway:udp-listener`, identifying the namespace (`stunner` +here) and the name for the Gateway (`udp-gateway`), and the listener to connect to +(`udp-listener`), and `turncat` will do the heavy lifting. Note that your actual WebRTC clients do *not* need to use `turncat` to reach the cluster: all modern Web browsers and WebRTC clients come with a STUN/TURN client built in. Here, `turncat` is @@ -478,7 +470,7 @@ greeter) by STUNner. ```console kubectl apply -f - < **Warning** - Clients should never query the STUNner authentication service directly to obtain an ICE - config. Instead, the WebRTC application server should retrieve the ICE config in the name of the - client during session establishment and return the generated ICE config to the client. + The ICE configs generated by the [STUNner authentication service](https://github.com/l7mp/stunner-auth-service) are always up to date with the most recent dataplane configuration. This makes sure that whenever you modify the STUNner Gateway API configuration (say, switch from `static` authentication to `ephemeral`), your clients will always receive an ICE config that reflects these changes (that is, the username/password pair will provide a time-windowed ephemeral credential). - The ICE configs generated by the [STUNner authentication - service](https://github.com/l7mp/stunner-auth-service) are always up to date with the most - recent dataplane configuration. This makes sure that whenever you modify the STUNner Gateway API - configuration (say, switch from `static` authentication to `ephemeral`), your clients will - always receive an ICE config that reflects these changes (that is, the username/password pair - will provide a time-windowed ephemeral credential). - - For instance, the below will query the STUnner auth service, which is by default available at - the URL `http://stunner-auth.stunner-system:8088`, for a valid ICE config. + Below is a query to the STUnner auth service, by default available at the URL `http://stunner-auth.stunner-system:8088`, that returns a valid ICE config. ```console curl "http://stunner-auth.stunner-system:8088/ice?service=turn" @@ -65,24 +38,15 @@ The intended authentication workflow in STUNner is as follows. } ``` - Use the below to generate a valid STUNner credential for a user called `my-user` with a lifetime - of one hour (`ttl`, only makes sense when STUNner uses `ephemeral` authentication - credentials). In addition, we select the Gateway called `my-gateway` deployed into the - `my-namespace` namespace on which we intend to receive WebRTC media from the user: + Use the below query to generate a valid STUNner credential to access the Gateway called `my-gateway` deployed into the `my-namespace` namespace: ```console curl "http://stunner-auth.stunner-system:8088/ice?service=turn?ttl=3600&username=my-user&namespace=my-namespace&gateway=my-gateway" ``` -2. The clients *receive the ICE configuration* (usually, from the application server) over a secure - channel. This is outside the context of STUNner; our advice is to return the ICE configuration - during the session setup process, say, along with the initial configuration returned for clients - before starting the call. +2. The clients *receive the ICE configuration* (usually, from the application server) over a secure channel. This is outside the context of STUNner. Our advice is to return the ICE configuration during the session setup process, say, along with the initial configuration returned for clients before starting the call. -3. WebRTC clients are *configured with the ICE configuration* obtained above. The below snippet - shows how to initialize a WebRTC - [`PeerConnection`](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection) - to use the above ICE server configuration in order to use STUNner as the default TURN service. +3. WebRTC clients are *configured with the ICE configuration*. The below snippet shows how to initialize a WebRTC [`PeerConnection`](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection) to use the above ICE server configuration in order to use STUNner as the default TURN service. ``` var iceConfig = @@ -91,34 +55,21 @@ The intended authentication workflow in STUNner is as follows. ## Static authentication -In STUNner, `static` authentication is the simplest and least secure authentication mode, basically -corresponding to a traditional "log-in" username and password pair given to users. - -> **Note** -STUNner accepts (and sometimes reports) the alias `plaintext` to mean the `static` authentication mode; the use of `plaintext` is deprecated and will be removed in a later release. +In STUNner, `static` authentication is the simplest and least secure authentication mode, basically corresponding to a traditional "log-in" username and password pair given to users. -When STUNner is configured to use `static` authentication only a single username/password pair is -used for *all* clients. This makes configuration easy; e.g., the ICE server configuration can be -hardcoded into the static Javascript code served to clients. At the same time, `static` -authentication is prone to leaking the credentials: once an attacker learns a username/password -pair they can use it without limits to reach STUNner (until the administrator rolls the -credentials, see below). +When STUNner is configured to use `static` authentication only a single username/password pair is used for *all* clients. This makes configuration easy; e.g., the ICE server configuration can be hardcoded into the static Javascript code served to clients. At the same time, `static` authentication is prone to leaking credentials: once an attacker learns a username/password pair they can use it without limits to reach STUNner (until the administrator rolls the credentials, see below). -The first step of configuring STUNner for the `static` authentication mode is to create a -Kubernetes Secret to hold the username/password pair. The below will set the username to `my-user` -and the password to `my-password`. If no `type` is set then STUNner defaults to `static` -authentication. +The first step of configuring STUNner for the `static` authentication mode is to create a Kubernetes Secret to hold the username/password pair. The below will set the username to `my-user` and the password to `my-password`. If no `type` is set then STUNner defaults to `static` authentication. ```console kubectl -n stunner create secret generic stunner-auth-secret --from-literal=type=static \ --from-literal=username=my-user --from-literal=password=my-password ``` -Then, we create or update the current [GatewayConfig](REFERENCE.md) to refer STUNner to this secret -for setting the authentication credentials. +Then, we update the [GatewayConfig](REFERENCE.md) to refer STUNner to this Secret for setting authentication credentials. ```yaml -apiVersion: stunner.l7mp.io/v1alpha1 +apiVersion: stunner.l7mp.io/v1 kind: GatewayConfig metadata: name: stunner-gatewayconfig @@ -130,54 +81,28 @@ spec: namespace: stunner ``` -The main use of static authentication is for testing. The reason for this is that static -authentication credentials are easily discoverable: since the WebRTC Javascript API uses the TURN -credentials unencrypted, an attacker can easily extract the STUNner credentials from the -client-side Javascript code. In order to mitigate the risk, it is a good security practice to reset -the username/password pair every once in a while. This can be done by simply updating the Secret -that holds the credentials. +It is a good security practice to reset the username/password pair every once in a while. This can be done by simply updating the Secret that holds the credentials. ```yaml kubectl -n stunner edit secret stunner-auth-secret ``` -> **Warning** -Modifying STUNner's credentials goes *without* restarting the TURN server but may affect existing -sessions, in that active sessions will not be able to refresh the TURN allocation established with -the old credentials. +> [!WARNING] +> +> Modifying STUNner's credentials goes *without* restarting the TURN server but may affect existing sessions, in that active sessions will not be able to refresh their TURN allocation any more. This will result in the disconnection of clients using the old credentials. ## Ephemeral authentication -For production use, STUNner provides the `ephemeral` authentication mode that uses per-client -time-limited STUN/TURN authentication credentials. Ephemeral credentials are dynamically generated -with a pre-configured lifetime and, once the lifetime expires, the credential cannot be used to -authenticate (or refresh) with STUNner any more. This authentication mode is more secure since -credentials are not shared between clients and come with a limited lifetime. Configuring -`ephemeral` authentication may be more complex though, since credentials must be dynamically -generated for each session and properly returned to clients. - -> **Note** -STUNner accepts (and sometimes reports) the alias `longterm` to mean the `ephemeral` authentication -mode; the use of `longterm` is deprecated and will be removed in a later release. The alias -`timewindowed` is also accepted. - -To implement this mode, STUNner adopts the [quasi-standard time-windowed TURN authentication -credential format](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00). In this -format, the TURN username consists of a colon-delimited combination of the expiration timestamp and -the user-id parameter, where the user-id is some application-specific id that is opaque to STUNner -and the timestamp specifies the date of expiry of the credential as a UNIX timestamp. The TURN -password is computed from the a secret key shared with the TURN server and the returned username -value, by performing `base64(HMAC-SHA1(secret key, username))`. STUNner extends this scheme -somewhat for maximizing interoperability with WebRTC apps, in that it allows the user-id and the -timestamp to appear in any order in the TURN username and it accepts usernames with a plain -timestamp, without the colon and/or the user-id. +STUNner provides the `ephemeral` authentication mode for production use, which uses per-client time-limited STUN/TURN authentication credentials. Ephemeral credentials are dynamically generated with a pre-configured lifetime and, once the lifetime expires, the credential cannot be used to authenticate (or refresh) with STUNner any more. This authentication mode is more secure since credentials are not shared between clients and come with a limited lifetime. Configuring `ephemeral` authentication may be more complex though, since credentials must be dynamically generated for each session and properly returned to clients. + +STUNner adopts the [quasi-standard time-windowed TURN authentication credential format](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00) for ephemeral authentication. The TURN username consists of a colon-delimited combination of the expiration timestamp and the user-id parameter, where the user-id is some application-specific id that is opaque to STUNner and the timestamp specifies the date of expiry of the credential as a UNIX timestamp. The TURN password is computed from the a secret key shared with the TURN server and the returned username value, by performing `base64(HMAC-SHA1(secret key, username))`. STUNner extends this scheme somewhat for maximizing interoperability with WebRTC apps, in that it allows the user-id and the timestamp to appear in any order in the TURN username and it accepts usernames with a plain timestamp, without the colon and/or the user-id. The advantage of this mechanism is that it is enough to know the shared secret for STUNner to be able to check the validity of a credential. -> **Warning** -The user-id is to ensure that the password generated per user-id is unique, but STUNner in no way -checks whether it identifies a valid user-id in the system. +> [!WARNING] +> +> The user-id is to ensure that the password generated per user-id is unique, but STUNner in no way checks whether it identifies a valid user-id in the system. In order to switch from `static` mode to `ephemeral` authentication, it is enough to update the Secret that holds the credentials. The below will set the shared secret `my-shared-secret` for the diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md index 7fb4f26f..87ed660e 100644 --- a/docs/CONCEPTS.md +++ b/docs/CONCEPTS.md @@ -1,27 +1,29 @@ # Concepts -In this guide we describe STUNner's architecture and the most important components of an operational STUNner installation. +This guide describes STUNner's architecture and the most important components of an operational installation. ## Architecture -A STUNner installation consists of two parts, a *control plane* and a *dataplane*. The control plane consists of declarative policies specifying the way STUNner should route WebRTC media traffic to the media servers, plus a gateway operator that renders the high-level policies into an actual dataplane configuration. The dataplane in turn comprises one or more `stunnerd` pods, responsible for actually ingesting media traffic into the cluster through a STUN/TURN server. +A STUNner installation consists of two parts, a *control plane* and a *data plane*. The control plane consists of declarative policies specifying the way STUNner should route WebRTC media traffic to the media servers, plus a gateway operator that renders the high-level policies into an actual dataplane configuration. The data plane in turn comprises one or more `stunnerd` pods, which are responsible for actually ingesting media traffic into the cluster. The dataplane pods are automatically provisioned by the gateway operator so they should come and go as you add and remove STUNner gateways. ![STUNner architecture](img/stunner_arch_big.svg) -The unit of the STUNner configuration is a [designated Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces) that holds the control plane configuration and the dataplane pods. You can run multiple STUNner deployments side-by-side by installing a separate dataplane into a each namespace and defining a distinct gateway hierarchy to configure each dataplane separately. - -### Control plane +## Control plane The STUNner control plane consists of the following components: -* **Gateway hierarchy:** A gateway hierarchy is a collection of [Kubernetes Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources) that together describe the way media traffic should enter the cluster, including public IP addresses and ports clients can use to reach STUNner, TURN credentials, forwarding rules, etc. The anchor of the gateway hierarchy is the GatewayClass object, and the rest of the resources form a complete hierarchy underneath it: the GatewayConfig describes general STUNner configuration, Gateways define the port and transport protocol for each TURN server listener, and UDPRoutes point to the backend services client traffic should be forwarded to. See [here](GATEWAY.md) for a full reference. +* **Gateway API resources:** The high-level STUNner configuration is a collection of [Gateway API](https://gateway-api.sigs.k8s.io) resources that together describe the way media traffic should enter the cluster. The anchor of the configuration hierarchy is the GatewayClass object, and the rest of the resources form a complete hierarchy underneath it: the GatewayConfig describes general STUNner configuration, Gateways define the port and transport protocol per each TURN server listener, and UDPRoutes point to the backend services client traffic should be forwarded to. See [here](GATEWAY.md) for a full reference. -* **Gateway operator:** The main purpose of the gateway operator is to watch gateway hierarchies for change and, once a custom resource is added or modified by the user, render a new dataplane configuration. This configuration is then mapped into the filesystem of the `stunnerd` pods running in the same namespace, so that each `stunnerd` instance will use the most recent configuration. The STUNner Helm chart [automatically installs](INSTALL.md) the gateway operator; more information can be found [here](https://github.com/l7mp/stunner-gateway-operator). +* **Gateway operator:** The main purpose of the gateway operator is to watch Gateway API resources and, once a Gateway API resource is added or modified by the user, update the dataplane accordingly (see below). -* **STUNner ConfigMap:** The STUNner ConfigMap contains the running dataplane configuration. Of course, we could let the `stunnerd` pods themselves to watch the control plane for changes, but this would run into scalability limitations for large deployments. Instead, we separate the control plane and the dataplane, which brings cool [benefits](https://en.wikipedia.org/wiki/Software-defined_networking). The STUNner ConfigMap is usually named as `stunnerd-config`, but you can override this from the GatewayConfig. +* **STUNner authentication service** (not shown on the figure): The auth service is an ancillary service that can be used to generate TURN credentials and complete [ICE server configurations](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#iceservers) to bootstrap clients. See more info [here](AUTH.md). ## Dataplane -The STUNner dataplane is comprised of a fleet of `stunnerd` pods. These pods actually implement the TURN server, using the configuration available in the STUNner ConfigMap which is mapped into the pods' filesystem dynamically. Then, `stunnerd` will watch for changes in the config file and, once a change is detected, it [reconciles](https://kubernetes.io/docs/concepts/architecture/controller) the dataplane to match the new user policies. +The STUNner dataplane is comprised of a fleet of `stunnerd` pods implementing the TURN servers that can be used by clients to create WebRTC connections, plus some additional configuration to expose the TURN services to clients. The complete dataplane configuration per each Gateway is as follows: + +* **`stunnerd` Deployment:** Once you create a new Gateway the gateway operator will spawn a new dataplane for the Gateway automatically. For each Gateway there will be `stunnerd` Deployment with the same name and namespace. The `stunnerd` daemon itself is a TURN server implemented on top of the [pion/turn](https://github.com/pion/turn) Go WebRTC framework. The daemon will instantiate a separate *TURN listener* for each Gateway listener in the gateway configuration to terminate clients' TURN sessions, a *cluster* per each UDPRoute to forward packets to the backend services (e.g., to the media servers), with some ancillary administrative and authentication mechanisms in place to check client credentials, logging, etc. Whenever you modify a Gateway (UDPRoute), the gateway operator renders a new dataplane configuration with the modified listener (cluster, respectively) specs and downloads it to the `stunnerd` pods, which in turn reconcile their internal state with respect the new configuration. You are free to scale the dataplane to as many `stunnerd` pods as you wish: Kubernetes will make sure that new client connections are distributed evenly over the scaled-out STUNner dataplane. + +* **LoadBalancer Service:** STUNner creates a separate LoadBalancer Service per each Gateway to expose the TURN listeners of the `stunnerd` pods to the outside world. Similarly to the case of the `stunnerd` Deployment, there will be a separate LoadBalancer Service per each Gateway with the same name and namespace. -The `stunnerd` daemon itself is essentially a simple TURN server on top of [pion/turn](https://github.com/pion/turn) written in Go. The daemon will instantiate a separate *TURN listener* for each Gateway listener in the gateway hierarchy to terminate clients' TURN sessions, a *cluster* per each UDPRoute to forward packets to the backend services (e.g., to the media servers), with some ancillary administrative and authentication mechanisms in place to check client credentials before admitting traffic into the cluster, logging, etc. There is a one-to-one mapping between the control-plane Gateway listeners and the `stunnerd` TURN listeners, as well as between the UDPRoute resources and `stunnerd`'s clusters. Whenever you modify a Gateway (UDPRoute), the gateway operator renders a new dataplane configuration with the modified listener (cluster, respectively) specs and the `stunnerd` pods reconcile their internal state to the new configuration. You are free to scale the dataplane to as many `stunnerd` pods as you like: Kubernetes will make sure that new client connections are distributed evenly over the scaled-out STUNner dataplane. +* **STUNner ConfigMap**: In order to simplify troubleshooting a STUNner setup, the dataplane configuration of each Gateway is always made available in a ConfigMap for human inspection. Again, the name and namespace of the ConfigMap is the same as those of the corresponding Gateway. Note that this ConfigMap is no longer used by the dataplane for reconciliation, it is there only fo debugging purposes and may be silently removed in a later release. You can use STUNner's own [Config Discovery Service client](https://pkg.go.dev/github.com/l7mp/stunner@v0.16.2/pkg/config/client) to obtain dataplane configuration right from the gateway operator instead. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 3e0dd317..24da4338 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -6,10 +6,9 @@ can act either as a simple headless STUN/TURN server or a fully fledged ingress an entire Kubernetes-based media server pool. Second, when STUNner is configured as an ingress gateway then there are multiple [ICE models](#ice-models), based on whether only the client connects via STUNner or both clients and media servers use STUNner to set up the media-plane -connection. Third, STUNner can run in one of several [control plane models](#control-plane-models), -based on whether the user manually supplies STUNner configuration or there is a separate STUNner -control plane that automatically reconciles the dataplane based on a high-level [declarative -API](https://gateway-api.sigs.k8s.io). +connection. Third, STUNner can run in one of several [data plane models](#data-plane-models), based +on whether the dataplane is automatically provisioned or the user has to manually supply the +dataplane pods for STUNner. ## Architectural models @@ -26,11 +25,11 @@ this case the STUN/TURN servers are deployed into Kubernetes. ![STUNner headless deployment architecture](img/stunner_standalone_arch.svg) -> **Warning** -For STUNner to be able to connect WebRTC clients and servers in the headless model *all* the -clients and servers *must* use STUNner as the TURN server. This is because STUNner opens the -transport relay connections *inside* the cluster, on a private IP address, and this address is -reachable only to STUNner itself, but not for external STUN/TURN servers. + + + + + ### Media-plane deployment model @@ -52,12 +51,18 @@ for clients' UDP transport streams then STUNner can be scaled freely, otherwise result the [disconnection of a small number of client connections](https://cilium.io/blog/2020/11/10/cilium-19/#maglev). -#### Asymmetric ICE mode +## ICE models -The standard mode to supply an ICE server configuration for clients and media servers in the -media-plane deployment model of STUNner is the *asymmetric ICE mode*. In this model the client is -configured with STUNner as the TURN server and media servers run with no STUN or TURN servers -whatsoever. +The peers willing to create a connection via STUNner (e.g., two clients as per the headless model, +or a client and a media server in the media-plane deployment model) need to decide how to create +ICE candidates. + +### Asymmetric ICE mode + +In *asymmetric ICE mode*, one peer is configured with STUNner as the TURN server and the other peer +runs with no STUN or TURN servers whatsoever. The first peer will create a TURN transport relay +connection via STUNner to which the other peer can directly join. Asymmetric ICE mode is the +recommended way for the media-plane deployment model. ![STUNner asymmetric ICE mode](img/stunner_asymmetric_ice.svg) @@ -71,37 +76,34 @@ connection. In contrast, servers run without any STUN/TURN server whatsoever, so only. Due to servers being deployed into ordinary Kubernetes pods, the server's host candidate will likewise contain a private pod IP address. Then, since in the Kubernetes networking model ["pods can communicate with all other pods on any other node without a -NAT"](https://kubernetes.io/docs/concepts/services-networking), clients' relay candidates and the -servers' host candidates will have direct connectivity in the Kubernetes private container network -and the ICE connectivity check will succeed. See more explanation +NAT"](https://kubernetes.io/docs/concepts/services-networking), the client's relay candidate and +the server's host candidate will have direct connectivity in the Kubernetes private container +network and the ICE connectivity check will succeed. See more explanation [here](examples/kurento-one2one-call/README.md#what-is-going-on-here). -> **Warning** -Refrain from configuring additional public STUN/TURN servers, apart from STUNner itself. The rules -to follow in setting the [ICE server +Refrain from configuring additional public STUN/TURN servers apart from STUNner itself. The rules +to follow for setting the [ICE server configuration](https://github.com/l7mp/stunner#configuring-webrtc-clients) in asymmetric ICE mode are as below: -> - on the client, set STUNner as the *only* TURN server and configure *no* STUN servers, whereas -> - on the server do *not* configure *any* STUN or TURN servers whatsoever. - -Most users will want to deploy STUNner using the asymmetric ICE mode. In the rest of the docs we -assume the asymmetric ICE mode with the media plane deployment model, unless noted otherwise. +- on the client, set STUNner as the *only* TURN server and configure *no* STUN servers, and +- on the server do *not* configure *any* STUN or TURN server whatsoever. -> **Warning** -Deviating from the above rules *might* work in certain cases, but may have uncanny and -hard-to-debug side-effects. For instance, configuring clients and servers with public STUN servers -in certain unlucky situations may allow them to connect via server-reflexive ICE candidates, -completely circumventing STUNner. This is on the one hand extremely fragile and, on the other hand, -a security vulnerability; remember, STUNner should be the *only* external access point to your -media plane. It is a good advice to set the `iceTransportPolicy` to `relay` on the clients to avoid -side-effects: this will prevent clients from generating host and server-reflexive ICE candidates, -leaving STUNner as the only option to obtain an ICE candidate from. +Deviating from these rules *might* work in certain cases, but may have uncanny and hard-to-debug +side-effects. For instance, configuring clients and servers with public STUN servers in certain +unlucky situations may allow them to connect via server-reflexive ICE candidates, completely +circumventing STUNner. This is on the one hand extremely fragile and, on the other hand, a security +vulnerability; remember, STUNner should be the *only* external access point to your media plane. It +is a good advice to set the `iceTransportPolicy` to `relay` on the clients to avoid side-effects: +this will prevent clients from generating host and server-reflexive ICE candidates, leaving STUNner +as the only option to obtain an ICE candidate from. -#### Symmetric ICE mode +### Symmetric ICE mode In the symmetric ICE mode both the client and the server obtain an ICE [relay candidate](https://developer.mozilla.org/en-US/docs/Web/API/RTCIceCandidate/type) from STUNner and -the connection occurs directly via STUNner. +the connection occurs directly via STUNner. This is the simplest mode for the headless deployment +model, but symmetric mode can also be used for the media-plane model as well to connect clients to +media servers. ![STUNner symmetric ICE mode](img/stunner_symmetric_ice.svg) @@ -118,7 +120,7 @@ priorities](https://www.ietf.org/rfc/rfc5245.txt) to different connection types) is a good practice to configure the STUNner TURN URI in the server-side ICE server configuration with the *internal* IP address and port used by STUNner (i.e., the ClusterIP of the `stunner` Kubernetes service and the corresponding port), otherwise the server might connect via the external -LoadBalancer IP causing an unnecessary roundtrip. +LoadBalancer IP causing an unnecessary roundtrip (hairpinning). The symmetric mode means more overhead compared to the asymmetric mode, since STUNner now performs TURN encapsulation/decapsulation for both sides. However, the symmetric mode comes with certain @@ -127,20 +129,10 @@ internal IP addresses in the ICE candidates from attackers; note that this is no but feel free to open an issue if [exposing internal IP addresses](SECURITY.md) is blocking you from adopting STUNner. -## Control plane models - -STUNner can run in one of several modes. - -In the default mode STUNner configuration is controlled by a *gateway-operator* component based on -high-level intent encoded in [Kubernetes Gateway API resources](https://gateway-api.sigs.k8s.io), -while in the *standalone model* the user configures STUNner manually. The standalone mode provides -perfect control over the way STUNner ingests media, but at the same time it requires users to deal -with the subtleties of internal STUNner APIs that are subject to change between subsequent -releases. As of v0.16, STUNner's operator-ful mode is feature complete and the standalone model is -considered obsolete. If still interested, comprehensive documentation for the standalone can be -found [here](OBSOLETE.md), but this mode is no longer supported. +## Data plane models -In addition, STUNner supports two dataplane provisioning modes. In the *legacy* mode the dataplane -is supposed to be deployed by the user manually (by installing the `stunner/stunner` Helm chart -into the target namespaces) while in the *managed* mode the dataplane pods are provisioned by the -gateway operator automatically. As of STUNner v0.16.0, the default is the *legacy* dataplane mode. +STUNner supports two dataplane provisioning modes. In the default *managed* mode, the dataplane +pods (i.e., the `stunnerd` pods) are provisioned automatically per each Gateway existing in the +cluster. In the *legacy* mode, the dataplane is supposed to be deployed by the user manually by +installing the `stunner/stunner` Helm chart into the target namespaces. Legacy mode is considered +obsolete at this point and it will be removed in a later release. diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 16742336..295ef4f1 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -1,23 +1,15 @@ # Reference -The [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) exposes the control plane configuration using the standard [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io). This allows to configure STUNner in the familiar YAML-engineering style via Kubernetes manifests. The below reference gives a quick overview of the Gateway API. Note that STUNner implements only a subset of the full [spec](GATEWAY.md), see [here](https://github.com/l7mp/stunner-gateway-operator#caveats) for a list of the most important simplifications. - -## Overview - -The main unit of the control plane configuration is the *gateway hierarchy*. Here, a Gateway hierarchy is a collection of [Kubernetes Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources) that together describe the way media traffic should enter the cluster via STUNner, including public IP addresses and ports clients can use to reach STUNner, TURN credentials, routing rules, etc. The anchor of the gateway hierarchy is the GatewayClass object, and the rest of the resources form a complete hierarchy underneath it. - -![Gateway hierarchy](img/gateway_api.svg) - -In general, the scope of a gateway hierarchy is a single namespace, but this is not strictly enforced: e.g., the GatewayClass is [cluster-scoped](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions) so it is outside the namespace, GatewayClasses can refer to GatewayConfigs across namespaces, Routes can attach to Gateways across a namespace boundary (if the Gateway [allows](https://gateway-api.sigs.k8s.io/guides/multiple-ns) this), etc. Still, it is a good practice to keep all control plane configuration, plus the actual dataplane pods, in a single namespace as much as possible. +The [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) exposes the control plane configuration using the standard [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io). This allows to configure STUNner in the familiar YAML-engineering style via Kubernetes manifests. The below reference gives an overview of the subset of the Gateway API supported by STUNner, see [here](https://github.com/l7mp/stunner-gateway-operator#caveats) for a list of the most important simplifications. ## GatewayClass -The GatewayClass resource provides the root of the gateway hierarchy. GatewayClass resources are cluster-scoped, so they can be attached to from any namespace, and we usually assume that each namespaced gateway hierarchy will have a separate global GatewayClass as the anchor. +The GatewayClass resource provides the root of a STUNner gateway configuration. GatewayClass resources are cluster-scoped, so they can be attached to from any namespace. -Below is a sample GatewayClass resource. Each GatewayClass must specify a controller that will manage the Gateway objects created under the hierarchy; this must be set to `stunner.l7mp.io/gateway-operator` for the STUNner gateway operator to pick up the GatewayClass. In addition, a GatewayClass can refer to further implementation-specific configuration via a `parametersRef`; in the case of STUNner this will always be a GatewayConfig object (see [below](#gatewayconfig)). +Below is a sample GatewayClass resource. Each GatewayClass specifies a controller that will manage the Gateway objects created under the class; this must be set to `stunner.l7mp.io/gateway-operator` for the STUNner gateway operator to pick up the GatewayClass. In addition, a GatewayClass can refer to further implementation-specific configuration via a `parametersRef`; in the case of STUNner this will always be a GatewayConfig object (see [below](#gatewayconfig)). ```yaml -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -41,12 +33,12 @@ Below is a quick reference of the most important fields of the GatewayClass [`sp ## GatewayConfig -The GatewayConfig resource provides general configuration for STUNner, most importantly the STUN/TURN authentication [credentials](AUTH.md) clients can use to connect to STUNner. GatewayClass resources attach a STUNner configuration to the hierarchy by specifying a particular GatewayConfig in the GatewayClass `parametersRef`. GatewayConfig resources are namespaced, and every hierarchy can contain at most one GatewayConfig. Failing to specify a GatewayConfig is an error because the authentication credentials cannot be learned by the dataplane otherwise. +The GatewayConfig resource provides general configuration for STUNner, most importantly the STUN/TURN authentication [credentials](AUTH.md) clients can use to connect to STUNner. GatewayClass resources attach a STUNner configuration to the hierarchy by specifying a particular GatewayConfig in the GatewayClass `parametersRef`. GatewayConfig resources are namespaced, and every hierarchy can contain at most one GatewayConfig. Failing to specify a GatewayConfig is an error because the authentication credentials cannot be learned otherwise. -The following example takes the [STUNner authentication settings](AUTH.md) from the Secret called `stunner-auth-secret` in the `stunner` namespace, sets the authentication realm to `stunner.l7mp.io`, sets the dataplane loglevel to `all:DEBUG,turn:INFO` (this will set all loggers to `DEBUG` level except the TURN protocol machinery's logger which is set to `INFO`), and sets the default URL for metric scraping. +The following example takes the [STUNner authentication settings](AUTH.md) from the Secret called `stunner-auth-secret` in the `stunner` namespace, sets the authentication realm to `stunner.l7mp.io`, and sets the dataplane loglevel to `all:DEBUG,turn:INFO` (this will set all loggers to `DEBUG` level except the TURN protocol machinery's logger which is set to `INFO`). ```yaml -apiVersion: stunner.l7mp.io/v1alpha1 +apiVersion: stunner.l7mp.io/v1 kind: GatewayConfig metadata: name: stunner-gatewayconfig @@ -54,42 +46,38 @@ metadata: spec: logLevel: "all:DEBUG,turn:INFO" realm: stunner.l7mp.io - authRef: + authRef: name: stunner-auth-secret namespace: stunner - metricsEndpoint: "http://0.0.0.0:8080/metrics" ``` -Below is a quick reference of the most important fields of the GatewayConfig [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects) +Below is a reference of the most important fields of the GatewayConfig [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects) | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | -| `stunnerConfig` | `string` | The name of the ConfigMap into which the operator renders the `stunnerd` running configuration. Default: `stunnerd-config`. | No | -| `logLevel` | `string` | Logging level for the dataplane daemon pods (`stunnerd`). Default: `all:INFO`. | No | +| `dataplane` | `string` | The name of the Dataplane template to use for provisioning `stunnerd` pods. Default: `default`. | No | +| `logLevel` | `string` | Logging level for the dataplane pods. Default: `all:INFO`. | No | | `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | | `authRef` | `reference` | Reference to a Secret (`namespace` and `name`) that defines the STUN/TURN authentication mechanism and the credentials. | No | | `authType` | `string` | Type of the STUN/TURN authentication mechanism. Valid only if `authRef` is not set. Default: `static`. | No | | `username` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | | `password` | `string` | The password for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | | `sharedSecret` | `string` | The shared secret for [`ephemeral` authentication](AUTH.md). Valid only if `authRef` is not set. | No | -| `metricsEndpoint` | `string` | The metrics server (Prometheus) endpoint URL for the `stunnerd` pods.| No | -| `healthCheckEndpoint` | `string` | HTTP health-check endpoint exposed by `stunnerd`. Liveness check will be available on path `/live` and readiness check on path `/ready`. Default is to enable health-checking on `http://0.0.0.0:8086/ready` and `http://0.0.0.0:8086/live`, use an empty string to disable.| No | | `authLifetime` | `int` | The lifetime of [`ephemeral` authentication](AUTH.md) credentials in seconds. Not used by STUNner.| No | -| `loadBalancerServiceAnnotations` | `map[string]string` | A list of annotations that will go into the LoadBalancer services created automatically by STUNner to obtain a public IP addresses. See more detail [here](https://github.com/l7mp/stunner/issues/32). | No | +| `loadBalancerServiceAnnotations` | `map[string]string` | A list of annotations that will go into the LoadBalancer services created automatically by STUNner to obtain a public IP address. See more detail [here](https://github.com/l7mp/stunner/issues/32). | No | -> **Warning** At least a valid username/password pair *must* be supplied for `static` authentication, or a `sharedSecret` for the `ephemeral` mode, either via an external Secret or inline in the GatewayConfig. External authentication settings override inline settings. Missing both is an error. -Except the TURN authentication realm, all GatewayConfig resources are safe for modification. That is, the `stunnerd` daemons know how to reconcile a change in the GatewayConfig without restarting listeners/TURN servers. Changing the realm, however, induces a *full* TURN server restart (see below). +Except the TURN authentication realm, all GatewayConfig resources are safe for modification. That is, the `stunnerd` daemons know how to reconcile a change in the GatewayConfig without restarting listeners/TURN servers. Changing the realm, however, induces a *full* dataplane restart. ## Gateway Gateways describe the STUN/TURN server listeners exposed to clients. -The below Gateway will configure STUNner to open a STUN/TURN listener over the UDP port 3478 and automatically expose it on a public IP address and port by creating a [LoadBalancer service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). The name and namespace of the automatically provisioned service are the same as those of the Gateway, and the service is automatically updated if the Gateway changes (e.g., a port changes). +The below Gateway resource will configure STUNner to open a STUN/TURN listener over the UDP port 3478 and make it available on a public IP address and port to clients. Each Gateway will have a `stunnerd` Deployment that will run the dataplane, a LoadBalancer Service that will expose the dataplane to the Internet, and an ancillary ConfigMap that will hold the corresponding configuration, each using the same name and namespace as the Gateway. Once the Gateway is removed, the corresponding resources are automatically garbage-collected. ```yaml -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway @@ -102,10 +90,10 @@ spec: protocol: TURN-UDP ``` -The below more complex example defines two TURN listeners: a TURN listener at the UDP:3478 port that accepts routes from any namespace, and a TURN listener at port TLS/TCP:443 that accepts routes from all namespaces labeled as `app:dev`. +The below example defines two TURN listeners: a TURN listener at the UDP:3478 port that accepts routes from any namespace (see below), and a TURN listener at port TLS/TCP:443 that accepts routes only from namespaces labeled with `app=dev`. ```yaml -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: complex-gateway @@ -142,7 +130,7 @@ spec: app: dev ``` -Below is a quick reference of the most important fields of the Gateway [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects). +Below is a reference of the most important fields of the Gateway [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects). | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | @@ -150,21 +138,35 @@ Below is a quick reference of the most important fields of the Gateway [`spec`]( | `listeners` | `list` | The list of TURN listeners. | Yes | | `addresses` | `list` | The list of manually hinted external IP addresses for the rendered service (only the first one is used). | No | -Each TURN `listener` is defined by a unique name, a transport protocol and a port. In addition, a -`tls` configuration is required for TURN-TLS and TURN-DTLS listeners. +> [!WARNING] +> +> Gateway resources are *not* safe for modification. This means that certain changes to a Gateway will restart the underlying TURN server listener, causing all active client sessions to terminate. The particular rules are as follows: +> - adding or removing a listener will start/stop *only* the TURN listener being created/removed, without affecting the rest of the listeners on the same Gateway; +> - changing the transport protocol, port or TLS keys/certs of an *existing* listener will restart the TURN listener but leave the rest of the listeners intact; +> - changing the TURN authentication realm will restart *all* TURN listeners. + +### Listener configuration + +Each TURN `listener` is defined by a unique name, a transport protocol and a port. In addition, a `tls` configuration is required for TURN-TLS and TURN-DTLS listeners. Per-listener configuration is as follows. | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | -| `name` | `string` | Name of the TURN listener. | Yes | +| `name` | `string` | Name of the TURN listener. Must be unique per Gateway. | Yes | | `port` | `int` | Network port for the TURN listener. | Yes | | `protocol` | `string` | Transport protocol for the TURN listener. Either TURN-UDP, TURN-TCP, TURN-TLS or TURN-DTLS. | Yes | | `tls` | `object` | [TLS configuration](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig).| Yes (for TURN-TLS/TURN-DTLS) | -| `allowedRoutes.from` | `object` | [Route attachment policy](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.AllowedRoutes), either `All`, `Selector`, or `Same` (default is `Same`) | No | +| `allowedRoutes.from` | `object` | [Route attachment policy](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.AllowedRoutes), either `All`, `Selector`, or `Same`. Default: `Same`. | No | For TURN-TLS/TURN-DTLS listeners, `tls.mode` must be set to `Terminate` or omitted (`Passthrough` does not make sense for TURN), and `tls.certificateRefs` must be a [reference to a Kubernetes Secret](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig) of type `tls` or `opaque` with exactly two keys: `tls.crt` must hold the TLS PEM certificate and `tls.key` must hold the TLS PEM key. +### Load balancer configuration + STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN over UDP and TURN over DTLS listeners are exposed as UDP services, TURN-TCP and TURN-TLS listeners are exposed as TCP. +STUNner implements two ways to customize the automatically created Service, both involving certain per-defined [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) added to the Service. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). + +By default, each key-value pair set in the GatewayConfig `loadBalancerServiceAnnotations` field will be copied verbatim into the Service. Service annotations can be customized on a per-Gateway basis as well, by adding the corresponding annotations to a Gateway resource. STUNner copies all annotations from the Gateway into the Service, overwriting the annotations specified in the GatewayConfig on conflict. + Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. | Field | Type | Description | Required | @@ -172,13 +174,14 @@ Manually hinted external address describes an address that can be bound to a Gat | `type` | `string` | Type of the address. Currently only `IPAddress` is supported. | Yes | | `value` | `string` | Address that should be bound to the Gateway's service. | Yes | -> **Warning** -Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if the cloud-provider or your Kubernetes install do not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. +> [!WARNING] +> +> Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if the cloud-provider or your Kubernetes install do not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. -Mixed multi-protocol Gateways are supported: this means if you want to expose a UDP and a TCP port on the same LoadBalancer service you can do it with a single Gateway. The below Gateway will expose both ports with their respective protocols. +[Mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) are supported: this means if you want to expose a UDP and a TCP port on the same IP you can do it with a single Gateway. The below Gateway will expose both ports with their respective protocols. ```yaml -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: mixed-protocol-gateway @@ -195,22 +198,32 @@ spec: protocol: TURN-TCP ``` -> **Warning** -> Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature for compatibility reasons. You can re-enable mixed-protocol LBs by annotating your Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. +> [!WARNING] +> +> Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature. You can enable mixed-protocol LBs by annotating a Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. -STUNner implements two ways to customize the automatically created Service, both involving adding certain [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) to the Service. First, if any annotation is set in the GatewayConfig `loadBalancerServiceAnnotations` field then those will be copied verbatim into the Service. Note that `loadBalancerServiceAnnotations` affect *all* LoadBalancer Services created by STUNner under the current Gateway hierarchy. Second, Service annotations can be customized on a per-Gateway basis as well by adding the annotations to Gateway resources. STUNner then copies all annotations from the Gateway verbatim into the Service, overwriting the annotations specified in the GatewayConfig on conflict. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). +## UDPRoute -> **Warning** -Gateway resources are *not* safe for modification. This means that certain changes to a Gateway will restart the underlying TURN server listener, causing all active client sessions to terminate. The particular rules are as follows: -> - adding or removing a listener will start/stop *only* the TURN listener to be started/stopped, without affecting the rest of the listeners on the same Gateway; -> - changing the transport protocol, port or TLS keys/certs of an *existing* listener will restart the TURN listener but leave the rest of the listeners intact; -> - changing the TURN authentication realm will restart *all* TURN listeners. +UDPRoute resources can be attached to Gateways in order to specify the backend services permitted to be reached via the Gateway. Multiple UDPRoutes can attach to the same Gateway, and each UDPRoute can specify multiple backend services; in this case access to *all* backends in *each* of the attached UDPRoutes is allowed. An UDPRoute can be attached to a Gateway by setting the `parentRef` to the Gateway's name and namespace. This is, however, contingent on whether the Gateway accepts routes from the given namespace: customize the `allowedRoutes` per each Gateway listener to control which namespaces the listener accepts routes from. -## UDPRoute +The below UDPRoute will configure STUNner to route client connections received on the Gateway called `udp-gateway` to *any UDP port* on the pods of the media server pool identified by the Kubernetes service `media-server-pool` in the `media-plane` namespace. -UDPRoute resources can be attached to Gateways in order to specify the backend services permitted to be reached via the Gateway. Multiple UDPRoutes can attach to the same Gateway, and each UDPRoute can specify multiple backend services; in this case access to *all* backends in *each* of the attached UDPRoutes is allowed. An UDPRoute can be attached only to a Gateway in any namespace by setting the `parentRef` to the Gateway's name and namespace. This is, however, contingent on whether the Gateway accepts routes from the given namespace: customize the `allowedRoutes` for each Gateway listener to control which namespaces the listener accepts routes from. +```yaml +apiVersion: stunner.l7mp.io/v1 +kind: UDPRoute +metadata: + name: media-plane-route + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - name: media-server-pool + namespace: media-plane +``` -The below UDPRoute will configure STUNner to route client connections received on the Gateway called `udp-gateway` to the media server pool identified by the Kubernetes service `media-server-pool` in the `media-plane` namespace. +Note that STUNner provides its own UDPRoute resource instead of the official UDPRoute resource available in the Gateway API. In contrast to the official version, still at version v1alpha2, STUNner's UDPRoutes can be considered stable and expected to be supported throughout the entire lifetime of STUNner v1. You can still use the official UDPRoute resource as well, by changing the API version and adding an arbitrary port to the backend references (this is required by the official API). Note that the port will be omitted. ```yaml apiVersion: gateway.networking.k8s.io/v1alpha2 @@ -225,26 +238,37 @@ spec: - backendRefs: - name: media-server-pool namespace: media-plane + port: 1 ``` -Below is a quick reference of the most important fields of the UDPRoute [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects). +Below is a reference of the most important fields of the STUNner UDPRoute `spec`. | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | | `parentRefs` | `list` | Name/namespace of the Gateways to attach the route to. If no namespace is given, then the Gateway will be searched in the UDPRoute's namespace. | Yes | -| `rules.backendRefs` | `list` | A list of `name`/`namespace` pairs specifying the backend Service(s) reachable through the UDPRoute. It is allowed to specify a service from a namespace other than the UDPRoute's own namespace. | No | +| `rules.backendRefs` | `list` | A list of backends (Services or StaticServices) reachable through the UDPRoute. It is allowed to specify a service from a namespace other than the UDPRoute's own namespace. | No | + +Backend reference configuration is as follows: + +| Field | Type | Description | Required | +| :--- | :---: | :--- | :---: | +| `group` | `string` | API group for the backend, either empty string for Service backends or `stunner.l7mp.io` for StaticService backends. Default: `""`. | No | +| `kind` | `string` | The kind of the backend resource, either `Service` or `StaticService`. Default: `Service`. | No | +| `name` | `string` | Name of the backend Service or StaticService. | Yes | +| `namespace` | `string` | Namespace of the backend Service or StaticService. | Yes | +| `port` | `int` | Port to use to reach the backend. If empty, make all ports available on the backend. Default: empty.| No | +| `endPort` | `int` | If port is also specified, then access to the backend is restricted to the port range [port, endPort] inclusive. If port and endPort are empty, make all ports available on the backend. If port is given but endPort is not, admit the singleton port range [port,port]. Default: empty.| No | UDPRoute resources are safe for modification: `stunnerd` knows how to reconcile modified routes without restarting any listeners/TURN servers. ## StaticService -When the target backend of a UDPRoute is running *inside* Kubernetes then the backend is always a proper Kubernetes Service. However, when the target is deployed *outside* Kubernetes then there is no Kubernetes Service that could be configured as a backend. This is particularly important when STUNner is used as a public TURN service. The StaticService resource provides a way to assign a routable IP address range to a UDPRoute for these cases. +When the target backend of a UDPRoute is running *inside* Kubernetes then the backend is always a proper Kubernetes Service. However, when the target is deployed *outside* Kubernetes then there is no Kubernetes Service that could be configured as a backend. This is particularly problematic in the cases when STUNner is used as a public TURN service. For such deployments, the StaticService resource provides a way to assign a routable IP address range to a UDPRoute. -The below StaticService represents a hypothetical Kubernetes Service backing a set of pods with IP -addresses in the range `192.0.2.0/24` or `198.51.100.0/24`. +The below StaticService represents a hypothetical Kubernetes Service backing a set of pods with IP addresses in the range `192.0.2.0/24` or `198.51.100.0/24`. ```yaml -apiVersion: stunner.l7mp.io/v1alpha1 +apiVersion: stunner.l7mp.io/v1 kind: StaticService metadata: name: static-svc @@ -258,7 +282,7 @@ spec: Assigning this StaticService to a UDPRoute will make sure allows access to *any* IP address in the specified ranges. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: media-plane-route @@ -275,16 +299,69 @@ spec: The StaticService `spec.prefixes` must be a list of proper IPv4 prefixes: any IP address in any of the listed prefixes will be whitelisted. Use the single prefix `0.0.0.0/0` to provide wildcard access via an UDPRoute. -> **Warning** -Never use StaticServices to access Services running *inside* Kubernetes, this may open up an unintended backdoor to your cluster. Use StaticServices only with *external* target backends. +> [!WARNING] +> +> Never use StaticServices to access Services running *inside* Kubernetes, this may open up an unintended backdoor to your cluster. Use StaticServices only with *external* target backends. -## Status +## Dataplane -Most Kubernetes resources contain a `status` subresource that describes the current state of the resource, supplied and updated by the Kubernetes system and its components. The Kubernetes control plane continually and actively manages every object's actual state to match the desired state you supplied and updates the status field to indicate whether any error was encountered during the reconciliation process. +The Dataplane resource is used as a template for provisioning of `stunnerd` pods. This is useful to choose the image origin and version, set custom command line arguments and environment variables to the `stunnerd` daemon, configure resource requests/limits, etc. -If you are not sure about whether the STUNner gateway operator successfully picked up your Gateways or UDPRoutes, it is worth checking the status to see what went wrong. +Below is the `default` Dataplane installed by STUNner. -```console -kubectl get -n -o jsonpath='{.status}' +```yaml +apiVersion: stunner.l7mp.io/v1 +kind: Dataplane +metadata: + name: default +spec: + command: + - stunnerd + args: + - -w + - --udp-thread-num=16 + image: l7mp/stunnerd:latest + resources: + limits: + cpu: 500m + memory: 512Mi + requests: + cpu: 100m + memory: 128Mi + terminationGracePeriodSeconds: 3600 ``` +Below is a reference of the most important fields of the Dataplane `spec` that can be used to customize the provisioning of `stunnerd` pods. + +| Field | Type | Description | Required | +|:--------------------------------|:----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| +| `image` | `string` | The container image. | Yes | +| `imagePullPolicy` | `string` | Policy for if/when to pull an image, can be either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if `:latest` tag is specified on the image, or `IfNotPresent` otherwise. | No | +| `command` | `list` | Entrypoint array. Default: `stunnerd`. | No | +| `args` | `list` | Arguments to the entrypoint. | Yes | +| `envFrom` | `list` | List of sources to populate environment variables in the container. Default: empty. | No | +| `env` | `list` | List of environment variables to set in the container. Default: empty. | No | +| `replicas` | `int` | Number of `stunnerd` pods per Gateway to provision. Not enforced if the `stunnerd` Deployment replica count is overwritten manually or by an autoscaler. Default: 1. | No | +| `hostNetwork` | `bool` | Deploy `stunnerd` into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | +| `resources` | `object` | Compute resources required by `stunnerd`. Default: whatever Kubernetes assigns. | No | +| `affinity` | `object` | Scheduling constraints. Default: none. | No | +| `tolerations` | `object` | Tolerations. Default: none. | No | +| `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | +| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | +| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | + +There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the corresponding GatewayConfig to choose the Dataplane for each STUNner install. + +> [!WARNING] +> +> A Dataplane resource called `default` must always be available in the cluster, otherwise the operator will not know how to provision dataplane pods. Removing the `default` template will break your STUNner installation. + + + + + + + + + + diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 48abb57a..f5f8f28c 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -2,112 +2,95 @@ ## Prerequisites -You need a Kubernetes cluster (>1.22), and the `kubectl` command-line tool must be installed and -configured to communicate with your cluster. STUNner should be compatible with *any* major hosted -Kubernetes service or any on-prem Kubernetes cluster; if not, please file an issue. +You need a Kubernetes cluster (>1.22), and the `kubectl` command-line tool must be installed and configured to communicate with your cluster. STUNner should be compatible with *any* major hosted Kubernetes service or any on-prem Kubernetes cluster; if not, please file an issue. -The simplest way to expose STUNner to clients is through Kubernetes [LoadBalancer -services](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer); -these are automatically managed by STUNner. This depends on a functional LoadBalancer integration -in your cluster (if using Minikube, try `minikube tunnel` to get an idea of how this -works). STUNner automatically detects if LoadBalancer service integration is functional and falls -back to using NodePorts when it is not; however, this may require manual tweaking of the firewall -rules to admit the UDP NodePort range into the cluster. +The simplest way to expose STUNner to clients is through Kubernetes [LoadBalancer services](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer); these are automatically managed by STUNner. This depends on a functional LoadBalancer integration in your cluster (if using Minikube, try `minikube tunnel` to get an idea of how this works). STUNner automatically detects if LoadBalancer service integration is functional and falls back to using NodePorts when it is not; however, this may require manual tweaking of the firewall rules to admit the UDP NodePort range into the cluster. -To recompile STUNner, at least Go v1.19 is required. Building the container images requires -[Docker](https://docker.io) or [Podman](https://podman.io). +To compile STUNner, at least Go v1.19 is required. Building the container images requires [Docker](https://docker.io) or [Podman](https://podman.io). -## Basic installation +## Installation -The simplest way to deploy the full STUNner distro, with the dataplane and the controller -automatically installed, is through [Helm](https://helm.sh). STUNner configuration parameters are -available for customization as [Helm -Values](https://helm.sh/docs/chart_template_guide/values_files). We recommend deploying each -STUNner dataplane into a separate Kubernetes namespace (e.g., `stunner`), while the gateway -operator should go into the `stunner-system` namespace (but effectively any namespace would work). +The simplest way to deploy STUNner is through [Helm](https://helm.sh). STUNner configuration parameters are available for customization as [Helm Values](https://helm.sh/docs/chart_template_guide/values_files); see the [STUNner-helm](https://github.com/l7mp/stunner-helm) repository for a list of the available customizations. -First, register the STUNner repository with Helm. +The first step is to register the STUNner repository with Helm. ```console helm repo add stunner https://l7mp.io/stunner helm repo update ``` -Install the control plane: +### Stable version + +The below will install the stable version of STUNner. In particular, the this will install only the STUNner control plane, i.e., the gateway operator and the authentication service, the dataplane will be automatically provisioned by the operator when needed (but see below). We recommend to use the `stunner-system` namespace to keep the full STUNner control plane in a single scope. ```console -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace \ + --namespace=stunner-system ``` -Install the dataplane: +And that's all: you don't need to install the dataplane separately, this is handled automatically by the operator. The `stunnerd` pods created by the operator can be customized using the Dataplane custom resource: you can specify the `stunnerd` container image version, provision resources per each `stunenrd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1alpha1#DataplaneSpec). + +### Development version + +By default, the Helm chart installs the stable version of STUNner. To track the bleeding edge, STUNner provides a `dev` release channel that tracks the latest development version. Use it at your own risk: we do not promise any stability for the dev-channel. ```console -helm install stunner stunner/stunner --create-namespace --namespace=stunner +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace \ + --namespace=stunner-system ``` -## Parallel deployments +### Legacy mode + +In the default *managed dataplane mode*, the STUNner gateway operator automatically provisions the dataplane, which substantially simplifies operations and removes lot of manual and repetitive work. For compatibility reasons the traditional operational model, called the *legacy mode*, is still available. In this mode the user is responsible for provisioning both the control plane, by installing the `stunner-gateway-operator` Helm chart, and the dataplane(s), by helm-installing the `stunner` chart possibly multiple times. + +```console +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace \ + --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy +helm install stunner stunner/stunner --create-namespace --namespace=stunner +``` -You can install multiple STUNner dataplanes side-by-side, provided that the corresponding -namespaces are different. For instance, to create a `prod` dataplane installation for your -production workload and a `dev` installation for experimentation, the below commands will install -two dataplanes, one into the `stunner-prod` and another one into the `stunner-dev` namespace. +You can install multiple legacy STUNner dataplanes side-by-side, provided that the corresponding namespaces are different. For instance, to create a `prod` dataplane installation for your production workload and a `dev` installation for experimentation, the below commands will install two dataplanes, one into the `stunner-prod` and another one into the `stunner-dev` namespace. ```console helm install stunner-prod stunner/stunner --create-namespace --namespace=stunner-prod helm install stunner-dev stunner/stunner --create-namespace --namespace=stunner-dev ``` -Now, you can build a separate [gateway hierarchy](CONCEPTS.md) per each namespace to supply a -distinct ingress gateway configuration per dataplane. +## Customization -For the list of available customizations, see the -[STUNner-helm](https://github.com/l7mp/stunner-helm) repository. For installing STUNner in the -standalone mode, consult the documentation [here](OBSOLETE.md). +The Helm charts let you fine-tune STUNner features, including [compute resources](#resources) provisioned for `stunnerd` pods, [UDP multithreading](#udp-multithreading), and[graceful shutdown](#graceful-shutdown). -## Development version +### Resources requests/limits -STUNner provides a `dev` release channel, which allows to track the latest development version. Use -it at your own risk: we do not promise any stability for STUNner installed from the dev-channel. +it is important to manage the [amount of CPU and memory resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers) available for each `stunnerd` pod. The [default](https://github.com/l7mp/stunner-helm/blob/main/helm/stunner-gateway-operator/values.yaml) resource request and limit is set as follows: -```console -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system -helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner -``` +```yaml +resources: + limits: + cpu: 2 + memory: 512Mi + requests: + cpu: 500m + memory: 128Mi +``` -## Managed mode +This means that every `stunnerd` pod will request 0.5 CPU cores and 128 Mibytes of memory. Note that the pods will start only if Kubernetes can successfully allocate the given amount of resources. In order to avoid stressing the Kubernetes scheduler, it is advised to keep the limits at the bare minimum and scale out by [increasing the number of running `stunnerd` pods](SCALING.md) if needed. -From v0.16.0 STUNner provides a new way to provision dataplane pods that is called the *managed mode*. In the traditional operational model (called the *legacy mode*), the user was responsible for provisioning both the control plane, by installing the `stunner-gateway-operator` Helm chart, and the dataplane(s), by helm-installing the `stunner` chart [possibly multiple times](#parallel-deployments). In the managed mode the operator *automatically* provisions the necessary dataplanes by creating a separate `stunnerd` Deployment per each Gateway, plus the usual LoadBalancer service to expose it. This substantially simplifies operations and removes lot of manual and repetitive work. +### UDP multithreading -To install the gateway operator using the new manged mode, start with a clean Kubernetes cluster and install the `stunner-gateway-operator` Helm chart, setting the flag `stunnerGatewayOperator.dataplane.mode` to `managed`. Observe that we do not install the `stunner` Helm chart separately; the operator will readily create the `stunnerd` pods as needed. +STUNner can run multiple UDP listeners over multiple parallel readloops for loadbalancing. Namely, ech `stunnerd` pod can create a configurable number of UDP server sockets using `SO_REUSEPORT` and then spawn a separate goroutine to run a parallel readloop per each. The kernel will load-balance allocations across the sockets/readloops per the IP 5-tuple, so the same allocation will always stay at the same CPU. This allows UDP listeners to scale to multiple CPUs, improving performance. Note that this is required only for UDP: TCP, TLS and DTLS listeners spawn a per-client readloop anyway. Also note that `SO_REUSEPORT` is not portable, so currently we enable this only for UNIX architectures. -```console -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace \ - --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=managed +The feature is exposed via the command line flag `--udp-thread-num=` in `stunnerd`. In the Helm chart, it can be enabled or disabled with the `--set stunner.deployment.container.stunnerd.udpMultithreading.enabled=true` flag. By default, UDP multithreading is enabled with 16 separate readloops per each UDP listener. + +```yaml +udpMultithreading: + enabled: true + readLoopsPerUDPListener: 16 ``` -The `stunnerd` pods created by the operator can be customized using the Dataplane CR: for instance you can specify the `stunnerd` container image version to be used as the dataplane, provision resources for each `stunenrd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1alpha1#DataplaneSpec). All gateways will use the `default` Dataplane; you can override this by creating a new Dataplane CR and setting the name in the [`spec.dataplane` field](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator@v0.15.2/api/v1alpha1#GatewayConfigSpec) of the corresponding GatewayConfig. +### Graceful shutdown + +STUNner has full support for [graceful shutdown](SCALING.md). This means that `stunner` pods will remain alive as long as there are active allocations in the embedded TURN server, and a pod will automatically remove itself once all allocations are deleted or time out. This enables the full support for graceful scale-down: the user can scale the number of `stunner` instances up and down and no harm should be made to active client connections meanwhile. + +The default termination period is set to 3600 seconds (1 hour). To modify, use the `--set stunner.deployment.container.terminationGracePeriodSeconds=` flag. -```console -kubectl get dataplanes.stunner.l7mp.io default -o yaml -apiVersion: stunner.l7mp.io/v1alpha1 -kind: Dataplane -metadata: - name: default -spec: - image: l7mp/stunnerd:latest - imagePullPolicy: Always - command: - - stunnerd - args: - - -w - - --udp-thread-num=16 - hostNetwork: false - resources: - limits: - cpu: 2 - memory: 512Mi - requests: - cpu: 500m - memory: 128Mi - terminationGracePeriodSeconds: 3600 -``` diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 9608d94d..d011279a 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -1,43 +1,18 @@ # Monitoring -STUNner can export various statistics into an external timeseries database like -[Prometheus](https://prometheus.io). This allows one to observe the state of the STUNner media -gateway instances, like CPU or memory use, as well as the amount of data received and sent, in -quasi-real-time. These statistics can then be presented to the operator in easy-to-use monitoring -dashboards in [Grafana](https://grafana.com). +STUNner can export various statistics into an external timeseries database like [Prometheus](https://prometheus.io). This allows one to observe the state of the STUNner media gateway instances, like CPU or memory use or the amount of data received and sent in quasi-real-time. These statistics can then be presented to the operator in a monitoring dashboard using, e.g., [Grafana](https://grafana.com). ## Configuration -Metrics collection is *not* enabled in the default installation. In order to open the -metrics-collection endpoint for a [gateway hierarchy](GATEWAY.md#overview), configure an -appropriate HTTP URL in the `metricsEndpoint` field of corresponding the -[GatewayConfig](GATEWAY.md#gatewayconfig) resource. - -For instance, the below GatewayConfig will expose the metrics-collection server on the URL -`http://:8080/metrics` in all the STUNner media gateway instances of the current gateway hierarchy. - -```yaml -apiVersion: stunner.l7mp.io/v1alpha1 -kind: GatewayConfig -metadata: - name: stunner-gatewayconfig - namespace: stunner -spec: - userName: "my-user" - password: "my-password" - metricsEndpoint: "http://:8080/metrics" -``` +Metrics collection is *not* enabled by default. To enable it, set the `enableMetricsEndpoint` field to true in the [Dataplane](GATEWAY.md#dataplane) template. This will configure the `stunnerd` dataplane pods to expose a HTTP metrics endpoint at port 8080 that Prometheus can scrape for metrics. ## Metrics -STUNner exports two types of metrics: the *Go collector metrics* describe the state of the Go -runtime, while the *Connection statistics* expose traffic monitoring data. +STUNner exports two types of metrics: the *Go collector metrics* describe the state of the Go runtime, while the *Connection statistics* expose traffic monitoring data. ### Go collector metrics -Each STUNner gateway instance exports a number of standard metrics that describe the state of the -current Go process runtime. Some notable metrics as listed below, see more in the -[documentation](https://github.com/prometheus/client_golang). +Each STUNner gateway instance exports a number of standard metrics that describe the state of the current Go process. Some notable metrics as listed below, see more in the [documentation](https://github.com/prometheus/client_golang). | Metric | Description | | :--- | :--- | @@ -50,8 +25,7 @@ current Go process runtime. Some notable metrics as listed below, see more in th ### Connection statistics -STUNner provides deep visibility into the amount of traffic sent and received on each listener -(downstream connections) and cluster (upstream connections). The particular metrics are as follows. +STUNner provides deep visibility into the amount of traffic sent and received on each listener (downstream connections) and cluster (upstream connections). The particular metrics are as follows. | Metric | Description | Type | Labels | | :--- | :--- | :--- | :--- | @@ -59,18 +33,16 @@ STUNner provides deep visibility into the amount of traffic sent and received on | `stunner_listener_connections_total` | Number of downstream connections at a listener. | counter | `name=` | | `stunner_listener_packets_total` | Number of datagrams sent or received at a listener. Unreliable for listeners running on a connection-oriented transport protocol (TCP/TLS). | counter | `direction=`, `name=`| | `stunner_listener_bytes_total` | Number of bytes sent or received at a listener. | counter | `direction=`, `name=` | -| `stunner_cluster_connections` | Number of *active* upstream connections on behalf of a listener. | gauge | `name=` | -| `stunner_cluster_connections_total` | Number of upstream connections on behalf of a listener. | counter | `name=` | -| `stunner_cluster_packets_total` | Number of datagrams sent to backends or received from backends on behalf of a listener. Unreliable for clusters running on a connection-oriented transport protocol (TCP/TLS).| counter | `direction=`, `name=` | -| `stunner_cluster_bytes_total` | Number of bytes sent to backends or received from backends on behalf of a listener. | counter | `direction=`, `name=` | +| `stunner_cluster_packets_total` | Number of datagrams sent to backends or received from backends of a cluster. Unreliable for clusters running on a connection-oriented transport protocol (TCP/TLS).| counter | `direction=`, `name=` | +| `stunner_cluster_bytes_total` | Number of bytes sent to backends or received from backends of a cluster. | counter | `direction=`, `name=` | ## Integration with Prometheus and Grafana -Collection and visualization of STUNner relies on Prometheus and Grafana services. The STUNer helm repository provides a way to [install](#installation) a ready-to-use Prometheus and Grafana stack. In addition, metrics visualization requires [user input](#configuration-and-usage) on configuring the plots; see below. +Collection and visualization of STUNner relies on Prometheus and Grafana services. The STUNer helm repository provides a way to [install](https://github.com/l7mp/stunner-helm#monitoring) a ready-to-use Prometheus and Grafana stack. In addition, metrics visualization requires [user input](#configuration) on configuring the plots; see below. ### Installation -A full-fledged Prometheus+Grafana helm chart is available in the STUNner helm repo. To use this chart, the installation steps involve enabling monitoring in STUNner, and installing the Prometheus+Grafana stack with helm. +A full-fledged Prometheus+Grafana helm chart is available in the [STUNner helm repo](https://github.com/l7mp/stunner-helm#monitoring). To use this chart, the installation steps involve enabling monitoring in STUNner, and installing the Prometheus+Grafana stack with helm. 1. Install STUNner with Prometheus support: @@ -78,7 +50,7 @@ A full-fledged Prometheus+Grafana helm chart is available in the STUNner helm re helm install stunner stunner/stunner --create-namespace --namespace=stunner --set stunner.deployment.monitoring.enabled=true ``` -2. Configure STUNner to expose the metrics by [exposing the STUNner metrics-collection server in the GatewayConfig](#configuration). +2. Configure STUNner to expose the metrics. 3. Install the Prometheus+Grafana stack with a helm chart. @@ -87,23 +59,14 @@ A full-fledged Prometheus+Grafana helm chart is available in the STUNner helm re ```console helm repo add stunner https://l7mp.io/stunner helm repo update - helm install prometheus stunner/stunner-prometheus ``` -### Configuration and Usage +### Configuration The helm chart deploys a ready-to-use Prometheus and Grafana stack, but leaves the Grafana dashboard empty to let the user pick metrics and configure their visualization. An interactive way to visualize STUNner metrics is to use the Grafana dashboard. -#### Access the Grafana dashboard - -To open the Grafana dashboard navigate a web browser to `grafana` NodePort service IP and port 80. - -The default username is **admin** with the password **admin**. - -At the first login you can change the password or leave as it is (use the *Skip* button). - -#### Visualize STUNner metrics +To open the Grafana dashboard navigate a web browser to `grafana` NodePort service IP and port 80. The default username is **admin** with the password **admin**. At the first login you can change the password or leave as it is (use the *Skip* button). As an example, let us plot the STUNner metric `stunner_listener_connections`. First step is to create a new panel, then to configure the plot parameters. @@ -131,10 +94,7 @@ Below is an example dashboard with data collected from the [simple-tunnel](examp Prometheus and Grafana both provide a dashboard to troubleshoot a running system, and to check the flow of metrics from STUNner to Prometheus, and from Prometheus to Grafana. -### Check Prometheus operations via its dashboard -The Prometheus dashboard is available as the `prometheus` NodePort service (use the node IP and node port to connect with a web browser). - -The dashboard enables checking running Prometheus configuration and testing the metrics collection. +The Prometheus dashboard is available as the `prometheus` NodePort service (use the node IP and node port to connect with a web browser). The dashboard enables checking running Prometheus configuration and testing the metrics collection. For example, to observe the `stunner_listener_connections` metric on the Prometheus dashboard: @@ -144,9 +104,7 @@ For example, to observe the `stunner_listener_connections` metric on the Prometh ![Prometheus Dashboard](img/prometheus-dashboard.png) -Note: some STUNner metrics are not available when they are inactive (e.g., there is no active cluster). - -#### Check Prometheus data source in Grafana +Note that some STUNner metrics may not be available when they are inactive (e.g., there is no active cluster). To configure/check the Prometheus data source in Grafana, first click on *Configuration* (1), then *Data sources* (2), as shown here: diff --git a/docs/OBSOLETE.md b/docs/OBSOLETE.md deleted file mode 100644 index 6beaa545..00000000 --- a/docs/OBSOLETE.md +++ /dev/null @@ -1,501 +0,0 @@ -# Standalone mode - -In order to gain full control over media ingestion, STUNner can be deployed without the gateway -operator component. In this standalone mode, the user is fully in charge of creating and -maintaining the configuration of the `stunnerd` pods. With the introduction of the STUNner gateway -operator *the standalone mode is considered obsolete* as of STUNner v0.11. The below documentation -is provided only for historical reference; before the gateway operator existed this was *the* -recommended way to interact with STUNner. - -## Table of contents - -- [Standalone mode](#standalone-mode) - - [Table of contents](#table-of-contents) - - [Prerequisites](#prerequisites) - - [Installation](#installation) - - [Installation with Helm](#installation-with-helm) - - [Manual installation](#manual-installation) - - [Configuration](#configuration) - - [Learning the external IP and port](#learning-the-external-ip-and-port) - - [Configuring WebRTC clients](#configuring-webrtc-clients) - - [Authentication](#authentication) - - [Access control](#access-control) - - [Enabling TURN transport over TCP](#enabling-turn-transport-over-tcp) - - [Enabling TURN transport over TLS and DTLS](#enabling-turn-transport-over-tls-and-dtls) - -## Prerequisites - -The below installation instructions require an operational cluster running a supported version of -Kubernetes (>1.22). Make sure that the cluster comes with a functional [load-balancer -integration](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer), -otherwise STUNner will not be able to allocate a public IP address for clients to reach your WebRTC -infra. In the standalone mode STUNner relies on Kubernetes ACLs (`NetworkPolicy`) with [port -ranges](https://kubernetes.io/docs/concepts/services-networking/network-policies/#targeting-a-range-of-ports) -to block malicious access; make sure your Kubernetes installation supports these. - -## Installation - -### Installation with Helm - -Use the [Helm charts](https://github.com/l7mp/stunner-helm) for installing STUNner, setting the -`standalone.enabled` feature gate to `true`: - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner stunner/stunner --set stunner.standalone.enabled=true -``` - -The below will create a new namespace named `stunner` and install the STUNner dataplane pods into that -namespace. - -```console -helm install stunner stunner/stunner --set stunner.standalone.enabled=true --create-namespace --namespace=stunner -``` - -Note that we do not install the usual control plane: in this mode we ourselves need to manually -provide the dataplane configuration for STUNner. - -### Manual installation - -If Helm is not an option, you can perform a manual installation using the static Kubernetes -manifests packaged with STUNner. - -First, clone the STUNner repository. - -```console -git clone https://github.com/l7mp/stunner.git -cd stunner -``` - -Then, customize the default settings in the STUNner service -[manifest](https://github.com/l7mp/stunner/blob/main/deploy/manifests/stunner-standalone.yaml) and deploy it via `kubectl`. - -```console -kubectl apply -f deploy/manifests/stunner-standalone.yaml -``` - -By default, all resources are created in the `default` namespace. - -## Configuration - -The default STUNner installation will create the below Kubernetes resources: - -1. a ConfigMap that stores STUNner local configuration, -2. a Deployment running one or more STUNner daemon replicas, -3. a LoadBalancer service to expose the STUNner deployment on a public IP address and UDP port - (by default, the port is UDP 3478), and finally -4. a NetworkPolicy, i.e., an ACL/firewall policy to control network communication from STUNner to - the rest of the Kubernetes workload. - -The installation scripts packaged with STUNner will use hard-coded configuration defaults that must -be customized prior to deployment. In particular, make absolutely sure to customize the access -tokens (`STUNNER_USERNAME` and `STUNNER_PASSWORD` for `plaintext` authentication, and -`STUNNER_SHARED_SECRET` and possibly `STUNNER_DURATION` for the `longterm` authentication mode), -otherwise STUNner will use hard-coded STUN/TURN credentials. This should not pose a major security -risk (see [here](SECURITY.md) for more info), but it is still safer to customize the access -tokens before exposing STUNner to the Internet. - -The most recent STUNner configuration is always available in the Kubernetes ConfigMap named -`stunnerd-config`. This configuration is made available to the `stunnerd` pods by -[mapping](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-container-environment-variables-using-configmap-data) -the `stunnerd-config` ConfigMap into the pods as environment variables. Note that changes to this -ConfigMap will take effect only once STUNner is restarted. - -The most important STUNner configuration settings are as follows. -* `STUNNER_PUBLIC_ADDR` (no default): The public IP address clients can use to reach STUNner. By - default, the public IP address will be dynamically assigned during installation. The installation - scripts take care of querying the external IP address from Kubernetes and automatically setting - `STUNNER_PUBLIC_ADDR`; for manual installation the external IP must be set by hand (see - [details](#learning-the-external-ip-and-port) below). -* `STUNNER_PUBLIC_PORT` (default: 3478): The public port used by clients to reach STUNner. Note - that the Helm installation scripts may overwrite this configuration if the installation falls - back to the `NodePort` service (i.e., when STUNner fails to obtain an external IP from the - Kubernetes ingress load balancer), see [details](#learning-the-external-ip-and-port) below. -* `STUNNER_PORT` (default: 3478): The internal port used by STUNner for communication inside the - cluster. It is safe to set this to the public port. -* `STUNNER_TRANSPORT_UDP_ENABLE` (default: "1", enabled): Enable UDP TURN transport. -* `STUNNER_TRANSPORT_TCP_ENABLE` (default: "", disabled): Enable TCP TURN transport. -* `STUNNER_REALM` (default: `stunner.l7mp.io`): the - [`REALM`](https://www.rfc-editor.org/rfc/rfc8489.html#section-14.9) used to guide the user agent - in authenticating with STUNner. -* `STUNNER_AUTH_TYPE` (default: `plaintext`): the STUN/TURN authentication mode, either `plaintext` - using the username/password pair `$STUNNER_USERNAME`/`$STUNNER_PASSWORD`, or `longterm`, using - the [STUN/TURN long-term credential](https://www.rfc-editor.org/rfc/rfc8489.html#section-9.2) - mechanism with the secret `$STUNNER_SHARED_SECRET`. -* `STUNNER_USERNAME` (default: `user`): the - [username](https://www.rfc-editor.org/rfc/rfc8489.html#section-14.3) attribute clients can use to - authenticate with STUNner over `plaintext` authentication. Make sure to customize! -* `STUNNER_PASSWORD` (default: `pass`): the password clients can use to authenticate with STUNner - in `plaintext` authentication. Make sure to customize! -* `STUNNER_SHARED_SECRET`: the shared secret used for `longterm` authentication mode. Make sure to - customize! -* `STUNNER_DURATION` (default: `86400` sec, i.e., one day): the lifetime of STUNner credentials in - `longterm` authentication. -* `STUNNER_LOGLEVEL` (default: `all:WARN`): the default log level used by the STUNner daemons. -* `STUNNER_MIN_PORT` (default: 10000): smallest relay transport port assigned by STUNner. -* `STUNNER_MAX_PORT` (default: 20000): highest relay transport port assigned by STUNner. - -The default configuration can be overridden by setting custom command line arguments when -[launching the STUNner pods](cmd/stunnerd.md). All examples below assume that STUNner is -deployed into the `default` namespace; see the installation notes below on how to override this. - -Note that changing in the configuration values becomes valid only once STUNner is restarted (see -below). - -## Learning the external IP and port - -There are two ways to expose the STUN/TURN ingress gateway service with STUNner: through a standard -Kubernetes [`LoadBalancer` -service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer) (the -default) or as a [`NodePort` -service](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport), used as a -fallback if an ingress load-balancer is not available. In both cases the external IP address and -port that WebRTC clients can use to reach STUNner may be set dynamically by Kubernetes. (Kubernetes -lets you use your own [fix IP address and domain -name](https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address), -but the default installation scripts do not support this.) - -In general, WebRTC clients will need to learn STUNner's external IP and port somehow. In order to -simplify the integration of STUNner into the WebRTC application server, STUNner stores the dynamic -IP address/port assigned by Kubernetes into the `stunnerd-config` ConfigMap under the key -`STUNNER_PUBLIC_IP` and `STUNNER_PUBLIC_PORT`. Then, WebRTC application pods can map this ConfigMap -as environment variables and communicate the IP address and port back to the clients (see an -[example](#configuring-webrtc-clients) below). - -The [Helm installation](#helm) scripts should take care of setting the IP address and port -automatically in the ConfigMap during installation. However, if later the LoadBalancer services -change for some reason then the new external IP address and port will need to be configured -manually in the ConfigMap. Similar is the case when using the static Kubernetes manifests to deploy -STUNner. The below instructions simplify this process. - -After a successful installation, you should see something similar to the below: - -```console -kubectl get all -NAME READY STATUS RESTARTS AGE -pod/stunner-XXXXXXXXXX-YYYYY 1/1 Running 0 8s - -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -service/kubernetes ClusterIP 10.72.128.1 443/TCP 6d4h -service/stunner ClusterIP 10.72.130.61 3478/UDP 81s -service/stunner-standalone-lb LoadBalancer 10.72.128.166 A.B.C.D 3478:30630/UDP 81s - -NAME READY UP-TO-DATE AVAILABLE AGE -deployment.apps/stunner 1/1 1 1 8s -``` - -Note the external IP address allocated by Kubernetes for the `stunner-standalone-lb` service -(`EXTERNAL-IP` marked with a placeholder `A.B.C.D` in the above): this will be the public STUN/TURN -access point that your WebRTC clients will need to use in order to access the WebRTC media service -via STUNner. - -Wait until Kubernetes assigns a valid external IP to STUNner and query the public IP address and -port used by STUNner from Kubernetes. - -```console -until [ -n "$(kubectl get svc stunner-standalone-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -export STUNNER_PUBLIC_ADDR=$(kubectl get svc stunner-standalone-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export STUNNER_PUBLIC_PORT=$(kubectl get svc stunner-standalone-lb -o jsonpath='{.spec.ports[0].port}') -``` - -If this hangs for minutes, then your Kubernetes load-balancer integration is not working (if using -[Minikube](https://github.com/kubernetes/minikube), make sure `minikube tunnel` is -[running](https://minikube.sigs.k8s.io/docs/handbook/accessing)). This may still allow STUNner to -be reached externally, using a Kubernetes `NodePort` service (provided that your [Kubernetes -supports -NodePorts](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#no_direct_external_inbound_connections_for_private_clusters)). In -this case, but only in this case!, set the IP address and port from the NodePort: - -```console -export STUNNER_PUBLIC_ADDR=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}') -export STUNNER_PUBLIC_PORT=$(kubectl get svc stunner-standalone-lb -o jsonpath='{.spec.ports[0].nodePort}') -``` - -Check that the IP address/port `${STUNNER_PUBLIC_ADDR}:${STUNNER_PUBLIC_PORT}` is reachable by your -WebRTC clients; some Kubernetes clusters are installed with private node IP addresses that may -prevent NodePort services to be reachable from the Internet. - -If all goes well, the STUNner service is now exposed on the IP address `$STUNNER_PUBLIC_ADDR` and -UDP port `$STUNNER_PUBLIC_PORT`. Finally, store the public IP address and port back into STUNner's -configuration, so that the WebRTC application server can learn this information and forward it to -the clients. - -```console -kubectl patch configmap/stunnerd-config --type merge \ - -p "{\"data\":{\"STUNNER_PUBLIC_ADDR\":\"${STUNNER_PUBLIC_ADDR}\",\"STUNNER_PUBLIC_PORT\":\"${STUNNER_PUBLIC_PORT}\"}}" -``` - -## Configuring WebRTC clients - -The last step is to configure your WebRTC clients to use STUNner as the TURN server. The below -JavaScript snippet will direct WebRTC clients to use STUNner; make sure to substitute the -placeholders (like ``) with the correct configuration from the above. - -```javascript -var ICE_config = { - 'iceServers': [ - { - 'url': "turn::?transport=udp', - 'username': , - 'credential': , - }, - ], -}; -var pc = new RTCPeerConnection(ICE_config); -``` - -## Authentication - -STUNner relies on the STUN [long-term credential -mechanism](https://www.rfc-editor.org/rfc/rfc8489.html#page-26) to provide user authentication. See -[here](AUTH.md) for more detail on STUNner's authentication modes. - -The below commands will configure STUNner to use `plaintext` authentication using the -username/password pair `my-user/my-password` and restart STUNner for the new configuration to take -effect. - -```console -kubectl patch configmap/stunnerd-config --type merge \ - -p "{\"data\":{\"STUNNER_AUTH_TYPE\":\"plaintext\",\"STUNNER_USERNAME\":\"my-user\",\"STUNNER_PASSWORD\":\"my-password\"}}" -kubectl rollout restart deployment/stunner -``` - -The below commands will configure STUNner to use `longterm` authentication mode, using the shared -secret `my-secret`. By default, STUNner credentials are valid for one day. - -```console -kubectl patch configmap/stunnerd-config --type merge \ - -p "{\"data\":{\"STUNNER_AUTH_TYPE\":\"longterm\",\"STUNNER_SHARED_SECRET\":\"my-secret\"}}" -kubectl rollout restart deployment/stunner -``` - -## Access control - -The security risks and best practices associated with STUNner are described -[here](SECURITY.md), below we summarize the only step that is specific to the standalone mode: -configuring access control. - -By default, a standalone STUNner installation comes with an open route: this essentially means -that, possessing a valid TURN credential, an attacker can reach *any* UDP service inside the -Kubernetes cluster via STUNner. This is because, without an operator, there is no control plane to -supply [endpoint-discovery -service](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/service_discovery#endpoint-discovery-service-eds) -for the dataplane and therefore `stunnerd` does not know whether the peer address a client wished -to reach belongs to the legitimate backend service or not. In order to prevent open access through -STUNner, the default standalone installation comes with a default-deny Kubernetes NetworkPolicy -that locks down *all* access from the STUNner pods to the rest of the workload. - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: stunner-network-policy -spec: - podSelector: - matchLabels: - app: stunner - policyTypes: - - Egress -``` - -In order for clients to reach a media server pod via STUNner the user must explicitly whitelist the -target service in this access control rule. Suppose that we want STUNner to reach the media server -pods labeled as `app=media-server` over the UDP port range `[10000:20000]`, but we don't want -connections via STUNner to succeed to any other pod. This will be enough to support WebRTC media, -but will not allow clients to, e.g., reach the Kubernetes DNS service. - -Assuming that the entire workload is deployed into the `default` namespace, the below -`NetworkPolicy` ensures that all access from any STUNner pod to any media server pod is allowed -over any UDP port between 10000 and 20000, and all other network access from STUNner is denied. - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: stunner-network-policy -spec: -# Choose the STUNner pods as source - podSelector: - matchLabels: - app: stunner - policyTypes: - - Egress - egress: - # Allow only this rule, everything else is denied - - to: - # Choose the media server pods as destination - - podSelector: - matchLabels: - app: media-server - ports: - # Only UDP ports 10000-20000 are allowed between - # the source-destination pairs - - protocol: UDP - port: 10000 - endPort: 20000 -``` - -If your Kubernetes CNIs does not support [network policies with port -ranges](https://kubernetes.io/docs/concepts/services-networking/network-policies/#targeting-a-range-of-ports), -then the below will provide an access control rule similar to the above, except that it opens up -*all* UDP ports on the media server instead of limiting access to the UDP port range -`[10000:20000]`. - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: stunner-network-policy -spec: - podSelector: - matchLabels: - app: stunner - policyTypes: - - Egress - egress: - - to: - - podSelector: - matchLabels: - app: media-server - ports: - - protocol: UDP -``` - -## Enabling TURN transport over TCP - -Some corporate firewalls block all UDP access from the private network, except DNS. To make sure -that clients can still reach STUNner, you can expose STUNner over a [TCP-based TURN -transport](https://www.rfc-editor.org/rfc/rfc6062). To maximize the chances of getting through a -zealous firewall, below we expose STUNner over the default HTTPS port 443. - -First, enable TURN transport over TCP in STUNner. - -```console -kubectl patch configmap/stunnerd-config --type merge -p "{\"data\":{\"STUNNER_TRANSPORT_TCP_ENABLE\":\"1\"}}" -``` - -Then, delete the default Kubernetes service that exposes STUNner over UDP and re-expose it over the -TCP port 443. -```console -kubectl delete service stunner-standalone-lb -kubectl expose deployment stunner-standalone-lb --protocol=TCP --port=443 --type=LoadBalancer -``` - -Wait until Kubernetes assigns a public IP address. -```console -until [ -n "$(kubectl get svc stunner-standalone-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -export STUNNER_PUBLIC_ADDR=$(kubectl get svc stunner-standalone-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export STUNNER_PUBLIC_PORT=$(kubectl get svc stunner-standalone-lb -o jsonpath='{.spec.ports[0].port}') -kubectl patch configmap/stunnerd-config --type merge \ - -p "{\"data\":{\"STUNNER_PUBLIC_ADDR\":\"${STUNNER_PUBLIC_ADDR}\",\"STUNNER_PUBLIC_PORT\":\"${STUNNER_PUBLIC_PORT}\"}}" -``` - -Restart STUNner with the new configuration. -```console -kubectl rollout restart deployment/stunner -``` - -Finally, direct your clients to the re-exposed STUNner TCP service with the below `PeerConnection` configuration; don't -forget to rewrite the TURN transport to TCP by adding the query `transport=tcp` to the -STUNner URI. -```javascript -var ICE_config = { - 'iceServers': [ - { - 'url': "turn::?transport=tcp", - 'username': , - 'credential': , - }, - ], -}; -var pc = new RTCPeerConnection(ICE_config); -``` - -## Enabling TURN transport over TLS and DTLS - -The ultimate tool to work around aggressive firewalls and middleboxes is exposing STUNner via TLS -and/or DTLS. Fixing the TLS listener port at 443 will make it impossible for the corporate firewall -to block TURN/TLS connections without blocking all external HTTPS access, so most probably at least -the TCP/443 port will be open to encrypted connections. - -Start with a fresh Kubernetes install. Below we create a self-signed certificate for testing; make -sure to replace the cert/key pair below with your own trusted credentials. - -```console -openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=example.domain.com" -kubectl create secret tls stunner-tls --key /tmp/tls.key --cert /tmp/tls.crt -``` - -Patch the TLS cert/key into the pre-configured static manifest and deploy the STUNner gateway. - -```console -cd stunner -cat deploy/manifests/stunner-standalone-tls.yaml.template | \ - perl -pe "s%XXXXXXX%`cat /tmp/tls.key | base64 -w 0`%g" | - perl -pe "s%YYYYYYY%`cat /tmp/tls.crt | base64 -w 0`%g" | - kubectl apply -f - -``` - -This will fire up STUNner with two TURN listeners, a TLS/TCP and a DTLS/UDP listener, both at port -443, and create two LoadBalancer services to expose these to clients. - -Wait until Kubernetes assigns a public IP address and learn the new public addresses. -```console -until [ -n "$(kubectl get svc stunner-tls -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -until [ -n "$(kubectl get svc stunner-dtls -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -export STUNNER_PUBLIC_ADDR_TLS=$(kubectl get svc stunner-tls -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export STUNNER_PUBLIC_ADDR_DTLS=$(kubectl get svc stunner-dtls -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -``` - -Check your configuration with the handy [`turncat`](cmd/turncat.md) utility and the [UDP -greeter](https://github.com/l7mp/stunner#testing) service. First, query the UDP greeter service via TLS/TCP. Here, the -`turncat` command line argument `-i` puts `turncat` into insecure mode in order to accept our -self-signed TURN sever TLS certificate. - -```console -cd stunner -go build -o turncat cmd/turncat/main.go -kubectl apply -f deploy/manifests/udp-greeter.yaml -export PEER_IP=$(kubectl get svc media-plane -o jsonpath='{.spec.clusterIP}') -export STUNNER_USERNAME=$(kubectl get cm stunner-config -o yaml -o jsonpath='{.data.STUNNER_USERNAME}') -export STUNNER_PASSWORD=$(kubectl get cm stunner-config -o yaml -o jsonpath='{.data.STUNNER_PASSWORD}') -./turncat -i - turn://${STUNNER_USERNAME}:${STUNNER_PASSWORD}@${STUNNER_PUBLIC_ADDR_TLS}:443?transport=tls udp://${PEER_IP}:9001 -Hello STUNner via TLS -Greetings from STUNner! -``` - -Type anything once `turncat` is running to receive a nice greeting from STUNner. DTLS/UDP should -also work fine: - -```console -./turncat -i - turn://${STUNNER_USERNAME}:${STUNNER_PASSWORD}@${STUNNER_PUBLIC_ADDR_DTLS}:443?transport=dtls udp://${PEER_IP}:9001 -Another hello STUNner, now via DTLS! -Greetings from STUNner! -``` - -Remember, you can always direct your clients to your TURN listeners by setting the TURN URIs in the -ICE server configuration on your `PeerConnection`s. - -```javascript -var ICE_config = { - 'iceServers': [ - { - 'url': "turn::443?transport=tls", - 'username': , - 'credential': , - }, - { - 'url': "turn::443?transport=dtls", - 'username': , - 'credential': , - }, - ], -}; -var pc = new RTCPeerConnection(ICE_config); -``` - -Note that the default Kubernetes manifest -['stunner-standalone-tls.yaml'](https://github.com/l7mp/stunner/blob/main/deploy/manifests/stunner-standalone-tls.yaml.template) opens up the -NetworkPolicy for the `media-plane/default` service only, make sure to configure this to your own -setup. diff --git a/docs/README.md b/docs/README.md index 33448b4b..9832cce7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,11 +18,11 @@ ## User guides -* [Gateway API reference](GATEWAY.md) * [Authentication](AUTH.md) * [Monitoring](MONITORING.md) * [Scaling](SCALING.md) * [Security](SECURITY.md) +* [Reference](GATEWAY.md) ## Tutorials @@ -49,7 +49,3 @@ * [`stunnerd` manual](cmd/stunnerd.md) * [`turncat` manual](cmd/turncat.md) * [`stunnerctl` manual](cmd/stunnerctl.md) - -## Obsolete features - -* [Standalone mode](OBSOLETE.md) diff --git a/docs/SCALING.md b/docs/SCALING.md index b615c501..98e9cfdd 100644 --- a/docs/SCALING.md +++ b/docs/SCALING.md @@ -1,85 +1,33 @@ # Scaling -[Autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale) is one of -the key features in Kubernetes. This means that Kubernetes will automatically increase the number -of pods that run a service as the demand for the service increases, and reduce the number of pods -when the demand drops. This improves service quality, simplifies management, and reduces -operational costs by avoiding the need to over-provision services to the peak load. Most -importantly, autoscaling saves you from having to guess the number of nodes or pods needed to run -your workload: Kubernetes will automatically and dynamically resize your workload based on demand. +[Autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale) is one of the key features in Kubernetes. This means that Kubernetes will automatically increase the number of pods that run a service as the demand for the service increases, and reduce the number of pods when the demand drops. This improves service quality, simplifies management, and reduces operational costs by avoiding the need to over-provision services to the peak load. Most importantly, autoscaling saves you from having to guess the number of nodes or pods needed to run your workload: Kubernetes will automatically and dynamically resize your workload based on demand. Further factors to autoscale your WebRTC workload are: - smaller load on each instance: this might result in better and more stable performance; - smaller blast radius: less calls will be affected if a pod fails for some reason. -Autoscaling a production service, especially one as sensitive to latency and performance as WebRTC, -can be challenging. This guide will provide the basics on autoscaling; see the [official -docs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale) for more detail. +Autoscaling a production service, especially one as sensitive to latency and performance as WebRTC, can be challenging. This guide will provide the basics on autoscaling; see the [official docs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale) for more detail. ## Horizontal scaling -It is a good practice to scale Kubernetes workloads -[horizontally](https://openmetal.io/docs/edu/openstack/horizontal-scaling-vs-vertical-scaling) -(that is, by adding or removing service pods) instead of vertically (that is, by migrating to a -more powerful server) when demand increases. Correspondingly it is a good advice to set the -[resource limits and -requests](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) to the -bare minimum and let Kubernetes to automatically scale out the service by adding more pods if -needed. Note that that HPA [uses the requested amount of -resources](https://pauldally.medium.com/horizontalpodautoscaler-uses-request-not-limit-to-determine-when-to-scale-97643d808997) -to determine when to scale-up or down the number of instances. - -STUNner comes with a full support for horizontal scaling using the the Kubernetes built-in -[HorizontalPodAutoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale) -(HPA). The triggering event can be based on arbitrary metric, say, the [number of active client -connections](#MONITORING.md) per STUNner dataplane pod. Below we use the CPU utilization for -simplicity. - -Scaling STUNner *up* occurs by Kubernetes adding more pods to the STUNner dataplane deployment and -load-balancing client requests across the running pods. This should (theoretically) never interrupt -existing calls, but new calls should be automatically routed by the cloud load balancer to the new -endpoint(s). Automatic scale-up means that STUNner should never become the bottleneck in the -system. Note that in certain cases scaling STUNner up would require adding new Kubernetes nodes to -your cluster: most modern hosted Kubernetes services provide horizontal node autoscaling out of the -box to support this. - -Scaling STUNner *down*, however, is trickier. Intuitively, when a running STUNner dataplane pod is -terminated on scale-down, all affected clients with active TURN allocations on the terminating pod -would be disconnected. This would then require clients to go through an [ICE -restart](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/restartIce) to -re-connect, which may cause prolonged connection interruption and may not even be supported by all -browsers. - -In order to avoid client disconnects on scale-down, STUNner supports a feature called [graceful -shutdown](https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace). This -means that `stunnerd` pods would refuse to terminate as long as there are active TURN allocations -on them, and automatically remove themselves only once all allocations are deleted or timed out. It -is important that *terminating* pods will not be counted by the HorizontalPodAutoscaler towards the -average CPU load, and hence would not affect autoscaling decisions. In addition, new TURN -allocation requests would never be routed by Kubernetes to terminating `stunnerd` pods. - -Graceful shutdown enables full support for scaling STUNner down without affecting active client -connections. As usual, however, some caveats apply: -1. Currently the max lifetime for `stunnerd` to remain alive is 1 hour after being deleted: this - means that `stunnerd` will remain active only for 1 hour after it has been deleted/scaled-down - even if active allocations would last longer. You can always set this by adjusting the - `terminationGracePeriod` on your `stunnerd` pods. -2. STUNner pods may remain alive well after the last client connection goes away. This occurs when - an TURN-UDP allocation is left open by a client (spontaneous UDP client-side connection closure - cannot be reliably detected by the server). As the default TURN refresh lifetime is [10 - minutes](https://www.rfc-editor.org/rfc/rfc8656#section-3.2-3), it may take 10 minutes until all - allocations time out, letting `stunnerd` to finally terminate. -3. If there are active (or very recent) TURN allocations then the `stunnerd` pod may refuse to be - removed after a `kubectl delete`. Use `kubectl delete pod --grace-period=0 --force stunner-XXX` - to force removal. +It is a good practice to scale your STUNner deployment [horizontally](https://openmetal.io/docs/edu/openstack/horizontal-scaling-vs-vertical-scaling) (that is, by adding or removing `stunnderd` pods) instead of vertically (that is, by increasing the resource limits of your pods) when demand increases. We advice to set the [resource limits and requests](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) of the `stunnerd` pods to the bare minimum (this can be set in the [Dataplane](GATEWAY.md#dataplane) template used for provisioning `stunnerd` pods) and let Kubernetes to automatically scale out the STUNner dataplane by adding more `stunnerd` pods if needed. Note that HPA uses the [requested amount of resources](https://pauldally.medium.com/horizontalpodautoscaler-uses-request-not-limit-to-determine-when-to-scale-97643d808997) to determine when to scale-up or down the number of instances. + +STUNner comes with a full support for horizontal scaling using the the Kubernetes built-in [HorizontalPodAutoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale) (HPA). The triggering event can be based on arbitrary metric, say, the [number of active client connections](#MONITORING.md) per STUNner dataplane pod. Below we use the CPU utilization for simplicity. + +Scaling STUNner *up* occurs by Kubernetes adding more pods to the STUNner dataplane deployment and load-balancing client requests across the running pods. This should (theoretically) never interrupt existing calls, but new calls should be automatically routed by the cloud load balancer to the new endpoint(s). Automatic scale-up means that STUNner should never become the bottleneck in the system. Note that in certain cases scaling STUNner up would require adding new Kubernetes nodes to your cluster: most modern hosted Kubernetes services provide horizontal node autoscaling out of the box to support this. + +Scaling STUNner *down*, however, is trickier. Intuitively, when a running STUNner dataplane pod is terminated on scale-down, all affected clients with active TURN allocations on the terminating pod would be disconnected. This would then require clients to go through an [ICE restart](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/restartIce) to re-connect, which may cause prolonged connection interruption and may not even be supported by all browsers. + +In order to avoid client disconnects on scale-down, STUNner supports a feature called [graceful shutdown](https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace). This means that `stunnerd` pods would refuse to terminate as long as there are active TURN allocations on them, and automatically remove themselves only once all allocations are deleted or timed out. It is important that *terminating* pods will not be counted by the HorizontalPodAutoscaler towards the average CPU load, and hence would not affect autoscaling decisions. In addition, new TURN allocation requests would never be routed by Kubernetes to terminating `stunnerd` pods. + +Graceful shutdown enables full support for scaling STUNner down without affecting active client connections. As usual, however, some caveats apply: +1. The default is to provision `stunnerd` pods with at most 2 CPU cores and 16 listener threads, both can be customized in the [Dataplane](GATEWAY.md#dataplane) template used to provision `stunnerd` pods. +2. Currently the max lifetime for `stunnerd` to remain alive is 1 hour after being deleted: this means that `stunnerd` will remain active only for 1 hour after it has been deleted/scaled-down even if active allocations would last longer. You can adjust the grace period in the `terminationGracePeriod` setting in the [Dataplane](GATEWAY.md#dataplane) template. +3. STUNner pods may remain alive well after the last client connection is gone. This occurs when an allocation is left open by a client (e.g., spontaneous UDP client-side connection closure cannot be reliably detected by the server). As the default TURN refresh lifetime is [10 minutes](https://www.rfc-editor.org/rfc/rfc8656#section-3.2-3) it may take 10 minutes until all allocations time out, letting `stunnerd` to finally terminate. In such cases `stunnerd` may refuse to stop after a `kubectl delete`. Use `kubectl delete pod --grace-period=0 --force stunner-XXX` to force removal. ### Example -Below is a simple -[HorizontalPodAutoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) -config for autoscaling `stunnerd`. The example assumes that the [Kubernetes metric -server](https://github.com/kubernetes-sigs/metrics-server#installation) is available in the -cluster. +Below is a simple [HorizontalPodAutoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) config for autoscaling `stunnerd`. The example assumes that the [Kubernetes metric server](https://github.com/kubernetes-sigs/metrics-server#installation) is available in the cluster. ```yaml apiVersion: autoscaling/v2 @@ -103,9 +51,7 @@ spec: averageUtilization: 300 ``` -Here, `scaleTargetRef` selects the STUNner Deployment named `stunnerd` as the scaling target and -the deployment will always run at least 1 pod and at most 10 pods. Understanding how Kubernetes -chooses the number of running pods is, however, a bit tricky. +Here, `scaleTargetRef` selects the STUNner Deployment named `stunnerd` as the scaling target and the deployment will always run at least 1 pod and at most 10 pods. Understanding how Kubernetes chooses the number of running pods is, however, a bit tricky. Suppose that the configured resources in the STUNner deployment are the following. @@ -119,12 +65,5 @@ resources: memory: 128Mi ``` -Suppose that, initially, there is only a single `stunnerd` pod in the cluster. As new calls come -in, CPU utilization is increasing. Scale out will be triggered when CPU usage of the `stunnerd` pod -reaches 1500 millicore CPU (three times the requested CPU). If more calls come and the total CPU -usage of the `stunnerd` pods reaches 3000 millicore, which amounts to 1500 millicore on average, -scale out would happen again. When users leave, load will drop and the total CPU utilization will -fall under 3000 millicore. At this point Kubernetes will automatically scale-in and remove one of -the `stunnerd` instances. Recall, this would never affect existing connections thanks to graceful -shutdown. +Initially, there is only a single `stunnerd` pod in the cluster. As new calls arrive, CPU utilization is increasing. Scale out will be triggered when CPU usage of the `stunnerd` pod reaches 1500 millicore CPU (three times the requested CPU). If more calls come and the total CPU usage of the `stunnerd` pods reaches 3000 millicore, which amounts to 1500 millicore on average, scale out would happen again. When users leave, load will drop and the total CPU utilization will fall under 3000 millicore. At this point Kubernetes will automatically scale-in and remove one of the `stunnerd` instances. Recall, this would never affect existing connections thanks to graceful shutdown. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 49ba8fb0..2131a6c7 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -1,30 +1,22 @@ # Security -Like any conventional gateway service, an improperly configured STUNner service may easily end up -exposing sensitive services to the Internet. The below security guidelines will allow to minimize -the risks associated with a misconfigured STUNner gateway service. +Like any conventional gateway service, an improperly configured STUNner service may easily end up exposing sensitive services to the Internet. The below security guidelines will allow to minimize the risks associated with a misconfigured STUNner gateway service. ## Threat -Before deploying STUNner, it is worth evaluating the potential [security -risks](https://www.rtcsec.com/article/slack-webrtc-turn-compromise-and-bug-bounty) a poorly -configured public STUN/TURN server poses. To demonstrate the risks, below we shall use the -[`turncat`](cmd/turncat.md) utility and `dig` to query the Kubernetes DNS service through a -misconfigured STUNner gateway. +Before deploying STUNner, it is worth evaluating the potential [security risks](https://www.rtcsec.com/article/slack-webrtc-turn-compromise-and-bug-bounty) a poorly configured public STUN/TURN server poses. To demonstrate the risks, below we shall use the [`turncat`](cmd/turncat.md) utility and `dig` to query the Kubernetes DNS service through a misconfigured STUNner gateway. -Start with a [fresh STUNner installation](INSTALL.md) into an empty namespace called `stunner` -and apply the below configuration. +Start with a [fresh STUNner installation](INSTALL.md) into an empty namespace called `stunner` and apply the below configuration. ```console cd stunner kubectl apply -f deploy/manifests/stunner-expose-kube-dns.yaml ``` -This will open a STUNner Gateway at port UDP:3478 and add a UDPRoute with the Kubernetes cluster -DNS service as the backend: +This will open a STUNner Gateway called `udp-gateway` at port UDP:3478 and add a UDPRoute with the Kubernetes cluster DNS service as the backend: ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: stunner-udproute @@ -44,28 +36,21 @@ Learn the virtual IP address (`ClusterIP`) assigned by Kubernetes to the cluster export KUBE_DNS_IP=$(kubectl get svc -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].spec.clusterIP}') ``` -Build `turncat`, the Swiss-army-knife [testing tool](cmd/turncat.md) for STUNner, fire up a -UDP listener on `localhost:5000`, and forward all received packets to the cluster DNS service -through STUNner. +Build `turncat`, the Swiss-army-knife [testing tool](cmd/turncat.md) for STUNner, fire up a UDP listener on `localhost:5000`, and forward all received packets to the cluster DNS service through STUNner. ```console -./turncat --log=all:DEBUG udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://${KUBE_DNS_IP}:53 +./turncat --log=all:DEBUG udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener udp://${KUBE_DNS_IP}:53 ``` Now, in another terminal query the Kubernetes DNS service through the `turncat` tunnel. ```console -dig +short @127.0.0.1 -p 5000 stunner.default.svc.cluster.local +dig +short @127.0.0.1 -p 5000 kubernetes.default.svc.cluster.local ``` -You should see the internal Cluster IP address allocated by Kubernetes for the STUNner dataplane -service. Experiment with other FQDNs, like `kubernetes.default.svc.cluster.local`, etc.; the -Kubernetes cluster DNS service will readily return the the corresponding internal service IP -addresses. +You should see the internal Cluster IP address for the Kubernetes API server. -This little experiment demonstrates the threats associated with a poorly configured STUNner -gateway: it may allow external access to *any* UDP service running inside your cluster. The -prerequisites for this: +This little experiment should demonstrate the threats associated with a poorly configured STUNner gateway: it may allow external access to *any* UDP service running inside your cluster. The prerequisites for this: 1. the target service *must* run over UDP (e.g., `kube-dns`), 2. the target service *must* be wrapped with a UDPRoute @@ -76,7 +61,7 @@ Should any of these prerequisites fail, STUNner will block access to the target Now rewrite the backend service in the UDPRoute to an arbitrary non-existent service. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: stunner-udproute @@ -89,39 +74,23 @@ spec: - name: dummy ``` -Repeat the above `dig` command to query the Kubernetes DNS service again and observe how the query -times out. This demonstrates that a properly locked down STUNner installation blocks all access -outside of the backend services explicitly opened up via a UDPRoute. +Repeat the above `dig` command to query the Kubernetes DNS service again and observe how the query times out. This demonstrates that a properly locked down STUNner installation blocks all access outside of the backend services explicitly opened up via a UDPRoute. ## Locking down STUNner -Unless properly locked down, STUNner may be used maliciously to open a tunnel to any UDP service -running inside a Kubernetes cluster. Accordingly, it is critical to tightly control the pods and -services exposed via STUNner. +Unless properly locked down, STUNner may be used maliciously to open a tunnel to any UDP service running inside a Kubernetes cluster. Accordingly, it is critical to tightly control the pods and services exposed via STUNner. STUNner's basic security model is as follows: -> In a properly configured deployment, STUNner provides the same level of security as a media -server pool exposed to the Internet over a public IP address, protected by a firewall that admits -only UDP access. A malicious attacker, even possessing a valid TURN credential, can reach only the -media servers deployed behind STUNner, but no other services. +> In a properly configured deployment, STUNner provides the same level of security as a media server pool exposed to the Internet over a public IP address, protected by a firewall that admits only UDP access. A malicious attacker, even possessing a valid TURN credential, can reach only the media servers deployed behind STUNner, but no other services. -The below security considerations will greatly reduce this attack surface even further. In any -case, use STUNner at your own risk. +The below security considerations will greatly reduce this attack surface even further. In any case, use STUNner at your own risk. ## Authentication -By default, STUNner uses a single static username/password pair for all clients and the password is -available in plain text at the clients (`static` authentication mode). Anyone with access to the -static STUNner credentials can open a UDP tunnel via STUNner, provided that they know the private -IP address of the target service or pod and provided that a UDPRoute exists that specifies the -target service as a backend. This means that a service is exposed only if STUNner is explicitly -configured so. +By default, STUNner uses a single static username/password pair for all clients and the password is available in plain text at the clients (`static` authentication mode). Anyone with access to the static STUNner credentials can open a UDP tunnel via STUNner, provided that they know the private IP address of the target service or pod and provided that a UDPRoute exists that specifies the target service as a backend. This means that a service is exposed only if STUNner is explicitly configured so. -For more security sensitive workloads, we recommend the `ephemeral` authentication mode, which uses -per-client fixed lifetime username/password pairs. This makes it more difficult for attackers to -steal and reuse STUNner's TURN credentials. See the [authentication guide](AUTH.md) for configuring -STUNner with `ephemeral` authentication. +For production deployments we recommend the `ephemeral` authentication mode, which uses per-client fixed lifetime username/password pairs. This makes it more difficult for attackers to steal and reuse STUNner's TURN credentials. See the [authentication guide](AUTH.md) for configuring STUNner with `ephemeral` authentication. ## Access control @@ -130,7 +99,7 @@ a proper UDPRoute. For instance, the below UDPRoute allows access *only* to the service in the `media-plane` namespace, and nothing else. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: stunner-udproute @@ -141,12 +110,9 @@ spec: rules: - backendRefs: - name: media-server - - namespace: media-plane + namespace: media-plane ``` -> **Note** -To avoid potential misuse, STUNner disables open wildcard access to the entire cluster unless explicitly requested to do so by a [specifying an open StaticService as a backend](GATEWAY.md#staticservice). - For hardened deployments, it is possible to add a second level of isolation between STUNner and the rest of the workload using the Kubernetes NetworkPolicy facility. Creating a NetworkPolicy will essentially implement a firewall, blocking all access from the source to the target workload except the services explicitly whitelisted by the user. The below example allows access from STUNner to *any* media server pod labeled as `app=media-server` in the `default` namespace over the UDP port range `[10000:20000]`, but nothing else. ```yaml @@ -176,24 +142,12 @@ spec: endPort: 20000 ``` -Kubernetes network policies can be easily [tested](https://banzaicloud.com/blog/network-policy) -before exposing STUNner publicly; e.g., the [`turncat` utility](cmd/turncat.md) packaged with -STUNner can be used conveniently for this [purpose](examples/simple-tunnel/README.md). +Kubernetes network policies can be easily [tested](https://banzaicloud.com/blog/network-policy) before exposing STUNner publicly; e.g., the [`turncat` utility](cmd/turncat.md) packaged with STUNner can be used conveniently for this [purpose](examples/simple-tunnel/README.md). ## Exposing internal IP addresses -The trick in STUNner is that both the TURN relay transport address and the media server address are -internal pod IP addresses, and pods in Kubernetes are guaranteed to be able to connect -[directly](https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/#kubernetes-networking-model), -without the involvement of a NAT. This makes it possible to host the entire WebRTC infrastructure -over the private internal pod network and still allow external clients to make connections to the -media servers via STUNner. At the same time, this also has the bitter consequence that internal IP -addresses are now exposed to the WebRTC clients in ICE candidates. - -The threat model is that, possessing the correct credentials, an attacker can scan the *private* IP -address of all STUNner pods and all media server pods. This should not pose a major security risk -though: remember, none of these private IP addresses can be reached externally. The attack surface -can be further reduced to the STUNner pods' private IP addresses by using the [symmetric ICE -mode](DEPLOYMENT.md#symmetric-ice-mode). +The trick in STUNner is that both the TURN relay transport address and the media server address are internal pod IP addresses, and pods in Kubernetes are guaranteed to be able to connect [directly](https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/#kubernetes-networking-model) without the involvement of a NAT. This makes it possible to host the entire WebRTC infrastructure over the private internal pod network and still allow external clients to make connections to the media servers via STUNner. At the same time, this also has the bitter consequence that internal IP addresses are now exposed to the WebRTC clients in ICE candidates. + +The threat model is that, possessing the correct credentials, an attacker can scan the *private* IP address of all STUNner pods and all media server pods. This should pose no major security risk though: remember, none of these private IP addresses can be reached externally. The attack surface can be further reduced to the STUNner pods' private IP addresses by using the [symmetric ICE mode](DEPLOYMENT.md#symmetric-ice-mode). Nevertheless, if worried about information exposure then STUNner may not be the best option at the moment. In later releases, we plan to implement a feature to obscure the relay transport addresses returned by STUNner. Please file an issue if you think this limitation is a blocker for your use case. diff --git a/docs/WHY.md b/docs/WHY.md index ce632d1b..5a663018 100644 --- a/docs/WHY.md +++ b/docs/WHY.md @@ -9,43 +9,40 @@ used outside of this context (e.g., as a regular STUN/TURN server), but this is ## The problem -The main pain points STUNner is trying to solve are all related to that Kubernetes and WebRTC are +The pain points STUNner is trying to solve are all related to that Kubernetes and WebRTC are currently foes, not friends. Kubernetes has been designed and optimized for the typical HTTP/TCP Web workload, which makes streaming workloads, and especially UDP/RTP based WebRTC media, feel like a foreign citizen. Most importantly, Kubernetes runs the media server pods/containers over a private L3 network over a -private IP address and the network dataplane applies several rounds of Network Address Translation -(NAT) steps to ingest media traffic into this private pod network. Most cloud load-balancers apply -a DNAT step to route packets to a node and then an SNAT step to put the packet to the private pod +private IP address and the several rounds of Network Address Translation (NAT) steps are required +to ingest media traffic into this private pod network. Most cloud load-balancers apply a DNAT step +to route packets to a Kubernetes node and then an SNAT step to inject a packet into the private pod network, so that by the time a media packet reaches a pod essentially all header fields in the [IP 5-tuple](https://www.techopedia.com/definition/28190/5-tuple) are modified except the destination port. Then, if any pod sends the packet over to another pod via a Kubernetes service load-balancer then the packet will again undergo a DNAT step, and so on. -The *Kubernetes dataplane teems with NATs*. This is not a big deal for the usual HTTP/TCP web -protocols Kubernetes was designed for, since an HTTP/TCP session contains an HTTP header that fully -describes it. Once an HTTP/TCP session is accepted by a server it does not need to re-identify the -client per each received packet, because it has session context. - -This is not the case with the prominent WebRTC media protocol encapsulation though, RTP over -UDP. RTP does not have anything remotely similar to an HTTP header. Consequently, the only -"semi-stable" connection identifier WebRTC servers can use to identify a client is by expecting the -client's packets to arrive from a negotiated IP source address and source port. When the IP 5-tuple -changes, for instance because there is a NAT in the datapath, then WebRTC media connections -break. Due to reasons which are mostly historical at this point, *UDP/RTP connections do not -survive not even a single NAT step*, let alone the 2-3 rounds of NATs a packet regularly undergoes -in the Kubernetes dataplane. +The *Kubernetes dataplane teems with NATs*. This is not a big deal for the web protocols Kubernetes +was designed for, since each HTTP/TCP connection involves a session context that can be used by a +server to identify clients. This is not the case with WebRTC media protocol stack though, since +UDP/RTP connections do not involve anything remotely similar to an HTTP context. Consequently, the +only "semi-stable" connection identifier WebRTC servers can use to identify a client is by +expecting the client's packets to arrive from a negotiated IP source address and source port. When +the IP 5-tuple changes, for instance because there is a NAT in the datapath, then WebRTC media +connections break. Due to reasons which are mostly historical at this point, *UDP/RTP connections +do not survive not even a single NAT step*, let alone the 2-3 rounds of NATs a packet regularly +undergoes in the Kubernetes dataplane. ## The state-of-the-art The current stance is that the only way to deploy a WebRTC media server into Kubernetes is to exploit a [well-documented Kubernetes anti-pattern](https://kubernetes.io/docs/concepts/configuration/overview): *running the media -server pods in the host network namespace* (using the `hostNetwork=true` setting in the pod's -container template). This way the media server shares the network namespace of the host (i.e., the -Kubernetes node) it is running on, inheriting the public address (if any) of the host and -(hopefully) sidestepping the private pod network with the involved NATs. +server pods in the host network namespace* of Kubernetes nodes (using the `hostNetwork=true` +setting in the pod's container template). This way the media server shares the network namespace of +the host (i.e., the Kubernetes node) it is running on, inheriting the public address (if any) of +the host and (hopefully) sidestepping the private pod network with the involved NATs. There are *lots* of reasons why this deployment model is less than ideal: diff --git a/docs/examples/benchmark/performance-stunner.yaml b/docs/examples/benchmark/performance-stunner.yaml index 7f786748..bd6dc97b 100644 --- a/docs/examples/benchmark/performance-stunner.yaml +++ b/docs/examples/benchmark/performance-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -12,7 +12,7 @@ spec: description: "STUNner is a WebRTC ingress gateway for Kubernetes" --- -apiVersion: stunner.l7mp.io/v1alpha1 +apiVersion: stunner.l7mp.io/v1 kind: GatewayConfig metadata: name: stunner-gatewayconfig @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway @@ -36,7 +36,7 @@ spec: port: 9001 protocol: UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: iperf-server diff --git a/docs/examples/cloudretro/README.md b/docs/examples/cloudretro/README.md index 15130a57..e7e1b75b 100644 --- a/docs/examples/cloudretro/README.md +++ b/docs/examples/cloudretro/README.md @@ -106,7 +106,7 @@ can connect from behind even the most over-zealous enterprise NAT or firewall. ```console kubectl apply -f - < **Warning** +> [!WARNING] +> > In case of [managed mode](/docs/INSTALL.md), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. This will expose STUNner on a public IP on UDP port 3478. A Kubernetes `LoadBalancer` assigns an diff --git a/docs/examples/neko/stunner.yaml b/docs/examples/neko/stunner.yaml index 98d57d13..d0db2009 100644 --- a/docs/examples/neko/stunner.yaml +++ b/docs/examples/neko/stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -12,7 +12,7 @@ spec: description: "STUNner is a WebRTC ingress gateway for Kubernetes" --- -apiVersion: stunner.l7mp.io/v1alpha1 +apiVersion: stunner.l7mp.io/v1 kind: GatewayConfig metadata: name: stunner-gatewayconfig @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway @@ -37,7 +37,7 @@ spec: protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: neko-plane diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index 25337d15..f0b49adf 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -65,7 +65,7 @@ that the UDPRoute specifies the `iperf-server` service as the `backendRef`, whic STUNner will forward the client connections received in any of the Gateways to the iperf server. ```yaml -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway @@ -78,7 +78,7 @@ spec: protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: tcp-gateway @@ -91,7 +91,7 @@ spec: protocol: TURN-TCP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: iperf-server diff --git a/docs/examples/simple-tunnel/iperf-stunner.yaml b/docs/examples/simple-tunnel/iperf-stunner.yaml index d30ff989..f21e3ff7 100644 --- a/docs/examples/simple-tunnel/iperf-stunner.yaml +++ b/docs/examples/simple-tunnel/iperf-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -12,7 +12,7 @@ spec: description: "STUNner is a WebRTC ingress gateway for Kubernetes" --- -apiVersion: stunner.l7mp.io/v1alpha1 +apiVersion: stunner.l7mp.io/v1 kind: GatewayConfig metadata: name: stunner-gatewayconfig @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway @@ -37,7 +37,7 @@ spec: protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1beta1 +apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: tcp-gateway @@ -50,7 +50,7 @@ spec: protocol: TURN-TCP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: iperf-server diff --git a/docs/img/stunner_arch_big.svg b/docs/img/stunner_arch_big.svg index 9c4f54da..d659fa75 100644 --- a/docs/img/stunner_arch_big.svg +++ b/docs/img/stunner_arch_big.svg @@ -2,9 +2,9 @@ + + + + + + + + + + + + + transform="translate(-101.99988,-74.454498)"> + style="opacity:0.92;fill:#ffffff;stroke-width:0.264999;stroke-miterlimit:4;stroke-dasharray:none" + id="rect17965" + width="145.36844" + height="124.38023" + x="101.99988" + y="74.454498" /> + y="86.053436" /> GatewayClass + y="91.999321">GatewayClass + y="101.32552" /> GatewayConfig + y="107.27142">GatewayConfig + y="102.0098" /> Gateway + y="106.36816">Gateway + gw-ns/gw + + Deployment + + LB Service + + ConfigMap + y="118.40133" /> UDPRoute + y="124.34724">UDPRoute + y="134.04221" /> Service + y="139.98804">Service + y="134.11505" /> Service + y="140.06088">Service - STUNner - ConfigMap - - - stunnerd. - conf + x="119.60569" + y="157.10823" /> + y="118.31358" /> UDPRoute + y="124.25938">UDPRoute + y="134.02731" /> Service + y="139.97322">Service @@ -480,24 +543,31 @@ style="fill:none;stroke:#000000;stroke-width:0.2;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:50;stroke-dasharray:none;stroke-opacity:1;paint-order:fill markers stroke" id="rect11705" width="92.906738" - height="66.294807" + height="62.699524" x="145.10664" y="84.031242" /> + STUNner Gateway Hierarchy + x="145.85818" + y="82.809853">Gateway API + transform="matrix(0.7,0,0,0.7,27.506081,50.259234)" + style="opacity:0.92;stroke-width:1.42857"> stunnerd + x="161.94887" + y="160.68379">stunnerd + sodipodi:nodetypes="cc" /> + Render + x="126.29139" + y="101.20586">Control Map + x="134.42651" + y="88.119209">Watch Watch + x="105.49356" + y="173.49677">STUN/TURN + + UDP + + UDP + x="130.05907" + y="172.40094" /> + x="150.91322" + y="166.99174" /> Cluster + x="154.32851" + y="172.00801">Cluster Listener + x="133.41063" + y="177.0959">Listener + x="150.91328" + y="179.03008" /> Cluster + x="154.32864" + y="184.04634">Cluster + d="m 126.88437,161.82693 h 44.42234 a 3.0382178,4.4979167 0 0 1 3.03822,4.49792 v 19.75365 a 3.0382178,4.4979167 0 0 1 -3.03822,4.49792 h -44.42234 a 3.0382178,4.4979167 0 0 1 -3.03822,-4.49792 v -19.75365 a 3.0382178,4.4979167 0 0 1 3.03822,-4.49792 z" + style="opacity:0.92;fill:none;fill-opacity:1;stroke:#000000;stroke-width:0.285714;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:normal" /> + gw-ns/gw + gw-ns/gw + gw-ns/gw From 69b5865c9e9de928157eb2f7f61371378b2c8349 Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Tue, 2 Jan 2024 17:19:47 +0100 Subject: [PATCH 081/243] Add mediasoup demo to rtd --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index c34fb616..be3a8d85 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - 'Kurento Magic Mirror': 'examples/kurento-magic-mirror/README.md' - 'LiveKit': 'examples/livekit/README.md' - 'Jitsi': 'examples/jitsi/README.md' + - 'mediasoup': 'examples/mediasoup/README.md' - 'Cloud Retro': 'examples/cloudretro/README.md' - 'N.eko': 'examples/neko/README.md' - 'Benchmark': 'examples/benchmark/README.md' From fbf7fdf73c0fd47efbbf0e1e73087995bd310f2e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 2 Jan 2024 16:49:20 +0100 Subject: [PATCH 082/243] doc: Add standalone TURN server config --- deploy/manifests/stunner-standalone.yaml | 208 +++++++++-------------- 1 file changed, 76 insertions(+), 132 deletions(-) diff --git a/deploy/manifests/stunner-standalone.yaml b/deploy/manifests/stunner-standalone.yaml index e73d1823..229a09f5 100644 --- a/deploy/manifests/stunner-standalone.yaml +++ b/deploy/manifests/stunner-standalone.yaml @@ -1,147 +1,91 @@ -# Stunner: An ingress gateway for WebRTC --- -# STUN/TURN server config: Make sure to customize the below configurations, see the README.pm for -# more details -apiVersion: v1 -kind: ConfigMap +apiVersion: stunner.l7mp.io/v1 +kind: Dataplane metadata: - name: stunner-config - namespace: default -data: - # * `STUNNER_PUBLIC_ADDR` (no default): The public IP address clients can use to reach - # STUNner. By default, the public IP address will be dynamically assigned by the Kubernetes - # LoadBalancer service. The Helm installation script takes care of updating the configuration - # with the correct value. However, if installing from the static manifests then the external IP - # must be set manually. - STUNNER_PUBLIC_ADDR: "A.B.C.D" - - # * `STUNNER_PUBLIC_PORT` (default: 3478): The public port used by clients to reach STUNner. It - # is important that applications use the public port as found in the configuration, since the - # Helm installation scripts may overwrite this configuration. This occurs when the installation - # falls back to a NodePort service (i.e., when STUNner fails to obtain an external IP from the - # load-balancer). - STUNNER_PUBLIC_PORT: "3478" - - # * `STUNNER_PORT` (default: 3478): The internal port used by STUNner for communication inside - # the cluster. It is safe to set this to the public port. - STUNNER_PORT: "3478" - - # * `STUNNER_REALM` (default `stunner.l7mp.io`): the REALM used to guide the user agent in - # authenticating with STUNner. - STUNNER_REALM: "stunner.l7mp.io" - - # * `STUNNER_AUTH_TYPE` (default: `plaintext`): the STUN/TURN authentication mode, either - # "plaintext" over the username/password pair $STUNNER_USERNAME/$STUNNER_PASSWORD, or - # "longterm", using $STUNNER_SECRET. Make sure to customize! - STUNNER_AUTH_TYPE: "plaintext" - - # * `STUNNER_USERNAME` (default: `user`): the USERNAME attribute clients can use the authenticate - # with STUNner over plain-text authentication. Make sure to customize! - STUNNER_USERNAME: "user1" - - # * `STUNNER_PASSWORD` (default: `pass`): the password clients can use to authenticate with - # STUNner over plain-text authentication. Make sure to customize! - STUNNER_PASSWORD: "passwd1" - - # * `STUNNER_SHARED_SECRET`: the shared secret used for longterm authentication. - STUNNER_SHARED_SECRET: "my-shared-secret" - - # * `STUNNER_DURATION` (default: `86400`, i.e., one day): the lifetime of STUNner credentials - # * over longterm authentication. - STUNNER_DURATION: "86400" - - # * `STUNNER_LOGLEVEL` (default: `all:WARN`): the default log level used by the STUNner daemons. - STUNNER_LOGLEVEL: "all:INFO" - - # * `STUNNER_MIN_PORT` (default: 10000): smallest relay transport port assigned by STUNner. - STUNNER_MIN_PORT: "10000" - - # * `STUNNER_MAX_PORT` (default: 20000): highest relay transport port assigned by STUNner. - STUNNER_MAX_PORT: "20000" - + name: host-net +spec: + command: + - stunnerd + args: + - -w + - --udp-thread-num=16 + image: l7mp/stunnerd:dev + hostNetwork: true + resources: + limits: + cpu: 200m + memory: 512Mi + requests: + cpu: 100m + memory: 128Mi --- -apiVersion: apps/v1 -kind: Deployment +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass metadata: - name: stunner - namespace: default + name: stunner-gatewayclass spec: - selector: - matchLabels: - app: stunner - replicas: 1 - template: - metadata: - labels: - app: stunner - spec: - containers: - - name: stunnerd - image: l7mp/stunnerd:latest - imagePullPolicy: Always - command: ["stunnerd"] - args: ["-c", "/stunnerd.conf"] - # args: ["-c", "/stunnerd.conf"] - envFrom: - - configMapRef: - name: stunner-config - env: - - name: STUNNER_ADDR # we use the POD IP - valueFrom: - fieldRef: - fieldPath: status.podIP - # Uncomment this if you want to deploy a sidecar container with stunner to sniff traffic - # - name: net-debug - # image: l7mp/net-debug:latest - # command: ["/bin/sh"] - # args: ["-c", "while true; do echo hello; sleep 10;done"] - + controllerName: "stunner.l7mp.io/gateway-operator" + parametersRef: + group: "stunner.l7mp.io" + kind: GatewayConfig + name: stunner-gatewayconfig + namespace: stunner + description: "STUNner is a WebRTC media gateway for Kubernetes" --- -apiVersion: v1 -kind: Service +apiVersion: stunner.l7mp.io/v1 +kind: GatewayConfig metadata: - name: stunner - namespace: default - labels: - app: stunner + name: stunner-gatewayconfig + namespace: stunner spec: - ports: - - port: 3478 - targetPort: 3478 - # nodePort: 30478 - protocol: UDP - name: stunner-udp - type: LoadBalancer - selector: - app: stunner - + dataplane: host-net + realm: stunner.l7mp.io + authRef: + name: stunner-auth-secret + namespace: stunner --- apiVersion: v1 -kind: Service +kind: Secret +metadata: + name: stunner-auth-secret + namespace: stunner +type: Opaque +stringData: + type: static + username: user-1 + password: pass-1 +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: turn-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: turn-listener + port: 3478 + protocol: TURN-UDP +--- +apiVersion: stunner.l7mp.io/v1 +kind: UDPRoute metadata: - name: stunner-tcp - labels: - app: stunner + name: open-route + namespace: stunner spec: - ports: - - port: 3478 - targetPort: 3478 - protocol: TCP - name: stunner-tcp - type: LoadBalancer - selector: - app: stunner - + parentRefs: + - name: turn-gateway + rules: + - backendRefs: + - group: stunner.l7mp.io + kind: StaticService + name: wildcard-backend --- -# lock down access from the TURN server to anywhere! -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy +apiVersion: stunner.l7mp.io/v1 +kind: StaticService metadata: - name: stunner-network-policy - namespace: default + name: wildcard-backend + namespace: stunner spec: - podSelector: - matchLabels: - app: stunner - policyTypes: - - Egress + prefixes: + - "0.0.0.0/0" From c9e70b5524adf27dcfd52ee6c997768b25f9aa9e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 2 Jan 2024 16:49:43 +0100 Subject: [PATCH 083/243] doc: Add ToC to the config reference --- docs/GATEWAY.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 295ef4f1..5ed7ddcd 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -2,6 +2,13 @@ The [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) exposes the control plane configuration using the standard [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io). This allows to configure STUNner in the familiar YAML-engineering style via Kubernetes manifests. The below reference gives an overview of the subset of the Gateway API supported by STUNner, see [here](https://github.com/l7mp/stunner-gateway-operator#caveats) for a list of the most important simplifications. +1. [GatewayClass](#gatewayclass) +1. [GatewayConfig](#gatewayconfig) +1. [Gateway](#gateway) +1. [UDPRoute](#udproute) +1. [StaticService](#staticservice) +1. [Dataplane](#dataplane) + ## GatewayClass The GatewayClass resource provides the root of a STUNner gateway configuration. GatewayClass resources are cluster-scoped, so they can be attached to from any namespace. @@ -206,7 +213,7 @@ spec: UDPRoute resources can be attached to Gateways in order to specify the backend services permitted to be reached via the Gateway. Multiple UDPRoutes can attach to the same Gateway, and each UDPRoute can specify multiple backend services; in this case access to *all* backends in *each* of the attached UDPRoutes is allowed. An UDPRoute can be attached to a Gateway by setting the `parentRef` to the Gateway's name and namespace. This is, however, contingent on whether the Gateway accepts routes from the given namespace: customize the `allowedRoutes` per each Gateway listener to control which namespaces the listener accepts routes from. -The below UDPRoute will configure STUNner to route client connections received on the Gateway called `udp-gateway` to *any UDP port* on the pods of the media server pool identified by the Kubernetes service `media-server-pool` in the `media-plane` namespace. +The below UDPRoute will configure STUNner to route client connections received on the Gateway called `udp-gateway` to *any UDP port* on the pods of the media server pool identified by the Kubernetes service `media-server-pool` in the `media-plane` namespace. ```yaml apiVersion: stunner.l7mp.io/v1 @@ -323,10 +330,10 @@ spec: image: l7mp/stunnerd:latest resources: limits: - cpu: 500m + cpu: 2 memory: 512Mi requests: - cpu: 100m + cpu: 500m memory: 128Mi terminationGracePeriodSeconds: 3600 ``` From cd40fc0b3ead7eb6ab5cb7c83d464ba4e85150bd Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 2 Jan 2024 18:08:06 +0100 Subject: [PATCH 084/243] fix: Unbreak legacy config file watcher The config file watcher no longer emits a zero-config on startup. It is expected that the config discovery client will bootstrap itself with a zero-config if that is the intended behavior. --- config_test.go | 30 ------------------------------ pkg/config/client/file_client.go | 6 ------ 2 files changed, 36 deletions(-) diff --git a/config_test.go b/config_test.go index 251fc980..dac58b49 100644 --- a/config_test.go +++ b/config_test.go @@ -2,7 +2,6 @@ package stunner import ( "context" - "encoding/json" "fmt" "net" "net/http" @@ -17,7 +16,6 @@ import ( "sigs.k8s.io/yaml" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" - cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) @@ -187,13 +185,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { // // wait a bit so that the watcher has time to react // time.Sleep(50 * time.Millisecond) - // first read should yield a zeroconfig c2, ok := <-conf - assert.True(t, ok, "zeroconfig emitted") - checkZeroConfig(t, &c2, stunner.GetId()) - - // second read yields the real config - c2, ok = <-conf assert.True(t, ok, "config emitted") checkDefaultConfig(t, &c2, "TURN-UDP") @@ -285,13 +277,7 @@ func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { _, err = f.WriteString(testConfigV1) assert.NoError(t, err, "write config to temp file") - // first read should yield a zeroconfig c2, ok := <-conf - assert.True(t, ok, "zeroconfig emitted") - checkZeroConfig(t, &c2, stunner.GetId()) - - // second read yields the real config - c2, ok = <-conf assert.True(t, ok, "config emitted") assert.Equal(t, stnrv1.ApiVersion, c2.ApiVersion, "version") @@ -382,12 +368,6 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { return conn.WriteMessage(websocket.PongMessage, []byte("keepalive")) }) - // send initial config - z := cdsclient.ZeroConfig("ns1/tester") - j, err := json.Marshal(z) - assert.NoError(t, err, "json marshal zeroconfig") - assert.NoError(t, conn.WriteMessage(websocket.TextMessage, j), "write zeroconfig") - // send v1config assert.NoError(t, conn.WriteMessage(websocket.TextMessage, []byte(testConfigV1)), "write config v1") @@ -420,13 +400,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { log.Debug("init config poller") assert.NoError(t, stunner.WatchConfig(ctx, origin, conf), "creating config poller") - // first read should yield a zeroconfig c2, ok := <-conf - assert.True(t, ok, "zeroconfig emitted") - checkZeroConfig(t, &c2, "ns1/tester") - - // second read yields a v1 config - c2, ok = <-conf assert.True(t, ok, "config emitted") assert.Equal(t, stnrv1.ApiVersion, c2.ApiVersion, "version") @@ -532,7 +506,3 @@ func checkDefaultConfig(t *testing.T, c *stnrv1.StunnerConfig, proto string) { assert.Len(t, c.Clusters[0].Endpoints, 1, "cluster endpoint len") assert.Equal(t, "0.0.0.0/0", c.Clusters[0].Endpoints[0], "endpoint") } - -func checkZeroConfig(t *testing.T, c *stnrv1.StunnerConfig, id string) { - assert.True(t, c.DeepEqual(cdsclient.ZeroConfig(id)), "zeroconfig ok") -} diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index 9144b1aa..9e6bbe7c 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -55,12 +55,6 @@ func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerCo return errors.New("uninitialized config file path") } - // emit an empty config: this bootstraps stunner the default resources (above all, starts - // the health-checker) - w.log.Debug("bootstrapping with zero configuration") - initConf := ZeroConfig(w.id) - ch <- *initConf - go func() { for { // try to watch From 4167d92bab3c9ae9f4047de9dd46879a53801131 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 2 Jan 2024 20:18:33 +0100 Subject: [PATCH 085/243] doc: Update the simple-tunnel tutorial --- docs/GATEWAY.md | 28 +++---- docs/examples/simple-tunnel/README.md | 111 ++++++++++++++------------ 2 files changed, 73 insertions(+), 66 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 5ed7ddcd..5c401baf 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -312,7 +312,7 @@ The StaticService `spec.prefixes` must be a list of proper IPv4 prefixes: any IP ## Dataplane -The Dataplane resource is used as a template for provisioning of `stunnerd` pods. This is useful to choose the image origin and version, set custom command line arguments and environment variables to the `stunnerd` daemon, configure resource requests/limits, etc. +The Dataplane resource is used as a template for provisioning dataplane (`stunnerd`) pods that actually implement TURN media ingestion. This is useful to choose the `stunnerd` image origin and version, set custom command line arguments and environment variables, configure resource requests/limits, etc. Below is the `default` Dataplane installed by STUNner. @@ -338,26 +338,26 @@ spec: terminationGracePeriodSeconds: 3600 ``` -Below is a reference of the most important fields of the Dataplane `spec` that can be used to customize the provisioning of `stunnerd` pods. +The following fields can be set in the Dataplane `spec` to customize the provisioning of `stunnerd` pods. | Field | Type | Description | Required | |:--------------------------------|:----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| | `image` | `string` | The container image. | Yes | -| `imagePullPolicy` | `string` | Policy for if/when to pull an image, can be either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if `:latest` tag is specified on the image, or `IfNotPresent` otherwise. | No | -| `command` | `list` | Entrypoint array. Default: `stunnerd`. | No | -| `args` | `list` | Arguments to the entrypoint. | Yes | -| `envFrom` | `list` | List of sources to populate environment variables in the container. Default: empty. | No | -| `env` | `list` | List of environment variables to set in the container. Default: empty. | No | -| `replicas` | `int` | Number of `stunnerd` pods per Gateway to provision. Not enforced if the `stunnerd` Deployment replica count is overwritten manually or by an autoscaler. Default: 1. | No | -| `hostNetwork` | `bool` | Deploy `stunnerd` into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | -| `resources` | `object` | Compute resources required by `stunnerd`. Default: whatever Kubernetes assigns. | No | -| `affinity` | `object` | Scheduling constraints. Default: none. | No | -| `tolerations` | `object` | Tolerations. Default: none. | No | +| `imagePullPolicy` | `string` | Policy for if/when to pull an image, can be either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if the `latest` tag is specified on the image, or `IfNotPresent` otherwise. | No | +| `command` | `list` | Entrypoints for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). . | No | +| `args` | `list` | Commane line arguments for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | +| `envFrom` | `list` | List of sources to populate environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | +| `env` | `list` | List of environment variablesfor the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | +| `replicas` | `int` | Number of dataplane pods per Gateway to provision. Not enforced if the `stunnerd` Deployment replica count is overwritten, either manually or by an autoscaler. Default: 1. | No | +| `hostNetwork` | `bool` | Deploy the dataplane into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | +| `resources` | `object` | Compute resources per dataplane pod. | No | +| `affinity` | `object` | Scheduling constraints for the dataplane pods. Default: none. | No | +| `tolerations` | `object` | Tolerations for the dataplane pods. Default: none. | No | | `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | -| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | +| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | | `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | -There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the corresponding GatewayConfig to choose the Dataplane for each STUNner install. +There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the GatewayConfig to choose the Dataplane per each STUNner install. > [!WARNING] > diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index f0b49adf..da47f13a 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -111,93 +111,95 @@ spec: Check whether you have all the necessary STUNner resources installed namespace. ```console -kubectl get gatewayconfigs,gateways,udproutes -n stunner -NAME REALM AUTH AGE -gatewayconfig.stunner.l7mp.io/stunner-gatewayconfig stunner.l7mp.io plaintext 3m53s +kubectl get gatewayconfigs,gateways,udproutes.stunner.l7mp.io -n stunner +NAME REALM DATAPLANE AGE +gatewayconfig.stunner.l7mp.io/stunner-gatewayconfig stunner.l7mp.io default 139m -NAME CLASS ADDRESS READY AGE -gateway.gateway.networking.k8s.io/tcp-gateway stunner-gatewayclass True 14s -gateway.gateway.networking.k8s.io/udp-gateway stunner-gatewayclass True 14s +NAME CLASS ADDRESS PROGRAMMED AGE +gateway.gateway.networking.k8s.io/tcp-gateway stunner-gatewayclass 35.187.97.94 True 139m +gateway.gateway.networking.k8s.io/udp-gateway stunner-gatewayclass 35.205.10.190 True 139m -NAME AGE -udproute.gateway.networking.k8s.io/iperf-server 14s +NAME AGE +udproute.stunner.l7mp.io/iperf-server 139m ``` -You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. +You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration for the +UDP gateway. ``` console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext +cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway +STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 Listener 1 - Name: udp-listener - Listener: udp-listener - Protocol: UDP - Public address: 34.116.220.190 - Public port: 30501 -Listener 2 - Name: tcp-listener - Listener: tcp-listener - Protocol: TCP - Public address: 34.118.93.28 + Name: stunner/udp-gateway/udp-listener + Listener: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address: 35.205.10.190 Public port: 3478 ``` -NOTE: It usually takes 30-60 seconds for Kubernetes to assign an external IP address to STUNner -gateways. As long as the external address is in `` status, STUNner exposes the Gateway on -a NodePort: in the above example the UDP Gateway's `udp-listener` is exposed on a node IP -(`34.116.220.190`) and the NodePort 30501. Once Kubernetes finishes the exposition of the Gateway -service, STUNner picks up the new address/port and updates the config accordingly. The end -result should be something similar to the below; observe how the `udp-listener` public port has -changed to the requested port 3478 and the public address is updated as well. +Likewise, the below will dump the config for the TCP gateway. ``` console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -[...] +cmd/stunnerctl/stunnerctl running-config stunner/tcp-gateway +STUN/TURN authentication type: static +STUN/TURN username: user-1 +STUN/TURN password: pass-1 Listener 1 - Name: udp-listener - Listener: udp-listener - Protocol: UDP - Public address: 34.118.16.31 + Name: stunner/tcp-gateway/tcp-listener + Listener: stunner/tcp-gateway/tcp-listener + Protocol: TURN-TCP + Public address: 35.187.97.94 Public port: 3478 -[...] ``` +NOTE: It usually takes 30-60 seconds for Kubernetes to assign an external IP address to STUNner +gateways. As long as the external address is in `` status, STUNner exposes the Gateway on +a NodePort. Once Kubernetes finishes the exposition of the Gateway service, STUNner will pick up +the new address/port and update the config accordingly. + If in doubt, you can always query Kubernetes for the service statuses. ``` console kubectl get -n stunner services -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -stunner ClusterIP 10.120.4.118 3478/UDP 2d -tcp-gateway LoadBalancer 10.120.11.196 34.118.93.28 3478:30959/TCP 14h -udp-gateway LoadBalancer 10.120.3.228 34.118.16.31 3478:30501/UDP 6m42s +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +stunner ClusterIP 10.0.9.70 3478/UDP 15m +tcp-gateway LoadBalancer 10.0.3.91 35.187.97.94 3478:31781/TCP 143m +udp-gateway LoadBalancer 10.0.14.218 35.205.10.190 3478:31048/UDP 143m ``` ### Run the benchmark We will need to learn the ClusterIP assigned by Kubernetes to the `iperf-server` service: this will be the peer address to which `turncat` will ask STUNner to relay the iperf test traffic. + ``` console export IPERF_ADDR=$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}") ``` -Next, set up `turncat` to listen on `UDP:127.0.0.1:5000` and tunnel connections from this -listener via the STUNner STUN/TURN listener `udp-listener` to the iperf server. Luckily, `turncat` -is clever enough to [parse the running STUNner configuration](../../cmd/turncat) from Kubernetes and set -the STUN/TURN server public address/port and the authentication credentials -accordingly. +Next, set up `turncat` to listen on `UDP:127.0.0.1:5000` and tunnel connections from this listener +via the STUNner STUN/TURN listener `udp-listener` to the iperf server. Luckily, `turncat` is clever +enough to [parse the running STUNner configuration](../../cmd/turncat) from Kubernetes and set the +STUN/TURN server public address/port and the authentication credentials accordingly. + ``` console -./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener \ +./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener \ udp://$IPERF_ADDR:5001 ``` -Fire up an iperf client from another terminal to start the benchmark. +The most important part here is the TURN meta-URI: `k8s://stunner/udp-gateway:udp-listener` +instructs `turncat` to look for the Gateway called `udp-gateway` in the `stunner` namespace and +create a connection to the TURN listener called `udp-listener` of the Gateway. + +Fire up an iperf client from another terminal that will connect to STUNner via `turncat` and start +the benchmark. ```console iperf -c localhost -p 5000 -u -i 1 -l 100 -b 800000 -t 10 ``` If successful, the iperf server logs should contain the benchmark results. + ```console kubectl logs $(kubectl get pods -l app=iperf-server -o jsonpath='{.items[0].metadata.name}') ------------------------------------------------------------ @@ -207,21 +209,25 @@ UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 1] local 10.116.2.30%eth0 port 5001 connected with 10.116.1.21 port 56439 (peer 2.1.7) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS inP NetPwr +... [ 1] 0.0000-9.9204 sec 977 KBytes 807 Kbits/sec 1.426 ms 0/10003 (0%) 14.256/10.791/97.428/ 4.993 ms 1008 pps 1.40 KByte 7.07 ``` -The results show that we have managed to squeeze 1000 packets/sec through STUNner without packet -loss, at an average one-way latency of 14.2 ms and average jitter 1.426 ms. Not bad from a +The results show that we have managed to send 1000 packets/sec through STUNner to the iperf server +without packet loss, at an average one-way latency of 14.2 ms and 1.426 ms jitter. Not bad from a Kubernetes cluster running in some remote datacenter! -Repeating the test, this time with a STUN/TURN over TCP, casts a somewhat more negative -picture. Change the STUN/TURN URI in the `turncat` CLI to connect via the `tcp-listener`. +Repeating the test, this time with a STUN/TURN over TCP, casts a somewhat different picture. Notice +the new meta-URI: `k8s://stunner/tcp-gateway:tcp-listener` to select the TURN server exposed on +TCP for`turncat`. + ``` console -./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:tcp-listener \ +./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/tcp-gateway:tcp-listener \ udp://$IPERF_ADDR:5001 ``` Run the benchmark again at 10kpps and watch the logs. + ``` console iperf -c localhost -p 5000 -u -l 100 -b 8000000 -o /dev/null -t 10 && \ kubectl logs $(kubectl get pods -l app=iperf-server -o jsonpath='{.items[0].metadata.name}') | tail -n 1 @@ -229,13 +235,14 @@ iperf -c localhost -p 5000 -u -l 100 -b 8000000 -o /dev/null -t 10 && \ ``` It seems that average latency has jumped to 148 ms, with a max latency of close to 460 ms! That's -why you should try to [avoid TCP at all +why you should [avoid TCP at all cost](https://bloggeek.me/why-you-should-prefer-udp-over-tcp-for-your-webrtc-sessions) in real-time communications. ### Cleaning up Stop `turncat` and wipe all Kubernetes configuration. + ```console kubectl delete -f docs/examples/simple-tunnel/iperf-server.yaml kubectl delete -f docs/examples/simple-tunnel/iperf-stunner.yaml From 366ca069b5d78b5f7a95c5076709171ed98844cc Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 2 Jan 2024 21:03:21 +0100 Subject: [PATCH 086/243] doc: Update user manuals --- cmd/stunnerctl/README.md | 22 +++++++++------ cmd/stunnerd/README.md | 61 ++++++++++++---------------------------- cmd/turncat/README.md | 33 +++++++--------------- 3 files changed, 41 insertions(+), 75 deletions(-) diff --git a/cmd/stunnerctl/README.md b/cmd/stunnerctl/README.md index 87ba4c19..6982ded4 100644 --- a/cmd/stunnerctl/README.md +++ b/cmd/stunnerctl/README.md @@ -4,17 +4,21 @@ A CLI tool to simplify the interaction with STUNner. ## Usage -Dump the running config from a live STUNner deployment in human-readable format. +Dump the running config of a STUNner gateway in human-readable format. + +The below will select the Gateway called `tcp-gateway` in the `stunner` namespace: ```console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener: udp-listener -Protocol: UDP -Public address: 34.118.36.108 -Public port: 3478 +cmd/stunnerctl/stunnerctl running-config stunner/stunner-gateway +STUN/TURN authentication type: static +STUN/TURN username: user-1 +STUN/TURN password: pass-1 +Listener 1 + Name: stunner/tcp-gateway/tcp-listener + Listener: stunner/tcp-gateway/tcp-listener + Protocol: TURN-TCP + Public address: 35.187.97.94 + Public port: 3478 ``` ## License diff --git a/cmd/stunnerd/README.md b/cmd/stunnerd/README.md index b28382aa..5710efa7 100644 --- a/cmd/stunnerd/README.md +++ b/cmd/stunnerd/README.md @@ -4,14 +4,14 @@ The `stunnerd` daemon implements the STUNner gateway dataplane. The daemon supports two basic modes. For quick tests `stunnerd` can be configured as a TURN server by specifying a TURN network URI on the command line. For more complex scenarios, and especially -for use in a Kubernetes cluster, `stunnerd` can take configuration from a config file. In addition, -`stunnerd` implements a watch-mode, so that it can actively monitor the config file for updates -and, once the config file has changed, automatically reconcile the TURN server to the new -configuration. This mode is intended for use with the [STUNner Kubernetes gateway +for use in a Kubernetes cluster, `stunnerd` can take configuration from a config origin, that can +be any file or a remote server. In addition, `stunnerd` implements a watch-mode, so that it can +actively monitor the config origin for updates and automatically reconcile the TURN server to any +new configuration. This mode is intended for use with the [STUNner Kubernetes gateway operator](https://github.com/l7mp/stunner-gateway-operator): the operator watches the Kubernetes -[Gateway API](https://gateway-api.sigs.k8s.io) resources and renders the active control plane -configuration into a ConfigMap, which is then mapped into the `stunnerd` pod's filesystem so that -the daemon can pick up the latest configuration using the watch mode. +[Gateway API](https://gateway-api.sigs.k8s.io) resources, renders the active control plane +configuration per each `stunnerd` pod and dynamically updates the dataplane using STUNner's config +discovery service. ## Features @@ -26,6 +26,7 @@ the daemon can pick up the latest configuration using the watch mode. * Two authentication modes via the long-term STUN/TURN credential mechanism: `static` using a static username/password pair, and `ephemeral` with dynamically generated time-scoped credentials. +* Peer port range filtering. ## Getting Started @@ -39,36 +40,25 @@ go build -o stunnerd cmd/stunnerd/main.go ### Usage -The below command will open a `stunnerd` UDP listener at `127.0.0.1:5000`, set `plaintext` -authentication using the username/password pair `user1/passwrd1`, and raises the debug level to the -maximum. +The below command will open a `stunnerd` UDP listener at `127.0.0.1:5000`, set `static` authentication using the username/password pair `user1/passwrd1`, and raise the debug level to the maximum. ```console ./stunnerd --log=all:TRACE turn://user1:passwd1@127.0.0.1:5000 ``` -Alternatively, run `stunnerd` in verbose mode with the config file taken from -`cmd/stunnerd/stunnerd.conf`. Adding the flag `-w` will enable watch mode. +Alternatively, run `stunnerd` in verbose mode with the config file taken from `cmd/stunnerd/stunnerd.conf`. Adding the flag `-w` will enable watch mode. ```console ./stunnerd -v -w -c cmd/stunnerd/stunnerd.conf ``` -Type `./stunnerd` to see a short description of the command line arguments supported by `stunnerd`. +Type `./stunnerd` to see a short description of the most important command line arguments. -In practice, you'll rarely need to run `stunnerd` directly: just fire up the [prebuilt container -image](https://hub.docker.com/repository/docker/l7mp/stunnerd) in Kubernetes and you should be good -to go. +In practice, you'll rarely need to run `stunnerd` directly: just fire up the [prebuilt container image](https://hub.docker.com/repository/docker/l7mp/stunnerd) in Kubernetes and you should be good to go. Or better yet, [install](/docs/INSTALL.md) the STUNner Kubernetes gateway operator that will readily manage the `stunnerd` pods for each Gateway you create. ## Configuration -Using the below configuration, `stunnerd` will open 4 STUNner listeners: two for accepting -unencrypted connections at UDP/3478 and TCP/3478, and two for encrypted connections at TLS/TCP/3479 -and DTLS/UDP/3479. For easier debugging, the port for the transport relay connections opened by -`stunnerd` will be taken from [10000:19999] for the UDP listener, [20000:29999] for the TCP -listener, etc. The daemon will use `longterm` authentication, with the shared secret read from the -environment variable `$STUNNER_SHARED_SECRET` during initialization. The relay address is taken -from the `$STUNNER_ADDR` environment variable. +Using the below configuration, `stunnerd` will open 4 STUNner listeners: two for accepting unencrypted connections at UDP/3478 and TCP/3478, and two for encrypted connections at TLS/TCP/3479 and DTLS/UDP/3479. The daemon will use `ephemeral` authentication, with the shared secret taken from the environment variable `$STUNNER_SHARED_SECRET` during initialization. The relay address will be taken from the `$STUNNER_ADDR` environment variable. ``` yaml version: v1alpha1 @@ -86,48 +76,34 @@ static: address: "$STUNNER_ADDR" protocol: udp port: 3478 - minPort: 10000 - maxPort: 19999 - name: stunnerd-tcp address: "$STUNNER_ADDR" protocol: tcp port: 3478 - minPort: 20000 - maxPort: 29999 - name: stunnerd-tls + address: "$STUNNER_ADDR" protocol: tls port: 3479 - minPort: 30000 - maxPort: 39999 cert: "my-cert.cert" key: "my-key.key" - name: stunnerd-dtls + address: "$STUNNER_ADDR" protocol: dtls port: 3479 cert: "my-cert.cert" key: "my-key.key" - minPort: 40000 - maxPort: 49999 ``` ## Advanced features ### TURN/UDP listener CPU scaling -STUNner can run multiple parallel readloops for TURN/UDP listeners, which allows it to scale to any -practical number of CPUs and brings massive performance improvements on UDP workloads. This is -achieved by creating a configurable number of UDP server sockets using the `SO_REUSEPORT` socket -option and spawn a separate goroutine to run a parallel readloop per each listener. The kernel will -load-balance allocations across the sockets/readloops per the IP 5-tuple, therefore the same -allocation will always stay at the same CPU which is important for correct operations. +STUNner can run multiple parallel readloops for TURN/UDP listeners, which allows it to scale to practically any number of CPUs and brings massive performance improvements for UDP workloads. This can be achieved by creating a configurable number of UDP server sockets using the `SO_REUSEPORT` socket option and spawning a separate goroutine to run a parallel readloop per each listener. The kernel will load-balance allocations across the sockets/readloops per the IP 5-tuple, thus the same allocation will always stay at the same CPU. This is important for correct TURN operations. -The feature is exposed via the command line flag `--udp-thread-num=`. The below -starts `stunnerd` watching the config file in `/etc/stunnerd/stunnerd.conf` using 32 parallel UDP -readloops (the default is 16). +The feature is exposed via the command line flag `--udp-thread-num=`. The below starts `stunnerd` watching the config file in `/etc/stunnerd/stunnerd.conf` using 32 parallel UDP readloops (the default is 16). ``` sh ./stunnerd -w -c /etc/stunnerd/stunnerd.conf --udp-thread-num=32 - ``` ## License @@ -138,5 +114,4 @@ MIT License - see [LICENSE](../../LICENSE) for full text. ## Acknowledgments -Initial code adopted from [pion/stun](https://github.com/pion/stun) and -[pion/turn](https://github.com/pion/turn). +Initial code adopted from [pion/stun](https://github.com/pion/stun) and [pion/turn](https://github.com/pion/turn). diff --git a/cmd/turncat/README.md b/cmd/turncat/README.md index 9d4d5f2d..9b0e6dda 100644 --- a/cmd/turncat/README.md +++ b/cmd/turncat/README.md @@ -1,9 +1,6 @@ # turncat: Swiss-army-knife testing tool for STUNner -`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote -address/port. The main use is to open a local tunnel endpoint to any service running inside a -Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl proxy`, but it -uses STUN/TURN to enter the cluster. +`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster. ## Getting Started @@ -18,33 +15,24 @@ go build -o turncat cmd/turncat/main.go ### Usage -Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets -through the TURN server located at `192.0.2.1:3478` to the UDP server located at -`192.0.2.2:53`. Use the longterm STUN/TURN credential mechanism to authenticate with the TURN -server and set the user/passwd to `test/test`: +Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP listener located at `192.0.2.2:53`. Use the [`static` STUN/TURN credential mechanism](/docs/AUTH.md) to authenticate with the TURN server and set the user/passwd to `test/test`: ```console -./turncat --log=all:INFO,turncat:DEBUG udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478 udp://192.0.2.2:53 +./turncat --log=all:INFO,turncat:DEBUG udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478 \ + udp://192.0.2.2:53 ``` -TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accept self-signed TLS -certificates and `--verbose` is equivalent to setting all `turncat` loggers to DEBUG mode (`-l -all:DEBUG`). +TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accept self-signed TLS certificates and `--verbose` is equivalent to setting all `turncat` loggers to DEBUG mode (`-l all:DEBUG`). ```console -./turncat --verbose --insecure udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 +./turncat --verbose --insecure udp://127.0.0.1:5000 \ + turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 ``` -Alternatively, specify the special TURN server URI `k8s://stunner/stunnerd-config:udp-listener` to -let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI -directs `turncat` to read the STUNner config from the ConfigMap named `stunnerd-config` in the -`stunner` namespace, and connect to the STUNner listener named `udp-listener`. The CLI flag `-` -instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent -via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will -enable verbose logging. +Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace, and connect to the TURN listener named `udp-listener`. The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will enable verbose logging. ```console -./turncat -v - k8s://stunner/stunnerd-config:udp-listener udp://10.0.0.1:9001 +./turncat -v - k8s://stunner/udp-gateway:udp-listener udp://10.0.0.1:9001 ``` ## License @@ -55,5 +43,4 @@ MIT License - see [LICENSE](../../LICENSE) for full text. ## Acknowledgments -Initial code adopted from [pion/stun](https://github.com/pion/stun) and -[pion/turn](https://github.com/pion/turn). +Initial code adopted from [pion/stun](https://github.com/pion/stun) and [pion/turn](https://github.com/pion/turn). From 41ac3b9cdd7255d56d1bbd4ca6a3b161bf699e72 Mon Sep 17 00:00:00 2001 From: Valniae <74309145+bbalint105@users.noreply.github.com> Date: Wed, 3 Jan 2024 14:46:00 +0100 Subject: [PATCH 087/243] doc: Update CloudRetro tutorial (#109) --- docs/examples/cloudretro/README.md | 14 ++++---------- .../cloudretro/cloudretro-stunner-cleanup.yaml | 2 +- docs/examples/cloudretro/coordinator-config.sh | 2 +- docs/examples/cloudretro/stunner-gwcc.yaml | 2 +- .../cloudretro/stunner-setup-for-cloudretro.sh | 2 +- 5 files changed, 8 insertions(+), 14 deletions(-) diff --git a/docs/examples/cloudretro/README.md b/docs/examples/cloudretro/README.md index e7e1b75b..e0d10442 100644 --- a/docs/examples/cloudretro/README.md +++ b/docs/examples/cloudretro/README.md @@ -79,14 +79,8 @@ the CloudRetro servers running on a private pod IP address. That is where STUNne ### STUNner -Use the official [Helm charts](../../INSTALL.md#installation) to install STUNner. - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace stunner-system -helm install stunner stunner/stunner --namespace stunner -``` +You will need to install Stunner on Your cluster, you can do this by following the official [installation guide](../../INSTALL.md#installation). +Wait until all the necessary resources are up and running, then you are ready to continue. Next, register STUNner with the Kubernetes Gateway API. @@ -94,7 +88,7 @@ Next, register STUNner with the Kubernetes Gateway API. kubectl apply -f stunner-gwcc.yaml ``` -The default configuration uses the `plaintext` STUN/TURN authentication mode with the +The default configuration uses the `static` STUN/TURN authentication mode with the username/password pair `user-1/pass-1`; make sure to [customize](../../AUTH.md) these defaults. Next, we expose the CloudRetro media services over STUNner. The below Gateway specification will @@ -116,7 +110,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP EOF ``` diff --git a/docs/examples/cloudretro/cloudretro-stunner-cleanup.yaml b/docs/examples/cloudretro/cloudretro-stunner-cleanup.yaml index 307ca0b9..a7cea0b3 100644 --- a/docs/examples/cloudretro/cloudretro-stunner-cleanup.yaml +++ b/docs/examples/cloudretro/cloudretro-stunner-cleanup.yaml @@ -9,7 +9,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: stunner.l7mp.io/v1 kind: UDPRoute diff --git a/docs/examples/cloudretro/coordinator-config.sh b/docs/examples/cloudretro/coordinator-config.sh index 024dccd0..95ac08b6 100755 --- a/docs/examples/cloudretro/coordinator-config.sh +++ b/docs/examples/cloudretro/coordinator-config.sh @@ -24,7 +24,7 @@ done username=$(kubectl get gatewayconfig -n stunner stunner-gatewayconfig -o jsonpath='{.spec.userName}' --context $secondary_context) credential=$(kubectl get gatewayconfig -n stunner stunner-gatewayconfig -o jsonpath='{.spec.password}' --context $secondary_context) -gwip=$(kubectl get service -n stunner stunner-gateway-udp-gateway-svc -o jsonpath='{.status.loadBalancer.ingress[0].ip}' --context $secondary_context) +gwip=$(kubectl get service -n stunner udp-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}' --context $secondary_context) kubectl patch configmap -n cloudretro cloudretro-config-c --context $primary_context --patch-file=/dev/stdin <<-EOF data: diff --git a/docs/examples/cloudretro/stunner-gwcc.yaml b/docs/examples/cloudretro/stunner-gwcc.yaml index 74b7ceb2..bfb46182 100644 --- a/docs/examples/cloudretro/stunner-gwcc.yaml +++ b/docs/examples/cloudretro/stunner-gwcc.yaml @@ -19,6 +19,6 @@ metadata: namespace: stunner spec: realm: stunner.l7mp.io - authType: plaintext + authType: static userName: "user-1" password: "pass-1" diff --git a/docs/examples/cloudretro/stunner-setup-for-cloudretro.sh b/docs/examples/cloudretro/stunner-setup-for-cloudretro.sh index 8a3323e1..4b7a2ad8 100644 --- a/docs/examples/cloudretro/stunner-setup-for-cloudretro.sh +++ b/docs/examples/cloudretro/stunner-setup-for-cloudretro.sh @@ -23,7 +23,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP EOF kubectl apply $context -f - < Date: Wed, 3 Jan 2024 18:55:03 +0100 Subject: [PATCH 088/243] doc: Kurento examples updated --- docs/examples/direct-one2one-call/README.md | 90 ++++++------ docs/examples/kurento-one2one-call/README.md | 141 +++++++++---------- 2 files changed, 113 insertions(+), 118 deletions(-) diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index 538b6010..265f09a2 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -66,7 +66,7 @@ change the ICE configuration. We solve this problem by (1) generating a new ICE configuration every time a new client registers with the application server and (2) sending the ICE configuration back to the client in the -`regiterResponse` message. Note that this choice is suboptimal for time-locked STUNner +`registerResponse` message. Note that this choice is suboptimal for time-locked STUNner authentication modes (i.e., the `ephemeral` mode, see below), because clients' STUN/TURN credentials might expire by the time they decide to connect. It is up to the application server developer to make sure that clients' ICE server configuration is periodically updated. @@ -123,10 +123,9 @@ application to generate an ICE config for each client. [...] ``` -1. Modify the application server code to query the STUNner authentication server every time a a - valid ICE config in needed. In particular, the code will return the ICE configuration before - returning a `registerResponse` to the client, so that the generated ICE configuration can be - piggy-backed on the response message. +1. Modify the application server code to query the STUNner authentication server every time a valid + ICE config in needed. In particular, the code will generate a new ICE configuration before + returning a `registerResponse` to the client and piggyback it on the response message. ```js function register(id, name, ws, callback) { @@ -235,9 +234,11 @@ spec: For later convenience we also create a TCP Gateway that runs on port TCP:3478. -In order to realize the headless deployment model, we set STUNner's own service as the backend in -the UDPRoute. This way, STUNner will loop back client connections to itself. The rest, that is, -cross-connecting the clients' media streams between each other, is just pure TURN magic. +Observe that we set STUNner's own service as the backend in the UDPRoute. This way, STUNner will be +allowed to loop back client connections to itself. This is required by the headless model, since in +this case the only way for clients to connect is via the transport relay connections created by +STUNner. The rest, that is, cross-connecting the clients' media streams between each other, is just +pure TURN magic. Here is the corresponding UDPRoute. Note that the route attaches itself to both the UDP and the TCP Gateway, so no matter on which gateway the client connects via our UDPRoute will apply. @@ -270,36 +271,31 @@ kubectl expose deployment -n stunner stunner --port 3478 --protocol UDP Check whether you have all the necessary objects installed into the `stunner` namespace. ```console -kubectl get gatewayconfigs,gateways,udproutes -n stunner -NAME REALM AUTH AGE -gatewayconfig.stunner.l7mp.io/stunner-gatewayconfig stunner.l7mp.io plaintext 4s +kubectl get gatewayconfigs,gateways,udproutes.stunner.l7mp.io -n stunner +NAME REALM DATAPLANE AGE +gatewayconfig.stunner.l7mp.io/stunner-gatewayconfig stunner.l7mp.io default 2m50s -NAME CLASS ADDRESS READY AGE -gateway.gateway.networking.k8s.io/tcp-gateway stunner-gatewayclass True 4s -gateway.gateway.networking.k8s.io/udp-gateway stunner-gatewayclass True 4s +NAME CLASS ADDRESS PROGRAMMED AGE +gateway.gateway.networking.k8s.io/tcp-gateway stunner-gatewayclass 34.118.104.235 True 2m50s +gateway.gateway.networking.k8s.io/udp-gateway stunner-gatewayclass 34.116.255.91 True 2m50s -NAME AGE -udproute.gateway.networking.k8s.io/stunner-headless 3s +NAME AGE +udproute.stunner.l7mp.io/stunner-headless 2m50s ``` -You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. +You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. For +instance, below is the human-readable config of the Gateway called `udp-gateway`: ``` console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config +cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 Listener 1 - Name: udp-listener - Listener: udp-listener + Name: stunner/udp-gateway/udp-listener + Listener: stunner/udp-gateway/udp-listener Protocol: TURN-UDP - Public address: 34.118.82.225 - Public port: 3478 -Listener 2 - Name: tcp-listener - Listener: tcp-listener - Protocol: TURN-TCP - Public address: 34.118.89.139 + Public address: 34.116.255.91 Public port: 3478 ``` @@ -332,23 +328,23 @@ with STUNner. ```js { - "id": "registerResponse", - "response": "accepted", - "iceConfiguration": { - "iceServers": [ - { - "url": "turn:34.118.82.225:3478?transport=UDP", - "username": "user-1", - "credential": "pass-1" - }, - { - "url": "turn:34.118.8.3:3478?transport=TCP", - "username": "user-1", - "credential": "pass-1" - } - ], - "iceTransportPolicy": "relay" - } + "id": "registerResponse", + "response": "accepted", + "iceConfiguration": { + "iceServers": [ + { + "urls": ["turn:34.118.104.235:3478?transport=tcp"], + "username": "user-1", + "credential": "pass-1" + }, + { + "urls": ["turn:34.116.255.91:3478?transport=udp"], + "username": "user-1", + "credential": "pass-1" + } + ], + "iceTransportPolicy": "relay" + } } ``` @@ -361,10 +357,10 @@ with STUNner. them over verbatim to the other client. ```console - Sending message: {[...] "candidate:0 1 UDP 91889663 10.116.1.21 36930 typ relay raddr 10.116.1.21 rport 36930" [...]} + Sending message: {[...] "candidate:0 1 UDP 92020735 10.76.0.15 44703 typ relay raddr 10.76.0.15 rport 44703" [...]} ``` - Observe that the ICE candidate contains a private IP address (`10.116.1.21` in this case) as the + Observe that the ICE candidate contains a private IP address (`10.76.0.15` in this case) as the TURN relay connection address: this just happens to be the IP address of the STUNner pod that receives the TURN allocation request from the browser. @@ -373,7 +369,7 @@ with STUNner. parties will have a set of local and remote ICE candidates they can probe for connectivity. ```console - Received message: { [...] "candidate:0 1 UDP 91889663 10.116.1.21 36930 typ relay raddr 10.116.1.21 rport 36930" [...]} + Received message: {[...] "candidate:0 1 UDP 92020735 10.76.0.15 57511 typ relay raddr 10.76.0.15 rport 57511", [...]} ``` - Once ICE candidates are exchanged, the browsers perform a connectivity check on each diff --git a/docs/examples/kurento-one2one-call/README.md b/docs/examples/kurento-one2one-call/README.md index bff6cb3a..cf7fc06b 100644 --- a/docs/examples/kurento-one2one-call/README.md +++ b/docs/examples/kurento-one2one-call/README.md @@ -55,14 +55,13 @@ PeerConnection, generates an SDP Offer and sends it back the application server At this point the application server has both party's SDP Offer, so the next step is to set up the media pipeline in Kurento and process the SDP Offers through the media server. This is done by -opening a WebSocket connection to the URI `ws://kms.default.svc.cluster.local:8888/kurento` -(this is set on the server's command line in the Deployment -[manifest](kurento-one2one-call-server.yaml)): here, -`kms.default.svc.cluster.local` is the DNS name assigned by Kubernetes to the `kms` service -(recall, this is the service associated with the media server Deployment) and Kurento listens on -the TCP port 8888 for control connections. Note that this call gets load-balanced through the -Kubernetes CNI's service load-balancer so it will hit one random media server replica (if there are -more). This ensures that new calls are distributed evenly across media servers. +opening a WebSocket connection to the URI `ws://kms.default.svc.cluster.local:8888/kurento` (this +is set on the server's command line in the Deployment +[manifest](kurento-one2one-call-server.yaml)): here, `kms.default.svc.cluster.local` is the DNS +name assigned by Kubernetes to the `kms` service and Kurento listens on the TCP port 8888 for +control connections. Note that this call gets load-balanced through the Kubernetes CNI's service +load-balancer so it will hit one random media server replica (if there are more). This ensures that +new calls are distributed evenly across media servers. The media server responds with an SDP Answer for both the callee and the caller, which the application server immediately sends back to the appropriate clients: the caller receives the SDP @@ -79,13 +78,12 @@ documentation](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutor In order to start the ICE conversation using STUNner as the STUN/TURN server, the browsers will need to learn an ICE server configuration from the application server with STUNner's external IP -addresses/ports and the required STUN/TURN credentials. This must happen *before* the -PeerConnection is created in the clients: once the PeerConnection is running we can no longer -change the ICE configuration. +addresses/ports and STUN/TURN credentials. This must happen *before* the PeerConnection is created +in the clients: once the PeerConnection is running we can no longer change the ICE configuration. We solve this problem by (1) generating a new ICE configuration every time a new client registers with the application server and (2) sending the ICE configuration back to the client in the -`regiterResponse` message. Note that this choice is suboptimal for time-locked STUNner +`registerResponse` message. Note that this choice is suboptimal for time-locked STUNner authentication modes (i.e., the `ephemeral` mode, see below), because clients' STUN/TURN credentials might expire by the time they decide to connect. It is up to the application server developer to make sure that clients' ICE server configuration is periodically updated. @@ -256,9 +254,9 @@ to modify *anything* in the demo and it will just work. ### STUNner configuration -Next, we deploy STUNner into the Kubernetes. The manifest below will set up a minimal STUNner -gateway hierarchy to do just that: the setup includes a Gateway listener at UDP:3478 and a -UDPRoute to forward incoming calls into the cluster. +Next, we deploy STUNner into Kubernetes. The manifest below will set up a minimal STUNner gateway +hierarchy to do just that: the setup includes a Gateway listener at UDP:3478 and a UDPRoute to +forward incoming calls into the cluster. ```console kubectl apply -f docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml @@ -279,17 +277,15 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP ``` -In order to realize the media-plane deployment model we set the `kms` service, which wraps the -Kurento media server deployment, as the target in the UDPRoute. Note that the target service lives -in another namespace (the UDPRoute is in `stunner` whereas the `kms` service is in the `default` -namespace), STUNner will still be able to forward connections (this is a small departure from the -[Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) spec, which requires you to install a -TargetRef into the target namespace; currently STUNner ignores this for simplicity). The rest, that -is, cross-connecting the clients' media streams with Kurento's WebRTC endpoints, is just pure TURN -magic. +We set the `kms` service, which wraps the Kurento media server deployment, as the target in the +UDPRoute. Note that the target service lives in another namespace (the UDPRoute is in `stunner` +whereas the `kms` service is in the `default` namespace), STUNner will still be able to forward +connections (this is a small departure from the [Kubernetes Gateway +API](https://gateway-api.sigs.k8s.io) spec). The rest, that is, cross-connecting the clients' media +streams with Kurento's WebRTC endpoints, is just pure TURN magic. Below is the corresponding UDPRoute. @@ -312,29 +308,29 @@ spec: Check whether you have all the necessary objects installed into the `stunner` namespace. ```console -kubectl get gatewayconfigs,gateways,udproutes -n stunner -NAME REALM AUTH AGE -gatewayconfig.stunner.l7mp.io/stunner-gatewayconfig stunner.l7mp.io plaintext 95m +kubectl get gatewayconfigs,gateways,udproutes.stunner.l7mp.io -n stunner +NAME REALM DATAPLANE AGE +gatewayconfig.stunner.l7mp.io/stunner-gatewayconfig stunner.l7mp.io default 84s -NAME CLASS ADDRESS READY AGE -gateway.gateway.networking.k8s.io/udp-gateway stunner-gatewayclass True 95m +NAME CLASS ADDRESS PROGRAMMED AGE +gateway.gateway.networking.k8s.io/udp-gateway stunner-gatewayclass 34.118.112.176 True 84s -NAME AGE -udproute.gateway.networking.k8s.io/kms-media-plane 95m +NAME AGE +udproute.stunner.l7mp.io/kms-media-plane 84s ``` You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. ```console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext +cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway +STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 Listener 1 - Name: udp-listener - Listener: udp-listener - Protocol: UDP - Public address: 34.118.18.210 + Name: stunner/udp-gateway/udp-listener + Listener: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address: 34.118.112.176 Public port: 3478 ``` @@ -366,14 +362,18 @@ with STUNner. ```js { - "iceServers": [ - { - "url": "turn:34.118.18.210:3478?transport=UDP", - "username": "user-1", - "credential": "pass-1" - } - ], - "iceTransportPolicy": "relay" + "id": "registerResponse", + "response": "accepted", + "iceConfiguration": { + "iceServers": [ + { + "urls": ["turn:34.118.112.176:3478?transport=udp"], + "username": "user-1", + "credential": "pass-1" + } + ], + "iceTransportPolicy": "relay" + } } ``` @@ -386,10 +386,10 @@ with STUNner. in turn passes them over verbatim to Kurento. ```console - Sending message: {[...] "candidate:0 1 UDP 91889663 10.116.1.42 51510 typ relay raddr 10.116.1.42 rport 51510" [...]} + Sending message: {[...] "candidate:0 1 UDP 92020735 10.76.0.19 49944 typ relay raddr 10.76.0.19 rport 49944" [...]} ``` - Observe that the ICE candidate contains a private IP address (`10.116.1.42` in this case) as the + Observe that the ICE candidate contains a private IP address (`10.76.0.19` in this case) as the TURN relay connection address: this just happens to be the IP address of the STUNner pod that receives the TURN allocation request from the browser. @@ -398,11 +398,11 @@ with STUNner. clients as remote ICE candidates. ```console - Received message: {[...] "candidate:1 1 UDP 2015363327 10.116.2.44 17325 typ host" [...]} + Received message: {[...] "candidate:1 1 UDP 2015363327 10.76.0.17 13081 typ host" [...]} ``` - Observe that the ICE candidate again contains a private IP: in fact, `10.116.2.44` is the pod IP - address belonging to the Kurento media server instance that received the call setup request from + Observe that the ICE candidate again contains a private IP: in fact, `10.76.0.17` is the pod IP + address that belongs to the Kurento media server pod that received the call setup request from the application server. - Once ICE candidates are exchanged, both clients have a set of local and remote ICE candidates @@ -443,12 +443,11 @@ applications with STUNner. ## Update STUN/TURN credentials As exemplified by `stunnerctl` output, STUNner currently runs with fairly poor security: using -`static` authentication (note that `static` is an alias to the legacy `plaintext` authentication -type you see above), sharing a single username/password pair between all active sessions. +`static` authentication, sharing a single username/password pair between all active sessions. ``` console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext +cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway +STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 ... @@ -466,13 +465,12 @@ In other words, *STUNner's default security model is exactly the same as if we p servers and media servers on public-facing physical servers*. Still, it would be nice to use per-session passwords. STUNner allows you to do that, by changing -the authentication type to `ephemeral` (the legacy alias is `longterm`, but this is deprecated) -instead of `static`. Even better: STUNner's ephemeral TURN credentials are valid only for a -specified time (one day by default, but you can override this querying the [authentication -service](https://github.com/l7mp/stunner-auth-service)), after which they expire and attackers can -no longer reuse them. And to make things even better we don't even have to work too much to switch -STUNner to the `ephemeral` authentication mode: it is enough to update the GatewayConfig and -everything should happen from this point automagically. +the authentication type to `ephemeral` instead of `static`. Even better: STUNner's ephemeral TURN +credentials are valid only for a specified time (one day by default, but you can override this via +the [authentication service](https://github.com/l7mp/stunner-auth-service)), after which they +expire and attackers can no longer reuse them. And to make things even better we don't even have to +work too much to switch STUNner to the `ephemeral` authentication mode: it is enough to update the +GatewayConfig and everything should happen from this point automagically. ```console kubectl apply -f - < Date: Wed, 3 Jan 2024 20:01:01 +0100 Subject: [PATCH 089/243] doc: Update neko example Changes: - update steps for the upcoming v0.17 release - remove support for legacy mode - wait for the neko deployment to become available Tested on k8s v1.28.3 (minikube). --- docs/examples/neko/README.md | 9 ++------- docs/examples/neko/stunner.yaml | 4 +--- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/docs/examples/neko/README.md b/docs/examples/neko/README.md index 75d070f9..84b31ff1 100644 --- a/docs/examples/neko/README.md +++ b/docs/examples/neko/README.md @@ -27,7 +27,6 @@ Install the STUNner gateway operator and STUNner ([more info](https://github.com helm repo add stunner https://l7mp.io/stunner helm repo update helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system -helm install stunner stunner/stunner ``` Configure STUNner to act as a STUN server towards clients, and to let media reach the media server. @@ -38,10 +37,6 @@ cd stunner/docs/examples/neko kubectl apply -f stunner.yaml ``` -> [!WARNING] -> -> In case of [managed mode](/docs/INSTALL.md), update the `neko-plane` UDPRoute by replacing `stunner` in backendRefs with the generated deployment, e.g., `udp-gateway`. - This will expose STUNner on a public IP on UDP port 3478. A Kubernetes `LoadBalancer` assigns an ephemeral public IP address to the service, so first we need to learn the external IP. @@ -59,10 +54,10 @@ You can do that by hand, or by this fancy `sed` command: sed -i "s/1.1.1.1/$STUNNERIP/g" neko.yaml ``` -Now apply the Neko manifests: +Now apply the Neko manifests and wait for the `neko` deployment to be available (should take a couple of seconds): ```console kubectl apply -f neko.yaml -kubectl get pods +kubectl wait --for=condition=Available deployment neko --timeout 5m ``` In this setup we use `ingress` to expose the Neko UI. Feel free to customize the `ingress` resource to your setup. diff --git a/docs/examples/neko/stunner.yaml b/docs/examples/neko/stunner.yaml index d0db2009..5cf09326 100644 --- a/docs/examples/neko/stunner.yaml +++ b/docs/examples/neko/stunner.yaml @@ -19,7 +19,7 @@ metadata: namespace: default spec: realm: stunner.l7mp.io - authType: plaintext + authType: static userName: "user-1" password: "pass-1" @@ -49,5 +49,3 @@ spec: - backendRefs: - name: neko-udp namespace: default - - name: stunner - namespace: default From 827807cfbc8f9eb84fbff944d4dc9f4dd08e0e15 Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Thu, 4 Jan 2024 14:39:20 +0100 Subject: [PATCH 090/243] update(doc): Jitsi example --- docs/examples/jitsi/README.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/examples/jitsi/README.md b/docs/examples/jitsi/README.md index 4a82c6cf..1905ed0f 100644 --- a/docs/examples/jitsi/README.md +++ b/docs/examples/jitsi/README.md @@ -78,10 +78,10 @@ Then add the Helm repository, which contains the cert-manager Helm chart, and in ```console helm repo add cert-manager https://charts.jetstack.io helm repo update -helm install my-cert-manager cert-manager/cert-manager \ - --create-namespace \ - --namespace cert-manager \ - --version v1.8.0 +helm install cert-manager jetstack/cert-manager --namespace cert-manager \ + --create-namespace --set global.leaderElection.namespace=cert-manager \ + --set installCRDs=true --set featureGates=ExperimentalGatewayAPISupport=true \ + --timeout 600s --debug ``` At this point we have all the necessary boilerplate set up to automate TLS issuance for Jitsi. @@ -92,13 +92,22 @@ Now comes the fun part. The simplest way to run this demo is to clone the [STUNn Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): +Legacy mode: + ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy helm install stunner stunner/stunner --create-namespace --namespace=stunner ``` +Managed mode: +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system +``` + Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Jitsi server pods. ```console From abcebbd349a74ddb45de2a7c946b8ebe07a365b5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Thu, 4 Jan 2024 16:03:03 +0100 Subject: [PATCH 091/243] doc(jitsi): Update protocol name --- docs/examples/jitsi/jitsi-call-stunner.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/examples/jitsi/jitsi-call-stunner.yaml b/docs/examples/jitsi/jitsi-call-stunner.yaml index 4d2dba04..08f387ce 100644 --- a/docs/examples/jitsi/jitsi-call-stunner.yaml +++ b/docs/examples/jitsi/jitsi-call-stunner.yaml @@ -30,7 +30,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: stunner.l7mp.io/v1 kind: UDPRoute From 99b9d6289b2d6a0c80f5f3523e978e466c719242 Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Thu, 4 Jan 2024 20:52:02 +0100 Subject: [PATCH 092/243] doc: Update livekit example --- docs/examples/jitsi/README.md | 18 ++++-------- docs/examples/jitsi/jitsi-call-stunner.yaml | 2 +- docs/examples/livekit/README.md | 29 ++++++++++--------- .../livekit/livekit-call-stunner.yaml | 2 +- 4 files changed, 24 insertions(+), 27 deletions(-) diff --git a/docs/examples/jitsi/README.md b/docs/examples/jitsi/README.md index 1905ed0f..62b6aebd 100644 --- a/docs/examples/jitsi/README.md +++ b/docs/examples/jitsi/README.md @@ -67,21 +67,14 @@ export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. -First, install cert-manager's CRDs. - -```console -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml -``` - -Then add the Helm repository, which contains the cert-manager Helm chart, and install the charts: +Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: ```console helm repo add cert-manager https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --namespace cert-manager \ --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set installCRDs=true --set featureGates=ExperimentalGatewayAPISupport=true \ - --timeout 600s --debug + --set installCRDs=true --timeout 600s --debug ``` At this point we have all the necessary boilerplate set up to automate TLS issuance for Jitsi. @@ -97,15 +90,16 @@ Legacy mode: ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy -helm install stunner stunner/stunner --create-namespace --namespace=stunner +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy +helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner ``` Managed mode: + ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system ``` Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Jitsi server pods. diff --git a/docs/examples/jitsi/jitsi-call-stunner.yaml b/docs/examples/jitsi/jitsi-call-stunner.yaml index 08f387ce..8ef8e935 100644 --- a/docs/examples/jitsi/jitsi-call-stunner.yaml +++ b/docs/examples/jitsi/jitsi-call-stunner.yaml @@ -17,7 +17,7 @@ metadata: name: stunner-gatewayconfig namespace: stunner spec: - authType: longterm + authType: ephemeral sharedSecret: "my-shared-secret" --- apiVersion: gateway.networking.k8s.io/v1 diff --git a/docs/examples/livekit/README.md b/docs/examples/livekit/README.md index 2480e295..8def4e38 100644 --- a/docs/examples/livekit/README.md +++ b/docs/examples/livekit/README.md @@ -65,21 +65,14 @@ export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. -First, install cert-manager's CRDs. - -```console -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml -``` - -Then add the Helm repository, which contains the cert-manager Helm chart, and install the charts: +Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: ```console helm repo add cert-manager https://charts.jetstack.io helm repo update -helm install my-cert-manager cert-manager/cert-manager \ - --create-namespace \ - --namespace cert-manager \ - --version v1.8.0 +helm install cert-manager jetstack/cert-manager --namespace cert-manager \ + --create-namespace --set global.leaderElection.namespace=cert-manager \ + --set installCRDs=true --timeout 600s --debug ``` At this point we have all the necessary boilerplate set up to automate TLS issuance for LiveKit. @@ -90,11 +83,21 @@ Now comes the fun part. The simplest way to run this demo is to clone the [STUNn Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): +Legacy mode: + +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy +helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner +``` + +Managed mode: + ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system -helm install stunner stunner/stunner --create-namespace --namespace=stunner +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system ``` Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the LiveKit server pods. diff --git a/docs/examples/livekit/livekit-call-stunner.yaml b/docs/examples/livekit/livekit-call-stunner.yaml index 6371ef83..c168a3c7 100644 --- a/docs/examples/livekit/livekit-call-stunner.yaml +++ b/docs/examples/livekit/livekit-call-stunner.yaml @@ -19,7 +19,7 @@ metadata: namespace: stunner spec: realm: stunner.l7mp.io - authType: plaintext + authType: static userName: "user-1" password: "pass-1" From 83f1bc945593d641d05c32c39a41512d2d26d850 Mon Sep 17 00:00:00 2001 From: Kornel David Date: Fri, 5 Jan 2024 11:10:13 +0100 Subject: [PATCH 093/243] doc: Remove dev tags from charts --- docs/examples/jitsi/README.md | 6 +++--- docs/examples/livekit/README.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/examples/jitsi/README.md b/docs/examples/jitsi/README.md index 62b6aebd..47629026 100644 --- a/docs/examples/jitsi/README.md +++ b/docs/examples/jitsi/README.md @@ -90,8 +90,8 @@ Legacy mode: ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy -helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy +helm install stunner stunner/stunner --create-namespace --namespace=stunner ``` Managed mode: @@ -99,7 +99,7 @@ Managed mode: ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system ``` Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Jitsi server pods. diff --git a/docs/examples/livekit/README.md b/docs/examples/livekit/README.md index 8def4e38..88f4a052 100644 --- a/docs/examples/livekit/README.md +++ b/docs/examples/livekit/README.md @@ -88,8 +88,8 @@ Legacy mode: ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy -helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy +helm install stunner stunner/stunner --create-namespace --namespace=stunner ``` Managed mode: @@ -97,7 +97,7 @@ Managed mode: ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system ``` Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the LiveKit server pods. From 0253a76e5f8c996830a3c0aa67abdc682b6cdcf3 Mon Sep 17 00:00:00 2001 From: Kornel David Date: Fri, 5 Jan 2024 13:31:20 +0100 Subject: [PATCH 094/243] doc: Update mediasoup and remove debug tag --- docs/examples/jitsi/README.md | 2 +- docs/examples/livekit/README.md | 2 +- docs/examples/mediasoup/README.md | 40 ++++++++++--------- .../mediasoup/mediasoup-call-stunner.yaml | 10 ++--- 4 files changed, 29 insertions(+), 25 deletions(-) diff --git a/docs/examples/jitsi/README.md b/docs/examples/jitsi/README.md index 47629026..a295629a 100644 --- a/docs/examples/jitsi/README.md +++ b/docs/examples/jitsi/README.md @@ -74,7 +74,7 @@ helm repo add cert-manager https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --namespace cert-manager \ --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set installCRDs=true --timeout 600s --debug + --set installCRDs=true --timeout 600s ``` At this point we have all the necessary boilerplate set up to automate TLS issuance for Jitsi. diff --git a/docs/examples/livekit/README.md b/docs/examples/livekit/README.md index 88f4a052..41a3c857 100644 --- a/docs/examples/livekit/README.md +++ b/docs/examples/livekit/README.md @@ -72,7 +72,7 @@ helm repo add cert-manager https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --namespace cert-manager \ --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set installCRDs=true --timeout 600s --debug + --set installCRDs=true --timeout 600s ``` At this point we have all the necessary boilerplate set up to automate TLS issuance for LiveKit. diff --git a/docs/examples/mediasoup/README.md b/docs/examples/mediasoup/README.md index 31260b88..ec0ed408 100644 --- a/docs/examples/mediasoup/README.md +++ b/docs/examples/mediasoup/README.md @@ -19,7 +19,7 @@ The recommended way to install mediasoup ([link](https://mediasoup.discourse.gro The figure below shows mediasoup deployed into regular Kubernetes pods behind STUNner without the host-networking hack. Here, mediasoup is deployed behind STUNner in the [*media-plane deployment model*](../../DEPLOYMENT.md), so that STUNner acts as a "local" STUN/TURN server for mediasoup, saving the overhead of using public a 3rd party STUN/TURN server for NAT traversal. -In this tutorial we deploy a video room example using [mediasoup's demo application](https://github.com/versatica/mediasoup-demo/) with slight modifications (more on these below), the [mediasoup server](https://github.com/versatica/mediasoup/) for media exchange, a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the LiveKit server pool to clients. +In this tutorial we deploy a video room example using [mediasoup's demo application](https://github.com/versatica/mediasoup-demo/) with slight modifications (more on these below), the [mediasoup server](https://github.com/versatica/mediasoup/) for media exchange, a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the mediasoup server pool to clients. ### Modifications on the mediasoup demo @@ -83,21 +83,14 @@ export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. -First, install cert-manager's CRDs. - -```console -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml -``` - -Then add the Helm repository, which contains the cert-manager Helm chart, and install the charts: +Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: ```console helm repo add cert-manager https://charts.jetstack.io helm repo update -helm install my-cert-manager cert-manager/cert-manager \ - --create-namespace \ - --namespace cert-manager \ - --version v1.8.0 +helm install cert-manager jetstack/cert-manager --namespace cert-manager \ + --create-namespace --set global.leaderElection.namespace=cert-manager \ + --set installCRDs=true --timeout 600s ``` At this point we have all the necessary boilerplate set up to automate TLS issuance for mediasoup. @@ -106,13 +99,23 @@ At this point we have all the necessary boilerplate set up to automate TLS issua Now comes the fun part. -Install the STUNner gateway operator using the [managed dataplane mode](https://github.com/l7mp/stunner/blob/main/docs/INSTALL.md#managed-mode) via [Helm](https://github.com/l7mp/stunner-helm): +Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): + +Legacy mode: + +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy +helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner-system +``` + +Managed mode (recommended): ```console helm repo add stunner https://l7mp.io/stunner helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace - --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed --set stunnerGatewayOperator.dataplane.spec.image=l7mp/stunnerd:latest +helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system ``` Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the mediasoup server pods. @@ -141,7 +144,7 @@ spec: apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: - name: livekit-media-plane + name: mediasoup-media-plane namespace: stunner spec: parentRefs: @@ -160,8 +163,8 @@ Wait until Kubernetes assigns an external IP and store the external IP assigned STUNner in an environment variable for later use. ```console -until [ -n "$(kubectl get svc udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -export STUNNERIP=$(kubectl get service udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +until [ -n "$(kubectl get svc udp-gateway -n stunner-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +export STUNNERIP=$(kubectl get service udp-gateway -n stunner-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` ### mediasoup @@ -179,6 +182,7 @@ sed -i "s/ingressserviceip/$INGRESSIP/g" docs/examples/mediasoup/mediasoup-serve Finally, fire up mediasoup. ```console +kubectl create ns mediasoup kubectl apply -f docs/examples/mediasoup/mediasoup-server.yaml ``` diff --git a/docs/examples/mediasoup/mediasoup-call-stunner.yaml b/docs/examples/mediasoup/mediasoup-call-stunner.yaml index e975ee84..2e16f70b 100644 --- a/docs/examples/mediasoup/mediasoup-call-stunner.yaml +++ b/docs/examples/mediasoup/mediasoup-call-stunner.yaml @@ -9,17 +9,17 @@ spec: group: "stunner.l7mp.io" kind: GatewayConfig name: stunner-gatewayconfig - namespace: stunner + namespace: stunner-system description: "STUNner is a WebRTC ingress gateway for Kubernetes" --- apiVersion: stunner.l7mp.io/v1 kind: GatewayConfig metadata: name: stunner-gatewayconfig - namespace: stunner + namespace: stunner-system spec: realm: stunner.l7mp.io - authType: plaintext + authType: static userName: "user-1" password: "pass-1" --- @@ -27,7 +27,7 @@ apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway - namespace: stunner + namespace: stunner-system spec: gatewayClassName: stunner-gatewayclass listeners: @@ -39,7 +39,7 @@ apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: name: livekit-media-plane - namespace: stunner + namespace: stunner-system spec: parentRefs: - name: udp-gateway From 7bb004862a01e42f357191c604b6a49eaaf6c4e5 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 5 Jan 2024 17:43:15 +0100 Subject: [PATCH 095/243] doc: Update user manuals --- cmd/stunnerd/README.md | 32 +++++++--------- cmd/turncat/README.md | 4 +- docs/cmd/stunnerctl.md | 29 ++++++++++----- docs/cmd/stunnerd.md | 84 ++++++++++++++++++++++-------------------- docs/cmd/turncat.md | 40 ++++++++++---------- 5 files changed, 99 insertions(+), 90 deletions(-) diff --git a/cmd/stunnerd/README.md b/cmd/stunnerd/README.md index 5710efa7..5d532836 100644 --- a/cmd/stunnerd/README.md +++ b/cmd/stunnerd/README.md @@ -4,14 +4,14 @@ The `stunnerd` daemon implements the STUNner gateway dataplane. The daemon supports two basic modes. For quick tests `stunnerd` can be configured as a TURN server by specifying a TURN network URI on the command line. For more complex scenarios, and especially -for use in a Kubernetes cluster, `stunnerd` can take configuration from a config origin, that can -be any file or a remote server. In addition, `stunnerd` implements a watch-mode, so that it can -actively monitor the config origin for updates and automatically reconcile the TURN server to any -new configuration. This mode is intended for use with the [STUNner Kubernetes gateway -operator](https://github.com/l7mp/stunner-gateway-operator): the operator watches the Kubernetes -[Gateway API](https://gateway-api.sigs.k8s.io) resources, renders the active control plane -configuration per each `stunnerd` pod and dynamically updates the dataplane using STUNner's config -discovery service. +for use in a Kubernetes cluster, `stunnerd` can take configuration from a config origin, which can +either be a config file or from a remote server reached over WebSocket. In addition, `stunnerd` +implements a watch-mode, so that it can actively monitor the config origin for updates and +automatically reconcile the TURN server to any new configuration. This mode is intended for use +with the [STUNner Kubernetes gateway operator](https://github.com/l7mp/stunner-gateway-operator): +the operator watches the Kubernetes [Gateway API](https://gateway-api.sigs.k8s.io) resources, +renders the active control plane configuration per each `stunnerd` pod and dynamically updates the +dataplane using STUNner's config discovery service. ## Features @@ -52,7 +52,7 @@ Alternatively, run `stunnerd` in verbose mode with the config file taken from `c ./stunnerd -v -w -c cmd/stunnerd/stunnerd.conf ``` -Type `./stunnerd` to see a short description of the most important command line arguments. +Type `./stunnerd -h` to get a short description of the supported command line arguments. In practice, you'll rarely need to run `stunnerd` directly: just fire up the [prebuilt container image](https://hub.docker.com/repository/docker/l7mp/stunnerd) in Kubernetes and you should be good to go. Or better yet, [install](/docs/INSTALL.md) the STUNner Kubernetes gateway operator that will readily manage the `stunnerd` pods for each Gateway you create. @@ -74,31 +74,27 @@ static: listeners: - name: stunnerd-udp address: "$STUNNER_ADDR" - protocol: udp + protocol: turn-udp port: 3478 - name: stunnerd-tcp address: "$STUNNER_ADDR" - protocol: tcp + protocol: turn-tcp port: 3478 - name: stunnerd-tls address: "$STUNNER_ADDR" - protocol: tls + protocol: turn-tls port: 3479 cert: "my-cert.cert" key: "my-key.key" - name: stunnerd-dtls address: "$STUNNER_ADDR" - protocol: dtls + protocol: turn-dtls port: 3479 cert: "my-cert.cert" key: "my-key.key" ``` -## Advanced features - -### TURN/UDP listener CPU scaling - -STUNner can run multiple parallel readloops for TURN/UDP listeners, which allows it to scale to practically any number of CPUs and brings massive performance improvements for UDP workloads. This can be achieved by creating a configurable number of UDP server sockets using the `SO_REUSEPORT` socket option and spawning a separate goroutine to run a parallel readloop per each listener. The kernel will load-balance allocations across the sockets/readloops per the IP 5-tuple, thus the same allocation will always stay at the same CPU. This is important for correct TURN operations. +STUNner can run multiple parallel readloops for TURN/UDP listeners, which allows it to scale to practically any number of CPUs and brings massive performance improvements for UDP workloads. This can be achieved by creating a configurable number of UDP readloop threads over the same TURN listener. The kernel will load-balance allocations across the readloops per the IP 5-tuple and so the same allocation will always stay at the same CPU, which is important for correct TURN operations. The feature is exposed via the command line flag `--udp-thread-num=`. The below starts `stunnerd` watching the config file in `/etc/stunnerd/stunnerd.conf` using 32 parallel UDP readloops (the default is 16). diff --git a/cmd/turncat/README.md b/cmd/turncat/README.md index 9b0e6dda..8e2c3552 100644 --- a/cmd/turncat/README.md +++ b/cmd/turncat/README.md @@ -1,6 +1,6 @@ # turncat: Swiss-army-knife testing tool for STUNner -`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster. +`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster which is much faster than the TCP connection used by kubectl. ## Getting Started @@ -29,7 +29,7 @@ TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accep turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 ``` -Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace, and connect to the TURN listener named `udp-listener`. The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will enable verbose logging. +Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace and connect to the TURN listener named `udp-listener`. The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will enable verbose logging. ```console ./turncat -v - k8s://stunner/udp-gateway:udp-listener udp://10.0.0.1:9001 diff --git a/docs/cmd/stunnerctl.md b/docs/cmd/stunnerctl.md index 9c4debcf..6982ded4 100644 --- a/docs/cmd/stunnerctl.md +++ b/docs/cmd/stunnerctl.md @@ -4,14 +4,25 @@ A CLI tool to simplify the interaction with STUNner. ## Usage -Dump the running config from a live STUNner deployment in human-readable format. +Dump the running config of a STUNner gateway in human-readable format. + +The below will select the Gateway called `tcp-gateway` in the `stunner` namespace: + ```console -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener: udp-listener -Protocol: UDP -Public address: 34.118.36.108 -Public port: 3478 +cmd/stunnerctl/stunnerctl running-config stunner/stunner-gateway +STUN/TURN authentication type: static +STUN/TURN username: user-1 +STUN/TURN password: pass-1 +Listener 1 + Name: stunner/tcp-gateway/tcp-listener + Listener: stunner/tcp-gateway/tcp-listener + Protocol: TURN-TCP + Public address: 35.187.97.94 + Public port: 3478 ``` + +## License + +Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). + +MIT License - see [LICENSE](../../LICENSE) for full text. diff --git a/docs/cmd/stunnerd.md b/docs/cmd/stunnerd.md index e801e660..5d532836 100644 --- a/docs/cmd/stunnerd.md +++ b/docs/cmd/stunnerd.md @@ -4,14 +4,14 @@ The `stunnerd` daemon implements the STUNner gateway dataplane. The daemon supports two basic modes. For quick tests `stunnerd` can be configured as a TURN server by specifying a TURN network URI on the command line. For more complex scenarios, and especially -for use in a Kubernetes cluster, `stunnerd` can take configuration from a config file. In addition, -`stunnerd` implements a watch-mode, so that it can actively monitor the config file for updates -and, once the config file has changed, automatically reconcile the TURN server to the new -configuration. This mode is intended for use with the [STUNner Kubernetes gateway -operator](https://github.com/l7mp/stunner-gateway-operator): the operator watches the Kubernetes -[Gateway API](https://gateway-api.sigs.k8s.io) resources and renders the active control plane -configuration into a ConfigMap, which is then mapped into the `stunnerd` pod's filesystem so that -the daemon can pick up the latest configuration using the watch mode. +for use in a Kubernetes cluster, `stunnerd` can take configuration from a config origin, which can +either be a config file or from a remote server reached over WebSocket. In addition, `stunnerd` +implements a watch-mode, so that it can actively monitor the config origin for updates and +automatically reconcile the TURN server to any new configuration. This mode is intended for use +with the [STUNner Kubernetes gateway operator](https://github.com/l7mp/stunner-gateway-operator): +the operator watches the Kubernetes [Gateway API](https://gateway-api.sigs.k8s.io) resources, +renders the active control plane configuration per each `stunnerd` pod and dynamically updates the +dataplane using STUNner's config discovery service. ## Features @@ -22,8 +22,11 @@ the daemon can pick up the latest configuration using the watch mode. * [RFC 6062](https://tools.ietf.org/html/rfc6062): Traversal Using Relays around NAT (TURN) Extensions for TCP Allocations * TURN transport over UDP, TCP, TLS/TCP and DTLS/UDP. -* Two authentication modes via the long-term STUN/TURN credential mechanism: `plaintext` using a - static username/password pair, and `longterm` with dynamically generated time-scoped credentials. +* TURN/UDP listener CPU scaling. +* Two authentication modes via the long-term STUN/TURN credential mechanism: `static` using a + static username/password pair, and `ephemeral` with dynamically generated time-scoped + credentials. +* Peer port range filtering. ## Getting Started @@ -37,39 +40,28 @@ go build -o stunnerd cmd/stunnerd/main.go ### Usage -The below command will open a `stunnerd` UDP listener at `127.0.0.1:5000`, set `plaintext` -authentication using the username/password pair `user1/passwrd1`, and raises the debug level to the -maximum. +The below command will open a `stunnerd` UDP listener at `127.0.0.1:5000`, set `static` authentication using the username/password pair `user1/passwrd1`, and raise the debug level to the maximum. ```console ./stunnerd --log=all:TRACE turn://user1:passwd1@127.0.0.1:5000 ``` -Alternatively, run `stunnerd` in verbose mode with the config file taken from -`cmd/stunnerd/stunnerd.conf`. Adding the flag `-w` will enable watch mode. +Alternatively, run `stunnerd` in verbose mode with the config file taken from `cmd/stunnerd/stunnerd.conf`. Adding the flag `-w` will enable watch mode. ```console -$ ./stunnerd -v -w -c cmd/stunnerd/stunnerd.conf +./stunnerd -v -w -c cmd/stunnerd/stunnerd.conf ``` -Type `./stunnerd` to see a short description of the command line arguments supported by `stunnerd`. +Type `./stunnerd -h` to get a short description of the supported command line arguments. -In practice, you'll rarely need to run `stunnerd` directly: just fire up the [prebuilt container -image](https://hub.docker.com/repository/docker/l7mp/stunnerd) in Kubernetes and you should be good -to go. +In practice, you'll rarely need to run `stunnerd` directly: just fire up the [prebuilt container image](https://hub.docker.com/repository/docker/l7mp/stunnerd) in Kubernetes and you should be good to go. Or better yet, [install](/docs/INSTALL.md) the STUNner Kubernetes gateway operator that will readily manage the `stunnerd` pods for each Gateway you create. ## Configuration -Using the below configuration, `stunnerd` will open 4 STUNner listeners: two for accepting -unencrypted connections at UDP/3478 and TCP/3478, and two for encrypted connections at TLS/TCP/3479 -and DTLS/UDP/3479. Transport relay connections opened by `stunnerd` will be restricted to peer -ports in the range [10000:19999] for the UDP listener, [20000:29999] for the TCP listener, etc. -The daemon will use `ephemeral` authentication, with the shared secret read from the environment -variable `$STUNNER_SHARED_SECRET` during initialization. The relay address is taken from the -`$STUNNER_ADDR` environment variable. +Using the below configuration, `stunnerd` will open 4 STUNner listeners: two for accepting unencrypted connections at UDP/3478 and TCP/3478, and two for encrypted connections at TLS/TCP/3479 and DTLS/UDP/3479. The daemon will use `ephemeral` authentication, with the shared secret taken from the environment variable `$STUNNER_SHARED_SECRET` during initialization. The relay address will be taken from the `$STUNNER_ADDR` environment variable. ``` yaml -version: v1 +version: v1alpha1 admin: name: my-stunnerd logLevel: all:DEBUG @@ -82,28 +74,40 @@ static: listeners: - name: stunnerd-udp address: "$STUNNER_ADDR" - protocol: udp + protocol: turn-udp port: 3478 - minPort: 10000 - maxPort: 19999 - name: stunnerd-tcp address: "$STUNNER_ADDR" - protocol: tcp + protocol: turn-tcp port: 3478 - minPort: 20000 - maxPort: 29999 - name: stunnerd-tls - protocol: tls + address: "$STUNNER_ADDR" + protocol: turn-tls port: 3479 - minPort: 30000 - maxPort: 39999 cert: "my-cert.cert" key: "my-key.key" - name: stunnerd-dtls - protocol: dtls + address: "$STUNNER_ADDR" + protocol: turn-dtls port: 3479 cert: "my-cert.cert" key: "my-key.key" - minPort: 40000 - maxPort: 49999 ``` + +STUNner can run multiple parallel readloops for TURN/UDP listeners, which allows it to scale to practically any number of CPUs and brings massive performance improvements for UDP workloads. This can be achieved by creating a configurable number of UDP readloop threads over the same TURN listener. The kernel will load-balance allocations across the readloops per the IP 5-tuple and so the same allocation will always stay at the same CPU, which is important for correct TURN operations. + +The feature is exposed via the command line flag `--udp-thread-num=`. The below starts `stunnerd` watching the config file in `/etc/stunnerd/stunnerd.conf` using 32 parallel UDP readloops (the default is 16). + +``` sh +./stunnerd -w -c /etc/stunnerd/stunnerd.conf --udp-thread-num=32 +``` + +## License + +Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). + +MIT License - see [LICENSE](../../LICENSE) for full text. + +## Acknowledgments + +Initial code adopted from [pion/stun](https://github.com/pion/stun) and [pion/turn](https://github.com/pion/turn). diff --git a/docs/cmd/turncat.md b/docs/cmd/turncat.md index 83ea9355..8e2c3552 100644 --- a/docs/cmd/turncat.md +++ b/docs/cmd/turncat.md @@ -1,9 +1,6 @@ # turncat: Swiss-army-knife testing tool for STUNner -`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote -address/port. The main use is to open a local tunnel endpoint to any service running inside a -Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl proxy`, but it -uses STUN/TURN to enter the cluster. +`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster which is much faster than the TCP connection used by kubectl. ## Getting Started @@ -18,31 +15,32 @@ go build -o turncat cmd/turncat/main.go ### Usage -Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets -through the TURN server located at `192.0.2.1:3478` to the UDP server located at -`192.0.2.2:53`. Use the `static` STUN/TURN credential mechanism to authenticate with the TURN -server and set the user/passwd to `test/test`: +Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP listener located at `192.0.2.2:53`. Use the [`static` STUN/TURN credential mechanism](/docs/AUTH.md) to authenticate with the TURN server and set the user/passwd to `test/test`: ```console -./turncat --log=all:INFO,turncat:DEBUG udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478 udp://192.0.2.2:53 +./turncat --log=all:INFO,turncat:DEBUG udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478 \ + udp://192.0.2.2:53 ``` -TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accept self-signed TLS -certificates and `--verbose` is equivalent to setting all `turncat` loggers to DEBUG mode (`-l -all:DEBUG`). +TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accept self-signed TLS certificates and `--verbose` is equivalent to setting all `turncat` loggers to DEBUG mode (`-l all:DEBUG`). ```console -./turncat --verbose --insecure udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 +./turncat --verbose --insecure udp://127.0.0.1:5000 \ + turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 ``` -Alternatively, specify the special TURN server URI `k8s://stunner/stunnerd-config:udp-listener` to -let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI -directs `turncat` to read the STUNner config from the ConfigMap named `stunnerd-config` in the -`stunner` namespace, and connect to the STUNner listener named `udp-listener`. The CLI flag `-` -instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent -via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will -enable verbose logging. +Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace and connect to the TURN listener named `udp-listener`. The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will enable verbose logging. ```console -./turncat -v - k8s://stunner/stunnerd-config:udp-listener udp://10.0.0.1:9001 +./turncat -v - k8s://stunner/udp-gateway:udp-listener udp://10.0.0.1:9001 ``` + +## License + +Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). + +MIT License - see [LICENSE](../../LICENSE) for full text. + +## Acknowledgments + +Initial code adopted from [pion/stun](https://github.com/pion/stun) and [pion/turn](https://github.com/pion/turn). From dfad892b163fc0c3d49d1429ad982bd37766ae6e Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 5 Jan 2024 19:15:05 +0100 Subject: [PATCH 096/243] docs(MONITORING): Update with managed-mode steps --- docs/MONITORING.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/MONITORING.md b/docs/MONITORING.md index d011279a..0cec627e 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -21,6 +21,7 @@ Each STUNner gateway instance exports a number of standard metrics that describe | `go_goroutines` | Number of goroutines that currently exist. | | `go_threads` | Number of OS threads created. | | `process_open_fds` | Number of open file descriptors.| +| `process_resident_memory_bytes` | Resident memory size in bytes. | | `process_virtual_memory_bytes` | Virtual memory size in bytes. | ### Connection statistics @@ -44,17 +45,21 @@ Collection and visualization of STUNner relies on Prometheus and Grafana service A full-fledged Prometheus+Grafana helm chart is available in the [STUNner helm repo](https://github.com/l7mp/stunner-helm#monitoring). To use this chart, the installation steps involve enabling monitoring in STUNner, and installing the Prometheus+Grafana stack with helm. -1. Install STUNner with Prometheus support: +1. Install stunner-gateway-operator with Prometheus support: - ```console - helm install stunner stunner/stunner --create-namespace --namespace=stunner --set stunner.deployment.monitoring.enabled=true - ``` +```console +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.spec.enableMetricsEndpoint=true +``` + +Alternatively, you can enable it on existing installations by setting `enableMetricsEndpoint: true` in your [Dataplane](./CONCEPTS.md#dataplane) objects. + +> [!NOTE] +> Metrics are exposed at `http://:8080/metrics` on each STUNner pod -2. Configure STUNner to expose the metrics. -3. Install the Prometheus+Grafana stack with a helm chart. +2. Install the Prometheus+Grafana stack with a helm chart. - The below creates a ready-to-use Prometheus+Grafana stack in the `monitoring` namespace: Prometheus, along with the prometheus-operator, is installed for metrics scarping, Grafana is set up for visualization, and the Prometheus is configured as a datasource for Grafana. +The below creates a ready-to-use Prometheus+Grafana stack in the `monitoring` namespace: Prometheus, along with the prometheus-operator, is installed for metrics scarping, Grafana is set up for visualization, and the Prometheus is configured as a datasource for Grafana. ```console helm repo add stunner https://l7mp.io/stunner From f687843899e4be54d0031878f8fe1bfdfa83cdd8 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 5 Jan 2024 19:16:33 +0100 Subject: [PATCH 097/243] doc(MONITORING): Unify 'Dataplane' links --- docs/MONITORING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 0cec627e..884f2052 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -51,7 +51,7 @@ A full-fledged Prometheus+Grafana helm chart is available in the [STUNner helm r helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.spec.enableMetricsEndpoint=true ``` -Alternatively, you can enable it on existing installations by setting `enableMetricsEndpoint: true` in your [Dataplane](./CONCEPTS.md#dataplane) objects. +Alternatively, you can enable it on existing installations by setting `enableMetricsEndpoint: true` in your [Dataplane](GATEWAY.md#dataplane) objects. > [!NOTE] > Metrics are exposed at `http://:8080/metrics` on each STUNner pod From 4af9fb532ddde08e104fe3f9e31b2d7ecd169ce2 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 5 Jan 2024 19:28:25 +0100 Subject: [PATCH 098/243] chore: Module dependencies --- go.mod | 3 +-- go.sum | 2 -- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/go.mod b/go.mod index 3897a2ec..f2ac185b 100644 --- a/go.mod +++ b/go.mod @@ -25,6 +25,7 @@ require ( golang.org/x/time v0.4.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 + k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 sigs.k8s.io/controller-runtime v0.12.3 sigs.k8s.io/yaml v1.3.0 ) @@ -45,7 +46,6 @@ require ( github.com/golang/protobuf v1.5.3 // indirect github.com/google/gnostic v0.5.7-v3refs // indirect github.com/google/gofuzz v1.1.0 // indirect - github.com/hashicorp/golang-lru v1.0.2 // indirect github.com/imdario/mergo v0.3.12 // indirect github.com/invopop/yaml v0.1.0 // indirect github.com/josharian/intern v1.0.0 // indirect @@ -82,7 +82,6 @@ require ( k8s.io/client-go v0.24.2 // indirect k8s.io/klog/v2 v2.60.1 // indirect k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 // indirect - k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 // indirect sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect ) diff --git a/go.sum b/go.sum index 5dbd6e72..7dc510b4 100644 --- a/go.sum +++ b/go.sum @@ -200,8 +200,6 @@ github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZH github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= -github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c= -github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb h1:tsEKRC3PU9rMw18w/uAptoijhgG4EvlA5kfJPtwrMDk= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb/go.mod h1:NtmN9h8vrTveVQRLHcX2HQ5wIPBDCsZ351TGbZWgg38= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= From 74d1136bdb48f310837d198cd38eae6b1fee2ee6 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 5 Jan 2024 19:29:08 +0100 Subject: [PATCH 099/243] doc: Remove doc disclaimer --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index c7b3f20e..a65130b2 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,6 @@

-*This is the documentation for the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the From 36a739b3371fb4677d4397f5e71dd21121d231e4 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 5 Jan 2024 20:27:33 +0100 Subject: [PATCH 100/243] doc: Readd doc disclaimer --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index a65130b2..c7b3f20e 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*This is the documentation for the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the From c40128c8ecae1abd557f5dfe7cbba0690516ed62 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 5 Jan 2024 20:40:19 +0100 Subject: [PATCH 101/243] chore(CI/CD): Update stable e2e test --- .github/workflows/e2e-test-stable.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index 48ac7997..a758d74a 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -43,8 +43,7 @@ jobs: run: | helm repo add stunner https://l7mp.io/stunner helm repo update - helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner - helm install stunner stunner/stunner --create-namespace --namespace=stunner + helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m --set stunnerGatewayOperator.dataplane.spec.resources.requests.cpu=100m - name: Deploy iperf server run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml @@ -61,12 +60,13 @@ jobs: - name: Wait for LoadBalancer IP run: | - for n in {1..300}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done + for n in {1..60}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done + kubectl wait -n stunner --for=condition=Available deployment udp-gateway --timeout 5m kubectl get all -A - name: Start turncat run: | - ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & + ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & sleep 1 - name: Run iperf client From d7448e60151ff531afdcc0657b951b57c0717d69 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Tue, 9 Jan 2024 14:39:17 +0100 Subject: [PATCH 102/243] feat(turncat): Extend help message in --help --- cmd/turncat/main.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 899b7e87..773daafb 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -43,8 +43,14 @@ func main() { // var passwd = flag.StringP("log", "l", "all:WARN", "Log level (default: all:WARN).") var insecure = flag.BoolP("insecure", "i", false, "Insecure TLS mode, accept self-signed certificates (default: false).") var verbose = flag.BoolP("verbose", "v", false, "Verbose logging, identical to -l all:DEBUG.") + var help = flag.BoolP("help", "h", false, "Display this help text and exit") flag.Parse() + if *help { + Usage() + os.Exit(0) + } + if flag.NArg() != 3 { Usage() os.Exit(1) From 0a94ce59eb9fdb9c5a5430f45c5218b32222eaca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Tue, 9 Jan 2024 16:52:26 +0100 Subject: [PATCH 103/243] fix(turncat): Fix build on Windows (#111) --- reuseaddr_unix.go | 12 ++++++++++++ reuseaddr_windows.go | 12 ++++++++++++ uri.go | 8 -------- 3 files changed, 24 insertions(+), 8 deletions(-) create mode 100644 reuseaddr_unix.go create mode 100644 reuseaddr_windows.go diff --git a/reuseaddr_unix.go b/reuseaddr_unix.go new file mode 100644 index 00000000..fa545c48 --- /dev/null +++ b/reuseaddr_unix.go @@ -0,0 +1,12 @@ +//go:build unix + +package stunner + +import "syscall" + +func reuseAddr(network, address string, conn syscall.RawConn) error { + return conn.Control(func(descriptor uintptr) { + _ = syscall.SetsockoptInt(int(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1) + // syscall.SetsockoptInt(int(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEPORT, 1) + }) +} diff --git a/reuseaddr_windows.go b/reuseaddr_windows.go new file mode 100644 index 00000000..eedc298c --- /dev/null +++ b/reuseaddr_windows.go @@ -0,0 +1,12 @@ +//go:build windows + +package stunner + +import "syscall" + +func reuseAddr(network, address string, conn syscall.RawConn) error { + return conn.Control(func(descriptor uintptr) { + _ = syscall.SetsockoptInt(syscall.Handle(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1) + // syscall.SetsockoptInt(syscall.Handle(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEPORT, 1) + }) +} diff --git a/uri.go b/uri.go index ec4cf88f..d4ac5817 100644 --- a/uri.go +++ b/uri.go @@ -7,7 +7,6 @@ import ( "os" "strconv" "strings" - "syscall" "github.com/l7mp/stunner/internal/util" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" @@ -166,10 +165,3 @@ func getStunnerProtoForURI(u *url.URL) (string, error) { return "", fmt.Errorf("Invalid scheme/protocol in URI %q", u.String()) } - -func reuseAddr(network, address string, conn syscall.RawConn) error { - return conn.Control(func(descriptor uintptr) { - _ = syscall.SetsockoptInt(int(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1) - // syscall.SetsockoptInt(int(descriptor), syscall.SOL_SOCKET, syscall.SO_REUSEPORT, 1) - }) -} From 68eac15d9a672e03893ea11327cbaa6ca9d6b286 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 12 Jan 2024 13:10:08 +0100 Subject: [PATCH 104/243] chore: Bump copyright year --- LICENSE | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/LICENSE b/LICENSE index 621376f6..116904bc 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2023 l7mp +Copyright (c) 2024 l7mp Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal From 22c2575ea2719e3709fb31d3f90a732d7232d342 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 18 Jan 2024 09:33:59 +0100 Subject: [PATCH 105/243] chore: Bump prometheus/client_golang to v1.18.0 --- go.mod | 12 ++++++------ go.sum | 24 ++++++++++++------------ 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/go.mod b/go.mod index f2ac185b..aaa6c927 100644 --- a/go.mod +++ b/go.mod @@ -17,11 +17,11 @@ require ( github.com/pion/logging v0.2.2 github.com/pion/transport/v3 v3.0.1 github.com/pion/turn/v3 v3.0.1 - github.com/prometheus/client_golang v1.16.0 + github.com/prometheus/client_golang v1.18.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 go.uber.org/zap v1.26.0 - golang.org/x/sys v0.13.0 + golang.org/x/sys v0.15.0 golang.org/x/time v0.4.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 @@ -51,7 +51,7 @@ require ( github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/mailru/easyjson v0.7.7 // indirect - github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect + github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect @@ -62,14 +62,14 @@ require ( github.com/pion/transport/v2 v2.2.1 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect - github.com/prometheus/client_model v0.4.0 // indirect - github.com/prometheus/common v0.44.0 // indirect + github.com/prometheus/client_model v0.5.0 // indirect + github.com/prometheus/common v0.45.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect go.uber.org/multierr v1.10.0 // indirect golang.org/x/crypto v0.14.0 // indirect golang.org/x/mod v0.12.0 // indirect golang.org/x/net v0.17.0 // indirect - golang.org/x/oauth2 v0.8.0 // indirect + golang.org/x/oauth2 v0.12.0 // indirect golang.org/x/term v0.13.0 // indirect golang.org/x/text v0.13.0 // indirect golang.org/x/tools v0.12.0 // indirect diff --git a/go.sum b/go.sum index 7dc510b4..6fd198bb 100644 --- a/go.sum +++ b/go.sum @@ -232,8 +232,8 @@ github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= -github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo= -github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= +github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg= +github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -283,13 +283,13 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.16.0 h1:yk/hx9hDbrGHovbci4BY+pRMfSuuat626eFsHb7tmT8= -github.com/prometheus/client_golang v1.16.0/go.mod h1:Zsulrv/L9oM40tJ7T815tM89lFEugiJ9HzIqaAx4LKc= +github.com/prometheus/client_golang v1.18.0 h1:HzFfmkOzH5Q8L8G+kSJKUx5dtG87sewO+FoDDqP5Tbk= +github.com/prometheus/client_golang v1.18.0/go.mod h1:T+GXkCk5wSJyOqMIzVgvvjFDlkOQntgjkJWKrN5txjA= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/client_model v0.4.0 h1:5lQXD3cAg1OXBf4Wq03gTrXHeaV0TQvGfUooCfx1yqY= -github.com/prometheus/client_model v0.4.0/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU= -github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY= -github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY= +github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= +github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= +github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM= +github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY= github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= @@ -444,8 +444,8 @@ golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8= -golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE= +golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4= +golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -512,8 +512,8 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE= -golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc= +golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= From 60021e7ea3b101eed6465f6d90b4aca00320a261 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 19 Jan 2024 10:40:48 +0100 Subject: [PATCH 106/243] refactor: Rewrite config discovery in strict OpenAPI mode --- cmd/stunnerd/main.go | 32 ++-- config.go | 12 +- config_test.go | 44 +++-- handlers_test.go | 2 +- pkg/apis/v1/auth.go | 4 + pkg/apis/v1/cluster.go | 5 + pkg/apis/v1/default.go | 4 + pkg/apis/v1/listener.go | 11 +- pkg/apis/v1/stunner.go | 25 +-- pkg/config/api/stunner_openapi.yaml | 96 +++++++--- pkg/config/cds_test.go | 232 +++++++++++++++++++---- pkg/config/client/api/client.gen.go | 100 +++++++++- pkg/config/client/cds_api.go | 27 +-- pkg/config/client/cds_client.go | 28 +-- pkg/config/client/client.go | 4 +- pkg/config/client/file_client.go | 30 +-- pkg/config/server/api/server.gen.go | 283 ++++++++++++++++++++++++++-- pkg/config/server/cfg.yaml | 1 + pkg/config/server/config.go | 12 +- pkg/config/server/conn.go | 9 +- pkg/config/server/error.go | 34 ---- pkg/config/server/handler.go | 144 ++++++-------- pkg/config/server/middleware.go | 94 +++++++++ pkg/config/server/server.go | 90 ++++----- reconcile.go | 14 +- reconcile_test.go | 8 +- server_unix_test.go | 2 +- stunner.go | 9 +- stunner_test.go | 19 +- turncat_test.go | 4 +- 30 files changed, 989 insertions(+), 390 deletions(-) delete mode 100644 pkg/config/server/error.go create mode 100644 pkg/config/server/middleware.go diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 52bf399e..c2f46a9d 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -17,17 +17,9 @@ import ( // usage: stunnerd -v turn://user1:passwd1@127.0.0.1:3478?transport=udp -const ( - defaultLoglevel = "all:INFO" - defaultConfigDiscoveryAddress = "http://localhost:13478" - envVarName = "STUNNER_NAME" - envVarNamespace = "STUNNER_NAMESPACE" - envVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" -) - func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", fmt.Sprintf("Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: %s).", defaultConfigDiscoveryAddress)) + var config = flag.StringP("config", "c", "", fmt.Sprintf("Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: %s).", stnrv1.DefaultConfigDiscoveryAddress)) var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO).") var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: ).") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false).") @@ -37,7 +29,7 @@ func main() { var verbose = flag.BoolP("verbose", "v", false, "Verbose logging, identical to <-l all:DEBUG>.") flag.Parse() - logLevel := defaultLoglevel + logLevel := stnrv1.DefaultLogLevel if *verbose { logLevel = "all:DEBUG" } @@ -46,8 +38,8 @@ func main() { logLevel = *level } - configOrigin := defaultConfigDiscoveryAddress - if origin, ok := os.LookupEnv(envVarConfigOrigin); ok { + configOrigin := stnrv1.DefaultConfigDiscoveryAddress + if origin, ok := os.LookupEnv(stnrv1.DefaultEnvVarConfigOrigin); ok { configOrigin = origin } if *config != "" { @@ -55,15 +47,15 @@ func main() { } if *id == "" { - name, ok1 := os.LookupEnv(envVarName) - namespace, ok2 := os.LookupEnv(envVarNamespace) + name, ok1 := os.LookupEnv(stnrv1.DefaultEnvVarName) + namespace, ok2 := os.LookupEnv(stnrv1.DefaultEnvVarNamespace) if ok1 && ok2 { *id = fmt.Sprintf("%s/%s", namespace, name) } } st := stunner.NewStunner(stunner.Options{ - Id: *id, + Name: *id, LogLevel: logLevel, DryRun: *dryRun, UDPListenerThreadNum: *udpThreadNum, @@ -74,7 +66,7 @@ func main() { log.Infof("starting stunnerd instance %q", st.GetId()) - conf := make(chan stnrv1.StunnerConfig, 1) + conf := make(chan *stnrv1.StunnerConfig, 1) defer close(conf) var cancelConfigLoader context.CancelFunc @@ -88,7 +80,7 @@ func main() { os.Exit(1) } - conf <- *c + conf <- c } else if !*watch { log.Infof("loading configuration from origin %q", configOrigin) @@ -99,12 +91,12 @@ func main() { os.Exit(1) } - conf <- *c + conf <- c } else if *watch { log.Info("bootstrapping with minimal config") z := cdsclient.ZeroConfig(st.GetId()) - conf <- *z + conf <- z log.Infof("watching configuration at origin %q", configOrigin) ctx, cancel := context.WithCancel(context.Background()) @@ -157,7 +149,7 @@ func main() { }() case c := <-conf: - log.Trace("new configuration file available") + log.Trace("new configuration available") // command line loglevel overrides config if *verbose || *level != "" { diff --git a/config.go b/config.go index d46ea99c..601e5039 100644 --- a/config.go +++ b/config.go @@ -15,8 +15,8 @@ import ( // Options defines various options for the STUNner server. type Options struct { - // Id is the identifier of this stunnerd daemon instance. Defaults to hostname. - Id string + // Name is the identifier of this stunnerd daemon instance. Defaults to hostname. + Name string // DryRun suppresses sideeffects: STUNner will not initialize listener sockets and bring up // the TURN server, and it will not fire up the health-check and the metrics // servers. Intended for testing, default is false. @@ -144,8 +144,8 @@ func (s *Stunner) GetConfig() *stnrv1.StunnerConfig { } // LoadConfig loads a configuration from an origin. This is a shim wrapper around ConfigOrigin.Load. -func (s *Stunner) LoadConfig(config string) (*stnrv1.StunnerConfig, error) { - client, err := client.New(config, s.id, s.logger) +func (s *Stunner) LoadConfig(origin string) (*stnrv1.StunnerConfig, error) { + client, err := client.New(origin, s.name, s.logger) if err != nil { return nil, err } @@ -154,8 +154,8 @@ func (s *Stunner) LoadConfig(config string) (*stnrv1.StunnerConfig, error) { } // WatchConfig watches a configuration from an origin. This is a shim wrapper around ConfigOrigin.Watch. -func (s *Stunner) WatchConfig(ctx context.Context, origin string, ch chan<- stnrv1.StunnerConfig) error { - client, err := client.New(origin, s.id, s.logger) +func (s *Stunner) WatchConfig(ctx context.Context, origin string, ch chan<- *stnrv1.StunnerConfig) error { + client, err := client.New(origin, s.name, s.logger) if err != nil { return err } diff --git a/config_test.go b/config_test.go index dac58b49..5e87dcd2 100644 --- a/config_test.go +++ b/config_test.go @@ -66,7 +66,7 @@ func TestStunnerDefaultServerVNet(t *testing.T) { }) log.Debug("starting stunnerd") - assert.NoError(t, stunner.Reconcile(*c), "starting server") + assert.NoError(t, stunner.Reconcile(c), "starting server") log.Debug("creating a client") lconn, err := v.wan.ListenPacket("udp4", "0.0.0.0:0") @@ -145,7 +145,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel}) log.Debug("starting watcher") - conf := make(chan stnrv1.StunnerConfig, 1) + conf := make(chan *stnrv1.StunnerConfig, 1) defer close(conf) log.Debug("init watcher with nonexistent config file") @@ -187,9 +187,9 @@ func TestStunnerConfigFileWatcher(t *testing.T) { c2, ok := <-conf assert.True(t, ok, "config emitted") - checkDefaultConfig(t, &c2, "TURN-UDP") + checkDefaultConfig(t, c2, "TURN-UDP") - log.Debug("write a wrong config file (WatchConfig does not validate)") + log.Debug("write a wrong config file: WatchConfig validates") c2.Listeners[0].Protocol = "dummy" y, err = yaml.Marshal(c2) @@ -204,12 +204,20 @@ func TestStunnerConfigFileWatcher(t *testing.T) { // this makes sure that we do not share anything with ConfigWatch c2.Listeners[0].PublicAddr = "AAAAAAAAAAAAAa" - c3 := <-conf - checkDefaultConfig(t, &c3, "dummy") + // we should not read anything so that channel should not br redable + time.Sleep(50 * time.Millisecond) + readable := false + select { + case _, ok := <-conf: + readable = ok + default: + readable = false + } + assert.False(t, readable, "wrong config file does not trigger a watch event") log.Debug("update the config file and check") - c3.Listeners[0].Protocol = "TURN-TCP" - y, err = yaml.Marshal(c3) + c2.Listeners[0].Protocol = "TURN-TCP" + y, err = yaml.Marshal(c2) assert.NoError(t, err, "marshal config file") err = f.Truncate(0) assert.NoError(t, err, "truncate temp file") @@ -218,15 +226,15 @@ func TestStunnerConfigFileWatcher(t *testing.T) { _, err = f.Write(y) assert.NoError(t, err, "write config to temp file") - c4 := <-conf - checkDefaultConfig(t, &c4, "TURN-TCP") + c3 := <-conf + checkDefaultConfig(t, c3, "TURN-TCP") stunner.Close() } const ( testConfigV1 = `{"version":"v1","admin":{"name":"ns1/tester", "loglevel":"all:ERROR"},"auth":{"type":"static","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` - testConfigV1A1 = `{"version":"v1alpha1","admin":{"name":"ns1/tester", "loglevel":"all:ERROR"},"auth":{"type":"longterm","credentials":{"password":"passwd1","username":"user1"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` + testConfigV1A1 = `{"version":"v1alpha1","admin":{"name":"ns1/tester", "loglevel":"all:ERROR"},"auth":{"type":"ephemeral","credentials":{"secret":"test-secret"}},"listeners":[{"name":"udp","protocol":"turn-udp","address":"1.2.3.4","port":3478,"routes":["echo-server-cluster"]}],"clusters":[{"name":"echo-server-cluster","type":"STATIC","endpoints":["1.2.3.5"]}]}` ) // test with v1alpha1 and v1 @@ -251,7 +259,7 @@ func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel}) log.Debug("starting watcher") - conf := make(chan stnrv1.StunnerConfig, 1) + conf := make(chan *stnrv1.StunnerConfig, 1) defer close(conf) log.Debug("init watcher with nonexistent config file") @@ -285,7 +293,7 @@ func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") assert.Len(t, c2.Listeners, 1, "listeners len") assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") - assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, "TURN-UDP", c2.Listeners[0].Protocol, "listener proto") assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") @@ -310,7 +318,7 @@ func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") assert.Len(t, c2.Listeners, 1, "listeners len") assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") - assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, "TURN-UDP", c2.Listeners[0].Protocol, "listener proto") assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") @@ -391,10 +399,10 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { time.Sleep(50 * time.Millisecond) log.Debug("creating a stunnerd") - stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel, Id: "ns1/tester"}) + stunner := NewStunner(Options{LogLevel: stunnerTestLoglevel, Name: "ns1/tester"}) log.Debug("starting watcher") - conf := make(chan stnrv1.StunnerConfig, 1) + conf := make(chan *stnrv1.StunnerConfig, 1) defer close(conf) log.Debug("init config poller") @@ -408,7 +416,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") assert.Len(t, c2.Listeners, 1, "listeners len") assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") - assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, "TURN-UDP", c2.Listeners[0].Protocol, "listener proto") assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") @@ -427,7 +435,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { assert.True(t, c2.Auth.Type == "static" || c2.Auth.Type == "ephemeral", "loglevel") assert.Len(t, c2.Listeners, 1, "listeners len") assert.Equal(t, "udp", c2.Listeners[0].Name, "listener name") - assert.Equal(t, "turn-udp", c2.Listeners[0].Protocol, "listener proto") + assert.Equal(t, "TURN-UDP", c2.Listeners[0].Protocol, "listener proto") assert.Equal(t, 3478, c2.Listeners[0].Port, "listener port") assert.Len(t, c2.Listeners[0].Routes, 1, "routes len") assert.Equal(t, "echo-server-cluster", c2.Listeners[0].Routes[0], "route name") diff --git a/handlers_test.go b/handlers_test.go index a35ff00d..7eef6269 100644 --- a/handlers_test.go +++ b/handlers_test.go @@ -230,7 +230,7 @@ func TestStunnerAuthServerVNet(t *testing.T) { }) log.Debug("starting stunnerd") - assert.NoError(t, stunner.Reconcile(c), "starting server") + assert.NoError(t, stunner.Reconcile(&c), "starting server") log.Debug("creating a client") lconn, err := v.wan.ListenPacket("udp4", "0.0.0.0:0") diff --git a/pkg/apis/v1/auth.go b/pkg/apis/v1/auth.go index c3173098..ab23fd9d 100644 --- a/pkg/apis/v1/auth.go +++ b/pkg/apis/v1/auth.go @@ -55,6 +55,10 @@ func (req *AuthConfig) Validate() error { req.Realm = DefaultRealm } + if req.Credentials == nil { + req.Credentials = map[string]string{} + } + return nil } diff --git a/pkg/apis/v1/cluster.go b/pkg/apis/v1/cluster.go index 29c3a789..24dcc5cf 100644 --- a/pkg/apis/v1/cluster.go +++ b/pkg/apis/v1/cluster.go @@ -64,7 +64,12 @@ func (req *ClusterConfig) Validate() error { } } + if req.Endpoints == nil { + req.Endpoints = []string{} + } + sort.Strings(req.Endpoints) + return nil } diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 51b5bd8f..60712741 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -19,3 +19,7 @@ const DefaultMetricsPort int = 8080 const DefaultHealthCheckPort int = 8086 const DefaultConfigDiscoveryAddress = ":13478" +const DefaultEnvVarName = "STUNNER_NAME" +const DefaultEnvVarNamespace = "STUNNER_NAMESPACE" +const DefaultEnvVarNodeName = "STUNNER_NODENAME" +const DefaultEnvVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" diff --git a/pkg/apis/v1/listener.go b/pkg/apis/v1/listener.go index c972ed47..2fbb1207 100644 --- a/pkg/apis/v1/listener.go +++ b/pkg/apis/v1/listener.go @@ -71,6 +71,10 @@ func (req *ListenerConfig) Validate() error { } } + if req.Routes == nil { + req.Routes = []string{} + } + sort.Strings(req.Routes) return nil } @@ -102,7 +106,12 @@ func (req *ListenerConfig) String() string { n = req.Name } - status = append(status, fmt.Sprintf("turn://0.0.0.0:%d", req.Port)) + addr := "0.0.0.0" + if req.Addr != "" && req.Addr != "$STUNNER_ADDR" { + addr = req.Addr + } + + status = append(status, fmt.Sprintf("turn://%s:%d", addr, req.Port)) a, p := "-", "-" if req.PublicAddr != "" { diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go index 23e5054b..824a402b 100644 --- a/pkg/apis/v1/stunner.go +++ b/pkg/apis/v1/stunner.go @@ -67,37 +67,38 @@ func (req *StunnerConfig) ConfigName() string { } // DeepEqual compares two configurations. -func (req *StunnerConfig) DeepEqual(conf Config) bool { - other, ok := conf.(*StunnerConfig) +func (a *StunnerConfig) DeepEqual(conf Config) bool { + b, ok := conf.(*StunnerConfig) if !ok { return false } - if req.ApiVersion != other.ApiVersion { + if a.ApiVersion != b.ApiVersion { return false } - if !req.Admin.DeepEqual(&other.Admin) { + + if !a.Admin.DeepEqual(&b.Admin) { return false } - if !req.Auth.DeepEqual(&other.Auth) { + + if !a.Auth.DeepEqual(&b.Auth) { return false } - if len(req.Listeners) != len(other.Listeners) { + if len(a.Listeners) != len(b.Listeners) { return false } - for i := range req.Listeners { - if !req.Listeners[i].DeepEqual(&other.Listeners[i]) { + for i := range a.Listeners { + if !a.Listeners[i].DeepEqual(&b.Listeners[i]) { return false } } - if len(req.Clusters) != len(other.Clusters) { + if len(a.Clusters) != len(b.Clusters) { return false } - - for i := range req.Clusters { - if !req.Clusters[i].DeepEqual(&other.Clusters[i]) { + for i := range a.Clusters { + if !a.Clusters[i].DeepEqual(&b.Clusters[i]) { return false } } diff --git a/pkg/config/api/stunner_openapi.yaml b/pkg/config/api/stunner_openapi.yaml index d2932f21..e983d432 100644 --- a/pkg/config/api/stunner_openapi.yaml +++ b/pkg/config/api/stunner_openapi.yaml @@ -20,17 +20,21 @@ paths: schema: type: boolean responses: - "501": - description: Unimplemented - # "200": - # description: OK - # content: - # application/json: - # schema: - # $ref: '#/components/schemas/v1.ConfigList' + "200": + description: OK + content: + application/json: + schema: + $ref: '#/components/schemas/v1.ConfigList' # application/json;stream=watch: # schema: # $ref: '#/components/schemas/v1.ConfigList' + "500": + description: Internal server error. + content: + application/json: + schema: + $ref: '#/components/schemas/v1.Error' /api/v1/configs/{namespace}: get: description: API to list or watch config objects in a specific namespace. @@ -48,14 +52,24 @@ paths: schema: type: boolean responses: - "501": - description: Unimplemented - # "200": - # description: OK - # content: - # application/json: - # schema: - # $ref: '#/components/schemas/v1.ConfigList' + "200": + description: OK + content: + application/json: + schema: + $ref: '#/components/schemas/v1.ConfigList' + "400": + description: Bad request. + content: + application/json: + schema: + $ref: '#/components/schemas/v1.Error' + "500": + description: Internal server error. + content: + application/json: + schema: + $ref: '#/components/schemas/v1.Error' # application/json;stream=watch: # schema: # $ref: '#/components/schemas/v1.ConfigList' @@ -81,6 +95,11 @@ paths: description: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. schema: type: boolean + - name: node + in: query + description: Name of the node the client runs on. + schema: + type: string responses: "200": description: OK @@ -88,15 +107,40 @@ paths: application/json: schema: $ref: '#/components/schemas/v1.Config' - application/json;stream=watch: + "400": + description: Bad request. + content: + application/json: schema: - $ref: '#/components/schemas/v1.Config' + $ref: '#/components/schemas/v1.Error' + "500": + description: Internal server error. + content: + application/json: + schema: + $ref: '#/components/schemas/v1.Error' + # application/json;stream=watch: + # schema: + # $ref: '#/components/schemas/v1.Config' components: schemas: + v1.ClientInfo: + description: Client description. + type: object + required: + - id + properties: + id: + description: Client id. + type: string + node: + description: Name of the node client is deployed to. + type: string v1.ConfigList: description: ConfigList is a list of Configs. type: object required: + - version - items properties: version: @@ -107,12 +151,22 @@ components: type: array items: $ref: '#/components/schemas/v1.Config' - kind: - description: Kind is a string value representing the REST resource this object represents. - type: string v1.Config: description: 'Config provides a STUNner config. Schema is defined in https://github.com/l7mp/stunner/tree/main/pkg/apis/v1' x-go-type: "stunnerv1.StunnerConfig" x-go-type-import: name: stunnerv1 path: "github.com/l7mp/stunner/pkg/apis/v1" + v1.Error: + description: API error. + type: object + required: + - code + properties: + code: + description: Error code. + type: integer + format: int32 + message: + description: Error message. + type: string diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index 9603317f..28734f4a 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -29,7 +29,6 @@ func init() { client.PongWait = 800 * time.Millisecond client.WriteWait = 200 * time.Millisecond client.RetryPeriod = 250 * time.Millisecond - } func TestServerLoad(t *testing.T) { @@ -47,7 +46,7 @@ func TestServerLoad(t *testing.T) { defer cancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(ctx) assert.NoError(t, err, "start") @@ -83,9 +82,11 @@ func TestServerLoad(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") @@ -133,7 +134,7 @@ func TestServerPoll(t *testing.T) { defer cancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(ctx) assert.NoError(t, err, "start") @@ -149,11 +150,11 @@ func TestServerPoll(t *testing.T) { assert.NoError(t, err, "client 3") testLog.Debug("poll: no result") - ch1 := make(chan stnrv1.StunnerConfig, 8) + ch1 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch1) - ch2 := make(chan stnrv1.StunnerConfig, 8) + ch2 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch2) - ch3 := make(chan stnrv1.StunnerConfig, 8) + ch3 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch3) go func() { @@ -186,9 +187,11 @@ func TestServerPoll(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") @@ -233,7 +236,7 @@ func TestServerWatch(t *testing.T) { serverCtx, serverCancel := context.WithCancel(context.Background()) testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -247,11 +250,11 @@ func TestServerWatch(t *testing.T) { assert.NoError(t, err, "client 3") testLog.Debug("watch: no result") - ch1 := make(chan stnrv1.StunnerConfig, 8) + ch1 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch1) - ch2 := make(chan stnrv1.StunnerConfig, 8) + ch2 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch2) - ch3 := make(chan stnrv1.StunnerConfig, 8) + ch3 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch3) clientCtx, clientCancel := context.WithCancel(context.Background()) @@ -280,9 +283,11 @@ func TestServerWatch(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") @@ -307,12 +312,15 @@ func TestServerWatch(t *testing.T) { assert.Len(t, cs, 3, "snapshot len") sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") sc2 = srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") sc3 = srv.GetConfigStore().Get("ns1/gw3") assert.NotNil(t, sc3, "get 3") + assert.NoError(t, sc3.Validate(), "valid") // loaders validate assert.True(t, c3.Config.DeepEqual(sc3), "deepeq 3") // poll should have fed the configs to the channels @@ -331,7 +339,7 @@ func TestServerWatch(t *testing.T) { time.Sleep(50 * time.Millisecond) serverCtx, serverCancel = context.WithCancel(context.Background()) defer serverCancel() - srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -357,11 +365,13 @@ func TestServerWatch(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") sc2 = srv.GetConfigStore().Get("ns1/gw2") assert.Nil(t, sc2, "get 2") sc3 = srv.GetConfigStore().Get("ns1/gw3") assert.NotNil(t, sc3, "get 3") + assert.NoError(t, sc3.Validate(), "valid") // loaders validate assert.True(t, c3.Config.DeepEqual(sc3), "deepeq 3") s = watchConfig(ch1, 50*time.Millisecond) @@ -403,7 +413,7 @@ func TestServerWatchBootstrap(t *testing.T) { defer serverCancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -422,15 +432,17 @@ func TestServerWatchBootstrap(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc3 := srv.GetConfigStore().Get("ns1/gw3") assert.Nil(t, sc3, "get 3") testLog.Debug("watch: 1 result") - ch1 := make(chan stnrv1.StunnerConfig, 8) + ch1 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch1) clientCtx, clientCancel := context.WithCancel(context.Background()) @@ -455,9 +467,11 @@ func TestServerWatchBootstrap(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") sc2 = srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") s = watchConfig(ch1, 500*time.Millisecond) @@ -491,7 +505,7 @@ func TestServerAPI(t *testing.T) { serverCtx, serverCancel := context.WithCancel(context.Background()) testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -507,13 +521,13 @@ func TestServerAPI(t *testing.T) { assert.NoError(t, err, "client 4") testLog.Debug("watch: no result") - ch1 := make(chan stnrv1.StunnerConfig, 8) + ch1 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch1) - ch2 := make(chan stnrv1.StunnerConfig, 8) + ch2 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch2) - ch3 := make(chan stnrv1.StunnerConfig, 8) + ch3 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch3) - ch4 := make(chan stnrv1.StunnerConfig, 8) + ch4 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch4) clientCtx, clientCancel := context.WithCancel(context.Background()) @@ -548,10 +562,12 @@ func TestServerAPI(t *testing.T) { cs := srv.GetConfigStore().Snapshot() assert.Len(t, cs, 2, "snapshot len") sc1 := srv.GetConfigStore().Get("ns1/gw1") - assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate + assert.True(t, c1.Config.DeepEqual(sc1), "deepeq 1") sc2 := srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq 2") testLog.Debug("load") @@ -632,12 +648,15 @@ func TestServerAPI(t *testing.T) { assert.Len(t, cs, 3, "snapshot len") sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 = srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") sc3 := srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc3, "get 3") + assert.NoError(t, sc3.Validate(), "valid") // loaders validate assert.True(t, c3.Config.DeepEqual(sc3), "deepeq") // all-configs should result sc1 and sc2 and sc3 @@ -719,7 +738,7 @@ func TestServerAPI(t *testing.T) { time.Sleep(50 * time.Millisecond) serverCtx, serverCancel = context.WithCancel(context.Background()) defer serverCancel() - srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -730,12 +749,15 @@ func TestServerAPI(t *testing.T) { assert.Len(t, cs, 3, "snapshot len") sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 = srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") sc3 = srv.GetConfigStore().Get("ns1/gw2") assert.NotNil(t, sc3, "get 3") + assert.NoError(t, sc3.Validate(), "valid") // loaders validate assert.True(t, c3.Config.DeepEqual(sc3), "deepeq") // all-configs should result sc1 and sc2 and sc3 @@ -830,12 +852,15 @@ func TestServerAPI(t *testing.T) { assert.Len(t, cs, 3, "snapshot len") sc1 = srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") sc2 = srv.GetConfigStore().Get("ns2/gw1") assert.NotNil(t, sc2, "get 2") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, c2.Config.DeepEqual(sc2), "deepeq") sc4 := srv.GetConfigStore().Get("ns3/gw1") assert.NotNil(t, sc3, "get 3") + assert.NoError(t, sc3.Validate(), "valid") // loaders validate assert.True(t, c4.Config.DeepEqual(sc4), "deepeq") // all-configs should result sc1 and sc2 and sc4 @@ -915,7 +940,7 @@ func TestClientReconnect(t *testing.T) { defer serverCancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -925,7 +950,7 @@ func TestClientReconnect(t *testing.T) { assert.NoError(t, err, "client 1") testLog.Debug("watch: no result") - ch1 := make(chan stnrv1.StunnerConfig, 8) + ch1 := make(chan *stnrv1.StunnerConfig, 8) defer close(ch1) clientCtx, clientCancel := context.WithCancel(context.Background()) @@ -945,6 +970,7 @@ func TestClientReconnect(t *testing.T) { assert.Len(t, cs, 1, "snapshot len") sc1 := srv.GetConfigStore().Get("ns1/gw1") assert.NotNil(t, sc1, "get 1") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, c1.Config.DeepEqual(sc1), "deepeq") // poll should have fed the config to the channels @@ -986,12 +1012,13 @@ func TestServerUpdate(t *testing.T) { defer serverCancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, log) + srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") - oldC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/udp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","username":"a","password":"b"},"listeners":[{"name": "stunner/udp-gateway/udp-listener", "protocol":"turn-udp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[]}`)) + oldC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/udp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","credentials":{"username":"a","password":"b"}},"listeners":[{"name": "stunner/udp-gateway/udp-listener", "protocol":"turn-udp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[]}`)) + assert.NoError(t, oldC.Validate(), "validate") assert.NoError(t, err, "parse 1") srv.UpsertConfig("stunner/udp-gateway", oldC) @@ -1000,6 +1027,7 @@ func TestServerUpdate(t *testing.T) { assert.Len(t, cs, 1, "snapshot len") sc1 := srv.GetConfigStore().Get("stunner/udp-gateway") assert.NotNil(t, sc1, "get") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, sc1.DeepEqual(oldC), "deepeq") // reapply - no change @@ -1010,10 +1038,12 @@ func TestServerUpdate(t *testing.T) { assert.Len(t, cs, 1, "snapshot len") sc1 = srv.GetConfigStore().Get("stunner/udp-gateway") assert.NotNil(t, sc1, "get") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, sc1.DeepEqual(oldC), "deepeq") // add another config - tcpC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/tcp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","username":"a","password":"b"},"listeners":[{"name": "stunner/tcp-gateway/tcp-listener", "protocol":"turn-tcp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[{"name":"stunner/media-plane", "type":"STATIC","protocol":"UDP","endpoints":["0.0.0.0/0"]}]}`)) + tcpC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/tcp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","credentials":{"username":"a","password":"b"}},"listeners":[{"name": "stunner/tcp-gateway/tcp-listener", "protocol":"turn-tcp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[{"name":"stunner/media-plane", "type":"STATIC","protocol":"UDP","endpoints":["0.0.0.0/0"]}]}`)) + assert.NoError(t, tcpC.Validate(), "validate") assert.NoError(t, err, "parse") srv.UpsertConfig("stunner/tcp-gateway", tcpC) @@ -1023,14 +1053,17 @@ func TestServerUpdate(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 = srv.GetConfigStore().Get("stunner/udp-gateway") assert.NotNil(t, sc1, "get") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, sc1.DeepEqual(oldC), "deepeq") sc2 := srv.GetConfigStore().Get("stunner/tcp-gateway") assert.NotNil(t, sc2, "get") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, sc2.DeepEqual(tcpC), "deepeq") // add a cluster - newC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/udp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","username":"a","password":"b"},"listeners":[{"name": "stunner/udp-gateway/udp-listener", "protocol":"turn-udp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[{"name": "stunner/media-plane", "type":"STATIC","protocol":"UDP","endpoints":["0.0.0.0/0"]}]}`)) + newC, err := client.ParseConfig([]byte(`{"version":"v1","admin":{"name":"stunner/udp-gateway","logLevel":"all:INFO","health-check":"http://:8086"},"auth":{"realm":"stunner.l7mp.io","type":"static","credentials":{"username":"a","password":"b"}},"listeners":[{"name": "stunner/udp-gateway/udp-listener", "protocol":"turn-udp","address":"0.0.0.0","port":3478,"routes":["stunner/media-plane"]}],"clusters":[{"name": "stunner/media-plane", "type":"STATIC","protocol":"UDP","endpoints":["0.0.0.0/0"]}]}`)) assert.NoError(t, err, "parse 1") + assert.NoError(t, newC.Validate(), "validate") assert.False(t, oldC.DeepEqual(newC), "deepeq") // process in a single go @@ -1043,43 +1076,162 @@ func TestServerUpdate(t *testing.T) { assert.Len(t, cs, 2, "snapshot len") sc1 = srv.GetConfigStore().Get("stunner/udp-gateway") assert.NotNil(t, sc1, "get") + assert.NoError(t, sc1.Validate(), "valid") // loaders validate assert.True(t, sc1.DeepEqual(newC), "deepeq") sc2 = srv.GetConfigStore().Get("stunner/tcp-gateway") assert.NotNil(t, sc2, "get") + assert.NoError(t, sc2.Validate(), "valid") // loaders validate assert.True(t, sc2.DeepEqual(tcpC), "deepeq") } +// func TestServerPatcher(t *testing.T) { +// zc := zap.NewProductionConfig() +// zc.Level = zap.NewAtomicLevelAt(testerLogLevel) +// z, err := zc.Build() +// assert.NoError(t, err, "logger created") +// zlogger := zapr.NewLogger(z) +// log := zlogger.WithName("tester") + +// logger := logger.NewLoggerFactory(stunnerLogLevel) +// testLog := logger.NewLogger("test") + +// ctx, cancel := context.WithCancel(context.Background()) +// defer cancel() + +// testLog.Debug("create server") +// patcher := func(conf *stnrv1.StunnerConfig, node string) (*stnrv1.StunnerConfig, error) { +// testLog.Debugf("patching config: %s", conf.String()) +// if conf == nil { +// return nil, fmt.Errorf("config patcher: nil config received") +// } +// for i := range conf.Listeners { +// conf.Listeners[i].Addr = node +// } +// testLog.Tracef(" patching ready: %s", conf.String()) +// return conf, nil +// } +// srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, patcher, log) +// assert.NotNil(t, srv, "server") +// err = srv.Start(ctx) +// assert.NoError(t, err, "start") + +// time.Sleep(20 * time.Millisecond) + +// c := testConfigListener("ns1/gw1", "realm1", "1.2.3.4") +// err = srv.UpdateConfig([]server.Config{c}) +// assert.NoError(t, err, "update") +// // expected result +// p := testConfigListener("ns1/gw1", "realm1", "10.11.12.13") +// p.Config.Listeners[0].Addr = "10.11.12.13" +// p.Config.Listeners[1].Addr = "10.11.12.13" +// p.Config.Listeners[2].Addr = "10.11.12.13" + +// testLog.Debug("client w/o node IP") +// loader1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) +// assert.NoError(t, err, "client") +// sc1, err := loader1.Load() +// assert.NoError(t, err, "load") +// assert.True(t, sc1.DeepEqual(c.Config), "deepeq") + +// testLog.Debug("client w/ node IP") +// loader2, err := client.New("127.0.0.1:13478", "ns1/gw1", map[string]string{"node": "10.11.12.13"}, logger) +// assert.NoError(t, err, "client") +// sc2, err := loader2.Load() +// assert.NoError(t, err, "load") +// assert.True(t, sc2.DeepEqual(p.Config), "deepeq") + +// watchCtx, watchCancel := context.WithCancel(context.Background()) +// defer watchCancel() + +// testLog.Debug("watcher1 w/o node IP") +// watcher1, err := client.New("127.0.0.1:13478", "ns1/gw1", nil, logger) +// assert.NoError(t, err, "client") +// ch1 := make(chan *stnrv1.StunnerConfig, 8) +// defer close(ch1) + +// err = watcher1.Watch(watchCtx, ch1) +// assert.NoError(t, err, "client watch") + +// s := watchConfig(ch1, 100*time.Millisecond) +// assert.NotNil(t, s, "watch-config") +// assert.True(t, s.DeepEqual(c.Config), "deepeq") + +// testLog.Debug("watcher2 w/ node IP") +// watcher2, err := client.New("127.0.0.1:13478", "ns1/gw1", map[string]string{"node": "10.11.12.13"}, logger) +// assert.NoError(t, err, "client") +// ch2 := make(chan *stnrv1.StunnerConfig, 8) +// defer close(ch2) + +// err = watcher2.Watch(watchCtx, ch2) +// assert.NoError(t, err, "client watch") + +// s = watchConfig(ch2, 100*time.Millisecond) +// assert.NotNil(t, s, "watch-config") +// assert.True(t, s.DeepEqual(p.Config), "deepeq") + +// // testing update +// c = testConfigListener("ns1/gw1", "realm1", "8.7.6.5") +// err = srv.UpdateConfig([]server.Config{c}) +// assert.NoError(t, err, "update") + +// testLog.Debug("client w/o node IP") +// sc1, err = loader1.Load() +// assert.NoError(t, err, "load") +// assert.True(t, sc1.DeepEqual(c.Config), "deepeq") + +// testLog.Debug("client w/ node IP") +// sc2, err = loader2.Load() +// assert.NoError(t, err, "load") +// assert.True(t, sc2.DeepEqual(p.Config), "deepeq") + +// s = watchConfig(ch1, 100*time.Millisecond) +// assert.NotNil(t, s, "watch-config") +// assert.True(t, s.DeepEqual(c.Config), "deepeq") + +// s = watchConfig(ch2, 100*time.Millisecond) +// assert.NotNil(t, s, "watch-config") +// assert.True(t, s.DeepEqual(p.Config), "deepeq") +// } + // only differ in id and realm func testConfig(id, realm string) server.Config { c := client.ZeroConfig(id) c.Auth.Realm = realm - + _ = c.Validate() // make sure deepeq works return server.Config{Id: id, Config: c} } +// func testConfigListener(id, realm, addr string) server.Config { +// c := client.ZeroConfig(id) +// c.Auth.Realm = realm +// c.Listeners = []stnrv1.ListenerConfig{{ +// Name: "l-1", +// Addr: addr, +// Port: 1, +// }, { +// Name: "l-2", +// Port: 2, +// }, { +// Name: "l-3", +// Addr: "101.102.103.104", +// Port: 3, +// }} +// _ = c.Validate() // make sure deepeq works +// return server.Config{Id: id, Config: c} +// } + // wait for some configurable time for a watch element -func watchConfig(ch chan stnrv1.StunnerConfig, d time.Duration) *stnrv1.StunnerConfig { +func watchConfig(ch chan *stnrv1.StunnerConfig, d time.Duration) *stnrv1.StunnerConfig { select { case c := <-ch: // fmt.Println("++++++++++++ got config ++++++++++++: ", c.String()) - return &c + return c case <-time.After(d): // fmt.Println("++++++++++++ timeout ++++++++++++") return nil } } -// func watchServerConfig(ch chan server.Config, d time.Duration) *stnrv1.StunnerConfig { -// select { -// case c := <-ch: -// // fmt.Println("++++++++++++ got config ++++++++++++: ", c.String()) -// return c.Config -// case <-time.After(d): -// // fmt.Println("++++++++++++ timeout ++++++++++++") -// return nil -// } -// } - func findConfById(cs []*stnrv1.StunnerConfig, id string) *stnrv1.StunnerConfig { for _, c := range cs { if c != nil && c.Admin.Name == id { diff --git a/pkg/config/client/api/client.gen.go b/pkg/config/client/api/client.gen.go index d7cd5cdd..35d3965b 100644 --- a/pkg/config/client/api/client.gen.go +++ b/pkg/config/client/api/client.gen.go @@ -19,6 +19,24 @@ import ( // V1Config Config provides a STUNner config. Schema is defined in https://github.com/l7mp/stunner/tree/main/pkg/apis/v1 type V1Config = stunnerv1.StunnerConfig +// V1ConfigList ConfigList is a list of Configs. +type V1ConfigList struct { + // Items Items is the list of Config objects in the list. + Items []V1Config `json:"items"` + + // Version version defines the versioned schema of this object. + Version string `json:"version"` +} + +// V1Error API error. +type V1Error struct { + // Code Error code. + Code int32 `json:"code"` + + // Message Error message. + Message *string `json:"message,omitempty"` +} + // ListV1ConfigsParams defines parameters for ListV1Configs. type ListV1ConfigsParams struct { // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. @@ -35,6 +53,9 @@ type ListV1ConfigsNamespaceParams struct { type GetV1ConfigNamespaceNameParams struct { // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` + + // Node Name of the node the client runs on. + Node *string `form:"node,omitempty" json:"node,omitempty"` } // RequestEditorFn is the function signature for the RequestEditor callback function @@ -313,6 +334,22 @@ func NewGetV1ConfigNamespaceNameRequest(server string, namespace string, name st } + if params.Node != nil { + + if queryFrag, err := runtime.StyleParamWithLocation("form", true, "node", runtime.ParamLocationQuery, *params.Node); err != nil { + return nil, err + } else if parsed, err := url.ParseQuery(queryFrag); err != nil { + return nil, err + } else { + for k, v := range parsed { + for _, v2 := range v { + queryValues.Add(k, v2) + } + } + } + + } + queryURL.RawQuery = queryValues.Encode() } @@ -380,6 +417,8 @@ type ClientWithResponsesInterface interface { type ListV1ConfigsResponse struct { Body []byte HTTPResponse *http.Response + JSON200 *V1ConfigList + JSON500 *V1Error } // Status returns HTTPResponse.Status @@ -401,6 +440,9 @@ func (r ListV1ConfigsResponse) StatusCode() int { type ListV1ConfigsNamespaceResponse struct { Body []byte HTTPResponse *http.Response + JSON200 *V1ConfigList + JSON400 *V1Error + JSON500 *V1Error } // Status returns HTTPResponse.Status @@ -423,6 +465,8 @@ type GetV1ConfigNamespaceNameResponse struct { Body []byte HTTPResponse *http.Response JSON200 *V1Config + JSON400 *V1Error + JSON500 *V1Error } // Status returns HTTPResponse.Status @@ -481,6 +525,23 @@ func ParseListV1ConfigsResponse(rsp *http.Response) (*ListV1ConfigsResponse, err HTTPResponse: rsp, } + switch { + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 200: + var dest V1ConfigList + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON200 = &dest + + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 500: + var dest V1Error + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON500 = &dest + + } + return response, nil } @@ -497,6 +558,30 @@ func ParseListV1ConfigsNamespaceResponse(rsp *http.Response) (*ListV1ConfigsName HTTPResponse: rsp, } + switch { + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 200: + var dest V1ConfigList + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON200 = &dest + + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 400: + var dest V1Error + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON400 = &dest + + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 500: + var dest V1Error + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON500 = &dest + + } + return response, nil } @@ -521,8 +606,19 @@ func ParseGetV1ConfigNamespaceNameResponse(rsp *http.Response) (*GetV1ConfigName } response.JSON200 = &dest - case rsp.StatusCode == 200: - // Content-type (application/json;stream=watch) unsupported + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 400: + var dest V1Error + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON400 = &dest + + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 500: + var dest V1Error + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSON500 = &dest } diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go index a61a775c..9960cf35 100644 --- a/pkg/config/client/cds_api.go +++ b/pkg/config/client/cds_api.go @@ -24,16 +24,16 @@ type ConfigList struct { Items []*stnrv1.StunnerConfig `json:"items"` } -type CDSAPI interface { +type CdsApi interface { // Endpoint returns the address of the server plus the WebSocket API endpoint. Endpoint() (string, string) // Get loads the config(s) from the API endpoint. Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error) // Watch watches config(s) from the API endpoint of a CDS server. If the server is not // available watch will retry, and if the connection goes away it will create a new one. - Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error // Poll creates a one-shot config watcher without the retry mechanincs of Watch. - Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error logging.LeveledLogger } @@ -44,7 +44,7 @@ type AllConfigsAPI struct { logging.LeveledLogger } -func NewAllConfigsAPI(addr string, logger logging.LeveledLogger) (CDSAPI, error) { +func NewAllConfigsAPI(addr string, logger logging.LeveledLogger) (CdsApi, error) { httpuri, err := getURI(addr) if err != nil { return nil, err @@ -88,12 +88,12 @@ func (a *AllConfigsAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error return decodeConfigList(r.Body) } -func (a *AllConfigsAPI) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (a *AllConfigsAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { a.Debugf("WATCH: watching all configs from CDS server %s", a.wsURI) return watch(ctx, a, ch) } -func (a *AllConfigsAPI) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (a *AllConfigsAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { a.Debugf("POLL: polling all configs from CDS server %s", a.wsURI) return poll(ctx, a, ch) } @@ -105,7 +105,7 @@ type ConfigsNamespaceAPI struct { logging.LeveledLogger } -func NewConfigsNamespaceAPI(addr, namespace string, logger logging.LeveledLogger) (CDSAPI, error) { +func NewConfigsNamespaceAPI(addr, namespace string, logger logging.LeveledLogger) (CdsApi, error) { httpuri, err := getURI(addr) if err != nil { return nil, err @@ -152,13 +152,13 @@ func (a *ConfigsNamespaceAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, return decodeConfigList(r.Body) } -func (a *ConfigsNamespaceAPI) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (a *ConfigsNamespaceAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { a.Debugf("WATCH: watching all configs in namespace %s from CDS server %s", a.namespace, a.wsURI) return watch(ctx, a, ch) } -func (a *ConfigsNamespaceAPI) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (a *ConfigsNamespaceAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { a.Debugf("POLL: polling all configs in namespace %s from CDS server %s", a.namespace, a.wsURI) return poll(ctx, a, ch) @@ -170,7 +170,7 @@ type ConfigNamespaceNameAPI struct { logging.LeveledLogger } -func NewConfigNamespaceNameAPI(addr, namespace, name string, logger logging.LeveledLogger) (CDSAPI, error) { +func NewConfigNamespaceNameAPI(addr, namespace, name string, logger logging.LeveledLogger) (CdsApi, error) { httpuri, err := getURI(addr) if err != nil { return nil, err @@ -205,7 +205,8 @@ func (a *ConfigNamespaceNameAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConf a.Debugf("GET: loading config for gateway %s/%s from CDS server %s", a.namespace, a.name, a.addr) - r, err := a.client.GetV1ConfigNamespaceNameWithResponse(ctx, a.namespace, a.name, nil) + var params *api.GetV1ConfigNamespaceNameParams + r, err := a.client.GetV1ConfigNamespaceNameWithResponse(ctx, a.namespace, a.name, params) if err != nil { return []*stnrv1.StunnerConfig{}, err } @@ -218,13 +219,13 @@ func (a *ConfigNamespaceNameAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConf return decodeConfig(r.Body) } -func (a *ConfigNamespaceNameAPI) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (a *ConfigNamespaceNameAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { a.Debugf("WATCH: watching config for gateway %s/%s from CDS server %s", a.namespace, a.name, a.wsURI) return watch(ctx, a, ch) } -func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { a.Debugf("POLL: polling config for gateway %s/%s from CDS server %s", a.namespace, a.name, a.wsURI) return poll(ctx, a, ch) diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go index ee4cffaa..79f7bbf3 100644 --- a/pkg/config/client/cds_client.go +++ b/pkg/config/client/cds_client.go @@ -16,7 +16,7 @@ import ( // CDSClient is a client for the config discovery service that knows how to poll configs for a // specific gateway. Use the CDSAPI to access the general CDS client set. type CDSClient struct { - CDSAPI + CdsApi addr, id string } @@ -31,7 +31,7 @@ func NewCDSClient(addr, id string, logger logging.LeveledLogger) (Client, error) return nil, err } - return &CDSClient{CDSAPI: client, addr: addr, id: id}, nil + return &CDSClient{CdsApi: client, addr: addr, id: id}, nil } func (p *CDSClient) String() string { @@ -39,7 +39,7 @@ func (p *CDSClient) String() string { } func (p *CDSClient) Load() (*stnrv1.StunnerConfig, error) { - configs, err := p.CDSAPI.Get(context.Background()) + configs, err := p.CdsApi.Get(context.Background()) if err != nil { return nil, err } @@ -47,13 +47,17 @@ func (p *CDSClient) Load() (*stnrv1.StunnerConfig, error) { return nil, fmt.Errorf("expected exactly one config, got %d", len(configs)) } - return configs[0], nil + c := configs[0] + if err := c.Validate(); err != nil { + return nil, fmt.Errorf("invalid config: %w", err) + } + + return c, nil } -func watch(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error { +func watch(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { go func() { for { - // try to watch if err := poll(ctx, a, ch); err != nil { _, wsuri := a.Endpoint() a.Errorf("failed to init CDS watcher (url: %s): %s", wsuri, err.Error()) @@ -70,7 +74,7 @@ func watch(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error return nil } -func poll(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error { +func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { _, url := a.Endpoint() a.Tracef("poll: trying to open connection to CDS server at %s", url) @@ -150,12 +154,14 @@ func poll(ctx context.Context, a CDSAPI, ch chan<- stnrv1.StunnerConfig) error { continue } - confCopy := stnrv1.StunnerConfig{} - c.DeepCopyInto(&confCopy) + if err := c.Validate(); err != nil { + a.Warnf("invalid config: %s", err.Error()) + continue + } - a.Debugf("new config received from %q: %q", url, confCopy.String()) + a.Debugf("new config received from %q: %q", url, c.String()) - ch <- confCopy + ch <- c } }() diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index 7d054ff5..e2049047 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -36,9 +36,9 @@ type Client interface { // Watch grabs new configs from a config origin (config file or CDS server) and returns // them on the channel. The context cancels the watcher. If the origin is not available // watch will retry. - Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error // Poll creates a one-shot config watcher without the retry mechanincs of Watch. - Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error + Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error fmt.Stringer } diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index 9e6bbe7c..ff9dc4a8 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -45,12 +45,21 @@ func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { return nil, errFileTruncated } - return ParseConfig(b) + c, err := ParseConfig(b) + if err != nil { + return nil, fmt.Errorf("could not parse config: %w", err) + } + + if err := c.Validate(); err != nil { + return nil, fmt.Errorf("invalid config: %w", err) + } + + return c, nil } // WatchConfig watches a configuration file for changes. If no file exists at the given path, // WatchConfig will periodically retry until the file appears. -func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { if w.configFile == "" { return errors.New("uninitialized config file path") } @@ -76,7 +85,7 @@ func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- stnrv1.StunnerCo // Poll watches the config file and emits new configs on the specified channel. Returns an error if // further action is needed (tryWatchConfig is to be started) or nil on normal exit. -func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- stnrv1.StunnerConfig) error { +func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { w.log.Tracef("configWatcher") // create a new watcher @@ -97,13 +106,9 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- stnrv1.StunnerCon return err } - // send a deepcopy over the channel - confCopy := stnrv1.StunnerConfig{} - c.DeepCopyInto(&confCopy) - - w.log.Debugf("initial config file successfully loaded from %q: %s", config, confCopy.String()) + w.log.Debugf("initial config file successfully loaded from %q: %s", config, c.String()) - ch <- confCopy + ch <- c // save deepcopy so that we can filter repeated events prev := stnrv1.StunnerConfig{} @@ -152,12 +157,9 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- stnrv1.StunnerCon continue } - confCopy := stnrv1.StunnerConfig{} - c.DeepCopyInto(&confCopy) - - w.log.Debugf("config file successfully loaded from %q: %s", config, confCopy.String()) + w.log.Debugf("config file successfully loaded from %q: %s", config, c.String()) - ch <- confCopy + ch <- c // save deepcopy so that we can filter repeated events c.DeepCopyInto(&prev) diff --git a/pkg/config/server/api/server.gen.go b/pkg/config/server/api/server.gen.go index ab4afe57..09db75ac 100644 --- a/pkg/config/server/api/server.gen.go +++ b/pkg/config/server/api/server.gen.go @@ -6,7 +6,9 @@ package api import ( "bytes" "compress/gzip" + "context" "encoding/base64" + "encoding/json" "fmt" "net/http" "net/url" @@ -17,11 +19,30 @@ import ( "github.com/gorilla/mux" stunnerv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/oapi-codegen/runtime" + strictnethttp "github.com/oapi-codegen/runtime/strictmiddleware/nethttp" ) // V1Config Config provides a STUNner config. Schema is defined in https://github.com/l7mp/stunner/tree/main/pkg/apis/v1 type V1Config = stunnerv1.StunnerConfig +// V1ConfigList ConfigList is a list of Configs. +type V1ConfigList struct { + // Items Items is the list of Config objects in the list. + Items []V1Config `json:"items"` + + // Version version defines the versioned schema of this object. + Version string `json:"version"` +} + +// V1Error API error. +type V1Error struct { + // Code Error code. + Code int32 `json:"code"` + + // Message Error message. + Message *string `json:"message,omitempty"` +} + // ListV1ConfigsParams defines parameters for ListV1Configs. type ListV1ConfigsParams struct { // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. @@ -38,6 +59,9 @@ type ListV1ConfigsNamespaceParams struct { type GetV1ConfigNamespaceNameParams struct { // Watch Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Watch *bool `form:"watch,omitempty" json:"watch,omitempty"` + + // Node Name of the node the client runs on. + Node *string `form:"node,omitempty" json:"node,omitempty"` } // ServerInterface represents all server handlers. @@ -162,6 +186,14 @@ func (siw *ServerInterfaceWrapper) GetV1ConfigNamespaceName(w http.ResponseWrite return } + // ------------- Optional query parameter "node" ------------- + + err = runtime.BindQueryParameter("form", true, false, "node", r.URL.Query(), ¶ms.Node) + if err != nil { + siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "node", Err: err}) + return + } + handler := http.Handler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { siw.Handler.GetV1ConfigNamespaceName(w, r, namespace, name, params) })) @@ -295,22 +327,247 @@ func HandlerWithOptions(si ServerInterface, options GorillaServerOptions) http.H return r } +type ListV1ConfigsRequestObject struct { + Params ListV1ConfigsParams +} + +type ListV1ConfigsResponseObject interface { + VisitListV1ConfigsResponse(w http.ResponseWriter) error +} + +type ListV1Configs200JSONResponse V1ConfigList + +func (response ListV1Configs200JSONResponse) VisitListV1ConfigsResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(200) + + return json.NewEncoder(w).Encode(response) +} + +type ListV1Configs500JSONResponse V1Error + +func (response ListV1Configs500JSONResponse) VisitListV1ConfigsResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(500) + + return json.NewEncoder(w).Encode(response) +} + +type ListV1ConfigsNamespaceRequestObject struct { + Namespace string `json:"namespace"` + Params ListV1ConfigsNamespaceParams +} + +type ListV1ConfigsNamespaceResponseObject interface { + VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error +} + +type ListV1ConfigsNamespace200JSONResponse V1ConfigList + +func (response ListV1ConfigsNamespace200JSONResponse) VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(200) + + return json.NewEncoder(w).Encode(response) +} + +type ListV1ConfigsNamespace400JSONResponse V1Error + +func (response ListV1ConfigsNamespace400JSONResponse) VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(400) + + return json.NewEncoder(w).Encode(response) +} + +type ListV1ConfigsNamespace500JSONResponse V1Error + +func (response ListV1ConfigsNamespace500JSONResponse) VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(500) + + return json.NewEncoder(w).Encode(response) +} + +type GetV1ConfigNamespaceNameRequestObject struct { + Namespace string `json:"namespace"` + Name string `json:"name"` + Params GetV1ConfigNamespaceNameParams +} + +type GetV1ConfigNamespaceNameResponseObject interface { + VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error +} + +type GetV1ConfigNamespaceName200JSONResponse V1Config + +func (response GetV1ConfigNamespaceName200JSONResponse) VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(200) + + return json.NewEncoder(w).Encode(response) +} + +type GetV1ConfigNamespaceName400JSONResponse V1Error + +func (response GetV1ConfigNamespaceName400JSONResponse) VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(400) + + return json.NewEncoder(w).Encode(response) +} + +type GetV1ConfigNamespaceName500JSONResponse V1Error + +func (response GetV1ConfigNamespaceName500JSONResponse) VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(500) + + return json.NewEncoder(w).Encode(response) +} + +// StrictServerInterface represents all server handlers. +type StrictServerInterface interface { + + // (GET /api/v1/configs) + ListV1Configs(ctx context.Context, request ListV1ConfigsRequestObject) (ListV1ConfigsResponseObject, error) + + // (GET /api/v1/configs/{namespace}) + ListV1ConfigsNamespace(ctx context.Context, request ListV1ConfigsNamespaceRequestObject) (ListV1ConfigsNamespaceResponseObject, error) + + // (GET /api/v1/configs/{namespace}/{name}) + GetV1ConfigNamespaceName(ctx context.Context, request GetV1ConfigNamespaceNameRequestObject) (GetV1ConfigNamespaceNameResponseObject, error) +} + +type StrictHandlerFunc = strictnethttp.StrictHttpHandlerFunc +type StrictMiddlewareFunc = strictnethttp.StrictHttpMiddlewareFunc + +type StrictHTTPServerOptions struct { + RequestErrorHandlerFunc func(w http.ResponseWriter, r *http.Request, err error) + ResponseErrorHandlerFunc func(w http.ResponseWriter, r *http.Request, err error) +} + +func NewStrictHandler(ssi StrictServerInterface, middlewares []StrictMiddlewareFunc) ServerInterface { + return &strictHandler{ssi: ssi, middlewares: middlewares, options: StrictHTTPServerOptions{ + RequestErrorHandlerFunc: func(w http.ResponseWriter, r *http.Request, err error) { + http.Error(w, err.Error(), http.StatusBadRequest) + }, + ResponseErrorHandlerFunc: func(w http.ResponseWriter, r *http.Request, err error) { + http.Error(w, err.Error(), http.StatusInternalServerError) + }, + }} +} + +func NewStrictHandlerWithOptions(ssi StrictServerInterface, middlewares []StrictMiddlewareFunc, options StrictHTTPServerOptions) ServerInterface { + return &strictHandler{ssi: ssi, middlewares: middlewares, options: options} +} + +type strictHandler struct { + ssi StrictServerInterface + middlewares []StrictMiddlewareFunc + options StrictHTTPServerOptions +} + +// ListV1Configs operation middleware +func (sh *strictHandler) ListV1Configs(w http.ResponseWriter, r *http.Request, params ListV1ConfigsParams) { + var request ListV1ConfigsRequestObject + + request.Params = params + + handler := func(ctx context.Context, w http.ResponseWriter, r *http.Request, request interface{}) (interface{}, error) { + return sh.ssi.ListV1Configs(ctx, request.(ListV1ConfigsRequestObject)) + } + for _, middleware := range sh.middlewares { + handler = middleware(handler, "ListV1Configs") + } + + response, err := handler(r.Context(), w, r, request) + + if err != nil { + sh.options.ResponseErrorHandlerFunc(w, r, err) + } else if validResponse, ok := response.(ListV1ConfigsResponseObject); ok { + if err := validResponse.VisitListV1ConfigsResponse(w); err != nil { + sh.options.ResponseErrorHandlerFunc(w, r, err) + } + } else if response != nil { + sh.options.ResponseErrorHandlerFunc(w, r, fmt.Errorf("unexpected response type: %T", response)) + } +} + +// ListV1ConfigsNamespace operation middleware +func (sh *strictHandler) ListV1ConfigsNamespace(w http.ResponseWriter, r *http.Request, namespace string, params ListV1ConfigsNamespaceParams) { + var request ListV1ConfigsNamespaceRequestObject + + request.Namespace = namespace + request.Params = params + + handler := func(ctx context.Context, w http.ResponseWriter, r *http.Request, request interface{}) (interface{}, error) { + return sh.ssi.ListV1ConfigsNamespace(ctx, request.(ListV1ConfigsNamespaceRequestObject)) + } + for _, middleware := range sh.middlewares { + handler = middleware(handler, "ListV1ConfigsNamespace") + } + + response, err := handler(r.Context(), w, r, request) + + if err != nil { + sh.options.ResponseErrorHandlerFunc(w, r, err) + } else if validResponse, ok := response.(ListV1ConfigsNamespaceResponseObject); ok { + if err := validResponse.VisitListV1ConfigsNamespaceResponse(w); err != nil { + sh.options.ResponseErrorHandlerFunc(w, r, err) + } + } else if response != nil { + sh.options.ResponseErrorHandlerFunc(w, r, fmt.Errorf("unexpected response type: %T", response)) + } +} + +// GetV1ConfigNamespaceName operation middleware +func (sh *strictHandler) GetV1ConfigNamespaceName(w http.ResponseWriter, r *http.Request, namespace string, name string, params GetV1ConfigNamespaceNameParams) { + var request GetV1ConfigNamespaceNameRequestObject + + request.Namespace = namespace + request.Name = name + request.Params = params + + handler := func(ctx context.Context, w http.ResponseWriter, r *http.Request, request interface{}) (interface{}, error) { + return sh.ssi.GetV1ConfigNamespaceName(ctx, request.(GetV1ConfigNamespaceNameRequestObject)) + } + for _, middleware := range sh.middlewares { + handler = middleware(handler, "GetV1ConfigNamespaceName") + } + + response, err := handler(r.Context(), w, r, request) + + if err != nil { + sh.options.ResponseErrorHandlerFunc(w, r, err) + } else if validResponse, ok := response.(GetV1ConfigNamespaceNameResponseObject); ok { + if err := validResponse.VisitGetV1ConfigNamespaceNameResponse(w); err != nil { + sh.options.ResponseErrorHandlerFunc(w, r, err) + } + } else if response != nil { + sh.options.ResponseErrorHandlerFunc(w, r, fmt.Errorf("unexpected response type: %T", response)) + } +} + // Base64 encoded, gzipped, json marshaled Swagger object var swaggerSpec = []string{ - "H4sIAAAAAAAC/+xWTY/kNBD9K6WCYybuZoWQgpBYjRBaLRrQTA97QHtwO9WJl8T22pXMjkb576ic9Md2", - "L1rEcEFwaiddqffqvWfZT2h8H7wjxwmrJ0ympV7n5bgur73b2UYeakom2sDWO6xwfg8h+tHWlEDD3eb+", - "xlEEk/8p4S73AZugpp11VIN10DKHVCnVWG6HbWl8r7pv+qASD85RVByJVK+tU+H3RulgkxrXWOCHq8Zf", - "8WMgrHCpHdfl3bxaOJ5UXdk++MhC2+n+o4+wwKC5xQr/jMMp8jRNU4HW7fylBJuWDkPXmnXotCN4+csr", - "oA/BpyxKpFmOIeptR9APHdurED1747vlsbOJSZps7m9vIFEcrSHY+QhvaHu7uYaeaqvBuoaSQJdYIFvu", - "ZKzbH+42GVLKL8mYvTIjxTSzXpercoVTgT6Q08FihS/KVfli0SX7LuOrca3mz/OrhvhSAQFmDzIA+AgP", - "mk27YILfviPDSUzXXQdiQwraUBL6PlDU0uRVjRX+ZBP/ur5ewIRH1D0xxYTVb+eYbzKIjGtaLZIIA24J", - "5rIt1RAp+SEaMcDJEw/RSUkPWjxJHEn34Heg67qAIdSaqVhqez8SOM92Z01mmPlaQX4/UHzEYp+oPC0W", - "y34RcZZ8br3vSDucprcFRkrBu0RZxK9X60sR753tQ0c9OaZ6CdyZAerpIN/0bDMgBTIy3tGTz1hys6/7", - "nDeHQhFXPGk004N+zHY9tPZIyCaI9H6gxFQfBM778qCvO0GVWhupxorjQJ/QPHG0rsFpKv7Py8d5mdfP", - "i42od5Kbvat/OU0/0iFMh4jI4t+YJ+H0T9H57yb7q9VKfox3TC6nUofQLUDqXZIBn04afhlphxV+oY53", - "FbVcVNTxliJb4bzPt/OQ380k/1bTqThT/ufXed9NBcppvc/uEDusEGVW1s2n8vwSXg9bio6Y0nKqn0Zo", - "Pu5LuPaOteEK5N7xvdxNSuuPci/HPE5vpz8CAAD//xcDuTG8CQAA", + "H4sIAAAAAAAC/+xXUW/bNhD+KwS3R0eKmw0D9LSuKIagQzYk7vpQ5IGmzhI7iWSPJ6dBoP8+HElLbuw0", + "BVIUKNo3ijzefXf3fWf6TmrXe2fBUpDVnQy6hV7F5XZZvHB2Yxr+qCFoNJ6Ms7KSaV94dFtTQxBKXK1e", + "X1hAoeNJIa6iH2GCqGFjLNTCWNES+VCVZWOoHdaFdn3Z/db7MtBgLWBJCFD2ytjS/9eUyptQbpdyIT+c", + "NO6Ebj3ISmbb7bK4SquMcc/qxPTeITFsq/qPLsmF9IpaWcmHMOxHHsfFXIW/TKCHKsFnnKsSHa/cRqT9", + "UHBAdB6QDMSqGoI+HPo55212QS3ccyLc+h1oClzB3Sn7nTz9jLCRlfypnFtZ5j6WcxPHhcwlVIjqlr+3", + "gCGGv48mH+TeJVB5D2qRfDNAak3I8BjR1CI0tonVQ3g/GIRaVm+naDvk19OF5EGmar9EdHgI6fk/5wL4", + "6LCk2tVweCH6EXzGNzYOe0WyksbS2bMZq7EEDSDH7iEE1TzoKR8/nmeEc5gcmxm7cYf+Vy1M+qkVKd8p", + "CyIm/MG7EPWFkJQ1oFp3IPqhI3Pi0ZHTrsufTAxgJ6vXlxciAG6NBrFxKN7A+nL1QvRQGyWMbSBw6JiK", + "oY5RXr68WsWQbH4IRu9ENlFGLovT4pTr5jxY5Y2s5FlxWpxlicXOsJLK7bJM1+NWA3S8ueQy71HcKNJt", + "jrnPftV1ghUdvNIQtcU0UOzkvJaVZBn+u8zSizhQ9UCAQVZv78d8E4NwurpVXBJGwERPZmuoBUJwA2pu", + "gOUvGjBKsBeKexIIQfWsA1XXCzH4WhEssm3vtiCsI7MxOiKMeA1Hfj8A3srFbjjFbOUij14uTmbO2rkO", + "lJXjeM0MC97ZkBj/7PQ0Ed8S2FhP5X2XA5XvQtL07PCzJkQccJGmHxfq71fc5V+/bMik8iPRzi0BWtVF", + "/gLuND9G03t8Ku8mNoxP5pYIHjR3a6bYIwy72Nk9RrXJMM1MEI0iuFG3kX03rZkBmSB4lEAgqCe+xF+s", + "iS52L+o8dggHOEKheUT9oP+T6P/LV6L/H6reMaD4FmSX1k9TH5NwT347cXy2KP+ESZOT0njxLcqSMX0p", + "ON/PgPhkGa2rIS50Z8CSwMEGkV4/x8KxufxUpb7KNPoxiY5MonEh02ZS84CdrKTkhpBqjin8uXg1rAEt", + "EIT8+t0XVXoWF/w/i5SmSvD7/Hf+O1gYNzMiP4fleD3+HwAA//9wiPtGLw8AAA==", } // GetSwagger returns the content of the embedded swagger specification file diff --git a/pkg/config/server/cfg.yaml b/pkg/config/server/cfg.yaml index d98f1968..793c0f74 100644 --- a/pkg/config/server/cfg.yaml +++ b/pkg/config/server/cfg.yaml @@ -3,6 +3,7 @@ generate: gorilla-server: true models: true embedded-spec: true + strict-server: true output: api/server.gen.go compatibility: apply-gorilla-middleware-first-to-last: true diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go index 35e18d1d..dff44634 100644 --- a/pkg/config/server/config.go +++ b/pkg/config/server/config.go @@ -5,12 +5,10 @@ import ( "sync" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/config/client/api" ) -type ConfigList struct { - Version string `json:"version"` - Items []*stnrv1.StunnerConfig `json:"items"` -} +type ConfigList = api.V1ConfigList type Config struct { Id string @@ -23,8 +21,10 @@ func (e Config) String() string { // UpsertConfig upserts a single config in the server. func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { - s.configs.Upsert(id, c) - s.configCh <- Config{Id: id, Config: c} + cpy := &stnrv1.StunnerConfig{} + c.DeepCopyInto(cpy) + s.configs.Upsert(id, cpy) + s.configCh <- Config{Id: id, Config: cpy} } // DeleteConfig should remove a config from the client. Theoretically, this would be done by diff --git a/pkg/config/server/conn.go b/pkg/config/server/conn.go index 49feaf8a..ca460880 100644 --- a/pkg/config/server/conn.go +++ b/pkg/config/server/conn.go @@ -5,20 +5,25 @@ import ( "sync" "github.com/gorilla/websocket" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) +type ClientConfigPatcher func(conf *stnrv1.StunnerConfig) (*stnrv1.StunnerConfig, error) + // Conn represents a client WebSocket connection. type Conn struct { *websocket.Conn - Filter FilterConfig + Filter ConfigFilter + patch ClientConfigPatcher readLock, writeLock sync.Mutex // for writemessage } // NewConn wraps a WebSocket connection. -func NewConn(conn *websocket.Conn, filter FilterConfig) *Conn { +func NewConn(conn *websocket.Conn, filter ConfigFilter, patch ClientConfigPatcher) *Conn { return &Conn{ Conn: conn, Filter: filter, + patch: patch, } } diff --git a/pkg/config/server/error.go b/pkg/config/server/error.go deleted file mode 100644 index bf0dae65..00000000 --- a/pkg/config/server/error.go +++ /dev/null @@ -1,34 +0,0 @@ -package server - -import ( - "encoding/json" - "net/http" -) - -// Error defines model for Error. -type Error struct { - // Code Error code - Code int32 `json:"code"` - - // Message Error message - Message string `json:"message"` -} - -func (e Error) Error() string { return e.Message } - -// sendServerError wraps sending of an error in the Error format, and handling the failure to -// marshal that. -func sendServerError(w http.ResponseWriter, message string, code int) { - err := &Error{ - Code: int32(code), - Message: message, - } - sendServerErrorRaw(w, err) -} - -// sendServerError wraps sending of an error in the Error format, and handling the failure to -// marshal that. -func sendServerErrorRaw(w http.ResponseWriter, err *Error) { - w.WriteHeader(int(err.Code)) - _ = json.NewEncoder(w).Encode(err) -} diff --git a/pkg/config/server/handler.go b/pkg/config/server/handler.go index 9ac4829c..a94deb97 100644 --- a/pkg/config/server/handler.go +++ b/pkg/config/server/handler.go @@ -1,9 +1,8 @@ package server import ( - "encoding/json" + "context" "fmt" - "net/http" "strings" "github.com/l7mp/stunner/pkg/config/server/api" @@ -12,112 +11,81 @@ import ( ) // make sure the server satisfies the generate OpenAPI server interface -var _ api.ServerInterface = &Server{} +var _ api.StrictServerInterface = (*Server)(nil) -// ResponseGen is a callback to generate a response to a request (also used for sending the initial config dump for watches). -type ResponseGen func() ([]byte, *Error) +// ConfigFilter is a callback to filter config updates for a client. +type ConfigFilter func(confId string) bool -// FilterConfig is a callback to filter config updates for a client. -type FilterConfig func(confId string) bool +// ConfigPatcher is a callback to patch config updates for a client. +type ConfigPatcher func(conf *stnrv1.StunnerConfig, node string) (*stnrv1.StunnerConfig, error) // (GET /api/v1/configs) -func (s *Server) ListV1Configs(w http.ResponseWriter, r *http.Request, params api.ListV1ConfigsParams) { - endpoint := "/api/v1/configs" - - responder := func() ([]byte, *Error) { - configs := s.configs.Snapshot() - response := ConfigList{Version: "v1", Items: []*stnrv1.StunnerConfig{}} - for _, c := range configs { - response.Items = append(response.Items, c.Config) - } - - json, err := json.Marshal(response) - if err != nil { - return []byte{}, &Error{ - Code: http.StatusInternalServerError, - Message: "Could not JSON marshal config list", - } - } - - return json, nil +func (s *Server) ListV1Configs(ctx context.Context, request api.ListV1ConfigsRequestObject) (api.ListV1ConfigsResponseObject, error) { + s.log.V(1).Info("handling ListV1Configs API call") + + configs := s.configs.Snapshot() + response := ConfigList{Version: "v1", Items: []stnrv1.StunnerConfig{}} + for _, c := range configs { + cpy := stnrv1.StunnerConfig{} + c.Config.DeepCopyInto(&cpy) + response.Items = append(response.Items, cpy) } - filter := func(confId string) bool { - return true - } + s.log.V(3).Info("ListV1Configs API handler: ready", + "configlist-len", len(configs)) - if params.Watch != nil && *params.Watch { - s.handleConn(w, r, endpoint, responder, filter) - } else { - s.handleReq(w, r, endpoint, responder) - } + return api.ListV1Configs200JSONResponse(response), nil } // (GET /api/v1/configs/{namespace}) -func (s *Server) ListV1ConfigsNamespace(w http.ResponseWriter, r *http.Request, namespace string, params api.ListV1ConfigsNamespaceParams) { - endpoint := fmt.Sprintf("/api/v1/configs/%s", namespace) - - responder := func() ([]byte, *Error) { - configs := s.configs.Snapshot() - response := ConfigList{Version: "v1", Items: []*stnrv1.StunnerConfig{}} - for _, c := range configs { - ps := strings.Split(c.Id, "/") - if len(ps) == 2 && ps[0] == namespace { - response.Items = append(response.Items, c.Config) - } +func (s *Server) ListV1ConfigsNamespace(ctx context.Context, request api.ListV1ConfigsNamespaceRequestObject) (api.ListV1ConfigsNamespaceResponseObject, error) { + s.log.V(1).Info("handling ListV1ConfigsNamespace API call", + "namespace", request.Namespace) + + configs := s.configs.Snapshot() + response := ConfigList{Version: "v1", Items: []stnrv1.StunnerConfig{}} + for _, c := range configs { + ps := strings.Split(c.Id, "/") + if len(ps) == 2 && ps[0] == request.Namespace { + cpy := stnrv1.StunnerConfig{} + c.Config.DeepCopyInto(&cpy) + response.Items = append(response.Items, cpy) } - - json, err := json.Marshal(response) - if err != nil { - return []byte{}, &Error{ - Code: http.StatusInternalServerError, - Message: "Could not JSON marshal config list", - } - } - - return json, nil - } - filter := func(confId string) bool { - ps := strings.Split(confId, "/") - return len(ps) == 2 && ps[0] == namespace } - if params.Watch != nil && *params.Watch { - s.handleConn(w, r, endpoint, responder, filter) - } else { - s.handleReq(w, r, endpoint, responder) - } + s.log.V(3).Info("ListV1ConfigsNamespace API handler: ready", + "configlist-len", len(configs)) + + return api.ListV1ConfigsNamespace200JSONResponse(response), nil } // (GET /api/v1/configs/{namespace}/{name}) -func (s *Server) GetV1ConfigNamespaceName(w http.ResponseWriter, r *http.Request, namespace string, name string, params api.GetV1ConfigNamespaceNameParams) { +func (s *Server) GetV1ConfigNamespaceName(ctx context.Context, request api.GetV1ConfigNamespaceNameRequestObject) (api.GetV1ConfigNamespaceNameResponseObject, error) { + namespace, name := request.Namespace, request.Name + s.log.V(1).Info("handling GetV1ConfigNamespaceName API call", "namespace", namespace, + "name", name) + id := fmt.Sprintf("%s/%s", namespace, name) - endpoint := fmt.Sprintf("/api/v1/configs/%s/%s", namespace, name) - - responder := func() ([]byte, *Error) { - c := s.configs.Get(id) - if c == nil { - return []byte{}, &Error{ - Code: http.StatusBadRequest, - Message: "Config not found", - } - } + c := s.configs.Get(id) + if c == nil { + s.log.V(1).Info("GetV1ConfigNamespaceName: config not found", "client", id) + return nil, fmt.Errorf("config not found for id %q", id) + } + + ret := &stnrv1.StunnerConfig{} + c.DeepCopyInto(ret) - json, err := json.Marshal(c) + if s.patch != nil && request.Params.Node != nil { + conf, err := s.patch(ret, *request.Params.Node) if err != nil { - return []byte{}, &Error{ - Code: http.StatusInternalServerError, - Message: "Config not found", - } + s.log.Error(err, "GetV1ConfigNamespaceName: patch config failed") + return nil, err } - - return json, nil + ret = conf } - filter := func(confId string) bool { return confId == id } - if params.Watch != nil && *params.Watch { - s.handleConn(w, r, endpoint, responder, filter) - } else { - s.handleReq(w, r, endpoint, responder) - } + s.log.V(3).Info("GetV1ConfigNamespaceName API handler: ready", + "config", ret.String()) + + return api.GetV1ConfigNamespaceName200JSONResponse(*ret), nil } diff --git a/pkg/config/server/middleware.go b/pkg/config/server/middleware.go new file mode 100644 index 00000000..4d727c7e --- /dev/null +++ b/pkg/config/server/middleware.go @@ -0,0 +1,94 @@ +package server + +import ( + "context" + "fmt" + "net/http" + "strings" + + "github.com/gorilla/websocket" + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/config/server/api" +) + +func (s *Server) WSUpgradeMiddleware(next api.StrictHandlerFunc, operationID string) api.StrictHandlerFunc { + return func(ctx context.Context, w http.ResponseWriter, r *http.Request, request interface{}) (interface{}, error) { + var filter ConfigFilter + var patcher ClientConfigPatcher + watch := false + + switch operationID { + case "GetV1ConfigNamespaceName": + param, ok := request.(api.GetV1ConfigNamespaceNameRequestObject) + if !ok { + return nil, fmt.Errorf("unexpected parameters in API operation %q", + operationID) + } + + filter = func(confID string) bool { + id := fmt.Sprintf("%s/%s", param.Namespace, param.Name) + return confID == id + } + + watch = param.Params.Watch != nil && *param.Params.Watch + + if s.patch != nil && param.Params.Node != nil { + patcher = func(conf *stnrv1.StunnerConfig) (*stnrv1.StunnerConfig, error) { + return s.patch(conf, *param.Params.Node) + } + } + + case "ListV1ConfigsNamespace": + param, ok := request.(api.ListV1ConfigsNamespaceRequestObject) + if !ok { + return nil, fmt.Errorf("unexpected parameters in API operation %q", + operationID) + } + + filter = func(confID string) bool { + ps := strings.Split(confID, "/") + return len(ps) == 2 && ps[0] == param.Namespace + } + + watch = param.Params.Watch != nil && *param.Params.Watch + + case "ListV1Configs": + param, ok := request.(api.ListV1ConfigsRequestObject) + if !ok { + return nil, fmt.Errorf("unexpected parameters in API operation %q", + operationID) + } + + filter = func(confID string) bool { + return true + } + + watch = param.Params.Watch != nil && *param.Params.Watch + + default: + return nil, fmt.Errorf("invalid API operation %q", + operationID) + } + + if !watch { + return next(ctx, w, r, request) + } + + s.log.V(4).Info("WS upgrade middleware: upgrading connection", "client", r.RemoteAddr) + + // upgrade to webSocket + upgrader := websocket.Upgrader{ + ReadBufferSize: 1024, + WriteBufferSize: 1024, + } + + conn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + return nil, err + } + + s.handleConn(ctx, conn, operationID, filter, patcher) + + return nil, nil + } +} diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index 16221cba..ab1a4193 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -5,7 +5,6 @@ import ( "context" "encoding/json" "errors" - "fmt" "net" "net/http" @@ -20,37 +19,40 @@ import ( // Server is a generic config discovery server implementation. type Server struct { *http.Server + router *mux.Router addr string conns *ConnTrack configs *ConfigStore configCh chan Config + patch ConfigPatcher log logr.Logger } // New creates a new config discovery server instance for the specified address. -func New(addr string, logger logr.Logger) *Server { +func New(addr string, patch ConfigPatcher, logger logr.Logger) *Server { if addr == "" { addr = stnrv1.DefaultConfigDiscoveryAddress } - cds := &Server{ + return &Server{ + router: mux.NewRouter(), conns: NewConnTrack(), configs: NewConfigStore(), configCh: make(chan Config, 8), addr: addr, + patch: patch, log: logger, } - return cds } // Start let the config discovery server listen to new client connections. func (s *Server) Start(ctx context.Context) error { - r := mux.NewRouter() - api.HandlerFromMux(s, r) - s.Server = &http.Server{Addr: s.addr, Handler: r} + handler := api.NewStrictHandler(s, []api.StrictMiddlewareFunc{s.WSUpgradeMiddleware}) + api.HandlerFromMux(handler, s.router) + s.Server = &http.Server{Addr: s.addr, Handler: s.router} go func() { - s.log.Info("Starting CDS server", "address", s.addr) + s.log.Info("starting CDS server", "address", s.addr, "patch", s.patch != nil) err := s.ListenAndServe() if err != nil { @@ -114,41 +116,8 @@ func (s *Server) RemoveClient(id string) { } } -func (s *Server) handleReq(w http.ResponseWriter, r *http.Request, endpoint string, responder ResponseGen) { - s.log.V(1).Info("received new request", "api", endpoint, "client", r.RemoteAddr) - - response, err := responder() - if err != nil { - s.log.Info("client config not found", "api", endpoint, "client", r.RemoteAddr, - "code", err.Code, "message", err.Message) - sendServerErrorRaw(w, err) - } - - s.log.V(2).Info("sending response to client", "api", endpoint, "client", r.RemoteAddr, - "response", string(response)) - - if _, err := w.Write(response); err != nil { - s.log.Error(err, "could not write config", "api", endpoint, "client", r.RemoteAddr) - http.Error(w, "Could not write config", http.StatusInternalServerError) - return - } -} - -func (s *Server) handleConn(w http.ResponseWriter, r *http.Request, endpoint string, responder ResponseGen, filter FilterConfig) { - // upgrade to webSocket - upgrader := websocket.Upgrader{ - ReadBufferSize: 1024, - WriteBufferSize: 1024, - } - - wsConn, err := upgrader.Upgrade(w, r, nil) - if err != nil { - msg := fmt.Sprintf("could not upgrade HTTP connection for client %s: %s", - r.RemoteAddr, err.Error()) - sendServerError(w, msg, http.StatusInternalServerError) - return - } - conn := NewConn(wsConn, filter) +func (s *Server) handleConn(ctx context.Context, wsConn *websocket.Conn, operationID string, filter ConfigFilter, patch ClientConfigPatcher) { + conn := NewConn(wsConn, filter, patch) s.conns.Upsert(conn) // a dummy reader that drops everything it receives: this must be there for the @@ -167,44 +136,49 @@ func (s *Server) handleConn(w http.ResponseWriter, r *http.Request, endpoint str return conn.WriteMessage(websocket.PongMessage, []byte("keepalive")) }) - s.log.V(1).Info("new config stream connection", "api", endpoint, "client", conn.Id()) + s.log.V(1).Info("new config stream connection", "api", operationID, "client", conn.Id()) // send initial config(list) for _, conf := range s.configs.Snapshot() { if filter(conf.Id) { - s.sendConfig(conn, conf) + s.sendConfig(conn, conf.Config) } } // wait until client closes the connection or the server is cancelled (which will kill all // the running connections) - <-r.Context().Done() + <-ctx.Done() - s.log.V(1).Info("client connection closed", "api", endpoint, - "client", r.RemoteAddr) + s.log.V(1).Info("client connection closed", "api", operationID, "client", conn.Id()) conn.Close() } // iterate through all connections and send response if needed func (s *Server) broadcastConfig(e Config) { - json, err := json.Marshal(e.Config) - if err != nil { - s.log.Error(err, "error JSON marshaling config", "event", e.String()) - return - } - for _, conn := range s.conns.Snapshot() { if conn.Filter(e.Id) { - s.sendJSONConfig(conn, json) + s.sendConfig(conn, e.Config) } } } -func (s *Server) sendConfig(conn *Conn, e Config) { - json, err := json.Marshal(e.Config) +func (s *Server) sendConfig(conn *Conn, e *stnrv1.StunnerConfig) { + c := &stnrv1.StunnerConfig{} + e.DeepCopyInto(c) + + if conn.patch != nil { + newC, err := conn.patch(c) + if err != nil { + s.log.Error(err, "cannot patch config", "event", e.String()) + return + } + c = newC + } + + json, err := json.Marshal(c) if err != nil { - s.log.Error(err, "error JSON marshaling config", "event", e.String()) + s.log.Error(err, "cannor JSON serialize config", "event", e.String()) return } diff --git a/reconcile.go b/reconcile.go index cce6d19c..2f33c3d9 100644 --- a/reconcile.go +++ b/reconcile.go @@ -15,11 +15,11 @@ import ( // least one internal object (usually, a listener) for the new config (unless DryRun is enabled), // and an error if an error has occurred during reconciliation, in which case it will rollback the // last working configuration (unless SuppressRollback is on). -func (s *Stunner) Reconcile(req stnrv1.StunnerConfig) error { +func (s *Stunner) Reconcile(req *stnrv1.StunnerConfig) error { return s.reconcileWithRollback(req, false) } -func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback bool) error { +func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bool) error { var errFinal error new, deleted, changed := 0, 0, 0 @@ -33,7 +33,7 @@ func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback boo toBeStarted, toBeRestarted := []object.Object{}, []object.Object{} // admin - adminState, err := s.adminManager.PrepareReconciliation([]stnrv1.Config{&req.Admin}, &req) + adminState, err := s.adminManager.PrepareReconciliation([]stnrv1.Config{&req.Admin}, req) if err != nil { return fmt.Errorf("error preparing reconciliation for admin config: %s", err.Error()) @@ -44,7 +44,7 @@ func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback boo deleted += len(adminState.DeletedJobQueue) // auth - authState, err := s.authManager.PrepareReconciliation([]stnrv1.Config{&req.Auth}, &req) + authState, err := s.authManager.PrepareReconciliation([]stnrv1.Config{&req.Auth}, req) if err != nil { return fmt.Errorf("error preparing reconciliation for auth config: %s", err.Error()) @@ -59,7 +59,7 @@ func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback boo for i := range req.Listeners { lconf[i] = &(req.Listeners[i]) } - listenerState, err := s.listenerManager.PrepareReconciliation(lconf, &req) + listenerState, err := s.listenerManager.PrepareReconciliation(lconf, req) if err != nil { return fmt.Errorf("error preparing reconciliation for listener config: %s", err.Error()) } @@ -73,7 +73,7 @@ func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback boo for i := range req.Clusters { cconf[i] = &(req.Clusters[i]) } - clusterState, err := s.clusterManager.PrepareReconciliation(cconf, &req) + clusterState, err := s.clusterManager.PrepareReconciliation(cconf, req) if err != nil { return fmt.Errorf("error preparing reconciliation for cluster config: %s", err.Error()) } @@ -193,7 +193,7 @@ func (s *Stunner) reconcileWithRollback(req stnrv1.StunnerConfig, inRollback boo rollback: if !s.suppressRollback { s.log.Infof("rolling back to previous configuration: %s", rollback.String()) - return s.reconcileWithRollback(*rollback, true) + return s.reconcileWithRollback(rollback, true) } return errFinal diff --git a/reconcile_test.go b/reconcile_test.go index 25e8f1df..d7579a18 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -1741,7 +1741,7 @@ func TestStunnerReconcile(t *testing.T) { }) log.Debug("starting stunnerd") - assert.NoError(t, s.Reconcile(*conf), "starting server") + assert.NoError(t, s.Reconcile(conf), "starting server") runningConf := s.GetConfig() assert.NotNil(t, runningConf, "default stunner get config ok") @@ -1764,7 +1764,7 @@ func TestStunnerReconcile(t *testing.T) { assert.True(t, conf.DeepEqual(runningConf), "default stunner config ok") - err = s.Reconcile(c.config) + err = s.Reconcile(&c.config) c.tester(t, s, err) s.Close() @@ -1827,14 +1827,14 @@ func testStunnerReconcileWithVNet(t *testing.T, testcases []StunnerTestReconcile }) log.Debug("starting stunnerd") - assert.NoError(t, s.Reconcile(*conf), "starting server") + assert.NoError(t, s.Reconcile(conf), "starting server") for _, c := range testcases { t.Run(c.testName, func(t *testing.T) { log.Debugf("-------------- Running test: %s -------------", c.testName) log.Debug("reconciling server") - err := s.Reconcile(c.config) + err := s.Reconcile(&c.config) if c.restart { assert.ErrorContains(t, err, "restart", "starting server") } else { diff --git a/server_unix_test.go b/server_unix_test.go index a2b16c96..1075ea44 100644 --- a/server_unix_test.go +++ b/server_unix_test.go @@ -69,7 +69,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { defer stunner.Close() log.Debug("starting stunnerd") - err := stunner.Reconcile(stnrv1.StunnerConfig{ + err := stunner.Reconcile(&stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: stunnerTestLoglevel, diff --git a/stunner.go b/stunner.go index 728261f3..f07da99f 100644 --- a/stunner.go +++ b/stunner.go @@ -25,8 +25,7 @@ var DefaultInstanceId = fmt.Sprintf("default/stunnerd-%s", uuid.New().String()) // Stunner is an instance of the STUNner deamon. type Stunner struct { - id string - version string + name, version string adminManager, authManager, listenerManager, clusterManager manager.Manager suppressRollback, dryRun bool resolver resolver.DnsResolver @@ -73,7 +72,7 @@ func NewStunner(options Options) *Stunner { udpThreadNum = options.UDPListenerThreadNum } - id := options.Id + id := options.Name if id == "" { if h, err := os.Hostname(); err != nil { id = DefaultInstanceId @@ -83,7 +82,7 @@ func NewStunner(options Options) *Stunner { } s := &Stunner{ - id: id, + name: id, version: stnrv1.ApiVersion, logger: logger, log: log, @@ -117,7 +116,7 @@ func NewStunner(options Options) *Stunner { // GetId returns the id of the current stunnerd instance. func (s *Stunner) GetId() string { - return s.id + return s.name } // GetVersion returns the STUNner API version. diff --git a/stunner_test.go b/stunner_test.go index 09d4077e..6c949259 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -37,6 +37,7 @@ var stunnerTestLoglevel string = "all:ERROR" // var stunnerTestLoglevel string = stnrv1.DefaultLogLevel // var stunnerTestLoglevel string = "all:INFO" // var stunnerTestLoglevel string = "all:TRACE" + // var stunnerTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" var certPem, keyPem, _ = GenerateSelfSignedKey() @@ -557,7 +558,7 @@ func testStunnerLocalhost(t *testing.T, udpThreadNum int, tests []TestStunnerCon // assert.False(t, stunner.IsReady(), "lifecycle 1: not-ready") log.Debug("starting stunnerd") - assert.NoError(t, stunner.Reconcile(c), "starting server") + assert.NoError(t, stunner.Reconcile(&c), "starting server") assert.False(t, stunner.shutdown, "lifecycle 2: alive") assert.True(t, stunner.ready, "lifecycle 2: ready") @@ -1130,7 +1131,7 @@ func TestStunnerClusterWithVNet(t *testing.T) { }) log.Debug("starting stunnerd") - assert.NoError(t, stunner.Reconcile(c.config), "starting server") + assert.NoError(t, stunner.Reconcile(&c.config), "starting server") var u, p string auth := c.config.Auth.Type @@ -1595,7 +1596,7 @@ func TestStunnerPortRangeWithVNet(t *testing.T) { }) log.Debug("starting stunnerd") - assert.NoError(t, stunner.Reconcile(c.config), "starting server") + assert.NoError(t, stunner.Reconcile(&c.config), "starting server") var u, p string auth := c.config.Auth.Type @@ -1875,7 +1876,7 @@ func TestStunnerLifecycle(t *testing.T) { } log.Debug("reconciling empty server") - err = s.Reconcile(conf) + err = s.Reconcile(&conf) assert.NoError(t, err, "reconcile empty server") status, err := doLivenessCheck("http://127.0.0.1:8086") @@ -1892,7 +1893,7 @@ func TestStunnerLifecycle(t *testing.T) { log.Debug("reconciling server") conf.Admin.HealthCheckEndpoint = c.hcEndpoint - err := s.Reconcile(conf) + err := s.Reconcile(&conf) assert.NoError(t, err, "cannot reconcile") // obtain hc address @@ -1926,7 +1927,7 @@ func TestStunnerLifecycle(t *testing.T) { // make sure health-check is running h := "0.0.0.0" conf.Admin.HealthCheckEndpoint = &h - assert.NoError(t, s.Reconcile(conf), "cannot reconcile") + assert.NoError(t, s.Reconcile(&conf), "cannot reconcile") status, err = doLivenessCheck("http://127.0.0.1:8086") assert.NoError(t, err, "liveness test before graceful-shutdown: running") @@ -2046,7 +2047,7 @@ func TestStunnerMetrics(t *testing.T) { } log.Debug("reconciling empty server") - err := s.Reconcile(conf) + err := s.Reconcile(&conf) assert.NoError(t, err, "reconcile empty server") assert.True(t, s.IsReady(), "server ready") @@ -2057,7 +2058,7 @@ func TestStunnerMetrics(t *testing.T) { log.Debug("reconciling server") conf.Admin.MetricsEndpoint = c.mcEndpoint - err := s.Reconcile(conf) + err := s.Reconcile(&conf) assert.NoError(t, err, "cannot reconcile") // obtain metric address @@ -2227,7 +2228,7 @@ func TestStunnerConfigV1Alpha1(t *testing.T) { assert.True(t, len(config.Clusters) > 0, "clusters len") log.Debug("starting stunnerd") - assert.NoError(t, stunner.Reconcile(*config), "starting server") + assert.NoError(t, stunner.Reconcile(config), "starting server") var u, p string auth := config.Auth.Type diff --git a/turncat_test.go b/turncat_test.go index ec82d8f4..dcaf07a4 100644 --- a/turncat_test.go +++ b/turncat_test.go @@ -111,7 +111,7 @@ func TestTurncatPlaintext(t *testing.T) { SuppressRollback: true, }) - err := stunner.Reconcile(stnrv1.StunnerConfig{ + err := stunner.Reconcile(&stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: turncatTestLoglevel, @@ -227,7 +227,7 @@ func TestTurncatLongterm(t *testing.T) { LogLevel: turncatTestLoglevel, SuppressRollback: true, }) - err := stunner.Reconcile(stnrv1.StunnerConfig{ + err := stunner.Reconcile(&stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ LogLevel: turncatTestLoglevel, From 55ea83e8bf93331128d0620f5c285c10da5631e5 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 19 Jan 2024 12:40:12 +0100 Subject: [PATCH 107/243] doc: Add docs for CDS client --- cmd/stunnerd/main.go | 2 +- pkg/config/client/cds_client.go | 4 ++++ pkg/config/client/client.go | 1 + pkg/config/client/file_client.go | 8 ++++++-- 4 files changed, 12 insertions(+), 3 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index c2f46a9d..88d692f5 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -149,7 +149,7 @@ func main() { }() case c := <-conf: - log.Trace("new configuration available") + log.Infof("new configuration available: %q", c.String()) // command line loglevel overrides config if *verbose || *level != "" { diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go index 79f7bbf3..647374b0 100644 --- a/pkg/config/client/cds_client.go +++ b/pkg/config/client/cds_client.go @@ -20,6 +20,8 @@ type CDSClient struct { addr, id string } +// NewCDSClient creates a config discovery service client that can be used to load or watch STUNner +// configurations from a CDS remote server. func NewCDSClient(addr, id string, logger logging.LeveledLogger) (Client, error) { ps := strings.Split(id, "/") if len(ps) != 2 { @@ -34,10 +36,12 @@ func NewCDSClient(addr, id string, logger logging.LeveledLogger) (Client, error) return &CDSClient{CdsApi: client, addr: addr, id: id}, nil } +// String outputs the status of the client. func (p *CDSClient) String() string { return fmt.Sprintf("config discovery client %q: using server %s", p.id, p.addr) } +// Load grabs a new configuration from the config doscovery server. func (p *CDSClient) Load() (*stnrv1.StunnerConfig, error) { configs, err := p.CdsApi.Get(context.Background()) if err != nil { diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index e2049047..2579494b 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -42,6 +42,7 @@ type Client interface { fmt.Stringer } +// New creates a generic config client. func New(origin string, id string, logger logging.LoggerFactory) (Client, error) { u, err := getURI(origin) if err != nil { diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index ff9dc4a8..b16172b3 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -22,6 +22,8 @@ type ConfigFileClient struct { log logging.LeveledLogger } +// NewConfigFileClient creates a client that load or watch STUNner configurations from a local +// file. func NewConfigFileClient(origin, id string, logger logging.LeveledLogger) (Client, error) { return &ConfigFileClient{ configFile: origin, @@ -31,10 +33,12 @@ func NewConfigFileClient(origin, id string, logger logging.LeveledLogger) (Clien } +// String outputs the status of the client. func (w *ConfigFileClient) String() string { return fmt.Sprintf("config client using file %q", w.configFile) } +// Load grabs a new configuration from a config file. func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { b, err := os.ReadFile(w.configFile) if err != nil { @@ -57,8 +61,8 @@ func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { return c, nil } -// WatchConfig watches a configuration file for changes. If no file exists at the given path, -// WatchConfig will periodically retry until the file appears. +// Watch watches a configuration file for changes. If no file exists at the given path, it will +// periodically retry until the file appears. func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { if w.configFile == "" { return errors.New("uninitialized config file path") From e1d3b8c30a17289ab12f8fd8398a1c8f95ba202c Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 19 Jan 2024 13:40:32 +0100 Subject: [PATCH 108/243] chore: Update module dependencies --- go.mod | 54 +++++++++++++++------------- go.sum | 111 ++++++++++++++++++++++++++++++--------------------------- 2 files changed, 88 insertions(+), 77 deletions(-) diff --git a/go.mod b/go.mod index aaa6c927..4248fb69 100644 --- a/go.mod +++ b/go.mod @@ -4,16 +4,16 @@ go 1.19 require ( github.com/deepmap/oapi-codegen/v2 v2.0.0 - github.com/fsnotify/fsnotify v1.6.0 - github.com/getkin/kin-openapi v0.118.0 + github.com/fsnotify/fsnotify v1.7.0 + github.com/getkin/kin-openapi v0.122.0 github.com/go-logr/logr v1.3.0 github.com/go-logr/zapr v1.3.0 - github.com/google/uuid v1.4.0 + github.com/google/uuid v1.5.0 github.com/gorilla/mux v1.8.0 github.com/gorilla/websocket v1.5.1 github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb - github.com/oapi-codegen/runtime v1.1.0 - github.com/pion/dtls/v2 v2.2.7 + github.com/oapi-codegen/runtime v1.1.1 + github.com/pion/dtls/v2 v2.2.9 github.com/pion/logging v0.2.2 github.com/pion/transport/v3 v3.0.1 github.com/pion/turn/v3 v3.0.1 @@ -21,13 +21,13 @@ require ( github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 go.uber.org/zap v1.26.0 - golang.org/x/sys v0.15.0 - golang.org/x/time v0.4.0 + golang.org/x/sys v0.16.0 + golang.org/x/time v0.5.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 - k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 + k8s.io/utils v0.0.0-20240102154912-e7106e64919e sigs.k8s.io/controller-runtime v0.12.3 - sigs.k8s.io/yaml v1.3.0 + sigs.k8s.io/yaml v1.4.0 ) require ( @@ -39,48 +39,52 @@ require ( github.com/davecgh/go-spew v1.1.1 // indirect github.com/emicklei/go-restful v2.9.5+incompatible // indirect github.com/evanphx/json-patch v4.12.0+incompatible // indirect - github.com/go-openapi/jsonpointer v0.19.5 // indirect + github.com/go-openapi/jsonpointer v0.20.2 // indirect github.com/go-openapi/jsonreference v0.19.5 // indirect - github.com/go-openapi/swag v0.19.14 // indirect + github.com/go-openapi/swag v0.22.7 // indirect + github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/golang/protobuf v1.5.3 // indirect github.com/google/gnostic v0.5.7-v3refs // indirect + github.com/google/go-cmp v0.6.0 // indirect github.com/google/gofuzz v1.1.0 // indirect + github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect github.com/imdario/mergo v0.3.12 // indirect - github.com/invopop/yaml v0.1.0 // indirect + github.com/invopop/yaml v0.2.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/mailru/easyjson v0.7.7 // indirect - github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/perimeterx/marshmallow v1.1.4 // indirect + github.com/onsi/ginkgo/v2 v2.15.0 // indirect + github.com/onsi/gomega v1.30.0 // indirect + github.com/perimeterx/marshmallow v1.1.5 // indirect github.com/pion/randutil v0.1.0 // indirect github.com/pion/stun/v2 v2.0.0 // indirect - github.com/pion/transport/v2 v2.2.1 // indirect + github.com/pion/transport/v2 v2.2.4 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.5.0 // indirect - github.com/prometheus/common v0.45.0 // indirect + github.com/prometheus/common v0.46.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.14.0 // indirect - golang.org/x/mod v0.12.0 // indirect - golang.org/x/net v0.17.0 // indirect - golang.org/x/oauth2 v0.12.0 // indirect - golang.org/x/term v0.13.0 // indirect - golang.org/x/text v0.13.0 // indirect - golang.org/x/tools v0.12.0 // indirect + golang.org/x/crypto v0.18.0 // indirect + golang.org/x/mod v0.14.0 // indirect + golang.org/x/net v0.20.0 // indirect + golang.org/x/oauth2 v0.16.0 // indirect + golang.org/x/term v0.16.0 // indirect + golang.org/x/text v0.14.0 // indirect + golang.org/x/tools v0.17.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/protobuf v1.31.0 // indirect + google.golang.org/protobuf v1.32.0 // indirect gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect k8s.io/client-go v0.24.2 // indirect - k8s.io/klog/v2 v2.60.1 // indirect + k8s.io/klog/v2 v2.80.1 // indirect k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 // indirect sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect diff --git a/go.sum b/go.sum index 6fd198bb..278652a1 100644 --- a/go.sum +++ b/go.sum @@ -91,11 +91,11 @@ github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoD github.com/form3tech-oss/jwt-go v3.2.3+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= -github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY= -github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw= +github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= +github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg= -github.com/getkin/kin-openapi v0.118.0 h1:z43njxPmJ7TaPpMSCQb7PN0dEYno4tyBPQcrFdHoLuM= -github.com/getkin/kin-openapi v0.118.0/go.mod h1:l5e9PaFUo9fyLJCPGQeXI2ML8c3P8BHOEV2VaAVf/pc= +github.com/getkin/kin-openapi v0.122.0 h1:WB9Jbl0Hp/T79/JF9xlSW5Kl9uYdk/AWD0yAd9HOM10= +github.com/getkin/kin-openapi v0.122.0/go.mod h1:PCWw/lfBrJY4HcdqE3jj+QFkaFK8ABoqo7PvqVhXXqw= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= @@ -108,16 +108,19 @@ github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ4 github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q= +github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs= github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8= github.com/go-openapi/jsonreference v0.19.5 h1:1WJP/wi4OjB4iV8KVbH73rQaoialJrqv8gitZLxGLtM= github.com/go-openapi/jsonreference v0.19.5/go.mod h1:RdybgQwPxbL4UEjuAruzK1x3nE69AqPYEJeo/TWfEeg= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= -github.com/go-openapi/swag v0.19.14 h1:gm3vOOXfiuw5i9p5N9xJvfjvuofpyvLA9Wr6QfK5Fng= github.com/go-openapi/swag v0.19.14/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= +github.com/go-openapi/swag v0.22.7 h1:JWrc1uc/P9cSomxfnsFSVWoE1FW6bNbrVPmpQYpCcR8= +github.com/go-openapi/swag v0.22.7/go.mod h1:Gl91UqO+btAM0plGGxHqJcQZ1ZTy6jbmridBTsDy8A0= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= -github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= @@ -168,7 +171,9 @@ github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/ github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= +github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -186,10 +191,12 @@ github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 h1:yAJXTCF9TqKcTiHJAE8dj7HMvPfh66eeA2JYW7eFpSE= +github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4= -github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU= +github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI= @@ -208,8 +215,8 @@ github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1: github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= -github.com/invopop/yaml v0.1.0 h1:YW3WGUoJEXYfzWBjn00zIlrw7brGVD0fUKRYDPAPhrc= -github.com/invopop/yaml v0.1.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q= +github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY= +github.com/invopop/yaml v0.2.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= @@ -232,8 +239,6 @@ github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= -github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg= -github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -251,30 +256,35 @@ github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= -github.com/oapi-codegen/runtime v1.1.0 h1:rJpoNUawn5XTvekgfkvSZr0RqEnoYpFkyvrzfWeFKWM= -github.com/oapi-codegen/runtime v1.1.0/go.mod h1:BeSfBkWWWnAnGdyS+S/GnlbmHKzf8/hwkvelJZDeKA8= +github.com/oapi-codegen/runtime v1.1.1 h1:EXLHh0DXIJnWhdRPN2w4MXAzFyE4CskzhNLUmtpMYro= +github.com/oapi-codegen/runtime v1.1.1/go.mod h1:SK9X900oXmPWilYR5/WKPzt3Kqxn/uS/+lbpREv+eCg= github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= +github.com/onsi/ginkgo/v2 v2.15.0 h1:79HwNRBAZHOEwrczrgSOPy+eFTTlIGELKy5as+ClttY= +github.com/onsi/ginkgo/v2 v2.15.0/go.mod h1:HlxMHtYF57y6Dpf+mc5529KKmSq9h2FpCF+/ZkwUxKM= github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= -github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE= -github.com/perimeterx/marshmallow v1.1.4 h1:pZLDH9RjlLGGorbXhcaQLhfuV0pFMNfPO55FuFkxqLw= -github.com/perimeterx/marshmallow v1.1.4/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= +github.com/onsi/gomega v1.30.0 h1:hvMK7xYz4D3HapigLTeGdId/NcfQx1VHMJc60ew99+8= +github.com/onsi/gomega v1.30.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= +github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s= +github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= -github.com/pion/dtls/v2 v2.2.7 h1:cSUBsETxepsCSFSxC3mc/aDo14qQLMSL+O6IjG28yV8= github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= +github.com/pion/dtls/v2 v2.2.9 h1:K+D/aVf9/REahQvqk6G5JavdrD8W1PWDKC11UlwN7ts= +github.com/pion/dtls/v2 v2.2.9/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8= github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= -github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= +github.com/pion/transport/v2 v2.2.4 h1:41JJK6DZQYSeVLxILA2+F4ZkKb4Xd/tFJZRFZQ9QAlo= +github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= github.com/pion/turn/v3 v3.0.1 h1:wLi7BTQr6/Q20R0vt/lHbjv6y4GChFtC33nkYbasoT8= @@ -288,8 +298,8 @@ github.com/prometheus/client_golang v1.18.0/go.mod h1:T+GXkCk5wSJyOqMIzVgvvjFDlk github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= -github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM= -github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY= +github.com/prometheus/common v0.46.0 h1:doXzt5ybi1HBKpsZOL0sSkaNHJJqkyfEWZGGqqScV0Y= +github.com/prometheus/common v0.46.0/go.mod h1:Tp0qkxpb9Jsg54QMe+EAmqXkSV7Evdy1BTn+g2pa/hQ= github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= @@ -309,13 +319,9 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/ugorji/go v1.2.7 h1:qYhyWUUd6WbiM+C6JZAUkIJt/1WrjzNHY9+KCIjVqTo= -github.com/ugorji/go v1.2.7/go.mod h1:nF9osbDWLy6bDVv/Rtoh6QgnvNDpmCalQV5urGCCS6M= -github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY= github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -345,8 +351,8 @@ golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5y golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc= -golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4= +golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= +golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -384,8 +390,8 @@ golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.12.0 h1:rmsUpXtvNzj340zd98LZ4KntptpfRHwpFOHG188oHXc= -golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= +golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -430,8 +436,8 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM= -golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= +golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo= +golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -444,8 +450,8 @@ golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4= -golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4= +golang.org/x/oauth2 v0.16.0 h1:aDkGMBSYxElaoP81NpoUoz2oo2R2wHdZpGToUxfyQrQ= +golang.org/x/oauth2 v0.16.0/go.mod h1:hqZ+0LWXsiVoZpeld6jVt06P3adbS2Uu911W1SsJv2o= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -507,21 +513,20 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc= -golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= +golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= -golang.org/x/term v0.13.0 h1:bb+I9cTfFazGW51MZqBVmZy7+JEJMouUHTUSKVQLBek= -golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U= +golang.org/x/term v0.16.0 h1:m+B6fahuftsE9qjo0VWp2FW0mB3MTJvR0BaMQrq0pmE= +golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -534,14 +539,14 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= -golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= -golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20220210224613-90d013bbcef8/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.4.0 h1:Z81tqI5ddIoXDPvVQ7/7CC9TnLM7ubaFG2qXYd5BbYY= -golang.org/x/time v0.4.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= +golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= +golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= @@ -594,8 +599,8 @@ golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.12.0 h1:YW6HUoUmYBpwSgyaGaZq1fHjrBjX1rlpZ54T6mu2kss= -golang.org/x/tools v0.12.0/go.mod h1:Sc0INKfu04TlqNoRA1hgpFZbhYXHPr4V5DzpSBTPqQM= +golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc= +golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -701,8 +706,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= -google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I= +google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 h1:FVCohIoYO7IJoDDVpV2pdq7SgrMH6wHnuTyrdrxJNoY= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0/go.mod h1:OdE7CF6DbADk7lN8LIKRzRJTTZXIjtWgA5THM5lhBAw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -748,13 +753,15 @@ k8s.io/client-go v0.24.2/go.mod h1:zg4Xaoo+umDsfCWr4fCnmLEtQXyCNXCvJuSsglNcV30= k8s.io/gengo v0.0.0-20210813121822-485abfe95c7c/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E= k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= -k8s.io/klog/v2 v2.60.1 h1:VW25q3bZx9uE3vvdL6M8ezOX79vA2Aq1nEWLqNQclHc= k8s.io/klog/v2 v2.60.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= +k8s.io/klog/v2 v2.80.1 h1:atnLQ121W371wYYFawwYx1aEY2eUfs4l3J72wtgAwV4= +k8s.io/klog/v2 v2.80.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 h1:Gii5eqf+GmIEwGNKQYQClCayuJCe2/4fZUvF7VG99sU= k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42/go.mod h1:Z/45zLw8lUo4wdiUkI+v/ImEGAvu3WatcZl3lPMR4Rk= k8s.io/utils v0.0.0-20210802155522-efc7438f0176/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= -k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 h1:HNSDgDCrr/6Ly3WEGKZftiE7IY19Vz2GdbOCyI4qqhc= k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +k8s.io/utils v0.0.0-20240102154912-e7106e64919e h1:eQ/4ljkx21sObifjzXwlPKpdGLrCfRziVtos3ofG/sQ= +k8s.io/utils v0.0.0-20240102154912-e7106e64919e/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= @@ -766,5 +773,5 @@ sigs.k8s.io/structured-merge-diff/v4 v4.0.2/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK sigs.k8s.io/structured-merge-diff/v4 v4.2.1 h1:bKCqE9GvQ5tiVHn5rfn1r+yao3aLQEaLzkkmAkf+A6Y= sigs.k8s.io/structured-merge-diff/v4 v4.2.1/go.mod h1:j/nl6xW8vLS49O8YvXW1ocPhZawJtm+Yrr7PPRQ0Vg4= sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= -sigs.k8s.io/yaml v1.3.0 h1:a2VclLzOGrwOHDiV8EfBGhvjHvP46CtW5j6POvhYGGo= -sigs.k8s.io/yaml v1.3.0/go.mod h1:GeOyir5tyXNByN85N/dRIT9es5UQNerPYEKK56eTBm8= +sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E= +sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY= From 1b205384c604ed08c06b4dd3c26f227e4fe3e037 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Fri, 19 Jan 2024 16:24:17 +0100 Subject: [PATCH 109/243] feat: Add `turncat` binaries as release assets (#115) --- .github/workflows/publish.yml | 54 +++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index e8537c6b..4283cdc4 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -65,3 +65,57 @@ jobs: repo: stunner-helm client_payload: '{"tag": "${{ steps.vars.outputs.tag }}", "type": "stunner"}' workflow_file_name: publish.yaml + + add_turncat_binaries: + name: Add turncat binaries to release assets + needs: run_tests + runs-on: ubuntu-latest + strategy: + matrix: + include: + - os: linux + arch: amd64 + file_end: "" + - os: linux + arch: arm64 + file_end: "" + + - os: darwin + arch: amd64 + file_end: "" + - os: darwin + arch: arm64 + file_end: "" + + - os: windows + arch: amd64 + file_end: ".exe" + - os: windows + arch: arm64 + file_end: ".exe" + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Get version + id: vars + run: echo tag=$(echo ${GITHUB_REF:11}) >> $GITHUB_OUTPUT + + - name: Install Go + uses: actions/setup-go@v5 + with: + go-version: '1.21' + + - name: Build turncat binary + run: | + CGO_ENABLED=0 GOARCH=${{ matrix.arch }} GOOS=${{ matrix.os }} go build -ldflags="-w -s" -trimpath -o turncat cmd/turncat/main.go + mv turncat turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + + - name: Release + uses: svenstaro/upload-release-action@v2 + with: + repo_token: ${{ secrets.GITHUB_TOKEN }} + file: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + tag: ${{ github.ref_name }} + asset_name: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} From 6a4b9ac659eaaaed886257745b7cdeee1c449289 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 23 Jan 2024 17:54:39 +0100 Subject: [PATCH 110/243] doc: Improve package docs --- pkg/config/server/config.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go index dff44634..1571ed8a 100644 --- a/pkg/config/server/config.go +++ b/pkg/config/server/config.go @@ -40,6 +40,7 @@ func (s *Server) DeleteConfig(id string) { // UpdateConfig receives a set of ids and newConfigs that represent the state-of-the-world at a // particular instance of time and generates an update per each change. func (s *Server) UpdateConfig(newConfigs []Config) error { + s.log.V(4).Info("processing config updates", "num-configs", len(newConfigs)) oldConfigs := s.configs.Snapshot() for _, oldC := range oldConfigs { @@ -50,6 +51,10 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { s.log.V(2).Info("updating config", "client", newC.Id, "config", newC.Config.String()) s.UpsertConfig(newC.Id, newC.Config) + } else { + s.log.V(2).Info("config not updated", "client", newC.Id, + "old-config", oldC.Config.String(), + "new-config", newC.Config.String()) } found = true break From 08eeec446e21ff8f84118675b8fa98c8c15af79b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 23 Jan 2024 18:46:49 +0100 Subject: [PATCH 111/243] fix: Remove race condition on CDS client poller --- pkg/config/client/cds_client.go | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go index 647374b0..de79b048 100644 --- a/pkg/config/client/cds_client.go +++ b/pkg/config/client/cds_client.go @@ -6,6 +6,7 @@ import ( "fmt" "net/http" "strings" + "sync" "time" "github.com/gorilla/websocket" @@ -90,13 +91,19 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error a.Infof("connection successfully opened to config discovery server at %s", url) - errCh := make(chan error, 1) pingTicker := time.NewTicker(PingPeriod) closePinger := make(chan any) defer close(closePinger) + // wait until all threads are closed and we can remove the error channel + errCh := make(chan error, 1) + var wg sync.WaitGroup + wg.Add(2) + go func() { defer pingTicker.Stop() + defer wg.Done() + for { select { case <-pingTicker.C: @@ -116,7 +123,7 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error // poller go func() { - defer close(errCh) + defer wg.Done() // the next pong must arrive within the PongWait period conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck @@ -147,10 +154,6 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error continue } - // fmt.Println("++++++++++++++++++++") - // fmt.Println(string(msg)) - // fmt.Println("++++++++++++++++++++") - c, err := ParseConfig(msg) if err != nil { // assume it is a YAML/JSON syntax error: report and ignore @@ -176,6 +179,8 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck conn.Close() closePinger <- struct{}{} + wg.Wait() + close(errCh) }() select { From 20c394cd5b89f0097d272fa80ce3333fe2ea38f8 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 25 Jan 2024 12:09:52 +0100 Subject: [PATCH 112/243] fix: Make sure CDS server listens on all local interfaces --- pkg/apis/v1/default.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 60712741..9c8c416c 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -18,7 +18,7 @@ const DefaultAuthName = "default-auth-config" const DefaultMetricsPort int = 8080 const DefaultHealthCheckPort int = 8086 -const DefaultConfigDiscoveryAddress = ":13478" +const DefaultConfigDiscoveryAddress = "0.0.0.0:13478" const DefaultEnvVarName = "STUNNER_NAME" const DefaultEnvVarNamespace = "STUNNER_NAMESPACE" const DefaultEnvVarNodeName = "STUNNER_NODENAME" From 5a55b15f7d96457f20415e06d1789703ad966baa Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 25 Jan 2024 15:00:24 +0100 Subject: [PATCH 113/243] fix: Make sure CDS server listens on localhost --- go.mod | 8 ++++---- go.sum | 15 +++++++++------ pkg/apis/v1/default.go | 2 +- 3 files changed, 14 insertions(+), 11 deletions(-) diff --git a/go.mod b/go.mod index 4248fb69..362e35d5 100644 --- a/go.mod +++ b/go.mod @@ -18,6 +18,7 @@ require ( github.com/pion/transport/v3 v3.0.1 github.com/pion/turn/v3 v3.0.1 github.com/prometheus/client_golang v1.18.0 + github.com/spf13/cobra v1.8.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 go.uber.org/zap v1.26.0 @@ -25,6 +26,7 @@ require ( golang.org/x/time v0.5.0 k8s.io/api v0.24.3 k8s.io/apimachinery v0.24.3 + k8s.io/client-go v0.24.2 k8s.io/utils v0.0.0-20240102154912-e7106e64919e sigs.k8s.io/controller-runtime v0.12.3 sigs.k8s.io/yaml v1.4.0 @@ -42,23 +44,22 @@ require ( github.com/go-openapi/jsonpointer v0.20.2 // indirect github.com/go-openapi/jsonreference v0.19.5 // indirect github.com/go-openapi/swag v0.22.7 // indirect - github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/golang/protobuf v1.5.3 // indirect github.com/google/gnostic v0.5.7-v3refs // indirect github.com/google/go-cmp v0.6.0 // indirect github.com/google/gofuzz v1.1.0 // indirect - github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect github.com/imdario/mergo v0.3.12 // indirect + github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/invopop/yaml v0.2.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/mailru/easyjson v0.7.7 // indirect + github.com/moby/spdystream v0.2.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/onsi/ginkgo/v2 v2.15.0 // indirect github.com/onsi/gomega v1.30.0 // indirect github.com/perimeterx/marshmallow v1.1.5 // indirect github.com/pion/randutil v0.1.0 // indirect @@ -83,7 +84,6 @@ require ( gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect - k8s.io/client-go v0.24.2 // indirect k8s.io/klog/v2 v2.80.1 // indirect k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 // indirect sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect diff --git a/go.sum b/go.sum index 278652a1..ed10a42e 100644 --- a/go.sum +++ b/go.sum @@ -53,6 +53,7 @@ github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdko github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= +github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= @@ -68,6 +69,7 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= +github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= @@ -75,6 +77,7 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/deepmap/oapi-codegen/v2 v2.0.0 h1:3TS7w3r+XnjKFXcbFbc16pTWzfTy0OLPkCsutEHjWDA= github.com/deepmap/oapi-codegen/v2 v2.0.0/go.mod h1:7zR+ZL3WzLeCkr2k8oWTxEa0v8y/F25ane0l6A5UjLA= github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= +github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 h1:yUdfgN0XgIJw7foRItutHYUIhlcKzcSf5vDpdhQAKTc= github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= github.com/emicklei/go-restful v2.9.5+incompatible h1:spTtZBk5DYEvbxMVutUuTyh1Ao2r4iyvLdACqsl/Ljk= @@ -118,8 +121,6 @@ github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh github.com/go-openapi/swag v0.19.14/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= github.com/go-openapi/swag v0.22.7 h1:JWrc1uc/P9cSomxfnsFSVWoE1FW6bNbrVPmpQYpCcR8= github.com/go-openapi/swag v0.22.7/go.mod h1:Gl91UqO+btAM0plGGxHqJcQZ1ZTy6jbmridBTsDy8A0= -github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= -github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= @@ -191,8 +192,6 @@ github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 h1:yAJXTCF9TqKcTiHJAE8dj7HMvPfh66eeA2JYW7eFpSE= -github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU= @@ -215,6 +214,8 @@ github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1: github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= +github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= +github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY= github.com/invopop/yaml v0.2.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= @@ -240,6 +241,7 @@ github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJ github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8= github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= @@ -263,8 +265,6 @@ github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= -github.com/onsi/ginkgo/v2 v2.15.0 h1:79HwNRBAZHOEwrczrgSOPy+eFTTlIGELKy5as+ClttY= -github.com/onsi/ginkgo/v2 v2.15.0/go.mod h1:HlxMHtYF57y6Dpf+mc5529KKmSq9h2FpCF+/ZkwUxKM= github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= @@ -304,7 +304,10 @@ github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= +github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= +github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 9c8c416c..60712741 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -18,7 +18,7 @@ const DefaultAuthName = "default-auth-config" const DefaultMetricsPort int = 8080 const DefaultHealthCheckPort int = 8086 -const DefaultConfigDiscoveryAddress = "0.0.0.0:13478" +const DefaultConfigDiscoveryAddress = ":13478" const DefaultEnvVarName = "STUNNER_NAME" const DefaultEnvVarNamespace = "STUNNER_NAMESPACE" const DefaultEnvVarNodeName = "STUNNER_NODENAME" From 602ed95ae201a363dbc34e9832d1e510a5611b3a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 1 Feb 2024 19:20:31 +0100 Subject: [PATCH 114/243] refactor: Rewrite stunnerctl and turncat to use the CDS API, fix #81 So far stunnerctl and turncat have used the ConfigMap rendered by the operator to load the running config of the requested Gateway. Since the ConfigMap is slated to be removed, this rewrite makes sure that the CLI tools access dataplane configs from the CDS API served by the operator. --- README.md | 31 +- cmd/stunnerctl/README.md | 91 ++- cmd/stunnerctl/main.go | 233 ++++++++ cmd/stunnerctl/{stunnerctl => stunnerctl.sh} | 21 +- cmd/turncat/README.md | 41 +- cmd/turncat/main.go | 106 ++-- docs/README.md | 1 + docs/cmd/stunnerctl.md | 91 ++- docs/cmd/turncat.md | 41 +- docs/examples/benchmark/README.md | 42 +- docs/examples/benchmark/benchmark.sh | 14 +- docs/examples/direct-one2one-call/README.md | 23 +- docs/examples/kurento-one2one-call/README.md | 58 +- docs/examples/livekit/README.md | 22 +- docs/examples/simple-tunnel/README.md | 40 +- go.mod | 69 ++- go.sum | 592 ++++--------------- pkg/apis/v1/default.go | 2 + pkg/apis/v1/stunner.go | 74 ++- pkg/config/client/cds_api.go | 68 +-- pkg/config/client/k8s_client.go | 167 ++++++ pkg/config/client/util.go | 57 ++ 22 files changed, 1095 insertions(+), 789 deletions(-) create mode 100644 cmd/stunnerctl/main.go rename cmd/stunnerctl/{stunnerctl => stunnerctl.sh} (73%) create mode 100644 pkg/config/client/k8s_client.go create mode 100644 pkg/config/client/util.go diff --git a/README.md b/README.md index c7b3f20e..00c2c327 100644 --- a/README.md +++ b/README.md @@ -349,30 +349,23 @@ The current STUNner dataplane configuration is always made available in a conven has the same name and namespace as the Gateway it belongs to (so this is supposed to be `stunner/udp-gateway` as per our example). -STUNner comes with a small utility to dump the running configuration in human readable format (you -must have [`jq`](https://stedolan.github.io/jq) installed in your PATH to be able to use it). Issue -the below from the main STUNner directory. +STUNner comes with a small CLI utility called [`stunnerctl`](/cmd/stunnerctl/README.md) to dump the +running configuration in human readable format. ```console -cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener 1 - Name: stunner/udp-gateway/udp-listener - Listener: stunner/udp-gateway/udp-listener - Protocol: TURN-UDP - Public address: 34.34.150.65 - Public port: 3478 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: ephemeral, shared-secret: my-very-secure-secret +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] ``` As it turns out, STUNner has successfully assigned a public IP and port to our Gateway and set the -STUN/TURN credentials based on the GatewayConfig. You can use the below to dump the entire running -configuration; `jq` is there just to pretty-print JSON. - -```console -kubectl get cm -n stunner udp-gateway -o jsonpath="{.data.stunnerd\.conf}" | jq . -``` +STUN/TURN credentials based on the GatewayConfig. ### Testing diff --git a/cmd/stunnerctl/README.md b/cmd/stunnerctl/README.md index 6982ded4..4e017234 100644 --- a/cmd/stunnerctl/README.md +++ b/cmd/stunnerctl/README.md @@ -1,15 +1,74 @@ # stunnerctl: Command line toolbox for STUNner -A CLI tool to simplify the interaction with STUNner. +A CLI tool to simplify the interaction with STUNner. +The prominent use of `stunnerctl` is to load or watch STUNner dataplane configurations from a Kubernetes cluster for debugging and troubleshooting, or just for checking whether everything is configured the way it should be. + +## Installation + +Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. + +```console +go install github.com/l7mp/stunner/cmd/stunnerctl@latest +``` + +You can also enforce a specific OS, CPU architecture, and STUNner version: + +```console +GOOS=windows GOARCH=amd64 go install github.com/l7mp/stunner/cmd/stunnerctl@v0.17.5 +``` + +Building from source is as easy as it usually gets with Go: + +```console +cd stunner +go build -o stunnerctl cmd/stunnerctl/main.go +``` ## Usage -Dump the running config of a STUNner gateway in human-readable format. +Type `stunnerctl` to get a glimpse of the features provided. Below are some common usage examples. + +- Dump a summary of the running config of the STUNner gateway called `udp-gateway` deployed into the `stunner` namespace: + + ```console + stunnerctl -n stunner config udp-gateway + Gateway: stunner/udp-gateway (loglevel: "all:INFO") + Authentication type: static, username/password: user-1/pass-1 + Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:9001 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.3, 10.80.7.104] + ``` + +- Dump a the running config of all gateways in the `stunner` namespace in JSON format (YAML is also available using `-o yaml`): -The below will select the Gateway called `tcp-gateway` in the `stunner` namespace: + ```console + stunnerctl -n stunner config -o json + {"version":"v1","admin":{"name":"stunner/tcp-gateway",...}} + {"version":"v1","admin":{"name":"stunner/udp-gateway",...}}} + ``` + +- Watch all STUNner configs as they are being refreshed and dump only the name of the STUNner gateway whose config changes: + + ```console + stunnerctl config --all-namespaces -o jsonpath='{.admin.name}' -w + stunner/tcp-gateway + stunner/udp-gateway + ... + ``` + +## Fallback + +For those who don't have the Go toolchain available to run `go install`, STUNner provides a minimalistic `stunnerctl` replacement called `stunnerctl.sh`. +This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. + +The below will dump the running config of `tcp-gateway` deployed into the `stunner` namespace: ```console -cmd/stunnerctl/stunnerctl running-config stunner/stunner-gateway +cd stunner +cmd/stunnerctl/stunnerctl.sh running-config stunner/stunner-gateway STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 @@ -21,6 +80,30 @@ Listener 1 Public port: 3478 ``` +## Last resort + +You can use `kubectl port-forward` to load or watch STUNner configs manually. +Open a port-forwarded connection to the STUNner gateway operator: + +``` console +export CDS_SERVER_NAME=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.name}') +export CDS_SERVER_NAMESPACE=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.namespace}') +kubectl -n $CDS_SERVER_NAMESPACE port-forward pod/${CDS_SERVER_NAME} 63478:13478 & +``` + +If all goes well, you can now connect to the STUNner config discovery API served by the gateway operator directly, just using `curl`. +The below will load the config of the `udp-gateway` in the `stunner` namespace: + +``` console +curl -s http://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway +``` + +If you happen to have a WebSocket client like the wonderful [`websocat`](https://github.com/vi/websocat) tool installed, you can also watch the configs as they are being rendered by the operator en live. + +``` console +websocat ws://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway?watch=true - +``` + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go new file mode 100644 index 00000000..ed7f3ab4 --- /dev/null +++ b/cmd/stunnerctl/main.go @@ -0,0 +1,233 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "os" + "os/signal" + "regexp" + "strings" + "syscall" + + "github.com/pion/logging" + "github.com/spf13/cobra" + cliopt "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/util/jsonpath" + "sigs.k8s.io/yaml" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" + "github.com/l7mp/stunner/pkg/logger" +) + +// list all configs: stunnerctl get config --all-namespaces +// watch all configs in namesapce stunner: stunnerctl -n stunner get config --watch +// get short-form config for stunner/udp-gateway: stunnerctl -n stunner get config udp-gateway +// get config for stunner/udp-gateway in yaml format: stunnerctl -n stunner get config udp-gateway --output yaml + +var ( + output string + watch, all, verbose bool + jsonQuery *jsonpath.JSONPath + k8sConfigFlags *cliopt.ConfigFlags + cdsConfigFlags *cdsclient.CDSConfigFlags + loggerFactory *logger.LeveledLoggerFactory + log logging.LeveledLogger + + rootCmd = &cobra.Command{ + Use: "stunnerctl", + Short: "A command line utility to inspect STUNner dataplane configurations.", + Long: "The stunnerctl tool is a CLI for inspecting, watching and troublehssooting the configuration of STUNner gateways", + DisableAutoGenTag: true, + } +) + +var ( + configCmd = &cobra.Command{ + Use: "config", + Aliases: []string{"stunner-config"}, + Short: "Gets or watches STUNner configs", + Args: cobra.RangeArgs(0, 1), + DisableAutoGenTag: true, + Run: func(cmd *cobra.Command, args []string) { + if err := runConfig(cmd, args); err != nil { + fmt.Println(err) + os.Exit(1) + } + }, + } +) + +func init() { + rootCmd.PersistentFlags().BoolVarP(&watch, "watch", "w", false, "Watch for config updates from server") + rootCmd.PersistentFlags().BoolVarP(&all, "all-namespaces", "a", false, "Consider all namespaces") + rootCmd.PersistentFlags().StringVarP(&output, "output", "o", "summary", "Output format") + rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") + + // Kubernetes config flags + k8sConfigFlags = cliopt.NewConfigFlags(true) + k8sConfigFlags.AddFlags(rootCmd.PersistentFlags()) + + // CDS server discovery flags + cdsConfigFlags = cdsclient.NewCDSConfigFlags() + cdsConfigFlags.AddFlags(rootCmd.PersistentFlags()) + + rootCmd.AddCommand(configCmd) +} + +func main() { + if err := rootCmd.Execute(); err != nil { + fmt.Fprintf(os.Stderr, "Whoops. There was an error while executing your CLI '%s'", err) + os.Exit(1) + } +} + +func runConfig(cmd *cobra.Command, args []string) error { + loglevel := "all:WARN" + if verbose { + loglevel = "all:TRACE" + } + loggerFactory = logger.NewLoggerFactory(loglevel) + log = loggerFactory.NewLogger("stunnerctl") + + gwNs := "default" + if k8sConfigFlags.Namespace != nil { + gwNs = *k8sConfigFlags.Namespace + } + + if strings.HasPrefix(output, "jsonpath") { + as := strings.Split(output, "=") + if len(as) != 2 || as[0] != "jsonpath" { + return fmt.Errorf("invalid jsonpath output definition %q", output) + } + + jsonQuery = jsonpath.New("output") + + // Parse and print jsonpath + fields, err := RelaxedJSONPathExpression(as[1]) + if err != nil { + return fmt.Errorf("invalid jsonpath query %w", err) + } + + if err := jsonQuery.Parse(fields); err != nil { + return fmt.Errorf("cannor parse jsonpath query %w", err) + } + output = "jsonpath" + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + log.Debug("Searching for CDS server") + cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, + loggerFactory.NewLogger("cds-fwd")) + if err != nil { + return fmt.Errorf("error searching for CDS server: %w", err) + } + + var cds cdsclient.CdsApi + cdslog := loggerFactory.NewLogger("cds-client") + if all { + cds, err = cdsclient.NewAllConfigsAPI(cdsAddr, cdslog) + } else if len(args) == 0 { + cds, err = cdsclient.NewConfigsNamespaceAPI(cdsAddr, gwNs, cdslog) + } else { + gwName := args[0] + cds, err = cdsclient.NewConfigNamespaceNameAPI(cdsAddr, gwNs, gwName, cdslog) + } + + if err != nil { + return fmt.Errorf("error creating CDS client: %w", err) + } + + confChan := make(chan *stnrv1.StunnerConfig, 8) + if watch { + err := cds.Watch(ctx, confChan) + if err != nil { + close(confChan) + return err + } + + go func() { + sigs := make(chan os.Signal, 1) + signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM) + <-sigs + close(confChan) + }() + } else { + resp, err := cds.Get(ctx) + if err != nil { + close(confChan) + return err + } + for _, c := range resp { + confChan <- c + } + + close(confChan) + } + + for c := range confChan { + switch output { + case "yaml": + if out, err := yaml.Marshal(c); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "json": + if out, err := json.Marshal(c); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "jsonpath": + values, err := jsonQuery.FindResults(c) + if err != nil { + return err + } + + if len(values) == 0 || len(values[0]) == 0 { + fmt.Println("") + } + + for arrIx := range values { + for valIx := range values[arrIx] { + fmt.Printf("%v\n", values[arrIx][valIx].Interface()) + } + } + case "summary": + fmt.Print(string(c.Summary())) + case "status": + fallthrough + default: + fmt.Println(c.String()) + } + } + + return nil +} + +var jsonRegexp = regexp.MustCompile(`^\{\.?([^{}]+)\}$|^\.?([^{}]+)$`) + +// k8s.io/kubectl/pkg/cmd/get +func RelaxedJSONPathExpression(pathExpression string) (string, error) { + if len(pathExpression) == 0 { + return pathExpression, nil + } + submatches := jsonRegexp.FindStringSubmatch(pathExpression) + if submatches == nil { + return "", fmt.Errorf("unexpected path string, expected a 'name1.name2' or '.name1.name2' or '{name1.name2}' or '{.name1.name2}'") + } + if len(submatches) != 3 { + return "", fmt.Errorf("unexpected submatch list: %v", submatches) + } + var fieldSpec string + if len(submatches[1]) != 0 { + fieldSpec = submatches[1] + } else { + fieldSpec = submatches[2] + } + return fmt.Sprintf("{.%s}", fieldSpec), nil +} diff --git a/cmd/stunnerctl/stunnerctl b/cmd/stunnerctl/stunnerctl.sh similarity index 73% rename from cmd/stunnerctl/stunnerctl rename to cmd/stunnerctl/stunnerctl.sh index a9b1b88c..fa7272fb 100755 --- a/cmd/stunnerctl/stunnerctl +++ b/cmd/stunnerctl/stunnerctl.sh @@ -6,6 +6,9 @@ USAGE="stunnerctl running-config " COMMAND="$1" ARG="$2" +# stop the port-forwarder +trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT + jq=$(which jq) if [ -z "$jq" ] ; then echo "Error: cannot find jq in PATH" && exit 0 @@ -19,12 +22,24 @@ running_config () { name=${args[1]} [ -z $namespace -o -z $name ] && echo "cannot parse argument" && exit 0 - [ $(kubectl get cm -n $namespace -o json| jq ".items | map(select(.metadata.name==\"${name}\"))|length") -eq 0 ] && \ - echo "STUNner configmap ${namespace}/${name} not found" && exit 1 + # find the CDS server + CDS_SERVER_NAME=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.name}') + CDS_SERVER_NAMESPACE=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.namespace}') + [ -z $CDS_SERVER_NAME -o -z $CDS_SERVER_NAMESPACE ] && echo "Could not find CDS server" && exit 1 + + # start the port-forwarder + kubectl -n $CDS_SERVER_NAMESPACE port-forward pod/${CDS_SERVER_NAME} 63478:13478 >/dev/null 2>&1 & + # query the cds server + sleep 1 tmpfile=$(mktemp "./stunnerd-config.XXXXXX") - kubectl get cm -n $namespace $name -o jsonpath="{.data.stunnerd\.conf}" > $tmpfile + curl -s http://127.0.0.1:63478/api/v1/configs/${namespace}/${name} > $tmpfile + if grep -q "onfig not found" $tmpfile >/dev/null 2>&1; then + cat $tmpfile + exit 1 + fi + local AUTH_TYPE=$($jq ".auth.type" $tmpfile) [ $AUTH_TYPE == "plaintext" ] && AUTH_TYPE="static" [ $AUTH_TYPE == "longterm" ] && AUTH_TYPE="ephemeral" diff --git a/cmd/turncat/README.md b/cmd/turncat/README.md index 8e2c3552..9589affe 100644 --- a/cmd/turncat/README.md +++ b/cmd/turncat/README.md @@ -1,40 +1,65 @@ # turncat: Swiss-army-knife testing tool for STUNner -`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster which is much faster than the TCP connection used by kubectl. +`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. +The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. +This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster. +This is much faster than the TCP connection used by `kubectl`. -## Getting Started +## Installation -### Installation +Install the `turncat` binary using the standard Go toolchain and add it to `$PATH`. -As simple as it gets: +```console +go install github.com/l7mp/stunner/cmd/turncat@latest +``` + +You can also enforce a specific OS, CPU architecture, and STUNner version like below: + +```console +GOOS=windows GOARCH=amd64 go install github.com/l7mp/stunner/cmd/turncat@v0.17.5 +``` + +Building from source is as easy as it usually gets with Go: ```console cd stunner go build -o turncat cmd/turncat/main.go ``` -### Usage +## Usage -Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP listener located at `192.0.2.2:53`. Use the [`static` STUN/TURN credential mechanism](/docs/AUTH.md) to authenticate with the TURN server and set the user/passwd to `test/test`: +Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP listener located at `192.0.2.2:53`. +Use the [`static` STUN/TURN credential mechanism](/docs/AUTH.md) to authenticate with the TURN server and set the user/passwd to `test/test`: ```console ./turncat --log=all:INFO,turncat:DEBUG udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478 \ udp://192.0.2.2:53 ``` -TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accept self-signed TLS certificates and `--verbose` is equivalent to setting all `turncat` loggers to DEBUG mode (`-l all:DEBUG`). +TLS/DTLS should also work. +Below `--insecure` allows `turncat` to accept self-signed TLS certificates and `--verbose` is equivalent to setting all loggers to DEBUG mode (`-l all:DEBUG`). ```console ./turncat --verbose --insecure udp://127.0.0.1:5000 \ turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 ``` -Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace and connect to the TURN listener named `udp-listener`. The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will enable verbose logging. +Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. +The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace and connect to the TURN listener named `udp-listener`. +The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). +The CLI flag `-v` will enable verbose logging. ```console ./turncat -v - k8s://stunner/udp-gateway:udp-listener udp://10.0.0.1:9001 ``` +Note that the standard `kubectl` command line flags are available. +For instance, the below will use the context `prod-europe` from the kubeconfig file `kube-prod.conf`: + +```console +./turncat --kubeconfig=kube-prod.conf --context prod-europe -v - k8s://... udp://... +``` + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 773daafb..f3e98a23 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -2,7 +2,6 @@ package main import ( "context" - "encoding/json" "fmt" "os" "os/signal" @@ -14,21 +13,28 @@ import ( "github.com/pion/turn/v3" flag "github.com/spf13/pflag" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/types" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/client/config" + cliopt "k8s.io/cli-runtime/pkg/genericclioptions" "github.com/l7mp/stunner" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) -const usage = "turncat [-l|--log ] [-i|--insecure] client server peer\n\tclient: ://:\n\tserver: @: | /:listener\n\tpeer: udp://:\n\tauth: \n" -const defaultStunnerdConfigfileName = "stunnerd.conf" - -var log logging.LeveledLogger -var defaultDuration time.Duration +const usage = `turncat [options] + client-addr: ://: + turn-server-addr: @: | /: + peer-addr: udp://: + auth: +` + +var ( + k8sConfigFlags *cliopt.ConfigFlags + cdsConfigFlags *cdsclient.CDSConfigFlags + log logging.LeveledLogger + defaultDuration time.Duration + loggerFactory *logger.LeveledLoggerFactory +) func main() { var Usage = func() { @@ -38,12 +44,20 @@ func main() { os.Args[0] = "turncat" defaultDuration, _ = time.ParseDuration("1h") - var level = flag.StringP("log", "l", "all:WARN", "Log level (default: all:WARN).") - // var user = flag.StringP("user", "u", "", "Set username. Auth fields in the TURN URI override this.") - // var passwd = flag.StringP("log", "l", "all:WARN", "Log level (default: all:WARN).") - var insecure = flag.BoolP("insecure", "i", false, "Insecure TLS mode, accept self-signed certificates (default: false).") - var verbose = flag.BoolP("verbose", "v", false, "Verbose logging, identical to -l all:DEBUG.") + + // Kubernetes config flags + k8sConfigFlags = cliopt.NewConfigFlags(true) + k8sConfigFlags.AddFlags(flag.CommandLine) + + // CDS server discovery flags + cdsConfigFlags = cdsclient.NewCDSConfigFlags() + cdsConfigFlags.AddFlags(flag.CommandLine) + + var level = flag.StringP("log", "l", "all:WARN", "Log level") + var insecure = flag.BoolP("insecure", "i", false, "Insecure TLS mode, accept self-signed TURN server certificates (default: false)") + var verbose = flag.BoolP("verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") var help = flag.BoolP("help", "h", false, "Display this help text and exit") + flag.Parse() if *help { @@ -60,8 +74,8 @@ func main() { *level = "all:DEBUG" } - logger := logger.NewLoggerFactory(*level) - log = logger.NewLogger("turncat-cli") + loggerFactory = logger.NewLoggerFactory(*level) + log = loggerFactory.NewLogger("turncat-cli") uri := flag.Arg(1) log.Debugf("Reading STUNner config from URI %q", uri) @@ -93,7 +107,7 @@ func main() { Realm: config.Auth.Realm, AuthGen: authGen, InsecureMode: *insecure, - LoggerFactory: logger, + LoggerFactory: loggerFactory, } t, err := stunner.NewTurncat(cfg) if err != nil { @@ -143,37 +157,31 @@ func getStunnerConfFromK8s(def string) (*stnrv1.StunnerConfig, error) { return nil, err } - ctx := context.Background() - cfg := config.GetConfigOrDie() + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() - cli, err := client.New(cfg, client.Options{}) + log.Debug("Searching for CDS server") + cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, + loggerFactory.NewLogger("cds-fwd")) if err != nil { - return nil, err - } - - // get the configmap - lookupKey := types.NamespacedName{ - Namespace: namespace, - Name: name, + return nil, fmt.Errorf("error searching for CDS server: %w", err) } - cm := &corev1.ConfigMap{} - err = cli.Get(ctx, lookupKey, cm) + cds, err := cdsclient.NewConfigNamespaceNameAPI(cdsAddr, namespace, name, + loggerFactory.NewLogger("cds-client")) if err != nil { - return nil, err + return nil, fmt.Errorf("error creating CDS client: %w", err) } - //parse out the stunnerconf - jsonConf, found := cm.Data[defaultStunnerdConfigfileName] - if !found { - return nil, fmt.Errorf("error unpacking STUNner configmap: %s not found", - defaultStunnerdConfigfileName) + confs, err := cds.Get(ctx) + if err != nil { + return nil, fmt.Errorf("error obtaining config from CDS client: %w", err) } - - conf := stnrv1.StunnerConfig{} - if err := json.Unmarshal([]byte(jsonConf), &conf); err != nil { - return nil, err + if len(confs) != 1 { + return nil, fmt.Errorf("invalid number of configs returned from CDS client: %d", + len(confs)) } + conf := confs[0] // remove all but the named listener ls := []stnrv1.ListenerConfig{} @@ -202,10 +210,10 @@ func getStunnerConfFromK8s(def string) (*stnrv1.StunnerConfig, error) { "specified TURN server URI", listener) } - conf.Listeners = []stnrv1.ListenerConfig{{}} + conf.Listeners = make([]stnrv1.ListenerConfig, 1) copy(conf.Listeners, ls) - return &conf, nil + return conf, nil } func getStunnerConfFromCLI(def string) (*stnrv1.StunnerConfig, error) { @@ -292,11 +300,17 @@ func getStunnerURI(config *stnrv1.StunnerConfig) (string, error) { } func parseK8sDef(def string) (string, string, string, error) { - re := regexp.MustCompile(`([0-9A-Za-z_-]+)/([0-9A-Za-z_-]+):([0-9A-Za-z_-]+)`) + re := regexp.MustCompile(`^/([0-9A-Za-z_-]+):([0-9A-Za-z_-]+)$`) xs := re.FindStringSubmatch(def) - if len(xs) != 4 { - return "", "", "", fmt.Errorf("cannot parse STUNner configmap def: %q", def) + if len(xs) == 3 && k8sConfigFlags.Namespace != nil { + return *k8sConfigFlags.Namespace, xs[1], xs[2], nil + } + + re = regexp.MustCompile(`^([0-9A-Za-z_-]+)/([0-9A-Za-z_-]+):([0-9A-Za-z_-]+)$`) + xs = re.FindStringSubmatch(def) + if len(xs) == 4 { + return xs[1], xs[2], xs[3], nil } - return xs[1], xs[2], xs[3], nil + return "", "", "", fmt.Errorf("cannot parse STUNner K8s URI: %q", def) } diff --git a/docs/README.md b/docs/README.md index 9832cce7..5adbc162 100644 --- a/docs/README.md +++ b/docs/README.md @@ -49,3 +49,4 @@ * [`stunnerd` manual](cmd/stunnerd.md) * [`turncat` manual](cmd/turncat.md) * [`stunnerctl` manual](cmd/stunnerctl.md) +* [Benchmarking](examples/benchmark) diff --git a/docs/cmd/stunnerctl.md b/docs/cmd/stunnerctl.md index 6982ded4..4e017234 100644 --- a/docs/cmd/stunnerctl.md +++ b/docs/cmd/stunnerctl.md @@ -1,15 +1,74 @@ # stunnerctl: Command line toolbox for STUNner -A CLI tool to simplify the interaction with STUNner. +A CLI tool to simplify the interaction with STUNner. +The prominent use of `stunnerctl` is to load or watch STUNner dataplane configurations from a Kubernetes cluster for debugging and troubleshooting, or just for checking whether everything is configured the way it should be. + +## Installation + +Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. + +```console +go install github.com/l7mp/stunner/cmd/stunnerctl@latest +``` + +You can also enforce a specific OS, CPU architecture, and STUNner version: + +```console +GOOS=windows GOARCH=amd64 go install github.com/l7mp/stunner/cmd/stunnerctl@v0.17.5 +``` + +Building from source is as easy as it usually gets with Go: + +```console +cd stunner +go build -o stunnerctl cmd/stunnerctl/main.go +``` ## Usage -Dump the running config of a STUNner gateway in human-readable format. +Type `stunnerctl` to get a glimpse of the features provided. Below are some common usage examples. + +- Dump a summary of the running config of the STUNner gateway called `udp-gateway` deployed into the `stunner` namespace: + + ```console + stunnerctl -n stunner config udp-gateway + Gateway: stunner/udp-gateway (loglevel: "all:INFO") + Authentication type: static, username/password: user-1/pass-1 + Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:9001 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.3, 10.80.7.104] + ``` + +- Dump a the running config of all gateways in the `stunner` namespace in JSON format (YAML is also available using `-o yaml`): -The below will select the Gateway called `tcp-gateway` in the `stunner` namespace: + ```console + stunnerctl -n stunner config -o json + {"version":"v1","admin":{"name":"stunner/tcp-gateway",...}} + {"version":"v1","admin":{"name":"stunner/udp-gateway",...}}} + ``` + +- Watch all STUNner configs as they are being refreshed and dump only the name of the STUNner gateway whose config changes: + + ```console + stunnerctl config --all-namespaces -o jsonpath='{.admin.name}' -w + stunner/tcp-gateway + stunner/udp-gateway + ... + ``` + +## Fallback + +For those who don't have the Go toolchain available to run `go install`, STUNner provides a minimalistic `stunnerctl` replacement called `stunnerctl.sh`. +This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. + +The below will dump the running config of `tcp-gateway` deployed into the `stunner` namespace: ```console -cmd/stunnerctl/stunnerctl running-config stunner/stunner-gateway +cd stunner +cmd/stunnerctl/stunnerctl.sh running-config stunner/stunner-gateway STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 @@ -21,6 +80,30 @@ Listener 1 Public port: 3478 ``` +## Last resort + +You can use `kubectl port-forward` to load or watch STUNner configs manually. +Open a port-forwarded connection to the STUNner gateway operator: + +``` console +export CDS_SERVER_NAME=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.name}') +export CDS_SERVER_NAMESPACE=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.namespace}') +kubectl -n $CDS_SERVER_NAMESPACE port-forward pod/${CDS_SERVER_NAME} 63478:13478 & +``` + +If all goes well, you can now connect to the STUNner config discovery API served by the gateway operator directly, just using `curl`. +The below will load the config of the `udp-gateway` in the `stunner` namespace: + +``` console +curl -s http://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway +``` + +If you happen to have a WebSocket client like the wonderful [`websocat`](https://github.com/vi/websocat) tool installed, you can also watch the configs as they are being rendered by the operator en live. + +``` console +websocat ws://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway?watch=true - +``` + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). diff --git a/docs/cmd/turncat.md b/docs/cmd/turncat.md index 8e2c3552..9589affe 100644 --- a/docs/cmd/turncat.md +++ b/docs/cmd/turncat.md @@ -1,40 +1,65 @@ # turncat: Swiss-army-knife testing tool for STUNner -`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster which is much faster than the TCP connection used by kubectl. +`turncat` is a STUN/TURN client to open a connection through a TURN server to an arbitrary remote address/port. +The main use is to open a local tunnel endpoint to any service running inside a Kubernetes cluster via STUNner. +This is very similar in functionality to `kubectl port-forward`, but it uses STUN/TURN to enter the cluster. +This is much faster than the TCP connection used by `kubectl`. -## Getting Started +## Installation -### Installation +Install the `turncat` binary using the standard Go toolchain and add it to `$PATH`. -As simple as it gets: +```console +go install github.com/l7mp/stunner/cmd/turncat@latest +``` + +You can also enforce a specific OS, CPU architecture, and STUNner version like below: + +```console +GOOS=windows GOARCH=amd64 go install github.com/l7mp/stunner/cmd/turncat@v0.17.5 +``` + +Building from source is as easy as it usually gets with Go: ```console cd stunner go build -o turncat cmd/turncat/main.go ``` -### Usage +## Usage -Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP listener located at `192.0.2.2:53`. Use the [`static` STUN/TURN credential mechanism](/docs/AUTH.md) to authenticate with the TURN server and set the user/passwd to `test/test`: +Listen to client connections on the UDP listener `127.0.0.1:5000` and tunnel the received packets through the TURN server located at `192.0.2.1:3478` to the UDP listener located at `192.0.2.2:53`. +Use the [`static` STUN/TURN credential mechanism](/docs/AUTH.md) to authenticate with the TURN server and set the user/passwd to `test/test`: ```console ./turncat --log=all:INFO,turncat:DEBUG udp://127.0.0.1:5000 turn://test:test@192.0.2.1:3478 \ udp://192.0.2.2:53 ``` -TLS/DTLS should also work fine; note that `--insecure` allows `turncat` to accept self-signed TLS certificates and `--verbose` is equivalent to setting all `turncat` loggers to DEBUG mode (`-l all:DEBUG`). +TLS/DTLS should also work. +Below `--insecure` allows `turncat` to accept self-signed TLS certificates and `--verbose` is equivalent to setting all loggers to DEBUG mode (`-l all:DEBUG`). ```console ./turncat --verbose --insecure udp://127.0.0.1:5000 \ turn://test:test@192.0.2.1:3478?transport=tls udp://192.0.2.2:53 ``` -Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace and connect to the TURN listener named `udp-listener`. The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). The CLI flag `-v` will enable verbose logging. +Alternatively, you can specify the special TURN server meta-URI `k8s://stunner/udp-gateway:udp-listener` to let `turncat` parse the running STUNner configuration from the active Kubernetes cluster. +The URI directs `turncat` to read the config of the STUNner Gateway called `udp-gateway` in the `stunner` namespace and connect to the TURN listener named `udp-listener`. +The CLI flag `-` instructs `turncat` to listen on the standard input: anything you type in the terminal will be sent via STUNner to the peer `udp://10.0.0.1:9001` (after you press Enter). +The CLI flag `-v` will enable verbose logging. ```console ./turncat -v - k8s://stunner/udp-gateway:udp-listener udp://10.0.0.1:9001 ``` +Note that the standard `kubectl` command line flags are available. +For instance, the below will use the context `prod-europe` from the kubeconfig file `kube-prod.conf`: + +```console +./turncat --kubeconfig=kube-prod.conf --context prod-europe -v - k8s://... udp://... +``` + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). diff --git a/docs/examples/benchmark/README.md b/docs/examples/benchmark/README.md index 3b110a64..605abe70 100644 --- a/docs/examples/benchmark/README.md +++ b/docs/examples/benchmark/README.md @@ -1,29 +1,25 @@ # Performance Benchmarking -With the help of this guide you are able to take performance measurements in your setup using STUNner. Both running STUNner locally (outside of Kubernetes) and running STUNner in Kubernetes can be evaluated. +This guide will help you to take performance measurements in your setup using STUNner. +Both running STUNner locally (outside of Kubernetes) and running STUNner in Kubernetes can be evaluated. +You can then compare the locally measured result with the result obtained from Kubernetes and figure out performance bottlenecks. -Compare the locally measured result to the result measured in Kubernetes and figure out the overhead cost. The extra cost of your cluster's networking may surprise you in terms of extra delay or more packet drops using the same bandwidth. - -Locally there is no installation needed, it should take less than a minute to measure. +Locally there is no installation needed, it should take less than a minute to make a measurements. If you have a Kubernetes cluster up and running, the installation and measurement should take a few minutes max. ## Tools The tools used in the measurement are the following: -* `iperf` Using to create traffic flows between the clients and server -* `turncat` Using to open a connection through STUNner to the iperf server -* `STUNner` Acting as a STUN server towards `turncat` clients - -### Caveats - -When measuring latency with `iperf` you might be fooled because it is [measuring one-way latency](https://stackoverflow.com/questions/63793030/iperf2-latency-is-a-two-way-or-one-way-latency) which requires the clocks to be synchronized. This means you might see corrupted latencies such as negative ones. +* `iperf`: Used for creating traffic flows between the clients and the server. +* `turncat`: Used for opening a connection through STUNner to the iperf server. +* `STUNner`: Our TURN server exposed to `turncat` clients. ## Measurement Setup ### Local setup -All the components are running locally. All of them are using `127.0.0.1` addresses. +All the components are running locally using localhost to simulate the network. ![STUNner benchmark local test architecture](../../img/stunner_benchmark_local.svg) @@ -35,7 +31,7 @@ All the components are running locally. All of them are using `127.0.0.1` addres ## Prerequisites -You must have [`iperfv2`](https://iperf.fr), [`jq`](https://stedolan.github.io/jq/) and most importantly [Go](https://go.dev/doc/install) installed locally to run this tutorial. +You must have [`iperfv2`](https://iperf.fr), [`jq`](https://stedolan.github.io/jq/), and [Go](https://go.dev/doc/install) installed locally to run this tutorial. For Kubernetes benchmarks, you also need a running [STUNner installation](/docs/INSTALL.md). ## Install locally @@ -43,17 +39,6 @@ You are good to go. No installation steps required. ## Install on Kubernetes -Install it in case you would like to benchmark your Kubernetes setup. If you want to benchmark locally skip this step. -Note that the benchmarking script does not support the standalone deployment. -Install the STUNner Gateway operator and STUNner ([more info](https://github.com/l7mp/stunner-helm)): - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator -create-namespace --namespace=stunner-system -helm install stunner stunner/stunner -create-namespace --namespace=stunner -``` - Configure STUNner to act as a STUN server towards [`turncat`](../../cmd/turncat.md) clients, and to let `iperf` client's traffic reach the `iperf` server. ``` @@ -176,10 +161,7 @@ Results Notice that the average packets/second rate will be slightly lower in case of a hosted Kubernetes cluster than in case of a local `STUNner` installation. -## Tips and Tricks - -* It is advised to repeat the measurment with different packet sizes. - -Recommended packet sizes in bytes are 64, 128, 256, 512, 1024, and 1200. +## Caveats -**Effect of packet sizes:** With smallish packets (e.g., 64B), the average packets/second rate will be higher than with largish packets (e.g., 1200B). Small packet sizes result lower effective throughput (when packet drop is < 1%). You should definitely change the arguments to test the performance of your setup ideally. \ No newline at end of file +* It is advised to repeat the measurment with different packet sizes. Recommended packet sizes in bytes are 64, 128, 256, 512, 1024, and 1200. Small packet sizes result lower effective throughput (when packet drop is < 1%). +* Measuring [measuring one-way latency](https://stackoverflow.com/questions/63793030/iperf2-latency-is-a-two-way-or-one-way-latency) with `iperf` requires the clocks at the iperf client and server to be synchronized. Without this the results may be corrupted, and you may even see negative latencies. diff --git a/docs/examples/benchmark/benchmark.sh b/docs/examples/benchmark/benchmark.sh index fa61c67a..cb74aceb 100755 --- a/docs/examples/benchmark/benchmark.sh +++ b/docs/examples/benchmark/benchmark.sh @@ -72,16 +72,16 @@ if [[ $platform == "local" ]]; then UDP_ECHO_IP="127.0.0.1" IPERF_PORT="5000" - go run ../../cmd/stunnerd/main.go --log=all:INFO \ + go run ../../../cmd/stunnerd/main.go --log=all:INFO \ turn://${STUNNER_USERNAME}:${STUNNER_PASSWORD}@${STUNNER_PUBLIC_ADDR}:${STUNNER_PUBLIC_PORT} &> /dev/null 2>&1 & iperf -s -p 5000 -u -e > log.tmp 2>&1 & sleep 2 elif [[ $platform == "k8s" ]]; then - STUNNER_PUBLIC_ADDR=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .listeners[0].public_address) - STUNNER_PUBLIC_PORT=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .listeners[0].public_port) - STUNNER_PASSWORD=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .auth.credentials.password) - STUNNER_USERNAME=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .auth.credentials.username) + # STUNNER_PUBLIC_ADDR=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .listeners[0].public_address) + # STUNNER_PUBLIC_PORT=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .listeners[0].public_port) + # STUNNER_PASSWORD=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .auth.credentials.password) + # STUNNER_USERNAME=$(kubectl get cm stunnerd-config -n stunner -o jsonpath='{.data.stunnerd\.conf}' | jq -r .auth.credentials.username) UDP_ECHO_IP=$(kubectl get svc iperf-server -o jsonpath='{.spec.clusterIP}') IPERF_PORT="5000" else @@ -92,8 +92,8 @@ fi for i in $(seq "$num_of_processes"); do port=$((8999+i)) - go run ../../cmd/turncat/main.go --log=all:INFO udp://127.0.0.1:$port \ - turn://"${STUNNER_USERNAME}":"${STUNNER_PASSWORD}"@"${STUNNER_PUBLIC_ADDR}":"${STUNNER_PUBLIC_PORT}" udp://"${UDP_ECHO_IP}":$IPERF_PORT &> /dev/null 2>&1 & + go run ../../../cmd/turncat/main.go --log=all:INFO udp://127.0.0.1:$port \ + k8s://stunner/udp-gateway:udp-listener udp://"${UDP_ECHO_IP}":$IPERF_PORT >/dev/null 2>&1 & done sleep 2 diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index 265f09a2..491d5196 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -283,20 +283,19 @@ NAME AGE udproute.stunner.l7mp.io/stunner-headless 2m50s ``` -You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. For +You can also use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. For instance, below is the human-readable config of the Gateway called `udp-gateway`: ``` console -cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener 1 - Name: stunner/udp-gateway/udp-listener - Listener: stunner/udp-gateway/udp-listener - Protocol: TURN-UDP - Public address: 34.116.255.91 - Public port: 3478 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] ``` ### Run the test @@ -389,7 +388,7 @@ WebRTC service and many things can go wrong. Below is a list of steps to help de * No ICE candidate appears: Most probably this occurs because the browser's ICE configuration does not match the running STUNner config. Check that the ICE configuration returned by the application server in the `registerResponse` message matches the output of `stunnerctl - running-config`. Examine the `stunner` pods' logs (`kubectl logs...`): permission-denied messages + config`. Examine the `stunner` pods' logs (`kubectl logs...`): permission-denied messages typically indicate that STUN/TURN authentication was unsuccessful. * No video-connection: This is most probably due to a communication issue between your client and STUNner. Try disabling STUNner's UDP Gateway and force the browser to use TCP. diff --git a/docs/examples/kurento-one2one-call/README.md b/docs/examples/kurento-one2one-call/README.md index cf7fc06b..4068791b 100644 --- a/docs/examples/kurento-one2one-call/README.md +++ b/docs/examples/kurento-one2one-call/README.md @@ -293,7 +293,7 @@ Below is the corresponding UDPRoute. apiVersion: stunner.l7mp.io/v1 kind: UDPRoute metadata: - name: stunner-headless + name: kms-media-plane namespace: stunner spec: parentRefs: @@ -319,19 +319,18 @@ NAME AGE udproute.stunner.l7mp.io/kms-media-plane 84s ``` -You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. +You can also use the handy CLI tool called [`stunnerctl`](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. ```console -cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener 1 - Name: stunner/udp-gateway/udp-listener - Listener: stunner/udp-gateway/udp-listener - Protocol: TURN-UDP - Public address: 34.118.112.176 - Public port: 3478 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.112.176:3478 + Routes: [stunner/kms-media-plane] + Endpoints: [10.76.1.4, 10.80.4.47] ``` ### Run the test @@ -442,14 +441,13 @@ applications with STUNner. ## Update STUN/TURN credentials -As exemplified by `stunnerctl` output, STUNner currently runs with fairly poor security: using +As shown in the `stunnerctl` output, STUNner currently runs with fairly poor security: using `static` authentication, sharing a single username/password pair between all active sessions. ``` console -cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 ... ``` @@ -458,8 +456,8 @@ credentials on the client side for potentially nefarious purposes. Note that att be able to make too much harm with these credentials, since the only Kubernetes service they can reach via STUNner is the Kurento media server pool. This is why we have installed the UDPRoute: STUNner will allow clients to connect *only* to the backend service(s) of the UDPRoute, and nothing -else. Then, the attackers would need access to the application-server to open WebRTC endpoints on -the media server for their own purposes, but application servers should be secure by default no? +else. Then, attackers would need access to the application-server to open WebRTC endpoints on the +media server for their own purposes, but media servers should be secure by default no? In other words, *STUNner's default security model is exactly the same as if we put the application servers and media servers on public-facing physical servers*. @@ -468,7 +466,7 @@ Still, it would be nice to use per-session passwords. STUNner allows you to do t the authentication type to `ephemeral` instead of `static`. Even better: STUNner's ephemeral TURN credentials are valid only for a specified time (one day by default, but you can override this via the [authentication service](https://github.com/l7mp/stunner-auth-service)), after which they -expire and attackers can no longer reuse them. And to make things even better we don't even have to +expire and attackers can no longer reuse them. To make things even better, we don't even have to work too much to switch STUNner to the `ephemeral` authentication mode: it is enough to update the GatewayConfig and everything should happen from this point automagically. @@ -496,15 +494,15 @@ this goes without having to restart the server. Check that the running config indeed is updated correctly. ```console -cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway -STUN/TURN authentication type: ephemeral -STUN/TURN secret: my-very-secure-secret -Listener 1 - Name: stunner/udp-gateway/udp-listener - Listener: stunner/udp-gateway/udp-listener - Protocol: TURN-UDP - Public address: 34.118.112.176 - Public port: 3478 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: ephemeral, shared-secret: my-very-secure-secret +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] ``` Reload the browser client and re-register: you should see an updated ICE configuration with the @@ -524,7 +522,7 @@ new, per-session STUN/TURN credentials. ``` Ephemeral credentials expire in one day, after which they are either refreshed (e.g., by forcing -the users to re-register) or become useless. See more in the STUNner [authentication +the users to re-register) or become useless. See more on this in the STUNner [authentication guide](/doc/AUTH.md). ## Clean up diff --git a/docs/examples/livekit/README.md b/docs/examples/livekit/README.md index 41a3c857..60a3468e 100644 --- a/docs/examples/livekit/README.md +++ b/docs/examples/livekit/README.md @@ -168,20 +168,18 @@ rtc: port: 3478 ``` -This will make sure that LiveKit is started with STUNner as the STUN/TURN server. If unsure about the STUNner settings to use, you can always use the handy `stunnerctl` CLI tool to dump the running STUNner configuration. +This will make sure that LiveKit is started with STUNner as the STUN/TURN server. If unsure about the STUNner settings to use, you can always use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. ``` console -cd stunner -cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config -STUN/TURN authentication type: plaintext -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener 1 - Name: udp-listener - Listener: udp-listener - Protocol: UDP - Public address: 1.2.3.4 - Public port: 3478 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] ``` Note that LiveKit itself will not use STUNner (that would amount to a less efficient [symmetric ICE mode](../../DEPLOYMENT.md)); with the above configuration we are just telling LiveKit to instruct its clients to use STUNner to reach the LiveKit media servers. diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index da47f13a..a533fbfc 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -123,35 +123,33 @@ NAME AGE udproute.stunner.l7mp.io/iperf-server 139m ``` -You can also use the handy `stunnerctl` CLI tool to dump the running STUNner configuration for the +You can also use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration for the UDP gateway. ``` console -cmd/stunnerctl/stunnerctl running-config stunner/udp-gateway -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener 1 - Name: stunner/udp-gateway/udp-listener - Listener: stunner/udp-gateway/udp-listener - Protocol: TURN-UDP - Public address: 35.205.10.190 - Public port: 3478 +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] ``` Likewise, the below will dump the config for the TCP gateway. ``` console -cmd/stunnerctl/stunnerctl running-config stunner/tcp-gateway -STUN/TURN authentication type: static -STUN/TURN username: user-1 -STUN/TURN password: pass-1 -Listener 1 - Name: stunner/tcp-gateway/tcp-listener - Listener: stunner/tcp-gateway/tcp-listener - Protocol: TURN-TCP - Public address: 35.187.97.94 - Public port: 3478 +stunnerctl -n stunner config tcp-gateway +Gateway: stunner/tcp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 +Listeners: + - Name: stunner/tcp-gateway/tcp-listener + Protocol: TURN-TCP + Public address:port: 34.116.180.89:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] ``` NOTE: It usually takes 30-60 seconds for Kubernetes to assign an external IP address to STUNner diff --git a/go.mod b/go.mod index 362e35d5..3265572d 100644 --- a/go.mod +++ b/go.mod @@ -1,15 +1,17 @@ module github.com/l7mp/stunner -go 1.19 +go 1.21 + +toolchain go1.21.5 require ( github.com/deepmap/oapi-codegen/v2 v2.0.0 github.com/fsnotify/fsnotify v1.7.0 - github.com/getkin/kin-openapi v0.122.0 + github.com/getkin/kin-openapi v0.123.0 github.com/go-logr/logr v1.3.0 github.com/go-logr/zapr v1.3.0 github.com/google/uuid v1.5.0 - github.com/gorilla/mux v1.8.0 + github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.1 github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb github.com/oapi-codegen/runtime v1.1.1 @@ -24,70 +26,83 @@ require ( go.uber.org/zap v1.26.0 golang.org/x/sys v0.16.0 golang.org/x/time v0.5.0 - k8s.io/api v0.24.3 - k8s.io/apimachinery v0.24.3 - k8s.io/client-go v0.24.2 + k8s.io/api v0.29.1 + k8s.io/apimachinery v0.29.1 + k8s.io/cli-runtime v0.29.1 + k8s.io/client-go v0.29.1 k8s.io/utils v0.0.0-20240102154912-e7106e64919e sigs.k8s.io/controller-runtime v0.12.3 sigs.k8s.io/yaml v1.4.0 ) require ( - github.com/PuerkitoBio/purell v1.1.1 // indirect - github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect + github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect - github.com/emicklei/go-restful v2.9.5+incompatible // indirect + github.com/emicklei/go-restful/v3 v3.11.0 // indirect github.com/evanphx/json-patch v4.12.0+incompatible // indirect + github.com/go-errors/errors v1.4.2 // indirect github.com/go-openapi/jsonpointer v0.20.2 // indirect - github.com/go-openapi/jsonreference v0.19.5 // indirect - github.com/go-openapi/swag v0.22.7 // indirect + github.com/go-openapi/jsonreference v0.20.2 // indirect + github.com/go-openapi/swag v0.22.8 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/golang/protobuf v1.5.3 // indirect - github.com/google/gnostic v0.5.7-v3refs // indirect - github.com/google/go-cmp v0.6.0 // indirect - github.com/google/gofuzz v1.1.0 // indirect + github.com/google/btree v1.0.1 // indirect + github.com/google/gnostic-models v0.6.8 // indirect + github.com/google/gofuzz v1.2.0 // indirect + github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect + github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect github.com/imdario/mergo v0.3.12 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/invopop/yaml v0.2.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect + github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect github.com/mailru/easyjson v0.7.7 // indirect + github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect github.com/moby/spdystream v0.2.0 // indirect + github.com/moby/term v0.0.0-20221205130635-1aeaba878587 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect + github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/onsi/gomega v1.30.0 // indirect + github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect github.com/perimeterx/marshmallow v1.1.5 // indirect + github.com/peterbourgon/diskv v2.0.1+incompatible // indirect github.com/pion/randutil v0.1.0 // indirect github.com/pion/stun/v2 v2.0.0 // indirect - github.com/pion/transport/v2 v2.2.4 // indirect + github.com/pion/transport/v2 v2.2.1 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.5.0 // indirect - github.com/prometheus/common v0.46.0 // indirect + github.com/prometheus/common v0.45.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect + github.com/xlab/treeprint v1.2.0 // indirect + go.starlark.net v0.0.0-20230525235612-a134d8f9ddca // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.18.0 // indirect + golang.org/x/crypto v0.17.0 // indirect golang.org/x/mod v0.14.0 // indirect - golang.org/x/net v0.20.0 // indirect - golang.org/x/oauth2 v0.16.0 // indirect - golang.org/x/term v0.16.0 // indirect + golang.org/x/net v0.19.0 // indirect + golang.org/x/oauth2 v0.12.0 // indirect + golang.org/x/sync v0.5.0 // indirect + golang.org/x/term v0.15.0 // indirect golang.org/x/text v0.14.0 // indirect - golang.org/x/tools v0.17.0 // indirect + golang.org/x/tools v0.16.1 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/protobuf v1.32.0 // indirect + google.golang.org/protobuf v1.31.0 // indirect gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect - k8s.io/klog/v2 v2.80.1 // indirect - k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 // indirect - sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect - sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect + k8s.io/klog/v2 v2.110.1 // indirect + k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00 // indirect + sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect + sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3 // indirect + sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3 // indirect + sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect ) // replace github.com/pion/turn/v2 => github.com/l7mp/turn/v2 v2.0.11 diff --git a/go.sum b/go.sum index ed10a42e..361fc1a7 100644 --- a/go.sum +++ b/go.sum @@ -1,61 +1,12 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= -cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= -cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= -cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= -cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= -cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To= -cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4= -cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M= -cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc= -cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk= -cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs= -cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc= -cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY= -cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI= -cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk= -cloud.google.com/go v0.78.0/go.mod h1:QjdrLG0uq+YwhjoVOLsS1t7TW8fs36kLs4XO5R5ECHg= -cloud.google.com/go v0.79.0/go.mod h1:3bzgcEeQlzbuEAYu4mrWhKqWjmpprinYgKJLgKHnbb8= -cloud.google.com/go v0.81.0/go.mod h1:mk/AM35KwGk/Nm2YSeZbxXdrNK3KZOYHmLkOqC2V6E0= -cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= -cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= -cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= -cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg= -cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc= -cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ= -cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= -cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= -cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= -cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= -cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= -cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU= -cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= -cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= -cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= -cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs= -cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0= -dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= -github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24= -github.com/Azure/go-autorest/autorest v0.11.18/go.mod h1:dSiJPy22c3u0OtOKDNttNgqpNFY/GeWa7GH/Pz56QRA= -github.com/Azure/go-autorest/autorest/adal v0.9.13/go.mod h1:W/MM4U6nLxnIskrw4UwWzlHfGjwUS50aOsc/I3yuU8M= -github.com/Azure/go-autorest/autorest/date v0.3.0/go.mod h1:BI0uouVdmngYNUzGWeSYnokU+TrmwEsOqdt8Y6sso74= -github.com/Azure/go-autorest/autorest/mocks v0.4.1/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k= -github.com/Azure/go-autorest/logger v0.2.1/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8= -github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU= +github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8= +github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= -github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= -github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= -github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= -github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= -github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= -github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= -github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bmatcuk/doublestar v1.1.1/go.mod h1:UD6OnuiIn0yFxxA2le/rnRU1G4RaI4UvFv1sNto9p6w= @@ -66,152 +17,91 @@ github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWR github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= -github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= -github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= -github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY= +github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/deepmap/oapi-codegen/v2 v2.0.0 h1:3TS7w3r+XnjKFXcbFbc16pTWzfTy0OLPkCsutEHjWDA= github.com/deepmap/oapi-codegen/v2 v2.0.0/go.mod h1:7zR+ZL3WzLeCkr2k8oWTxEa0v8y/F25ane0l6A5UjLA= -github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= -github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 h1:yUdfgN0XgIJw7foRItutHYUIhlcKzcSf5vDpdhQAKTc= -github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= -github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= -github.com/emicklei/go-restful v2.9.5+incompatible h1:spTtZBk5DYEvbxMVutUuTyh1Ao2r4iyvLdACqsl/Ljk= -github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= -github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= +github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= -github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= -github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po= -github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84= github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= -github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k= -github.com/form3tech-oss/jwt-go v3.2.3+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k= -github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= -github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= -github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg= -github.com/getkin/kin-openapi v0.122.0 h1:WB9Jbl0Hp/T79/JF9xlSW5Kl9uYdk/AWD0yAd9HOM10= -github.com/getkin/kin-openapi v0.122.0/go.mod h1:PCWw/lfBrJY4HcdqE3jj+QFkaFK8ABoqo7PvqVhXXqw= -github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= -github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= -github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= -github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= -github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= -github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= -github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mNrPhs8= +github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM= +github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA= +github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= github.com/go-logr/logr v1.3.0 h1:2y3SDp0ZXuc6/cjLSZ+Q3ir+QB9T/iG5yYRXqsagWSY= github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= -github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q= github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs= -github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8= -github.com/go-openapi/jsonreference v0.19.5 h1:1WJP/wi4OjB4iV8KVbH73rQaoialJrqv8gitZLxGLtM= -github.com/go-openapi/jsonreference v0.19.5/go.mod h1:RdybgQwPxbL4UEjuAruzK1x3nE69AqPYEJeo/TWfEeg= -github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= -github.com/go-openapi/swag v0.19.14/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= -github.com/go-openapi/swag v0.22.7 h1:JWrc1uc/P9cSomxfnsFSVWoE1FW6bNbrVPmpQYpCcR8= -github.com/go-openapi/swag v0.22.7/go.mod h1:Gl91UqO+btAM0plGGxHqJcQZ1ZTy6jbmridBTsDy8A0= +github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= +github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= +github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= +github.com/go-openapi/swag v0.22.8 h1:/9RjDSQ0vbFR+NyjGMkFTsA1IA0fmhKSThmfGZjicbw= +github.com/go-openapi/swag v0.22.8/go.mod h1:6QT22icPLEqAM/z/TChgb4WAveCHF92+2gF0CNjHpPI= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= +github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= -github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= -github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= -github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= -github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= -github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= -github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= -github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4= -github.com/golang/mock v1.5.0/go.mod h1:CWnOUgYIOo4TcNZ0wHX3YZCqsaM1I1Jvs6v3mP3KVu8= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= -github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= -github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk= github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= -github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= -github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.1/go.mod h1:DopwsBzvsk0Fs44TXzsVbJyPhcCPeIwnvohx4u74HPM= -github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= -github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= -github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4= github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA= -github.com/google/gnostic v0.5.7-v3refs h1:FhTMOKj2VhjpouxvWJAV1TL304uMlb9zcDqkl6cEI54= -github.com/google/gnostic v0.5.7-v3refs/go.mod h1:73MKFl6jIHelAJNaBGFzt3SPtZULs9dYrGFt8OiIsHQ= +github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= +github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= -github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= -github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= -github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= -github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= -github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= -github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= -github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= +github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/pprof v0.0.0-20211214055906-6f57359322fd h1:1FjCyPC+syAzJ5/2S8fqdZK1R22vvA0J7JZKcuOIQ7Y= +github.com/google/pprof v0.0.0-20211214055906-6f57359322fd/go.mod h1:KgnwoLYCZ8IQu3XUZ8Nc/bM9CCZFOyjUNOSygVozoDg= +github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4= +github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU= github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= -github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= -github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI= -github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= +github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= +github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY= github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY= +github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= -github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= -github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb h1:tsEKRC3PU9rMw18w/uAptoijhgG4EvlA5kfJPtwrMDk= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb/go.mod h1:NtmN9h8vrTveVQRLHcX2HQ5wIPBDCsZ351TGbZWgg38= -github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= -github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= -github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= -github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= @@ -220,58 +110,54 @@ github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY= github.com/invopop/yaml v0.2.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= -github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= -github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= -github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= -github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0= +github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= -github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg= +github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k= github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8= github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= +github.com/moby/term v0.0.0-20221205130635-1aeaba878587 h1:HfkjXDfhgVaN5rmueG8cL8KKeFNecRCXFhaJ2qZ5SKA= +github.com/moby/term v0.0.0-20221205130635-1aeaba878587/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 h1:RWengNIwukTxcDr9M+97sNutRR1RKhG96O6jWumTTnw= github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8= -github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/2gBQ3RWajuToeY6ZtZTIKv2v7ThUy5KKusIT0yc0= +github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= -github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= -github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= +github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU= github.com/oapi-codegen/runtime v1.1.1 h1:EXLHh0DXIJnWhdRPN2w4MXAzFyE4CskzhNLUmtpMYro= github.com/oapi-codegen/runtime v1.1.1/go.mod h1:SK9X900oXmPWilYR5/WKPzt3Kqxn/uS/+lbpREv+eCg= -github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= -github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= -github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= -github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= -github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= -github.com/onsi/gomega v1.30.0 h1:hvMK7xYz4D3HapigLTeGdId/NcfQx1VHMJc60ew99+8= -github.com/onsi/gomega v1.30.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= +github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU= +github.com/onsi/ginkgo/v2 v2.13.0 h1:0jY9lJquiL8fcf3M4LAXN5aMlS/b2BV86HFFPCPMgE4= +github.com/onsi/ginkgo/v2 v2.13.0/go.mod h1:TE309ZR8s5FsKKpuB1YAQYBzCaAfUgatB/xlT/ETL/o= +github.com/onsi/gomega v1.29.0 h1:KIA/t2t5UBzoirT4H9tsML45GEbo3ouUnBHsCfD2tVg= +github.com/onsi/gomega v1.29.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s= github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= +github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= github.com/pion/dtls/v2 v2.2.9 h1:K+D/aVf9/REahQvqk6G5JavdrD8W1PWDKC11UlwN7ts= @@ -282,9 +168,8 @@ github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8= github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= +github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= -github.com/pion/transport/v2 v2.2.4 h1:41JJK6DZQYSeVLxILA2+F4ZkKb4Xd/tFJZRFZQ9QAlo= -github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= github.com/pion/turn/v3 v3.0.1 h1:wLi7BTQr6/Q20R0vt/lHbjv6y4GChFtC33nkYbasoT8= @@ -298,223 +183,105 @@ github.com/prometheus/client_golang v1.18.0/go.mod h1:T+GXkCk5wSJyOqMIzVgvvjFDlk github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= -github.com/prometheus/common v0.46.0 h1:doXzt5ybi1HBKpsZOL0sSkaNHJJqkyfEWZGGqqScV0Y= -github.com/prometheus/common v0.46.0/go.mod h1:Tp0qkxpb9Jsg54QMe+EAmqXkSV7Evdy1BTn+g2pa/hQ= +github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM= +github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY= github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= -github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= -github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= +github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= -github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= +github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= +github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= -github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= -github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= -github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= -github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU= -github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg= +github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= +github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= -go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= -go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= -go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= -go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= -go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= -go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E= +go.starlark.net v0.0.0-20230525235612-a134d8f9ddca h1:VdD38733bfYv5tUZwEIskMM93VanwNIi5bIKnDrJdEY= +go.starlark.net v0.0.0-20230525235612-a134d8f9ddca/go.mod h1:jxU+3+j+71eXOW14274+SmmuW82qJzl6iZSeqEtTGds= go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= +go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo= go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= -golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= +golang.org/x/crypto v0.17.0 h1:r8bRNjWL3GshPW3gkd+RpvzWrZAwPS49OmTGZ/uhM4k= +golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= -golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= -golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= -golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= -golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= -golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= -golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= -golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= -golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= -golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= -golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= -golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= -golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= -golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= -golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= -golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= -golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= -golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= -golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= -golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= -golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= -golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc= -golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= -golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo= -golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= +golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c= +golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= -golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.16.0 h1:aDkGMBSYxElaoP81NpoUoz2oo2R2wHdZpGToUxfyQrQ= -golang.org/x/oauth2 v0.16.0/go.mod h1:hqZ+0LWXsiVoZpeld6jVt06P3adbS2Uu911W1SsJv2o= +golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4= +golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE= +golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210220050731-9a76102bfb43/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -524,257 +291,108 @@ golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= -golang.org/x/term v0.16.0 h1:m+B6fahuftsE9qjo0VWp2FW0mB3MTJvR0BaMQrq0pmE= -golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= -golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/term v0.15.0 h1:y/Oo/a/q3IXu26lQgl04j/gjuBDOBlx7X6Om1j2CPW4= +golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20220210224613-90d013bbcef8/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= -golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= -golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= -golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20200505023115-26f46d2f7ef8/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= -golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= -golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= -golang.org/x/tools v0.0.0-20200904185747-39188db58858/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE= -golang.org/x/tools v0.0.0-20201110124207-079ba7bd75cd/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20201201161351-ac6f37ff4c2a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20201208233053-a543418bbed2/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20210105154028-b0ab187a4818/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= -golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc= -golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps= +golang.org/x/tools v0.16.1 h1:TLyB3WofjdOEepBHAU20JdNC1Zbg87elYofWYAY5oZA= +golang.org/x/tools v0.16.1/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gomodules.xyz/jsonpatch/v2 v2.2.0 h1:4pT439QV83L+G9FkcCriY6EkpcK6r6bK+A5FBUMI7qY= -google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= -google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= -google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= -google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= -google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= -google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= -google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= -google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= -google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= -google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= -google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= -google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= -google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= -google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= -google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM= -google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc= -google.golang.org/api v0.35.0/go.mod h1:/XrVsuzM0rZmrsbjJutiuftIzeuTQcEeaYcSk/mQ1dg= -google.golang.org/api v0.36.0/go.mod h1:+z5ficQTmoYpPn8LCUNVpK5I7hwkpjbcgqA7I34qYtE= -google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjRCQ8= -google.golang.org/api v0.41.0/go.mod h1:RkxM5lITDfTzmyKFPt+wGrCJbVfniCr2ool8kTBzRTU= -google.golang.org/api v0.43.0/go.mod h1:nQsDGjRXMo4lvh5hP0TKqF244gqhGcr/YSIykhUk/94= +gomodules.xyz/jsonpatch/v2 v2.2.0/go.mod h1:WXp+iVDkoLQqPudfQ9GBlwB2eZ5DKOnjQZCYdOS8GPY= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= -google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= -google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0= -google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= -google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= -google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= -google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= -google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= -google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= -google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= -google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= -google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= -google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= -google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA= -google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= -google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA= -google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201019141844-1ed22bb0c154/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201109203340-2640f1f9cdfb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201201144952-b05cb90ed32e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210222152913-aa3ee6e6a81c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210303154014-9728d6b83eeb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210310155132-4ce2db91004e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210319143718-93e7006c17a6/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210402141018-6c239bbf2bb1/go.mod h1:9lPAdzaEmUacj36I+k7YKbEc5CXzPIeORRgDAUOu28A= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= -google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= -google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= -google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= -google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= -google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= -google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= -google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk= -google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= -google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= -google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= -google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= -google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8= -google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= -google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= -google.golang.org/grpc v1.36.1/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= -google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= -google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I= -google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 h1:FVCohIoYO7IJoDDVpV2pdq7SgrMH6wHnuTyrdrxJNoY= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0/go.mod h1:OdE7CF6DbADk7lN8LIKRzRJTTZXIjtWgA5THM5lhBAw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= -gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= -gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= -gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= -honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= -honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= -k8s.io/api v0.24.2/go.mod h1:AHqbSkTm6YrQ0ObxjO3Pmp/ubFF/KuM7jU+3khoBsOg= -k8s.io/api v0.24.3 h1:tt55QEmKd6L2k5DP6G/ZzdMQKvG5ro4H4teClqm0sTY= -k8s.io/api v0.24.3/go.mod h1:elGR/XSZrS7z7cSZPzVWaycpJuGIw57j9b95/1PdJNI= +k8s.io/api v0.29.1 h1:DAjwWX/9YT7NQD4INu49ROJuZAAAP/Ijki48GUPzxqw= +k8s.io/api v0.29.1/go.mod h1:7Kl10vBRUXhnQQI8YR/R327zXC8eJ7887/+Ybta+RoQ= k8s.io/apiextensions-apiserver v0.24.2 h1:/4NEQHKlEz1MlaK/wHT5KMKC9UKYz6NZz6JE6ov4G6k= -k8s.io/apimachinery v0.24.2/go.mod h1:82Bi4sCzVBdpYjyI4jY6aHX+YCUchUIrZrXKedjd2UM= -k8s.io/apimachinery v0.24.3 h1:hrFiNSA2cBZqllakVYyH/VyEh4B581bQRmqATJSeQTg= -k8s.io/apimachinery v0.24.3/go.mod h1:82Bi4sCzVBdpYjyI4jY6aHX+YCUchUIrZrXKedjd2UM= -k8s.io/client-go v0.24.2 h1:CoXFSf8if+bLEbinDqN9ePIDGzcLtqhfd6jpfnwGOFA= -k8s.io/client-go v0.24.2/go.mod h1:zg4Xaoo+umDsfCWr4fCnmLEtQXyCNXCvJuSsglNcV30= -k8s.io/gengo v0.0.0-20210813121822-485abfe95c7c/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E= -k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= -k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= -k8s.io/klog/v2 v2.60.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= -k8s.io/klog/v2 v2.80.1 h1:atnLQ121W371wYYFawwYx1aEY2eUfs4l3J72wtgAwV4= -k8s.io/klog/v2 v2.80.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= -k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 h1:Gii5eqf+GmIEwGNKQYQClCayuJCe2/4fZUvF7VG99sU= -k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42/go.mod h1:Z/45zLw8lUo4wdiUkI+v/ImEGAvu3WatcZl3lPMR4Rk= -k8s.io/utils v0.0.0-20210802155522-efc7438f0176/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= -k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +k8s.io/apiextensions-apiserver v0.24.2/go.mod h1:e5t2GMFVngUEHUd0wuCJzw8YDwZoqZfJiGOW6mm2hLQ= +k8s.io/apimachinery v0.29.1 h1:KY4/E6km/wLBguvCZv8cKTeOwwOBqFNjwJIdMkMbbRc= +k8s.io/apimachinery v0.29.1/go.mod h1:6HVkd1FwxIagpYrHSwJlQqZI3G9LfYWRPAkUvLnXTKU= +k8s.io/cli-runtime v0.29.1 h1:By3WVOlEWYfyxhGko0f/IuAOLQcbBSMzwSaDren2JUs= +k8s.io/cli-runtime v0.29.1/go.mod h1:vjEY9slFp8j8UoMhV5AlO8uulX9xk6ogfIesHobyBDU= +k8s.io/client-go v0.29.1 h1:19B/+2NGEwnFLzt0uB5kNJnfTsbV8w6TgQRz9l7ti7A= +k8s.io/client-go v0.29.1/go.mod h1:TDG/psL9hdet0TI9mGyHJSgRkW3H9JZk2dNEUS7bRks= +k8s.io/klog/v2 v2.110.1 h1:U/Af64HJf7FcwMcXyKm2RPM22WZzyR7OSpYj5tg3cL0= +k8s.io/klog/v2 v2.110.1/go.mod h1:YGtd1984u+GgbuZ7e08/yBuAfKLSO0+uR1Fhi6ExXjo= +k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00 h1:aVUu9fTY98ivBPKR9Y5w/AuzbMm96cd3YHRTU83I780= +k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00/go.mod h1:AsvuZPBlUDVuCdzJ87iajxtXuR9oktsTctW/R9wwouA= k8s.io/utils v0.0.0-20240102154912-e7106e64919e h1:eQ/4ljkx21sObifjzXwlPKpdGLrCfRziVtos3ofG/sQ= k8s.io/utils v0.0.0-20240102154912-e7106e64919e/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= -rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= -rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= -rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= sigs.k8s.io/controller-runtime v0.12.3 h1:FCM8xeY/FI8hoAfh/V4XbbYMY20gElh9yh+A98usMio= sigs.k8s.io/controller-runtime v0.12.3/go.mod h1:qKsk4WE6zW2Hfj0G4v10EnNB2jMG1C+NTb8h+DwCoU0= -sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 h1:kDi4JBNAsJWfz1aEXhO8Jg87JJaPNLh5tIzYHgStQ9Y= -sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2/go.mod h1:B+TnT182UBxE84DiCz4CVE26eOSDAeYCpfDnC2kdKMY= -sigs.k8s.io/structured-merge-diff/v4 v4.0.2/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= -sigs.k8s.io/structured-merge-diff/v4 v4.2.1 h1:bKCqE9GvQ5tiVHn5rfn1r+yao3aLQEaLzkkmAkf+A6Y= -sigs.k8s.io/structured-merge-diff/v4 v4.2.1/go.mod h1:j/nl6xW8vLS49O8YvXW1ocPhZawJtm+Yrr7PPRQ0Vg4= -sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= +sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= +sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= +sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3 h1:XX3Ajgzov2RKUdc5jW3t5jwY7Bo7dcRm+tFxT+NfgY0= +sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3/go.mod h1:9n16EZKMhXBNSiUC5kSdFQJkdH3zbxS/JoO619G1VAY= +sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3 h1:W6cLQc5pnqM7vh3b7HvGNfXrJ/xL6BDMS0v1V/HHg5U= +sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3/go.mod h1:JWP1Fj0VWGHyw3YUPjXSQnRnrwezrZSrApfX5S0nIag= +sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= +sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08= sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E= sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY= diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 60712741..0a2940cb 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -23,3 +23,5 @@ const DefaultEnvVarName = "STUNNER_NAME" const DefaultEnvVarNamespace = "STUNNER_NAMESPACE" const DefaultEnvVarNodeName = "STUNNER_NODENAME" const DefaultEnvVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" +const DefaultCDSServiceLabelKey = "stunner.l7mp.io/config-discovery-service" +const DefaultCDSServiceLabelValue = "enabled" diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go index 824a402b..63f1b39f 100644 --- a/pkg/apis/v1/stunner.go +++ b/pkg/apis/v1/stunner.go @@ -124,6 +124,28 @@ func (req *StunnerConfig) DeepCopyInto(dst Config) { } } +// GetListenerConfig finds a Listener by name in a StunnerConfig or returns an error. +func (req *StunnerConfig) GetListenerConfig(name string) (ListenerConfig, error) { + for _, l := range req.Listeners { + if l.Name == name { + return l, nil + } + } + + return ListenerConfig{}, ErrNoSuchListener +} + +// GetClusterConfig finds a Cluster by name in a StunnerConfig or returns an error. +func (req *StunnerConfig) GetClusterConfig(name string) (ClusterConfig, error) { + for _, c := range req.Clusters { + if c.Name == name { + return c, nil + } + } + + return ClusterConfig{}, ErrNoSuchCluster +} + // String stringifies the configuration. func (req *StunnerConfig) String() string { status := []string{} @@ -146,24 +168,48 @@ func (req *StunnerConfig) String() string { return fmt.Sprintf("{%s}", strings.Join(status, ",")) } -// GetListenerConfig finds a Listener by name in a StunnerConfig or returns an error. -func (req *StunnerConfig) GetListenerConfig(name string) (ListenerConfig, error) { - for _, l := range req.Listeners { - if l.Name == name { - return l, nil +// Summary returns a stringified configuration. +func (req *StunnerConfig) Summary() string { + // isEnabled = func(b bool) string { if b {return "enabled"} else {return "disabled"}} + strOrNone := func(s string) string { + if s != "" { + return s + } else { + return "" + } + } + intOrNone := func(s int) string { + if s != 0 { + return fmt.Sprintf("%d", s) + } else { + return "" + } + } + status := fmt.Sprintf("Gateway: %s (loglevel: %q)\n", req.Admin.Name, req.Admin.LogLevel) + if t, err := NewAuthType(req.Auth.Type); err == nil { + if t == AuthTypeStatic { + status += fmt.Sprintf("Authentication type: static, username/password: %s/%s\n", + req.Auth.Credentials["username"], req.Auth.Credentials["password"]) + } else { + status += fmt.Sprintf("Authentication type: ephemeral, shared-secret: %s\n", + req.Auth.Credentials["secret"]) } } - return ListenerConfig{}, ErrNoSuchListener -} - -// GetClusterConfig finds a Cluster by name in a StunnerConfig or returns an error. -func (req *StunnerConfig) GetClusterConfig(name string) (ClusterConfig, error) { - for _, c := range req.Clusters { - if c.Name == name { - return c, nil + status += "Listeners:\n" + for _, l := range req.Listeners { + status += fmt.Sprintf(" - Name: %s\n", l.Name) + status += fmt.Sprintf(" Protocol: %s\n", l.Protocol) + status += fmt.Sprintf(" Public address:port: %s:%s\n", strOrNone(l.PublicAddr), intOrNone(l.PublicPort)) + status += fmt.Sprintf(" Routes: [%s]\n", strings.Join(l.Routes, ", ")) + ep := []string{} + for _, r := range l.Routes { + if c, err := req.GetClusterConfig(r); err == nil { + ep = append(ep, c.Endpoints...) + } } + status += fmt.Sprintf(" Endpoints: [%s]\n", strings.Join(ep, ", ")) } - return ClusterConfig{}, ErrNoSuchCluster + return status } diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go index 9960cf35..28871034 100644 --- a/pkg/config/client/cds_api.go +++ b/pkg/config/client/cds_api.go @@ -3,10 +3,8 @@ package client import ( "context" - "encoding/json" "fmt" "net/http" - "net/url" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/config/client/api" @@ -24,6 +22,9 @@ type ConfigList struct { Items []*stnrv1.StunnerConfig `json:"items"` } +type ClientOption = api.ClientOption +type HttpRequestDoer = api.HttpRequestDoer + type CdsApi interface { // Endpoint returns the address of the server plus the WebSocket API endpoint. Endpoint() (string, string) @@ -37,6 +38,8 @@ type CdsApi interface { logging.LeveledLogger } +func WithHTTPClient(doer HttpRequestDoer) ClientOption { return api.WithHTTPClient(doer) } + // AllConfigsAPI is the API for listing all configs in a namespace. type AllConfigsAPI struct { addr, httpURI, wsURI string @@ -44,7 +47,7 @@ type AllConfigsAPI struct { logging.LeveledLogger } -func NewAllConfigsAPI(addr string, logger logging.LeveledLogger) (CdsApi, error) { +func NewAllConfigsAPI(addr string, logger logging.LeveledLogger, opts ...ClientOption) (CdsApi, error) { httpuri, err := getURI(addr) if err != nil { return nil, err @@ -55,7 +58,7 @@ func NewAllConfigsAPI(addr string, logger logging.LeveledLogger) (CdsApi, error) return nil, err } - client, err := api.NewClientWithResponses(httpuri.String()) + client, err := api.NewClientWithResponses(httpuri.String(), opts...) if err != nil { return nil, err } @@ -105,7 +108,7 @@ type ConfigsNamespaceAPI struct { logging.LeveledLogger } -func NewConfigsNamespaceAPI(addr, namespace string, logger logging.LeveledLogger) (CdsApi, error) { +func NewConfigsNamespaceAPI(addr, namespace string, logger logging.LeveledLogger, opts ...ClientOption) (CdsApi, error) { httpuri, err := getURI(addr) if err != nil { return nil, err @@ -116,7 +119,7 @@ func NewConfigsNamespaceAPI(addr, namespace string, logger logging.LeveledLogger return nil, err } - client, err := api.NewClientWithResponses(httpuri.String()) + client, err := api.NewClientWithResponses(httpuri.String(), opts...) if err != nil { return nil, err } @@ -170,7 +173,7 @@ type ConfigNamespaceNameAPI struct { logging.LeveledLogger } -func NewConfigNamespaceNameAPI(addr, namespace, name string, logger logging.LeveledLogger) (CdsApi, error) { +func NewConfigNamespaceNameAPI(addr, namespace, name string, logger logging.LeveledLogger, opts ...ClientOption) (CdsApi, error) { httpuri, err := getURI(addr) if err != nil { return nil, err @@ -181,7 +184,7 @@ func NewConfigNamespaceNameAPI(addr, namespace, name string, logger logging.Leve return nil, err } - client, err := api.NewClientWithResponses(httpuri.String()) + client, err := api.NewClientWithResponses(httpuri.String(), opts...) if err != nil { return nil, err } @@ -230,52 +233,3 @@ func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- *stnrv1.Stu a.namespace, a.name, a.wsURI) return poll(ctx, a, ch) } - -func decodeConfig(r []byte) ([]*stnrv1.StunnerConfig, error) { - c := stnrv1.StunnerConfig{} - if err := json.Unmarshal(r, &c); err != nil { - return nil, err - } - - // copy - - return []*stnrv1.StunnerConfig{&c}, nil -} - -func decodeConfigList(r []byte) ([]*stnrv1.StunnerConfig, error) { - l := ConfigList{} - if err := json.Unmarshal(r, &l); err != nil { - return nil, err - } - return l.Items, nil -} - -// getURI tries to parse an address or an URL or a file name into an URL. -func getURI(addr string) (*url.URL, error) { - url, err := url.Parse(addr) - if err != nil { - // try to parse with a http scheme as a last resort - u, err2 := url.Parse("http://" + addr) - if err2 != nil { - return nil, err - } - url = u - } - return url, nil -} - -// wsURI returns a websocket url from a HTTP URI. -func wsURI(addr, endpoint string) (string, error) { - uri, err := getURI(addr) - if err != nil { - return "", err - } - - uri.Scheme = "ws" - uri.Path = endpoint - v := url.Values{} - v.Set("watch", "true") - uri.RawQuery = v.Encode() - - return uri.String(), nil -} diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go new file mode 100644 index 00000000..a903cb08 --- /dev/null +++ b/pkg/config/client/k8s_client.go @@ -0,0 +1,167 @@ +package client + +import ( + "bytes" + "context" + "fmt" + "net/http" + "os" + "strconv" + + "github.com/pion/logging" + "github.com/spf13/pflag" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + cliopt "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/tools/portforward" + "k8s.io/client-go/transport/spdy" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +const ( + DefaultCDSServerPort = 13478 + CDSServerAddrEnv = "CDS_SERVER_ADDR" + CDSServerNamespaceEnv = "CDS_SERVER_NAME" + CDSServerPortEnv = "CDS_SERVER_PORT" +) + +// CDSConfigFlags composes a set of flags for CDS server discovery +type CDSConfigFlags struct { + // ServerAddr is an explicit IP address for the CDS server. + ServerAddr string + // ServerNamespace is the namespace of the CDS server pod. + ServerNamespace string + // ServerPort is the port of the CDS server pod. + ServerPort int +} + +// NewCDSConfigFlags returns CDS service discovery flags with default values set. +func NewCDSConfigFlags() *CDSConfigFlags { + port := DefaultCDSServerPort + if os.Getenv(CDSServerPortEnv) != "" { + p, err := strconv.Atoi(os.Getenv(CDSServerPortEnv)) + if err != nil { + port = p + } + } + return &CDSConfigFlags{ + ServerAddr: os.Getenv(CDSServerAddrEnv), + ServerPort: port, + ServerNamespace: os.Getenv(CDSServerNamespaceEnv), + } +} + +// AddFlags binds CDS server discovery configuration flags to a given flagset. +func (f *CDSConfigFlags) AddFlags(flags *pflag.FlagSet) { + flags.StringVar(&f.ServerAddr, "cds-server-address", f.ServerAddr, + "Config discovery service address (overriders cds-namesapce/name and disables CDS service discovery)") + flags.StringVar(&f.ServerNamespace, "cds-server-namespace", f.ServerNamespace, + "Config discovery service namespace (disables CDS service discovery)") + flags.IntVar(&f.ServerPort, "cds-server-port", f.ServerPort, + "Config discovery service port") +} + +// DiscoverK8sCDSServer discovers a CDS Server located in a Kubernetes cluster and returns an +// address that a CDS client can be opened to for reaching that CDS server. If necessary, opens a +// port-forward connection to the remote cluster. +func DiscoverK8sCDSServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, cdsFlags *CDSConfigFlags, log logging.LeveledLogger) (string, error) { + // if CDS server address is specified, return it + if cdsFlags.ServerAddr != "" { + return cdsFlags.ServerAddr, nil + } + + ns := "" + nsLog := "" + if cdsFlags.ServerNamespace != "" { + ns = cdsFlags.ServerNamespace + nsLog = ns + } + + log.Debug("Obtaining kubeconfig") + config, err := k8sFlags.ToRESTConfig() + if err != nil { + return "", fmt.Errorf("error building Kubernetes config: %w", err) + } + + log.Debug("Creating a Kubernetes client") + cs, err := kubernetes.NewForConfig(config) + if err != nil { + return "", fmt.Errorf("error creating http client: %w", err) + } + + label := fmt.Sprintf("%s=%s", stnrv1.DefaultCDSServiceLabelKey, stnrv1.DefaultCDSServiceLabelValue) + log.Debugf("Querying CDS server pods in namespace %q using label-selector %q", nsLog, label) + pods, err := cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{ + LabelSelector: label, + }) + + if err != nil { + return "", err + } + + if len(pods.Items) == 0 { + return "", fmt.Errorf("no CDS server found") + } + + if len(pods.Items) > 1 { + return "", fmt.Errorf("too many CDS servers") + } + + name := pods.Items[0].GetName() + namespace := pods.Items[0].GetNamespace() + log.Debugf("Found CDS server: %s/%s", namespace, name) + req := cs.RESTClient(). + Post(). + Prefix("api/v1"). + Resource("pods"). + Namespace(namespace). + Name(name). + SubResource("portforward") + + log.Debugf("Creating a SPDY stream to API server using URL %q", req.URL().String()) + transport, upgrader, err := spdy.RoundTripperFor(config) + if err != nil { + return "", fmt.Errorf("error getting transport/upgrader from restconfig: %w", err) + } + + dialer := spdy.NewDialer(upgrader, &http.Client{Transport: transport}, http.MethodPost, req.URL()) + + log.Debug("Creating a port-forwarder to CDS server") + stopChan, readyChan := make(chan struct{}, 1), make(chan struct{}, 1) + out, errOut := new(bytes.Buffer), new(bytes.Buffer) + + fw, err := portforward.New(dialer, []string{fmt.Sprintf("0:%d", cdsFlags.ServerPort)}, stopChan, readyChan, out, errOut) + if err != nil { + return "", fmt.Errorf("error creating port-forwarder: %w", err) + } + + go func() { + if err := fw.ForwardPorts(); err != nil { + log.Errorf("error setting up port-forwarder: %s", err.Error()) + os.Exit(1) + } + }() + + log.Debug("Waiting for port-forwarder...") + <-readyChan + + localPort, err := fw.GetPorts() + if err != nil { + return "", fmt.Errorf("error obtaining local forwarder port: %w", err) + } + + if len(localPort) != 1 { + return "", fmt.Errorf("error setting up port-forwarder: required port pairs (1) "+ + "does not match the length of port forwarder port pairs (%d)", len(localPort)) + } + + go func() { + <-ctx.Done() + close(stopChan) + }() + + addr := fmt.Sprintf("127.0.0.1:%d", localPort[0].Local) + log.Debugf("CDS server reachable on address %q", addr) + return addr, nil +} diff --git a/pkg/config/client/util.go b/pkg/config/client/util.go new file mode 100644 index 00000000..666b81a2 --- /dev/null +++ b/pkg/config/client/util.go @@ -0,0 +1,57 @@ +package client + +import ( + "encoding/json" + "net/url" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +func decodeConfig(r []byte) ([]*stnrv1.StunnerConfig, error) { + c := stnrv1.StunnerConfig{} + if err := json.Unmarshal(r, &c); err != nil { + return nil, err + } + + // copy + + return []*stnrv1.StunnerConfig{&c}, nil +} + +func decodeConfigList(r []byte) ([]*stnrv1.StunnerConfig, error) { + l := ConfigList{} + if err := json.Unmarshal(r, &l); err != nil { + return nil, err + } + return l.Items, nil +} + +// getURI tries to parse an address or an URL or a file name into an URL. +func getURI(addr string) (*url.URL, error) { + url, err := url.Parse(addr) + if err != nil { + // try to parse with a http scheme as a last resort + u, err2 := url.Parse("http://" + addr) + if err2 != nil { + return nil, err + } + url = u + } + return url, nil +} + +// wsURI returns a websocket url from a HTTP URI. +func wsURI(addr, endpoint string) (string, error) { + uri, err := getURI(addr) + if err != nil { + return "", err + } + + uri.Scheme = "ws" + uri.Path = endpoint + v := url.Values{} + v.Set("watch", "true") + uri.RawQuery = v.Encode() + + return uri.String(), nil +} From d93e41c1177a5c9f6e3e0271cc8ae70b97fbeba7 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 2 Feb 2024 13:33:08 +0100 Subject: [PATCH 115/243] fix: Streamline environment and CLI flags in CDS discovery --- cmd/stunnerctl/README.md | 4 +-- cmd/stunnerd/main.go | 14 ++++---- pkg/apis/v1/default.go | 57 ++++++++++++++++++++------------- pkg/config/client/k8s_client.go | 25 +++++++++------ 4 files changed, 58 insertions(+), 42 deletions(-) diff --git a/cmd/stunnerctl/README.md b/cmd/stunnerctl/README.md index 4e017234..5d3c7d5c 100644 --- a/cmd/stunnerctl/README.md +++ b/cmd/stunnerctl/README.md @@ -68,7 +68,7 @@ The below will dump the running config of `tcp-gateway` deployed into the `stunn ```console cd stunner -cmd/stunnerctl/stunnerctl.sh running-config stunner/stunner-gateway +cmd/stunnerctl/stunnerctl.sh running-config stunner/tcp-gateway STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 @@ -82,7 +82,7 @@ Listener 1 ## Last resort -You can use `kubectl port-forward` to load or watch STUNner configs manually. +You can also use `kubectl port-forward` to load or watch STUNner configs manually. Open a port-forwarded connection to the STUNner gateway operator: ``` console diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 88d692f5..20ecb82f 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -19,14 +19,14 @@ import ( func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", fmt.Sprintf("Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN, default: %s).", stnrv1.DefaultConfigDiscoveryAddress)) - var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO).") - var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: ).") - var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false).") + var config = flag.StringP("config", "c", "", "Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN)") + var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO)") + var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: )") + var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false)") var udpThreadNum = flag.IntP("udp-thread-num", "u", 0, - "Number of readloop threads (CPU cores) per UDP listener. Zero disables UDP multithreading (default: 0).") - var dryRun = flag.BoolP("dry-run", "d", false, "Suppress side-effects, intended for testing (default: false).") - var verbose = flag.BoolP("verbose", "v", false, "Verbose logging, identical to <-l all:DEBUG>.") + "Number of readloop threads (CPU cores) per UDP listener. Zero disables UDP multithreading (default: 0)") + var dryRun = flag.BoolP("dry-run", "d", false, "Suppress side-effects, intended for testing (default: false)") + var verbose = flag.BoolP("verbose", "v", false, "Verbose logging, identical to <-l all:DEBUG>") flag.Parse() logLevel := stnrv1.DefaultLogLevel diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 0a2940cb..6dac5136 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -1,27 +1,38 @@ package v1 -const ApiVersion string = "v1" -const DefaultStunnerName = "default-stunnerd" -const DefaultProtocol = "turn-udp" -const DefaultClusterProtocol = "udp" -const DefaultPort int = 3478 -const DefaultLogLevel = "all:INFO" -const DefaultRealm = "stunner.l7mp.io" -const DefaultAuthType = "static" -const DefaultMinRelayPort int = 1 -const DefaultMaxRelayPort int = 1<<16 - 1 -const DefaultClusterType = "STATIC" +// stunnerd defaults +const ( + ApiVersion string = "v1" + DefaultStunnerName = "default-stunnerd" + DefaultProtocol = "turn-udp" + DefaultClusterProtocol = "udp" + DefaultPort int = 3478 + DefaultLogLevel = "all:INFO" + DefaultRealm = "stunner.l7mp.io" + DefaultAuthType = "static" + DefaultMinRelayPort int = 1 + DefaultMaxRelayPort int = 1<<16 - 1 + DefaultClusterType = "STATIC" + DefaultAdminName = "default-admin-config" + DefaultAuthName = "default-auth-config" +) -const DefaultAdminName = "default-admin-config" -const DefaultAuthName = "default-auth-config" +// health-check and metrics reporting defaults +const ( + DefaultMetricsPort int = 8080 + DefaultHealthCheckPort int = 8086 +) -const DefaultMetricsPort int = 8080 -const DefaultHealthCheckPort int = 8086 - -const DefaultConfigDiscoveryAddress = ":13478" -const DefaultEnvVarName = "STUNNER_NAME" -const DefaultEnvVarNamespace = "STUNNER_NAMESPACE" -const DefaultEnvVarNodeName = "STUNNER_NODENAME" -const DefaultEnvVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" -const DefaultCDSServiceLabelKey = "stunner.l7mp.io/config-discovery-service" -const DefaultCDSServiceLabelValue = "enabled" +// CDS defaults +const ( + DefaultConfigDiscoveryAddress = ":13478" + DefaultEnvVarName = "STUNNER_NAME" + DefaultEnvVarNamespace = "STUNNER_NAMESPACE" + DefaultEnvVarNodeName = "STUNNER_NODENAME" + DefaultEnvVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" + DefaultCDSServiceLabelKey = "stunner.l7mp.io/config-discovery-service" + DefaultCDSServiceLabelValue = "enabled" + DefaultCDSServerAddrEnv = "CDS_SERVER_ADDR" + DefaultCDSServerNamespaceEnv = "CDS_SERVER_NAMESPACE" + DefaultCDSServerPortEnv = "CDS_SERVER_PORT" +) diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go index a903cb08..a25d6d1f 100644 --- a/pkg/config/client/k8s_client.go +++ b/pkg/config/client/k8s_client.go @@ -7,6 +7,7 @@ import ( "net/http" "os" "strconv" + "strings" "github.com/pion/logging" "github.com/spf13/pflag" @@ -19,12 +20,16 @@ import ( stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) -const ( - DefaultCDSServerPort = 13478 - CDSServerAddrEnv = "CDS_SERVER_ADDR" - CDSServerNamespaceEnv = "CDS_SERVER_NAME" - CDSServerPortEnv = "CDS_SERVER_PORT" -) +var DefaultCDSServerPort = 13478 + +func init() { + as := strings.Split(stnrv1.DefaultConfigDiscoveryAddress, ":") + if len(as) == 2 { + if port, err := strconv.Atoi(as[1]); err != nil { + DefaultCDSServerPort = port + } + } +} // CDSConfigFlags composes a set of flags for CDS server discovery type CDSConfigFlags struct { @@ -39,16 +44,16 @@ type CDSConfigFlags struct { // NewCDSConfigFlags returns CDS service discovery flags with default values set. func NewCDSConfigFlags() *CDSConfigFlags { port := DefaultCDSServerPort - if os.Getenv(CDSServerPortEnv) != "" { - p, err := strconv.Atoi(os.Getenv(CDSServerPortEnv)) + if os.Getenv(stnrv1.DefaultCDSServerPortEnv) != "" { + p, err := strconv.Atoi(os.Getenv(stnrv1.DefaultCDSServerPortEnv)) if err != nil { port = p } } return &CDSConfigFlags{ - ServerAddr: os.Getenv(CDSServerAddrEnv), + ServerAddr: os.Getenv(stnrv1.DefaultCDSServerAddrEnv), ServerPort: port, - ServerNamespace: os.Getenv(CDSServerNamespaceEnv), + ServerNamespace: os.Getenv(stnrv1.DefaultCDSServerNamespaceEnv), } } From 00c9c0e97b0870823159ad703d4fa0c11b79d249 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 2 Feb 2024 17:10:05 +0100 Subject: [PATCH 116/243] chore: Streamline CDS client error messages --- pkg/config/client/cds_api.go | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go index 28871034..481ee7ab 100644 --- a/pkg/config/client/cds_api.go +++ b/pkg/config/client/cds_api.go @@ -5,6 +5,7 @@ import ( "context" "fmt" "net/http" + "strings" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/config/client/api" @@ -85,7 +86,9 @@ func (a *AllConfigsAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error } if r.HTTPResponse.StatusCode != http.StatusOK { - return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error: %s", r.HTTPResponse.Status) + body := strings.TrimSpace(string(r.Body)) + return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error (status: %s): %s", + r.HTTPResponse.Status, body) } return decodeConfigList(r.Body) @@ -148,8 +151,9 @@ func (a *ConfigsNamespaceAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, } if r.HTTPResponse.StatusCode != http.StatusOK { - return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error: %s", - r.HTTPResponse.Status) + body := strings.TrimSpace(string(r.Body)) + return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error (status: %s): %s", + r.HTTPResponse.Status, body) } return decodeConfigList(r.Body) @@ -215,8 +219,9 @@ func (a *ConfigNamespaceNameAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConf } if r.HTTPResponse.StatusCode != http.StatusOK { - return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error: %s", - r.HTTPResponse.Status) + body := strings.TrimSpace(string(r.Body)) + return []*stnrv1.StunnerConfig{}, fmt.Errorf("HTTP error (status: %s): %s", + r.HTTPResponse.Status, body) } return decodeConfig(r.Body) From 1bb46b71873db44818e8c9260eccb36ea7141ef0 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 2 Feb 2024 19:03:11 +0100 Subject: [PATCH 117/243] fix: Restore credentials after config env substitution, fix #102 ParseConfig applies environment substitution on the config file being parsed. This interferes with credential parsing, which may contain valid `$` symbols, and messes up credentials otherwise perfectly valid credentials (we should base-64 encode credentials at some point). This fix restores the credentials after env substitution to the original ones. --- config_test.go | 42 +++++++++++++++++++++++++++++++++++++ pkg/config/client/config.go | 37 ++++++++++++++++++++++++++------ 2 files changed, 73 insertions(+), 6 deletions(-) diff --git a/config_test.go b/config_test.go index 5e87dcd2..35970fbb 100644 --- a/config_test.go +++ b/config_test.go @@ -501,6 +501,48 @@ func TestStunnerURIParser(t *testing.T) { } } +// make sure credentials are excempt from env-substitution in ParseConfig +func TestCredentialParser(t *testing.T) { + lim := test.TimeOut(time.Second * 30) + defer lim.Stop() + + report := test.CheckRoutines(t) + defer report() + + loggerFactory := logger.NewLoggerFactory(stunnerTestLoglevel) + log := loggerFactory.NewLogger("test") + + for _, testConf := range []struct { + name string + config []byte + user, pass, secret string + }{ + {"plain", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"pass","username":"user"}}}`), "user", "pass", ""}, + // user name with $ + {"username_with_leading_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"pass","username":"$user"}}}`), "$user", "pass", ""}, + {"username_with_trailing_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"pass","username":"user$"}}}`), "user$", "pass", ""}, + {"username_with_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"pass","username":"us$er"}}}`), "us$er", "pass", ""}, + // passwd with $ + {"passwd_with_leading_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"$pass","username":"user"}}}`), "user", "$pass", ""}, + {"passwd_with_trailing_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"pass$","username":"user"}}}`), "user", "pass$", ""}, + {"passwd_with_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"password":"pa$ss","username":"user"}}}`), "user", "pa$ss", ""}, + // secret with $ + {"secret_with_leading_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"secret":"$secret","username":"user"}}}`), "user", "", "$secret"}, + {"secret_with_trailing_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"secret":"secret$","username":"user"}}}`), "user", "", "secret$"}, + {"secret_with_$", []byte(`{"version":"v1","admin":{"name":"ns1/tester"},"auth":{"type":"static","credentials":{"secret":"sec$ret","username":"user"}}}`), "user", "", "sec$ret"}, + } { + testName := fmt.Sprintf("TestCredentialParser:%s", testConf.name) + t.Run(testName, func(t *testing.T) { + log.Debugf("-------------- Running test: %s -------------", testName) + c, err := cdsclient.ParseConfig(testConf.config) + assert.NoError(t, err, "parser") + assert.Equal(t, testConf.user, c.Auth.Credentials["username"], "username") + assert.Equal(t, testConf.pass, c.Auth.Credentials["password"], "password") + assert.Equal(t, testConf.secret, c.Auth.Credentials["secret"], "secret") + }) + } +} + func checkDefaultConfig(t *testing.T, c *stnrv1.StunnerConfig, proto string) { assert.Equal(t, "static", c.Auth.Type, "auth-type") assert.Equal(t, "user1", c.Auth.Credentials["username"], "username") diff --git a/pkg/config/client/config.go b/pkg/config/client/config.go index a05f9b61..001ceeba 100644 --- a/pkg/config/client/config.go +++ b/pkg/config/client/config.go @@ -3,6 +3,7 @@ package client import ( "encoding/json" "fmt" + "maps" "os" "regexp" "strconv" @@ -54,12 +55,36 @@ func ParseConfig(c []byte) (*stnrv1.StunnerConfig, error) { os.Setenv("STUNNER_PORT", fmt.Sprintf("%d", publicPort)) } + // make sure credentials are not affected by environment substitution + + // parse up before env substitution is applied + confRaw, err := parseRaw(c) + if err != nil { + return nil, err + } + + // save credentials + credRaw := make(map[string]string) + maps.Copy(credRaw, confRaw.Auth.Credentials) + + // apply env substitution and parse again e := os.ExpandEnv(string(c)) + confExp, err := parseRaw([]byte(e)) + if err != nil { + return nil, err + } + + // restore credentials + maps.Copy(confExp.Auth.Credentials, credRaw) + + return confExp, nil +} +func parseRaw(c []byte) (*stnrv1.StunnerConfig, error) { // try to parse only the config version first k := ConfigSkeleton{} - if err := yaml.Unmarshal([]byte(e), &k); err != nil { - if errJ := json.Unmarshal([]byte(e), &k); err != nil { + if err := yaml.Unmarshal([]byte(c), &k); err != nil { + if errJ := json.Unmarshal([]byte(c), &k); err != nil { return nil, fmt.Errorf("could not parse config file API version: "+ "YAML parse error: %s, JSON parse error: %s\n", err.Error(), errJ.Error()) @@ -70,8 +95,8 @@ func ParseConfig(c []byte) (*stnrv1.StunnerConfig, error) { switch k.ApiVersion { case stnrv1.ApiVersion: - if err := yaml.Unmarshal([]byte(e), &s); err != nil { - if errJ := json.Unmarshal([]byte(e), &s); err != nil { + if err := yaml.Unmarshal([]byte(c), &s); err != nil { + if errJ := json.Unmarshal([]byte(c), &s); errJ != nil { return nil, fmt.Errorf("could not parse config file: "+ "YAML parse error: %s, JSON parse error: %s\n", err.Error(), errJ.Error()) @@ -79,8 +104,8 @@ func ParseConfig(c []byte) (*stnrv1.StunnerConfig, error) { } case stnrv1a1.ApiVersion: a := stnrv1a1.StunnerConfig{} - if err := yaml.Unmarshal([]byte(e), &a); err != nil { - if errJ := json.Unmarshal([]byte(e), &a); err != nil { + if err := yaml.Unmarshal([]byte(c), &a); err != nil { + if errJ := json.Unmarshal([]byte(c), &a); errJ != nil { return nil, fmt.Errorf("could not parse config file: "+ "YAML parse error: %s, JSON parse error: %s\n", err.Error(), errJ.Error()) From bc1bfb991303979c883e21d96b06d6e825484523 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 2 Feb 2024 19:06:26 +0100 Subject: [PATCH 118/243] test: Avoid CDS server port clashes in tests --- config_test.go | 5 ++-- pkg/config/cds_test.go | 58 ++++++++++++++++++++++-------------------- 2 files changed, 33 insertions(+), 30 deletions(-) diff --git a/config_test.go b/config_test.go index 35970fbb..a7b65acf 100644 --- a/config_test.go +++ b/config_test.go @@ -16,6 +16,7 @@ import ( "sigs.k8s.io/yaml" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) @@ -342,7 +343,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { log.Debugf("-------------- Running test: %s -------------", testName) log.Debug("creating a mock CDS server") - addr := "localhost:13478" + addr := "localhost:63479" origin := "ws://" + addr ctx, cancel := context.WithCancel(context.Background()) @@ -426,7 +427,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { assert.Len(t, c2.Clusters[0].Endpoints, 1, "endpoints len") assert.Equal(t, "1.2.3.5", c2.Clusters[0].Endpoints[0], "cluster port") - // thrird read yields a v1alpha1 config + // next read yields a v1alpha1 config c2, ok = <-conf assert.True(t, ok, "config emitted") diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index 28734f4a..401d49fb 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -23,6 +23,8 @@ var testerLogLevel = zapcore.ErrorLevel // const stunnerLogLevel = "all:TRACE" const stunnerLogLevel = "all:ERROR" +const testCDSAddr = ":63478" + func init() { // setup a fast pinger so that we get a timely error notification client.PingPeriod = 500 * time.Millisecond @@ -46,7 +48,7 @@ func TestServerLoad(t *testing.T) { defer cancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(ctx) assert.NoError(t, err, "start") @@ -54,12 +56,12 @@ func TestServerLoad(t *testing.T) { time.Sleep(20 * time.Millisecond) testLog.Debug("create client") - client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + client1, err := client.New(testCDSAddr, "ns1/gw1", logger) assert.NoError(t, err, "client 1") - client2, err := client.New("127.0.0.1:13478", "ns1/gw2", logger) + client2, err := client.New(testCDSAddr, "ns1/gw2", logger) assert.NoError(t, err, "client 2") // nonexistent - client3, err := client.New("127.0.0.1:13478", "ns1/gw3", logger) + client3, err := client.New(testCDSAddr, "ns1/gw3", logger) assert.NoError(t, err, "client 3") testLog.Debug("load: error") @@ -134,7 +136,7 @@ func TestServerPoll(t *testing.T) { defer cancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(ctx) assert.NoError(t, err, "start") @@ -142,11 +144,11 @@ func TestServerPoll(t *testing.T) { time.Sleep(20 * time.Millisecond) testLog.Debug("create client") - client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + client1, err := client.New(testCDSAddr, "ns1/gw1", logger) assert.NoError(t, err, "client 1") - client2, err := client.New("127.0.0.1:13478", "ns1/gw2", logger) + client2, err := client.New(testCDSAddr, "ns1/gw2", logger) assert.NoError(t, err, "client 2") - client3, err := client.New("127.0.0.1:13478", "ns1/gw3", logger) + client3, err := client.New(testCDSAddr, "ns1/gw3", logger) assert.NoError(t, err, "client 3") testLog.Debug("poll: no result") @@ -236,17 +238,17 @@ func TestServerWatch(t *testing.T) { serverCtx, serverCancel := context.WithCancel(context.Background()) testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") testLog.Debug("create client") - client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + client1, err := client.New(testCDSAddr, "ns1/gw1", logger) assert.NoError(t, err, "client 1") - client2, err := client.New("127.0.0.1:13478", "ns1/gw2", logger) + client2, err := client.New(testCDSAddr, "ns1/gw2", logger) assert.NoError(t, err, "client 2") - client3, err := client.New("127.0.0.1:13478", "ns1/gw3", logger) + client3, err := client.New(testCDSAddr, "ns1/gw3", logger) assert.NoError(t, err, "client 3") testLog.Debug("watch: no result") @@ -339,7 +341,7 @@ func TestServerWatch(t *testing.T) { time.Sleep(50 * time.Millisecond) serverCtx, serverCancel = context.WithCancel(context.Background()) defer serverCancel() - srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv = server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -413,13 +415,13 @@ func TestServerWatchBootstrap(t *testing.T) { defer serverCancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") testLog.Debug("create client") - client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + client1, err := client.New(testCDSAddr, "ns1/gw1", logger) assert.NoError(t, err, "client 1") testLog.Debug("bootstrap") @@ -505,19 +507,19 @@ func TestServerAPI(t *testing.T) { serverCtx, serverCancel := context.WithCancel(context.Background()) testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") testLog.Debug("create client") - client1, err := client.NewAllConfigsAPI("127.0.0.1:13478", logger.NewLogger("all-config-client")) + client1, err := client.NewAllConfigsAPI(testCDSAddr, logger.NewLogger("all-config-client")) assert.NoError(t, err, "client 1") - client2, err := client.NewConfigsNamespaceAPI("127.0.0.1:13478", "ns1", logger.NewLogger("ns-config-client-ns1")) + client2, err := client.NewConfigsNamespaceAPI(testCDSAddr, "ns1", logger.NewLogger("ns-config-client-ns1")) assert.NoError(t, err, "client 2") - client3, err := client.NewConfigsNamespaceAPI("127.0.0.1:13478", "ns2", logger.NewLogger("ns-config-client-ns2")) + client3, err := client.NewConfigsNamespaceAPI(testCDSAddr, "ns2", logger.NewLogger("ns-config-client-ns2")) assert.NoError(t, err, "client 3") - client4, err := client.NewConfigNamespaceNameAPI("127.0.0.1:13478", "ns1", "gw1", logger.NewLogger("gw-config-client")) + client4, err := client.NewConfigNamespaceNameAPI(testCDSAddr, "ns1", "gw1", logger.NewLogger("gw-config-client")) assert.NoError(t, err, "client 4") testLog.Debug("watch: no result") @@ -738,7 +740,7 @@ func TestServerAPI(t *testing.T) { time.Sleep(50 * time.Millisecond) serverCtx, serverCancel = context.WithCancel(context.Background()) defer serverCancel() - srv = server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv = server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -940,13 +942,13 @@ func TestClientReconnect(t *testing.T) { defer serverCancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") testLog.Debug("create client") - client1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) + client1, err := client.New(testCDSAddr, "ns1/gw1", logger) assert.NoError(t, err, "client 1") testLog.Debug("watch: no result") @@ -1012,7 +1014,7 @@ func TestServerUpdate(t *testing.T) { defer serverCancel() testLog.Debug("create server") - srv := server.New(stnrv1.DefaultConfigDiscoveryAddress, nil, log) + srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) assert.NoError(t, err, "start") @@ -1127,14 +1129,14 @@ func TestServerUpdate(t *testing.T) { // p.Config.Listeners[2].Addr = "10.11.12.13" // testLog.Debug("client w/o node IP") -// loader1, err := client.New("127.0.0.1:13478", "ns1/gw1", logger) +// loader1, err := client.New(testCDSAddr, "ns1/gw1", logger) // assert.NoError(t, err, "client") // sc1, err := loader1.Load() // assert.NoError(t, err, "load") // assert.True(t, sc1.DeepEqual(c.Config), "deepeq") // testLog.Debug("client w/ node IP") -// loader2, err := client.New("127.0.0.1:13478", "ns1/gw1", map[string]string{"node": "10.11.12.13"}, logger) +// loader2, err := client.New(testCDSAddr, "ns1/gw1", map[string]string{"node": "10.11.12.13"}, logger) // assert.NoError(t, err, "client") // sc2, err := loader2.Load() // assert.NoError(t, err, "load") @@ -1144,7 +1146,7 @@ func TestServerUpdate(t *testing.T) { // defer watchCancel() // testLog.Debug("watcher1 w/o node IP") -// watcher1, err := client.New("127.0.0.1:13478", "ns1/gw1", nil, logger) +// watcher1, err := client.New(testCDSAddr, "ns1/gw1", nil, logger) // assert.NoError(t, err, "client") // ch1 := make(chan *stnrv1.StunnerConfig, 8) // defer close(ch1) @@ -1157,7 +1159,7 @@ func TestServerUpdate(t *testing.T) { // assert.True(t, s.DeepEqual(c.Config), "deepeq") // testLog.Debug("watcher2 w/ node IP") -// watcher2, err := client.New("127.0.0.1:13478", "ns1/gw1", map[string]string{"node": "10.11.12.13"}, logger) +// watcher2, err := client.New(testCDSAddr, "ns1/gw1", map[string]string{"node": "10.11.12.13"}, logger) // assert.NoError(t, err, "client") // ch2 := make(chan *stnrv1.StunnerConfig, 8) // defer close(ch2) From dabdb348f659a7271464cb512417b01df015cf3e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Mon, 5 Feb 2024 15:27:59 +0100 Subject: [PATCH 119/243] chore(CI/CD): Build and release stunnerctl too (#121) --- .github/workflows/publish--add-binaries.yml | 72 ++++++++++++++++++++ .github/workflows/publish--push-charts.yml | 40 +++++++++++ .github/workflows/publish-dev.yaml | 13 +--- .github/workflows/publish.yml | 74 ++------------------- .github/workflows/test.yml | 2 +- 5 files changed, 123 insertions(+), 78 deletions(-) create mode 100644 .github/workflows/publish--add-binaries.yml create mode 100644 .github/workflows/publish--push-charts.yml diff --git a/.github/workflows/publish--add-binaries.yml b/.github/workflows/publish--add-binaries.yml new file mode 100644 index 00000000..f9416b5c --- /dev/null +++ b/.github/workflows/publish--add-binaries.yml @@ -0,0 +1,72 @@ +name: "publish: Add binaries to release assets" + +on: + workflow_call: + +jobs: + add_binaries: + name: Add binaries to release assets + runs-on: ubuntu-latest + strategy: + matrix: + include: + - os: linux + arch: amd64 + file_end: "" + - os: linux + arch: arm64 + file_end: "" + + - os: darwin + arch: amd64 + file_end: "" + - os: darwin + arch: arm64 + file_end: "" + + - os: windows + arch: amd64 + file_end: ".exe" + - os: windows + arch: arm64 + file_end: ".exe" + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Get version + id: vars + run: echo tag=$(echo ${GITHUB_REF:11}) >> $GITHUB_OUTPUT + + - name: Install Go + uses: actions/setup-go@v5 + with: + go-version: '1.21' + + - name: Build turncat binary + run: | + CGO_ENABLED=0 GOARCH=${{ matrix.arch }} GOOS=${{ matrix.os }} go build -ldflags="-w -s" -trimpath -o turncat cmd/turncat/main.go + mv turncat turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + + - name: Release turncat binary + uses: svenstaro/upload-release-action@v2 + with: + repo_token: ${{ secrets.GITHUB_TOKEN }} + file: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + tag: ${{ github.ref_name }} + asset_name: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + + + - name: Build stunnerctl binary + run: | + CGO_ENABLED=0 GOARCH=${{ matrix.arch }} GOOS=${{ matrix.os }} go build -ldflags="-w -s" -trimpath -o stunnerctl cmd/stunnerctl/main.go + mv stunnerctl stunnerctl-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + + - name: Release stunnerctl binary + uses: svenstaro/upload-release-action@v2 + with: + repo_token: ${{ secrets.GITHUB_TOKEN }} + file: stunnerctl-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + tag: ${{ github.ref_name }} + asset_name: stunnerctl-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} diff --git a/.github/workflows/publish--push-charts.yml b/.github/workflows/publish--push-charts.yml new file mode 100644 index 00000000..7a28fe20 --- /dev/null +++ b/.github/workflows/publish--push-charts.yml @@ -0,0 +1,40 @@ +name: "publish: Push Helm charts to web" + +on: + workflow_call: + with: + inputs: + dev: + description: Whether to release a dev version + required: true + type: boolean + +jobs: + push_charts: + name: Push Helm charts to web + runs-on: ubuntu-latest + steps: + - name: Get version for non-dev release + if: ${{ inputs.dev == false || inputs.dev == 'false' }} + id: vars + run: echo tag=$(echo ${GITHUB_REF:11}) >> $GITHUB_OUTPUT + + - name: Trigger release workflow in the stunner-helm repo + if: ${{ inputs.dev == false || inputs.dev == 'false' }} + uses: convictional/trigger-workflow-and-wait@v1.6.5 + with: + github_token: ${{ secrets.WEB_PAT_TOKEN }} + owner: l7mp + repo: stunner-helm + client_payload: '{"tag": "${{ steps.vars.outputs.tag }}", "type": "stunner"}' + workflow_file_name: publish.yaml + + - name: Trigger release workflow in the stunner-helm repo + if: ${{ inputs.dev == true || inputs.dev == 'true' }} + uses: convictional/trigger-workflow-and-wait@v1.6.5 + with: + github_token: ${{ secrets.WEB_PAT_TOKEN }} + owner: l7mp + repo: stunner-helm + client_payload: '{"tag": "dev", "type": "stunner"}' + workflow_file_name: publish.yaml diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index ca0f16c9..92ffd39a 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -56,13 +56,6 @@ jobs: push_chart: name: Push helm charts to the repo needs: push_to_registry - runs-on: ubuntu-latest - steps: - - name: Triggering release workflow in the stunner-helm repo - uses: convictional/trigger-workflow-and-wait@v1.6.5 - with: - github_token: ${{ secrets.WEB_PAT_TOKEN }} - owner: l7mp - repo: stunner-helm - client_payload: '{"tag": "dev", "type": "stunner"}' - workflow_file_name: publish.yaml + uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main + with: + dev: true diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 4283cdc4..6217f7d8 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -49,73 +49,13 @@ jobs: labels: ${{ steps.meta.outputs.labels }} push_chart: - name: Push charts to the web + name: Push helm charts to the repo needs: push_to_registry - runs-on: ubuntu-latest - steps: - - name: Get version - id: vars - run: echo tag=$(echo ${GITHUB_REF:11}) >> $GITHUB_OUTPUT + uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main + with: + dev: false - - name: Triggering release workflow in the stunner-helm repo - uses: convictional/trigger-workflow-and-wait@v1.6.5 - with: - github_token: ${{ secrets.WEB_PAT_TOKEN }} - owner: l7mp - repo: stunner-helm - client_payload: '{"tag": "${{ steps.vars.outputs.tag }}", "type": "stunner"}' - workflow_file_name: publish.yaml - - add_turncat_binaries: - name: Add turncat binaries to release assets + add_binaries: + name: Add binaries to release assets + uses: l7mp/stunner/.github/workflows/publish--add-binaries.yml@main needs: run_tests - runs-on: ubuntu-latest - strategy: - matrix: - include: - - os: linux - arch: amd64 - file_end: "" - - os: linux - arch: arm64 - file_end: "" - - - os: darwin - arch: amd64 - file_end: "" - - os: darwin - arch: arm64 - file_end: "" - - - os: windows - arch: amd64 - file_end: ".exe" - - os: windows - arch: arm64 - file_end: ".exe" - - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Get version - id: vars - run: echo tag=$(echo ${GITHUB_REF:11}) >> $GITHUB_OUTPUT - - - name: Install Go - uses: actions/setup-go@v5 - with: - go-version: '1.21' - - - name: Build turncat binary - run: | - CGO_ENABLED=0 GOARCH=${{ matrix.arch }} GOOS=${{ matrix.os }} go build -ldflags="-w -s" -trimpath -o turncat cmd/turncat/main.go - mv turncat turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} - - - name: Release - uses: svenstaro/upload-release-action@v2 - with: - repo_token: ${{ secrets.GITHUB_TOKEN }} - file: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} - tag: ${{ github.ref_name }} - asset_name: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 4197274d..e4dac3e2 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -46,7 +46,7 @@ jobs: - name: Convert coverage.out to coverage.lcov uses: jandelgado/gcov2lcov-action@v1 - name: Coveralls - uses: coverallsapp/github-action@v1 + uses: coverallsapp/github-action@v2 with: github-token: ${{ secrets.github_token }} path-to-lcov: coverage.lcov From 4e2d950c3a84ad70683b753770b130171b4164e1 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 5 Feb 2024 15:41:36 +0100 Subject: [PATCH 120/243] chore(CI/CD): Fix syntax errors --- .github/workflows/publish-dev.yaml | 7 ++++--- .github/workflows/publish.yml | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 92ffd39a..6fb83dec 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -56,6 +56,7 @@ jobs: push_chart: name: Push helm charts to the repo needs: push_to_registry - uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main - with: - dev: true + steps: + - uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main + with: + dev: true diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 6217f7d8..82b35331 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -51,9 +51,10 @@ jobs: push_chart: name: Push helm charts to the repo needs: push_to_registry - uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main - with: - dev: false + steps: + - uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main + - with: + dev: false add_binaries: name: Add binaries to release assets From 73632e60d150605440637266b253482f40306541 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 5 Feb 2024 15:57:40 +0100 Subject: [PATCH 121/243] chore(CI/CD): Revert 4e2d950 --- .github/workflows/publish-dev.yaml | 7 +++---- .github/workflows/publish.yml | 7 +++---- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 6fb83dec..58960cb6 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -56,7 +56,6 @@ jobs: push_chart: name: Push helm charts to the repo needs: push_to_registry - steps: - - uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main - with: - dev: true + uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main + with: + dev: 'true' diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 82b35331..8ffb95b0 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -51,10 +51,9 @@ jobs: push_chart: name: Push helm charts to the repo needs: push_to_registry - steps: - - uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main - - with: - dev: false + uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main + with: + dev: 'false' add_binaries: name: Add binaries to release assets From d4466b6090effc76fc73f6c684d890ccba912309 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 5 Feb 2024 16:01:06 +0100 Subject: [PATCH 122/243] chore(CI/CD): Fix syntax errors in push_charts --- .github/workflows/publish--push-charts.yml | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/.github/workflows/publish--push-charts.yml b/.github/workflows/publish--push-charts.yml index 7a28fe20..f39fdf7e 100644 --- a/.github/workflows/publish--push-charts.yml +++ b/.github/workflows/publish--push-charts.yml @@ -2,12 +2,11 @@ name: "publish: Push Helm charts to web" on: workflow_call: - with: - inputs: - dev: - description: Whether to release a dev version - required: true - type: boolean + inputs: + dev: + description: Whether to release a dev version + required: true + type: boolean jobs: push_charts: From 9f1624daa54ae6b7ef87e0deea54dcfd98560743 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 5 Feb 2024 16:02:23 +0100 Subject: [PATCH 123/243] chore(CI/CD): Fix type errors in publish workflows --- .github/workflows/publish-dev.yaml | 2 +- .github/workflows/publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 58960cb6..92ffd39a 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -58,4 +58,4 @@ jobs: needs: push_to_registry uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: - dev: 'true' + dev: true diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 8ffb95b0..6217f7d8 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -53,7 +53,7 @@ jobs: needs: push_to_registry uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: - dev: 'false' + dev: false add_binaries: name: Add binaries to release assets From ea4648090fe2b4c04b685395095a8e1a9cf10d02 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 5 Feb 2024 16:18:33 +0100 Subject: [PATCH 124/243] chore(CI/CD): Fix missing secrets in push_charts --- .github/workflows/publish--push-charts.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/publish--push-charts.yml b/.github/workflows/publish--push-charts.yml index f39fdf7e..b11c208c 100644 --- a/.github/workflows/publish--push-charts.yml +++ b/.github/workflows/publish--push-charts.yml @@ -7,6 +7,7 @@ on: description: Whether to release a dev version required: true type: boolean + secrets: inherit jobs: push_charts: From 8ba9074efdba488166a04fece9e332e3c8239c1e Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 5 Feb 2024 16:20:58 +0100 Subject: [PATCH 125/243] chore(CI/CD): Fix missing secrets in push_charts --- .github/workflows/publish--push-charts.yml | 1 - .github/workflows/publish-dev.yaml | 1 + .github/workflows/publish.yml | 1 + 3 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/publish--push-charts.yml b/.github/workflows/publish--push-charts.yml index b11c208c..f39fdf7e 100644 --- a/.github/workflows/publish--push-charts.yml +++ b/.github/workflows/publish--push-charts.yml @@ -7,7 +7,6 @@ on: description: Whether to release a dev version required: true type: boolean - secrets: inherit jobs: push_charts: diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 92ffd39a..7e98ace5 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -59,3 +59,4 @@ jobs: uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: dev: true + secrets: inherit diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 6217f7d8..ff6d0fe1 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -54,6 +54,7 @@ jobs: uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: dev: false + secrets: inherit add_binaries: name: Add binaries to release assets From b16223c08d35889268dd2c37b00f89b96ca3728e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sat, 3 Feb 2024 11:39:52 +0100 Subject: [PATCH 126/243] chore: Streamline CDS server error handling --- go.mod | 59 +++++++++--------- go.sum | 62 +++++++++++++++++++ pkg/config/api/stunner_openapi.yaml | 34 ++++------- pkg/config/client/api/client.gen.go | 38 ++++++------ pkg/config/server/api/server.gen.go | 94 ++++++++++++++++------------- pkg/config/server/handler.go | 13 +++- 6 files changed, 186 insertions(+), 114 deletions(-) diff --git a/go.mod b/go.mod index 3265572d..ed859099 100644 --- a/go.mod +++ b/go.mod @@ -5,17 +5,17 @@ go 1.21 toolchain go1.21.5 require ( - github.com/deepmap/oapi-codegen/v2 v2.0.0 + github.com/deepmap/oapi-codegen/v2 v2.1.0 github.com/fsnotify/fsnotify v1.7.0 github.com/getkin/kin-openapi v0.123.0 - github.com/go-logr/logr v1.3.0 + github.com/go-logr/logr v1.4.1 github.com/go-logr/zapr v1.3.0 - github.com/google/uuid v1.5.0 + github.com/google/uuid v1.6.0 github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.1 github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb github.com/oapi-codegen/runtime v1.1.1 - github.com/pion/dtls/v2 v2.2.9 + github.com/pion/dtls/v2 v2.2.10 github.com/pion/logging v0.2.2 github.com/pion/transport/v3 v3.0.1 github.com/pion/turn/v3 v3.0.1 @@ -36,25 +36,25 @@ require ( ) require ( - github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect + github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect - github.com/emicklei/go-restful/v3 v3.11.0 // indirect - github.com/evanphx/json-patch v4.12.0+incompatible // indirect - github.com/go-errors/errors v1.4.2 // indirect + github.com/emicklei/go-restful/v3 v3.11.2 // indirect + github.com/evanphx/json-patch v5.9.0+incompatible // indirect + github.com/go-errors/errors v1.5.1 // indirect github.com/go-openapi/jsonpointer v0.20.2 // indirect - github.com/go-openapi/jsonreference v0.20.2 // indirect - github.com/go-openapi/swag v0.22.8 // indirect + github.com/go-openapi/jsonreference v0.20.4 // indirect + github.com/go-openapi/swag v0.22.9 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/golang/protobuf v1.5.3 // indirect - github.com/google/btree v1.0.1 // indirect + github.com/google/btree v1.1.2 // indirect github.com/google/gnostic-models v0.6.8 // indirect github.com/google/gofuzz v1.2.0 // indirect github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect - github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect - github.com/imdario/mergo v0.3.12 // indirect + github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect + github.com/imdario/mergo v0.3.16 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/invopop/yaml v0.2.0 // indirect github.com/josharian/intern v1.0.0 // indirect @@ -63,7 +63,7 @@ require ( github.com/mailru/easyjson v0.7.7 // indirect github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect github.com/moby/spdystream v0.2.0 // indirect - github.com/moby/term v0.0.0-20221205130635-1aeaba878587 // indirect + github.com/moby/term v0.5.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect @@ -74,34 +74,35 @@ require ( github.com/peterbourgon/diskv v2.0.1+incompatible // indirect github.com/pion/randutil v0.1.0 // indirect github.com/pion/stun/v2 v2.0.0 // indirect - github.com/pion/transport/v2 v2.2.1 // indirect + github.com/pion/transport/v2 v2.2.4 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.5.0 // indirect - github.com/prometheus/common v0.45.0 // indirect + github.com/prometheus/common v0.46.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect github.com/xlab/treeprint v1.2.0 // indirect - go.starlark.net v0.0.0-20230525235612-a134d8f9ddca // indirect + go.starlark.net v0.0.0-20240123142251-f86470692795 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.17.0 // indirect + golang.org/x/crypto v0.18.0 // indirect golang.org/x/mod v0.14.0 // indirect - golang.org/x/net v0.19.0 // indirect - golang.org/x/oauth2 v0.12.0 // indirect - golang.org/x/sync v0.5.0 // indirect - golang.org/x/term v0.15.0 // indirect + golang.org/x/net v0.20.0 // indirect + golang.org/x/oauth2 v0.16.0 // indirect + golang.org/x/sync v0.6.0 // indirect + golang.org/x/term v0.16.0 // indirect golang.org/x/text v0.14.0 // indirect - golang.org/x/tools v0.16.1 // indirect - google.golang.org/appengine v1.6.7 // indirect - google.golang.org/protobuf v1.31.0 // indirect + golang.org/x/tools v0.17.0 // indirect + google.golang.org/appengine v1.6.8 // indirect + google.golang.org/protobuf v1.32.0 // indirect gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect + gopkg.in/evanphx/json-patch.v5 v5.9.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect - k8s.io/klog/v2 v2.110.1 // indirect - k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00 // indirect + k8s.io/klog/v2 v2.120.1 // indirect + k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec // indirect sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect - sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3 // indirect - sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3 // indirect + sigs.k8s.io/kustomize/api v0.16.0 // indirect + sigs.k8s.io/kustomize/kyaml v0.16.0 // indirect sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect ) diff --git a/go.sum b/go.sum index 361fc1a7..e5268dc9 100644 --- a/go.sum +++ b/go.sum @@ -1,6 +1,8 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= +github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0= +github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= @@ -26,20 +28,30 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/deepmap/oapi-codegen/v2 v2.0.0 h1:3TS7w3r+XnjKFXcbFbc16pTWzfTy0OLPkCsutEHjWDA= github.com/deepmap/oapi-codegen/v2 v2.0.0/go.mod h1:7zR+ZL3WzLeCkr2k8oWTxEa0v8y/F25ane0l6A5UjLA= +github.com/deepmap/oapi-codegen/v2 v2.1.0 h1:I/NMVhJCtuvL9x+S2QzZKpSjGi33oDZwPRdemvOZWyQ= +github.com/deepmap/oapi-codegen/v2 v2.1.0/go.mod h1:R1wL226vc5VmCNJUvMyYr3hJMm5reyv25j952zAVXZ8= github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= +github.com/emicklei/go-restful/v3 v3.11.2 h1:1onLa9DcsMYO9P+CXaL0dStDqQ2EHHXLiz+BtnqkLAU= +github.com/emicklei/go-restful/v3 v3.11.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84= github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/evanphx/json-patch v5.9.0+incompatible h1:fBXyNpNMuTTDdquAq/uisOr2lShz4oaXpDTX2bLe7ls= +github.com/evanphx/json-patch v5.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mNrPhs8= github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM= github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA= github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= +github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk= +github.com/go-errors/errors v1.5.1/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= github.com/go-logr/logr v1.3.0 h1:2y3SDp0ZXuc6/cjLSZ+Q3ir+QB9T/iG5yYRXqsagWSY= github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ= +github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= @@ -47,9 +59,13 @@ github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbX github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs= github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= +github.com/go-openapi/jsonreference v0.20.4 h1:bKlDxQxQJgwpUSgOENiMPzCTBVuc7vTdXSSgNeAhojU= +github.com/go-openapi/jsonreference v0.20.4/go.mod h1:5pZJyJP2MnYCpoeoMAql78cCHauHj0V9Lhc506VOpw4= github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= github.com/go-openapi/swag v0.22.8 h1:/9RjDSQ0vbFR+NyjGMkFTsA1IA0fmhKSThmfGZjicbw= github.com/go-openapi/swag v0.22.8/go.mod h1:6QT22icPLEqAM/z/TChgb4WAveCHF92+2gF0CNjHpPI= +github.com/go-openapi/swag v0.22.9 h1:XX2DssF+mQKM2DHsbgZK74y/zj4mo9I99+89xUmuZCE= +github.com/go-openapi/swag v0.22.9/go.mod h1:3/OXnFfnMAwBD099SwYRk7GD3xOrr1iL7d/XNLXVVwE= github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= @@ -68,10 +84,13 @@ github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:W github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= +github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4= github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA= +github.com/google/btree v1.1.2 h1:xf4v41cLI2Z6FxbKm+8Bu+m8ifhj15JuZ9sa0jZCMUU= +github.com/google/btree v1.1.2/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= @@ -93,6 +112,8 @@ github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaU github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU= github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ= github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= @@ -100,10 +121,14 @@ github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/ github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= +github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 h1:+ngKgrYPPJrOjhax5N+uePQ0Fh1Z7PheYoUI/0nzkPA= +github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb h1:tsEKRC3PU9rMw18w/uAptoijhgG4EvlA5kfJPtwrMDk= github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb/go.mod h1:NtmN9h8vrTveVQRLHcX2HQ5wIPBDCsZ351TGbZWgg38= github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= +github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4= +github.com/imdario/mergo v0.3.16/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY= @@ -132,6 +157,8 @@ github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8 github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= github.com/moby/term v0.0.0-20221205130635-1aeaba878587 h1:HfkjXDfhgVaN5rmueG8cL8KKeFNecRCXFhaJ2qZ5SKA= github.com/moby/term v0.0.0-20221205130635-1aeaba878587/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= +github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0= +github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -162,6 +189,8 @@ github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= github.com/pion/dtls/v2 v2.2.9 h1:K+D/aVf9/REahQvqk6G5JavdrD8W1PWDKC11UlwN7ts= github.com/pion/dtls/v2 v2.2.9/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= +github.com/pion/dtls/v2 v2.2.10 h1:u2Axk+FyIR1VFTPurktB+1zoEPGIW3bmyj3LEFrXjAA= +github.com/pion/dtls/v2 v2.2.10/go.mod h1:d9SYc9fch0CqK90mRk1dC7AkzzpwJj6u2GU3u+9pqFE= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= @@ -170,6 +199,8 @@ github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= +github.com/pion/transport/v2 v2.2.4 h1:41JJK6DZQYSeVLxILA2+F4ZkKb4Xd/tFJZRFZQ9QAlo= +github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= github.com/pion/turn/v3 v3.0.1 h1:wLi7BTQr6/Q20R0vt/lHbjv6y4GChFtC33nkYbasoT8= @@ -185,6 +216,8 @@ github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cY github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM= github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY= +github.com/prometheus/common v0.46.0 h1:doXzt5ybi1HBKpsZOL0sSkaNHJJqkyfEWZGGqqScV0Y= +github.com/prometheus/common v0.46.0/go.mod h1:Tp0qkxpb9Jsg54QMe+EAmqXkSV7Evdy1BTn+g2pa/hQ= github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= @@ -218,6 +251,8 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= go.starlark.net v0.0.0-20230525235612-a134d8f9ddca h1:VdD38733bfYv5tUZwEIskMM93VanwNIi5bIKnDrJdEY= go.starlark.net v0.0.0-20230525235612-a134d8f9ddca/go.mod h1:jxU+3+j+71eXOW14274+SmmuW82qJzl6iZSeqEtTGds= +go.starlark.net v0.0.0-20240123142251-f86470692795 h1:LmbG8Pq7KDGkglKVn8VpZOZj6vb9b8nKEGcg9l03epM= +go.starlark.net v0.0.0-20240123142251-f86470692795/go.mod h1:LcLNIzVOMp4oV+uusnpk+VU+SzXaJakUuBjoCSWH5dM= go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo= go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= @@ -232,6 +267,8 @@ golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= golang.org/x/crypto v0.17.0 h1:r8bRNjWL3GshPW3gkd+RpvzWrZAwPS49OmTGZ/uhM4k= golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= +golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= +golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= @@ -261,9 +298,13 @@ golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c= golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U= +golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo= +golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4= golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4= +golang.org/x/oauth2 v0.16.0 h1:aDkGMBSYxElaoP81NpoUoz2oo2R2wHdZpGToUxfyQrQ= +golang.org/x/oauth2 v0.16.0/go.mod h1:hqZ+0LWXsiVoZpeld6jVt06P3adbS2Uu911W1SsJv2o= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -273,6 +314,8 @@ golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE= golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ= +golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -298,10 +341,13 @@ golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= golang.org/x/term v0.15.0 h1:y/Oo/a/q3IXu26lQgl04j/gjuBDOBlx7X6Om1j2CPW4= golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= +golang.org/x/term v0.16.0 h1:m+B6fahuftsE9qjo0VWp2FW0mB3MTJvR0BaMQrq0pmE= +golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= @@ -321,6 +367,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/tools v0.16.1 h1:TLyB3WofjdOEepBHAU20JdNC1Zbg87elYofWYAY5oZA= golang.org/x/tools v0.16.1/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0= +golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc= +golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -331,6 +379,8 @@ google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9Ywl google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAsM= +google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= @@ -349,11 +399,15 @@ google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp0 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I= +google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 h1:FVCohIoYO7IJoDDVpV2pdq7SgrMH6wHnuTyrdrxJNoY= gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0/go.mod h1:OdE7CF6DbADk7lN8LIKRzRJTTZXIjtWgA5THM5lhBAw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/evanphx/json-patch.v5 v5.9.0 h1:hx1VU2SGj4F8r9b8GUwJLdc8DNO8sy79ZGui0G05GLo= +gopkg.in/evanphx/json-patch.v5 v5.9.0/go.mod h1:/kvTRh1TVm5wuM6OkHxqXtE/1nUZZpihg29RtuIyfvk= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= @@ -380,8 +434,12 @@ k8s.io/client-go v0.29.1 h1:19B/+2NGEwnFLzt0uB5kNJnfTsbV8w6TgQRz9l7ti7A= k8s.io/client-go v0.29.1/go.mod h1:TDG/psL9hdet0TI9mGyHJSgRkW3H9JZk2dNEUS7bRks= k8s.io/klog/v2 v2.110.1 h1:U/Af64HJf7FcwMcXyKm2RPM22WZzyR7OSpYj5tg3cL0= k8s.io/klog/v2 v2.110.1/go.mod h1:YGtd1984u+GgbuZ7e08/yBuAfKLSO0+uR1Fhi6ExXjo= +k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw= +k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00 h1:aVUu9fTY98ivBPKR9Y5w/AuzbMm96cd3YHRTU83I780= k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00/go.mod h1:AsvuZPBlUDVuCdzJ87iajxtXuR9oktsTctW/R9wwouA= +k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec h1:iGTel2aR8vCZdxJDgmbeY0zrlXy9Qcvyw4R2sB4HLrA= +k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec/go.mod h1:Pa1PvrP7ACSkuX6I7KYomY6cmMA0Tx86waBhDUgoKPw= k8s.io/utils v0.0.0-20240102154912-e7106e64919e h1:eQ/4ljkx21sObifjzXwlPKpdGLrCfRziVtos3ofG/sQ= k8s.io/utils v0.0.0-20240102154912-e7106e64919e/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= sigs.k8s.io/controller-runtime v0.12.3 h1:FCM8xeY/FI8hoAfh/V4XbbYMY20gElh9yh+A98usMio= @@ -390,8 +448,12 @@ sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMm sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3 h1:XX3Ajgzov2RKUdc5jW3t5jwY7Bo7dcRm+tFxT+NfgY0= sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3/go.mod h1:9n16EZKMhXBNSiUC5kSdFQJkdH3zbxS/JoO619G1VAY= +sigs.k8s.io/kustomize/api v0.16.0 h1:/zAR4FOQDCkgSDmVzV2uiFbuy9bhu3jEzthrHCuvm1g= +sigs.k8s.io/kustomize/api v0.16.0/go.mod h1:MnFZ7IP2YqVyVwMWoRxPtgl/5hpA+eCCrQR/866cm5c= sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3 h1:W6cLQc5pnqM7vh3b7HvGNfXrJ/xL6BDMS0v1V/HHg5U= sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3/go.mod h1:JWP1Fj0VWGHyw3YUPjXSQnRnrwezrZSrApfX5S0nIag= +sigs.k8s.io/kustomize/kyaml v0.16.0 h1:6J33uKSoATlKZH16unr2XOhDI+otoe2sR3M8PDzW3K0= +sigs.k8s.io/kustomize/kyaml v0.16.0/go.mod h1:xOK/7i+vmE14N2FdFyugIshB8eF6ALpy7jI87Q2nRh4= sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08= sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E= diff --git a/pkg/config/api/stunner_openapi.yaml b/pkg/config/api/stunner_openapi.yaml index e983d432..66195d87 100644 --- a/pkg/config/api/stunner_openapi.yaml +++ b/pkg/config/api/stunner_openapi.yaml @@ -26,11 +26,8 @@ paths: application/json: schema: $ref: '#/components/schemas/v1.ConfigList' - # application/json;stream=watch: - # schema: - # $ref: '#/components/schemas/v1.ConfigList' - "500": - description: Internal server error. + default: + description: Unexpected error content: application/json: schema: @@ -58,21 +55,12 @@ paths: application/json: schema: $ref: '#/components/schemas/v1.ConfigList' - "400": - description: Bad request. - content: - application/json: - schema: - $ref: '#/components/schemas/v1.Error' - "500": - description: Internal server error. + default: + description: Unexpected error content: application/json: schema: $ref: '#/components/schemas/v1.Error' - # application/json;stream=watch: - # schema: - # $ref: '#/components/schemas/v1.ConfigList' /api/v1/configs/{namespace}/{name}: get: description: API to list or watch config objects for a specific gateway in a specific namespace. @@ -107,8 +95,8 @@ paths: application/json: schema: $ref: '#/components/schemas/v1.Config' - "400": - description: Bad request. + "404": + description: Not found. content: application/json: schema: @@ -119,9 +107,12 @@ paths: application/json: schema: $ref: '#/components/schemas/v1.Error' - # application/json;stream=watch: - # schema: - # $ref: '#/components/schemas/v1.Config' + default: + description: Unexpected error + content: + application/json: + schema: + $ref: '#/components/schemas/v1.Error' components: schemas: v1.ClientInfo: @@ -162,6 +153,7 @@ components: type: object required: - code + - message properties: code: description: Error code. diff --git a/pkg/config/client/api/client.gen.go b/pkg/config/client/api/client.gen.go index 35d3965b..f4b100ca 100644 --- a/pkg/config/client/api/client.gen.go +++ b/pkg/config/client/api/client.gen.go @@ -1,6 +1,6 @@ // Package api provides primitives to interact with the openapi HTTP API. // -// Code generated by github.com/deepmap/oapi-codegen/v2 version v2.0.0 DO NOT EDIT. +// Code generated by github.com/deepmap/oapi-codegen/v2 version v2.1.0 DO NOT EDIT. package api import ( @@ -34,7 +34,7 @@ type V1Error struct { Code int32 `json:"code"` // Message Error message. - Message *string `json:"message,omitempty"` + Message string `json:"message"` } // ListV1ConfigsParams defines parameters for ListV1Configs. @@ -418,7 +418,7 @@ type ListV1ConfigsResponse struct { Body []byte HTTPResponse *http.Response JSON200 *V1ConfigList - JSON500 *V1Error + JSONDefault *V1Error } // Status returns HTTPResponse.Status @@ -441,8 +441,7 @@ type ListV1ConfigsNamespaceResponse struct { Body []byte HTTPResponse *http.Response JSON200 *V1ConfigList - JSON400 *V1Error - JSON500 *V1Error + JSONDefault *V1Error } // Status returns HTTPResponse.Status @@ -465,8 +464,9 @@ type GetV1ConfigNamespaceNameResponse struct { Body []byte HTTPResponse *http.Response JSON200 *V1Config - JSON400 *V1Error + JSON404 *V1Error JSON500 *V1Error + JSONDefault *V1Error } // Status returns HTTPResponse.Status @@ -533,12 +533,12 @@ func ParseListV1ConfigsResponse(rsp *http.Response) (*ListV1ConfigsResponse, err } response.JSON200 = &dest - case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 500: + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && true: var dest V1Error if err := json.Unmarshal(bodyBytes, &dest); err != nil { return nil, err } - response.JSON500 = &dest + response.JSONDefault = &dest } @@ -566,19 +566,12 @@ func ParseListV1ConfigsNamespaceResponse(rsp *http.Response) (*ListV1ConfigsName } response.JSON200 = &dest - case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 400: - var dest V1Error - if err := json.Unmarshal(bodyBytes, &dest); err != nil { - return nil, err - } - response.JSON400 = &dest - - case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 500: + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && true: var dest V1Error if err := json.Unmarshal(bodyBytes, &dest); err != nil { return nil, err } - response.JSON500 = &dest + response.JSONDefault = &dest } @@ -606,12 +599,12 @@ func ParseGetV1ConfigNamespaceNameResponse(rsp *http.Response) (*GetV1ConfigName } response.JSON200 = &dest - case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 400: + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 404: var dest V1Error if err := json.Unmarshal(bodyBytes, &dest); err != nil { return nil, err } - response.JSON400 = &dest + response.JSON404 = &dest case strings.Contains(rsp.Header.Get("Content-Type"), "json") && rsp.StatusCode == 500: var dest V1Error @@ -620,6 +613,13 @@ func ParseGetV1ConfigNamespaceNameResponse(rsp *http.Response) (*GetV1ConfigName } response.JSON500 = &dest + case strings.Contains(rsp.Header.Get("Content-Type"), "json") && true: + var dest V1Error + if err := json.Unmarshal(bodyBytes, &dest); err != nil { + return nil, err + } + response.JSONDefault = &dest + } return response, nil diff --git a/pkg/config/server/api/server.gen.go b/pkg/config/server/api/server.gen.go index 09db75ac..16a047ac 100644 --- a/pkg/config/server/api/server.gen.go +++ b/pkg/config/server/api/server.gen.go @@ -1,6 +1,6 @@ // Package api provides primitives to interact with the openapi HTTP API. // -// Code generated by github.com/deepmap/oapi-codegen/v2 version v2.0.0 DO NOT EDIT. +// Code generated by github.com/deepmap/oapi-codegen/v2 version v2.1.0 DO NOT EDIT. package api import ( @@ -40,7 +40,7 @@ type V1Error struct { Code int32 `json:"code"` // Message Error message. - Message *string `json:"message,omitempty"` + Message string `json:"message"` } // ListV1ConfigsParams defines parameters for ListV1Configs. @@ -123,7 +123,7 @@ func (siw *ServerInterfaceWrapper) ListV1ConfigsNamespace(w http.ResponseWriter, // ------------- Path parameter "namespace" ------------- var namespace string - err = runtime.BindStyledParameter("simple", false, "namespace", mux.Vars(r)["namespace"], &namespace) + err = runtime.BindStyledParameterWithOptions("simple", "namespace", mux.Vars(r)["namespace"], &namespace, runtime.BindStyledParameterOptions{Explode: false, Required: true}) if err != nil { siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "namespace", Err: err}) return @@ -160,7 +160,7 @@ func (siw *ServerInterfaceWrapper) GetV1ConfigNamespaceName(w http.ResponseWrite // ------------- Path parameter "namespace" ------------- var namespace string - err = runtime.BindStyledParameter("simple", false, "namespace", mux.Vars(r)["namespace"], &namespace) + err = runtime.BindStyledParameterWithOptions("simple", "namespace", mux.Vars(r)["namespace"], &namespace, runtime.BindStyledParameterOptions{Explode: false, Required: true}) if err != nil { siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "namespace", Err: err}) return @@ -169,7 +169,7 @@ func (siw *ServerInterfaceWrapper) GetV1ConfigNamespaceName(w http.ResponseWrite // ------------- Path parameter "name" ------------- var name string - err = runtime.BindStyledParameter("simple", false, "name", mux.Vars(r)["name"], &name) + err = runtime.BindStyledParameterWithOptions("simple", "name", mux.Vars(r)["name"], &name, runtime.BindStyledParameterOptions{Explode: false, Required: true}) if err != nil { siw.ErrorHandlerFunc(w, r, &InvalidParamFormatError{ParamName: "name", Err: err}) return @@ -344,13 +344,16 @@ func (response ListV1Configs200JSONResponse) VisitListV1ConfigsResponse(w http.R return json.NewEncoder(w).Encode(response) } -type ListV1Configs500JSONResponse V1Error +type ListV1ConfigsdefaultJSONResponse struct { + Body V1Error + StatusCode int +} -func (response ListV1Configs500JSONResponse) VisitListV1ConfigsResponse(w http.ResponseWriter) error { +func (response ListV1ConfigsdefaultJSONResponse) VisitListV1ConfigsResponse(w http.ResponseWriter) error { w.Header().Set("Content-Type", "application/json") - w.WriteHeader(500) + w.WriteHeader(response.StatusCode) - return json.NewEncoder(w).Encode(response) + return json.NewEncoder(w).Encode(response.Body) } type ListV1ConfigsNamespaceRequestObject struct { @@ -371,22 +374,16 @@ func (response ListV1ConfigsNamespace200JSONResponse) VisitListV1ConfigsNamespac return json.NewEncoder(w).Encode(response) } -type ListV1ConfigsNamespace400JSONResponse V1Error - -func (response ListV1ConfigsNamespace400JSONResponse) VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(400) - - return json.NewEncoder(w).Encode(response) +type ListV1ConfigsNamespacedefaultJSONResponse struct { + Body V1Error + StatusCode int } -type ListV1ConfigsNamespace500JSONResponse V1Error - -func (response ListV1ConfigsNamespace500JSONResponse) VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error { +func (response ListV1ConfigsNamespacedefaultJSONResponse) VisitListV1ConfigsNamespaceResponse(w http.ResponseWriter) error { w.Header().Set("Content-Type", "application/json") - w.WriteHeader(500) + w.WriteHeader(response.StatusCode) - return json.NewEncoder(w).Encode(response) + return json.NewEncoder(w).Encode(response.Body) } type GetV1ConfigNamespaceNameRequestObject struct { @@ -408,11 +405,11 @@ func (response GetV1ConfigNamespaceName200JSONResponse) VisitGetV1ConfigNamespac return json.NewEncoder(w).Encode(response) } -type GetV1ConfigNamespaceName400JSONResponse V1Error +type GetV1ConfigNamespaceName404JSONResponse V1Error -func (response GetV1ConfigNamespaceName400JSONResponse) VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error { +func (response GetV1ConfigNamespaceName404JSONResponse) VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error { w.Header().Set("Content-Type", "application/json") - w.WriteHeader(400) + w.WriteHeader(404) return json.NewEncoder(w).Encode(response) } @@ -426,6 +423,18 @@ func (response GetV1ConfigNamespaceName500JSONResponse) VisitGetV1ConfigNamespac return json.NewEncoder(w).Encode(response) } +type GetV1ConfigNamespaceNamedefaultJSONResponse struct { + Body V1Error + StatusCode int +} + +func (response GetV1ConfigNamespaceNamedefaultJSONResponse) VisitGetV1ConfigNamespaceNameResponse(w http.ResponseWriter) error { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(response.StatusCode) + + return json.NewEncoder(w).Encode(response.Body) +} + // StrictServerInterface represents all server handlers. type StrictServerInterface interface { @@ -439,8 +448,8 @@ type StrictServerInterface interface { GetV1ConfigNamespaceName(ctx context.Context, request GetV1ConfigNamespaceNameRequestObject) (GetV1ConfigNamespaceNameResponseObject, error) } -type StrictHandlerFunc = strictnethttp.StrictHttpHandlerFunc -type StrictMiddlewareFunc = strictnethttp.StrictHttpMiddlewareFunc +type StrictHandlerFunc = strictnethttp.StrictHTTPHandlerFunc +type StrictMiddlewareFunc = strictnethttp.StrictHTTPMiddlewareFunc type StrictHTTPServerOptions struct { RequestErrorHandlerFunc func(w http.ResponseWriter, r *http.Request, err error) @@ -552,22 +561,23 @@ func (sh *strictHandler) GetV1ConfigNamespaceName(w http.ResponseWriter, r *http // Base64 encoded, gzipped, json marshaled Swagger object var swaggerSpec = []string{ - "H4sIAAAAAAAC/+xXUW/bNhD+KwS3R0eKmw0D9LSuKIagQzYk7vpQ5IGmzhI7iWSPJ6dBoP8+HElLbuw0", - "BVIUKNo3ijzefXf3fWf6TmrXe2fBUpDVnQy6hV7F5XZZvHB2Yxr+qCFoNJ6Ms7KSaV94dFtTQxBKXK1e", - "X1hAoeNJIa6iH2GCqGFjLNTCWNES+VCVZWOoHdaFdn3Z/db7MtBgLWBJCFD2ytjS/9eUyptQbpdyIT+c", - "NO6Ebj3ISmbb7bK4SquMcc/qxPTeITFsq/qPLsmF9IpaWcmHMOxHHsfFXIW/TKCHKsFnnKsSHa/cRqT9", - "UHBAdB6QDMSqGoI+HPo55212QS3ccyLc+h1oClzB3Sn7nTz9jLCRlfypnFtZ5j6WcxPHhcwlVIjqlr+3", - "gCGGv48mH+TeJVB5D2qRfDNAak3I8BjR1CI0tonVQ3g/GIRaVm+naDvk19OF5EGmar9EdHgI6fk/5wL4", - "6LCk2tVweCH6EXzGNzYOe0WyksbS2bMZq7EEDSDH7iEE1TzoKR8/nmeEc5gcmxm7cYf+Vy1M+qkVKd8p", - "CyIm/MG7EPWFkJQ1oFp3IPqhI3Pi0ZHTrsufTAxgJ6vXlxciAG6NBrFxKN7A+nL1QvRQGyWMbSBw6JiK", - "oY5RXr68WsWQbH4IRu9ENlFGLovT4pTr5jxY5Y2s5FlxWpxlicXOsJLK7bJM1+NWA3S8ueQy71HcKNJt", - "jrnPftV1ghUdvNIQtcU0UOzkvJaVZBn+u8zSizhQ9UCAQVZv78d8E4NwurpVXBJGwERPZmuoBUJwA2pu", - "gOUvGjBKsBeKexIIQfWsA1XXCzH4WhEssm3vtiCsI7MxOiKMeA1Hfj8A3srFbjjFbOUij14uTmbO2rkO", - "lJXjeM0MC97ZkBj/7PQ0Ed8S2FhP5X2XA5XvQtL07PCzJkQccJGmHxfq71fc5V+/bMik8iPRzi0BWtVF", - "/gLuND9G03t8Ku8mNoxP5pYIHjR3a6bYIwy72Nk9RrXJMM1MEI0iuFG3kX03rZkBmSB4lEAgqCe+xF+s", - "iS52L+o8dggHOEKheUT9oP+T6P/LV6L/H6reMaD4FmSX1k9TH5NwT347cXy2KP+ESZOT0njxLcqSMX0p", - "ON/PgPhkGa2rIS50Z8CSwMEGkV4/x8KxufxUpb7KNPoxiY5MonEh02ZS84CdrKTkhpBqjin8uXg1rAEt", - "EIT8+t0XVXoWF/w/i5SmSvD7/Hf+O1gYNzMiP4fleD3+HwAA//9wiPtGLw8AAA==", + "H4sIAAAAAAAC/+xXTW/cNhD9KwTboyztJikK6NQgCAojhVv4ozkEPnClkcRUIpnhaG3D0H8vhuRKG6+3", + "DhAj6NeN4se8x5n3Zrn3srKDswYMeVneS191MKgw3K7zN9Y0uuWPGnyF2pG2RpYyzguHdqtr8EKJi8ur", + "MwMoqrCSi4sQR2gvami0gVpoIzoi58uiaDV14yav7FD0Pw6u8DQaA1gQAhSD0qZwf7SFctoX27XM5O1J", + "a0/ozoEsZdq7XecXcZQ47u060YOzSEzbqOGzQzKTTlEnS3mMwz7yNGVLFn7Rno5lgtf4rkr0PLKNiPM+", + "Z0C0DpA0hKxqgsEfxjnlaQ5BHTwIIuzmI1TkOYO7VY47R/oeoZGl/K5YSlmkOhZLEadMphQqRHXH31tA", + "H+AfskkLqXaRVJqDWsTYTJA67RM9ZjSXCLVpQ/YQPo0aoZblhxltx/x6PhAjyJjtt4gWDym9/u1UAC8d", + "prSyNRweCHEEr/GJxuKgSJZSG3r5YuGqDUELyNgDeK/ao5HS8tP3DHSWcIfX5APaNPYQ6bKD2Um1IuV6", + "ZUCEq98664PTEKLHRlSbHsQw9qRPHFqyle3TJ0sEOMjl1fmZ8IBbXYFoLIr3sDm/fCMGqLUS2rTgGTpc", + "SlPPLM/fXlwGSN5+SKba2W0Wj1znq3zFGbQOjHJalvJlvspfJrOFGrGniu26iMfDVAv0eJnJJgeguFFU", + "dQlz3weq7wV72ztVQXAZC0JxkNNalpIN+fs6mTDwQDUAAXpZfniI+T6A8HWrTnFKmAFLPm7bQC0QvB2x", + "4gIY/qIRgxkHobgmnhDUwI5QdZ2J0dWKIEt7B7sFYSzpRleBYeCrGfnTCHgns12bCreVWWrCnJyknI21", + "PSgjp+mateadNT5q/8VqFS1gCEzIp3KuT0DFRx/dvQT8ol4RWl2Q6eeJ+vWdDHONGnt6Ttjo+UcQrwzc", + "OqgI6mh+3jNlD9VU3M9amL5aWcI7qLhWi8Ce0NfZbt9TQps3xt4JolUEN+ouaO+m0wsh7QW3FPAE9ayW", + "8Ms1i8XsoS7th3CERwS0tKr/xf8vFn8cf50HWAp7JthJ9Iut8TPMzpj1zoN/ojmY03PR+e/Y9C/TaGwN", + "YVD1GgwJHI0X8QXyGJyJj6njmfomPeF4P3i1evVNesGZJdHY0dQ5o/7wvJc8inpqCNCoPjwhAXcP8L9D", + "H5wyGUnFXjJiL0spWQ6k2sf6y2vxbtwAGiDw6f27b+n4MM75PxepikrBL/Sf+K9hru2ix/QgltP19GcA", + "AAD//2H3j/07DwAA", } // GetSwagger returns the content of the embedded swagger specification file diff --git a/pkg/config/server/handler.go b/pkg/config/server/handler.go index a94deb97..5f72e2e6 100644 --- a/pkg/config/server/handler.go +++ b/pkg/config/server/handler.go @@ -3,6 +3,7 @@ package server import ( "context" "fmt" + "net/http" "strings" "github.com/l7mp/stunner/pkg/config/server/api" @@ -68,8 +69,11 @@ func (s *Server) GetV1ConfigNamespaceName(ctx context.Context, request api.GetV1 id := fmt.Sprintf("%s/%s", namespace, name) c := s.configs.Get(id) if c == nil { - s.log.V(1).Info("GetV1ConfigNamespaceName: config not found", "client", id) - return nil, fmt.Errorf("config not found for id %q", id) + s.log.V(1).Info("GetV1ConfigNamespaceName: Config not found", "client", id) + return api.GetV1ConfigNamespaceName404JSONResponse{ + Code: http.StatusNotFound, + Message: fmt.Sprintf("Config not found for ID %q", id), + }, nil } ret := &stnrv1.StunnerConfig{} @@ -79,7 +83,10 @@ func (s *Server) GetV1ConfigNamespaceName(ctx context.Context, request api.GetV1 conf, err := s.patch(ret, *request.Params.Node) if err != nil { s.log.Error(err, "GetV1ConfigNamespaceName: patch config failed") - return nil, err + return api.GetV1ConfigNamespaceName500JSONResponse{ + Code: http.StatusInternalServerError, + Message: fmt.Sprintf("Config patch failed: %s", err.Error()), + }, nil } ret = conf } From 8a9c7adf9c8172734318bc72a9ec94e51b3d4cc8 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 6 Feb 2024 23:46:59 +0100 Subject: [PATCH 127/243] test: Run CDS client-server tests on a random port --- pkg/config/cds_test.go | 68 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 60 insertions(+), 8 deletions(-) diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index 401d49fb..001884ba 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -2,6 +2,8 @@ package server import ( "context" + "fmt" + "math/rand" "testing" "time" @@ -23,7 +25,11 @@ var testerLogLevel = zapcore.ErrorLevel // const stunnerLogLevel = "all:TRACE" const stunnerLogLevel = "all:ERROR" -const testCDSAddr = ":63478" +// run on random port +func getRandCDSAddr() string { + rndPort := rand.Intn(10000) + 50000 + return fmt.Sprintf(":%d", rndPort) +} func init() { // setup a fast pinger so that we get a timely error notification @@ -44,10 +50,16 @@ func TestServerLoad(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // suppress deletions + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = 0 + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + ctx, cancel := context.WithCancel(context.Background()) defer cancel() - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(ctx) @@ -132,10 +144,16 @@ func TestServerPoll(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // suppress deletions + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = 0 + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + ctx, cancel := context.WithCancel(context.Background()) defer cancel() - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(ctx) @@ -235,9 +253,15 @@ func TestServerWatch(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // suppress deletions + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = 0 + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + serverCtx, serverCancel := context.WithCancel(context.Background()) - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) @@ -411,10 +435,16 @@ func TestServerWatchBootstrap(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // suppress deletions + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = 0 + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + serverCtx, serverCancel := context.WithCancel(context.Background()) defer serverCancel() - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) @@ -504,9 +534,15 @@ func TestServerAPI(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // suppress deletions + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = time.Millisecond + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + serverCtx, serverCancel := context.WithCancel(context.Background()) - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) @@ -935,13 +971,19 @@ func TestClientReconnect(t *testing.T) { zlogger := zapr.NewLogger(z) log := zlogger.WithName("tester") + // suppress deletions + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = 0 + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") serverCtx, serverCancel := context.WithCancel(context.Background()) defer serverCancel() - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) @@ -1010,10 +1052,16 @@ func TestServerUpdate(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // make sure deletions are suppressed + deleteDelay := server.ConfigDeletionUpdateDelay + server.ConfigDeletionUpdateDelay = 0 + defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + serverCtx, serverCancel := context.WithCancel(context.Background()) defer serverCancel() - testLog.Debug("create server") + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) srv := server.New(testCDSAddr, nil, log) assert.NotNil(t, srv, "server") err = srv.Start(serverCtx) @@ -1023,6 +1071,7 @@ func TestServerUpdate(t *testing.T) { assert.NoError(t, oldC.Validate(), "validate") assert.NoError(t, err, "parse 1") + testLog.Debug("upsert stunner/udp-gateway") srv.UpsertConfig("stunner/udp-gateway", oldC) cs := srv.GetConfigStore().Snapshot() @@ -1033,6 +1082,7 @@ func TestServerUpdate(t *testing.T) { assert.True(t, sc1.DeepEqual(oldC), "deepeq") // reapply - no change + testLog.Debug("re-apply stunner/udp-gateway") srv.UpsertConfig("stunner/udp-gateway", oldC) time.Sleep(20 * time.Millisecond) // let the server process @@ -1048,6 +1098,7 @@ func TestServerUpdate(t *testing.T) { assert.NoError(t, tcpC.Validate(), "validate") assert.NoError(t, err, "parse") + testLog.Debug("upsert stunner/tcp-gateway") srv.UpsertConfig("stunner/tcp-gateway", tcpC) time.Sleep(20 * time.Millisecond) // let the server process @@ -1069,6 +1120,7 @@ func TestServerUpdate(t *testing.T) { assert.False(t, oldC.DeepEqual(newC), "deepeq") // process in a single go + testLog.Debug("modify stunner/udp-gateway using UpdateConfig") err = srv.UpdateConfig([]server.Config{{Id: "stunner/udp-gateway", Config: newC}, {Id: "stunner/tcp-gateway", Config: tcpC}}) assert.NoError(t, err, "parse 1") From ef76f94220af9e5c96e0cd8207d0ad3676af2eaf Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 6 Feb 2024 23:48:27 +0100 Subject: [PATCH 128/243] fix: Make sure CDS clients get an update on config deletion So far we have suppressed CDS updates on config deletion to avoid interference with graceful shutdown. This commit makes sure the update is indeed sent (actually we send a zero-config after a configurable delay, 5 sec by default). Since stunnerd graceful shutdown now disables the config watcher, the deleted configs will not cause a premature termination of active listeners. --- pkg/config/cds_test.go | 11 ++++++++--- pkg/config/server/config.go | 25 +++++++++++++++++++------ pkg/config/server/server.go | 28 ++++++++++++++++++++++++++-- 3 files changed, 53 insertions(+), 11 deletions(-) diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index 001884ba..fdfb6a08 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -939,19 +939,24 @@ func TestServerAPI(t *testing.T) { s2 = watchConfig(ch1, 500*time.Millisecond) assert.NotNil(t, s2) s3 = watchConfig(ch1, 500*time.Millisecond) - assert.Nil(t, s3) - lst = []*stnrv1.StunnerConfig{s1, s2} + assert.NotNil(t, s3) + lst = []*stnrv1.StunnerConfig{s1, s2, s3} assert.NotNil(t, findConfById(lst, "ns1/gw1")) assert.True(t, findConfById(lst, "ns1/gw1").DeepEqual(sc1), "deepeq") assert.NotNil(t, findConfById(lst, "ns3/gw1")) assert.True(t, findConfById(lst, "ns3/gw1").DeepEqual(sc4), "deepeq") + assert.NotNil(t, findConfById(lst, "ns1/gw2")) + gw2zero := client.ZeroConfig("ns1/gw2") // deleted! + assert.NoError(t, gw2zero.Validate()) + assert.True(t, findConfById(lst, "ns1/gw2").DeepEqual(gw2zero), "deepeq") // 1 config from client2 watch (removed config never updated) s1 = watchConfig(ch2, 50*time.Millisecond) assert.NotNil(t, s1) s2 = watchConfig(ch2, 50*time.Millisecond) - assert.Nil(t, s2) + assert.NotNil(t, s2) assert.True(t, s1.DeepEqual(sc1), "deepeq") + assert.True(t, s2.DeepEqual(gw2zero), "deepeq") // deleted! // no config from client3 watch s = watchConfig(ch3, 50*time.Millisecond) diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go index 1571ed8a..fe0b6c4a 100644 --- a/pkg/config/server/config.go +++ b/pkg/config/server/config.go @@ -5,6 +5,7 @@ import ( "sync" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/config/client/api" ) @@ -27,14 +28,26 @@ func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { s.configCh <- Config{Id: id, Config: cpy} } -// DeleteConfig should remove a config from the client. Theoretically, this would be done by -// sending the client a zero-config. However, in order to avoid that a client being removed and -// entering the graceful shutdown cycle receive a zeroconfig and abruprly kill all listeners with -// all active connections allocated to it, currently we suppress the config update. +// DeleteConfig removes a config from the client. Theoretically, this should send the client a +// zero-config immediately. However, in order to avoid that a client being removed and entering the +// graceful shutdown cycle receive a zeroconfig and abruprly kill all listeners with all active +// connections allocated to them, we actually delay sending the zeroconfig with a configurable time +// (default is 5 sec, but a zero delay will suppress sending the zero-config all together). This +// should allow the client comfortable time to enter the grafeul shutdown cycle. Note that clients +// should stop actively reconciling config updates once they initiated graceful shutdown for this +// to work. func (s *Server) DeleteConfig(id string) { s.configs.Delete(id) - s.log.Info("suppressing config update for terminating client", "client", id) - // s.configCh <- Config{Id: id, Config: client.ZeroConfig(id)} + if ConfigDeletionUpdateDelay == 0 { + s.log.Info("suppressing config update for deleted config", + "client", id) + return + } + + s.log.Info("delaying config update for deleted config", + "client", id, "delay", ConfigDeletionUpdateDelay) + + s.deleteCh <- Config{Id: id, Config: client.ZeroConfig(id)} } // UpdateConfig receives a set of ids and newConfigs that represent the state-of-the-world at a diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index ab1a4193..bc25c732 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -7,6 +7,7 @@ import ( "errors" "net" "net/http" + "time" "github.com/go-logr/logr" "github.com/gorilla/mux" @@ -16,6 +17,13 @@ import ( stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) +var ( + // ConfigDeletionUpdateDelay is the delay between deleting a config from the server and + // sending the corresponing zero-config to the client. Set this to zero to suppress sending + // the zero-config all together. + ConfigDeletionUpdateDelay = 5 * time.Second +) + // Server is a generic config discovery server implementation. type Server struct { *http.Server @@ -24,6 +32,7 @@ type Server struct { conns *ConnTrack configs *ConfigStore configCh chan Config + deleteCh chan Config patch ConfigPatcher log logr.Logger } @@ -39,6 +48,7 @@ func New(addr string, patch ConfigPatcher, logger logr.Logger) *Server { conns: NewConnTrack(), configs: NewConfigStore(), configCh: make(chan Config, 8), + deleteCh: make(chan Config, 8), addr: addr, patch: patch, log: logger, @@ -67,12 +77,26 @@ func (s *Server) Start(ctx context.Context) error { go func() { defer close(s.configCh) + defer close(s.deleteCh) defer s.Close() for { select { - case e := <-s.configCh: - s.broadcastConfig(e) + case c := <-s.configCh: + s.broadcastConfig(c) + + case c := <-s.deleteCh: + // delayed config deletion + go func() { + select { + case <-ctx.Done(): + return + case <-time.After(ConfigDeletionUpdateDelay): + s.configCh <- Config{Id: c.Id, Config: c.Config} + return + } + }() + case <-ctx.Done(): return } From 8b7f5459ee31a222f7fd052f651d1cff67494114 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 7 Feb 2024 11:35:10 +0100 Subject: [PATCH 129/243] fix: Tight control of CDS server connection handler lifetime --- pkg/config/server/conn.go | 5 ++++- pkg/config/server/server.go | 17 +++++++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/pkg/config/server/conn.go b/pkg/config/server/conn.go index ca460880..74a962f1 100644 --- a/pkg/config/server/conn.go +++ b/pkg/config/server/conn.go @@ -1,6 +1,7 @@ package server import ( + "context" "fmt" "sync" @@ -15,15 +16,17 @@ type Conn struct { *websocket.Conn Filter ConfigFilter patch ClientConfigPatcher + cancel context.CancelFunc readLock, writeLock sync.Mutex // for writemessage } // NewConn wraps a WebSocket connection. -func NewConn(conn *websocket.Conn, filter ConfigFilter, patch ClientConfigPatcher) *Conn { +func NewConn(conn *websocket.Conn, filter ConfigFilter, patch ClientConfigPatcher, cancel context.CancelFunc) *Conn { return &Conn{ Conn: conn, Filter: filter, patch: patch, + cancel: cancel, } } diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index bc25c732..1f3d15db 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -140,8 +140,12 @@ func (s *Server) RemoveClient(id string) { } } -func (s *Server) handleConn(ctx context.Context, wsConn *websocket.Conn, operationID string, filter ConfigFilter, patch ClientConfigPatcher) { - conn := NewConn(wsConn, filter, patch) +func (s *Server) handleConn(reqCtx context.Context, wsConn *websocket.Conn, operationID string, filter ConfigFilter, patch ClientConfigPatcher) { + // since wsConn is hijacked, reqCtx is unreliable in that it may not be canceled when the + // connection is closed, so we create our own connection context that we can cancel + // explicitly + ctx, cancel := context.WithCancel(reqCtx) + conn := NewConn(wsConn, filter, patch, cancel) s.conns.Upsert(conn) // a dummy reader that drops everything it receives: this must be there for the @@ -210,8 +214,7 @@ func (s *Server) sendConfig(conn *Conn, e *stnrv1.StunnerConfig) { } func (s *Server) sendJSONConfig(conn *Conn, json []byte) { - s.log.V(2).Info("sending configuration to client", "client", conn.Id(), - "config", string(json)) + s.log.V(2).Info("sending configuration to client", "client", conn.Id()) if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { s.log.Error(err, "error sending config update", "client", conn.Id()) @@ -223,6 +226,12 @@ func (s *Server) closeConn(conn *Conn) { s.log.V(1).Info("closing client connection", "client", conn.Id()) conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck + + if conn.cancel != nil { + conn.cancel() + conn.cancel = nil // make sure we can cancel multiple times + } + s.conns.Delete(conn) conn.Close() } From 530320c53bc44c74fa5d10452cb5ce86376c2fe0 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 7 Feb 2024 13:59:14 +0100 Subject: [PATCH 130/243] feat: Report deleted configs in stunnerctl --- cmd/stunnerctl/main.go | 4 ++++ pkg/config/cds_test.go | 6 ++---- pkg/config/client/config.go | 16 ++++++++++++++++ pkg/config/server/server.go | 12 +++++------- 4 files changed, 27 insertions(+), 11 deletions(-) diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go index ed7f3ab4..a874f976 100644 --- a/cmd/stunnerctl/main.go +++ b/cmd/stunnerctl/main.go @@ -169,6 +169,10 @@ func runConfig(cmd *cobra.Command, args []string) error { } for c := range confChan { + if cdsclient.IsConfigDeleted(c) { + fmt.Printf("Gateway: %s \n", c.Admin.Name) + continue + } switch output { case "yaml": if out, err := yaml.Marshal(c); err != nil { diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index fdfb6a08..c53daa7b 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -946,9 +946,7 @@ func TestServerAPI(t *testing.T) { assert.NotNil(t, findConfById(lst, "ns3/gw1")) assert.True(t, findConfById(lst, "ns3/gw1").DeepEqual(sc4), "deepeq") assert.NotNil(t, findConfById(lst, "ns1/gw2")) - gw2zero := client.ZeroConfig("ns1/gw2") // deleted! - assert.NoError(t, gw2zero.Validate()) - assert.True(t, findConfById(lst, "ns1/gw2").DeepEqual(gw2zero), "deepeq") + assert.True(t, client.IsConfigDeleted(findConfById(lst, "ns1/gw2")), "deepeq") // 1 config from client2 watch (removed config never updated) s1 = watchConfig(ch2, 50*time.Millisecond) @@ -956,7 +954,7 @@ func TestServerAPI(t *testing.T) { s2 = watchConfig(ch2, 50*time.Millisecond) assert.NotNil(t, s2) assert.True(t, s1.DeepEqual(sc1), "deepeq") - assert.True(t, s2.DeepEqual(gw2zero), "deepeq") // deleted! + assert.True(t, client.IsConfigDeleted(s2), "deepeq") // deleted! // no config from client3 watch s = watchConfig(ch3, 50*time.Millisecond) diff --git a/pkg/config/client/config.go b/pkg/config/client/config.go index 001ceeba..f25916c2 100644 --- a/pkg/config/client/config.go +++ b/pkg/config/client/config.go @@ -122,3 +122,19 @@ func parseRaw(c []byte) (*stnrv1.StunnerConfig, error) { return &s, nil } + +// IsConfigDeleted is a helper that allows to decide whether a config is being deleted. When a +// config is being removed (say, because the corresponding Gateway is deleted), the CDS server +// sends a validated zero-config for the client. This function is a quick helper to decide whether +// the config received is such a zero-config. +func IsConfigDeleted(conf *stnrv1.StunnerConfig) bool { + if conf == nil { + return false + } + zeroConf := ZeroConfig(conf.Admin.Name) + // zeroconfs have to be explcitly validated before deepEq (the cds client validates) + if err := zeroConf.Validate(); err != nil { + return false + } + return conf.DeepEqual(zeroConf) +} diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index 1f3d15db..b900913e 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -86,7 +86,8 @@ func (s *Server) Start(ctx context.Context) error { s.broadcastConfig(c) case c := <-s.deleteCh: - // delayed config deletion + s.log.V(2).Info("initiating deleyed config deletion", "id", c.Id) + go func() { select { case <-ctx.Done(): @@ -206,15 +207,12 @@ func (s *Server) sendConfig(conn *Conn, e *stnrv1.StunnerConfig) { json, err := json.Marshal(c) if err != nil { - s.log.Error(err, "cannor JSON serialize config", "event", e.String()) + s.log.Error(err, "cannot JSON serialize config", "event", e.String()) return } - s.sendJSONConfig(conn, json) -} - -func (s *Server) sendJSONConfig(conn *Conn, json []byte) { - s.log.V(2).Info("sending configuration to client", "client", conn.Id()) + s.log.V(2).Info("sending configuration to client", "client", conn.Id(), + "config", c.String()) if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { s.log.Error(err, "error sending config update", "client", conn.Id()) From 65a9f1b4e3659c053817782b10cb1fc734be768e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 7 Feb 2024 16:08:54 +0100 Subject: [PATCH 131/243] fix: Make sure DNS domain names properly parse as a CDS server URI --- cmd/stunnerd/main.go | 2 +- config_test.go | 7 +++++-- pkg/config/client/client.go | 4 +++- pkg/config/client/client_test.go | 2 +- pkg/config/client/file_client.go | 3 +++ pkg/config/client/util.go | 15 +++++++++------ stunner_test.go | 1 - 7 files changed, 22 insertions(+), 12 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 20ecb82f..d3bdcf09 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -19,7 +19,7 @@ import ( func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", "Config origin, either a valid IP address or URL to the CDS server, or a file name (overrides: STUNNER_CONFIG_ORIGIN)") + var config = flag.StringP("config", "c", "", "Config origin, either a valid address in the format IP:port, or HTTP URL to the CDS server, or a proper file name URI in the format file:// (overrides: STUNNER_CONFIG_ORIGIN)") var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO)") var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: )") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false)") diff --git a/config_test.go b/config_test.go index a7b65acf..96c62086 100644 --- a/config_test.go +++ b/config_test.go @@ -153,7 +153,8 @@ func TestStunnerConfigFileWatcher(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - err = stunner.WatchConfig(ctx, file, conf) + url := "file://" + file + err = stunner.WatchConfig(ctx, url, conf) assert.NoError(t, err, "creating config watcher") // nothing should happen here: wait a bit so that the watcher has comfortable time to start @@ -266,7 +267,9 @@ func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { log.Debug("init watcher with nonexistent config file") ctx, cancel := context.WithCancel(context.Background()) defer cancel() - err = stunner.WatchConfig(ctx, file, conf) + + url := "file://" + file + err = stunner.WatchConfig(ctx, url, conf) assert.NoError(t, err, "creating config watcher") // nothing should happen here: wait a bit so that the watcher has comfortable time to start diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index 2579494b..1b38605c 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -42,7 +42,9 @@ type Client interface { fmt.Stringer } -// New creates a generic config client. +// New creates a generic config client. Origin is either a network address in the form +// ":" or a proper HTTP/WS URI, in which case a CDS client is returned, or a proper file +// URL "file:///" in which case a config file watcher is returned. func New(origin string, id string, logger logging.LoggerFactory) (Client, error) { u, err := getURI(origin) if err != nil { diff --git a/pkg/config/client/client_test.go b/pkg/config/client/client_test.go index 97765a17..16fb31bf 100644 --- a/pkg/config/client/client_test.go +++ b/pkg/config/client/client_test.go @@ -10,7 +10,7 @@ func TestParseURI(t *testing.T) { // file u, err := getURI("/tmp/a") assert.NoError(t, err, "file URI parse") - assert.Equal(t, "", u.Scheme, "file URI scheme") + assert.Equal(t, "http", u.Scheme, "file URI scheme") assert.Equal(t, "", u.Host, "file URI host") assert.Equal(t, "/tmp/a", u.Path, "file URI path") diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index b16172b3..dedd7a01 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "os" + "strings" "time" "github.com/fsnotify/fsnotify" @@ -25,6 +26,8 @@ type ConfigFileClient struct { // NewConfigFileClient creates a client that load or watch STUNner configurations from a local // file. func NewConfigFileClient(origin, id string, logger logging.LeveledLogger) (Client, error) { + origin = strings.TrimPrefix(origin, "file://") // returns original if there is no "file://" prefix + return &ConfigFileClient{ configFile: origin, id: id, diff --git a/pkg/config/client/util.go b/pkg/config/client/util.go index 666b81a2..c0c7f4fd 100644 --- a/pkg/config/client/util.go +++ b/pkg/config/client/util.go @@ -3,6 +3,7 @@ package client import ( "encoding/json" "net/url" + "strings" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) @@ -28,14 +29,16 @@ func decodeConfigList(r []byte) ([]*stnrv1.StunnerConfig, error) { // getURI tries to parse an address or an URL or a file name into an URL. func getURI(addr string) (*url.URL, error) { + // make sure we have a working HTTP scheme + if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") && + !strings.HasPrefix(addr, "ws://") && !strings.HasPrefix(addr, "wss://") && + !strings.HasPrefix(addr, "file://") { + addr = "http://" + addr + } + url, err := url.Parse(addr) if err != nil { - // try to parse with a http scheme as a last resort - u, err2 := url.Parse("http://" + addr) - if err2 != nil { - return nil, err - } - url = u + return nil, err } return url, nil } diff --git a/stunner_test.go b/stunner_test.go index 6c949259..efea6a3f 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -37,7 +37,6 @@ var stunnerTestLoglevel string = "all:ERROR" // var stunnerTestLoglevel string = stnrv1.DefaultLogLevel // var stunnerTestLoglevel string = "all:INFO" // var stunnerTestLoglevel string = "all:TRACE" - // var stunnerTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" var certPem, keyPem, _ = GenerateSelfSignedKey() From 24f59606582d87de85a047ee5b76570560cf821e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 7 Feb 2024 17:39:29 +0100 Subject: [PATCH 132/243] doc: Remove all references of the ConfigMap that is now gone --- README.md | 9 ++-- docs/CONCEPTS.md | 2 - docs/GATEWAY.md | 2 +- docs/img/stunner_arch_big.svg | 93 +++++++++++++++-------------------- 4 files changed, 44 insertions(+), 62 deletions(-) diff --git a/README.md b/README.md index 00c2c327..85e972fb 100644 --- a/README.md +++ b/README.md @@ -345,12 +345,9 @@ modifications and update the underlying dataplane. Kubernetes is beautiful, isn' ### Check your config -The current STUNner dataplane configuration is always made available in a convenient ConfigMap that -has the same name and namespace as the Gateway it belongs to (so this is supposed to be -`stunner/udp-gateway` as per our example). - -STUNner comes with a small CLI utility called [`stunnerctl`](/cmd/stunnerctl/README.md) to dump the -running configuration in human readable format. +The current STUNner dataplane configuration is always made available via the convenient +[`stunnerctl`](/cmd/stunnerctl/README.md) CLI utility. The below will dump the config of the UDP +gateway in human readable format. ```console stunnerctl -n stunner config udp-gateway diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md index 87ed660e..d8cc646a 100644 --- a/docs/CONCEPTS.md +++ b/docs/CONCEPTS.md @@ -25,5 +25,3 @@ The STUNner dataplane is comprised of a fleet of `stunnerd` pods implementing th * **`stunnerd` Deployment:** Once you create a new Gateway the gateway operator will spawn a new dataplane for the Gateway automatically. For each Gateway there will be `stunnerd` Deployment with the same name and namespace. The `stunnerd` daemon itself is a TURN server implemented on top of the [pion/turn](https://github.com/pion/turn) Go WebRTC framework. The daemon will instantiate a separate *TURN listener* for each Gateway listener in the gateway configuration to terminate clients' TURN sessions, a *cluster* per each UDPRoute to forward packets to the backend services (e.g., to the media servers), with some ancillary administrative and authentication mechanisms in place to check client credentials, logging, etc. Whenever you modify a Gateway (UDPRoute), the gateway operator renders a new dataplane configuration with the modified listener (cluster, respectively) specs and downloads it to the `stunnerd` pods, which in turn reconcile their internal state with respect the new configuration. You are free to scale the dataplane to as many `stunnerd` pods as you wish: Kubernetes will make sure that new client connections are distributed evenly over the scaled-out STUNner dataplane. * **LoadBalancer Service:** STUNner creates a separate LoadBalancer Service per each Gateway to expose the TURN listeners of the `stunnerd` pods to the outside world. Similarly to the case of the `stunnerd` Deployment, there will be a separate LoadBalancer Service per each Gateway with the same name and namespace. - -* **STUNner ConfigMap**: In order to simplify troubleshooting a STUNner setup, the dataplane configuration of each Gateway is always made available in a ConfigMap for human inspection. Again, the name and namespace of the ConfigMap is the same as those of the corresponding Gateway. Note that this ConfigMap is no longer used by the dataplane for reconciliation, it is there only fo debugging purposes and may be silently removed in a later release. You can use STUNner's own [Config Discovery Service client](https://pkg.go.dev/github.com/l7mp/stunner@v0.16.2/pkg/config/client) to obtain dataplane configuration right from the gateway operator instead. diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 5c401baf..538bb915 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -81,7 +81,7 @@ Except the TURN authentication realm, all GatewayConfig resources are safe for m Gateways describe the STUN/TURN server listeners exposed to clients. -The below Gateway resource will configure STUNner to open a STUN/TURN listener over the UDP port 3478 and make it available on a public IP address and port to clients. Each Gateway will have a `stunnerd` Deployment that will run the dataplane, a LoadBalancer Service that will expose the dataplane to the Internet, and an ancillary ConfigMap that will hold the corresponding configuration, each using the same name and namespace as the Gateway. Once the Gateway is removed, the corresponding resources are automatically garbage-collected. +The below Gateway resource will configure STUNner to open a STUN/TURN listener over the UDP port 3478 and make it available on a public IP address and port to clients. Each Gateway will have a `stunnerd` Deployment that will run the dataplane and a LoadBalancer Service that will expose the gateway to the Internet, both using the same name and namespace as the Gateway. Once the Gateway is removed, the corresponding resources are automatically garbage-collected. ```yaml apiVersion: gateway.networking.k8s.io/v1 diff --git a/docs/img/stunner_arch_big.svg b/docs/img/stunner_arch_big.svg index d659fa75..7f7d34c5 100644 --- a/docs/img/stunner_arch_big.svg +++ b/docs/img/stunner_arch_big.svg @@ -7,7 +7,7 @@ viewBox="0 0 145.36844 124.38022" version="1.1" id="svg3087" - inkscape:version="1.1.2 (0a00cf5339, 2022-02-04)" + inkscape:version="1.2.2 (b0a8486541, 2022-12-01)" sodipodi:docname="stunner_arch_big.svg" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" @@ -24,10 +24,10 @@ inkscape:document-units="mm" showgrid="false" inkscape:zoom="1.3444423" - inkscape:cx="292.31451" - inkscape:cy="245.45493" + inkscape:cx="293.43022" + inkscape:cy="245.45494" inkscape:window-width="1912" - inkscape:window-height="1038" + inkscape:window-height="1035" inkscape:window-x="0" inkscape:window-y="0" inkscape:window-maximized="1" @@ -37,7 +37,9 @@ fit-margin-left="0" fit-margin-right="0" fit-margin-bottom="0" - units="mm" /> + units="mm" + inkscape:showpageshadow="2" + inkscape:deskcolor="#d1d1d1" /> + hide_knots="false" + nodesatellites_param="F,0,0,1,0,4.4979167,0,1 @ F,0,0,1,0,4.4979167,0,1 @ F,0,0,1,0,4.4979167,0,1 @ F,0,0,1,0,4.4979167,0,1" /> + y="134.60397" /> Deployment + y="120.47246" /> LB Service - - ConfigMap + y="124.30173">LB Service + y="115.8348" /> Control + y="113.37669">Control + Execute gw-ns/gw + y="142.23865">gw-ns/gw gw-ns/gw - gw-ns/gw + y="128.38521">gw-ns/gw From 01843e8ea44124a1703345337d678845702a267f Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 13 Feb 2024 17:44:34 +0100 Subject: [PATCH 133/243] doc(examples/neko): Update README Changes: - mention tested web browser - generalize `sed` command --- docs/examples/neko/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/examples/neko/README.md b/docs/examples/neko/README.md index 84b31ff1..2a641597 100644 --- a/docs/examples/neko/README.md +++ b/docs/examples/neko/README.md @@ -51,7 +51,7 @@ STUNNERIP=$(kubectl get service udp-gateway -n default -o jsonpath='{.status.loa We need to give this public IP the Neko configuration in the `NEKO_ICESERVERS` environment variable, inside the `json` content (basically this will tell you browser to use STUNner as a STUN/TURN server). You can do that by hand, or by this fancy `sed` command: ```console -sed -i "s/1.1.1.1/$STUNNERIP/g" neko.yaml +sed -i "s/turn:[\.0-9]*:3478/turn:$STUNNERIP:3478/g" neko.yaml ``` Now apply the Neko manifests and wait for the `neko` deployment to be available (should take a couple of seconds): @@ -65,6 +65,9 @@ If you don't have an ingress controller, you can use the `neko-tcp` service with Ideally, by opening your ingress controller in your browser, you should see the Neko UI. You can log in with the `admin`:`admin` credentials. The WebRTC stream then should be relayed through STUNner. +> [!NOTE] +> Tested with Chromium/Google Chrome. + ## Help STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). From 4a9d03c7dba1e019653284c733fdd048b54e7dec Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 15 Feb 2024 18:43:13 +0100 Subject: [PATCH 134/243] doc: README updates --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 85e972fb..48e33cfc 100644 --- a/README.md +++ b/README.md @@ -352,7 +352,7 @@ gateway in human readable format. ```console stunnerctl -n stunner config udp-gateway Gateway: stunner/udp-gateway (loglevel: "all:INFO") -Authentication type: ephemeral, shared-secret: my-very-secure-secret +Authentication type: static, username/password: user-1/pass-1 Listeners: - Name: stunner/udp-gateway/udp-listener Protocol: TURN-UDP @@ -390,9 +390,9 @@ a heartwarming welcome message. 1. We also need a STUN/TURN client to actually initiate a connection. STUNner comes with a handy STUN/TURN client called [`turncat`](cmd/turncat/README.md) for this purpose. Once - [built](cmd/turncat/README.md#installation), you can fire up `turncat` to listen on the standard - input and send everything it receives to STUNner. Type any input and press Enter, and you should - see a nice greeting from your cluster! + [installed](cmd/turncat/README.md#installation), you can fire up `turncat` to listen on the + standard input and send everything it receives to STUNner. Type any input and press Enter, and + you should see a nice greeting from your cluster! ```console ./turncat - k8s://stunner/udp-gateway:udp-listener udp://${PEER_IP}:9001 From 571ddc7671328cccb15b94e9b3dc9fbd140290f3 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 21 Feb 2024 14:17:08 +0100 Subject: [PATCH 135/243] chore(CI/CD): Bump azure/setup-helm to v4 --- .github/workflows/e2e-test-dev-legacy.yml | 2 +- .github/workflows/e2e-test-dev-managed.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml index fd81c34b..1f63ca3a 100644 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ b/.github/workflows/e2e-test-dev-legacy.yml @@ -36,7 +36,7 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v3 + uses: azure/setup-helm@v4 with: version: v3.12.3 diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev-managed.yaml index 091b8a06..81065ee8 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev-managed.yaml @@ -36,7 +36,7 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v3 + uses: azure/setup-helm@v4 with: version: v3.12.3 diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index a758d74a..9badbd0b 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -35,7 +35,7 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v3 + uses: azure/setup-helm@v4 with: version: v3.12.3 From 4a259e7da5ce2d0d88f8b9f0c3595107ab8d56c7 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 21 Feb 2024 14:18:18 +0100 Subject: [PATCH 136/243] chore(CI/CD): Remove legacy mode e2e test --- .github/workflows/e2e-test-dev-legacy.yml | 96 ------------------- ...est-dev-managed.yaml => e2e-test-dev.yaml} | 4 +- 2 files changed, 2 insertions(+), 98 deletions(-) delete mode 100644 .github/workflows/e2e-test-dev-legacy.yml rename .github/workflows/{e2e-test-dev-managed.yaml => e2e-test-dev.yaml} (92%) diff --git a/.github/workflows/e2e-test-dev-legacy.yml b/.github/workflows/e2e-test-dev-legacy.yml deleted file mode 100644 index 1f63ca3a..00000000 --- a/.github/workflows/e2e-test-dev-legacy.yml +++ /dev/null @@ -1,96 +0,0 @@ -name: Run End-to-End Test [legacy] (dev) - -on: - workflow_dispatch: - schedule: - - cron: '0 11 * * 1' - -jobs: - e2e_test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Set up Go - uses: actions/setup-go@v5 - with: - go-version: '1.21' - - - name: Download modules - run: go mod download - - name: Go install - run: go install - - name: Build turncat - run: CGO_ENABLED=0 go build -ldflags="-w -s" -o turncat cmd/turncat/main.go - - - name: Start minikube - uses: medyagh/setup-minikube@master - with: - driver: docker - container-runtime: containerd - cpus: max - wait: all - cache: false - - - name: Start minikube tunnel - run: minikube tunnel &>mktunnel.log & - - - name: Set up Helm - uses: azure/setup-helm@v4 - with: - version: v3.12.3 - - - name: Install STUNner - run: | - helm repo add stunner https://l7mp.io/stunner - helm repo update - helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=legacy --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m - helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner --set stunner.deployment.container.stunnerd.resources.requests.cpu=200m - - - name: Deploy iperf server - run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml - - - name: Configure STUNner - run: | - kubectl apply -f docs/examples/simple-tunnel/iperf-stunner.yaml - sleep 75 - - - name: Install iperf client - run: | - sudo apt-get update - sudo apt-get -y install iperf - - - name: Wait for LoadBalancer IP - run: | - for n in {1..60}; do [[ ! -z $(kubectl get svc udp-gateway -n stunner -o jsonpath="{.status.loadBalancer.ingress[0].ip}") ]] && break; echo "Waiting for LoadBalancer IP"; sleep 2; done - echo "* wait for the deployment" - kubectl get all -A - kubectl wait -n stunner --for=condition=Available deployment stunner --timeout 5m - echo "* EVERYTHING UP" - kubectl get all -A - - - name: Start turncat - run: | - ./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/stunnerd-config:udp-listener udp://$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}"):5001 &>turncat.log & - sleep 1 - - - name: Run iperf client - run: | - iperf -c 127.0.0.1 -p 5000 -u -l 100 -b 5M -t 5 | tee iperf.log - - - name: Show logs - run: | - echo "* IPERF" - echo "** Client" - cat iperf.log - echo "** Server" - kubectl logs $(kubectl get pods -l app=iperf-server -o jsonpath='{.items[0].metadata.name}') - echo "* MINIKUBE TUNNEL" - cat mktunnel.log - echo "* TURNCAT" - cat turncat.log - echo "* STUNNER" - kubectl logs -n stunner $(kubectl get pods -n stunner -l app=stunner -o jsonpath='{.items[0].metadata.name}') - - - name: Check iperf conectivity - run: grep "Server Report" iperf.log diff --git a/.github/workflows/e2e-test-dev-managed.yaml b/.github/workflows/e2e-test-dev.yaml similarity index 92% rename from .github/workflows/e2e-test-dev-managed.yaml rename to .github/workflows/e2e-test-dev.yaml index 81065ee8..6680a201 100644 --- a/.github/workflows/e2e-test-dev-managed.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -1,4 +1,4 @@ -name: Run End-to-End Test [managed] (dev) +name: Run End-to-End Test (dev) on: workflow_dispatch: @@ -44,7 +44,7 @@ jobs: run: | helm repo add stunner https://l7mp.io/stunner helm repo update - helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.dataplane.mode=managed --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m --set stunnerGatewayOperator.dataplane.spec.resources.requests.cpu=100m + helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner --set stunnerGatewayOperator.deployment.container.manager.resources.requests.cpu=200m --set stunnerGatewayOperator.dataplane.spec.resources.requests.cpu=100m - name: Deploy iperf server run: kubectl apply -f docs/examples/simple-tunnel/iperf-server.yaml From 4ab0374fad7f5e4080bdb36a15b9d05e613f6608 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 21 Feb 2024 14:29:56 +0100 Subject: [PATCH 137/243] chore(CI/CD): Bump golangci/golangci-lint-action to v4 --- .github/workflows/lint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index e3ba6e9e..b34e4bdd 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -24,6 +24,6 @@ jobs: - name: Checkout code uses: actions/checkout@v4 - name: Run linters - uses: golangci/golangci-lint-action@v3 + uses: golangci/golangci-lint-action@v4 with: args: --timeout 5m From 69f4ada3ed8db130b40ea75cf023211f5b208cba Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 21 Feb 2024 15:14:30 +0100 Subject: [PATCH 138/243] feat: Serve /status requests on the health-check server --- docs/img/stunner_neko.svg | 1169 +++++++++++++++++++++++++++++++++++ go.mod | 4 - go.sum | 156 +---- handlers.go | 5 + internal/object/admin.go | 64 +- internal/object/auth.go | 5 + internal/object/cluster.go | 7 +- internal/object/listener.go | 36 +- internal/object/object.go | 9 +- pkg/apis/v1/admin.go | 27 + pkg/apis/v1/auth.go | 11 +- pkg/apis/v1/cluster.go | 2 + pkg/apis/v1/config.go | 6 + pkg/apis/v1/listener.go | 2 + pkg/apis/v1/stunner.go | 33 + reconcile.go | 2 +- reconcile_test.go | 52 ++ stunner.go | 48 +- 18 files changed, 1433 insertions(+), 205 deletions(-) create mode 100644 docs/img/stunner_neko.svg diff --git a/docs/img/stunner_neko.svg b/docs/img/stunner_neko.svg new file mode 100644 index 00000000..961ab1a1 --- /dev/null +++ b/docs/img/stunner_neko.svg @@ -0,0 +1,1169 @@ + +audio/videouser input (keys pressed)HTTPS/WebSocket Kubernetes clusterApplication ServerSTUNner AuthenticationServiceAUTH SERVICEHTTPREST diff --git a/go.mod b/go.mod index ed859099..827c49e8 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,6 @@ require ( github.com/google/uuid v1.6.0 github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.1 - github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb github.com/oapi-codegen/runtime v1.1.1 github.com/pion/dtls/v2 v2.2.10 github.com/pion/logging v0.2.2 @@ -31,7 +30,6 @@ require ( k8s.io/cli-runtime v0.29.1 k8s.io/client-go v0.29.1 k8s.io/utils v0.0.0-20240102154912-e7106e64919e - sigs.k8s.io/controller-runtime v0.12.3 sigs.k8s.io/yaml v1.4.0 ) @@ -61,7 +59,6 @@ require ( github.com/json-iterator/go v1.1.12 // indirect github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect github.com/mailru/easyjson v0.7.7 // indirect - github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect github.com/moby/spdystream v0.2.0 // indirect github.com/moby/term v0.5.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect @@ -93,7 +90,6 @@ require ( golang.org/x/tools v0.17.0 // indirect google.golang.org/appengine v1.6.8 // indirect google.golang.org/protobuf v1.32.0 // indirect - gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect gopkg.in/evanphx/json-patch.v5 v5.9.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/go.sum b/go.sum index e5268dc9..d3bb53cb 100644 --- a/go.sum +++ b/go.sum @@ -1,9 +1,5 @@ -cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8= -github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0= github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= -github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= @@ -12,58 +8,34 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bmatcuk/doublestar v1.1.1/go.mod h1:UD6OnuiIn0yFxxA2le/rnRU1G4RaI4UvFv1sNto9p6w= -github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= -github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= -github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= -github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY= github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/deepmap/oapi-codegen/v2 v2.0.0 h1:3TS7w3r+XnjKFXcbFbc16pTWzfTy0OLPkCsutEHjWDA= -github.com/deepmap/oapi-codegen/v2 v2.0.0/go.mod h1:7zR+ZL3WzLeCkr2k8oWTxEa0v8y/F25ane0l6A5UjLA= github.com/deepmap/oapi-codegen/v2 v2.1.0 h1:I/NMVhJCtuvL9x+S2QzZKpSjGi33oDZwPRdemvOZWyQ= github.com/deepmap/oapi-codegen/v2 v2.1.0/go.mod h1:R1wL226vc5VmCNJUvMyYr3hJMm5reyv25j952zAVXZ8= -github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= -github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= github.com/emicklei/go-restful/v3 v3.11.2 h1:1onLa9DcsMYO9P+CXaL0dStDqQ2EHHXLiz+BtnqkLAU= github.com/emicklei/go-restful/v3 v3.11.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= -github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= -github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= -github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84= -github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/evanphx/json-patch v5.9.0+incompatible h1:fBXyNpNMuTTDdquAq/uisOr2lShz4oaXpDTX2bLe7ls= github.com/evanphx/json-patch v5.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mNrPhs8= github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM= -github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA= -github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk= github.com/go-errors/errors v1.5.1/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= -github.com/go-logr/logr v1.3.0 h1:2y3SDp0ZXuc6/cjLSZ+Q3ir+QB9T/iG5yYRXqsagWSY= -github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ= github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= -github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q= github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs= -github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= -github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= github.com/go-openapi/jsonreference v0.20.4 h1:bKlDxQxQJgwpUSgOENiMPzCTBVuc7vTdXSSgNeAhojU= github.com/go-openapi/jsonreference v0.20.4/go.mod h1:5pZJyJP2MnYCpoeoMAql78cCHauHj0V9Lhc506VOpw4= -github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= -github.com/go-openapi/swag v0.22.8 h1:/9RjDSQ0vbFR+NyjGMkFTsA1IA0fmhKSThmfGZjicbw= -github.com/go-openapi/swag v0.22.8/go.mod h1:6QT22icPLEqAM/z/TChgb4WAveCHF92+2gF0CNjHpPI= github.com/go-openapi/swag v0.22.9 h1:XX2DssF+mQKM2DHsbgZK74y/zj4mo9I99+89xUmuZCE= github.com/go-openapi/swag v0.22.9/go.mod h1:3/OXnFfnMAwBD099SwYRk7GD3xOrr1iL7d/XNLXVVwE= github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= @@ -72,33 +44,14 @@ github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= -github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= -github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= -github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= -github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= -github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= -github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= -github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= -github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4= -github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA= github.com/google/btree v1.1.2 h1:xf4v41cLI2Z6FxbKm+8Bu+m8ifhj15JuZ9sa0jZCMUU= github.com/google/btree v1.1.2/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= -github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= -github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= -github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= -github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= @@ -106,12 +59,10 @@ github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeN github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/pprof v0.0.0-20211214055906-6f57359322fd h1:1FjCyPC+syAzJ5/2S8fqdZK1R22vvA0J7JZKcuOIQ7Y= -github.com/google/pprof v0.0.0-20211214055906-6f57359322fd/go.mod h1:KgnwoLYCZ8IQu3XUZ8Nc/bM9CCZFOyjUNOSygVozoDg= +github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec= +github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= -github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU= -github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= @@ -119,14 +70,8 @@ github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWS github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY= github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY= -github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM= -github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 h1:+ngKgrYPPJrOjhax5N+uePQ0Fh1Z7PheYoUI/0nzkPA= github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= -github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb h1:tsEKRC3PU9rMw18w/uAptoijhgG4EvlA5kfJPtwrMDk= -github.com/heptiolabs/healthcheck v0.0.0-20211123025425-613501dd5deb/go.mod h1:NtmN9h8vrTveVQRLHcX2HQ5wIPBDCsZ351TGbZWgg38= -github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= -github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4= github.com/imdario/mergo v0.3.16/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= @@ -140,23 +85,16 @@ github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHm github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0= github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= -github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg= -github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k= github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8= github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= -github.com/moby/term v0.0.0-20221205130635-1aeaba878587 h1:HfkjXDfhgVaN5rmueG8cL8KKeFNecRCXFhaJ2qZ5SKA= -github.com/moby/term v0.0.0-20221205130635-1aeaba878587/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0= github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -172,12 +110,8 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= -github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= -github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU= github.com/oapi-codegen/runtime v1.1.1 h1:EXLHh0DXIJnWhdRPN2w4MXAzFyE4CskzhNLUmtpMYro= github.com/oapi-codegen/runtime v1.1.1/go.mod h1:SK9X900oXmPWilYR5/WKPzt3Kqxn/uS/+lbpREv+eCg= -github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= -github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU= github.com/onsi/ginkgo/v2 v2.13.0 h1:0jY9lJquiL8fcf3M4LAXN5aMlS/b2BV86HFFPCPMgE4= github.com/onsi/ginkgo/v2 v2.13.0/go.mod h1:TE309ZR8s5FsKKpuB1YAQYBzCaAfUgatB/xlT/ETL/o= github.com/onsi/gomega v1.29.0 h1:KIA/t2t5UBzoirT4H9tsML45GEbo3ouUnBHsCfD2tVg= @@ -187,8 +121,6 @@ github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0V github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= -github.com/pion/dtls/v2 v2.2.9 h1:K+D/aVf9/REahQvqk6G5JavdrD8W1PWDKC11UlwN7ts= -github.com/pion/dtls/v2 v2.2.9/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= github.com/pion/dtls/v2 v2.2.10 h1:u2Axk+FyIR1VFTPurktB+1zoEPGIW3bmyj3LEFrXjAA= github.com/pion/dtls/v2 v2.2.10/go.mod h1:d9SYc9fch0CqK90mRk1dC7AkzzpwJj6u2GU3u+9pqFE= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= @@ -197,7 +129,6 @@ github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8= github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= -github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= github.com/pion/transport/v2 v2.2.4 h1:41JJK6DZQYSeVLxILA2+F4ZkKb4Xd/tFJZRFZQ9QAlo= github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= @@ -211,11 +142,8 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/prometheus/client_golang v1.18.0 h1:HzFfmkOzH5Q8L8G+kSJKUx5dtG87sewO+FoDDqP5Tbk= github.com/prometheus/client_golang v1.18.0/go.mod h1:T+GXkCk5wSJyOqMIzVgvvjFDlkOQntgjkJWKrN5txjA= -github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= -github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM= -github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY= github.com/prometheus/common v0.46.0 h1:doXzt5ybi1HBKpsZOL0sSkaNHJJqkyfEWZGGqqScV0Y= github.com/prometheus/common v0.46.0/go.mod h1:Tp0qkxpb9Jsg54QMe+EAmqXkSV7Evdy1BTn+g2pa/hQ= github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= @@ -238,7 +166,6 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= @@ -249,12 +176,10 @@ github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -go.starlark.net v0.0.0-20230525235612-a134d8f9ddca h1:VdD38733bfYv5tUZwEIskMM93VanwNIi5bIKnDrJdEY= -go.starlark.net v0.0.0-20230525235612-a134d8f9ddca/go.mod h1:jxU+3+j+71eXOW14274+SmmuW82qJzl6iZSeqEtTGds= go.starlark.net v0.0.0-20240123142251-f86470692795 h1:LmbG8Pq7KDGkglKVn8VpZOZj6vb9b8nKEGcg9l03epM= go.starlark.net v0.0.0-20240123142251-f86470692795/go.mod h1:LcLNIzVOMp4oV+uusnpk+VU+SzXaJakUuBjoCSWH5dM= -go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= -go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= @@ -265,14 +190,8 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/crypto v0.17.0 h1:r8bRNjWL3GshPW3gkd+RpvzWrZAwPS49OmTGZ/uhM4k= -golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= -golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= -golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= -golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -281,12 +200,7 @@ golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= -golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= @@ -296,27 +210,17 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c= -golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U= golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo= golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= -golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= -golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4= -golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4= golang.org/x/oauth2 v0.16.0 h1:aDkGMBSYxElaoP81NpoUoz2oo2R2wHdZpGToUxfyQrQ= golang.org/x/oauth2 v0.16.0/go.mod h1:hqZ+0LWXsiVoZpeld6jVt06P3adbS2Uu911W1SsJv2o= -golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE= -golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ= golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= -golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -324,7 +228,6 @@ golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -334,17 +237,13 @@ golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= -golang.org/x/term v0.15.0 h1:y/Oo/a/q3IXu26lQgl04j/gjuBDOBlx7X6Om1j2CPW4= -golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= golang.org/x/term v0.16.0 h1:m+B6fahuftsE9qjo0VWp2FW0mB3MTJvR0BaMQrq0pmE= golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= @@ -356,53 +255,23 @@ golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= -golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.16.1 h1:TLyB3WofjdOEepBHAU20JdNC1Zbg87elYofWYAY5oZA= -golang.org/x/tools v0.16.1/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0= golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc= golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -gomodules.xyz/jsonpatch/v2 v2.2.0 h1:4pT439QV83L+G9FkcCriY6EkpcK6r6bK+A5FBUMI7qY= -gomodules.xyz/jsonpatch/v2 v2.2.0/go.mod h1:WXp+iVDkoLQqPudfQ9GBlwB2eZ5DKOnjQZCYdOS8GPY= -google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= -google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= -google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= -google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAsM= google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds= -google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= -google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= -google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= -google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= -google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= -google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= -google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= -google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= -google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= -google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= -google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= -google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= -google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= -google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= -google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I= google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= -gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 h1:FVCohIoYO7IJoDDVpV2pdq7SgrMH6wHnuTyrdrxJNoY= -gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0/go.mod h1:OdE7CF6DbADk7lN8LIKRzRJTTZXIjtWgA5THM5lhBAw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= @@ -410,48 +279,31 @@ gopkg.in/evanphx/json-patch.v5 v5.9.0 h1:hx1VU2SGj4F8r9b8GUwJLdc8DNO8sy79ZGui0G0 gopkg.in/evanphx/json-patch.v5 v5.9.0/go.mod h1:/kvTRh1TVm5wuM6OkHxqXtE/1nUZZpihg29RtuIyfvk= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= -gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= -gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= k8s.io/api v0.29.1 h1:DAjwWX/9YT7NQD4INu49ROJuZAAAP/Ijki48GUPzxqw= k8s.io/api v0.29.1/go.mod h1:7Kl10vBRUXhnQQI8YR/R327zXC8eJ7887/+Ybta+RoQ= -k8s.io/apiextensions-apiserver v0.24.2 h1:/4NEQHKlEz1MlaK/wHT5KMKC9UKYz6NZz6JE6ov4G6k= -k8s.io/apiextensions-apiserver v0.24.2/go.mod h1:e5t2GMFVngUEHUd0wuCJzw8YDwZoqZfJiGOW6mm2hLQ= k8s.io/apimachinery v0.29.1 h1:KY4/E6km/wLBguvCZv8cKTeOwwOBqFNjwJIdMkMbbRc= k8s.io/apimachinery v0.29.1/go.mod h1:6HVkd1FwxIagpYrHSwJlQqZI3G9LfYWRPAkUvLnXTKU= k8s.io/cli-runtime v0.29.1 h1:By3WVOlEWYfyxhGko0f/IuAOLQcbBSMzwSaDren2JUs= k8s.io/cli-runtime v0.29.1/go.mod h1:vjEY9slFp8j8UoMhV5AlO8uulX9xk6ogfIesHobyBDU= k8s.io/client-go v0.29.1 h1:19B/+2NGEwnFLzt0uB5kNJnfTsbV8w6TgQRz9l7ti7A= k8s.io/client-go v0.29.1/go.mod h1:TDG/psL9hdet0TI9mGyHJSgRkW3H9JZk2dNEUS7bRks= -k8s.io/klog/v2 v2.110.1 h1:U/Af64HJf7FcwMcXyKm2RPM22WZzyR7OSpYj5tg3cL0= -k8s.io/klog/v2 v2.110.1/go.mod h1:YGtd1984u+GgbuZ7e08/yBuAfKLSO0+uR1Fhi6ExXjo= k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw= k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= -k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00 h1:aVUu9fTY98ivBPKR9Y5w/AuzbMm96cd3YHRTU83I780= -k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00/go.mod h1:AsvuZPBlUDVuCdzJ87iajxtXuR9oktsTctW/R9wwouA= k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec h1:iGTel2aR8vCZdxJDgmbeY0zrlXy9Qcvyw4R2sB4HLrA= k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec/go.mod h1:Pa1PvrP7ACSkuX6I7KYomY6cmMA0Tx86waBhDUgoKPw= k8s.io/utils v0.0.0-20240102154912-e7106e64919e h1:eQ/4ljkx21sObifjzXwlPKpdGLrCfRziVtos3ofG/sQ= k8s.io/utils v0.0.0-20240102154912-e7106e64919e/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= -sigs.k8s.io/controller-runtime v0.12.3 h1:FCM8xeY/FI8hoAfh/V4XbbYMY20gElh9yh+A98usMio= -sigs.k8s.io/controller-runtime v0.12.3/go.mod h1:qKsk4WE6zW2Hfj0G4v10EnNB2jMG1C+NTb8h+DwCoU0= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= -sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3 h1:XX3Ajgzov2RKUdc5jW3t5jwY7Bo7dcRm+tFxT+NfgY0= -sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3/go.mod h1:9n16EZKMhXBNSiUC5kSdFQJkdH3zbxS/JoO619G1VAY= sigs.k8s.io/kustomize/api v0.16.0 h1:/zAR4FOQDCkgSDmVzV2uiFbuy9bhu3jEzthrHCuvm1g= sigs.k8s.io/kustomize/api v0.16.0/go.mod h1:MnFZ7IP2YqVyVwMWoRxPtgl/5hpA+eCCrQR/866cm5c= -sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3 h1:W6cLQc5pnqM7vh3b7HvGNfXrJ/xL6BDMS0v1V/HHg5U= -sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3/go.mod h1:JWP1Fj0VWGHyw3YUPjXSQnRnrwezrZSrApfX5S0nIag= sigs.k8s.io/kustomize/kyaml v0.16.0 h1:6J33uKSoATlKZH16unr2XOhDI+otoe2sR3M8PDzW3K0= sigs.k8s.io/kustomize/kyaml v0.16.0/go.mod h1:xOK/7i+vmE14N2FdFyugIshB8eF6ALpy7jI87Q2nRh4= sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= diff --git a/handlers.go b/handlers.go index 6bcab5ca..7daa3023 100644 --- a/handlers.go +++ b/handlers.go @@ -114,3 +114,8 @@ func (s *Stunner) NewRealmHandler() object.RealmHandler { return "" } } + +// NewStatusHandler creates a helper function for printing the status of STUNner. +func (s *Stunner) NewStatusHandler() object.StatusHandler { + return func() stnrv1.Status { return s.Status() } +} diff --git a/internal/object/admin.go b/internal/object/admin.go index 49a4c980..418449c2 100644 --- a/internal/object/admin.go +++ b/internal/object/admin.go @@ -2,6 +2,7 @@ package object import ( "context" + "encoding/json" "errors" "fmt" "net" @@ -12,8 +13,6 @@ import ( "github.com/pion/logging" "github.com/prometheus/client_golang/prometheus/promhttp" - health "github.com/heptiolabs/healthcheck" - stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) @@ -25,12 +24,12 @@ type Admin struct { DryRun bool MetricsEndpoint, HealthCheckEndpoint string metricsServer, healthCheckServer *http.Server - health health.Handler + health *http.ServeMux log logging.LeveledLogger } // NewAdmin creates a new Admin object. -func NewAdmin(conf stnrv1.Config, dryRun bool, rc health.Check, logger logging.LoggerFactory) (Object, error) { +func NewAdmin(conf stnrv1.Config, dryRun bool, rc ReadinessHandler, status StatusHandler, logger logging.LoggerFactory) (Object, error) { req, ok := conf.(*stnrv1.AdminConfig) if !ok { return nil, stnrv1.ErrInvalidConf @@ -38,15 +37,44 @@ func NewAdmin(conf stnrv1.Config, dryRun bool, rc health.Check, logger logging.L admin := Admin{ DryRun: dryRun, - health: health.NewHandler(), + health: http.NewServeMux(), log: logger.NewLogger("stunner-admin"), } admin.log.Tracef("NewAdmin: %s", req.String()) // health checker // liveness probe always succeeds once we got here - admin.health.AddLivenessCheck("server-alive", func() error { return nil }) - admin.health.AddReadinessCheck("server-ready", rc) + admin.health.HandleFunc("/live", func(w http.ResponseWriter, req *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + w.WriteHeader(http.StatusOK) + w.Write([]byte("{}\n")) //nolint:errcheck + }) + // readniness checker calls the checker from the factory + admin.health.HandleFunc("/ready", func(w http.ResponseWriter, req *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + if err := rc(); err != nil { + w.WriteHeader(http.StatusServiceUnavailable) + + w.Write([]byte(fmt.Sprintf("{\"status\":%d,\"message\":\"%s\"}\n", //nolint:errcheck + http.StatusServiceUnavailable, err.Error()))) + } else { + w.WriteHeader(http.StatusOK) + w.Write([]byte(fmt.Sprintf("{\"status\":%d,\"message\":\"%s\"}\n", //nolint:errcheck + http.StatusOK, "READY"))) + } + }) + // status handler returns the status + admin.health.HandleFunc("/status", func(w http.ResponseWriter, req *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + if js, err := json.Marshal(status()); err != nil { + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(fmt.Sprintf("{\"status\":%d,\"message\":\"%s\"}\n", //nolint:errcheck + http.StatusInternalServerError, err.Error()))) + } else { + w.WriteHeader(http.StatusOK) + w.Write(js) //nolint:errcheck + } + }) if err := admin.Reconcile(req); err != nil && !errors.Is(err, ErrRestartRequired) { return nil, err @@ -142,6 +170,19 @@ func (a *Admin) Close() error { return nil } +// Status returns the status of the object. +func (a *Admin) Status() stnrv1.Status { + s := stnrv1.AdminStatus{ + Name: a.Name, + LogLevel: a.LogLevel, + MetricsEndpoint: a.MetricsEndpoint, + HealthCheckEndpoint: a.HealthCheckEndpoint, + } + + // add licensing status here + return &s +} + func (a *Admin) reconcileMetrics(req *stnrv1.AdminConfig) error { a.log.Trace("reconcileMetrics") @@ -275,13 +316,14 @@ end: // AdminFactory can create now Admin objects type AdminFactory struct { dry bool - rc health.Check + rc ReadinessHandler + status StatusHandler logger logging.LoggerFactory } // NewAdminFactory creates a new factory for Admin objects -func NewAdminFactory(dryRun bool, rc health.Check, logger logging.LoggerFactory) Factory { - return &AdminFactory{dry: dryRun, rc: rc, logger: logger} +func NewAdminFactory(dryRun bool, rc ReadinessHandler, status StatusHandler, logger logging.LoggerFactory) Factory { + return &AdminFactory{dry: dryRun, rc: rc, status: status, logger: logger} } // New can produce a new Admin object from the given configuration. A nil config will create an @@ -291,7 +333,7 @@ func (f *AdminFactory) New(conf stnrv1.Config) (Object, error) { return &Admin{}, nil } - return NewAdmin(conf, f.dry, f.rc, f.logger) + return NewAdmin(conf, f.dry, f.rc, f.status, f.logger) } func getHealthAddr(e string) string { diff --git a/internal/object/auth.go b/internal/object/auth.go index da400d1c..aba8e04d 100644 --- a/internal/object/auth.go +++ b/internal/object/auth.go @@ -105,6 +105,11 @@ func (auth *Auth) Close() error { return nil } +// Status returns the status of the object. +func (auth *Auth) Status() stnrv1.Status { + return auth.GetConfig() +} + // AuthFactory can create now Auth objects type AuthFactory struct { logger logging.LoggerFactory diff --git a/internal/object/cluster.go b/internal/object/cluster.go index cc4654fc..1aec7b03 100644 --- a/internal/object/cluster.go +++ b/internal/object/cluster.go @@ -92,7 +92,7 @@ func (c *Cluster) Reconcile(conf stnrv1.Config) error { c.Endpoints = append(c.Endpoints, ep) } case stnrv1.ClusterTypeStrictDNS: - // TOD: port-range support for DNS clusters + // TODO: port-range support for DNS clusters if c.Resolver == nil { return fmt.Errorf("STRICT_DNS cluster %q initialized with no DNS resolver", c.Name) } @@ -164,6 +164,11 @@ func (c *Cluster) Close() error { return nil } +// Status returns the status of the object. +func (c *Cluster) Status() stnrv1.Status { + return c.GetConfig() +} + // Route decides whether a peer IP appears among the permitted endpoints of a cluster. func (c *Cluster) Route(peer net.IP) bool { return c.Match(peer, 0) diff --git a/internal/object/listener.go b/internal/object/listener.go index 3274a9ef..b07f2ca3 100644 --- a/internal/object/listener.go +++ b/internal/object/listener.go @@ -22,6 +22,8 @@ type Listener struct { Proto stnrv1.ListenerProtocol Addr net.IP Port, MinPort, MaxPort int + PublicAddr string // for GetConfig() + PublicPort int // for GetConfig() rawAddr string // net.IP.String() may rewrite the string representation Cert, Key []byte Conns []any // either a set of turn.ListenerConfigs or turn.PacketConnConfigs @@ -46,12 +48,14 @@ func NewListener(conf stnrv1.Config, net transport.Net, realmHandler RealmHandle } l := Listener{ - Name: req.Name, - Net: net, - getRealm: realmHandler, - Conns: []any{}, - logger: logger, - log: logger.NewLogger(fmt.Sprintf("stunner-listener-%s", req.Name)), + Name: req.Name, + PublicAddr: req.PublicAddr, + PublicPort: req.PublicPort, + Net: net, + getRealm: realmHandler, + Conns: []any{}, + logger: logger, + log: logger.NewLogger(fmt.Sprintf("stunner-listener-%s", req.Name)), } l.log.Tracef("NewListener: %s", req.String()) @@ -88,7 +92,7 @@ func (l *Listener) Inspect(old, new, full stnrv1.Config) (bool, error) { return false, fmt.Errorf("invalid TLS key: base64-decode error: %w", err) } - // the only chance we don't need a restart if only the Routes change + // the only chance we don't need a restart if only the Routes and/or PublicIP/PublicPort change restart := ErrRestartRequired if l.Name == req.Name && // name unchanged (should always be true) l.Proto == proto && // protocol unchanged @@ -150,6 +154,9 @@ func (l *Listener) Reconcile(conf stnrv1.Config) error { } l.Realm = l.getRealm() + l.PublicAddr = req.PublicAddr + l.PublicPort = req.PublicPort + l.Routes = make([]string, len(req.Routes)) copy(l.Routes, req.Routes) @@ -179,10 +186,12 @@ func (l *Listener) GetConfig() stnrv1.Config { sort.Strings(l.Routes) c := &stnrv1.ListenerConfig{ - Name: l.Name, - Protocol: l.Proto.String(), - Addr: l.rawAddr, - Port: l.Port, + Name: l.Name, + Protocol: l.Proto.String(), + Addr: l.rawAddr, + Port: l.Port, + PublicAddr: l.PublicAddr, + PublicPort: l.PublicPort, } c.Cert = string(l.Cert) @@ -240,6 +249,11 @@ func (l *Listener) Close() error { return nil } +// Status returns the status of the object. +func (l *Listener) Status() stnrv1.Status { + return l.GetConfig() +} + // /////////// // ListenerFactory can create now Listener objects type ListenerFactory struct { diff --git a/internal/object/object.go b/internal/object/object.go index 4a917bec..c80edab8 100644 --- a/internal/object/object.go +++ b/internal/object/object.go @@ -16,11 +16,13 @@ type Object interface { GetConfig() stnrv1.Config // Close closes the object, may return ErrRestartRequired. Close() error + // Status returns the status of the object. + Status() stnrv1.Status } -// Factory can create new objects +// Factory can create new objects. type Factory interface { - // New will spawn a new object from the factory + // New will spawn a new object from the factory. New(conf stnrv1.Config) (Object, error) } @@ -29,3 +31,6 @@ type ReadinessHandler = func() error // RealmHandler is a callback that allows an object to find out the authentication realm. type RealmHandler = func() string + +// StatusHandler is a callback that allows an object to obtain the status of STUNNer. +type StatusHandler = func() stnrv1.Status diff --git a/pkg/apis/v1/admin.go b/pkg/apis/v1/admin.go index 808b99cc..9f95ff38 100644 --- a/pkg/apis/v1/admin.go +++ b/pkg/apis/v1/admin.go @@ -94,3 +94,30 @@ func (req *AdminConfig) String() string { } return fmt.Sprintf("admin:{%s}", strings.Join(status, ",")) } + +// AdminStatus represents the administrative status. +type AdminStatus struct { + Name string `json:"name,omitempty"` + LogLevel string `json:"loglevel,omitempty"` + MetricsEndpoint string `json:"metrics_endpoint,omitempty"` + HealthCheckEndpoint string `json:"healthcheck_endpoint,omitempty"` + // licencing status comes here +} + +// String returns a string reprsentation of the administrative status. +func (a *AdminStatus) String() string { + status := []string{} + if a.LogLevel != "" { + status = append(status, fmt.Sprintf("logLevel=%q", a.LogLevel)) + } + if a.MetricsEndpoint != "" { + status = append(status, fmt.Sprintf("metrics=%q", a.MetricsEndpoint)) + } + if a.HealthCheckEndpoint != "" { + status = append(status, fmt.Sprintf("health-check=%q", a.HealthCheckEndpoint)) + } + + // add licencing status here + + return fmt.Sprintf("%s:{%s}", a.Name, strings.Join(status, ",")) +} diff --git a/pkg/apis/v1/auth.go b/pkg/apis/v1/auth.go index ab23fd9d..4f37c5a3 100644 --- a/pkg/apis/v1/auth.go +++ b/pkg/apis/v1/auth.go @@ -113,8 +113,7 @@ func (req *AuthConfig) String() string { } else { p = "-" } - status = append(status, fmt.Sprintf("type=%q,username=%q,password=%q", - atype.String(), u, p)) + status = append(status, fmt.Sprintf("username=%q,password=%q", u, p)) case AuthTypeEphemeral: s, secretFound := req.Credentials["secret"] @@ -128,10 +127,12 @@ func (req *AuthConfig) String() string { s = "-" } - status = append(status, fmt.Sprintf("type=%q,shared-secret=%q", - atype.String(), s)) + status = append(status, fmt.Sprintf("secret=%q", s)) } } - return fmt.Sprintf("auth:{%s}", strings.Join(status, ",")) + return fmt.Sprintf("%s-auth:{%s}", req.Type, strings.Join(status, ",")) } + +// AuthStatus represents the authentication status. +type AuthStatus = AuthConfig diff --git a/pkg/apis/v1/cluster.go b/pkg/apis/v1/cluster.go index 24dcc5cf..1f8746f9 100644 --- a/pkg/apis/v1/cluster.go +++ b/pkg/apis/v1/cluster.go @@ -113,3 +113,5 @@ func (req *ClusterConfig) String() string { return fmt.Sprintf("%q:{%s}", n, strings.Join(status, ",")) } + +type ClusterStatus = ClusterConfig diff --git a/pkg/apis/v1/config.go b/pkg/apis/v1/config.go index 1d614949..b32986f8 100644 --- a/pkg/apis/v1/config.go +++ b/pkg/apis/v1/config.go @@ -14,3 +14,9 @@ type Config interface { // String stringifies the configuration. String() string } + +// Status holds the status of a component. +type Status interface { + // String stringifies the status. + String() string +} diff --git a/pkg/apis/v1/listener.go b/pkg/apis/v1/listener.go index 2fbb1207..5d08e3d8 100644 --- a/pkg/apis/v1/listener.go +++ b/pkg/apis/v1/listener.go @@ -179,3 +179,5 @@ func (req *ListenerConfig) GetListenerURI(rfc7065 bool) (string, error) { } return uri, nil } + +type ListenerStatus = ListenerConfig diff --git a/pkg/apis/v1/stunner.go b/pkg/apis/v1/stunner.go index 63f1b39f..ebcff12e 100644 --- a/pkg/apis/v1/stunner.go +++ b/pkg/apis/v1/stunner.go @@ -213,3 +213,36 @@ func (req *StunnerConfig) Summary() string { return status } + +// StunnerStatus represents the status of the STUnner daemon. +type StunnerStatus struct { + ApiVersion string `json:"version"` + Admin *AdminStatus `json:"admin"` + Auth *AuthStatus `json:"auth"` + Listeners []*ListenerStatus `json:"listeners"` + Clusters []*ClusterStatus `json:"clusters"` + AllocationCount int `json:"allocationCount"` + Status string `json:"status"` +} + +// String stringifies the status. +func (s *StunnerStatus) String() string { + ls := []string{} + for _, l := range s.Listeners { + ls = append(ls, l.String()) + } + cs := []string{} + for _, c := range s.Clusters { + cs = append(cs, c.String()) + } + + return fmt.Sprintf("%s/%s/%s/%s/allocs:%d/status=%s", + s.Admin.String(), s.Auth.String(), ls, cs, s.AllocationCount, s.Status) +} + +// String summarizes the status. +func (s *StunnerStatus) Summary() string { + return fmt.Sprintf("%s\n\t%s\n\tlisteners:%d/clusters:%d\n\tallocs:%d/status=%s", + s.Admin.String(), s.Auth.String(), len(s.Listeners), len(s.Clusters), + s.AllocationCount, s.Status) +} diff --git a/reconcile.go b/reconcile.go index 2f33c3d9..165ef456 100644 --- a/reconcile.go +++ b/reconcile.go @@ -177,7 +177,7 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo "deleted objects: %d, started objects: %d, restarted objects: %d", new, changed, deleted, len(toBeStarted), len(toBeRestarted)) - s.log.Info(s.Status()) + s.log.Infof("new dataplane status: %s", s.Status().String()) if len(toBeRestarted) > 0 { names := make([]string, len(toBeRestarted)) diff --git a/reconcile_test.go b/reconcile_test.go index d7579a18..89c02086 100644 --- a/reconcile_test.go +++ b/reconcile_test.go @@ -1253,6 +1253,58 @@ var testReconcileDefault = []StunnerReconcileTestConfig{ }, }, + { + name: "reconcile-test: reconcile existing listener with new public IP and port", + config: stnrv1.StunnerConfig{ + ApiVersion: stnrv1.ApiVersion, + Admin: stnrv1.AdminConfig{ + LogLevel: stunnerTestLoglevel, + }, + Auth: stnrv1.AuthConfig{ + Credentials: map[string]string{ + "username": "user", + "password": "pass", + }, + }, + Listeners: []stnrv1.ListenerConfig{{ + Name: "default-listener", + Addr: "127.0.0.1", + Protocol: "TURN-UDP", + PublicAddr: "127.0.0.2", + PublicPort: 33478, + Routes: []string{"allow-any"}, + }}, + Clusters: []stnrv1.ClusterConfig{{ + Name: "allow-any", + Endpoints: []string{"0.0.0.0/0"}, + }}, + }, + tester: func(t *testing.T, s *Stunner, err error) { + // does not require a restart! + assert.NoError(t, err, "restart") + + assert.Len(t, s.listenerManager.Keys(), 1, "listenerManager keys") + + l := s.GetListener("default-listener") + assert.NotNil(t, l, "listener found") + assert.IsType(t, l, &object.Listener{}, "listener type ok") + assert.Equal(t, l.Proto, stnrv1.ListenerProtocolTURNUDP, "listener proto ok") + assert.Equal(t, l.Addr.String(), "127.0.0.1", "listener address ok") + assert.Equal(t, l.Port, stnrv1.DefaultPort, "listener port ok") + assert.Equal(t, l.PublicAddr, "127.0.0.2", "listener public address ok") + assert.Equal(t, l.PublicPort, 33478, "listener public port ok") + assert.Len(t, l.Routes, 1, "listener route count ok") + assert.Equal(t, l.Routes[0], "allow-any", "listener route name ok") + + c := s.GetCluster("allow-any") + assert.NotNil(t, c, "cluster found") + assert.IsType(t, c, &object.Cluster{}, "cluster type ok") + assert.Equal(t, c.Type, stnrv1.ClusterTypeStatic, "cluster mode ok") + assert.Len(t, c.Endpoints, 1, "cluster endpoint count ok") + _, n, _ := net.ParseCIDR("0.0.0.0/0") + assert.Equal(t, c.Endpoints[0].String(), n.String(), "cluster endpoint ok") + }, + }, { name: "reconcile-test: reconcile deleted listener", config: stnrv1.StunnerConfig{ diff --git a/stunner.go b/stunner.go index f07da99f..054d499b 100644 --- a/stunner.go +++ b/stunner.go @@ -4,7 +4,6 @@ package stunner import ( "fmt" "os" - "strings" "github.com/google/uuid" "github.com/pion/logging" @@ -94,7 +93,7 @@ func NewStunner(options Options) *Stunner { } s.adminManager = manager.NewManager("admin-manager", - object.NewAdminFactory(options.DryRun, s.NewReadinessHandler(), logger), logger) + object.NewAdminFactory(options.DryRun, s.NewReadinessHandler(), s.NewStatusHandler(), logger), logger) s.authManager = manager.NewManager("auth-manager", object.NewAuthFactory(logger), logger) s.listenerManager = manager.NewManager("listener-manager", @@ -205,30 +204,43 @@ func (s *Stunner) AllocationCount() int { return n } -// Status returns a short status description of the running STUNner instance. -func (s *Stunner) Status() string { - listeners := s.listenerManager.Keys() - ls := make([]string, len(listeners)) - for i, l := range listeners { - ls[i] = s.GetListener(l).String() +// Status returns the status for the running STUNner instance. +func (s *Stunner) Status() stnrv1.Status { + status := stnrv1.StunnerStatus{ApiVersion: s.version} + if admin := s.GetAdmin(); admin != nil { + status.Admin = admin.Status().(*stnrv1.AdminStatus) + } + if auth := s.GetAuth(); auth != nil { + status.Auth = auth.Status().(*stnrv1.AuthStatus) + } + + ls := s.listenerManager.Keys() + status.Listeners = make([]*stnrv1.ListenerStatus, len(ls)) + for i, lName := range ls { + if l := s.GetListener(lName); l != nil { + status.Listeners[i] = l.Status().(*stnrv1.ListenerStatus) + } } - str := "NONE" - if len(ls) > 0 { - str = strings.Join(ls, ", ") + + cs := s.clusterManager.Keys() + status.Clusters = make([]*stnrv1.ClusterStatus, len(cs)) + for i, cName := range cs { + if c := s.GetCluster(cName); c != nil { + status.Clusters[i] = c.Status().(*stnrv1.ClusterStatus) + } } - status := "READY" + status.AllocationCount = s.AllocationCount() + stat := "READY" if !s.ready { - status = "NOT-READY" + stat = "NOT-READY" } if s.shutdown { - status = "TERMINATING" + stat = "TERMINATING" } + status.Status = stat - auth := s.GetAuth() - return fmt.Sprintf("status: %s, realm: %s, authentication: %s, listeners: %s"+ - ", active allocations: %d", status, auth.Realm, auth.Type.String(), str, - s.AllocationCount()) + return &status } // Close stops the STUNner daemon, cleans up any internal state, and closes all connections From f9c624c3f9f7943507131257ac1e3386a9ce6d4a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 21 Feb 2024 15:15:25 +0100 Subject: [PATCH 139/243] feat: Implement a status sub-command in stunnerctl --- cmd/stunnerctl/main.go | 208 +++++++++++++++++---- cmd/stunnerd/main.go | 41 +++- cmd/turncat/main.go | 2 +- pkg/config/client/cds_api.go | 151 +++++++++++++++ pkg/config/client/cds_client.go | 149 --------------- pkg/config/client/k8s_client.go | 318 +++++++++++++++++++++++++------- 6 files changed, 612 insertions(+), 257 deletions(-) diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go index a874f976..147b6a64 100644 --- a/cmd/stunnerctl/main.go +++ b/cmd/stunnerctl/main.go @@ -4,11 +4,13 @@ import ( "context" "encoding/json" "fmt" + "net/http" "os" "os/signal" "regexp" "strings" "syscall" + "time" "github.com/pion/logging" "github.com/spf13/cobra" @@ -29,16 +31,16 @@ import ( var ( output string watch, all, verbose bool - jsonQuery *jsonpath.JSONPath k8sConfigFlags *cliopt.ConfigFlags cdsConfigFlags *cdsclient.CDSConfigFlags + podConfigFlags *cdsclient.PodConfigFlags loggerFactory *logger.LeveledLoggerFactory log logging.LeveledLogger rootCmd = &cobra.Command{ Use: "stunnerctl", - Short: "A command line utility to inspect STUNner dataplane configurations.", - Long: "The stunnerctl tool is a CLI for inspecting, watching and troublehssooting the configuration of STUNner gateways", + Short: "A command line utility to inspect STUNner dataplane .", + Long: "The stunnerctl tool is a CLI for inspecting, watching and troublehssooting STUNner gateways", DisableAutoGenTag: true, } ) @@ -47,7 +49,7 @@ var ( configCmd = &cobra.Command{ Use: "config", Aliases: []string{"stunner-config"}, - Short: "Gets or watches STUNner configs", + Short: "Get or watch dataplane configs", Args: cobra.RangeArgs(0, 1), DisableAutoGenTag: true, Run: func(cmd *cobra.Command, args []string) { @@ -57,23 +59,44 @@ var ( } }, } + statusCmd = &cobra.Command{ + Use: "status", + Aliases: []string{"dataplane-status"}, + Short: "Read status from dataplane pods", + Args: cobra.RangeArgs(0, 1), + DisableAutoGenTag: true, + Run: func(cmd *cobra.Command, args []string) { + if err := runStatus(cmd, args); err != nil { + fmt.Println(err) + os.Exit(1) + } + }, + } ) func init() { - rootCmd.PersistentFlags().BoolVarP(&watch, "watch", "w", false, "Watch for config updates from server") rootCmd.PersistentFlags().BoolVarP(&all, "all-namespaces", "a", false, "Consider all namespaces") rootCmd.PersistentFlags().StringVarP(&output, "output", "o", "summary", "Output format") rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") - // Kubernetes config flags + // Kubernetes config flags: persistent, all commands k8sConfigFlags = cliopt.NewConfigFlags(true) k8sConfigFlags.AddFlags(rootCmd.PersistentFlags()) - // CDS server discovery flags + // CDS server discovery flags: only for "config" command cdsConfigFlags = cdsclient.NewCDSConfigFlags() - cdsConfigFlags.AddFlags(rootCmd.PersistentFlags()) + cdsConfigFlags.AddFlags(configCmd.Flags()) + + // watch flag: only for config + configCmd.Flags().BoolVarP(&watch, "watch", "w", false, "Watch for config updates from server") + + // Pod discovery flags: only for "status" command + podConfigFlags = cdsclient.NewPodConfigFlags() + podConfigFlags.AddFlags(statusCmd.Flags()) + // Add commands rootCmd.AddCommand(configCmd) + rootCmd.AddCommand(statusCmd) } func main() { @@ -96,45 +119,31 @@ func runConfig(cmd *cobra.Command, args []string) error { gwNs = *k8sConfigFlags.Namespace } - if strings.HasPrefix(output, "jsonpath") { - as := strings.Split(output, "=") - if len(as) != 2 || as[0] != "jsonpath" { - return fmt.Errorf("invalid jsonpath output definition %q", output) - } - - jsonQuery = jsonpath.New("output") - - // Parse and print jsonpath - fields, err := RelaxedJSONPathExpression(as[1]) - if err != nil { - return fmt.Errorf("invalid jsonpath query %w", err) - } - - if err := jsonQuery.Parse(fields); err != nil { - return fmt.Errorf("cannor parse jsonpath query %w", err) - } - output = "jsonpath" + jsonQuery, output, err := ParseJSONPathFlag(output) + if err != nil { + return err } ctx, cancel := context.WithCancel(context.Background()) defer cancel() log.Debug("Searching for CDS server") - cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, + pod, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, loggerFactory.NewLogger("cds-fwd")) if err != nil { return fmt.Errorf("error searching for CDS server: %w", err) } + log.Debugf("Connecting to CDS server: %s", pod.String()) var cds cdsclient.CdsApi cdslog := loggerFactory.NewLogger("cds-client") if all { - cds, err = cdsclient.NewAllConfigsAPI(cdsAddr, cdslog) + cds, err = cdsclient.NewAllConfigsAPI(pod.Addr, cdslog) } else if len(args) == 0 { - cds, err = cdsclient.NewConfigsNamespaceAPI(cdsAddr, gwNs, cdslog) + cds, err = cdsclient.NewConfigsNamespaceAPI(pod.Addr, gwNs, cdslog) } else { gwName := args[0] - cds, err = cdsclient.NewConfigNamespaceNameAPI(cdsAddr, gwNs, gwName, cdslog) + cds, err = cdsclient.NewConfigNamespaceNameAPI(pod.Addr, gwNs, gwName, cdslog) } if err != nil { @@ -202,9 +211,7 @@ func runConfig(cmd *cobra.Command, args []string) error { } } case "summary": - fmt.Print(string(c.Summary())) - case "status": - fallthrough + fmt.Print(c.Summary()) default: fmt.Println(c.String()) } @@ -213,6 +220,116 @@ func runConfig(cmd *cobra.Command, args []string) error { return nil } +func runStatus(cmd *cobra.Command, args []string) error { + loglevel := "all:WARN" + if verbose { + loglevel = "all:TRACE" + } + loggerFactory = logger.NewLoggerFactory(loglevel) + log = loggerFactory.NewLogger("stunnerctl") + + jsonQuery, output, err := ParseJSONPathFlag(output) + if err != nil { + return err + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + gwNs := "default" + extraLog := "in namespace default" + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { + gwNs = *k8sConfigFlags.Namespace + extraLog = fmt.Sprintf("in namespace %s", gwNs) + } + // --all-namespaces overrides -n + if all { + gwNs = "" + extraLog = "in all namespaces" + } + + gw := "" + if len(args) > 0 { + gw = args[0] + } + if gwNs != "" && gw != "" { + extraLog += fmt.Sprintf("for gateway %s", gw) + } + + log.Debug("Searching for dataplane pods " + extraLog) + pods, err := cdsclient.DiscoverK8sStunnerdPods(ctx, k8sConfigFlags, podConfigFlags, + gwNs, gw, loggerFactory.NewLogger("stunnerd-fwd")) + if err != nil { + return fmt.Errorf("error searching for stunnerd pods: %w", err) + } + + for _, pod := range pods { + client := http.Client{ + Timeout: 5 * time.Second, + } + url := fmt.Sprintf("http://%s/status", pod.Addr) + res, err := client.Get(url) + if err != nil { + log.Errorf("Error querying status for stunnerd pod at URL %q on %s: %s", + url, pod.String(), err.Error()) + continue + } + + if res.StatusCode != http.StatusOK { + log.Errorf("Status query failed on %s with HTTP error code %s", + pod.String(), res.Status) + continue + } + + s := stnrv1.StunnerStatus{} + err = json.NewDecoder(res.Body).Decode(&s) + if err != nil { + log.Errorf("Could not decode status response: %s", err.Error()) + continue + } + + switch output { + case "yaml": + if out, err := yaml.Marshal(s); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "json": + if out, err := json.Marshal(s); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "jsonpath": + values, err := jsonQuery.FindResults(s) + if err != nil { + return err + } + + if len(values) == 0 || len(values[0]) == 0 { + fmt.Println("") + } + + for arrIx := range values { + for valIx := range values[arrIx] { + fmt.Printf("%v\n", values[arrIx][valIx].Interface()) + } + } + case "summary": + fallthrough + default: + if pod.Proxy { + fmt.Printf("%s/%s:\n\t%s\n", pod.Namespace, pod.Name, s.Summary()) + } else { + fmt.Printf("%s:\n\t%s\n", pod.Addr, s.Summary()) + } + } + } + + return nil +} + var jsonRegexp = regexp.MustCompile(`^\{\.?([^{}]+)\}$|^\.?([^{}]+)$`) // k8s.io/kubectl/pkg/cmd/get @@ -235,3 +352,28 @@ func RelaxedJSONPathExpression(pathExpression string) (string, error) { } return fmt.Sprintf("{.%s}", fieldSpec), nil } + +func ParseJSONPathFlag(output string) (*jsonpath.JSONPath, string, error) { + if !strings.HasPrefix(output, "jsonpath") { + return nil, output, nil + } + + as := strings.Split(output, "=") + if len(as) != 2 || as[0] != "jsonpath" { + return nil, output, fmt.Errorf("invalid jsonpath output definition %q", output) + } + + jsonQuery := jsonpath.New("output") + + // Parse and print jsonpath + fields, err := RelaxedJSONPathExpression(as[1]) + if err != nil { + return nil, output, fmt.Errorf("invalid jsonpath query %w", err) + } + + if err := jsonQuery.Parse(fields); err != nil { + return nil, output, fmt.Errorf("cannor parse jsonpath query %w", err) + } + + return jsonQuery, "jsonpath", nil +} diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index d3bdcf09..74f2a7bf 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -9,6 +9,7 @@ import ( "time" flag "github.com/spf13/pflag" + cliopt "k8s.io/cli-runtime/pkg/genericclioptions" "github.com/l7mp/stunner" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" @@ -19,7 +20,7 @@ import ( func main() { os.Args[0] = "stunnerd" - var config = flag.StringP("config", "c", "", "Config origin, either a valid address in the format IP:port, or HTTP URL to the CDS server, or a proper file name URI in the format file:// (overrides: STUNNER_CONFIG_ORIGIN)") + var config = flag.StringP("config", "c", "", "Config origin, either a valid address in the format IP:port, or HTTP URL to the CDS server, or literal \"k8s\" to discover the CDS server from Kubernetes, or a proper file name URI in the format file:// (overrides: STUNNER_CONFIG_ORIGIN)") var level = flag.StringP("log", "l", "", "Log level (format: :, overrides: PION_LOG_*, default: all:INFO)") var id = flag.StringP("id", "i", "", "Id for identifying with the CDS server (format: /, overrides: STUNNER_NAMESPACE/STUNNER_NAME, default: )") var watch = flag.BoolP("watch", "w", false, "Watch config file for updates (default: false)") @@ -27,6 +28,15 @@ func main() { "Number of readloop threads (CPU cores) per UDP listener. Zero disables UDP multithreading (default: 0)") var dryRun = flag.BoolP("dry-run", "d", false, "Suppress side-effects, intended for testing (default: false)") var verbose = flag.BoolP("verbose", "v", false, "Verbose logging, identical to <-l all:DEBUG>") + + // Kubernetes config flags + k8sConfigFlags := cliopt.NewConfigFlags(true) + k8sConfigFlags.AddFlags(flag.CommandLine) + + // CDS server discovery flags + cdsConfigFlags := cdsclient.NewCDSConfigFlags() + cdsConfigFlags.AddFlags(flag.CommandLine) + flag.Parse() logLevel := stnrv1.DefaultLogLevel @@ -83,13 +93,26 @@ func main() { conf <- c } else if !*watch { - log.Infof("loading configuration from origin %q", configOrigin) + ctx, cancel := context.WithCancel(context.Background()) + + if configOrigin == "k8s" { + log.Info("discovering configuration from Kubernetes") + cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, + st.GetLogger().NewLogger("cds-fwd")) + if err != nil { + log.Errorf("error searching for CDS server: %s", err.Error()) + os.Exit(1) + } + configOrigin = cdsAddr.Addr + } + log.Infof("loading configuration from origin %q", configOrigin) c, err := st.LoadConfig(configOrigin) if err != nil { log.Error(err.Error()) os.Exit(1) } + cancel() conf <- c @@ -98,12 +121,22 @@ func main() { z := cdsclient.ZeroConfig(st.GetId()) conf <- z - log.Infof("watching configuration at origin %q", configOrigin) ctx, cancel := context.WithCancel(context.Background()) defer cancel() cancelConfigLoader = cancel - // Watch closes the channel + if configOrigin == "k8s" { + log.Info("discovering configuration from Kubernetes") + cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, + st.GetLogger().NewLogger("cds-fwd")) + if err != nil { + log.Errorf("error searching for CDS server: %s", err.Error()) + os.Exit(1) + } + configOrigin = cdsAddr.Addr + } + + log.Infof("watching configuration at origin %q", configOrigin) if err := st.WatchConfig(ctx, configOrigin, conf); err != nil { log.Errorf("could not run config watcher: %s", err.Error()) os.Exit(1) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index f3e98a23..61c82756 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -167,7 +167,7 @@ func getStunnerConfFromK8s(def string) (*stnrv1.StunnerConfig, error) { return nil, fmt.Errorf("error searching for CDS server: %w", err) } - cds, err := cdsclient.NewConfigNamespaceNameAPI(cdsAddr, namespace, name, + cds, err := cdsclient.NewConfigNamespaceNameAPI(cdsAddr.Addr, namespace, name, loggerFactory.NewLogger("cds-client")) if err != nil { return nil, fmt.Errorf("error creating CDS client: %w", err) diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go index 481ee7ab..f83ea86e 100644 --- a/pkg/config/client/cds_api.go +++ b/pkg/config/client/cds_api.go @@ -6,7 +6,10 @@ import ( "fmt" "net/http" "strings" + "sync" + "time" + "github.com/gorilla/websocket" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/config/client/api" "github.com/pion/logging" @@ -238,3 +241,151 @@ func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- *stnrv1.Stu a.namespace, a.name, a.wsURI) return poll(ctx, a, ch) } + +func watch(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { + go func() { + for { + if err := poll(ctx, a, ch); err != nil { + _, wsuri := a.Endpoint() + a.Errorf("failed to init CDS watcher (url: %s): %s", wsuri, err.Error()) + } else { + // context got cancelled + return + } + + // wait between each attempt + time.Sleep(RetryPeriod) + } + }() + + return nil +} + +// //////////// +// API workers +// //////////// +func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { + _, url := a.Endpoint() + a.Tracef("poll: trying to open connection to CDS server at %s", url) + + conn, _, err := websocket.DefaultDialer.DialContext(ctx, url, makeHeader(url)) + if err != nil { + return err + } + defer conn.Close() // this will close the poller goroutine + + a.Infof("connection successfully opened to config discovery server at %s", url) + + pingTicker := time.NewTicker(PingPeriod) + closePinger := make(chan any) + defer close(closePinger) + + // wait until all threads are closed and we can remove the error channel + errCh := make(chan error, 1) + var wg sync.WaitGroup + wg.Add(2) + + go func() { + defer pingTicker.Stop() + defer wg.Done() + + for { + select { + case <-pingTicker.C: + // p.log.Tracef("++++ PING ++++ for CDS server %q at client %q", location, p.id) + conn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck + if err := conn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { + errCh <- fmt.Errorf("could not ping CDS server at %q: %w", + conn.RemoteAddr(), err) + return + } + case <-closePinger: + a.Tracef("closing ping handler to config discovery server at %q", url) + return + } + } + }() + + // poller + go func() { + defer wg.Done() + + // the next pong must arrive within the PongWait period + conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + // reinit the deadline when we get a pong + conn.SetPongHandler(func(string) error { + // a.Tracef("Got PONG from server %q", url) + conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + return nil + }) + + for { + // ping-pong deadline misses will end up being caught here as a read beyond + // the deadline + msgType, msg, err := conn.ReadMessage() + if err != nil { + errCh <- err + return + } + + if msgType != websocket.TextMessage { + errCh <- fmt.Errorf("unexpected message type (code: %d) from client %q", + msgType, conn.RemoteAddr().String()) + return + } + + if len(msg) == 0 { + a.Warn("ignoring zero-length config") + continue + } + + c, err := ParseConfig(msg) + if err != nil { + // assume it is a YAML/JSON syntax error: report and ignore + a.Warnf("could not parse config: %s", err.Error()) + continue + } + + if err := c.Validate(); err != nil { + a.Warnf("invalid config: %s", err.Error()) + continue + } + + a.Debugf("new config received from %q: %q", url, c.String()) + + ch <- c + } + }() + + // wait fo cancel + for { + defer func() { + a.Infof("closing connection for client %s", conn.RemoteAddr().String()) + conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck + conn.Close() + closePinger <- struct{}{} + wg.Wait() + close(errCh) + }() + + select { + case <-ctx.Done(): + // cancel: normal return + return nil + case err := <-errCh: + // error: return it + return err + } + } +} + +// creates an origin header +func makeHeader(uri string) http.Header { + header := http.Header{} + url, _ := getURI(uri) //nolint:errcheck + origin := *url + origin.Scheme = "http" + origin.Path = "" + header.Set("origin", origin.String()) + return header +} diff --git a/pkg/config/client/cds_client.go b/pkg/config/client/cds_client.go index de79b048..c11e9f75 100644 --- a/pkg/config/client/cds_client.go +++ b/pkg/config/client/cds_client.go @@ -4,12 +4,8 @@ package client import ( "context" "fmt" - "net/http" "strings" - "sync" - "time" - "github.com/gorilla/websocket" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/pion/logging" ) @@ -59,148 +55,3 @@ func (p *CDSClient) Load() (*stnrv1.StunnerConfig, error) { return c, nil } - -func watch(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { - go func() { - for { - if err := poll(ctx, a, ch); err != nil { - _, wsuri := a.Endpoint() - a.Errorf("failed to init CDS watcher (url: %s): %s", wsuri, err.Error()) - } else { - // context got cancelled - return - } - - // wait between each attempt - time.Sleep(RetryPeriod) - } - }() - - return nil -} - -func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { - _, url := a.Endpoint() - a.Tracef("poll: trying to open connection to CDS server at %s", url) - - conn, _, err := websocket.DefaultDialer.DialContext(ctx, url, makeHeader(url)) - if err != nil { - return err - } - defer conn.Close() // this will close the poller goroutine - - a.Infof("connection successfully opened to config discovery server at %s", url) - - pingTicker := time.NewTicker(PingPeriod) - closePinger := make(chan any) - defer close(closePinger) - - // wait until all threads are closed and we can remove the error channel - errCh := make(chan error, 1) - var wg sync.WaitGroup - wg.Add(2) - - go func() { - defer pingTicker.Stop() - defer wg.Done() - - for { - select { - case <-pingTicker.C: - // p.log.Tracef("++++ PING ++++ for CDS server %q at client %q", location, p.id) - conn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck - if err := conn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { - errCh <- fmt.Errorf("could not ping CDS server at %q: %w", - conn.RemoteAddr(), err) - return - } - case <-closePinger: - a.Tracef("closing ping handler to config discovery server at %q", url) - return - } - } - }() - - // poller - go func() { - defer wg.Done() - - // the next pong must arrive within the PongWait period - conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck - // reinit the deadline when we get a pong - conn.SetPongHandler(func(string) error { - // a.Tracef("Got PONG from server %q", url) - conn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck - return nil - }) - - for { - // ping-pong deadline misses will end up being caught here as a read beyond - // the deadline - msgType, msg, err := conn.ReadMessage() - if err != nil { - errCh <- err - return - } - - if msgType != websocket.TextMessage { - errCh <- fmt.Errorf("unexpected message type (code: %d) from client %q", - msgType, conn.RemoteAddr().String()) - return - } - - if len(msg) == 0 { - a.Warn("ignoring zero-length config") - continue - } - - c, err := ParseConfig(msg) - if err != nil { - // assume it is a YAML/JSON syntax error: report and ignore - a.Warnf("could not parse config: %s", err.Error()) - continue - } - - if err := c.Validate(); err != nil { - a.Warnf("invalid config: %s", err.Error()) - continue - } - - a.Debugf("new config received from %q: %q", url, c.String()) - - ch <- c - } - }() - - // wait fo cancel - for { - defer func() { - a.Infof("closing connection for client %s", conn.RemoteAddr().String()) - conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck - conn.Close() - closePinger <- struct{}{} - wg.Wait() - close(errCh) - }() - - select { - case <-ctx.Done(): - // cancel: normal return - return nil - case err := <-errCh: - // error: return it - return err - } - } -} - -// creates an origin header -func makeHeader(uri string) http.Header { - header := http.Header{} - url, _ := getURI(uri) //nolint:errcheck - origin := *url - origin.Scheme = "http" - origin.Path = "" - header.Set("origin", origin.String()) - return header -} diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go index a25d6d1f..78d63d7f 100644 --- a/pkg/config/client/k8s_client.go +++ b/pkg/config/client/k8s_client.go @@ -7,43 +7,36 @@ import ( "net/http" "os" "strconv" - "strings" + "sync" "github.com/pion/logging" "github.com/spf13/pflag" + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/selection" cliopt "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" "k8s.io/client-go/tools/portforward" "k8s.io/client-go/transport/spdy" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" ) -var DefaultCDSServerPort = 13478 - -func init() { - as := strings.Split(stnrv1.DefaultConfigDiscoveryAddress, ":") - if len(as) == 2 { - if port, err := strconv.Atoi(as[1]); err != nil { - DefaultCDSServerPort = port - } - } -} - -// CDSConfigFlags composes a set of flags for CDS server discovery +// CDSConfigFlags composes a set of flags for CDS server discovery. type CDSConfigFlags struct { - // ServerAddr is an explicit IP address for the CDS server. - ServerAddr string - // ServerNamespace is the namespace of the CDS server pod. - ServerNamespace string - // ServerPort is the port of the CDS server pod. - ServerPort int + // Addr is an explicit IP address for the CDS server. + Addr string + // Namespace is the namespace of the CDS server pod. + Namespace string + // Port is the port of the CDS server pod. + Port int } // NewCDSConfigFlags returns CDS service discovery flags with default values set. func NewCDSConfigFlags() *CDSConfigFlags { - port := DefaultCDSServerPort + port := stnrv1.DefaultConfigDiscoveryPort if os.Getenv(stnrv1.DefaultCDSServerPortEnv) != "" { p, err := strconv.Atoi(os.Getenv(stnrv1.DefaultCDSServerPortEnv)) if err != nil { @@ -51,113 +44,298 @@ func NewCDSConfigFlags() *CDSConfigFlags { } } return &CDSConfigFlags{ - ServerAddr: os.Getenv(stnrv1.DefaultCDSServerAddrEnv), - ServerPort: port, - ServerNamespace: os.Getenv(stnrv1.DefaultCDSServerNamespaceEnv), + Addr: os.Getenv(stnrv1.DefaultCDSServerAddrEnv), + Port: port, + Namespace: os.Getenv(stnrv1.DefaultCDSServerNamespaceEnv), } } -// AddFlags binds CDS server discovery configuration flags to a given flagset. +// AddFlags binds pod discovery configuration flags to a given flagset. func (f *CDSConfigFlags) AddFlags(flags *pflag.FlagSet) { - flags.StringVar(&f.ServerAddr, "cds-server-address", f.ServerAddr, - "Config discovery service address (overriders cds-namesapce/name and disables CDS service discovery)") - flags.StringVar(&f.ServerNamespace, "cds-server-namespace", f.ServerNamespace, - "Config discovery service namespace (disables CDS service discovery)") - flags.IntVar(&f.ServerPort, "cds-server-port", f.ServerPort, - "Config discovery service port") + flags.StringVar(&f.Addr, "cds-server-address", f.Addr, + "Config discovery service address (overriders cds-namesapce/name and disables service discovery)") + flags.StringVar(&f.Namespace, "cds-server-namespace", f.Namespace, + "Config discovery service namespace (disables service discovery)") + flags.IntVar(&f.Port, "cds-server-port", f.Port, "Config discovery service port") +} + +// PodConfigFlags composes a set of flags for pod discovery. +type PodConfigFlags struct { + // Addr is an explicit IP address for the pod. + Addr string + // Name is the name of the pod. + Name string + // Port is the port to use. + Port int +} + +// NewPodConfigFlags returns Stunnerd service discovery flags with default values set. +func NewPodConfigFlags() *PodConfigFlags { + return &PodConfigFlags{ + Port: stnrv1.DefaultHealthCheckPort, + } +} + +// AddFlags binds pod discovery configuration flags to a given flagset. +func (f *PodConfigFlags) AddFlags(flags *pflag.FlagSet) { + flags.StringVar(&f.Addr, "pod-address", f.Addr, + "Address of the stunnerd instance to connect to (overrides K8s pod discovery)") + flags.StringVar(&f.Name, "pod-name", f.Name, + "Name of the specific stunnerd pod to connect to (valid only if both -n and gateway name are specified)") + flags.IntVar(&f.Port, "pod-port", f.Port, "Port of the stunnerd pod to connect to") +} + +// PodConnector is a helper for discovering and connecting to pods in a Kubernetes cluster. +type PodConnector struct { + cs *kubernetes.Clientset + config *rest.Config + k8sFlags *cliopt.ConfigFlags + log logging.LeveledLogger +} + +// PodInfo allows to return a full pod descriptor to callers. +type PodInfo struct { + // Name is the name of the pod. Only valid + Name string + // Namespace is the Kubernetes namespace of the pod. + Namespace string + // Addr is the Kubernetes namespace of the pod. + Addr string + // Proxy is a boolean telling whether the connection is proxied over a port-forwarder. + Proxy bool +} + +func (p *PodInfo) String() string { + ret := "" + if p.Proxy { + ret += fmt.Sprintf("pod %s/%s at %s", p.Namespace, p.Name, p.Addr) + } else { + ret += p.Addr + } + return ret +} + +// NewK8sDiscoverer returns a new Kubernetes CDS discovery client. +func NewK8sDiscoverer(k8sFlags *cliopt.ConfigFlags, log logging.LeveledLogger) (*PodConnector, error) { + d := &PodConnector{ + k8sFlags: k8sFlags, + log: log, + } + + d.log.Debug("Obtaining kubeconfig") + config, err := d.k8sFlags.ToRESTConfig() + if err != nil { + return nil, fmt.Errorf("error building Kubernetes config: %w", err) + } + d.config = config + + d.log.Debug("Creating a Kubernetes client") + cs, err := kubernetes.NewForConfig(config) + if err != nil { + return nil, fmt.Errorf("error creating http client: %w", err) + } + d.cs = cs + + return d, nil } // DiscoverK8sCDSServer discovers a CDS Server located in a Kubernetes cluster and returns an // address that a CDS client can be opened to for reaching that CDS server. If necessary, opens a // port-forward connection to the remote cluster. -func DiscoverK8sCDSServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, cdsFlags *CDSConfigFlags, log logging.LeveledLogger) (string, error) { +func DiscoverK8sCDSServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, cdsFlags *CDSConfigFlags, log logging.LeveledLogger) (PodInfo, error) { // if CDS server address is specified, return it - if cdsFlags.ServerAddr != "" { - return cdsFlags.ServerAddr, nil + if cdsFlags.Addr != "" { + return PodInfo{ + Addr: fmt.Sprintf("%s:%d", cdsFlags.Addr, cdsFlags.Port), + Proxy: false, + }, nil } ns := "" nsLog := "" - if cdsFlags.ServerNamespace != "" { - ns = cdsFlags.ServerNamespace + if cdsFlags.Namespace != "" { + ns = cdsFlags.Namespace nsLog = ns } - log.Debug("Obtaining kubeconfig") - config, err := k8sFlags.ToRESTConfig() + d, err := NewK8sDiscoverer(k8sFlags, log) if err != nil { - return "", fmt.Errorf("error building Kubernetes config: %w", err) - } - - log.Debug("Creating a Kubernetes client") - cs, err := kubernetes.NewForConfig(config) - if err != nil { - return "", fmt.Errorf("error creating http client: %w", err) + return PodInfo{}, fmt.Errorf("failed to init CDS discovery client: %w", err) } label := fmt.Sprintf("%s=%s", stnrv1.DefaultCDSServiceLabelKey, stnrv1.DefaultCDSServiceLabelValue) - log.Debugf("Querying CDS server pods in namespace %q using label-selector %q", nsLog, label) - pods, err := cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{ + d.log.Debugf("Querying CDS server pods in namespace %q using label-selector %q", nsLog, label) + + pods, err := d.cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{ LabelSelector: label, }) - if err != nil { - return "", err + return PodInfo{}, fmt.Errorf("failed to query Kubernetes API server: %w", err) } if len(pods.Items) == 0 { - return "", fmt.Errorf("no CDS server found") + return PodInfo{}, fmt.Errorf("no CDS server found") } if len(pods.Items) > 1 { - return "", fmt.Errorf("too many CDS servers") + return PodInfo{}, fmt.Errorf("too many CDS servers") } - name := pods.Items[0].GetName() - namespace := pods.Items[0].GetNamespace() - log.Debugf("Found CDS server: %s/%s", namespace, name) - req := cs.RESTClient(). + return d.portfwd(ctx, &pods.Items[0], cdsFlags.Port) +} + +// DiscoverK8sStunnerdPods discovers the stunnerd pods in a Kubernetes cluster, opens a +// port-forwarded connection to each, and returns a local address that can be used to connect to +// each pod. If gateway is empty, return all stunnerd pods in a namespace. If no namespace is given +// (using the -n CLI flag), query all stunnerd pods in the cluster. +func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, podFlags *PodConfigFlags, gwNs, gw string, log logging.LeveledLogger) ([]PodInfo, error) { + var ps []PodInfo + + // direct connection + if podFlags.Addr != "" { + return []PodInfo{{ + Addr: fmt.Sprintf("%s:%d", podFlags.Addr, podFlags.Port), + Proxy: false, + }}, nil + } + + d, err := NewK8sDiscoverer(k8sFlags, log) + if err != nil { + return ps, fmt.Errorf("failed to init CDS discovery client: %w", err) + } + + selector := labels.NewSelector() + appLabel, err := labels.NewRequirement(stnrv1.DefaultAppLabelKey, + selection.Equals, []string{stnrv1.DefaultAppLabelValue}) + if err != nil { + return ps, fmt.Errorf("cannot create app label selector: %w", err) + } + selector = selector.Add(*appLabel) + + if gwNs != "" { + nsLabel, err := labels.NewRequirement(stnrv1.DefaultRelatedGatewayNamespace, + selection.Equals, []string{gwNs}) + if err != nil { + return ps, fmt.Errorf("cannot create namespace label selector: %w", err) + } + selector = selector.Add(*nsLabel) + + if gw != "" { + gwLabel, err := labels.NewRequirement(stnrv1.DefaultRelatedGatewayKey, + selection.Equals, []string{gw}) + if err != nil { + return ps, fmt.Errorf("cannot create namespace label selector: %w", err) + } + selector = selector.Add(*gwLabel) + } + } + + d.log.Debugf("Calling GET on api/pods using namespace %q and label selector %q", + gwNs, selector.String()) + pods, err := d.cs.CoreV1().Pods(gwNs).List(context.TODO(), metav1.ListOptions{ + LabelSelector: selector.String(), + }) + if err != nil { + return ps, fmt.Errorf("failed to query Kubernetes API server: %w", err) + } + + // filter by pod name + if gwNs != "" && gw != "" && podFlags.Name != "" { + found := false + for i, p := range pods.Items { + if p.GetName() == podFlags.Name { + // keep only the i-th pod + d.log.Debugf("Enforcing pod %s/%s for gateway %s/%s", *k8sFlags.Namespace, gwNs, gw) + pods.Items = pods.Items[i : i+1] + found = true + break + } + } + if !found { + return ps, fmt.Errorf("pod %q not found for gateway %s/%s", + podFlags.Name, gwNs, gw) + } + } + + // open port-forwarders in parallel + var wg sync.WaitGroup + var lock sync.Mutex + ps = make([]PodInfo, len(pods.Items)) + wg.Add(len(pods.Items)) + for i := range pods.Items { + go func(j int) { + defer wg.Done() + pod := pods.Items[j] + + p, err := d.portfwd(ctx, &pod, podFlags.Port) + if err != nil { + d.log.Errorf("failed to create port-forwarder to stunnerd pod %s/%s: %s", + pod.GetNamespace(), pod.GetName(), err.Error()) + return + } + + lock.Lock() + defer lock.Unlock() + ps[j] = p + }(i) + } + + wg.Wait() + + d.log.Debugf("Successfully opened %d port-forward connections", len(pods.Items)) + + return ps, nil +} + +func (d *PodConnector) portfwd(ctx context.Context, pod *corev1.Pod, port int) (PodInfo, error) { + p := PodInfo{ + Name: pod.GetName(), + Namespace: pod.GetNamespace(), + Proxy: true, + } + d.log.Debugf("Found pod: %s/%s", p.Namespace, p.Name) + req := d.cs.RESTClient(). Post(). Prefix("api/v1"). Resource("pods"). - Namespace(namespace). - Name(name). + Namespace(p.Namespace). + Name(p.Name). SubResource("portforward") - log.Debugf("Creating a SPDY stream to API server using URL %q", req.URL().String()) - transport, upgrader, err := spdy.RoundTripperFor(config) + d.log.Debugf("Creating a SPDY stream to API server using URL %q", req.URL().String()) + transport, upgrader, err := spdy.RoundTripperFor(d.config) if err != nil { - return "", fmt.Errorf("error getting transport/upgrader from restconfig: %w", err) + return PodInfo{}, fmt.Errorf("error getting transport/upgrader from restconfig: %w", err) } dialer := spdy.NewDialer(upgrader, &http.Client{Transport: transport}, http.MethodPost, req.URL()) - log.Debug("Creating a port-forwarder to CDS server") + d.log.Debugf("Creating a port-forwarder to pod") + remoteAddr := fmt.Sprintf("0:%d", port) stopChan, readyChan := make(chan struct{}, 1), make(chan struct{}, 1) out, errOut := new(bytes.Buffer), new(bytes.Buffer) - - fw, err := portforward.New(dialer, []string{fmt.Sprintf("0:%d", cdsFlags.ServerPort)}, stopChan, readyChan, out, errOut) + fw, err := portforward.New(dialer, []string{remoteAddr}, stopChan, readyChan, out, errOut) if err != nil { - return "", fmt.Errorf("error creating port-forwarder: %w", err) + return PodInfo{}, fmt.Errorf("error creating port-forwarder: %w", err) } go func() { if err := fw.ForwardPorts(); err != nil { - log.Errorf("error setting up port-forwarder: %s", err.Error()) + d.log.Errorf("error setting up port-forwarder: %s", err.Error()) os.Exit(1) } }() - log.Debug("Waiting for port-forwarder...") + d.log.Debug("Waiting for port-forwarder...") <-readyChan localPort, err := fw.GetPorts() if err != nil { - return "", fmt.Errorf("error obtaining local forwarder port: %w", err) + return PodInfo{}, fmt.Errorf("error obtaining local forwarder port: %w", err) } if len(localPort) != 1 { - return "", fmt.Errorf("error setting up port-forwarder: required port pairs (1) "+ + return PodInfo{}, fmt.Errorf("error setting up port-forwarder: required port pairs (1) "+ "does not match the length of port forwarder port pairs (%d)", len(localPort)) } @@ -166,7 +344,7 @@ func DiscoverK8sCDSServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, cds close(stopChan) }() - addr := fmt.Sprintf("127.0.0.1:%d", localPort[0].Local) - log.Debugf("CDS server reachable on address %q", addr) - return addr, nil + p.Addr = fmt.Sprintf("127.0.0.1:%d", localPort[0].Local) + d.log.Debugf("Port-forwarder connected to %s", p.String()) + return p, nil } From 3397486affdf0afeb33b5a8b08c299df36e0280d Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 21 Feb 2024 15:15:59 +0100 Subject: [PATCH 140/243] chore: Define CDS server K8s flags as per of the public API --- pkg/apis/v1/default.go | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 6dac5136..1c74bf4b 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -23,15 +23,26 @@ const ( DefaultHealthCheckPort int = 8086 ) +// Label/anotation defaults +const ( + DefaultCDSServiceLabelKey = "stunner.l7mp.io/config-discovery-service" + DefaultCDSServiceLabelValue = "enabled" + DefaultAppLabelKey = "app" + DefaultAppLabelValue = "stunner" + DefaultRelatedGatewayKey = "stunner.l7mp.io/related-gateway-name" + DefaultRelatedGatewayNamespace = "stunner.l7mp.io/related-gateway-namespace" + DefaultOwnedByLabelKey = "stunner.l7mp.io/owned-by" + DefaultOwnedByLabelValue = "stunner" +) + // CDS defaults const ( + DefaultConfigDiscoveryPort = 13478 DefaultConfigDiscoveryAddress = ":13478" DefaultEnvVarName = "STUNNER_NAME" DefaultEnvVarNamespace = "STUNNER_NAMESPACE" DefaultEnvVarNodeName = "STUNNER_NODENAME" DefaultEnvVarConfigOrigin = "STUNNER_CONFIG_ORIGIN" - DefaultCDSServiceLabelKey = "stunner.l7mp.io/config-discovery-service" - DefaultCDSServiceLabelValue = "enabled" DefaultCDSServerAddrEnv = "CDS_SERVER_ADDR" DefaultCDSServerNamespaceEnv = "CDS_SERVER_NAMESPACE" DefaultCDSServerPortEnv = "CDS_SERVER_PORT" From fd532d82622478e157e9655fa034fce2c3dcaf5a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 15 Feb 2024 18:43:13 +0100 Subject: [PATCH 141/243] doc: README updates --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 85e972fb..48e33cfc 100644 --- a/README.md +++ b/README.md @@ -352,7 +352,7 @@ gateway in human readable format. ```console stunnerctl -n stunner config udp-gateway Gateway: stunner/udp-gateway (loglevel: "all:INFO") -Authentication type: ephemeral, shared-secret: my-very-secure-secret +Authentication type: static, username/password: user-1/pass-1 Listeners: - Name: stunner/udp-gateway/udp-listener Protocol: TURN-UDP @@ -390,9 +390,9 @@ a heartwarming welcome message. 1. We also need a STUN/TURN client to actually initiate a connection. STUNner comes with a handy STUN/TURN client called [`turncat`](cmd/turncat/README.md) for this purpose. Once - [built](cmd/turncat/README.md#installation), you can fire up `turncat` to listen on the standard - input and send everything it receives to STUNner. Type any input and press Enter, and you should - see a nice greeting from your cluster! + [installed](cmd/turncat/README.md#installation), you can fire up `turncat` to listen on the + standard input and send everything it receives to STUNner. Type any input and press Enter, and + you should see a nice greeting from your cluster! ```console ./turncat - k8s://stunner/udp-gateway:udp-listener udp://${PEER_IP}:9001 From ce15c430adb95ba124e1db9e4c340a5cb4762393 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Wed, 21 Feb 2024 16:19:07 +0100 Subject: [PATCH 142/243] chore: Add helper scripts to download stunner tools (#124) * feat: Add helper script to download `stunnerctl` * feat: Add helper script to download `turncat` --- cmd/getstunner/getstunner.sh | 80 ++++++++++++++++++++++++++++++++++++ docs/cmd/stunnerctl.md | 11 ++++- docs/cmd/turncat.md | 11 ++++- 3 files changed, 100 insertions(+), 2 deletions(-) create mode 100755 cmd/getstunner/getstunner.sh diff --git a/cmd/getstunner/getstunner.sh b/cmd/getstunner/getstunner.sh new file mode 100755 index 00000000..d7853b6a --- /dev/null +++ b/cmd/getstunner/getstunner.sh @@ -0,0 +1,80 @@ +#!/bin/sh + +# STUNner tools downloader script +# +# inspired by https://raw.githubusercontent.com/istio/istio/master/release/downloadIstioCtl.sh + + +REPO=levaitamas/stunner + +# Determine OS +OS="${TARGET_OS:-$(uname)}" +if [ "${OS}" = "Darwin" ] ; then + OSEXT="darwin" +else + OSEXT="linux" +fi + +# Determine the latest STUNner version +if [ "${STUNNER_VERSION}" = "" ] ; then + STUNNER_VERSION="$(curl -Lsf https://api.github.com/repos/${REPO}/releases/latest \ + | grep -o '"tag_name": "v[0-9]*.[0-9]*.[0-9]*' \ + | awk -F'"' '{print $4}')" + STUNNER_VERSION="${STUNNER_VERSION##*/}" +fi + +# Determine build params +if [ "${STUNNER_VERSION}" = "" ] ; then + printf "Unable to get latest Stunner version. Set STUNNER_VERSION env var and re-run. For example: export STUNNER_VERSION=0.18.0" + exit 1; +fi + +LOCAL_ARCH=$(uname -m) +if [ "${TARGET_ARCH}" ]; then + LOCAL_ARCH=${TARGET_ARCH} +fi + +case "${LOCAL_ARCH}" in + x86_64|amd64) + STUNNER_ARCH=amd64 + ;; + armv8*|aarch64*|arm64) + STUNNER_ARCH=arm64 + ;; + *) + echo "This system's architecture, ${LOCAL_ARCH}, isn't supported" + exit 1 + ;; +esac + +# Download binaries +progs="stunnerctl turncat" +tmp=$(mktemp -d /tmp/stunner.XXXXXX) + +for prog in $progs; do + NAME="${prog}-${STUNNER_VERSION}" + URL="https://github.com/${REPO}/releases/download/${STUNNER_VERSION}/${prog}-${STUNNER_VERSION}-${OSEXT}-${STUNNER_ARCH}" + filename="${prog}-${STUNNER_VERSION}-${OSEXT}-${STUNNER_ARCH}" + + printf "\nDownloading %s from %s ...\n" "${NAME}" "$URL" + if ! curl -o /dev/null -sIf "$URL"; then + printf "\n%s is not found, please specify a valid STUNNER_VERSION and TARGET_ARCH\n" "$URL" + exit 1 + fi + curl -fsL -o "${tmp}/${filename}" "$URL" + printf "%s download complete!\n" "${filename}" + + mkdir -p "$HOME/.l7mp/bin" + mv "${tmp}/${filename}" "$HOME/.l7mp/bin/${prog}" + chmod +x "$HOME/.l7mp/bin/${prog}" +done + +rm -r "${tmp}" + +# Print final message +printf "\n" +printf "Add stunner tools to your path with:" +printf "\n" +printf " export PATH=\$HOME/.l7mp/bin:\$PATH \n" +printf "\n" +printf "Need more information? Visit https://docs.l7mp.io/en/${STUNNER_VERSION}/ \n" diff --git a/docs/cmd/stunnerctl.md b/docs/cmd/stunnerctl.md index 4e017234..0d5c230e 100644 --- a/docs/cmd/stunnerctl.md +++ b/docs/cmd/stunnerctl.md @@ -5,7 +5,16 @@ The prominent use of `stunnerctl` is to load or watch STUNner dataplane configur ## Installation -Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. +On Linux and macOS, Use [this script](/cmd/getstunner/getstunner.sh) to download the latest version of the `stunnerctl` binary: + +```console +curl -sL https://raw.githubusercontent.com/l7mp/stunner/main/cmd/getstunner/getstunner.sh | sh - +export PATH=$HOME/.l7mp/bin:$PATH +``` +> [!NOTE] +> The script installs `turncat` too. + +Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. ```console go install github.com/l7mp/stunner/cmd/stunnerctl@latest diff --git a/docs/cmd/turncat.md b/docs/cmd/turncat.md index 9589affe..8d2fd9ef 100644 --- a/docs/cmd/turncat.md +++ b/docs/cmd/turncat.md @@ -7,7 +7,16 @@ This is much faster than the TCP connection used by `kubectl`. ## Installation -Install the `turncat` binary using the standard Go toolchain and add it to `$PATH`. +On Linux and macOS, use [this script](/cmd/getstunner/getstunner.sh) to download the latest version of the `turncat` binary: + +```console +curl -sL https://raw.githubusercontent.com/l7mp/stunner/main/cmd/getstunner/getstunner.sh | sh - +export PATH=$HOME/.l7mp/bin:$PATH +``` +> [!NOTE] +> The script installs `stunnerctl` too. + +Install the `turncat` binary using the standard Go toolchain and add it to `$PATH`. ```console go install github.com/l7mp/stunner/cmd/turncat@latest From 0bcea5db993d9d912d9e849e9eb64da5aaf4d068 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 21 Feb 2024 16:28:32 +0100 Subject: [PATCH 143/243] fix(getstunner): Fix repo name --- cmd/getstunner/getstunner.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/getstunner/getstunner.sh b/cmd/getstunner/getstunner.sh index d7853b6a..bffbc12b 100755 --- a/cmd/getstunner/getstunner.sh +++ b/cmd/getstunner/getstunner.sh @@ -5,7 +5,7 @@ # inspired by https://raw.githubusercontent.com/istio/istio/master/release/downloadIstioCtl.sh -REPO=levaitamas/stunner +REPO=l7mp/stunner # Determine OS OS="${TARGET_OS:-$(uname)}" From c9ebbbfba888e865b31d22e1664241a36983053f Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 21 Feb 2024 18:05:51 +0100 Subject: [PATCH 144/243] feat: Implement a basic auth-service client in stunnerctl --- cmd/stunnerctl/main.go | 83 +++++++++++++++++++++++++++++++++ pkg/apis/v1/default.go | 6 ++- pkg/config/client/k8s_client.go | 71 ++++++++++++++++++++++++++++ 3 files changed, 158 insertions(+), 2 deletions(-) diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go index 147b6a64..b1c90b61 100644 --- a/cmd/stunnerctl/main.go +++ b/cmd/stunnerctl/main.go @@ -4,7 +4,9 @@ import ( "context" "encoding/json" "fmt" + "io" "net/http" + "net/url" "os" "os/signal" "regexp" @@ -33,6 +35,7 @@ var ( watch, all, verbose bool k8sConfigFlags *cliopt.ConfigFlags cdsConfigFlags *cdsclient.CDSConfigFlags + authConfigFlags *cdsclient.AuthConfigFlags podConfigFlags *cdsclient.PodConfigFlags loggerFactory *logger.LeveledLoggerFactory log logging.LeveledLogger @@ -72,6 +75,19 @@ var ( } }, } + authCmd = &cobra.Command{ + Use: "auth", + Aliases: []string{"get-credential"}, + Short: "Obtain authenticaction credentials to a gateway", + Args: cobra.RangeArgs(0, 1), + DisableAutoGenTag: true, + Run: func(cmd *cobra.Command, args []string) { + if err := runAuth(cmd, args); err != nil { + fmt.Println(err) + os.Exit(1) + } + }, + } ) func init() { @@ -94,9 +110,14 @@ func init() { podConfigFlags = cdsclient.NewPodConfigFlags() podConfigFlags.AddFlags(statusCmd.Flags()) + // Auth discovery flags: only for "auth" command + authConfigFlags = cdsclient.NewAuthConfigFlags() + authConfigFlags.AddFlags(authCmd.Flags()) + // Add commands rootCmd.AddCommand(configCmd) rootCmd.AddCommand(statusCmd) + rootCmd.AddCommand(authCmd) } func main() { @@ -330,6 +351,68 @@ func runStatus(cmd *cobra.Command, args []string) error { return nil } +func runAuth(cmd *cobra.Command, args []string) error { + loglevel := "all:WARN" + if verbose { + loglevel = "all:TRACE" + } + loggerFactory = logger.NewLoggerFactory(loglevel) + log = loggerFactory.NewLogger("stunnerctl") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + log.Debug("Searching for authentication server") + pod, err := cdsclient.DiscoverK8sAuthServer(ctx, k8sConfigFlags, authConfigFlags, + loggerFactory.NewLogger("auth-fwd")) + if err != nil { + return fmt.Errorf("error searching for auth service: %w", err) + } + + u := url.URL{ + Scheme: "http", + Host: pod.Addr, + Path: "/ice", + } + q := u.Query() + q.Set("service", "turn") + u.RawQuery = q.Encode() + + if authConfigFlags.TurnAuth { + // enforce TURN credential format + u.Path = "" + } + + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { + q := u.Query() + q.Set("namespace", *k8sConfigFlags.Namespace) + if len(args) > 0 { + q.Set("gateway", args[0]) + } + u.RawQuery = q.Encode() + } + + log.Debugf("Querying to authentication server %s using URL %q", pod.String(), u.String()) + res, err := http.Get(u.String()) + if err != nil { + return fmt.Errorf("error querying auth service %s: %w", pod.String(), err) + } + if res.StatusCode != http.StatusOK { + return fmt.Errorf("HTTP error querying auth service %s: expected status %d, got %d", + pod.String(), http.StatusOK, res.StatusCode) + } + + b, err := io.ReadAll(res.Body) + if err != nil { + return fmt.Errorf("cannot read HTTP response: %w", err) + } + + fmt.Println(string(b)) + + return nil +} + +// //////////////////////// var jsonRegexp = regexp.MustCompile(`^\{\.?([^{}]+)\}$|^\.?([^{}]+)$`) // k8s.io/kubectl/pkg/cmd/get diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 1c74bf4b..8207d38c 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -17,18 +17,20 @@ const ( DefaultAuthName = "default-auth-config" ) -// health-check and metrics reporting defaults +// default ports const ( DefaultMetricsPort int = 8080 DefaultHealthCheckPort int = 8086 + DefaultAuthServicePort int = 8088 ) -// Label/anotation defaults +// Label/annotation defaults const ( DefaultCDSServiceLabelKey = "stunner.l7mp.io/config-discovery-service" DefaultCDSServiceLabelValue = "enabled" DefaultAppLabelKey = "app" DefaultAppLabelValue = "stunner" + DefaultAuthAppLabelValue = "stunner-auth" DefaultRelatedGatewayKey = "stunner.l7mp.io/related-gateway-name" DefaultRelatedGatewayNamespace = "stunner.l7mp.io/related-gateway-namespace" DefaultOwnedByLabelKey = "stunner.l7mp.io/owned-by" diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go index 78d63d7f..fca5f310 100644 --- a/pkg/config/client/k8s_client.go +++ b/pkg/config/client/k8s_client.go @@ -85,6 +85,35 @@ func (f *PodConfigFlags) AddFlags(flags *pflag.FlagSet) { flags.IntVar(&f.Port, "pod-port", f.Port, "Port of the stunnerd pod to connect to") } +// AuthConfigFlags composes a set of flags for authentication service discovery. +type AuthConfigFlags struct { + // Addr is an explicit IP address for the server. + Addr string + // Namespace is the namespace of the server pod. + Namespace string + // Port is the port of the server pod. + Port int + // Enforce turn credential. + TurnAuth bool +} + +// NewAuthConfigFlags returns auth service discovery flags with default values set. +func NewAuthConfigFlags() *AuthConfigFlags { + return &AuthConfigFlags{ + Port: stnrv1.DefaultAuthServicePort, + } +} + +// AddFlags binds pod discovery configuration flags to a given flagset. +func (f *AuthConfigFlags) AddFlags(flags *pflag.FlagSet) { + flags.StringVar(&f.Addr, "auth-server-address", f.Addr, + "Auth service address (disables service discovery)") + flags.StringVar(&f.Namespace, "auth-service-namespace", f.Namespace, + "Auth service namespace (disables service discovery)") + flags.IntVar(&f.Port, "auth-service-port", f.Port, "Auth service port") + flags.BoolVar(&f.TurnAuth, "auth-turn-credential", f.TurnAuth, "Request TURN credentials (default: request ICE server config)") +} + // PodConnector is a helper for discovering and connecting to pods in a Kubernetes cluster. type PodConnector struct { cs *kubernetes.Clientset @@ -287,6 +316,48 @@ func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, return ps, nil } +// DiscoverK8sAuthServer discovers the cluster authentication service. +func DiscoverK8sAuthServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, authFlags *AuthConfigFlags, log logging.LeveledLogger) (PodInfo, error) { + if authFlags.Addr != "" { + return PodInfo{ + Addr: fmt.Sprintf("%s:%d", authFlags.Addr, authFlags.Port), + Proxy: false, + }, nil + } + + ns := "" + nsLog := "" + if authFlags.Namespace != "" { + ns = authFlags.Namespace + nsLog = ns + } + + d, err := NewK8sDiscoverer(k8sFlags, log) + if err != nil { + return PodInfo{}, fmt.Errorf("failed to init CDS discovery client: %w", err) + } + + label := fmt.Sprintf("%s=%s", stnrv1.DefaultAppLabelKey, stnrv1.DefaultAuthAppLabelValue) + d.log.Debugf("Querying auth service pods in namespace %q using label-selector %q", nsLog, label) + + pods, err := d.cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{ + LabelSelector: label, + }) + if err != nil { + return PodInfo{}, fmt.Errorf("failed to query Kubernetes API server: %w", err) + } + + if len(pods.Items) == 0 { + return PodInfo{}, fmt.Errorf("no authentication found") + } + + if len(pods.Items) > 1 { + d.log.Infof("mulitple (%d) authentication service instances found, using the first one", len(pods.Items)) + } + + return d.portfwd(ctx, &pods.Items[0], authFlags.Port) +} + func (d *PodConnector) portfwd(ctx context.Context, pod *corev1.Pod, port int) (PodInfo, error) { p := PodInfo{ Name: pod.GetName(), From 0b06ecde8a402e7d0bccf00bc8973229ac4c7892 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 21 Feb 2024 18:30:40 +0100 Subject: [PATCH 145/243] chore(CI/CD): Skip coverage calculation in forks --- .github/workflows/test.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index e4dac3e2..d799c289 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -32,6 +32,7 @@ jobs: run: go test -v -covermode=count coverage: runs-on: ubuntu-latest + if: github.repository == ‘l7mp/stunner’ steps: - name: Install Go if: success() From 7c6ca5dba641aa9131ea24151c7c21ba6030a4f5 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 21 Feb 2024 18:33:17 +0100 Subject: [PATCH 146/243] fix(CI/CD): Fix invalid characters in 'test' --- .github/workflows/test.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index d799c289..3cc8c3f4 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -32,7 +32,7 @@ jobs: run: go test -v -covermode=count coverage: runs-on: ubuntu-latest - if: github.repository == ‘l7mp/stunner’ + if: github.repository == 'l7mp/stunner' steps: - name: Install Go if: success() From 466b67b7cfea918d3d6f7cb2b45ddc310593c6b1 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 21 Feb 2024 22:54:34 +0100 Subject: [PATCH 147/243] doc: Document the new stunnerctl commands (status and auth) --- cmd/stunnerctl/README.md | 72 ++++++++++++++++++++++++++++------ docs/cmd/stunnerctl.md | 85 +++++++++++++++++++++++++++++----------- 2 files changed, 124 insertions(+), 33 deletions(-) diff --git a/cmd/stunnerctl/README.md b/cmd/stunnerctl/README.md index 5d3c7d5c..7b63962c 100644 --- a/cmd/stunnerctl/README.md +++ b/cmd/stunnerctl/README.md @@ -26,7 +26,11 @@ go build -o stunnerctl cmd/stunnerctl/main.go ## Usage -Type `stunnerctl` to get a glimpse of the features provided. Below are some common usage examples. +Type `stunnerctl` to get a glimpse of the sub-commands and features provided. + +### Config + +The `config` sub-command is used to load or watch running dataplane configs from the STUNner config discovery service (CDS) running in a remote Kubernetes cluster. Usually the CDS server role is fulfilled by the [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) but you can choose any CDS service you want (see the `--cds-server-*` CLI flags in the help). The main use of this command is to check the active dataplane configuration for troubleshooting connectivity problems. - Dump a summary of the running config of the STUNner gateway called `udp-gateway` deployed into the `stunner` namespace: @@ -42,7 +46,13 @@ Type `stunnerctl` to get a glimpse of the features provided. Below are some comm Endpoints: [10.76.1.3, 10.80.7.104] ``` -- Dump a the running config of all gateways in the `stunner` namespace in JSON format (YAML is also available using `-o yaml`): +- The same, but using the alternative Kubernetes config file `~/my-config.conf` to access the cluster. The rest of the usual `kubectl` flags (`--context`, `--token`, etc.) are also available to select the cluster to connect to. + + ``` console + stunnerctl --kubeconfig ~/my-config.conf -n stunner config udp-gateway + ``` + +- Dump the running config of all gateways in the `stunner` namespace in JSON format (YAML is also available using `-o yaml`): ```console stunnerctl -n stunner config -o json @@ -50,7 +60,7 @@ Type `stunnerctl` to get a glimpse of the features provided. Below are some comm {"version":"v1","admin":{"name":"stunner/udp-gateway",...}}} ``` -- Watch all STUNner configs as they are being refreshed and dump only the name of the STUNner gateway whose config changes: +- Watch STUNner configs as they are being refreshed by the operator and dump only the name of the gateway whose config changes: ```console stunnerctl config --all-namespaces -o jsonpath='{.admin.name}' -w @@ -59,8 +69,6 @@ Type `stunnerctl` to get a glimpse of the features provided. Below are some comm ... ``` -## Fallback - For those who don't have the Go toolchain available to run `go install`, STUNner provides a minimalistic `stunnerctl` replacement called `stunnerctl.sh`. This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. @@ -80,10 +88,7 @@ Listener 1 Public port: 3478 ``` -## Last resort - -You can also use `kubectl port-forward` to load or watch STUNner configs manually. -Open a port-forwarded connection to the STUNner gateway operator: +You can also use `kubectl port-forward` to load or watch STUNner configs manually. Open a port-forwarded connection to the STUNner gateway operator: ``` console export CDS_SERVER_NAME=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.name}') @@ -91,8 +96,7 @@ export CDS_SERVER_NAMESPACE=$(kubectl get pods -l stunner.l7mp.io/config-discove kubectl -n $CDS_SERVER_NAMESPACE port-forward pod/${CDS_SERVER_NAME} 63478:13478 & ``` -If all goes well, you can now connect to the STUNner config discovery API served by the gateway operator directly, just using `curl`. -The below will load the config of the `udp-gateway` in the `stunner` namespace: +If all goes well, you can now connect to the STUNner CDS API served by the gateway operator through the port-forwarded tunnel opened by `kubectl` just using `curl`. The below will load the config of the `udp-gateway` in the `stunner` namespace: ``` console curl -s http://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway @@ -104,6 +108,52 @@ If you happen to have a WebSocket client like the wonderful [`websocat`](https:/ websocat ws://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway?watch=true - ``` +### Status + +The `status` sub-command reports the status of the dataplane pods for a gateway, especially the runtime state of the `stunnerd` daemon. + +- Find all dataplane pods for the `udp-gateway` in the `stunner` namespace and dump a status summary: + + ``` console + stunnerctl -n stunner status udp-gateway + stunner/udp-gateway-856c9f4dc9-524hc: + stunner/udp-gateway:{logLevel="all:INFO",health-check="http://:8086"} + static-auth:{realm="stunner.l7mp.io",username="",password=""} + listeners:1/clusters:1 + allocs:3/status=READY + stunner/udp-gateway-856c9f4dc9-c7wcq: + stunner/udp-gateway:{logLevel="all:INFO",health-check="http://:8086"} + static-auth:{realm="stunner.l7mp.io",username="",password=""} + listeners:1/clusters:1 + allocs:2/status=READY + ``` + +- Same but report only the runtime status of the `stunnerd` pods in the `stunner` namespace: + + ``` console + stunnerctl -n stunner status -o jsonpath='{.status}' + READY + TERMINATING + ``` + +### Authentication + +The `auth` sub-command can be used to obtain a TURN credential or a full ICE server config for connecting to a specific gateway. The authentication service API is usually served by a separate [STUNner authentication server](https://github.com/l7mp/stunner-auth-service) deployed alongside the gateway operator. The main use of this command is to feed an ICE agent manually with the ICE server config to connect to a specific STUNner gateway. + +- Obtain a full ICE server config for `udp-gateway` deployed into the `stunner` namespace: + + ``` console + stunnerctl -n stunner auth udp-gateway + {"iceServers":[{"credential":"pass-1","urls":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"}],"iceTransportPolicy":"all"} + ``` + +- Request a plain [TURN credential](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00) using the authentication service deployed into the `stunner-system-prod` namespace: + + ``` console + stunnerctl -n stunner auth udp-gateway --auth-turn-credential --auth-service-namespace=stunner-system-prod + {"password":"pass-1","ttl":86400,"uris":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"} + ``` + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). diff --git a/docs/cmd/stunnerctl.md b/docs/cmd/stunnerctl.md index 0d5c230e..7b63962c 100644 --- a/docs/cmd/stunnerctl.md +++ b/docs/cmd/stunnerctl.md @@ -5,16 +5,7 @@ The prominent use of `stunnerctl` is to load or watch STUNner dataplane configur ## Installation -On Linux and macOS, Use [this script](/cmd/getstunner/getstunner.sh) to download the latest version of the `stunnerctl` binary: - -```console -curl -sL https://raw.githubusercontent.com/l7mp/stunner/main/cmd/getstunner/getstunner.sh | sh - -export PATH=$HOME/.l7mp/bin:$PATH -``` -> [!NOTE] -> The script installs `turncat` too. - -Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. +Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. ```console go install github.com/l7mp/stunner/cmd/stunnerctl@latest @@ -35,7 +26,11 @@ go build -o stunnerctl cmd/stunnerctl/main.go ## Usage -Type `stunnerctl` to get a glimpse of the features provided. Below are some common usage examples. +Type `stunnerctl` to get a glimpse of the sub-commands and features provided. + +### Config + +The `config` sub-command is used to load or watch running dataplane configs from the STUNner config discovery service (CDS) running in a remote Kubernetes cluster. Usually the CDS server role is fulfilled by the [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) but you can choose any CDS service you want (see the `--cds-server-*` CLI flags in the help). The main use of this command is to check the active dataplane configuration for troubleshooting connectivity problems. - Dump a summary of the running config of the STUNner gateway called `udp-gateway` deployed into the `stunner` namespace: @@ -51,7 +46,13 @@ Type `stunnerctl` to get a glimpse of the features provided. Below are some comm Endpoints: [10.76.1.3, 10.80.7.104] ``` -- Dump a the running config of all gateways in the `stunner` namespace in JSON format (YAML is also available using `-o yaml`): +- The same, but using the alternative Kubernetes config file `~/my-config.conf` to access the cluster. The rest of the usual `kubectl` flags (`--context`, `--token`, etc.) are also available to select the cluster to connect to. + + ``` console + stunnerctl --kubeconfig ~/my-config.conf -n stunner config udp-gateway + ``` + +- Dump the running config of all gateways in the `stunner` namespace in JSON format (YAML is also available using `-o yaml`): ```console stunnerctl -n stunner config -o json @@ -59,7 +60,7 @@ Type `stunnerctl` to get a glimpse of the features provided. Below are some comm {"version":"v1","admin":{"name":"stunner/udp-gateway",...}}} ``` -- Watch all STUNner configs as they are being refreshed and dump only the name of the STUNner gateway whose config changes: +- Watch STUNner configs as they are being refreshed by the operator and dump only the name of the gateway whose config changes: ```console stunnerctl config --all-namespaces -o jsonpath='{.admin.name}' -w @@ -68,8 +69,6 @@ Type `stunnerctl` to get a glimpse of the features provided. Below are some comm ... ``` -## Fallback - For those who don't have the Go toolchain available to run `go install`, STUNner provides a minimalistic `stunnerctl` replacement called `stunnerctl.sh`. This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. @@ -77,7 +76,7 @@ The below will dump the running config of `tcp-gateway` deployed into the `stunn ```console cd stunner -cmd/stunnerctl/stunnerctl.sh running-config stunner/stunner-gateway +cmd/stunnerctl/stunnerctl.sh running-config stunner/tcp-gateway STUN/TURN authentication type: static STUN/TURN username: user-1 STUN/TURN password: pass-1 @@ -89,10 +88,7 @@ Listener 1 Public port: 3478 ``` -## Last resort - -You can use `kubectl port-forward` to load or watch STUNner configs manually. -Open a port-forwarded connection to the STUNner gateway operator: +You can also use `kubectl port-forward` to load or watch STUNner configs manually. Open a port-forwarded connection to the STUNner gateway operator: ``` console export CDS_SERVER_NAME=$(kubectl get pods -l stunner.l7mp.io/config-discovery-service=enabled --all-namespaces -o jsonpath='{.items[0].metadata.name}') @@ -100,8 +96,7 @@ export CDS_SERVER_NAMESPACE=$(kubectl get pods -l stunner.l7mp.io/config-discove kubectl -n $CDS_SERVER_NAMESPACE port-forward pod/${CDS_SERVER_NAME} 63478:13478 & ``` -If all goes well, you can now connect to the STUNner config discovery API served by the gateway operator directly, just using `curl`. -The below will load the config of the `udp-gateway` in the `stunner` namespace: +If all goes well, you can now connect to the STUNner CDS API served by the gateway operator through the port-forwarded tunnel opened by `kubectl` just using `curl`. The below will load the config of the `udp-gateway` in the `stunner` namespace: ``` console curl -s http://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway @@ -113,6 +108,52 @@ If you happen to have a WebSocket client like the wonderful [`websocat`](https:/ websocat ws://127.0.0.1:63478/api/v1/configs/stunner/udp-gateway?watch=true - ``` +### Status + +The `status` sub-command reports the status of the dataplane pods for a gateway, especially the runtime state of the `stunnerd` daemon. + +- Find all dataplane pods for the `udp-gateway` in the `stunner` namespace and dump a status summary: + + ``` console + stunnerctl -n stunner status udp-gateway + stunner/udp-gateway-856c9f4dc9-524hc: + stunner/udp-gateway:{logLevel="all:INFO",health-check="http://:8086"} + static-auth:{realm="stunner.l7mp.io",username="",password=""} + listeners:1/clusters:1 + allocs:3/status=READY + stunner/udp-gateway-856c9f4dc9-c7wcq: + stunner/udp-gateway:{logLevel="all:INFO",health-check="http://:8086"} + static-auth:{realm="stunner.l7mp.io",username="",password=""} + listeners:1/clusters:1 + allocs:2/status=READY + ``` + +- Same but report only the runtime status of the `stunnerd` pods in the `stunner` namespace: + + ``` console + stunnerctl -n stunner status -o jsonpath='{.status}' + READY + TERMINATING + ``` + +### Authentication + +The `auth` sub-command can be used to obtain a TURN credential or a full ICE server config for connecting to a specific gateway. The authentication service API is usually served by a separate [STUNner authentication server](https://github.com/l7mp/stunner-auth-service) deployed alongside the gateway operator. The main use of this command is to feed an ICE agent manually with the ICE server config to connect to a specific STUNner gateway. + +- Obtain a full ICE server config for `udp-gateway` deployed into the `stunner` namespace: + + ``` console + stunnerctl -n stunner auth udp-gateway + {"iceServers":[{"credential":"pass-1","urls":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"}],"iceTransportPolicy":"all"} + ``` + +- Request a plain [TURN credential](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00) using the authentication service deployed into the `stunner-system-prod` namespace: + + ``` console + stunnerctl -n stunner auth udp-gateway --auth-turn-credential --auth-service-namespace=stunner-system-prod + {"password":"pass-1","ttl":86400,"uris":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"} + ``` + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). From cb58de8317dfcc95eead3623bd419da1b69dc585 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 26 Feb 2024 18:07:36 +0100 Subject: [PATCH 148/243] chore: Remove doc disclaimer before release --- README.md | 2 -- docs/README.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/README.md b/README.md index 48e33cfc..20c3d8b2 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,6 @@

-*This is the documentation for the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 5adbc162..03638016 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,6 @@ STUNner

-*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io).* - ## Overview * [Why STUNner](WHY.md) From 0680f3c76ed13857b133057628ed690b09899e39 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 26 Feb 2024 18:48:11 +0100 Subject: [PATCH 149/243] chore: Restore doc disclaimer --- README.md | 2 ++ docs/README.md | 2 ++ docs/RELEASE.md | 4 +--- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 20c3d8b2..48e33cfc 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*This is the documentation for the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 03638016..5adbc162 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,8 @@ STUNner

+*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io).* + ## Overview * [Why STUNner](WHY.md) diff --git a/docs/RELEASE.md b/docs/RELEASE.md index ce2baaa7..7321a879 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -30,10 +30,8 @@ Let the new version be vX.Y.Z. - `git pull` - bump `github.com/l7mp/stunner` version to vX.Y.Z in the `go.mod` -- bump `github.com/l7mp/stunner-gateway-operator` version to vX.Y.Z (or whatever you used when - releasing the operator) in the `go.mod` - `go mod tidy` -- make sure `make test` passes +- make sure `go test ./... -count 1` passes - make sure `golangci-lint run` passes - `git push` if there are local changes - release vX.Y.Z (try to have the same version as the main stunner repo, if possible) on github (so From aecff26b9af6bee3a93d1cd540a6cf26d75a9969 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 26 Feb 2024 19:10:49 +0100 Subject: [PATCH 150/243] chore: Update the release engineering docs The new step-by-step guide moves the operator release phase as the last step, so that the helm build CI step will find the required artifacts at docker hub. --- docs/RELEASE.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 7321a879..3496b5db 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -15,23 +15,26 @@ Let the new version be vX.Y.Z. - release vX.Y.Z on github (so that we can publish the release notes) - wait until the CI/CD pipeline goes green (may take a while) -## STUNner gateway operator +## STUNner auth service - `git pull` -- bump `github.com/l7mp/stunner` version to vX.Y.Z in the `go.mod` +- bump `github.com/l7mp/stunner` version to vX.Y.Z in the `go.mod` - `go mod tidy` -- make sure `make test` passes +- make sure `go test ./... -count 1` passes - make sure `golangci-lint run` passes - `git push` if there are local changes - release vX.Y.Z (try to have the same version as the main stunner repo, if possible) on github (so that we can publish the release notes) -## STUNner auth service +## STUNner gateway operator +- wait until both stunner and the auth-service releases finished building and the CI status goes + green on both repos (the gateway-operator helm chart CI build step needs the new artifact from + both, so it will fail until the other two finish the image push phase) - `git pull` -- bump `github.com/l7mp/stunner` version to vX.Y.Z in the `go.mod` +- bump `github.com/l7mp/stunner` version to vX.Y.Z in the `go.mod` - `go mod tidy` -- make sure `go test ./... -count 1` passes +- make sure `make test` passes - make sure `golangci-lint run` passes - `git push` if there are local changes - release vX.Y.Z (try to have the same version as the main stunner repo, if possible) on github (so From b962318fb66392373e683770f09d751fd84c603e Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 26 Feb 2024 19:22:24 +0100 Subject: [PATCH 151/243] fix(CI/CD): Fix azure/setup-helm version There is no `v4` tag yet, only `v4.0.0`. --- .github/workflows/e2e-test-dev.yaml | 4 ++-- .github/workflows/e2e-test-stable.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/e2e-test-dev.yaml b/.github/workflows/e2e-test-dev.yaml index 6680a201..d9422abd 100644 --- a/.github/workflows/e2e-test-dev.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -36,9 +36,9 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v4 + uses: azure/setup-helm@v4.0.0 with: - version: v3.12.3 + version: v3.14.2 - name: Install STUNner run: | diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index 9badbd0b..81ea2854 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -35,9 +35,9 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v4 + uses: azure/setup-helm@v4.0.0 with: - version: v3.12.3 + version: v3.14.2 - name: Install STUNner run: | From 1d3b1b899646ac94045ace797e5dea179cbfae9a Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 28 Feb 2024 23:38:55 +0100 Subject: [PATCH 152/243] chore(CI/CD): Use azure/setup-helm@v4 --- .github/workflows/e2e-test-dev.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/e2e-test-dev.yaml b/.github/workflows/e2e-test-dev.yaml index d9422abd..37190af4 100644 --- a/.github/workflows/e2e-test-dev.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -36,7 +36,7 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v4.0.0 + uses: azure/setup-helm@v4 with: version: v3.14.2 diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index 81ea2854..9b0fe13f 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -35,7 +35,7 @@ jobs: run: minikube tunnel &>mktunnel.log & - name: Set up Helm - uses: azure/setup-helm@v4.0.0 + uses: azure/setup-helm@v4 with: version: v3.14.2 From e64841b0c1740b22b12136a1aa85245cd2faa100 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 4 Mar 2024 20:48:37 +0100 Subject: [PATCH 153/243] chore(CI/CD): Bump go version to v1.22 --- .github/workflows/e2e-test-dev.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- .github/workflows/lint.yml | 2 +- .github/workflows/publish--add-binaries.yml | 2 +- .github/workflows/test.yml | 4 ++-- Dockerfile | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/e2e-test-dev.yaml b/.github/workflows/e2e-test-dev.yaml index 37190af4..caf82e40 100644 --- a/.github/workflows/e2e-test-dev.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: '1.21' + go-version: '1.22' - name: Download modules run: go mod download diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index 9b0fe13f..b2c8a8db 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: '1.21' + go-version: '1.22' - name: Download modules run: go mod download diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index b34e4bdd..4bf38dc7 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -20,7 +20,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: '1.21' + go-version: '1.22' - name: Checkout code uses: actions/checkout@v4 - name: Run linters diff --git a/.github/workflows/publish--add-binaries.yml b/.github/workflows/publish--add-binaries.yml index f9416b5c..b2753d91 100644 --- a/.github/workflows/publish--add-binaries.yml +++ b/.github/workflows/publish--add-binaries.yml @@ -42,7 +42,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: '1.21' + go-version: '1.22' - name: Build turncat binary run: | diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 3cc8c3f4..95a3174b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -23,7 +23,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: '1.21' + go-version: '1.22' - name: Download modules run: go mod download - name: Go install @@ -38,7 +38,7 @@ jobs: if: success() uses: actions/setup-go@v5 with: - go-version: '1.21' + go-version: '1.22' - name: Checkout code uses: actions/checkout@v4 - name: Calc coverage diff --git a/Dockerfile b/Dockerfile index b524bac0..0a6297a4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ ########### # BUILD -FROM golang:1.21-alpine as builder +FROM golang:1.22-alpine as builder WORKDIR /app From 82b0bb319e9c9123739ecb869676ffb0648f88b7 Mon Sep 17 00:00:00 2001 From: Teo Stocco Date: Mon, 25 Mar 2024 19:59:17 +0100 Subject: [PATCH 154/243] docs: Fix userName typo (#134) --- docs/GATEWAY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 538bb915..a3daf774 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -67,7 +67,7 @@ Below is a reference of the most important fields of the GatewayConfig [`spec`]( | `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | | `authRef` | `reference` | Reference to a Secret (`namespace` and `name`) that defines the STUN/TURN authentication mechanism and the credentials. | No | | `authType` | `string` | Type of the STUN/TURN authentication mechanism. Valid only if `authRef` is not set. Default: `static`. | No | -| `username` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | +| `userName` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | | `password` | `string` | The password for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | | `sharedSecret` | `string` | The shared secret for [`ephemeral` authentication](AUTH.md). Valid only if `authRef` is not set. | No | | `authLifetime` | `int` | The lifetime of [`ephemeral` authentication](AUTH.md) credentials in seconds. Not used by STUNner.| No | From a71f1ebff9942374219464b1ab116b97b231970e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 25 Mar 2024 21:52:15 +0100 Subject: [PATCH 155/243] chore: Create issue template --- .github/ISSUE_TEMPLATE/bug_report.md | 36 ++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..540f380e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,36 @@ +--- +name: Bug report +about: Create a report to help us improve STUNner +title: '' +labels: '' +assignees: '' + +--- + +### Description + +[Description of the problem] + +### Steps to Reproduce + +[Brief description of the steps you took to encounter the problem, if applicable] + +**Expected behavior:** [What you expected to happen] + +**Actual behavior:** [What actually happened] + +### Versions + +[Which version of STUNner you are using] + +### Info + +[Please copy-paste the output of the below commands and make sure to remove all sensitive information, like usernames, passwords, IP addresses, etc.] + +#### Gateway API status + +[Output of `kubectl get gateways,gatewayconfigs,gatewayclasses,udproutes.stunner.l7mp.io --all-namespaces -o yaml`] + +#### Operator logs + +[Output of `kubectl -n stunner-system logs $(kubectl get pods -l control-plane=stunner-gateway-operator-controller-manager --all-namespaces -o jsonpath='{.items[0].metadata.name}')`] From 75e1bffa731c07d57790d55409eb5e2b433f62cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Korn=C3=A9l=20D=C3=A1vid?= <47784437+davidkornel@users.noreply.github.com> Date: Tue, 16 Apr 2024 21:53:58 +0200 Subject: [PATCH 156/243] Update AUTH.md fixed stunner auth URL, parameter separator was accidentally a ? instead of & --- docs/AUTH.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/AUTH.md b/docs/AUTH.md index bf0dde84..360a719a 100644 --- a/docs/AUTH.md +++ b/docs/AUTH.md @@ -41,7 +41,7 @@ The authentication workflow of STUNner is as follows. Use the below query to generate a valid STUNner credential to access the Gateway called `my-gateway` deployed into the `my-namespace` namespace: ```console - curl "http://stunner-auth.stunner-system:8088/ice?service=turn?ttl=3600&username=my-user&namespace=my-namespace&gateway=my-gateway" + curl "http://stunner-auth.stunner-system:8088/ice?service=turn&ttl=3600&username=my-user&namespace=my-namespace&gateway=my-gateway" ``` 2. The clients *receive the ICE configuration* (usually, from the application server) over a secure channel. This is outside the context of STUNner. Our advice is to return the ICE configuration during the session setup process, say, along with the initial configuration returned for clients before starting the call. From ef35bfa54ce501c4e36802a775e690337d582ea5 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 24 Apr 2024 12:59:55 +0200 Subject: [PATCH 157/243] doc: Use an explicit UDPRoute backend Service in the direct-1to1 demo --- docs/examples/direct-one2one-call/README.md | 33 +++++++++---------- .../direct-one2one-call-stunner.yaml | 21 +++--------- handlers.go | 2 +- 3 files changed, 21 insertions(+), 35 deletions(-) diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index 491d5196..0358bc58 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -215,8 +215,8 @@ another one at TCP:3478, plus and a UDPRoute. kubectl apply -f docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml ``` -The most important component in the STUNner configuration is the TURN Gateway: this will expose a -public TURN server on the UDP port 3478 through which clients will connect to each other. +The most important component in the STUNner configuration is the Gateway: this will expose a public +TURN server on the UDP port 3478 through which clients will connect to each other. ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -234,14 +234,15 @@ spec: For later convenience we also create a TCP Gateway that runs on port TCP:3478. -Observe that we set STUNner's own service as the backend in the UDPRoute. This way, STUNner will be -allowed to loop back client connections to itself. This is required by the headless model, since in -this case the only way for clients to connect is via the transport relay connections created by -STUNner. The rest, that is, cross-connecting the clients' media streams between each other, is just -pure TURN magic. +We still need to ensure that STUNner will loop back client connections to itself. This is required +by the headless model, since in this case the only way for clients to connect to each other is via +the transport relay connections created by STUNner. The rest, that is, cross-connecting the +clients' media streams, is just pure TURN magic. Here is the corresponding UDPRoute. Note that the route attaches itself to both the UDP and the TCP -Gateway, so no matter on which gateway the client connects via our UDPRoute will apply. +Gateway, so no matter on which gateway the client connects via our UDPRoute will apply. Observe +also that the backend Services are the Gateway pods themselves: this makes sure that two clients +that happen to connect via different Gateway pods still connect to each other. ```yaml apiVersion: stunner.l7mp.io/v1 @@ -255,16 +256,14 @@ spec: - name: tcp-gateway rules: - backendRefs: - - name: stunner + - name: udp-gateway + namespace: stunner + - name: tcp-gateway namespace: stunner ``` -Note that the `stunner/stunner` service should exist for this to work. The manifest conveniently creates it, but -if you're doing things manually here is how to create the target service. - -```console -kubectl expose deployment -n stunner stunner --port 3478 --protocol UDP -``` +Noe that the `stunner/udp-gateway` and `stunner/tcp-gateway` Kubernetes Services are automatically +created by STUNner to expose your Gateways, here we merely reuse these as backends. ### Check your configuration @@ -294,8 +293,8 @@ Listeners: - Name: stunner/udp-gateway/udp-listener Protocol: TURN-UDP Public address:port: 34.118.88.91:3478 - Routes: [stunner/iperf-server] - Endpoints: [10.76.1.4, 10.80.4.47] + Routes: [stunner/stunner-headless] + Endpoints: ... ``` ### Run the test diff --git a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml index d6c936af..b741ff69 100644 --- a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml +++ b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml @@ -61,21 +61,8 @@ spec: - name: tcp-gateway rules: - backendRefs: - - name: stunner + - name: udp-gateway namespace: stunner - ---- -# make sure the stunner service exists, otherwise the above route will be useless -apiVersion: v1 -kind: Service -metadata: - name: stunner - namespace: stunner -spec: - selector: - app: stunner - ports: - - port: 3478 - protocol: UDP - targetPort: 3478 - type: ClusterIP + - name: tcp-gateway + namespace: stunner + diff --git a/handlers.go b/handlers.go index 7daa3023..60814cd8 100644 --- a/handlers.go +++ b/handlers.go @@ -67,7 +67,7 @@ func (s *Stunner) NewPermissionHandler(l *object.Listener) a12n.PermissionHandle return func(src net.Addr, peer net.IP) bool { // need auth for logging - // dynamic: authHandler might have changed behind ur back + // dynamic: authHandler might have changed behind our back auth := s.GetAuth() peerIP := peer.String() From 38b9c67f45cd07dc13edd87f95693c2bd1854147 Mon Sep 17 00:00:00 2001 From: Pierre Cavan Date: Thu, 25 Apr 2024 09:17:03 +0200 Subject: [PATCH 158/243] Add allocations active metric (#140) --- internal/telemetry/metrics.go | 49 +++++++++++++++++++---------------- stunner.go | 9 ++++--- 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/internal/telemetry/metrics.go b/internal/telemetry/metrics.go index 88236fd6..80fca0d4 100644 --- a/internal/telemetry/metrics.go +++ b/internal/telemetry/metrics.go @@ -1,6 +1,7 @@ package telemetry import ( + "github.com/pion/logging" "github.com/prometheus/client_golang/prometheus" ) @@ -11,6 +12,7 @@ const ( var ( ConnLabels = []string{"name"} CounterLabels = []string{"name", "direction"} + AllocActiveGauge prometheus.GaugeFunc ListenerPacketsTotal *prometheus.CounterVec ListenerBytesTotal *prometheus.CounterVec ListenerConnsTotal *prometheus.CounterVec @@ -134,30 +136,31 @@ func SubConnection(n string, c ConnType) { } } -// func RegisterMetrics(log logging.LeveledLogger, GetAllocationCount func() float64) { -// AllocActiveGauge = prometheus.NewGaugeFunc( -// prometheus.GaugeOpts{ -// Name: "stunner_allocations_active", -// Help: "Number of active allocations.", -// }, -// GetAllocationCount, -// ) -// if err := prometheus.Register(AllocActiveGauge); err == nil { -// log.Debug("GaugeFunc 'stunner_allocations_active' registered.") -// } else { -// log.Warn("GaugeFunc 'stunner_allocations_active' cannot be registered.") -// } -// } +func RegisterAllocationMetric(log logging.LeveledLogger, GetAllocationCount func() float64) { + AllocActiveGauge = prometheus.NewGaugeFunc( + prometheus.GaugeOpts{ + Namespace: stunnerNamespace, + Name: "allocations_active", + Help: "Number of active allocations.", + }, + GetAllocationCount, + ) + if err := prometheus.Register(AllocActiveGauge); err == nil { + log.Debug("GaugeFunc 'stunner_allocations_active' registered.") + } else { + log.Warn("GaugeFunc 'stunner_allocations_active' cannot be registered.") + } +} -// func UnregisterMetrics(log logging.LeveledLogger) { -// if AllocActiveGauge != nil { -// if success := prometheus.Unregister(AllocActiveGauge); success { -// log.Debug("GaugeFunc 'stunner_allocations_active' unregistered.") -// return -// } -// } -// log.Warn("GaugeFunc 'stunner_allocations_active' cannot be unregistered.") -// } +func UnregisterAllocationMetric(log logging.LeveledLogger) { + if AllocActiveGauge != nil { + if success := prometheus.Unregister(AllocActiveGauge); success { + log.Debug("GaugeFunc 'stunner_allocations_active' unregistered.") + return + } + } + log.Warn("GaugeFunc 'stunner_allocations_active' cannot be unregistered.") +} // func GetListenerPacketsTotal(ch chan prometheus.Metric) { // go func() { diff --git a/stunner.go b/stunner.go index 054d499b..0cd23bf7 100644 --- a/stunner.go +++ b/stunner.go @@ -104,7 +104,7 @@ func NewStunner(options Options) *Stunner { if !s.dryRun { s.resolver.Start() telemetry.Init() - // telemetry.RegisterMetrics(s.log, func() float64 { return s.GetActiveConnections() }) + telemetry.RegisterAllocationMetric(s.log, s.GetActiveConnections) } // TODO: remove this when STUNner gains self-managed dataplanes @@ -275,7 +275,7 @@ func (s *Stunner) Close() { } } - // telemetry.UnregisterMetrics(s.log) + telemetry.UnregisterAllocationMetric(s.log) if !s.dryRun { telemetry.Close() } @@ -284,4 +284,7 @@ func (s *Stunner) Close() { } // GetActiveConnections returns the number of active downstream (listener-side) TURN allocations. -func (s *Stunner) GetActiveConnections() float64 { return 0.0 } +func (s *Stunner) GetActiveConnections() float64 { + count := s.AllocationCount() + return float64(count) +} From 868704f7317c4219b4c363cc8581ffa78ad18c83 Mon Sep 17 00:00:00 2001 From: Richard Varadi Date: Mon, 29 Apr 2024 20:01:44 +0200 Subject: [PATCH 159/243] chore: Doc update --- docs/cmd/stunnerctl.md | 8 ++++---- docs/examples/jitsi/README.md | 5 +++-- docs/examples/livekit/livekit-server.yaml | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/cmd/stunnerctl.md b/docs/cmd/stunnerctl.md index 7b63962c..0a9e6761 100644 --- a/docs/cmd/stunnerctl.md +++ b/docs/cmd/stunnerctl.md @@ -1,11 +1,11 @@ # stunnerctl: Command line toolbox for STUNner -A CLI tool to simplify the interaction with STUNner. +A CLI tool to simplify the interaction with STUNner. The prominent use of `stunnerctl` is to load or watch STUNner dataplane configurations from a Kubernetes cluster for debugging and troubleshooting, or just for checking whether everything is configured the way it should be. ## Installation -Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. +Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. ```console go install github.com/l7mp/stunner/cmd/stunnerctl@latest @@ -26,7 +26,7 @@ go build -o stunnerctl cmd/stunnerctl/main.go ## Usage -Type `stunnerctl` to get a glimpse of the sub-commands and features provided. +Type `stunnerctl` to get a glimpse of the sub-commands and features provided. ### Config @@ -70,7 +70,7 @@ The `config` sub-command is used to load or watch running dataplane configs from ``` For those who don't have the Go toolchain available to run `go install`, STUNner provides a minimalistic `stunnerctl` replacement called `stunnerctl.sh`. -This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. +This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. The below will dump the running config of `tcp-gateway` deployed into the `stunner` namespace: diff --git a/docs/examples/jitsi/README.md b/docs/examples/jitsi/README.md index a295629a..380aaaf4 100644 --- a/docs/examples/jitsi/README.md +++ b/docs/examples/jitsi/README.md @@ -119,7 +119,7 @@ metadata: name: stunner-gatewayconfig namespace: stunner spec: - authType: longterm + authType: ephemeral sharedSecret: "my-shared-secret" --- apiVersion: gateway.networking.k8s.io/v1 @@ -132,7 +132,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: stunner.l7mp.io/v1 kind: UDPRoute @@ -145,6 +145,7 @@ spec: rules: - backendRefs: - name: jitsi-jvb + namespace: default ``` Once the Gateway resource is installed into Kubernetes, STUNner will create a Kubernetes LoadBalancer for the Gateway to expose the TURN server on UDP:3478 to clients. It can take up to a minute for Kubernetes to allocate a public external IP for the service. diff --git a/docs/examples/livekit/livekit-server.yaml b/docs/examples/livekit/livekit-server.yaml index cbbd9ba6..04f6ea02 100644 --- a/docs/examples/livekit/livekit-server.yaml +++ b/docs/examples/livekit/livekit-server.yaml @@ -10,7 +10,7 @@ spec: serviceName: redis replicas: 1 selector: - matchLabels: + matchLabels: app: redis template: metadata: From 374ecc5dad1b1fb1391a110089fdefbee34e00b9 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 10 May 2024 18:21:57 +0200 Subject: [PATCH 160/243] chore: Add test not to break legacy mode again --- pkg/config/client/client_test.go | 9 ++++++++- pkg/config/client/util.go | 2 +- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/pkg/config/client/client_test.go b/pkg/config/client/client_test.go index 16fb31bf..ec1f0f27 100644 --- a/pkg/config/client/client_test.go +++ b/pkg/config/client/client_test.go @@ -8,7 +8,14 @@ import ( func TestParseURI(t *testing.T) { // file - u, err := getURI("/tmp/a") + u, err := getURI("file:///tmp/a") + assert.NoError(t, err, "file URI parse") + assert.Equal(t, "file", u.Scheme, "file URI scheme") + assert.Equal(t, "", u.Host, "file URI host") + assert.Equal(t, "/tmp/a", u.Path, "file URI path") + + // default is http + u, err = getURI("/tmp/a") assert.NoError(t, err, "file URI parse") assert.Equal(t, "http", u.Scheme, "file URI scheme") assert.Equal(t, "", u.Host, "file URI host") diff --git a/pkg/config/client/util.go b/pkg/config/client/util.go index c0c7f4fd..7766e151 100644 --- a/pkg/config/client/util.go +++ b/pkg/config/client/util.go @@ -29,7 +29,7 @@ func decodeConfigList(r []byte) ([]*stnrv1.StunnerConfig, error) { // getURI tries to parse an address or an URL or a file name into an URL. func getURI(addr string) (*url.URL, error) { - // make sure we have a working HTTP scheme + // default URL scheme is "http" if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") && !strings.HasPrefix(addr, "ws://") && !strings.HasPrefix(addr, "wss://") && !strings.HasPrefix(addr, "file://") { From e1f30a618e2275ad13d268116675a8403fd5e1d4 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 10 May 2024 19:01:19 +0200 Subject: [PATCH 161/243] chore: Report git version/tag/hash in the startup logs --- Makefile | 28 +++++++++++++++++++--------- cmd/stunnerd/main.go | 10 ++++++++-- pkg/buildinfo/build_info.go | 13 +++++++++++++ 3 files changed, 40 insertions(+), 11 deletions(-) create mode 100644 pkg/buildinfo/build_info.go diff --git a/Makefile b/Makefile index c0598e7b..ebb17857 100644 --- a/Makefile +++ b/Makefile @@ -1,3 +1,17 @@ +# Build variables +PACKAGE = github.com/l7mp/stunner +BUILD_DIR ?= bin/ +VERSION ?= $(shell (git describe --tags --abbrev=8 --always --long) | tr "/" "-") +COMMIT_HASH ?= $(shell git rev-parse --short HEAD 2>/dev/null) +BUILD_DATE ?= $(shell date +%FT%T%z) +LDFLAGS += -X main.version=${VERSION} -X main.commitHash=${COMMIT_HASH} -X main.buildDate=${BUILD_DATE} + +ifeq (${VERBOSE}, 1) +ifeq ($(filter -v,${GOARGS}),) + GOARGS += -v +endif +endif + .PHONY: all all: build @@ -14,20 +28,16 @@ vet: ## Run go vet against code. go vet ./... .PHONY: test -test: manifests generate fmt vet test ## Run tests. +test: generate fmt vet + go test ./... -v ##@ Build .PHONY: build -build: generate fmt vet ## Build binary. - go build -o bin/stunnerd cmd/stunnerd/main.go - go build -o bin/turncat cmd/turncat/main.go - -.PHONY: run -run: manifests generate fmt vet ## Run a controller from your host. - go run cmd/stunnerd/main.go +build: generate fmt vet + go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerd cmd/stunnerd/main.go + go build ${GOARGS} -o ${BUILD_DIR}/turncat cmd/turncat/main.go -# clean up generated files .PHONY: clean clean: echo 'Use "make generate` to autogenerate server code' > pkg/server/server.go diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 74f2a7bf..ccab9821 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -13,10 +13,15 @@ import ( "github.com/l7mp/stunner" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/buildinfo" cdsclient "github.com/l7mp/stunner/pkg/config/client" ) -// usage: stunnerd -v turn://user1:passwd1@127.0.0.1:3478?transport=udp +var ( + version = "dev" + commitHash = "n/a" + buildDate = "" +) func main() { os.Args[0] = "stunnerd" @@ -74,7 +79,8 @@ func main() { log := st.GetLogger().NewLogger("stunnerd") - log.Infof("starting stunnerd instance %q", st.GetId()) + buildInfo := buildinfo.BuildInfo{Version: version, CommitHash: commitHash, BuildDate: buildDate} + log.Infof("starting stunnerd id %q, STUNner %s ", st.GetId(), buildInfo.String()) conf := make(chan *stnrv1.StunnerConfig, 1) defer close(conf) diff --git a/pkg/buildinfo/build_info.go b/pkg/buildinfo/build_info.go new file mode 100644 index 00000000..4665b91f --- /dev/null +++ b/pkg/buildinfo/build_info.go @@ -0,0 +1,13 @@ +package buildinfo + +import "fmt" + +type BuildInfo struct { + Version string + CommitHash string + BuildDate string +} + +func (i BuildInfo) String() string { + return fmt.Sprintf("version %s (%s) built on %s", i.Version, i.CommitHash, i.BuildDate) +} From 4df714c3c3308f6e9485832d1353f4766a272fbe Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 10 May 2024 19:23:47 +0200 Subject: [PATCH 162/243] chore: Report git version/tag/hash in the turncat startup logs too --- Makefile | 3 ++- cmd/turncat/main.go | 8 ++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index ebb17857..e91215bc 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,8 @@ test: generate fmt vet .PHONY: build build: generate fmt vet go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerd cmd/stunnerd/main.go - go build ${GOARGS} -o ${BUILD_DIR}/turncat cmd/turncat/main.go + go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/turncat cmd/turncat/main.go + go build ${GOARGS} -o ${BUILD_DIR}/stunnerctl cmd/stunnerctl/main.go .PHONY: clean clean: diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 61c82756..d1beeb55 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -17,6 +17,7 @@ import ( "github.com/l7mp/stunner" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/buildinfo" cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) @@ -34,6 +35,10 @@ var ( log logging.LeveledLogger defaultDuration time.Duration loggerFactory *logger.LeveledLoggerFactory + + version = "dev" + commitHash = "n/a" + buildDate = "" ) func main() { @@ -77,6 +82,9 @@ func main() { loggerFactory = logger.NewLoggerFactory(*level) log = loggerFactory.NewLogger("turncat-cli") + buildInfo := buildinfo.BuildInfo{Version: version, CommitHash: commitHash, BuildDate: buildDate} + log.Debugf("Starting turncat %s", buildInfo.String()) + uri := flag.Arg(1) log.Debugf("Reading STUNner config from URI %q", uri) config, err := getStunnerConf(uri) From 009ee7a1f1429df953e444517a73ee05826dbe7e Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Sat, 11 May 2024 11:34:33 +0200 Subject: [PATCH 163/243] chore(CI/CD): Use the new build process --- .github/workflows/publish--add-binaries.yml | 14 +++++--------- Dockerfile | 14 +++++++++----- Makefile | 1 + 3 files changed, 15 insertions(+), 14 deletions(-) diff --git a/.github/workflows/publish--add-binaries.yml b/.github/workflows/publish--add-binaries.yml index b2753d91..39882a2f 100644 --- a/.github/workflows/publish--add-binaries.yml +++ b/.github/workflows/publish--add-binaries.yml @@ -44,10 +44,12 @@ jobs: with: go-version: '1.22' - - name: Build turncat binary + - name: Build binaries run: | - CGO_ENABLED=0 GOARCH=${{ matrix.arch }} GOOS=${{ matrix.os }} go build -ldflags="-w -s" -trimpath -o turncat cmd/turncat/main.go - mv turncat turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + export CGO_ENABLED=0 GOOS=${{ matrix.os }} GOARCH=${{ matrix.arch }} + make build + mv bin/turncat turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} + mv bin/stunnerctl stunnerctl-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} - name: Release turncat binary uses: svenstaro/upload-release-action@v2 @@ -57,12 +59,6 @@ jobs: tag: ${{ github.ref_name }} asset_name: turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} - - - name: Build stunnerctl binary - run: | - CGO_ENABLED=0 GOARCH=${{ matrix.arch }} GOOS=${{ matrix.os }} go build -ldflags="-w -s" -trimpath -o stunnerctl cmd/stunnerctl/main.go - mv stunnerctl stunnerctl-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} - - name: Release stunnerctl binary uses: svenstaro/upload-release-action@v2 with: diff --git a/Dockerfile b/Dockerfile index 0a6297a4..56c8c2fb 100644 --- a/Dockerfile +++ b/Dockerfile @@ -6,21 +6,25 @@ WORKDIR /app COPY go.mod ./ COPY go.sum ./ -RUN go mod download COPY *.go ./ COPY internal/ internal/ COPY pkg/ pkg/ -COPY cmd/stunnerd/main.go cmd/stunnerd/ -COPY cmd/stunnerd/stunnerd.conf cmd/stunnerd/ +COPY cmd/ cmd/ + +COPY .git ./ +COPY Makefile ./ +RUN apk add --no-cache git make RUN apkArch="$(apk --print-arch)"; \ case "$apkArch" in \ aarch64) export GOARCH='arm64' ;; \ *) export GOARCH='amd64' ;; \ esac; \ - CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o stunnerd cmd/stunnerd/main.go + export CGO_ENABLED=0; \ + export GOOS=linux; \ + make build ########### # STUNNERD @@ -28,7 +32,7 @@ FROM scratch WORKDIR /app -COPY --from=builder /app/stunnerd /usr/bin/ +COPY --from=builder /app/bin/stunnerd /usr/bin/ COPY --from=builder /app/cmd/stunnerd/stunnerd.conf / EXPOSE 3478/udp diff --git a/Makefile b/Makefile index e91215bc..8e7973fd 100644 --- a/Makefile +++ b/Makefile @@ -5,6 +5,7 @@ VERSION ?= $(shell (git describe --tags --abbrev=8 --always --long) | tr "/" "-" COMMIT_HASH ?= $(shell git rev-parse --short HEAD 2>/dev/null) BUILD_DATE ?= $(shell date +%FT%T%z) LDFLAGS += -X main.version=${VERSION} -X main.commitHash=${COMMIT_HASH} -X main.buildDate=${BUILD_DATE} +GOARGS = -trimpath ifeq (${VERBOSE}, 1) ifeq ($(filter -v,${GOARGS}),) From 4e4d6a3213a45231a9261fa4876b74e9035c4505 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Sat, 11 May 2024 12:21:51 +0200 Subject: [PATCH 164/243] chore(CI/CD): Strip debug info from built binaries --- .github/workflows/publish--add-binaries.yml | 2 +- Dockerfile | 2 +- Makefile | 6 +++++- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/.github/workflows/publish--add-binaries.yml b/.github/workflows/publish--add-binaries.yml index 39882a2f..14c71c09 100644 --- a/.github/workflows/publish--add-binaries.yml +++ b/.github/workflows/publish--add-binaries.yml @@ -47,7 +47,7 @@ jobs: - name: Build binaries run: | export CGO_ENABLED=0 GOOS=${{ matrix.os }} GOARCH=${{ matrix.arch }} - make build + make build-bin mv bin/turncat turncat-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} mv bin/stunnerctl stunnerctl-v${{ steps.vars.outputs.tag }}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.file_end }} diff --git a/Dockerfile b/Dockerfile index 56c8c2fb..c2b2d18c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -24,7 +24,7 @@ RUN apkArch="$(apk --print-arch)"; \ esac; \ export CGO_ENABLED=0; \ export GOOS=linux; \ - make build + make build-bin ########### # STUNNERD diff --git a/Makefile b/Makefile index 8e7973fd..7b2389fe 100644 --- a/Makefile +++ b/Makefile @@ -4,6 +4,7 @@ BUILD_DIR ?= bin/ VERSION ?= $(shell (git describe --tags --abbrev=8 --always --long) | tr "/" "-") COMMIT_HASH ?= $(shell git rev-parse --short HEAD 2>/dev/null) BUILD_DATE ?= $(shell date +%FT%T%z) +LDFLAGS += -s -w LDFLAGS += -X main.version=${VERSION} -X main.commitHash=${COMMIT_HASH} -X main.buildDate=${BUILD_DATE} GOARGS = -trimpath @@ -35,7 +36,10 @@ test: generate fmt vet ##@ Build .PHONY: build -build: generate fmt vet +build: generate fmt vet build-bin + +.PHONY: build-bin +build-bin: go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerd cmd/stunnerd/main.go go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/turncat cmd/turncat/main.go go build ${GOARGS} -o ${BUILD_DIR}/stunnerctl cmd/stunnerctl/main.go From 0a5c095c96087bfb27cd978284a21e9ff19c0a7f Mon Sep 17 00:00:00 2001 From: Pierre Cavan Date: Mon, 13 May 2024 16:52:14 +0200 Subject: [PATCH 165/243] doc: Add stunner_allocations_active in the monitoring doc (#142) --- docs/MONITORING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 884f2052..ba830169 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -30,7 +30,8 @@ STUNner provides deep visibility into the amount of traffic sent and received on | Metric | Description | Type | Labels | | :--- | :--- | :--- | :--- | -| `stunner_listener_connections` | Number of *active* downstream connections at a listener. | gauge | `name=` | +| `stunner_allocations_active` | Number of active allocations. | gauge | none | +| `stunner_listener_connections` | Number of *active* downstream connections at a listener. Stays constant when using only UDP listeners. | gauge | `name=` | | `stunner_listener_connections_total` | Number of downstream connections at a listener. | counter | `name=` | | `stunner_listener_packets_total` | Number of datagrams sent or received at a listener. Unreliable for listeners running on a connection-oriented transport protocol (TCP/TLS). | counter | `direction=`, `name=`| | `stunner_listener_bytes_total` | Number of bytes sent or received at a listener. | counter | `direction=`, `name=` | From 24a87338127edf56f0b0d05bf6b7c9a9840af1b6 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 13 May 2024 19:29:47 +0200 Subject: [PATCH 166/243] chore: Bump pion and prometheus deps --- go.mod | 22 +++++++++++----------- go.sum | 45 ++++++++++++++++++++++++++++----------------- 2 files changed, 39 insertions(+), 28 deletions(-) diff --git a/go.mod b/go.mod index 827c49e8..2c3f3563 100644 --- a/go.mod +++ b/go.mod @@ -14,16 +14,16 @@ require ( github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.1 github.com/oapi-codegen/runtime v1.1.1 - github.com/pion/dtls/v2 v2.2.10 + github.com/pion/dtls/v2 v2.2.11 github.com/pion/logging v0.2.2 - github.com/pion/transport/v3 v3.0.1 - github.com/pion/turn/v3 v3.0.1 - github.com/prometheus/client_golang v1.18.0 + github.com/pion/transport/v3 v3.0.2 + github.com/pion/turn/v3 v3.0.3 + github.com/prometheus/client_golang v1.19.1 github.com/spf13/cobra v1.8.0 github.com/spf13/pflag v1.0.5 - github.com/stretchr/testify v1.8.4 + github.com/stretchr/testify v1.9.0 go.uber.org/zap v1.26.0 - golang.org/x/sys v0.16.0 + golang.org/x/sys v0.18.0 golang.org/x/time v0.5.0 k8s.io/api v0.29.1 k8s.io/apimachinery v0.29.1 @@ -75,21 +75,21 @@ require ( github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.5.0 // indirect - github.com/prometheus/common v0.46.0 // indirect + github.com/prometheus/common v0.48.0 // indirect github.com/prometheus/procfs v0.12.0 // indirect github.com/xlab/treeprint v1.2.0 // indirect go.starlark.net v0.0.0-20240123142251-f86470692795 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.18.0 // indirect + golang.org/x/crypto v0.21.0 // indirect golang.org/x/mod v0.14.0 // indirect - golang.org/x/net v0.20.0 // indirect + golang.org/x/net v0.22.0 // indirect golang.org/x/oauth2 v0.16.0 // indirect golang.org/x/sync v0.6.0 // indirect - golang.org/x/term v0.16.0 // indirect + golang.org/x/term v0.18.0 // indirect golang.org/x/text v0.14.0 // indirect golang.org/x/tools v0.17.0 // indirect google.golang.org/appengine v1.6.8 // indirect - google.golang.org/protobuf v1.32.0 // indirect + google.golang.org/protobuf v1.33.0 // indirect gopkg.in/evanphx/json-patch.v5 v5.9.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/go.sum b/go.sum index d3bb53cb..e2de97fc 100644 --- a/go.sum +++ b/go.sum @@ -121,8 +121,8 @@ github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0V github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= -github.com/pion/dtls/v2 v2.2.10 h1:u2Axk+FyIR1VFTPurktB+1zoEPGIW3bmyj3LEFrXjAA= -github.com/pion/dtls/v2 v2.2.10/go.mod h1:d9SYc9fch0CqK90mRk1dC7AkzzpwJj6u2GU3u+9pqFE= +github.com/pion/dtls/v2 v2.2.11 h1:9U/dpCYl1ySttROPWJgqWKEylUdT0fXp/xst6JwY5Ks= +github.com/pion/dtls/v2 v2.2.11/go.mod h1:d9SYc9fch0CqK90mRk1dC7AkzzpwJj6u2GU3u+9pqFE= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= @@ -132,20 +132,21 @@ github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLcc github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= github.com/pion/transport/v2 v2.2.4 h1:41JJK6DZQYSeVLxILA2+F4ZkKb4Xd/tFJZRFZQ9QAlo= github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= -github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= -github.com/pion/turn/v3 v3.0.1 h1:wLi7BTQr6/Q20R0vt/lHbjv6y4GChFtC33nkYbasoT8= -github.com/pion/turn/v3 v3.0.1/go.mod h1:MrJDKgqryDyWy1/4NT9TWfXWGMC7UHT6pJIv1+gMeNE= +github.com/pion/transport/v3 v3.0.2 h1:r+40RJR25S9w3jbA6/5uEPTzcdn7ncyU44RWCbHkLg4= +github.com/pion/transport/v3 v3.0.2/go.mod h1:nIToODoOlb5If2jF9y2Igfx3PFYWfuXi37m0IlWa/D0= +github.com/pion/turn/v3 v3.0.3 h1:1e3GVk8gHZLPBA5LqadWYV60lmaKUaHCkm9DX9CkGcE= +github.com/pion/turn/v3 v3.0.3/go.mod h1:vw0Dz420q7VYAF3J4wJKzReLHIo2LGp4ev8nXQexYsc= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.18.0 h1:HzFfmkOzH5Q8L8G+kSJKUx5dtG87sewO+FoDDqP5Tbk= -github.com/prometheus/client_golang v1.18.0/go.mod h1:T+GXkCk5wSJyOqMIzVgvvjFDlkOQntgjkJWKrN5txjA= +github.com/prometheus/client_golang v1.19.1 h1:wZWJDwK+NameRJuPGDhlnFgx8e8HN3XHQeLaYJFJBOE= +github.com/prometheus/client_golang v1.19.1/go.mod h1:mP78NwGzrVks5S2H6ab8+ZZGJLZUq1hoULYBAYBw1Ho= github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= -github.com/prometheus/common v0.46.0 h1:doXzt5ybi1HBKpsZOL0sSkaNHJJqkyfEWZGGqqScV0Y= -github.com/prometheus/common v0.46.0/go.mod h1:Tp0qkxpb9Jsg54QMe+EAmqXkSV7Evdy1BTn+g2pa/hQ= +github.com/prometheus/common v0.48.0 h1:QO8U2CdOzSn1BBsmXJXduaaW+dY/5QLjfB8svtSzKKE= +github.com/prometheus/common v0.48.0/go.mod h1:0/KsvlIEfPQCQ5I2iNSAWKPZziNCvRs5EC6ILDTlAPc= github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= @@ -160,15 +161,17 @@ github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= -github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY= +github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= +github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU= github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg= github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= @@ -190,8 +193,10 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= +golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= +golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA= +golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -210,8 +215,10 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo= golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= +golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= +golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc= +golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= golang.org/x/oauth2 v0.16.0 h1:aDkGMBSYxElaoP81NpoUoz2oo2R2wHdZpGToUxfyQrQ= golang.org/x/oauth2 v0.16.0/go.mod h1:hqZ+0LWXsiVoZpeld6jVt06P3adbS2Uu911W1SsJv2o= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -233,16 +240,20 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4= +golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= -golang.org/x/term v0.16.0 h1:m+B6fahuftsE9qjo0VWp2FW0mB3MTJvR0BaMQrq0pmE= golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= +golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk= +golang.org/x/term v0.18.0 h1:FcHjZXDMxI8mM3nwhX9HlKop4C0YQvCVCdwYl2wOtE8= +golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= @@ -270,8 +281,8 @@ google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAs google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I= -google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= +google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI= +google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= From 1792b5e688ddf24dae9509e0093e594e30ae7cd7 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 14 May 2024 21:20:22 +0200 Subject: [PATCH 167/243] chore(ci/cd): Print gw-operator logs in e2e tests --- .github/workflows/e2e-test-dev.yaml | 2 ++ .github/workflows/e2e-test-stable.yml | 2 ++ 2 files changed, 4 insertions(+) diff --git a/.github/workflows/e2e-test-dev.yaml b/.github/workflows/e2e-test-dev.yaml index caf82e40..fd7974de 100644 --- a/.github/workflows/e2e-test-dev.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -90,6 +90,8 @@ jobs: cat turncat.log echo "* STUNNER" kubectl logs -n stunner $(kubectl get pods -n stunner -l app=stunner -o jsonpath='{.items[0].metadata.name}') + echo "* STUNNER-GATEWAY=OPERATOR" + kubectl logs -n stunner $(kubectl get pods -n stunner -l control-plane=stunner-gateway-operator-controller-manager -o jsonpath='{.items[0].metadata.name}') - name: Check iperf conectivity run: grep "Server Report" iperf.log diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index b2c8a8db..e1630eab 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -86,6 +86,8 @@ jobs: cat turncat.log echo "* STUNNER" kubectl logs -n stunner $(kubectl get pods -n stunner -l app=stunner -o jsonpath='{.items[0].metadata.name}') + echo "* STUNNER-GATEWAY=OPERATOR" + kubectl logs -n stunner $(kubectl get pods -n stunner -l control-plane=stunner-gateway-operator-controller-manager -o jsonpath='{.items[0].metadata.name}') - name: Check iperf conectivity run: grep "Server Report" iperf.log From 92167d15f83128e189d1d4a2296bbe6ebaf9b127 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 15 May 2024 12:27:09 +0200 Subject: [PATCH 168/243] doc: Document the new settings in the Dataplane CRD --- docs/GATEWAY.md | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index a3daf774..2664a912 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -312,7 +312,7 @@ The StaticService `spec.prefixes` must be a list of proper IPv4 prefixes: any IP ## Dataplane -The Dataplane resource is used as a template for provisioning dataplane (`stunnerd`) pods that actually implement TURN media ingestion. This is useful to choose the `stunnerd` image origin and version, set custom command line arguments and environment variables, configure resource requests/limits, etc. +The Dataplane resource is used as a template for provisioning [`stunnerd`](/cmd/stunnerd/README.md) dataplane pods that implement TURN media ingestion. This is useful to choose the `stunnerd` image origin and version, set custom command line arguments and environment variables, configure resource requests/limits, etc. Below is the `default` Dataplane installed by STUNner. @@ -340,22 +340,26 @@ spec: The following fields can be set in the Dataplane `spec` to customize the provisioning of `stunnerd` pods. -| Field | Type | Description | Required | -|:--------------------------------|:----------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| -| `image` | `string` | The container image. | Yes | -| `imagePullPolicy` | `string` | Policy for if/when to pull an image, can be either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if the `latest` tag is specified on the image, or `IfNotPresent` otherwise. | No | -| `command` | `list` | Entrypoints for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). . | No | -| `args` | `list` | Commane line arguments for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | -| `envFrom` | `list` | List of sources to populate environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | -| `env` | `list` | List of environment variablesfor the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | -| `replicas` | `int` | Number of dataplane pods per Gateway to provision. Not enforced if the `stunnerd` Deployment replica count is overwritten, either manually or by an autoscaler. Default: 1. | No | -| `hostNetwork` | `bool` | Deploy the dataplane into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | -| `resources` | `object` | Compute resources per dataplane pod. | No | -| `affinity` | `object` | Scheduling constraints for the dataplane pods. Default: none. | No | -| `tolerations` | `object` | Tolerations for the dataplane pods. Default: none. | No | -| `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | -| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | -| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | +| Field | Type | Description | Required | +|:--------------------------------|:----------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| +| `image` | `string` | The container image. | Yes | +| `imagePullPolicy` | `string` | Policy for if/when to pull the [`stunnerd`](/cmd/stunnerd/README.md), either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if the `latest` tag is specified on the image, `IfNotPresent` otherwise. | No | +| `command` | `list` | Entrypoints for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | +| `args` | `list` | Command line arguments for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | +| `envFrom` | `list` | List of sources to populate environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | +| `env` | `list` | List of environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | +| `containerSecurityContext` | `object` | Container-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `replicas` | `int` | Number of dataplane pods per Gateway to provision. Not enforced if the [dataplane](/cmd/stunnerd/README.md) Deployment replica count is overwritten manually or by an autoscaler. Default: 1. | No | +| `imagePullSecrets` | `list` | List of Secret references to use for pulling the `stunnerd` image. Each ref is a secret name, namespace is the same as that of the Gateway on behalf of which the dataplane is deployed. | No | +| `hostNetwork` | `bool` | Deploy the [dataplane](/cmd/stunnerd/README.md) into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | +| `resources` | `object` | Compute resources per [dataplane](/cmd/stunnerd/README.md) pod. Default: none. | No | +| `affinity` | `object` | Scheduling constraints for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `tolerations` | `object` | Tolerations for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `securityContext` | `object` | Pod-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `topologySpreadConstraints` | `object` | Description of how the [dataplane](/cmd/stunnerd/README.md) pods for a Gateway ought to spread across topology domains. Default: none. | No | +| `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | +| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | +| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the GatewayConfig to choose the Dataplane per each STUNner install. From bf5599481924088086c4f564f09a26f652b60801 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 20 May 2024 18:27:11 +0200 Subject: [PATCH 169/243] doc: Document Gateway annotations --- docs/GATEWAY.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 2664a912..5f3ba9ac 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -185,7 +185,9 @@ Manually hinted external address describes an address that can be bound to a Gat > > Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if the cloud-provider or your Kubernetes install do not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. -[Mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) are supported: this means if you want to expose a UDP and a TCP port on the same IP you can do it with a single Gateway. The below Gateway will expose both ports with their respective protocols. +Currently, STUNner limits each Gateway to a single transport protocol, e.g., UDP or TCP. This is intended to improve the consistency across the Kubernetes services of different cloud providers, which provide varying support for [mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types). If you still want to expose a UDP and a TCP port on the same IP using a single Gateway, add the annotation `stunner.l7mp.io/enable-mixed-protocol-lb: true` to the Gateway. + +The below Gateway will expose both ports with their respective protocols. ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -209,6 +211,28 @@ spec: > > Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature. You can enable mixed-protocol LBs by annotating a Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. +Some use cases require maintaining the source IP address of the client for correct operation. For the intended use case of STUNner, as an ingress media gateway exposing the cluster's media services over the TURN protocol, this does not matter. However, some users wish to use STUNner as a STUN server, which requires the original source IP to be retained by the load balancer. This can be achieved by adding the annotation `stunner.l7mp.io/external-traffic-policy: local` to a Gateway. This will set the [`service.spec.externalTrafficPolicy`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) field in the Service created by STUNner for the Gateway to `local`, which will instruct Kubernetes to preserve the original source IP address in clients' packets. + +> [!WARNING] +> +> This feature comes with fairly complex [limitations](https://kubernetes.io/docs/tutorials/services/source-ip), use it at your own [risk](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#caveats-and-limitations-when-preserving-source-ips). + +### Manually provisioning the dataplane + +In some cases it may be useful to manually provision a dataplane for a Gateway, e.g., to deploy `stunnerd` in a DeamonSet instead of a Deployment. Adding the annotation `stunner.l7mp.io/disable-managed-dataplane: true` to a Gateway will prevent STUNner from spawning a dataplane for the Gateway. This then allows one to manually create a `stunnerd` dataplane and connect it to the CDS server exposed by the operator to obtain the dataplane configuration. Remove the annotation to revert to the default mode and let STUNner to manage the dataplane for the Gateway. + +> [!WARNING] +> +> Manually provisioning a dataplane for a Gateway requires intimate knowledge with the STUNner internals, use this feature only if you know what you are doing. + +The below table summarizes the Gateway annotations supported by STUNner. + +| Key/value | Description | Default | +|:----------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------:| +| `stunner.l7mp.io/service-type: ` | [Type of the Service](https://kubernetes.io/docs/concepts/services-networking/service) per Gateway, either `ClusterIP`, `NodePort`, or `LoadBalancer`. | `LoadBalancer` | +| `stunner.l7mp.io/enable-mixed-protocol-lb: ` | [Mixed protocol load balancer service](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) support. | False | +| `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway | False | + ## UDPRoute UDPRoute resources can be attached to Gateways in order to specify the backend services permitted to be reached via the Gateway. Multiple UDPRoutes can attach to the same Gateway, and each UDPRoute can specify multiple backend services; in this case access to *all* backends in *each* of the attached UDPRoutes is allowed. An UDPRoute can be attached to a Gateway by setting the `parentRef` to the Gateway's name and namespace. This is, however, contingent on whether the Gateway accepts routes from the given namespace: customize the `allowedRoutes` per each Gateway listener to control which namespaces the listener accepts routes from. From 09bcb5423ebfd79b82d631e6d55780cd0f1cb89a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 22 May 2024 15:02:02 +0200 Subject: [PATCH 170/243] fix: Let stunnerctl use the default namespace if no namespace given, fix #139 --- cmd/stunnerctl/main.go | 8 ++++---- docs/GATEWAY.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go index b1c90b61..41d2bf23 100644 --- a/cmd/stunnerctl/main.go +++ b/cmd/stunnerctl/main.go @@ -92,7 +92,7 @@ var ( func init() { rootCmd.PersistentFlags().BoolVarP(&all, "all-namespaces", "a", false, "Consider all namespaces") - rootCmd.PersistentFlags().StringVarP(&output, "output", "o", "summary", "Output format") + rootCmd.PersistentFlags().StringVarP(&output, "output", "o", "summary", "Output format, either json, yaml, summary or jsonpath=template") rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") // Kubernetes config flags: persistent, all commands @@ -127,7 +127,7 @@ func main() { } } -func runConfig(cmd *cobra.Command, args []string) error { +func runConfig(_ *cobra.Command, args []string) error { loglevel := "all:WARN" if verbose { loglevel = "all:TRACE" @@ -136,7 +136,7 @@ func runConfig(cmd *cobra.Command, args []string) error { log = loggerFactory.NewLogger("stunnerctl") gwNs := "default" - if k8sConfigFlags.Namespace != nil { + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { gwNs = *k8sConfigFlags.Namespace } @@ -241,7 +241,7 @@ func runConfig(cmd *cobra.Command, args []string) error { return nil } -func runStatus(cmd *cobra.Command, args []string) error { +func runStatus(_ *cobra.Command, args []string) error { loglevel := "all:WARN" if verbose { loglevel = "all:TRACE" diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 5f3ba9ac..450a3f8b 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -183,7 +183,7 @@ Manually hinted external address describes an address that can be bound to a Gat > [!WARNING] > -> Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if the cloud-provider or your Kubernetes install do not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. +> Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if your Kubernetes install does not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. Currently, STUNner limits each Gateway to a single transport protocol, e.g., UDP or TCP. This is intended to improve the consistency across the Kubernetes services of different cloud providers, which provide varying support for [mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types). If you still want to expose a UDP and a TCP port on the same IP using a single Gateway, add the annotation `stunner.l7mp.io/enable-mixed-protocol-lb: true` to the Gateway. @@ -231,7 +231,7 @@ The below table summarizes the Gateway annotations supported by STUNner. |:----------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------:| | `stunner.l7mp.io/service-type: ` | [Type of the Service](https://kubernetes.io/docs/concepts/services-networking/service) per Gateway, either `ClusterIP`, `NodePort`, or `LoadBalancer`. | `LoadBalancer` | | `stunner.l7mp.io/enable-mixed-protocol-lb: ` | [Mixed protocol load balancer service](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) support. | False | -| `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway | False | +| `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway. | False | ## UDPRoute From 897a4bf52cf1f80e3034a471441b238cfd25b590 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 23 May 2024 21:34:13 +0200 Subject: [PATCH 171/243] doc: Document the remaining Gateway annotations --- docs/GATEWAY.md | 90 +++++++++++++++++++++++-------------------------- 1 file changed, 43 insertions(+), 47 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 450a3f8b..eecb9def 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -152,6 +152,17 @@ Below is a reference of the most important fields of the Gateway [`spec`](https: > - changing the transport protocol, port or TLS keys/certs of an *existing* listener will restart the TURN listener but leave the rest of the listeners intact; > - changing the TURN authentication realm will restart *all* TURN listeners. +Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. + +| Field | Type | Description | Required | +|:--------|:--------:|:--------------------------------------------------------------|:--------:| +| `type` | `string` | Type of the address. Currently only `IPAddress` is supported. | Yes | +| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | + +> [!WARNING] +> +> Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if your Kubernetes install does not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. + ### Listener configuration Each TURN `listener` is defined by a unique name, a transport protocol and a port. In addition, a `tls` configuration is required for TURN-TLS and TURN-DTLS listeners. Per-listener configuration is as follows. @@ -168,62 +179,45 @@ For TURN-TLS/TURN-DTLS listeners, `tls.mode` must be set to `Terminate` or omitt ### Load balancer configuration -STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN over UDP and TURN over DTLS listeners are exposed as UDP services, TURN-TCP and TURN-TLS listeners are exposed as TCP. +STUNner will automatically generate a Kubernetes LoadBalancer Service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN over UDP and TURN over DTLS listeners are exposed as UDP services, TURN-TCP and TURN-TLS listeners are exposed as TCP. -STUNner implements two ways to customize the automatically created Service, both involving certain per-defined [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) added to the Service. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). +STUNner implements two ways to customize the automatically created Service, both involving certain pre-defined [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations). First, you can add global annotations to the `loadBalancerServiceAnnotations` field of the [GatewayConfig spec](#gatewayconfig), which affect the Service created for each Gateway that links to the GatewayConfig (via the GatewayClass `parametersRef`). To customize annotations on a per-gateway status, you can also add specific annotations to the Gateway itself. Gateway annotations override the global annotations on conflict. -By default, each key-value pair set in the GatewayConfig `loadBalancerServiceAnnotations` field will be copied verbatim into the Service. Service annotations can be customized on a per-Gateway basis as well, by adding the corresponding annotations to a Gateway resource. STUNner copies all annotations from the Gateway into the Service, overwriting the annotations specified in the GatewayConfig on conflict. +The following rules apply: +- Each annotation is copied verbatim into the Service created for any Gateway. This can be used, for instance, to specify health-check settings on the load-balancer Service (using the `service.beta.kubernetes.io/*-loadbalancer-healthcheck-*` annotations, see above). +- Annotations with the prefix `stunner.l7mp.io/...` have special meaning: apart from being copied into the Service these annotations also affect some specifics of the created Service, like the service type or the nodeports assigned to listeners. -Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. +STUNner defines the following special annotations: -| Field | Type | Description | Required | -|:--------|:--------:|:--------------------------------------------------------------|:--------:| -| `type` | `string` | Type of the address. Currently only `IPAddress` is supported. | Yes | -| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | +1. **Service type:** The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). -> [!WARNING] -> -> Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if your Kubernetes install does not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. +1. **Mixed-protocol support:** Currently, STUNner limits each Gateway to a single transport protocol, e.g., UDP or TCP. This is intended to improve the consistency across the Kubernetes services of different cloud providers, which provide varying support for [mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types). If you still want to expose a UDP and a TCP port on the same IP using a single Gateway, add the annotation `stunner.l7mp.io/enable-mixed-protocol-lb: true` to the Gateway. Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature. + + The below Gateway will expose both ports with their respective protocols. -Currently, STUNner limits each Gateway to a single transport protocol, e.g., UDP or TCP. This is intended to improve the consistency across the Kubernetes services of different cloud providers, which provide varying support for [mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types). If you still want to expose a UDP and a TCP port on the same IP using a single Gateway, add the annotation `stunner.l7mp.io/enable-mixed-protocol-lb: true` to the Gateway. + ```yaml + apiVersion: gateway.networking.k8s.io/v1 + kind: Gateway + metadata: + name: mixed-protocol-gateway + annotations: + stunner.l7mp.io/enable-mixed-protocol-lb: true + spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: TURN-UDP + - name: tcp-listener + port: 3479 + protocol: TURN-TCP + ``` -The below Gateway will expose both ports with their respective protocols. +1. **Retaining the source IP:** Normally, Kubernetes load balancers will apply a source IP address translation when ingesting packets into the cluster. However, some use cases require maintaining clients' source IP address for correct operation. For STUNner's intended use case, as an ingress media gateway exposing the cluster's media services over the TURN protocol, this does not matter. However, STUNner can also act as a STUN server, which requires clients' source IP to be retained by the load balancer. This can be achieved by adding the annotation `stunner.l7mp.io/external-traffic-policy: local` to a Gateway. This will set the [`service.spec.externalTrafficPolicy`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) field in the Service created by STUNner for the Gateway to `Local`. Note that this feature comes with fairly complex [limitations](https://kubernetes.io/docs/tutorials/services/source-ip), use it at your own [risk](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#caveats-and-limitations-when-preserving-source-ips). -```yaml -apiVersion: gateway.networking.k8s.io/v1 -kind: Gateway -metadata: - name: mixed-protocol-gateway - annotations: - stunner.l7mp.io/enable-mixed-protocol-lb: true -spec: - gatewayClassName: stunner-gatewayclass - listeners: - - name: udp-listener - port: 3478 - protocol: TURN-UDP - - name: tcp-listener - port: 3479 - protocol: TURN-TCP -``` +1. **Manually provisioning the dataplane:** In some cases it may be useful to manually provision a dataplane for a Gateway, e.g., to deploy `stunnerd` in a DeamonSet instead of a Deployment. Adding the annotation `stunner.l7mp.io/disable-managed-dataplane: true` to a Gateway will prevent STUNner from spawning a dataplane for the Gateway. This then allows one to manually create a `stunnerd` dataplane and connect it to the CDS server exposed by the operator to obtain the dataplane configuration. Remove the annotation to revert to the default mode and let STUNner to manage the dataplane for the Gateway. Manual dataplane provisioning requires intimate knowledge with the STUNner internals, use this feature only if you know what you are doing. -> [!WARNING] -> -> Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature. You can enable mixed-protocol LBs by annotating a Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. - -Some use cases require maintaining the source IP address of the client for correct operation. For the intended use case of STUNner, as an ingress media gateway exposing the cluster's media services over the TURN protocol, this does not matter. However, some users wish to use STUNner as a STUN server, which requires the original source IP to be retained by the load balancer. This can be achieved by adding the annotation `stunner.l7mp.io/external-traffic-policy: local` to a Gateway. This will set the [`service.spec.externalTrafficPolicy`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) field in the Service created by STUNner for the Gateway to `local`, which will instruct Kubernetes to preserve the original source IP address in clients' packets. - -> [!WARNING] -> -> This feature comes with fairly complex [limitations](https://kubernetes.io/docs/tutorials/services/source-ip), use it at your own [risk](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#caveats-and-limitations-when-preserving-source-ips). - -### Manually provisioning the dataplane - -In some cases it may be useful to manually provision a dataplane for a Gateway, e.g., to deploy `stunnerd` in a DeamonSet instead of a Deployment. Adding the annotation `stunner.l7mp.io/disable-managed-dataplane: true` to a Gateway will prevent STUNner from spawning a dataplane for the Gateway. This then allows one to manually create a `stunnerd` dataplane and connect it to the CDS server exposed by the operator to obtain the dataplane configuration. Remove the annotation to revert to the default mode and let STUNner to manage the dataplane for the Gateway. - -> [!WARNING] -> -> Manually provisioning a dataplane for a Gateway requires intimate knowledge with the STUNner internals, use this feature only if you know what you are doing. +1. **Selecting the NodePort:** By default, Kubernetes assigns a random external port from the range [32000-32767] for each listener of a Gateway exposed with a NodePort Service. This requires all ports in the [32000-32767] range to be opened on the external firewall, which may raise security concerns for hardened deployments. In order to assign specific nodeports to particular listeners, add the annotation `stunner.l7mp.io/nodeport:` `{listener_name_1:nodeport_1,listener_name_2:nodeport_2,...}` to the Gateway, where each key-value pair is a name of a listener and the selected (numeric) NodePort. The value itself must be proper a JSON map. Unknown listeners are silently ignored. Note that STUNner makes no specific effort to reconcile conflicting NodePorts: whenever the selected NodePort is unavailable Kubernetes will silently reject the Service, which can lead to hard-to-debug failures. Use this feature at your own risk. The below table summarizes the Gateway annotations supported by STUNner. @@ -231,7 +225,9 @@ The below table summarizes the Gateway annotations supported by STUNner. |:----------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------:| | `stunner.l7mp.io/service-type: ` | [Type of the Service](https://kubernetes.io/docs/concepts/services-networking/service) per Gateway, either `ClusterIP`, `NodePort`, or `LoadBalancer`. | `LoadBalancer` | | `stunner.l7mp.io/enable-mixed-protocol-lb: ` | [Mixed protocol load balancer service](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) support. | False | +| `stunner.l7mp.io/external-traffic-policy: ` | Se the value to `Local` to preserve clients' source IP at the load balancer. | `Cluster` | | `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway. | False | +| `stunner.l7mp.io/nodeport: ` | Request a specific NodePort for particular listeners. Value is a JSON map of listener-nodeport key-value pairs. | None | ## UDPRoute From b2b37eebe33cb6940627efdb9ebc66cfe7a6b905 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 24 May 2024 15:22:20 +0200 Subject: [PATCH 172/243] doc: Update examples --- docs/examples/direct-one2one-call/README.md | 194 +++--------- .../direct-one2one-call-server.yaml | 2 +- docs/examples/kurento-one2one-call/README.md | 284 ++++-------------- .../kurento-one2one-call-server.yaml | 4 +- docs/examples/simple-tunnel/README.md | 72 ++--- docs/examples/simple-tunnel/iperf-server.yaml | 2 +- 6 files changed, 129 insertions(+), 429 deletions(-) diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index 0358bc58..398ab618 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -2,8 +2,7 @@ ## Headless deployment: Direct one to one video call via STUNner -This tutorial showcases the *headless deployment model* of STUNner, that is, when WebRTC clients -connect to each other directly via STUNner, without going through a media server. +This tutorial showcases the *headless deployment model* of STUNner, that is, when WebRTC clients connect to each other directly via STUNner, without going through a media server. In this demo you will learn how to: @@ -16,83 +15,31 @@ In this demo you will learn how to: ### Prerequisites -The tutorial assumes a fresh STUNner installation; see the [STUNner installation and configuration -guide](../../INSTALL.md). Create a namespace called `stunner` if there is none. You need a -WebRTC-compatible browser to run this tutorial. Basically any modern browser will do; we usually -test our WebRTC applications with Firefox. +The tutorial assumes a fresh STUNner installation; see the [STUNner installation and configuration guide](../../INSTALL.md). Create a namespace called `stunner` if there is none. You need a WebRTC-compatible browser to run this tutorial. Basically any modern browser will do; we usually test our WebRTC applications with Firefox. ### Setup -The tutorial has been adopted from the [Kurento](https://www.kurento.org/) [one-to-one video call -tutorial](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutorial-one2one.html), but -we have [dropped the media -server](https://github.com/l7mp/kurento-tutorial-node/tree/master/direct-one2one-call) so that -clients will connect to each other directly via STUNner. We will deploy a -[Node.js](https://nodejs.org) application server into Kubernetes: this will serve the main HTML -page with the embedded video viewports and download the client-side JavaScript code to the -browsers. +The tutorial has been adopted from the [Kurento](https://www.kurento.org/) [one-to-one video call tutorial](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutorial-one2one.html), but we have [dropped the media server](https://github.com/l7mp/kurento-tutorial-node/tree/master/direct-one2one-call) so that clients will connect to each other directly via STUNner. We will deploy a [Node.js](https://nodejs.org) application server into Kubernetes: this will serve the main HTML page with the embedded video viewports and download the client-side JavaScript code to the browsers. ![STUNner standalone deployment architecture](../../img/stunner_direct_one2one_arch.svg) -Note that transcoding/transsizing and other media server goodies are not available in this setup: -the clients must be fully compatible to be able to establish an audio/video session in this -case. Consult the [one to one video call with Kurento](../kurento-one2one-call) tutorial to -learn how to set up a fully-fledged media server behind STUNner. +Note that transcoding/transsizing and other media server goodies are not available in this setup: the clients must be fully compatible to be able to establish an audio/video session in this case. Consult the [one to one video call with Kurento](../kurento-one2one-call) tutorial to learn how to set up a fully-fledged media server behind STUNner. ### Application server -The application server implements a simple JSON/WebSocket API two browser clients can call to -establish a two-party call. The caller and the callee will connect to each other via STUNner as -the TURN server, without the mediation of a media server. - -As the first step, each client registers a unique username with the application server by sending a -`register` message, which the server acknowledges in a `registerResponse` message. To start a call, -the caller sets up a [WebRTC -PeerConnection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection), generates an -SDP Offer, and sends it along to the application server in a `call` message. The application server -forwards the SDP Offer to the callee in an `incomingCall` message. If accepting the call, the -callee sets up a WebRTC PeerConnection, generates an SDP Answer, and returns it in an -`incomingCallResponse` message to the application server. The SDP Answer is then forwarded back to -the caller in a `callResponse` message. Meanwhile, the caller and the callee exchange ICE -candidates in the background. Once the ICE process connects, the caller and the callee start to -exchange audio/video frames via STUNner until one of the parties sends a `stop` message to the -application server to terminate the call. - -In order start the ICE conversation using STUNner as the STUN/TURN server, the browsers will need -to learn an ICE server configuration from the application server with STUNner's external IP -addresses/ports and the required STUN/TURN credentials. This must happen *before* the -PeerConnection is created in the clients: once the PeerConnection is running we can no longer -change the ICE configuration. - -We solve this problem by (1) generating a new ICE configuration every time a new client registers -with the application server and (2) sending the ICE configuration back to the client in the -`registerResponse` message. Note that this choice is suboptimal for time-locked STUNner -authentication modes (i.e., the `ephemeral` mode, see below), because clients' STUN/TURN -credentials might expire by the time they decide to connect. It is up to the application server -developer to make sure that clients' ICE server configuration is periodically updated. - -The default STUNner [install](/doc/INSTALL.md) contains a utility called the [STUNner -authentication service](https://github.com/l7mp/stunner-auth-service) that is purposed specifically -to generate ICE configurations for the application server. The service watches the running STUNner -configuration(s) from the Kubernetes API server and makes sure to generate STUN/TURN credentials -and ICE server configuration from the most recent STUNner config. - -The full application server code can be found -[here](https://github.com/l7mp/kurento-tutorial-node/tree/master/direct-one2one-call); below we -summarize the most important steps needed to call the STUNner authentication service in the -application to generate an ICE config for each client. - -1. Define the address and the port of the STUNner authentication service as environment variables - for the application server pod. This will allow the application server to query the STUNner - authentication server for TURN credentials. By default, the authentication service is available - at the address `stunner-auth.stunner-system.svc.cluster.local` on port TCP 8088 over HTTP, these - defaults can be overridden using the `STUNNER_AUTH_ADDR` and `STUNNER_AUTH_PORT` environment - variables. The ICE configuration returned by the auth service will contain an URI for each - public STUNner Gateway: you can filter on particular Kubernetes namespaces, gateways or gateway - listeners by the `STUNNER_NAMESPACE`, `STUNNER_GATEWAY` and `STUNNER_LISTENER` environment - variables. Since WebRTC media will be ingested into he cluster over the UDP listener called - `udp-listener` on the STUNner Gateway called `udp-gateway` deployed into the `stunner` - namespace, the Kubernetes manifest for the application server will look like the below. +The application server implements a simple JSON/WebSocket API for two browser clients to establish a two-party call. The caller and the callee will connect to each other via STUNner as the TURN server, without the mediation of a media server. + +As the first step, each client registers a unique username with the application server by sending a `register` message, which the server acknowledges in a `registerResponse` message. To start a call, the caller sets up a [WebRTC PeerConnection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection), generates an SDP Offer, and sends it along to the application server in a `call` message. The application server forwards the SDP Offer to the callee in an `incomingCall` message. If accepting the call, the callee sets up a WebRTC PeerConnection, generates an SDP Answer, and returns it in an `incomingCallResponse` message to the application server. The SDP Answer is then forwarded back to the caller in a `callResponse` message. Meanwhile, the caller and the callee exchange ICE candidates in the background. Once the ICE process connects, the caller and the callee start to exchange audio/video frames via STUNner until one of the parties sends a `stop` message to the application server to terminate the call. + +In order start the ICE conversation using STUNner as the STUN/TURN server, the browsers will need to learn an ICE server configuration from the application server with STUNner's external IP addresses/ports and the required STUN/TURN credentials. This must happen *before* the PeerConnection is created in the clients: once the PeerConnection is running we can no longer change the ICE configuration. + +We solve this problem by (1) generating a new ICE configuration every time a new client registers with the application server and (2) sending the ICE configuration back to the client in the `registerResponse` message. Note that this choice is suboptimal for [time-locked STUNner authentication modes](/doc/AUTH.md) (i.e., the `ephemeral` mode, see below), because clients' STUN/TURN credentials might expire by the time they decide to connect. It is up to the application server developer to make sure that clients' ICE server configuration is periodically updated. + +The default STUNner [install](/doc/INSTALL.md) contains a utility called the [STUNner authentication service](https://github.com/l7mp/stunner-auth-service) that is purposed specifically to generate ICE configurations for the application server. The service watches the running STUNner configuration(s) from the Kubernetes API server and makes sure to generate STUN/TURN credentials and ICE server configuration from the most recent STUNner config. + +The full application server code can be found [here](https://github.com/l7mp/kurento-tutorial-node/tree/master/direct-one2one-call); below we summarize the most important steps needed to call the STUNner authentication service in the application to generate an ICE config for each client. + +1. Define the address and the port of the STUNner authentication service as environment variables for the application server pod. This will allow the application server to query the STUNner authentication server for TURN credentials. By default, the authentication service is available at the address `stunner-auth.stunner-system.svc.cluster.local` on port TCP 8088 over HTTP, these defaults can be overridden using the `STUNNER_AUTH_ADDR` and `STUNNER_AUTH_PORT` environment variables. The ICE configuration returned by the auth service will contain an URI for each public STUNner Gateway: you can filter on particular Kubernetes namespaces, gateways or gateway listeners by the `STUNNER_NAMESPACE`, `STUNNER_GATEWAY` and `STUNNER_LISTENER` environment variables. Since WebRTC media will be ingested into he cluster over the UDP listener called `udp-listener` on the STUNner Gateway called `udp-gateway` deployed into the `stunner` namespace, the Kubernetes manifest for the application server will look like the below. ```yaml apiVersion: apps/v1 @@ -123,9 +70,7 @@ application to generate an ICE config for each client. [...] ``` -1. Modify the application server code to query the STUNner authentication server every time a valid - ICE config in needed. In particular, the code will generate a new ICE configuration before - returning a `registerResponse` to the client and piggyback it on the response message. +1. Modify the application server code to query the STUNner authentication server every time a valid ICE config in needed. In particular, the code will generate a new ICE configuration before returning a `registerResponse` to the client and piggyback it on the response message. ```js function register(id, name, ws, callback) { @@ -163,8 +108,7 @@ application to generate an ICE config for each client. } ``` -1. Next, modify the client-side JavaScript code to parse the ICE configuration received from the - application server from the `registerResponse` message. +1. Next, modify the client-side JavaScript code to parse the ICE configuration received from the application server from the `registerResponse` message. ```js var iceConfiguration; @@ -177,10 +121,7 @@ application to generate an ICE config for each client. } ``` -1. Then, every time the client calls the PeerConnection constructor pass in the stored [ICE - configuration](https://developer.mozilla.org/en-US/docs/Web/API/RTCIceServer). Note that - `kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv` is a small wrapper that makes it more convenient to - create PeerConnections with Kurento. +1. Then, every time the client calls the PeerConnection constructor pass in the stored [ICE configuration](https://developer.mozilla.org/en-US/docs/Web/API/RTCIceServer). Note that `kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv` is a small wrapper that makes it more convenient to create PeerConnections with Kurento. ```js var options = { @@ -195,11 +136,7 @@ Eventually, the call setup process will look like this. ![Call setup](sequence.png) -You can build the application server container locally from the tutorial -[repo](https://github.com/l7mp/kurento-tutorial-node/tree/master/direct-one2one-call), or you can -use the below manifest to fire up the prebuilt container image in a single step. This will deploy -the application server into the `stunner` namespace and exposes it in the Kubernetes LoadBalancer -service called `webrtc-server`. +You can build the application server container locally from the tutorial [repo](https://github.com/l7mp/kurento-tutorial-node/tree/master/direct-one2one-call), or you can use the below manifest to fire up the prebuilt container image in a single step. This will deploy the application server into the `stunner` namespace and exposes it in the Kubernetes LoadBalancer service called `webrtc-server`. ```console kubectl apply -f docs/examples/direct-one2one-call/direct-one2one-call-server.yaml @@ -207,16 +144,13 @@ kubectl apply -f docs/examples/direct-one2one-call/direct-one2one-call-server.ya ### STUNner configuration -Next, we deploy STUNner into Kubernetes. The manifest below will set up a minimal STUNner -gateway hierarchy to do just that: the setup includes two Gateway listeners, one at UDP:3478 and -another one at TCP:3478, plus and a UDPRoute. +Next, we deploy STUNner into Kubernetes. The manifest below will set up a minimal STUNner gateway hierarchy to do just that: the setup includes two Gateway listeners, one at UDP:3478 and another one at TCP:3478, plus and a UDPRoute. ```console kubectl apply -f docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml ``` -The most important component in the STUNner configuration is the Gateway: this will expose a public -TURN server on the UDP port 3478 through which clients will connect to each other. +The most important component in the STUNner configuration is the Gateway: this will expose a public TURN server on the UDP port 3478 through which clients will connect to each other. ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -234,15 +168,9 @@ spec: For later convenience we also create a TCP Gateway that runs on port TCP:3478. -We still need to ensure that STUNner will loop back client connections to itself. This is required -by the headless model, since in this case the only way for clients to connect to each other is via -the transport relay connections created by STUNner. The rest, that is, cross-connecting the -clients' media streams, is just pure TURN magic. +We still need to ensure that STUNner will loop back client connections to itself. This is required by the headless model, since in this case the only way for clients to connect to each other is via the transport relay connections created by STUNner. The rest, that is, cross-connecting the clients' media streams, is just pure TURN magic. -Here is the corresponding UDPRoute. Note that the route attaches itself to both the UDP and the TCP -Gateway, so no matter on which gateway the client connects via our UDPRoute will apply. Observe -also that the backend Services are the Gateway pods themselves: this makes sure that two clients -that happen to connect via different Gateway pods still connect to each other. +Here is the corresponding UDPRoute. Note that the route attaches itself to both the UDP and the TCP Gateway, so no matter on which gateway the client connects via our UDPRoute will apply. Observe also that the backend Services are the Gateway pods themselves: this makes sure that two clients that happen to connect via different Gateway pods still connect to each other. ```yaml apiVersion: stunner.l7mp.io/v1 @@ -262,8 +190,7 @@ spec: namespace: stunner ``` -Noe that the `stunner/udp-gateway` and `stunner/tcp-gateway` Kubernetes Services are automatically -created by STUNner to expose your Gateways, here we merely reuse these as backends. +Noe that the `stunner/udp-gateway` and `stunner/tcp-gateway` Kubernetes Services are automatically created by STUNner to expose your Gateways, here we merely reuse these as backends. ### Check your configuration @@ -282,11 +209,10 @@ NAME AGE udproute.stunner.l7mp.io/stunner-headless 2m50s ``` -You can also use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. For -instance, below is the human-readable config of the Gateway called `udp-gateway`: +You can also use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. Make sure to issue `make build` first to build `stunnerctl`, along with a set of other handy STUNner utilities, in the `bin/` directory. Below is the human-readable config of the Gateway called `udp-gateway`: ``` console -stunnerctl -n stunner config udp-gateway +bin/stunnerctl -n stunner config udp-gateway Gateway: stunner/udp-gateway (loglevel: "all:INFO") Authentication type: static, username/password: user-1/pass-1 Listeners: @@ -299,30 +225,19 @@ Listeners: ### Run the test -At this point, everything should be set up to make a video-call from your browser via -STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer service of the -application server. +At this point, everything should be set up to make a video-call from your browser via STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer service of the application server. ``` console export WEBRTC_SERVER_IP=$(kubectl get service webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` -Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, -register a user, repeat this process in an another browser window using a different user name, then -call one user from the other and enjoy a nice video-conference with yourself. +Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, register a user, repeat this process in an another browser window using a different user name, then call one user from the other and enjoy a nice video-conference with yourself. ### What is going on here? -The HTML page served by the application server contains a handy console port, which allows to track -the call setup process. We use the logs from one of the clients to demonstrate call establishment -with STUNner. +The HTML page served by the application server contains a handy console port, which allows to track the call setup process. We use the logs from one of the clients to demonstrate call establishment with STUNner. -- After registering with the application server, the console should show the content of the - `registerResponse` message. If all goes well, the response should show the ICE configuration - returned by the application server. The configuration should contain two TURN URIs for the two - gateways we have created in STUNner: one with UDP transport and another one with TCP. In - addition, the authentication credentials and the public IP addresses and ports should match - those in the output of `stunnerctl`. +- After registering with the application server, the console should show the content of the `registerResponse` message. If all goes well, the response should show the ICE configuration returned by the application server. The configuration should contain two TURN URIs for the two gateways we have created in STUNner: one with UDP transport and another one with TCP. In addition, the authentication credentials and the public IP addresses and ports should match those in the output of `stunnerctl`. ```js { @@ -346,55 +261,32 @@ with STUNner. } ``` -- Once configured with the above ICE server configuration, the browser will ask STUNner to open a - TURN transport relay connection for sending/receiving the media stream and generates a local - ICE candidate for each relay connection it creates. Note that only TURN-relay candidates are - generated: host and server-reflexive candidates would not work with STUNner anyway. (This is why - we set the `iceTransportPolicy` to type `relay` in the ICE server configuration above.) Locally - generated ICE candidates are sent by the browser to the application server, which in turn passes - them over verbatim to the other client. +- Once configured with the above ICE server configuration, the browser will ask STUNner to open a TURN transport relay connection for sending/receiving the media stream and generates a local ICE candidate for each relay connection it creates. Note that only TURN-relay candidates are generated: host and server-reflexive candidates would not work with STUNner anyway. (This is why we set the `iceTransportPolicy` to type `relay` in the ICE server configuration above.) Locally generated ICE candidates are sent by the browser to the application server, which in turn passes them over verbatim to the other client. ```console Sending message: {[...] "candidate:0 1 UDP 92020735 10.76.0.15 44703 typ relay raddr 10.76.0.15 rport 44703" [...]} ``` - Observe that the ICE candidate contains a private IP address (`10.76.0.15` in this case) as the - TURN relay connection address: this just happens to be the IP address of the STUNner pod that - receives the TURN allocation request from the browser. + Observe that the ICE candidate contains a private IP address (`10.76.0.15` in this case) as the TURN relay connection address: this just happens to be the IP address of the STUNner pod that receives the TURN allocation request from the browser. -- The ICE candidate generated by a client is received by the other client as a remote - ICE candidate. The process is executed in the other direction as well, so that eventually both - parties will have a set of local and remote ICE candidates they can probe for connectivity. +- The ICE candidate generated by a client is received by the other client as a remote ICE candidate. The process is executed in the other direction as well, so that eventually both parties will have a set of local and remote ICE candidates they can probe for connectivity. ```console Received message: {[...] "candidate:0 1 UDP 92020735 10.76.0.15 57511 typ relay raddr 10.76.0.15 rport 57511", [...]} ``` -- Once ICE candidates are exchanged, the browsers perform a connectivity check on each - candidate pair. With STUNner this usually succeeds with the first pair. +- Once ICE candidates are exchanged, the browsers perform a connectivity check on each candidate pair. With STUNner this usually succeeds with the first pair. -After connecting, video starts to flow between the two clients via the UDP/TURN connection opened -on STUNner. Note that browsers can be behind any type of NAT: STUNner makes sure that whatever -aggressive middlebox exists between itself and a client, media traffic will still flow seamlessly. +After connecting, audio and video start to flow between the two clients via the UDP/TURN connection opened on STUNner. Note that browsers can be behind any type of NAT: STUNner makes sure that whatever aggressive middlebox exists between itself and a client, media traffic will still flow seamlessly. ### Troubleshooting -Like in any sufficiently complex application, there are lots of moving parts in a Kubernetes-based -WebRTC service and many things can go wrong. Below is a list of steps to help debugging STUNner. - -* Cannot reach the application server: Make sure that the LoadBalancer IP is reachable and the TCP - port 8443 is available from your client. -* No ICE candidate appears: Most probably this occurs because the browser's ICE configuration does - not match the running STUNner config. Check that the ICE configuration returned by the - application server in the `registerResponse` message matches the output of `stunnerctl - config`. Examine the `stunner` pods' logs (`kubectl logs...`): permission-denied messages - typically indicate that STUN/TURN authentication was unsuccessful. -* No video-connection: This is most probably due to a communication issue between your client and - STUNner. Try disabling STUNner's UDP Gateway and force the browser to use TCP. -* Still no connection: follow the excellent [TURN troubleshooting - guide](https://www.giacomovacca.com/2022/05/troubleshooting-turn.html) to track down the - issue. Remember: your ultimate friends `tcpdump` and `Wireshark` are always there for you to - help! +Like in any sufficiently complex application, there are lots of moving parts in a Kubernetes-based WebRTC service and many things can go wrong. Below is a list of steps to help debugging STUNner. + +* Cannot reach the application server: Make sure that the LoadBalancer IP is reachable and the TCP port 8443 is available from your client. +* No ICE candidate appears: Most probably this occurs because the browser's ICE configuration does not match the running STUNner config. Check that the ICE configuration returned by the application server in the `registerResponse` message matches the output of `stunnerctl config`. Examine the `stunner` pods' logs (`kubectl logs...`): permission-denied messages typically indicate that STUN/TURN authentication was unsuccessful. +* No audio/video-connection: This is most probably due to a communication issue between your client and STUNner. Try disabling STUNner's UDP Gateway and force the browser to use TCP. +* Still no connection: follow the excellent [TURN troubleshooting guide](https://www.giacomovacca.com/2022/05/troubleshooting-turn.html) to track down the issue. Remember: your ultimate friends `tcpdump` and `Wireshark` are always there for you to help! ## Clean up diff --git a/docs/examples/direct-one2one-call/direct-one2one-call-server.yaml b/docs/examples/direct-one2one-call/direct-one2one-call-server.yaml index 069f4fa1..0fb8b3c8 100644 --- a/docs/examples/direct-one2one-call/direct-one2one-call-server.yaml +++ b/docs/examples/direct-one2one-call/direct-one2one-call-server.yaml @@ -29,7 +29,7 @@ spec: spec: containers: - name: webrtc-server - image: l7mp/direct-one2one-call-server + image: docker.io/l7mp/direct-one2one-call-server imagePullPolicy: Always # image: localhost/l7mp/direct-one2one-call-server # imagePullPolicy: Never diff --git a/docs/examples/kurento-one2one-call/README.md b/docs/examples/kurento-one2one-call/README.md index 4068791b..fa5c827c 100644 --- a/docs/examples/kurento-one2one-call/README.md +++ b/docs/examples/kurento-one2one-call/README.md @@ -2,9 +2,7 @@ ## Media-plane mode: One to one video call with Kurento via STUNner -This tutorial demonstrates the *media-plane deployment model* of STUNner, that is, when WebRTC -clients connect to each other via the Kurento media server deployed into Kubernetes. The media -servers are exposed to clients via a STUNner gateway. +This tutorial demonstrates the *media-plane deployment model* of STUNner, that is, when WebRTC clients connect to each other via the Kurento media server deployed into Kubernetes. The media servers are exposed to clients via a STUNner gateway. In this demo you will learn how to: @@ -14,102 +12,37 @@ In this demo you will learn how to: - configure STUNner to expose Kurento to clients, and - update the STUN/TURN credentials used by STUNner to improve security. - ## Installation ### Prerequisites -The tutorial assumes a fresh STUNner installation; see the [STUNner installation and configuration -guide](../../INSTALL.md). Create a namespace called `stunner` if there is none. You need a -WebRTC-compatible browser to run this tutorial. Basically any modern browser will do; we usually -test our WebRTC applications with Firefox. +The tutorial assumes a fresh STUNner installation; see the [STUNner installation and configuration guide](../../INSTALL.md). Create a namespace called `stunner` if there is none. You need a WebRTC-compatible browser to run this tutorial. Basically any modern browser will do; we usually test our WebRTC applications with Firefox. ### Setup -The tutorial has been adopted from the [Kurento](https://www.kurento.org/) [one-to-one video call -tutorial](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutorial-one2one.html), with -minimal -[modifications](https://github.com/l7mp/kurento-tutorial-node/tree/master/kurento-one2one-call) to -deploy it into Kubernetes and integrate it with STUNner. We will deploy a -[Node.js](https://nodejs.org) application server into Kubernetes for creating a browser-based -two-party WebRTC video-calls, the Kurento media server for media exchange and, potentially, -automatic audio/video transcoding, and configure STUNner to expose the Kurento server pool to -clients. +The tutorial has been adopted from the [Kurento](https://www.kurento.org/) [one-to-one video call tutorial](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutorial-one2one.html), with minimal [modifications](https://github.com/l7mp/kurento-tutorial-node/tree/master/kurento-one2one-call) to deploy it into Kubernetes and integrate it with STUNner. We will deploy a [Node.js](https://nodejs.org) application server into Kubernetes for creating a browser-based two-party WebRTC video-calls, the Kurento media server for media exchange and, potentially, automatic audio/video transcoding, and configure STUNner to expose the Kurento server pool to clients. ![STUNner media-plane deployment architecture](../../img/stunner_kurento_one2one_arch.svg) ### Application server -The application server implements a simple JSON/WebSocket API two browser clients can call to -establish a two-party call. The caller and the callee will connect to each other via STUNner as -the TURN server, using the Kurento media server to mediate audio/video calls. - -As the first step, each client registers a unique username with the application server by sending a -`register` message, which the server acknowledges in a `registerResponse` message. To start a call, -the caller sets up a [WebRTC -PeerConnection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection), generates an -SDP Offer, and sends it along to the application server in a `call` message. The application server -rings the callee in an `incomingCall` message. If accepting the call, the callee sets up a WebRTC -PeerConnection, generates an SDP Offer and sends it back the application server in an -`incomingCallResponse` message. - -At this point the application server has both party's SDP Offer, so the next step is to set up the -media pipeline in Kurento and process the SDP Offers through the media server. This is done by -opening a WebSocket connection to the URI `ws://kms.default.svc.cluster.local:8888/kurento` (this -is set on the server's command line in the Deployment -[manifest](kurento-one2one-call-server.yaml)): here, `kms.default.svc.cluster.local` is the DNS -name assigned by Kubernetes to the `kms` service and Kurento listens on the TCP port 8888 for -control connections. Note that this call gets load-balanced through the Kubernetes CNI's service -load-balancer so it will hit one random media server replica (if there are more). This ensures that -new calls are distributed evenly across media servers. - -The media server responds with an SDP Answer for both the callee and the caller, which the -application server immediately sends back to the appropriate clients: the caller receives the SDP -Answer in a `callResponse` message while the callee receives it in a `startCommunication` -message. Meanwhile, the caller and the callee generate local ICE candidates and send them over to -the application server, which forwards the candidates to Kurento. Kurento also generates a set of -ICE candidates, these will be passed by the application server to the clients. Note that the media -will flow between the clients and Kurento; the application server is there just to mediate the call -setup. Once the ICE process connects, the caller and the callee start to exchange audio/video -frames with Kurento via STUNner, until one of the parties sends a `stop` message to the application -server to terminate the call. For more info on the application server protocol, consult the -[Kurento -documentation](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutorial-one2one.html). - -In order to start the ICE conversation using STUNner as the STUN/TURN server, the browsers will -need to learn an ICE server configuration from the application server with STUNner's external IP -addresses/ports and STUN/TURN credentials. This must happen *before* the PeerConnection is created -in the clients: once the PeerConnection is running we can no longer change the ICE configuration. - -We solve this problem by (1) generating a new ICE configuration every time a new client registers -with the application server and (2) sending the ICE configuration back to the client in the -`registerResponse` message. Note that this choice is suboptimal for time-locked STUNner -authentication modes (i.e., the `ephemeral` mode, see below), because clients' STUN/TURN -credentials might expire by the time they decide to connect. It is up to the application server -developer to make sure that clients' ICE server configuration is periodically updated. - -The default STUNner [install](/doc/INSTALL.md) contains a utility called the [STUNner -authentication service](https://github.com/l7mp/stunner-auth-service) that is purposed specifically -to generate ICE configurations for the application server. The service watches the running STUNner -configuration(s) from the Kubernetes API server and makes sure to generate STUN/TURN credentials -and ICE server configuration from the most recent STUNner config. - -The full application server code can be found -[here](https://github.com/l7mp/kurento-tutorial-node/tree/master/kurento-one2one-call); below we -summarize the most important steps needed to call the STUNner authentication service in the -application to generate an ICE config for each client. - -1. Define the address and the port of the STUNner authentication service as environment variables - for the application server pod. This will allow the application server to query the STUNner - authentication server for TURN credentials. By default, the authentication service is available - at the address `stunner-auth.stunner-system.svc.cluster.local` on port TCP 8088 over HTTP, these - defaults can be overridden using the `STUNNER_AUTH_ADDR` and `STUNNER_AUTH_PORT` environment - variables. The ICE configuration returned by the auth service will contain an URI for each - public STUNner Gateway: you can filter on particular Kubernetes namespaces, gateways or gateway - listeners by the `STUNNER_NAMESPACE`, `STUNNER_GATEWAY` and `STUNNER_LISTENER` environment - variables. Since WebRTC media will be ingested into he cluster over the UDP listener called - `udp-listener` on the STUNner Gateway called `udp-gateway` deployed into the `stunner` - namespace, the Kubernetes manifest for the application server will look like the below. +The application server implements a simple JSON/WebSocket API for two browser clients to establish a two-party call. The caller and the callee will connect to each other via STUNner as the TURN server, using the Kurento media server to mediate audio/video calls. + +As the first step, each client registers a unique username with the application server by sending a `register` message, which the server acknowledges in a `registerResponse` message. To start a call, the caller sets up a [WebRTC PeerConnection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection), generates an SDP Offer, and sends it along to the application server in a `call` message. The application server rings the callee in an `incomingCall` message. If accepting the call, the callee sets up a WebRTC PeerConnection, generates an SDP Offer and sends it back the application server in an `incomingCallResponse` message. + +At this point the application server has both party's SDP Offer, so the next step is to set up the media pipeline in Kurento and process the SDP Offers through the media server. This is done by opening a WebSocket connection to the URI `ws://kms.default.svc.cluster.local:8888/kurento` (this is set on the server's command line in the Deployment [manifest](kurento-one2one-call-server.yaml)): here, `kms.default.svc.cluster.local` is the DNS name assigned by Kubernetes to the `kms` service and Kurento listens on the TCP port 8888 for control connections. Note that this call gets load-balanced through the Kubernetes CNI's service load-balancer so it will hit one random media server replica (if there are more). This ensures that new calls are distributed evenly across media servers. + +The media server responds with an SDP Answer for both the callee and the caller, which the application server immediately sends back to the appropriate clients: the caller receives the SDP Answer in a `callResponse` message while the callee receives it in a `startCommunication` message. Meanwhile, the caller and the callee generate local ICE candidates and send them over to the application server, which forwards the candidates to Kurento. Kurento also generates a set of ICE candidates, these will be passed by the application server to the clients. Note that the media will flow between the clients and Kurento; the application server is there just to mediate the call setup. Once the ICE process connects, the caller and the callee start to exchange audio/video frames with Kurento via STUNner, until one of the parties sends a `stop` message to the application server to terminate the call. For more info on the application server protocol, consult the [Kurento documentation](https://doc-kurento.readthedocs.io/en/latest/tutorials/node/tutorial-one2one.html). + +In order to start the ICE conversation using STUNner as the STUN/TURN server, the browsers will need to learn an ICE server configuration from the application server with STUNner's external IP addresses/ports and STUN/TURN credentials. This must happen *before* the PeerConnection is created in the clients: once the PeerConnection is running we can no longer change the ICE configuration. + +We solve this problem by (1) generating a new ICE configuration every time a new client registers with the application server and (2) sending the ICE configuration back to the client in the `registerResponse` message. Note that this choice is suboptimal for [time-locked STUNner authentication modes](/doc/AUTH.md) (i.e., the `ephemeral` mode, see below), because clients' STUN/TURN credentials might expire by the time they decide to connect. It is up to the application server developer to make sure that clients' ICE server configuration is periodically updated. + +The default STUNner [install](/doc/INSTALL.md) contains a utility called the [STUNner authentication service](https://github.com/l7mp/stunner-auth-service) that is purposed specifically to generate ICE configurations for the application server. The service watches the running STUNner configuration(s) from the Kubernetes API server and makes sure to generate STUN/TURN credentials and ICE server configuration from the most recent STUNner config. + +The full application server code can be found [here](https://github.com/l7mp/kurento-tutorial-node/tree/master/kurento-one2one-call); below we summarize the most important steps needed to call the STUNner authentication service in the application to generate an ICE config for each client. + +1. Define the address and the port of the STUNner authentication service as environment variables for the application server pod. This will allow the application server to query the STUNner authentication server for TURN credentials. By default, the authentication service is available at the address `stunner-auth.stunner-system.svc.cluster.local` on port TCP 8088 over HTTP, these defaults can be overridden using the `STUNNER_AUTH_ADDR` and `STUNNER_AUTH_PORT` environment variables. The ICE configuration returned by the auth service will contain an URI for each public STUNner Gateway: you can filter on particular Kubernetes namespaces, gateways or gateway listeners by the `STUNNER_NAMESPACE`, `STUNNER_GATEWAY` and `STUNNER_LISTENER` environment variables. Since WebRTC media will be ingested into he cluster over the UDP listener called `udp-listener` on the STUNner Gateway called `udp-gateway` deployed into the `stunner` namespace, the Kubernetes manifest for the application server will look like the below. ```yaml apiVersion: apps/v1 @@ -140,10 +73,7 @@ application to generate an ICE config for each client. [...] ``` -1. Modify the application server code to query the STUNner authentication server every time a a - valid ICE config in needed. In particular, the code will return the ICE configuration before - returning a `registerResponse` to the client, so that the generated ICE configuration can be - piggy-backed on the response message. +1. Modify the application server code to query the STUNner authentication server every time a a valid ICE config in needed. In particular, the code will return the ICE configuration before returning a `registerResponse` to the client, so that the generated ICE configuration can be piggy-backed on the response message. ```js function register(id, name, ws, callback) { @@ -181,10 +111,7 @@ application to generate an ICE config for each client. } ``` -1. Next, modify the [client-side JavaScript - code](https://github.com/l7mp/kurento-tutorial-node/blob/master/kurento-one2one-call/static/js/index.js) - to parse the ICE configuration received from the application server from the `registerResponse` - message. +1. Next, modify the [client-side JavaScript code](https://github.com/l7mp/kurento-tutorial-node/blob/master/kurento-one2one-call/static/js/index.js) to parse the ICE configuration received from the application server from the `registerResponse` message. ```js var iceConfiguration; @@ -197,12 +124,7 @@ application to generate an ICE config for each client. } ``` -1. Then, every time the client calls the [PeerConnection - constructor](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection), - pass in the stored [ICE - configuration](https://developer.mozilla.org/en-US/docs/Web/API/RTCIceServer). Note that - `kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv` is a small wrapper that makes it more convenient to - create PeerConnections with Kurento. +1. Then, every time the client calls the [PeerConnection constructor](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection), pass in the stored [ICE configuration](https://developer.mozilla.org/en-US/docs/Web/API/RTCIceServer). Note that `kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv` is a small wrapper that makes it more convenient to create PeerConnections with Kurento. ```js var options = { @@ -213,20 +135,13 @@ application to generate an ICE config for each client. webRtcPeer = kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options, ...); ``` -You can build the application server container locally from the tutorial -[repo](https://github.com/l7mp/kurento-tutorial-node/tree/master/kurento-one2one-call), or you can -use the below manifest to fire up the prebuilt container image in a single step. This will deploy -the application server into the `stunner` namespace and exposes it in the Kubernetes LoadBalancer -service called `webrtc-server`. +You can build the application server container locally from the tutorial [repo](https://github.com/l7mp/kurento-tutorial-node/tree/master/kurento-one2one-call), or you can use the below manifest to fire up the prebuilt container image in a single step. This will deploy the application server into the `stunner` namespace and exposes it in the Kubernetes LoadBalancer service called `webrtc-server`. ```console kubectl apply -f docs/examples/kurento-one2one-call/kurento-one2one-call-server.yaml ``` -Note that we disable STUN/TURN in Kurento: STUNner will make sure that your media servers will not -need to use NAT traversal (despite running with a private IP) and everything should still work just -fine. Below is the corresponding snippet from the [manifest](kurento-one2one-call-server.yaml), which sets -Kurento's environment variables accordingly: +Note that we disable STUN/TURN in Kurento: STUNner will make sure that your media servers will not need to use NAT traversal (despite running with a private IP) and everything should still work just fine. Below is the corresponding snippet from the [manifest](kurento-one2one-call-server.yaml), which sets Kurento's environment variables accordingly: ```yaml spec: @@ -247,24 +162,17 @@ spec: [...] ``` -And that's all. We added roughly 10-20 lines of fairly trivial code to the Kurento demo to make it -work with STUNner, with most of the changes needed to return the public STUN/TURN URI and -credentials to clients. If you allocate STUNner to a stable IP and domain name, you don't even need -to modify *anything* in the demo and it will just work. +And that's all. We added roughly 10-20 lines of fairly trivial code to the Kurento demo to make it work with STUNner, with most of the changes needed to return the public STUN/TURN URI and credentials to clients. If you allocate STUNner to a stable IP and domain name, you don't even need to modify *anything* in the demo and it will just work. ### STUNner configuration -Next, we deploy STUNner into Kubernetes. The manifest below will set up a minimal STUNner gateway -hierarchy to do just that: the setup includes a Gateway listener at UDP:3478 and a UDPRoute to -forward incoming calls into the cluster. +Next, we deploy STUNner into Kubernetes. The manifest below will set up a minimal STUNner gateway hierarchy to do just that: the setup includes a Gateway listener at UDP:3478 and a UDPRoute to forward incoming calls into the cluster. ```console kubectl apply -f docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml ``` -The most important component in the STUNner configuration is the TURN Gateway: this will expose a -public TURN server on the UDP port 3478 through which clients will connect to the Kurento media -server pods. +The most important component in the STUNner configuration is the TURN Gateway: this will expose a public TURN server on the UDP port 3478 through which clients will connect to the Kurento media server pods. ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -280,12 +188,7 @@ spec: protocol: TURN-UDP ``` -We set the `kms` service, which wraps the Kurento media server deployment, as the target in the -UDPRoute. Note that the target service lives in another namespace (the UDPRoute is in `stunner` -whereas the `kms` service is in the `default` namespace), STUNner will still be able to forward -connections (this is a small departure from the [Kubernetes Gateway -API](https://gateway-api.sigs.k8s.io) spec). The rest, that is, cross-connecting the clients' media -streams with Kurento's WebRTC endpoints, is just pure TURN magic. +We set the `kms` service, which wraps the Kurento media server deployment, as the target in the UDPRoute. Note that the target service lives in another namespace (the UDPRoute is in `stunner` whereas the `kms` service is in the `default` namespace), STUNner will still be able to forward connections (this is a small departure from the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) spec). The rest, that is, connecting clients' media streams to Kurento's WebRTC endpoints is just pure TURN magic. Below is the corresponding UDPRoute. @@ -307,6 +210,7 @@ spec: ### Check your configuration Check whether you have all the necessary objects installed into the `stunner` namespace. + ```console kubectl get gatewayconfigs,gateways,udproutes.stunner.l7mp.io -n stunner NAME REALM DATAPLANE AGE @@ -319,10 +223,10 @@ NAME AGE udproute.stunner.l7mp.io/kms-media-plane 84s ``` -You can also use the handy CLI tool called [`stunnerctl`](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. +You can also use the handy CLI tool called [`stunnerctl`](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. Make sure to issue `make build` first to build `stunnerctl`, along with a set of other handy STUNner utilities, in the `bin/` directory. ```console -stunnerctl -n stunner config udp-gateway +bin/stunnerctl -n stunner config udp-gateway Gateway: stunner/udp-gateway (loglevel: "all:INFO") Authentication type: static, username/password: user-1/pass-1 Listeners: @@ -335,29 +239,19 @@ Listeners: ### Run the test -At this point, everything should be set up to make a video-call from your browser via -STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer service of the -application server. +At this point, everything should be set up to make a video-call from your browser via STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer service of the application server. ``` console export WEBRTC_SERVER_IP=$(kubectl get service webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` -Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, -register a user, repeat this process in an another browser window using a different user name, then -call one user from the other and enjoy a nice video-conference with yourself. +Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, register a user, repeat this process in an another browser window using a different user name, then call one user from the other and enjoy a nice video-conference with yourself. ### What is going on here? -The HTML page served by the application server contains a handy console port, which allows to track -the call setup process. We use the logs from one of the clients to demonstrate call establishment -with STUNner. +The HTML page served by the application server contains a handy console port, which allows to track the call setup process. We use the logs from one of the clients to demonstrate call establishment with STUNner. -- After registering with the application server, the console should show the content of the - `registerResponse` message. If all goes well, the response should show the ICE configuration - returned by the application server. The configuration should contain a TURN URI for the UDP - Gateway we have created in STUNner. In addition, the authentication credentials and the public - IP addresses and ports should match those in the output of `stunnerctl`. +- After registering with the application server, the console should show the content of the `registerResponse` message. If all goes well, the response should show the ICE configuration returned by the application server. The configuration should contain a TURN URI for the UDP Gateway we have created in STUNner. In addition, the authentication credentials and the public IP addresses and ports should match those in the output of `stunnerctl`. ```js { @@ -376,99 +270,51 @@ with STUNner. } ``` -- Once bootstrapped with the above ICE server configuration, the browser will ask STUNner to open - a TURN transport relay connection for exchanging the video stream with Kurento and generates a - local ICE candidate for each relay connection it creates. Note that only TURN-relay candidates - are generated: host and server-reflexive candidates would not work with STUNner anyway. (This is - why we set the `iceTransportPolicy` to type `relay` in the ICE server configuration above.) - Locally generated ICE candidates are sent by the browser over to the application server, which - in turn passes them over verbatim to Kurento. +- Once bootstrapped with the above ICE server configuration, the browser will ask STUNner to open a TURN transport relay connection for exchanging the video stream with Kurento and generates a local ICE candidate for each relay connection it creates. Note that only TURN-relay candidates are generated: host and server-reflexive candidates would not work with STUNner anyway. (This is why we set the `iceTransportPolicy` to type `relay` in the ICE server configuration above.) Locally generated ICE candidates are sent by the browser over to the application server, which in turn passes them over verbatim to Kurento. ```console Sending message: {[...] "candidate:0 1 UDP 92020735 10.76.0.19 49944 typ relay raddr 10.76.0.19 rport 49944" [...]} ``` - Observe that the ICE candidate contains a private IP address (`10.76.0.19` in this case) as the - TURN relay connection address: this just happens to be the IP address of the STUNner pod that - receives the TURN allocation request from the browser. + Observe that the ICE candidate contains a private IP address (`10.76.0.19` in this case) as the TURN relay connection address: this just happens to be the IP address of the STUNner pod that receives the TURN allocation request from the browser. -- The media server generates ICE candidates as well. Since we disabled STUN/TURN in Kurento, only - host-type ICE candidates are generated by the media server. These will be sent back to the - clients as remote ICE candidates. +- The media server generates ICE candidates as well. Since we disabled STUN/TURN in Kurento, only host-type ICE candidates are generated by the media server. These will be sent back to the clients as remote ICE candidates. ```console Received message: {[...] "candidate:1 1 UDP 2015363327 10.76.0.17 13081 typ host" [...]} ``` - Observe that the ICE candidate again contains a private IP: in fact, `10.76.0.17` is the pod IP - address that belongs to the Kurento media server pod that received the call setup request from - the application server. - -- Once ICE candidates are exchanged, both clients have a set of local and remote ICE candidates - they can start to probe for connectivity. Local candidates were obtained from STUNner, these are - all relay-candidates and contain a pod IP address as the transport relay address, and the remote - candidates were generated by the media server. These are of host-type and likewise contain a pod - IP address. Since in the Kubernetes networking model ["pods can communicate with all other pods - on any other node without NAT"](https://kubernetes.io/docs/concepts/services-networking), all - local-remote ICE candidate pairs will have direct connectivity and ICE connectivity check will - succeed on the first candidate pair! - -After connecting, video starts to flow between the each client and the media server via the -UDP/TURN connection opened by STUNner, and the media server can perform all audio- and -video-processing the tasks a media server is expected to perform. Note that browsers may be behind -any type of NAT: STUNner makes sure that whatever aggressive middlebox exists between itself and a -client, media traffic will still flow seamlessly. + Observe that the ICE candidate again contains a private IP: in fact, `10.76.0.17` is the pod IP address that belongs to the Kurento media server pod that received the call setup request from the application server. + +- Once ICE candidates are exchanged, both clients have a set of local and remote ICE candidates they can start to probe for connectivity. Local candidates were obtained from STUNner, these are all relay-candidates and contain a pod IP address as the transport relay address, and the remote candidates were generated by the media server. These are of host-type and likewise contain a pod IP address. Since in the Kubernetes networking model ["pods can communicate with all other pods on any other node without NAT"](https://kubernetes.io/docs/concepts/services-networking), all local-remote ICE candidate pairs will have direct connectivity and ICE connectivity check will succeed on the first candidate pair! + +After connecting, audio and video start to flow between the each client and the media server via the UDP/TURN connection opened by STUNner, and the media server cross connects the audio- and video streams. Note that browsers may be behind any type of NAT: STUNner makes sure that whatever aggressive middlebox exists between itself and a client, media traffic will still flow seamlessly. ### Troubleshooting -Like in any sufficiently complex application, there are lots of moving parts in a Kubernetes-based -WebRTC service and many things can go wrong. Below is a list of steps to help debugging WebRTC -applications with STUNner. - -* Cannot reach the application server: Make sure that the LoadBalancer IP is reachable and the TCP - port 8443 is available from your client. -* No ICE candidate appears: Most probably this occurs because the browser's ICE configuration does - not match the running STUNner config. Check that the ICE configuration returned by the - application server in the `registerResponse` message matches the output of `stunnerctl - running-config`. Examine the `stunner` pods' logs (`kubectl logs...`): permission-denied messages - typically indicate that STUN/TURN authentication was unsuccessful. -* No video-connection: This is most probably due to a communication issue between your client and - STUNner. Try disabling STUNner's UDP Gateway and force the browser to use TCP. -* Still no connection: follow the excellent [TURN troubleshooting - guide](https://www.giacomovacca.com/2022/05/troubleshooting-turn.html) to track down the - issue. Remember: your ultimate friends `tcpdump` and `Wireshark` are always there for you to - help! +Like in any sufficiently complex application, there are lots of moving parts in a Kubernetes-based WebRTC service and many things can go wrong. Below is a list of steps to help debugging WebRTC applications with STUNner. + +* Cannot reach the application server: Make sure that the LoadBalancer IP is reachable and the TCP port 8443 is available from your client. +* No ICE candidate appears: Most probably this occurs because the browser's ICE configuration does not match the running STUNner config. Check that the ICE configuration returned by the application server in the `registerResponse` message matches the output of `stunnerctl running-config`. Examine the `stunner` pods' logs (`kubectl logs...`): permission-denied messages typically indicate that STUN/TURN authentication was unsuccessful. +* No audio/video-connection: This is most probably due to a communication issue between your client and STUNner. Try disabling STUNner's UDP Gateway and force the browser to use TCP. +* Still no connection: follow the excellent [TURN troubleshooting guide](https://www.giacomovacca.com/2022/05/troubleshooting-turn.html) to track down the issue. Remember: your ultimate friends `tcpdump` and `Wireshark` are always there for you to help! ## Update STUN/TURN credentials -As shown in the `stunnerctl` output, STUNner currently runs with fairly poor security: using -`static` authentication, sharing a single username/password pair between all active sessions. +As shown in the `stunnerctl` output, STUNner currently runs with fairly poor security: using `static` authentication, sharing a single username/password pair between all active sessions. ``` console -stunnerctl -n stunner config udp-gateway +bin/stunnerctl -n stunner config udp-gateway Gateway: stunner/udp-gateway (loglevel: "all:INFO") Authentication type: static, username/password: user-1/pass-1 ... ``` -Since static credentials are just what they are, static, it is easy to extract the STUN/TURN -credentials on the client side for potentially nefarious purposes. Note that attackers should not -be able to make too much harm with these credentials, since the only Kubernetes service they can -reach via STUNner is the Kurento media server pool. This is why we have installed the UDPRoute: -STUNner will allow clients to connect *only* to the backend service(s) of the UDPRoute, and nothing -else. Then, attackers would need access to the application-server to open WebRTC endpoints on the -media server for their own purposes, but media servers should be secure by default no? - -In other words, *STUNner's default security model is exactly the same as if we put the application -servers and media servers on public-facing physical servers*. - -Still, it would be nice to use per-session passwords. STUNner allows you to do that, by changing -the authentication type to `ephemeral` instead of `static`. Even better: STUNner's ephemeral TURN -credentials are valid only for a specified time (one day by default, but you can override this via -the [authentication service](https://github.com/l7mp/stunner-auth-service)), after which they -expire and attackers can no longer reuse them. To make things even better, we don't even have to -work too much to switch STUNner to the `ephemeral` authentication mode: it is enough to update the -GatewayConfig and everything should happen from this point automagically. +Since static credentials are just what they are, static, it is easy to extract the STUN/TURN credentials on the client side for potentially nefarious purposes. Note that attackers should not be able to make too much harm with these credentials, since the only Kubernetes service they can reach via STUNner is the Kurento media server pool. This is why we have installed the UDPRoute: STUNner will allow clients to connect *only* to the backend service(s) of the UDPRoute, and nothing else. Then, attackers would need access to the application-server to open WebRTC endpoints on the media server for their own purposes, but media servers should be secure by default no? + +In other words, *STUNner's default security model is exactly the same as if we put the application servers and media servers on public-facing physical servers*. + +That being said, it would be nice to use per-session passwords. STUNner allows you to do that, by changing the authentication type to `ephemeral` instead of `static`. Even better: STUNner's ephemeral TURN credentials are valid only for a specified time (one day by default, but you can override this via the [authentication service](https://github.com/l7mp/stunner-auth-service)), after which they expire and attackers can no longer reuse them. To make things even better, we don't even have to work too much to switch STUNner to the `ephemeral` authentication mode: it is enough to update the GatewayConfig and everything should happen from this point automagically. ```console kubectl apply -f - <` by Kubernetes); this makes sure that the -only way to reach this service from the local iperf speed-test client is through STUNner. +This will start an Deployment that runs the iperf server and wraps it in a Kubernetes service called `iperf-server` of type ClusterIP. Check this service and make sure that it is not exposed to the outside world (i.e., `EXTERNAL-IP` is set to `` by Kubernetes); this makes sure that the only way to reach this service from the local iperf speed-test client is through STUNner. ```console kubectl get service iperf-server -o wide @@ -52,17 +43,13 @@ iperf-server ClusterIP 10.120.5.36 5001/UDP,5001/TCP 19s ### STUNner configuration -Expose the service via the STUNner. The pre-compiled manifest below will create the required -GatewayClass and GateayConfig resources, fire up a Gateway listener at UDP:3478 and another one on -TCP:3478, and route client connections received on the gateways to the `iperf-server` -service. +Expose the service via the STUNner. The pre-compiled manifest below will create the required GatewayClass and GateayConfig resources, fire up a Gateway listener at UDP:3478 and another one on TCP:3478, and route client connections received on the gateways to the `iperf-server` service. + ```console kubectl apply -f docs/examples/simple-tunnel/iperf-stunner.yaml ``` -For convenience, below is a dump of the Gateway and UDPRoute resources the manifests create. Note -that the UDPRoute specifies the `iperf-server` service as the `backendRef`, which makes sure that -STUNner will forward the client connections received in any of the Gateways to the iperf server. +For convenience, below is a dump of the Gateway and UDPRoute resources the manifests create. Note that the UDPRoute specifies the `iperf-server` service as the `backendRef`, which makes sure that STUNner will forward the client connections received in any of the Gateways to the iperf server. ```yaml apiVersion: gateway.networking.k8s.io/v1 @@ -123,11 +110,10 @@ NAME AGE udproute.stunner.l7mp.io/iperf-server 139m ``` -You can also use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration for the -UDP gateway. +You can also use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration for the UDP gateway. Make sure to issue `make build` first to build `stunnerctl`, along with a set of other handy STUNner utilities, in the `bin/` directory. ``` console -stunnerctl -n stunner config udp-gateway +bin/stunnerctl -n stunner config udp-gateway Gateway: stunner/udp-gateway (loglevel: "all:INFO") Authentication type: static, username/password: user-1/pass-1 Listeners: @@ -141,7 +127,7 @@ Listeners: Likewise, the below will dump the config for the TCP gateway. ``` console -stunnerctl -n stunner config tcp-gateway +bin/stunnerctl -n stunner config tcp-gateway Gateway: stunner/tcp-gateway (loglevel: "all:INFO") Authentication type: static, username/password: user-1/pass-1 Listeners: @@ -152,12 +138,10 @@ Listeners: Endpoints: [10.76.1.4, 10.80.4.47] ``` -NOTE: It usually takes 30-60 seconds for Kubernetes to assign an external IP address to STUNner -gateways. As long as the external address is in `` status, STUNner exposes the Gateway on -a NodePort. Once Kubernetes finishes the exposition of the Gateway service, STUNner will pick up -the new address/port and update the config accordingly. +NOTE: It usually takes 30-60 seconds for Kubernetes to assign an external IP address to STUNner gateways. As long as the external address is in `` status, STUNner exposes the Gateway on a NodePort. Once Kubernetes finishes the exposition of the Gateway service, STUNner will pick up the new address/port and update the config accordingly. If in doubt, you can always query Kubernetes for the service statuses. + ``` console kubectl get -n stunner services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE @@ -168,29 +152,22 @@ udp-gateway LoadBalancer 10.0.14.218 35.205.10.190 3478:31048/UDP 143m ### Run the benchmark -We will need to learn the ClusterIP assigned by Kubernetes to the `iperf-server` service: this will -be the peer address to which `turncat` will ask STUNner to relay the iperf test traffic. +We will need to learn the ClusterIP assigned by Kubernetes to the `iperf-server` service: this will be the peer address to which `turncat` will ask STUNner to relay the iperf test traffic. ``` console export IPERF_ADDR=$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}") ``` -Next, set up `turncat` to listen on `UDP:127.0.0.1:5000` and tunnel connections from this listener -via the STUNner STUN/TURN listener `udp-listener` to the iperf server. Luckily, `turncat` is clever -enough to [parse the running STUNner configuration](../../cmd/turncat) from Kubernetes and set the -STUN/TURN server public address/port and the authentication credentials accordingly. +Next, set up `turncat` to listen on `UDP:127.0.0.1:5000` and tunnel connections from this listener via the STUNner STUN/TURN listener `udp-listener` to the iperf server. Luckily, `turncat` is clever enough to [parse the running STUNner configuration](../../cmd/turncat) from Kubernetes and set the STUN/TURN server public address/port and the authentication credentials accordingly. ``` console -./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener \ +bin/turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener \ udp://$IPERF_ADDR:5001 ``` -The most important part here is the TURN meta-URI: `k8s://stunner/udp-gateway:udp-listener` -instructs `turncat` to look for the Gateway called `udp-gateway` in the `stunner` namespace and -create a connection to the TURN listener called `udp-listener` of the Gateway. +The most important part here is the TURN meta-URI: `k8s://stunner/udp-gateway:udp-listener` instructs `turncat` to look for the Gateway called `udp-gateway` in the `stunner` namespace and create a connection to the TURN listener called `udp-listener` of the Gateway. -Fire up an iperf client from another terminal that will connect to STUNner via `turncat` and start -the benchmark. +Fire up an iperf client from another terminal that will connect to STUNner via `turncat` and start the benchmark. ```console iperf -c localhost -p 5000 -u -i 1 -l 100 -b 800000 -t 10 @@ -211,16 +188,12 @@ UDP buffer size: 208 KByte (default) [ 1] 0.0000-9.9204 sec 977 KBytes 807 Kbits/sec 1.426 ms 0/10003 (0%) 14.256/10.791/97.428/ 4.993 ms 1008 pps 1.40 KByte 7.07 ``` -The results show that we have managed to send 1000 packets/sec through STUNner to the iperf server -without packet loss, at an average one-way latency of 14.2 ms and 1.426 ms jitter. Not bad from a -Kubernetes cluster running in some remote datacenter! +The results show that we have managed to send 1000 packets/sec through STUNner to the iperf server without packet loss, at an average one-way latency of 14.2 ms and 1.426 ms jitter. Not bad from a Kubernetes cluster running in some remote datacenter! -Repeating the test, this time with a STUN/TURN over TCP, casts a somewhat different picture. Notice -the new meta-URI: `k8s://stunner/tcp-gateway:tcp-listener` to select the TURN server exposed on -TCP for`turncat`. +Repeating the test, this time with a STUN/TURN over TCP, casts a somewhat different picture. Notice the new meta-URI: `k8s://stunner/tcp-gateway:tcp-listener` to select the TURN server exposed on TCP for`turncat`. ``` console -./turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/tcp-gateway:tcp-listener \ +bin/turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/tcp-gateway:tcp-listener \ udp://$IPERF_ADDR:5001 ``` @@ -232,10 +205,7 @@ iperf -c localhost -p 5000 -u -l 100 -b 8000000 -o /dev/null -t 10 && \ [ 3] 0.0000-9.9365 sec 9.41 MBytes 7.94 Mbits/sec 0.085 ms 1361/100003 (1.4%) 148.261/21.098/454.266/73.704 ms 9927 pps 144 KByte 6.70 ``` -It seems that average latency has jumped to 148 ms, with a max latency of close to 460 ms! That's -why you should [avoid TCP at all -cost](https://bloggeek.me/why-you-should-prefer-udp-over-tcp-for-your-webrtc-sessions) in real-time -communications. +It seems that average latency has jumped to 148 ms, with a max latency of close to 460 ms! That's why you should [avoid TCP at all cost](https://bloggeek.me/why-you-should-prefer-udp-over-tcp-for-your-webrtc-sessions) in real-time communications. ### Cleaning up diff --git a/docs/examples/simple-tunnel/iperf-server.yaml b/docs/examples/simple-tunnel/iperf-server.yaml index e89b5895..84fb31ab 100644 --- a/docs/examples/simple-tunnel/iperf-server.yaml +++ b/docs/examples/simple-tunnel/iperf-server.yaml @@ -15,7 +15,7 @@ spec: app: iperf-server spec: containers: - - image: l7mp/net-debug:latest + - image: docker.io/l7mp/net-debug:latest name: net-debug command: ["/usr/bin/iperf"] args: ["-s", "-p", "5001", "-u", "-e"] From bd4199cfbb45044864d92bce0883d8b01d34ebf3 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 27 May 2024 19:41:36 +0200 Subject: [PATCH 173/243] chore: Updated release notes --- docs/GATEWAY.md | 4 ++-- docs/RELEASE.md | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index eecb9def..8b7ff618 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -213,11 +213,11 @@ STUNner defines the following special annotations: protocol: TURN-TCP ``` -1. **Retaining the source IP:** Normally, Kubernetes load balancers will apply a source IP address translation when ingesting packets into the cluster. However, some use cases require maintaining clients' source IP address for correct operation. For STUNner's intended use case, as an ingress media gateway exposing the cluster's media services over the TURN protocol, this does not matter. However, STUNner can also act as a STUN server, which requires clients' source IP to be retained by the load balancer. This can be achieved by adding the annotation `stunner.l7mp.io/external-traffic-policy: local` to a Gateway. This will set the [`service.spec.externalTrafficPolicy`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) field in the Service created by STUNner for the Gateway to `Local`. Note that this feature comes with fairly complex [limitations](https://kubernetes.io/docs/tutorials/services/source-ip), use it at your own [risk](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#caveats-and-limitations-when-preserving-source-ips). +1. **Retaining the source IP:** Normally, Kubernetes load balancers apply source IP address translation when ingesting packets into the cluster. This replaces clients' original IP address with a private IP address. For STUNner's intended use case, as an ingress media gateway exposing the cluster's media services over the TURN protocol, this does not matter. However, STUNner can also act as a STUN server, which requires clients' source IP to be retained at the load balancer. This can be achieved by adding the annotation `stunner.l7mp.io/external-traffic-policy: local` to a Gateway, which will set the [`service.spec.externalTrafficPolicy`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) field in the Service created by STUNner for the Gateway to `Local`. Note that this Kubernetes feature comes with fairly complex [limitations](https://kubernetes.io/docs/tutorials/services/source-ip): if a STUN or TURN request hits a Kubernetes node that is not running a `stunnerd` pod, then the request will silently fail. This is required for Kubernetes to retain the client IP, which otherwise would be lost when passing packets between nodes. Use this setting at your own [risk](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#caveats-and-limitations-when-preserving-source-ips). 1. **Manually provisioning the dataplane:** In some cases it may be useful to manually provision a dataplane for a Gateway, e.g., to deploy `stunnerd` in a DeamonSet instead of a Deployment. Adding the annotation `stunner.l7mp.io/disable-managed-dataplane: true` to a Gateway will prevent STUNner from spawning a dataplane for the Gateway. This then allows one to manually create a `stunnerd` dataplane and connect it to the CDS server exposed by the operator to obtain the dataplane configuration. Remove the annotation to revert to the default mode and let STUNner to manage the dataplane for the Gateway. Manual dataplane provisioning requires intimate knowledge with the STUNner internals, use this feature only if you know what you are doing. -1. **Selecting the NodePort:** By default, Kubernetes assigns a random external port from the range [32000-32767] for each listener of a Gateway exposed with a NodePort Service. This requires all ports in the [32000-32767] range to be opened on the external firewall, which may raise security concerns for hardened deployments. In order to assign specific nodeports to particular listeners, add the annotation `stunner.l7mp.io/nodeport:` `{listener_name_1:nodeport_1,listener_name_2:nodeport_2,...}` to the Gateway, where each key-value pair is a name of a listener and the selected (numeric) NodePort. The value itself must be proper a JSON map. Unknown listeners are silently ignored. Note that STUNner makes no specific effort to reconcile conflicting NodePorts: whenever the selected NodePort is unavailable Kubernetes will silently reject the Service, which can lead to hard-to-debug failures. Use this feature at your own risk. +1. **Selecting the NodePort:** By default, Kubernetes assigns a random external port from the range [32000-32767] to each listener of a Gateway exposed with a NodePort Service. This requires all ports in the [32000-32767] range to be opened on the external firewall, which may raise security concerns for hardened deployments. In order to assign specific nodeports to particular listeners, add the annotation `stunner.l7mp.io/nodeport:` `{listener_name_1:nodeport_1,listener_name_2:nodeport_2,...}` to the Gateway, where each key-value pair is a name of a listener and the selected (numeric) NodePort. The value itself must be proper a JSON map. Unknown listeners are silently ignored. Note that STUNner makes no specific effort to reconcile conflicting NodePorts: whenever the selected NodePort is unavailable Kubernetes will silently reject the Service, which can lead to hard-to-debug failures. Use this feature at your own risk. The below table summarizes the Gateway annotations supported by STUNner. diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 3496b5db..484e0353 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -17,6 +17,7 @@ Let the new version be vX.Y.Z. ## STUNner auth service +- this must finish before releasing the operator, otherwise the Helm build will fail - `git pull` - bump `github.com/l7mp/stunner` version to vX.Y.Z in the `go.mod` - `go mod tidy` From af70f3de0289d4e7caf24e98e90824ba5bf352b2 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 28 May 2024 13:03:48 +0200 Subject: [PATCH 174/243] doc(examples/cloudretro): Fix minor issues - use FQ docker image name to make it work with podman - fix external_ip query command - add stunner install instruction due to non-default namespace - detail testing by mentioning the URL to visit --- docs/examples/cloudretro/README.md | 12 +++++++++--- .../cloudretro/cloudretro-setup-coordinator.yaml | 6 +++--- .../cloudretro/cloudretro-setup-workers.yaml | 6 +++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/examples/cloudretro/README.md b/docs/examples/cloudretro/README.md index e0d10442..206aa256 100644 --- a/docs/examples/cloudretro/README.md +++ b/docs/examples/cloudretro/README.md @@ -66,7 +66,7 @@ the exposed service of the Coordinator, which clients will connect to. Running command will result the IP address assigned by the Kubernetes load-balancer: ```console -export EXTERNAL_IP=$(kubectl get service -n cloudretro coordinator-lb-svc -o jsonpath='{.status.loadBalancer.ingress[0].ip})' +export EXTERNAL_IP=$(kubectl get service -n cloudretro coordinator-lb-svc -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` If Kubernetes refuses to assign an external IP to your service after a couple of minutes, you will @@ -79,7 +79,13 @@ the CloudRetro servers running on a private pod IP address. That is where STUNne ### STUNner -You will need to install Stunner on Your cluster, you can do this by following the official [installation guide](../../INSTALL.md#installation). +First we install STUNner gateway operator with helm ([more info](https://github.com/l7mp/stunner-helm)): +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner +``` + Wait until all the necessary resources are up and running, then you are ready to continue. Next, register STUNner with the Kubernetes Gateway API. @@ -148,7 +154,7 @@ the CloudRetro config accordingly and restart CloudRetro to pick up the new conf ### Test -At this point, you should be able to play SuperMario on CloudRetro installed in Kubernetes. +At this point, you can play SuperMario on CloudRetro installed in Kubernetes by navigating to `http://${EXTERNAL-IP}:8000` in your web browser. ![Supre Maro let's go](mario-super.gif) diff --git a/docs/examples/cloudretro/cloudretro-setup-coordinator.yaml b/docs/examples/cloudretro/cloudretro-setup-coordinator.yaml index 4f6f4dfb..02feebac 100644 --- a/docs/examples/cloudretro/cloudretro-setup-coordinator.yaml +++ b/docs/examples/cloudretro/cloudretro-setup-coordinator.yaml @@ -3,7 +3,7 @@ # The following yaml script will install a cloud-retro setup in your Kubernetes cluster # with the needed services. With an invalid configuration of course, for this to properly work # You need the apply-config.sh minimalistic shell script to update the config with the unclear addresses. -# +# # # In short, no touching. # @@ -39,7 +39,7 @@ spec: spec: containers: - name: coordinator - image: valniae/snekyrepo:crdi + image: docker.io/valniae/snekyrepo:crdi envFrom: - configMapRef: name: cloudretro-config-c @@ -61,4 +61,4 @@ spec: - port: 8000 targetPort: 8000 #nodePort: 30001 - type: LoadBalancer \ No newline at end of file + type: LoadBalancer diff --git a/docs/examples/cloudretro/cloudretro-setup-workers.yaml b/docs/examples/cloudretro/cloudretro-setup-workers.yaml index c0fd002c..ded50826 100644 --- a/docs/examples/cloudretro/cloudretro-setup-workers.yaml +++ b/docs/examples/cloudretro/cloudretro-setup-workers.yaml @@ -3,7 +3,7 @@ # The following yaml script will install a cloud-retro setup in your Kubernetes cluster # with the needed services. With an invalid configuration of course, for this to properly work # You need the apply-config.sh minimalistic shell script to update the config with the unclear addresses. -# +# # Please not that this will only install the workers. For them to work, you will need a coordinator deployment as well. # # In short, no touching. @@ -42,7 +42,7 @@ spec: spec: containers: - name: worker - image: valniae/snekyrepo:crdi + image: docker.io/valniae/snekyrepo:crdi envFrom: - configMapRef: name: cloudretro-config-w @@ -80,4 +80,4 @@ spec: - protocol: UDP port: 8443 targetPort: 8443 - type: ClusterIP \ No newline at end of file + type: ClusterIP From 9aba8b6440fc3b294f3db9e81b8a89148f9a6a16 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 28 May 2024 18:42:26 +0200 Subject: [PATCH 175/243] chore: Update static CRDs --- deploy/manifests/static/stunner-crd.yaml | 3893 +++++++++++++--------- 1 file changed, 2304 insertions(+), 1589 deletions(-) diff --git a/deploy/manifests/static/stunner-crd.yaml b/deploy/manifests/static/stunner-crd.yaml index c7243451..f4a60abc 100644 --- a/deploy/manifests/static/stunner-crd.yaml +++ b/deploy/manifests/static/stunner-crd.yaml @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.13.0 + controller-gen.kubebuilder.io/version: v0.14.0 name: gatewayconfigs.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -34,14 +34,19 @@ spec: description: GatewayConfig is the Schema for the gatewayconfigs API properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -54,18 +59,18 @@ spec: format: int32 type: integer authRef: - description: 'Note that externally set credentials override any inline - auth credentials (AuthType, AuthUsername, etc.): if AuthRef is nonempty - then it is expected that the referenced Secret exists and *all* - authentication credentials are correctly set in the referenced Secret - (username/password or shared secret). Mixing of credential sources - (inline/external) is not supported.' + description: |- + Note that externally set credentials override any inline auth credentials (AuthType, + AuthUsername, etc.): if AuthRef is nonempty then it is expected that the referenced + Secret exists and *all* authentication credentials are correctly set in the referenced + Secret (username/password or shared secret). Mixing of credential sources + (inline/external) is not supported. properties: group: default: "" - description: Group is the group of the referent. For example, - "gateway.networking.k8s.io". When unspecified or empty string, - core API group is inferred. + description: |- + Group is the group of the referent. For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API group is inferred. maxLength: 253 pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string @@ -82,12 +87,18 @@ spec: minLength: 1 type: string namespace: - description: "Namespace is the namespace of the referenced object. - When unspecified, the local namespace is inferred. \n Note that - when a namespace different than the local namespace is specified, - a ReferenceGrant object is required in the referent namespace - to allow that namespace's owner to accept the reference. See - the ReferenceGrant documentation for details. \n Support: Core" + description: |- + Namespace is the namespace of the referenced object. When unspecified, the local + namespace is inferred. + + + Note that when a namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace to allow that + namespace's owner to accept the reference. See the ReferenceGrant + documentation for details. + + + Support: Core maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -103,23 +114,25 @@ spec: type: string dataplane: default: default - description: Dataplane defines the dataplane (stunnerd image, version, - etc) for STUNner gateways using this GatewayConfig. + description: |- + Dataplane defines the dataplane (stunnerd image, version, etc) for STUNner gateways + using this GatewayConfig. type: string loadBalancerServiceAnnotations: additionalProperties: type: string - description: "LoadBalancerServiceAnnotations is a list of annotations - that will go into the LoadBalancer services created automatically - by the operator to wrap Gateways. \n NOTE: removing annotations - from a GatewayConfig will not result in the removal of the corresponding - annotations from the LoadBalancer service, in order to prevent the - accidental removal of an annotation installed there by Kubernetes - or the cloud provider. If you really want to remove an annotation, - do this manually or simply remove all Gateways (which will remove - the corresponding LoadBalancer services), update the GatewayConfig - and then recreate the Gateways, so that the newly created LoadBalancer - services will contain the required annotations." + description: |- + LoadBalancerServiceAnnotations is a list of annotations that will go into the + LoadBalancer services created automatically by the operator to wrap Gateways. + + + NOTE: removing annotations from a GatewayConfig will not result in the removal of the + corresponding annotations from the LoadBalancer service, in order to prevent the + accidental removal of an annotation installed there by Kubernetes or the cloud + provider. If you really want to remove an annotation, do this manually or simply remove + all Gateways (which will remove the corresponding LoadBalancer services), update the + GatewayConfig and then recreate the Gateways, so that the newly created LoadBalancer + services will contain the required annotations. type: object logLevel: description: LogLevel specifies the default loglevel for the STUNner @@ -132,11 +145,13 @@ spec: type: string realm: default: stunner.l7mp.io - description: "Realm defines the STUN/TURN authentication realm to - be used for clients toauthenticate with STUNner. \n The realm must - consist of lower case alphanumeric characters or '-', and must start - and end with an alphanumeric character. No other punctuation is - allowed." + description: |- + Realm defines the STUN/TURN authentication realm to be used for clients toauthenticate + with STUNner. + + + The realm must consist of lower case alphanumeric characters or '-', and must start and + end with an alphanumeric character. No other punctuation is allowed. pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string sharedSecret: @@ -169,14 +184,19 @@ spec: description: GatewayConfig is the Schema for the gatewayconfigs API properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -189,18 +209,18 @@ spec: format: int32 type: integer authRef: - description: 'Note that externally set credentials override any inline - auth credentials (AuthType, AuthUsername, etc.): if AuthRef is nonempty - then it is expected that the referenced Secret exists and *all* - authentication credentials are correctly set in the referenced Secret - (username/password or shared secret). Mixing of credential sources - (inline/external) is not supported.' + description: |- + Note that externally set credentials override any inline auth credentials (AuthType, + AuthUsername, etc.): if AuthRef is nonempty then it is expected that the referenced + Secret exists and *all* authentication credentials are correctly set in the referenced + Secret (username/password or shared secret). Mixing of credential sources + (inline/external) is not supported. properties: group: default: "" - description: Group is the group of the referent. For example, - "gateway.networking.k8s.io". When unspecified or empty string, - core API group is inferred. + description: |- + Group is the group of the referent. For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API group is inferred. maxLength: 253 pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string @@ -217,12 +237,18 @@ spec: minLength: 1 type: string namespace: - description: "Namespace is the namespace of the referenced object. - When unspecified, the local namespace is inferred. \n Note that - when a namespace different than the local namespace is specified, - a ReferenceGrant object is required in the referent namespace - to allow that namespace's owner to accept the reference. See - the ReferenceGrant documentation for details. \n Support: Core" + description: |- + Namespace is the namespace of the referenced object. When unspecified, the local + namespace is inferred. + + + Note that when a namespace different than the local namespace is specified, + a ReferenceGrant object is required in the referent namespace to allow that + namespace's owner to accept the reference. See the ReferenceGrant + documentation for details. + + + Support: Core maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ @@ -238,31 +264,33 @@ spec: type: string dataplane: default: default - description: Dataplane defines the TURN server to set up for the STUNner - Gateways using this GatewayConfig. Can be used to select the stunnerd - image repo and version or deploy into the host-network namespace. + description: |- + Dataplane defines the TURN server to set up for the STUNner Gateways using this + GatewayConfig. Can be used to select the stunnerd image repo and version or deploy into + the host-network namespace. type: string healthCheckEndpoint: - description: HealthCheckEndpoint is the URI of the form `http://address:port` - exposed for external HTTP health-checking. A liveness probe responder - will be exposed on path `/live` and readiness probe on path `/ready`. - The scheme (`http://`) is mandatory, default is to enable health-checking - at "http://0.0.0.0:8086". + description: |- + HealthCheckEndpoint is the URI of the form `http://address:port` exposed for external + HTTP health-checking. A liveness probe responder will be exposed on path `/live` and + readiness probe on path `/ready`. The scheme (`http://`) is mandatory, default is to + enable health-checking at "http://0.0.0.0:8086". type: string loadBalancerServiceAnnotations: additionalProperties: type: string - description: "LoadBalancerServiceAnnotations is a list of annotations - that will go into the LoadBalancer services created automatically - by the operator to wrap Gateways. \n NOTE: removing annotations - from a GatewayConfig will not result in the removal of the corresponding - annotations from the LoadBalancer service, in order to prevent the - accidental removal of an annotation installed there by Kubernetes - or the cloud provider. If you really want to remove an annotation, - do this manually or simply remove all Gateways (which will remove - the corresponding LoadBalancer services), update the GatewayConfig - and then recreate the Gateways, so that the newly created LoadBalancer - services will contain the required annotations." + description: |- + LoadBalancerServiceAnnotations is a list of annotations that will go into the + LoadBalancer services created automatically by the operator to wrap Gateways. + + + NOTE: removing annotations from a GatewayConfig will not result in the removal of the + corresponding annotations from the LoadBalancer service, in order to prevent the + accidental removal of an annotation installed there by Kubernetes or the cloud + provider. If you really want to remove an annotation, do this manually or simply remove + all Gateways (which will remove the corresponding LoadBalancer services), update the + GatewayConfig and then recreate the Gateways, so that the newly created LoadBalancer + services will contain the required annotations. type: object logLevel: description: LogLevel specifies the default loglevel for the STUNner @@ -274,9 +302,10 @@ spec: format: int32 type: integer metricsEndpoint: - description: MetricsEndpoint is the URI in the form `http://address:port/path` - exposed for metric scraping (Prometheus). The scheme (`http://`) - is mandatory. Default is to expose no metric endpoint. + description: |- + MetricsEndpoint is the URI in the form `http://address:port/path` exposed for metric + scraping (Prometheus). The scheme (`http://`) is mandatory. Default is to expose no + metric endpoint. type: string minPort: description: MinRelayPort is the smallest relay port assigned for @@ -290,11 +319,13 @@ spec: type: string realm: default: stunner.l7mp.io - description: "Realm defines the STUN/TURN authentication realm to - be used for clients toauthenticate with STUNner. \n The realm must - consist of lower case alphanumeric characters or '-', and must start - and end with an alphanumeric character. No other punctuation is - allowed." + description: |- + Realm defines the STUN/TURN authentication realm to be used for clients toauthenticate + with STUNner. + + + The realm must consist of lower case alphanumeric characters or '-', and must start and + end with an alphanumeric character. No other punctuation is allowed. pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string sharedSecret: @@ -303,8 +334,9 @@ spec: type: string stunnerConfig: default: stunnerd-config - description: StunnerConfig specifies the name of the ConfigMap into - which the operator renders the stunnerd configfile. + description: |- + StunnerConfig specifies the name of the ConfigMap into which the operator renders the + stunnerd configfile. maxLength: 64 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ type: string @@ -323,7 +355,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.13.0 + controller-gen.kubebuilder.io/version: v0.14.0 name: staticservices.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -341,20 +373,25 @@ spec: - name: v1 schema: openAPIV3Schema: - description: StaticService is a set of static IP address prefixes STUNner - allows access to via a UDPRoute (or TCPRoute in the future). In contrast - to Kubernetes Services, StaticServices expose all ports on the given IPs. - See also https://github.com/kubernetes/enhancements/pull/2611. + description: |- + StaticService is a set of static IP address prefixes STUNner allows access to via a UDPRoute (or + TCPRoute in the future). In contrast to Kubernetes Services, StaticServices expose all ports on + the given IPs. See also https://github.com/kubernetes/enhancements/pull/2611. properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -376,20 +413,25 @@ spec: - name: v1alpha1 schema: openAPIV3Schema: - description: StaticService is a set of static IP address prefixes STUNner - allows access to via a Route. The purpose is to allow a Service-like CRD - containing a set of static IP address prefixes to be set as the backend - of a UDPRoute (or TCPRoute). + description: |- + StaticService is a set of static IP address prefixes STUNner allows access to via a Route. The + purpose is to allow a Service-like CRD containing a set of static IP address prefixes to be set + as the backend of a UDPRoute (or TCPRoute). properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -403,37 +445,45 @@ spec: description: ServicePort contains information on service's port. properties: appProtocol: - description: "The application protocol for this port. This is - used as a hint for implementations to offer richer behavior - for protocols that they understand. This field follows standard - Kubernetes label syntax. Valid values are either: \n * Un-prefixed - protocol names - reserved for IANA standard service names - (as per RFC-6335 and https://www.iana.org/assignments/service-names). - \n * Kubernetes-defined prefixed names: * 'kubernetes.io/h2c' - - HTTP/2 over cleartext as described in https://www.rfc-editor.org/rfc/rfc7540 - * 'kubernetes.io/ws' - WebSocket over cleartext as described - in https://www.rfc-editor.org/rfc/rfc6455 * 'kubernetes.io/wss' - - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455 - \n * Other protocols should use implementation-defined prefixed - names such as mycompany.com/my-custom-protocol." + description: |- + The application protocol for this port. + This is used as a hint for implementations to offer richer behavior for protocols that they understand. + This field follows standard Kubernetes label syntax. + Valid values are either: + + + * Un-prefixed protocol names - reserved for IANA standard service names (as per + RFC-6335 and https://www.iana.org/assignments/service-names). + + + * Kubernetes-defined prefixed names: + * 'kubernetes.io/h2c' - HTTP/2 prior knowledge over cleartext as described in https://www.rfc-editor.org/rfc/rfc9113.html#name-starting-http-2-with-prior- + * 'kubernetes.io/ws' - WebSocket over cleartext as described in https://www.rfc-editor.org/rfc/rfc6455 + * 'kubernetes.io/wss' - WebSocket over TLS as described in https://www.rfc-editor.org/rfc/rfc6455 + + + * Other protocols should use implementation-defined prefixed names such as + mycompany.com/my-custom-protocol. type: string name: - description: The name of this port within the service. This - must be a DNS_LABEL. All ports within a ServiceSpec must have - unique names. When considering the endpoints for a Service, - this must match the 'name' field in the EndpointPort. Optional - if only one ServicePort is defined on this service. + description: |- + The name of this port within the service. This must be a DNS_LABEL. + All ports within a ServiceSpec must have unique names. When considering + the endpoints for a Service, this must match the 'name' field in the + EndpointPort. + Optional if only one ServicePort is defined on this service. type: string nodePort: - description: 'The port on each node on which this service is - exposed when type is NodePort or LoadBalancer. Usually assigned - by the system. If a value is specified, in-range, and not - in use it will be used, otherwise the operation will fail. If - not specified, a port will be allocated if this Service requires - one. If this field is specified when creating a Service which - does not need it, creation will fail. This field will be wiped - when updating a Service to no longer need it (e.g. changing - type from NodePort to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport' + description: |- + The port on each node on which this service is exposed when type is + NodePort or LoadBalancer. Usually assigned by the system. If a value is + specified, in-range, and not in use it will be used, otherwise the + operation will fail. If not specified, a port will be allocated if this + Service requires one. If this field is specified when creating a + Service which does not need it, creation will fail. This field will be + wiped when updating a Service to no longer need it (e.g. changing type + from NodePort to ClusterIP). + More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport format: int32 type: integer port: @@ -442,21 +492,23 @@ spec: type: integer protocol: default: TCP - description: The IP protocol for this port. Supports "TCP", - "UDP", and "SCTP". Default is TCP. + description: |- + The IP protocol for this port. Supports "TCP", "UDP", and "SCTP". + Default is TCP. type: string targetPort: anyOf: - type: integer - type: string - description: 'Number or name of the port to access on the pods - targeted by the service. Number must be in the range 1 to - 65535. Name must be an IANA_SVC_NAME. If this is a string, - it will be looked up as a named port in the target Pod''s - container ports. If this is not specified, the value of the - ''port'' field is used (an identity map). This field is ignored - for services with clusterIP=None, and should be omitted or - set equal to the ''port'' field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service' + description: |- + Number or name of the port to access on the pods targeted by the service. + Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. + If this is a string, it will be looked up as a named port in the + target Pod's container ports. If this is not specified, the value + of the 'port' field is used (an identity map). + This field is ignored for services with clusterIP=None, and should be + omitted or set equal to the 'port' field. + More info: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service x-kubernetes-int-or-string: true required: - port @@ -483,7 +535,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.13.0 + controller-gen.kubebuilder.io/version: v0.14.0 name: dataplanes.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -501,19 +553,25 @@ spec: - name: v1 schema: openAPIV3Schema: - description: Dataplane is a collection of configuration parameters that can - be used for spawning a `stunnerd` instance for a Gateway. Labels and annotations - on the Dataplane object will be copied verbatim into the target Deployment. + description: |- + Dataplane is a collection of configuration parameters that can be used for spawning a `stunnerd` + instance for a Gateway. Labels and annotations on the Dataplane object will be copied verbatim + into the target Deployment. properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -528,22 +586,20 @@ spec: pod. properties: preferredDuringSchedulingIgnoredDuringExecution: - description: The scheduler will prefer to schedule pods to - nodes that satisfy the affinity expressions specified by - this field, but it may choose a node that violates one or - more of the expressions. The node that is most preferred - is the one with the greatest sum of weights, i.e. for each - node that meets all of the scheduling requirements (resource - request, requiredDuringScheduling affinity expressions, - etc.), compute a sum by iterating through the elements of - this field and adding "weight" to the sum if the node matches - the corresponding matchExpressions; the node(s) with the - highest sum are the most preferred. + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node matches the corresponding matchExpressions; the + node(s) with the highest sum are the most preferred. items: - description: An empty preferred scheduling term matches - all objects with implicit weight 0 (i.e. it's a no-op). - A null preferred scheduling term matches no objects (i.e. - is also a no-op). + description: |- + An empty preferred scheduling term matches all objects with implicit weight 0 + (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). properties: preference: description: A node selector term, associated with the @@ -553,30 +609,26 @@ spec: description: A list of node selector requirements by node's labels. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -589,30 +641,26 @@ spec: description: A list of node selector requirements by node's fields. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -634,50 +682,46 @@ spec: type: object type: array requiredDuringSchedulingIgnoredDuringExecution: - description: If the affinity requirements specified by this - field are not met at scheduling time, the pod will not be - scheduled onto the node. If the affinity requirements specified - by this field cease to be met at some point during pod execution - (e.g. due to an update), the system may or may not try to - eventually evict the pod from its node. + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to an update), the system + may or may not try to eventually evict the pod from its node. properties: nodeSelectorTerms: description: Required. A list of node selector terms. The terms are ORed. items: - description: A null or empty node selector term matches - no objects. The requirements of them are ANDed. The - TopologySelectorTerm type implements a subset of the - NodeSelectorTerm. + description: |- + A null or empty node selector term matches no objects. The requirements of + them are ANDed. + The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. properties: matchExpressions: description: A list of node selector requirements by node's labels. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -690,30 +734,26 @@ spec: description: A list of node selector requirements by node's fields. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -735,16 +775,15 @@ spec: this pod in the same node, zone, etc. as some other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: - description: The scheduler will prefer to schedule pods to - nodes that satisfy the affinity expressions specified by - this field, but it may choose a node that violates one or - more of the expressions. The node that is most preferred - is the one with the greatest sum of weights, i.e. for each - node that meets all of the scheduling requirements (resource - request, requiredDuringScheduling affinity expressions, - etc.), compute a sum by iterating through the elements of - this field and adding "weight" to the sum if the node has - pods which matches the corresponding podAffinityTerm; the + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm @@ -755,36 +794,33 @@ spec: with the corresponding weight. properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -797,51 +833,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied - to the union of the namespaces selected by this - field and the ones listed in the namespaces field. - null selector and null or empty namespaces list - means "this pod's namespace". An empty selector - ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -854,40 +913,37 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list - of namespace names that the term applies to. The - term is applied to the union of the namespaces - listed in this field and the ones selected by - namespaceSelector. null or empty namespaces list - and null namespaceSelector means "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods - matching the labelSelector in the specified namespaces, - where co-located is defined as running on a node - whose value of the label with key topologyKey - matches that of any node on which any of the selected - pods is running. Empty topologyKey is not allowed. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: - description: weight associated with matching the corresponding - podAffinityTerm, in the range 1-100. + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. format: int32 type: integer required: @@ -896,52 +952,51 @@ spec: type: object type: array requiredDuringSchedulingIgnoredDuringExecution: - description: If the affinity requirements specified by this - field are not met at scheduling time, the pod will not be - scheduled onto the node. If the affinity requirements specified - by this field cease to be met at some point during pod execution - (e.g. due to a pod label update), the system may or may - not try to eventually evict the pod from its node. When - there are multiple elements, the lists of nodes corresponding - to each podAffinityTerm are intersected, i.e. all terms - must be satisfied. + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. items: - description: Defines a set of pods (namely those matching - the labelSelector relative to the given namespace(s)) - that this pod should be co-located (affinity) or not co-located - (anti-affinity) with, where co-located is defined as running - on a node whose value of the label with key - matches that of any node on which a pod of the set of - pods is running + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -953,47 +1008,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied to the - union of the namespaces selected by this field and - the ones listed in the namespaces field. null selector - and null or empty namespaces list means "this pod's - namespace". An empty selector ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -1005,32 +1087,28 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list of namespace - names that the term applies to. The term is applied - to the union of the namespaces listed in this field - and the ones selected by namespaceSelector. null or - empty namespaces list and null namespaceSelector means - "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods matching - the labelSelector in the specified namespaces, where - co-located is defined as running on a node whose value - of the label with key topologyKey matches that of - any node on which any of the selected pods is running. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. Empty topologyKey is not allowed. type: string required: @@ -1044,16 +1122,15 @@ spec: other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: - description: The scheduler will prefer to schedule pods to - nodes that satisfy the anti-affinity expressions specified - by this field, but it may choose a node that violates one - or more of the expressions. The node that is most preferred - is the one with the greatest sum of weights, i.e. for each - node that meets all of the scheduling requirements (resource - request, requiredDuringScheduling anti-affinity expressions, - etc.), compute a sum by iterating through the elements of - this field and adding "weight" to the sum if the node has - pods which matches the corresponding podAffinityTerm; the + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the anti-affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm @@ -1064,36 +1141,33 @@ spec: with the corresponding weight. properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -1106,51 +1180,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied - to the union of the namespaces selected by this - field and the ones listed in the namespaces field. - null selector and null or empty namespaces list - means "this pod's namespace". An empty selector - ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -1163,40 +1260,37 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list - of namespace names that the term applies to. The - term is applied to the union of the namespaces - listed in this field and the ones selected by - namespaceSelector. null or empty namespaces list - and null namespaceSelector means "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods - matching the labelSelector in the specified namespaces, - where co-located is defined as running on a node - whose value of the label with key topologyKey - matches that of any node on which any of the selected - pods is running. Empty topologyKey is not allowed. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: - description: weight associated with matching the corresponding - podAffinityTerm, in the range 1-100. + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. format: int32 type: integer required: @@ -1205,52 +1299,51 @@ spec: type: object type: array requiredDuringSchedulingIgnoredDuringExecution: - description: If the anti-affinity requirements specified by - this field are not met at scheduling time, the pod will - not be scheduled onto the node. If the anti-affinity requirements - specified by this field cease to be met at some point during - pod execution (e.g. due to a pod label update), the system - may or may not try to eventually evict the pod from its - node. When there are multiple elements, the lists of nodes - corresponding to each podAffinityTerm are intersected, i.e. - all terms must be satisfied. + description: |- + If the anti-affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the anti-affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. items: - description: Defines a set of pods (namely those matching - the labelSelector relative to the given namespace(s)) - that this pod should be co-located (affinity) or not co-located - (anti-affinity) with, where co-located is defined as running - on a node whose value of the label with key - matches that of any node on which a pod of the set of - pods is running + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -1262,47 +1355,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied to the - union of the namespaces selected by this field and - the ones listed in the namespaces field. null selector - and null or empty namespaces list means "this pod's - namespace". An empty selector ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -1314,32 +1434,28 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list of namespace - names that the term applies to. The term is applied - to the union of the namespaces listed in this field - and the ones selected by namespaceSelector. null or - empty namespaces list and null namespaceSelector means - "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods matching - the labelSelector in the specified namespaces, where - co-located is defined as running on a node whose value - of the label with key topologyKey matches that of - any node on which any of the selected pods is running. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. Empty topologyKey is not allowed. type: string required: @@ -1358,15 +1474,184 @@ spec: items: type: string type: array + containerSecurityContext: + description: |- + ContainerSecurityContext holds container-level security attributes specifically for the + stunnerd container. + properties: + allowPrivilegeEscalation: + description: |- + AllowPrivilegeEscalation controls whether a process can gain more + privileges than its parent process. This bool directly controls if + the no_new_privs flag will be set on the container process. + AllowPrivilegeEscalation is true always when the container is: + 1) run as Privileged + 2) has CAP_SYS_ADMIN + Note that this field cannot be set when spec.os.name is windows. + type: boolean + capabilities: + description: |- + The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by the container runtime. + Note that this field cannot be set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities type + type: string + type: array + type: object + privileged: + description: |- + Run container in privileged mode. + Processes in privileged containers are essentially equivalent to root on the host. + Defaults to false. + Note that this field cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: |- + procMount denotes the type of proc mount to use for the containers. + The default is DefaultProcMount which uses the container runtime defaults for + readonly paths and masked paths. + This requires the ProcMountType feature flag to be enabled. + Note that this field cannot be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: |- + Whether this container has a read-only root filesystem. + Default is false. + Note that this field cannot be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: boolean + runAsUser: + description: |- + The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: |- + The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies to + the container. + type: string + role: + description: Role is a SELinux role label that applies to + the container. + type: string + type: + description: Type is a SELinux type label that applies to + the container. + type: string + user: + description: User is a SELinux user label that applies to + the container. + type: string + type: object + seccompProfile: + description: |- + The seccomp options to use by this container. If seccomp options are + provided at both the pod & container level, the container options + override the pod options. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. + type: string + type: + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. + type: string + required: + - type + type: object + windowsOptions: + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options from the PodSecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the GMSA + credential spec to use. + type: string + hostProcess: + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. + type: boolean + runAsUserName: + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: string + type: object + type: object disableHealthCheck: - description: 'Disable health-checking. Default is to enable HTTP health-checks - on port 8086: a liveness probe responder will be exposed on path - `/live` and readiness probe on path `/ready`.' + description: |- + Disable health-checking. Default is to enable HTTP health-checks on port 8086: a + liveness probe responder will be exposed on path `/live` and readiness probe on path + `/ready`. type: boolean enableMetricsEndpoint: - description: EnableMetricsEnpoint can be used to enable metrics scraping - (Prometheus). If enabled, a metrics endpoint will available at http://0.0.0.0:8080 - at all dataplane pods. Default is no metrics collection. + description: |- + EnableMetricsEnpoint can be used to enable metrics scraping (Prometheus). If enabled, a + metrics endpoint will be available at http://0.0.0.0:8080 at all dataplane pods. Default + is no metrics collection. type: boolean env: description: List of environment variables to set in the stunnerd @@ -1379,15 +1664,16 @@ spec: description: Name of the environment variable. Must be a C_IDENTIFIER. type: string value: - description: 'Variable references $(VAR_NAME) are expanded using - the previously defined environment variables in the container - and any service environment variables. If a variable cannot - be resolved, the reference in the input string will be unchanged. - Double $$ are reduced to a single $, which allows for escaping - the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the - string literal "$(VAR_NAME)". Escaped references will never - be expanded, regardless of whether the variable exists or - not. Defaults to "".' + description: |- + Variable references $(VAR_NAME) are expanded + using the previously defined environment variables in the container and + any service environment variables. If a variable cannot be resolved, + the reference in the input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. + "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". + Escaped references will never be expanded, regardless of whether the variable + exists or not. + Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot @@ -1400,8 +1686,10 @@ spec: description: The key to select. type: string name: - description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - TODO: Add other useful fields. apiVersion, kind, uid?' + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? type: string optional: description: Specify whether the ConfigMap or its key @@ -1412,10 +1700,9 @@ spec: type: object x-kubernetes-map-type: atomic fieldRef: - description: 'Selects a field of the pod: supports metadata.name, - metadata.namespace, `metadata.labels['''']`, `metadata.annotations['''']`, - spec.nodeName, spec.serviceAccountName, status.hostIP, - status.podIP, status.podIPs.' + description: |- + Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, + spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is @@ -1430,10 +1717,9 @@ spec: type: object x-kubernetes-map-type: atomic resourceFieldRef: - description: 'Selects a resource of the container: only - resources limits and requests (limits.cpu, limits.memory, - limits.ephemeral-storage, requests.cpu, requests.memory - and requests.ephemeral-storage) are currently supported.' + description: |- + Selects a resource of the container: only resources limits and requests + (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, @@ -1462,8 +1748,10 @@ spec: be a valid secret key. type: string name: - description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - TODO: Add other useful fields. apiVersion, kind, uid?' + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? type: string optional: description: Specify whether the Secret or its key must @@ -1488,8 +1776,10 @@ spec: description: The ConfigMap to select from properties: name: - description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - TODO: Add other useful fields. apiVersion, kind, uid?' + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? type: string optional: description: Specify whether the ConfigMap must be defined @@ -1504,8 +1794,10 @@ spec: description: The Secret to select from properties: name: - description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - TODO: Add other useful fields. apiVersion, kind, uid?' + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? type: string optional: description: Specify whether the Secret must be defined @@ -1515,9 +1807,9 @@ spec: type: object type: array hostNetwork: - description: Host networking requested for the stunnerd pod to use - the host's network namespace. Can be used to implement public TURN - servers with Kubernetes. Defaults to false. + description: |- + Host networking requested for the stunnerd pod to use the host's network namespace. + Can be used to implement public TURN servers with Kubernetes. Defaults to false. type: boolean image: description: Container image name. @@ -1525,27 +1817,56 @@ spec: imagePullPolicy: description: Image pull policy. One of Always, Never, IfNotPresent. type: string + imagePullSecrets: + description: |- + ImagePullSecrets is an optional list of references to secrets to use for pulling the + stunnerd image. Note that the referenced secrets are not watched by the operator, so + modifications will in effect only for newly created pods. Also note that the Secret is + always searched in the same namespace as the Gateway, which allows to use separate pull + secrets per each namespace. + items: + description: |- + LocalObjectReference contains enough information to let you locate the + referenced object inside the same namespace. + properties: + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? + type: string + type: object + x-kubernetes-map-type: atomic + type: array replicas: - description: Number of desired pods. This is a pointer to distinguish - between explicit zero and not specified. Defaults to 1. + description: |- + Number of desired pods. If empty or set to 1, use whatever is in the target Deployment. + Otherwise, enforce this setting, overwiting whatever is set in the Deployment (this may + block autoscaling the dataplane though). Defaults to 1. format: int32 type: integer resources: description: Resources required by stunnerd. properties: claims: - description: "Claims lists the names of resources, defined in - spec.resourceClaims, that are used by this container. \n This - is an alpha field and requires enabling the DynamicResourceAllocation - feature gate. \n This field is immutable. It can only be set - for containers." + description: |- + Claims lists the names of resources, defined in spec.resourceClaims, + that are used by this container. + + + This is an alpha field and requires enabling the + DynamicResourceAllocation feature gate. + + + This field is immutable. It can only be set for containers. items: description: ResourceClaim references one entry in PodSpec.ResourceClaims. properties: name: - description: Name must match the name of one entry in pod.spec.resourceClaims - of the Pod where this field is used. It makes that resource - available inside a container. + description: |- + Name must match the name of one entry in pod.spec.resourceClaims of + the Pod where this field is used. It makes that resource available + inside a container. type: string required: - name @@ -1561,8 +1882,9 @@ spec: - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true - description: 'Limits describes the maximum amount of compute resources - allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + description: |- + Limits describes the maximum amount of compute resources allowed. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object requests: additionalProperties: @@ -1571,11 +1893,11 @@ spec: - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true - description: 'Requests describes the minimum amount of compute - resources required. If Requests is omitted for a container, - it defaults to Limits if that is explicitly specified, otherwise - to an implementation-defined value. Requests cannot exceed Limits. - More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + description: |- + Requests describes the minimum amount of compute resources required. + If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. Requests cannot exceed Limits. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object type: object securityContext: @@ -1583,59 +1905,67 @@ spec: common container settings. properties: fsGroup: - description: "A special supplemental group that applies to all - containers in a pod. Some volume types allow the Kubelet to - change the ownership of that volume to be owned by the pod: - \n 1. The owning GID will be the FSGroup 2. The setgid bit is - set (new files created in the volume will be owned by FSGroup) - 3. The permission bits are OR'd with rw-rw---- \n If unset, - the Kubelet will not modify the ownership and permissions of - any volume. Note that this field cannot be set when spec.os.name - is windows." + description: |- + A special supplemental group that applies to all containers in a pod. + Some volume types allow the Kubelet to change the ownership of that volume + to be owned by the pod: + + + 1. The owning GID will be the FSGroup + 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- + + + If unset, the Kubelet will not modify the ownership and permissions of any volume. + Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer fsGroupChangePolicy: - description: 'fsGroupChangePolicy defines behavior of changing - ownership and permission of the volume before being exposed - inside Pod. This field will only apply to volume types which - support fsGroup based ownership(and permissions). It will have - no effect on ephemeral volume types such as: secret, configmaps - and emptydir. Valid values are "OnRootMismatch" and "Always". - If not specified, "Always" is used. Note that this field cannot - be set when spec.os.name is windows.' + description: |- + fsGroupChangePolicy defines behavior of changing ownership and permission of the volume + before being exposed inside Pod. This field will only apply to + volume types which support fsGroup based ownership(and permissions). + It will have no effect on ephemeral volume types such as: secret, configmaps + and emptydir. + Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. + Note that this field cannot be set when spec.os.name is windows. type: string runAsGroup: - description: The GID to run the entrypoint of the container process. - Uses runtime default if unset. May also be set in SecurityContext. If - set in both SecurityContext and PodSecurityContext, the value - specified in SecurityContext takes precedence for that container. + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: - description: Indicates that the container must run as a non-root - user. If true, the Kubelet will validate the image at runtime - to ensure that it does not run as UID 0 (root) and fail to start - the container if it does. If unset or false, no such validation - will be performed. May also be set in SecurityContext. If set - in both SecurityContext and PodSecurityContext, the value specified - in SecurityContext takes precedence. + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: - description: The UID to run the entrypoint of the container process. + description: |- + The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. - May also be set in SecurityContext. If set in both SecurityContext - and PodSecurityContext, the value specified in SecurityContext - takes precedence for that container. Note that this field cannot - be set when spec.os.name is windows. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. + Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxOptions: - description: The SELinux context to be applied to all containers. - If unspecified, the container runtime will allocate a random - SELinux context for each container. May also be set in SecurityContext. If - set in both SecurityContext and PodSecurityContext, the value - specified in SecurityContext takes precedence for that container. + description: |- + The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in SecurityContext. If set in + both SecurityContext and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. properties: level: @@ -1656,47 +1986,48 @@ spec: type: string type: object seccompProfile: - description: The seccomp options to use by the containers in this - pod. Note that this field cannot be set when spec.os.name is - windows. + description: |- + The seccomp options to use by the containers in this pod. + Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: - description: localhostProfile indicates a profile defined - in a file on the node should be used. The profile must be - preconfigured on the node to work. Must be a descending - path, relative to the kubelet's configured seccomp profile - location. Must be set if type is "Localhost". Must NOT be - set for any other type. + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: - description: "type indicates which kind of seccomp profile - will be applied. Valid options are: \n Localhost - a profile - defined in a file on the node should be used. RuntimeDefault - - the container runtime default profile should be used. - Unconfined - no profile should be applied." + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. type: string required: - type type: object supplementalGroups: - description: A list of groups applied to the first process run - in each container, in addition to the container's primary GID, - the fsGroup (if specified), and group memberships defined in - the container image for the uid of the container process. If - unspecified, no additional groups are added to any container. - Note that group memberships defined in the container image for - the uid of the container process are still effective, even if - they are not included in this list. Note that this field cannot - be set when spec.os.name is windows. + description: |- + A list of groups applied to the first process run in each container, in addition + to the container's primary GID, the fsGroup (if specified), and group memberships + defined in the container image for the uid of the container process. If unspecified, + no additional groups are added to any container. Note that group memberships + defined in the container image for the uid of the container process are still effective, + even if they are not included in this list. + Note that this field cannot be set when spec.os.name is windows. items: format: int64 type: integer type: array sysctls: - description: Sysctls hold a list of namespaced sysctls used for - the pod. Pods with unsupported sysctls (by the container runtime) - might fail to launch. Note that this field cannot be set when - spec.os.name is windows. + description: |- + Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported + sysctls (by the container runtime) might fail to launch. + Note that this field cannot be set when spec.os.name is windows. items: description: Sysctl defines a kernel parameter to be set properties: @@ -1712,36 +2043,35 @@ spec: type: object type: array windowsOptions: - description: The Windows specific settings applied to all containers. - If unspecified, the options within a container's SecurityContext - will be used. If set in both SecurityContext and PodSecurityContext, - the value specified in SecurityContext takes precedence. Note - that this field cannot be set when spec.os.name is linux. + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: - description: GMSACredentialSpec is where the GMSA admission - webhook (https://github.com/kubernetes-sigs/windows-gmsa) - inlines the contents of the GMSA credential spec named by - the GMSACredentialSpecName field. + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: - description: HostProcess determines if a container should - be run as a 'Host Process' container. All of a Pod's containers - must have the same effective HostProcess value (it is not - allowed to have a mix of HostProcess containers and non-HostProcess - containers). In addition, if HostProcess is true then HostNetwork - must also be set to true. + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: - description: The UserName in Windows to run the entrypoint - of the container process. Defaults to the user specified - in image metadata if unspecified. May also be set in PodSecurityContext. - If set in both SecurityContext and PodSecurityContext, the - value specified in SecurityContext takes precedence. + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object @@ -1753,43 +2083,225 @@ spec: tolerations: description: If specified, the pod's tolerations. items: - description: The pod this Toleration is attached to tolerates any - taint that matches the triple using the matching - operator . + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . properties: effect: - description: Effect indicates the taint effect to match. Empty - means match all taint effects. When specified, allowed values - are NoSchedule, PreferNoSchedule and NoExecute. + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. type: string key: - description: Key is the taint key that the toleration applies - to. Empty means match all taint keys. If the key is empty, - operator must be Exists; this combination means to match all - values and all keys. + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. type: string operator: - description: Operator represents a key's relationship to the - value. Valid operators are Exists and Equal. Defaults to Equal. - Exists is equivalent to wildcard for value, so that a pod - can tolerate all taints of a particular category. + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists and Equal. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. type: string tolerationSeconds: - description: TolerationSeconds represents the period of time - the toleration (which must be of effect NoExecute, otherwise - this field is ignored) tolerates the taint. By default, it - is not set, which means tolerate the taint forever (do not - evict). Zero and negative values will be treated as 0 (evict - immediately) by the system. + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. format: int64 type: integer value: - description: Value is the taint value the toleration matches - to. If the operator is Exists, the value should be empty, - otherwise just a regular string. + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. type: string type: object type: array + topologySpreadConstraints: + description: |- + TopologySpreadConstraints describes how stunnerd pods ought to spread across topology + domains. + items: + description: TopologySpreadConstraint specifies how to spread matching + pods among the given topology. + properties: + labelSelector: + description: |- + LabelSelector is used to find matching pods. + Pods that match this label selector are counted to determine the number of pods + in their corresponding topology domain. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select the pods over which + spreading will be calculated. The keys are used to lookup values from the + incoming pod labels, those key-value labels are ANDed with labelSelector + to select the group of existing pods over which spreading will be calculated + for the incoming pod. The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + MatchLabelKeys cannot be set when LabelSelector isn't set. + Keys that don't exist in the incoming pod labels will + be ignored. A null or empty list means only match against labelSelector. + + + This is a beta field and requires the MatchLabelKeysInPodTopologySpread feature gate to be enabled (enabled by default). + items: + type: string + type: array + x-kubernetes-list-type: atomic + maxSkew: + description: |- + MaxSkew describes the degree to which pods may be unevenly distributed. + When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference + between the number of matching pods in the target topology and the global minimum. + The global minimum is the minimum number of matching pods in an eligible domain + or zero if the number of eligible domains is less than MinDomains. + For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same + labelSelector spread as 2/2/1: + In this case, the global minimum is 1. + | zone1 | zone2 | zone3 | + | P P | P P | P | + - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; + scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) + violate MaxSkew(1). + - if MaxSkew is 2, incoming pod can be scheduled onto any zone. + When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence + to topologies that satisfy it. + It's a required field. Default value is 1 and 0 is not allowed. + format: int32 + type: integer + minDomains: + description: |- + MinDomains indicates a minimum number of eligible domains. + When the number of eligible domains with matching topology keys is less than minDomains, + Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. + And when the number of eligible domains with matching topology keys equals or greater than minDomains, + this value has no effect on scheduling. + As a result, when the number of eligible domains is less than minDomains, + scheduler won't schedule more than maxSkew Pods to those domains. + If value is nil, the constraint behaves as if MinDomains is equal to 1. + Valid values are integers greater than 0. + When value is not nil, WhenUnsatisfiable must be DoNotSchedule. + + + For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same + labelSelector spread as 2/2/2: + | zone1 | zone2 | zone3 | + | P P | P P | P P | + The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. + In this situation, new pod with the same labelSelector cannot be scheduled, + because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, + it will violate MaxSkew. + + + This is a beta field and requires the MinDomainsInPodTopologySpread feature gate to be enabled (enabled by default). + format: int32 + type: integer + nodeAffinityPolicy: + description: |- + NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector + when calculating pod topology spread skew. Options are: + - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. + - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations. + + + If this value is nil, the behavior is equivalent to the Honor policy. + This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag. + type: string + nodeTaintsPolicy: + description: |- + NodeTaintsPolicy indicates how we will treat node taints when calculating + pod topology spread skew. Options are: + - Honor: nodes without taints, along with tainted nodes for which the incoming pod + has a toleration, are included. + - Ignore: node taints are ignored. All nodes are included. + + + If this value is nil, the behavior is equivalent to the Ignore policy. + This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag. + type: string + topologyKey: + description: |- + TopologyKey is the key of node labels. Nodes that have a label with this key + and identical values are considered to be in the same topology. + We consider each as a "bucket", and try to put balanced number + of pods into each bucket. + We define a domain as a particular instance of a topology. + Also, we define an eligible domain as a domain whose nodes meet the requirements of + nodeAffinityPolicy and nodeTaintsPolicy. + e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. + And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. + It's a required field. + type: string + whenUnsatisfiable: + description: |- + WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy + the spread constraint. + - DoNotSchedule (default) tells the scheduler not to schedule it. + - ScheduleAnyway tells the scheduler to schedule the pod in any location, + but giving higher precedence to topologies that would help reduce the + skew. + A constraint is considered "Unsatisfiable" for an incoming pod + if and only if every possible node assignment for that pod would violate + "MaxSkew" on some topology. + For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same + labelSelector spread as 3/1/1: + | zone1 | zone2 | zone3 | + | P P P | P | P | + If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled + to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies + MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler + won't make it *more* imbalanced. + It's a required field. + type: string + required: + - maxSkew + - topologyKey + - whenUnsatisfiable + type: object + type: array type: object type: object served: true @@ -1797,19 +2309,25 @@ spec: - name: v1alpha1 schema: openAPIV3Schema: - description: Dataplane is a collection of configuration parameters that can - be used for spawning a `stunnerd` instance for a Gateway. Labels and annotations - on the Dataplane object will be copied verbatim into the target Deployment. + description: |- + Dataplane is a collection of configuration parameters that can be used for spawning a `stunnerd` + instance for a Gateway. Labels and annotations on the Dataplane object will be copied verbatim + into the target Deployment. properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -1824,22 +2342,20 @@ spec: pod. properties: preferredDuringSchedulingIgnoredDuringExecution: - description: The scheduler will prefer to schedule pods to - nodes that satisfy the affinity expressions specified by - this field, but it may choose a node that violates one or - more of the expressions. The node that is most preferred - is the one with the greatest sum of weights, i.e. for each - node that meets all of the scheduling requirements (resource - request, requiredDuringScheduling affinity expressions, - etc.), compute a sum by iterating through the elements of - this field and adding "weight" to the sum if the node matches - the corresponding matchExpressions; the node(s) with the - highest sum are the most preferred. + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node matches the corresponding matchExpressions; the + node(s) with the highest sum are the most preferred. items: - description: An empty preferred scheduling term matches - all objects with implicit weight 0 (i.e. it's a no-op). - A null preferred scheduling term matches no objects (i.e. - is also a no-op). + description: |- + An empty preferred scheduling term matches all objects with implicit weight 0 + (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). properties: preference: description: A node selector term, associated with the @@ -1849,30 +2365,26 @@ spec: description: A list of node selector requirements by node's labels. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -1885,30 +2397,26 @@ spec: description: A list of node selector requirements by node's fields. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -1930,50 +2438,46 @@ spec: type: object type: array requiredDuringSchedulingIgnoredDuringExecution: - description: If the affinity requirements specified by this - field are not met at scheduling time, the pod will not be - scheduled onto the node. If the affinity requirements specified - by this field cease to be met at some point during pod execution - (e.g. due to an update), the system may or may not try to - eventually evict the pod from its node. + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to an update), the system + may or may not try to eventually evict the pod from its node. properties: nodeSelectorTerms: description: Required. A list of node selector terms. The terms are ORed. items: - description: A null or empty node selector term matches - no objects. The requirements of them are ANDed. The - TopologySelectorTerm type implements a subset of the - NodeSelectorTerm. + description: |- + A null or empty node selector term matches no objects. The requirements of + them are ANDed. + The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. properties: matchExpressions: description: A list of node selector requirements by node's labels. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -1986,30 +2490,26 @@ spec: description: A list of node selector requirements by node's fields. items: - description: A node selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. properties: key: description: The label key that the selector applies to. type: string operator: - description: Represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists, DoesNotExist. Gt, and - Lt. + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. type: string values: - description: An array of string values. If - the operator is In or NotIn, the values - array must be non-empty. If the operator - is Exists or DoesNotExist, the values array - must be empty. If the operator is Gt or - Lt, the values array must have a single - element, which will be interpreted as an - integer. This array is replaced during a - strategic merge patch. + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. items: type: string type: array @@ -2031,16 +2531,15 @@ spec: this pod in the same node, zone, etc. as some other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: - description: The scheduler will prefer to schedule pods to - nodes that satisfy the affinity expressions specified by - this field, but it may choose a node that violates one or - more of the expressions. The node that is most preferred - is the one with the greatest sum of weights, i.e. for each - node that meets all of the scheduling requirements (resource - request, requiredDuringScheduling affinity expressions, - etc.), compute a sum by iterating through the elements of - this field and adding "weight" to the sum if the node has - pods which matches the corresponding podAffinityTerm; the + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm @@ -2051,36 +2550,33 @@ spec: with the corresponding weight. properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -2093,51 +2589,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied - to the union of the namespaces selected by this - field and the ones listed in the namespaces field. - null selector and null or empty namespaces list - means "this pod's namespace". An empty selector - ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -2150,40 +2669,37 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list - of namespace names that the term applies to. The - term is applied to the union of the namespaces - listed in this field and the ones selected by - namespaceSelector. null or empty namespaces list - and null namespaceSelector means "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods - matching the labelSelector in the specified namespaces, - where co-located is defined as running on a node - whose value of the label with key topologyKey - matches that of any node on which any of the selected - pods is running. Empty topologyKey is not allowed. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: - description: weight associated with matching the corresponding - podAffinityTerm, in the range 1-100. + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. format: int32 type: integer required: @@ -2192,52 +2708,51 @@ spec: type: object type: array requiredDuringSchedulingIgnoredDuringExecution: - description: If the affinity requirements specified by this - field are not met at scheduling time, the pod will not be - scheduled onto the node. If the affinity requirements specified - by this field cease to be met at some point during pod execution - (e.g. due to a pod label update), the system may or may - not try to eventually evict the pod from its node. When - there are multiple elements, the lists of nodes corresponding - to each podAffinityTerm are intersected, i.e. all terms - must be satisfied. + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. items: - description: Defines a set of pods (namely those matching - the labelSelector relative to the given namespace(s)) - that this pod should be co-located (affinity) or not co-located - (anti-affinity) with, where co-located is defined as running - on a node whose value of the label with key - matches that of any node on which a pod of the set of - pods is running + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -2249,47 +2764,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied to the - union of the namespaces selected by this field and - the ones listed in the namespaces field. null selector - and null or empty namespaces list means "this pod's - namespace". An empty selector ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -2301,32 +2843,28 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list of namespace - names that the term applies to. The term is applied - to the union of the namespaces listed in this field - and the ones selected by namespaceSelector. null or - empty namespaces list and null namespaceSelector means - "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods matching - the labelSelector in the specified namespaces, where - co-located is defined as running on a node whose value - of the label with key topologyKey matches that of - any node on which any of the selected pods is running. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. Empty topologyKey is not allowed. type: string required: @@ -2340,16 +2878,15 @@ spec: other pod(s)). properties: preferredDuringSchedulingIgnoredDuringExecution: - description: The scheduler will prefer to schedule pods to - nodes that satisfy the anti-affinity expressions specified - by this field, but it may choose a node that violates one - or more of the expressions. The node that is most preferred - is the one with the greatest sum of weights, i.e. for each - node that meets all of the scheduling requirements (resource - request, requiredDuringScheduling anti-affinity expressions, - etc.), compute a sum by iterating through the elements of - this field and adding "weight" to the sum if the node has - pods which matches the corresponding podAffinityTerm; the + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the anti-affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. items: description: The weights of all of the matched WeightedPodAffinityTerm @@ -2360,36 +2897,33 @@ spec: with the corresponding weight. properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -2402,51 +2936,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied - to the union of the namespaces selected by this - field and the ones listed in the namespaces field. - null selector and null or empty namespaces list - means "this pod's namespace". An empty selector - ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement - is a selector that contains values, a key, - and an operator that relates the key and - values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and - DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. - If the operator is Exists or DoesNotExist, - the values array must be empty. This - array is replaced during a strategic + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic merge patch. items: type: string @@ -2459,40 +3016,37 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is - "In", and the values array contains only "value". - The requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list - of namespace names that the term applies to. The - term is applied to the union of the namespaces - listed in this field and the ones selected by - namespaceSelector. null or empty namespaces list - and null namespaceSelector means "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods - matching the labelSelector in the specified namespaces, - where co-located is defined as running on a node - whose value of the label with key topologyKey - matches that of any node on which any of the selected - pods is running. Empty topologyKey is not allowed. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. type: string required: - topologyKey type: object weight: - description: weight associated with matching the corresponding - podAffinityTerm, in the range 1-100. + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. format: int32 type: integer required: @@ -2501,52 +3055,51 @@ spec: type: object type: array requiredDuringSchedulingIgnoredDuringExecution: - description: If the anti-affinity requirements specified by - this field are not met at scheduling time, the pod will - not be scheduled onto the node. If the anti-affinity requirements - specified by this field cease to be met at some point during - pod execution (e.g. due to a pod label update), the system - may or may not try to eventually evict the pod from its - node. When there are multiple elements, the lists of nodes - corresponding to each podAffinityTerm are intersected, i.e. - all terms must be satisfied. + description: |- + If the anti-affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the anti-affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. items: - description: Defines a set of pods (namely those matching - the labelSelector relative to the given namespace(s)) - that this pod should be co-located (affinity) or not co-located - (anti-affinity) with, where co-located is defined as running - on a node whose value of the label with key - matches that of any node on which a pod of the set of - pods is running + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running properties: labelSelector: - description: A label query over a set of resources, - in this case pods. + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -2558,47 +3111,74 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + Also, MatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector. + Also, MismatchLabelKeys cannot be set when LabelSelector isn't set. + This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate. + items: + type: string + type: array + x-kubernetes-list-type: atomic namespaceSelector: - description: A label query over the set of namespaces - that the term applies to. The term is applied to the - union of the namespaces selected by this field and - the ones listed in the namespaces field. null selector - and null or empty namespaces list means "this pod's - namespace". An empty selector ({}) matches all namespaces. + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: - description: A label selector requirement is a - selector that contains values, a key, and an - operator that relates the key and values. + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: - description: operator represents a key's relationship - to a set of values. Valid operators are - In, NotIn, Exists and DoesNotExist. + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If the - operator is Exists or DoesNotExist, the - values array must be empty. This array is - replaced during a strategic merge patch. + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. items: type: string type: array @@ -2610,32 +3190,28 @@ spec: matchLabels: additionalProperties: type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". The - requirements are ANDed. + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic namespaces: - description: namespaces specifies a static list of namespace - names that the term applies to. The term is applied - to the union of the namespaces listed in this field - and the ones selected by namespaceSelector. null or - empty namespaces list and null namespaceSelector means - "this pod's namespace". + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". items: type: string type: array topologyKey: - description: This pod should be co-located (affinity) - or not co-located (anti-affinity) with the pods matching - the labelSelector in the specified namespaces, where - co-located is defined as running on a node whose value - of the label with key topologyKey matches that of - any node on which any of the selected pods is running. + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. Empty topologyKey is not allowed. type: string required: @@ -2665,15 +3241,16 @@ spec: description: Name of the environment variable. Must be a C_IDENTIFIER. type: string value: - description: 'Variable references $(VAR_NAME) are expanded using - the previously defined environment variables in the container - and any service environment variables. If a variable cannot - be resolved, the reference in the input string will be unchanged. - Double $$ are reduced to a single $, which allows for escaping - the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the - string literal "$(VAR_NAME)". Escaped references will never - be expanded, regardless of whether the variable exists or - not. Defaults to "".' + description: |- + Variable references $(VAR_NAME) are expanded + using the previously defined environment variables in the container and + any service environment variables. If a variable cannot be resolved, + the reference in the input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. + "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". + Escaped references will never be expanded, regardless of whether the variable + exists or not. + Defaults to "". type: string valueFrom: description: Source for the environment variable's value. Cannot @@ -2686,8 +3263,10 @@ spec: description: The key to select. type: string name: - description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - TODO: Add other useful fields. apiVersion, kind, uid?' + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? type: string optional: description: Specify whether the ConfigMap or its key @@ -2698,10 +3277,9 @@ spec: type: object x-kubernetes-map-type: atomic fieldRef: - description: 'Selects a field of the pod: supports metadata.name, - metadata.namespace, `metadata.labels['''']`, `metadata.annotations['''']`, - spec.nodeName, spec.serviceAccountName, status.hostIP, - status.podIP, status.podIPs.' + description: |- + Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`, + spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. properties: apiVersion: description: Version of the schema the FieldPath is @@ -2716,10 +3294,9 @@ spec: type: object x-kubernetes-map-type: atomic resourceFieldRef: - description: 'Selects a resource of the container: only - resources limits and requests (limits.cpu, limits.memory, - limits.ephemeral-storage, requests.cpu, requests.memory - and requests.ephemeral-storage) are currently supported.' + description: |- + Selects a resource of the container: only resources limits and requests + (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported. properties: containerName: description: 'Container name: required for volumes, @@ -2748,8 +3325,10 @@ spec: be a valid secret key. type: string name: - description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names - TODO: Add other useful fields. apiVersion, kind, uid?' + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid? type: string optional: description: Specify whether the Secret or its key must @@ -2768,9 +3347,9 @@ spec: description: If specified, the health-check port. type: integer hostNetwork: - description: Host networking requested for the stunnerd pod to use - the host's network namespace. Can be used to implement public TURN - servers with Kubernetes. Defaults to false. + description: |- + Host networking requested for the stunnerd pod to use the host's network namespace. + Can be used to implement public TURN servers with Kubernetes. Defaults to false. type: boolean image: description: Container image name. @@ -2779,26 +3358,33 @@ spec: description: Image pull policy. One of Always, Never, IfNotPresent. type: string replicas: - description: Number of desired pods. This is a pointer to distinguish - between explicit zero and not specified. Defaults to 1. + description: |- + Number of desired pods. This is a pointer to distinguish between explicit zero and not + specified. Defaults to 1. format: int32 type: integer resources: description: Resources required by stunnerd. properties: claims: - description: "Claims lists the names of resources, defined in - spec.resourceClaims, that are used by this container. \n This - is an alpha field and requires enabling the DynamicResourceAllocation - feature gate. \n This field is immutable. It can only be set - for containers." + description: |- + Claims lists the names of resources, defined in spec.resourceClaims, + that are used by this container. + + + This is an alpha field and requires enabling the + DynamicResourceAllocation feature gate. + + + This field is immutable. It can only be set for containers. items: description: ResourceClaim references one entry in PodSpec.ResourceClaims. properties: name: - description: Name must match the name of one entry in pod.spec.resourceClaims - of the Pod where this field is used. It makes that resource - available inside a container. + description: |- + Name must match the name of one entry in pod.spec.resourceClaims of + the Pod where this field is used. It makes that resource available + inside a container. type: string required: - name @@ -2814,8 +3400,9 @@ spec: - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true - description: 'Limits describes the maximum amount of compute resources - allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + description: |- + Limits describes the maximum amount of compute resources allowed. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object requests: additionalProperties: @@ -2824,11 +3411,11 @@ spec: - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true - description: 'Requests describes the minimum amount of compute - resources required. If Requests is omitted for a container, - it defaults to Limits if that is explicitly specified, otherwise - to an implementation-defined value. Requests cannot exceed Limits. - More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + description: |- + Requests describes the minimum amount of compute resources required. + If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. Requests cannot exceed Limits. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ type: object type: object securityContext: @@ -2836,59 +3423,67 @@ spec: common container settings. properties: fsGroup: - description: "A special supplemental group that applies to all - containers in a pod. Some volume types allow the Kubelet to - change the ownership of that volume to be owned by the pod: - \n 1. The owning GID will be the FSGroup 2. The setgid bit is - set (new files created in the volume will be owned by FSGroup) - 3. The permission bits are OR'd with rw-rw---- \n If unset, - the Kubelet will not modify the ownership and permissions of - any volume. Note that this field cannot be set when spec.os.name - is windows." + description: |- + A special supplemental group that applies to all containers in a pod. + Some volume types allow the Kubelet to change the ownership of that volume + to be owned by the pod: + + + 1. The owning GID will be the FSGroup + 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- + + + If unset, the Kubelet will not modify the ownership and permissions of any volume. + Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer fsGroupChangePolicy: - description: 'fsGroupChangePolicy defines behavior of changing - ownership and permission of the volume before being exposed - inside Pod. This field will only apply to volume types which - support fsGroup based ownership(and permissions). It will have - no effect on ephemeral volume types such as: secret, configmaps - and emptydir. Valid values are "OnRootMismatch" and "Always". - If not specified, "Always" is used. Note that this field cannot - be set when spec.os.name is windows.' + description: |- + fsGroupChangePolicy defines behavior of changing ownership and permission of the volume + before being exposed inside Pod. This field will only apply to + volume types which support fsGroup based ownership(and permissions). + It will have no effect on ephemeral volume types such as: secret, configmaps + and emptydir. + Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. + Note that this field cannot be set when spec.os.name is windows. type: string runAsGroup: - description: The GID to run the entrypoint of the container process. - Uses runtime default if unset. May also be set in SecurityContext. If - set in both SecurityContext and PodSecurityContext, the value - specified in SecurityContext takes precedence for that container. + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer runAsNonRoot: - description: Indicates that the container must run as a non-root - user. If true, the Kubelet will validate the image at runtime - to ensure that it does not run as UID 0 (root) and fail to start - the container if it does. If unset or false, no such validation - will be performed. May also be set in SecurityContext. If set - in both SecurityContext and PodSecurityContext, the value specified - in SecurityContext takes precedence. + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. type: boolean runAsUser: - description: The UID to run the entrypoint of the container process. + description: |- + The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. - May also be set in SecurityContext. If set in both SecurityContext - and PodSecurityContext, the value specified in SecurityContext - takes precedence for that container. Note that this field cannot - be set when spec.os.name is windows. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. + Note that this field cannot be set when spec.os.name is windows. format: int64 type: integer seLinuxOptions: - description: The SELinux context to be applied to all containers. - If unspecified, the container runtime will allocate a random - SELinux context for each container. May also be set in SecurityContext. If - set in both SecurityContext and PodSecurityContext, the value - specified in SecurityContext takes precedence for that container. + description: |- + The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in SecurityContext. If set in + both SecurityContext and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. properties: level: @@ -2909,47 +3504,48 @@ spec: type: string type: object seccompProfile: - description: The seccomp options to use by the containers in this - pod. Note that this field cannot be set when spec.os.name is - windows. + description: |- + The seccomp options to use by the containers in this pod. + Note that this field cannot be set when spec.os.name is windows. properties: localhostProfile: - description: localhostProfile indicates a profile defined - in a file on the node should be used. The profile must be - preconfigured on the node to work. Must be a descending - path, relative to the kubelet's configured seccomp profile - location. Must be set if type is "Localhost". Must NOT be - set for any other type. + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. type: string type: - description: "type indicates which kind of seccomp profile - will be applied. Valid options are: \n Localhost - a profile - defined in a file on the node should be used. RuntimeDefault - - the container runtime default profile should be used. - Unconfined - no profile should be applied." + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. type: string required: - type type: object supplementalGroups: - description: A list of groups applied to the first process run - in each container, in addition to the container's primary GID, - the fsGroup (if specified), and group memberships defined in - the container image for the uid of the container process. If - unspecified, no additional groups are added to any container. - Note that group memberships defined in the container image for - the uid of the container process are still effective, even if - they are not included in this list. Note that this field cannot - be set when spec.os.name is windows. + description: |- + A list of groups applied to the first process run in each container, in addition + to the container's primary GID, the fsGroup (if specified), and group memberships + defined in the container image for the uid of the container process. If unspecified, + no additional groups are added to any container. Note that group memberships + defined in the container image for the uid of the container process are still effective, + even if they are not included in this list. + Note that this field cannot be set when spec.os.name is windows. items: format: int64 type: integer type: array sysctls: - description: Sysctls hold a list of namespaced sysctls used for - the pod. Pods with unsupported sysctls (by the container runtime) - might fail to launch. Note that this field cannot be set when - spec.os.name is windows. + description: |- + Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported + sysctls (by the container runtime) might fail to launch. + Note that this field cannot be set when spec.os.name is windows. items: description: Sysctl defines a kernel parameter to be set properties: @@ -2965,36 +3561,35 @@ spec: type: object type: array windowsOptions: - description: The Windows specific settings applied to all containers. - If unspecified, the options within a container's SecurityContext - will be used. If set in both SecurityContext and PodSecurityContext, - the value specified in SecurityContext takes precedence. Note - that this field cannot be set when spec.os.name is linux. + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. properties: gmsaCredentialSpec: - description: GMSACredentialSpec is where the GMSA admission - webhook (https://github.com/kubernetes-sigs/windows-gmsa) - inlines the contents of the GMSA credential spec named by - the GMSACredentialSpecName field. + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. type: string gmsaCredentialSpecName: description: GMSACredentialSpecName is the name of the GMSA credential spec to use. type: string hostProcess: - description: HostProcess determines if a container should - be run as a 'Host Process' container. All of a Pod's containers - must have the same effective HostProcess value (it is not - allowed to have a mix of HostProcess containers and non-HostProcess - containers). In addition, if HostProcess is true then HostNetwork - must also be set to true. + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. type: boolean runAsUserName: - description: The UserName in Windows to run the entrypoint - of the container process. Defaults to the user specified - in image metadata if unspecified. May also be set in PodSecurityContext. - If set in both SecurityContext and PodSecurityContext, the - value specified in SecurityContext takes precedence. + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. type: string type: object type: object @@ -3006,40 +3601,39 @@ spec: tolerations: description: If specified, the pod's tolerations. items: - description: The pod this Toleration is attached to tolerates any - taint that matches the triple using the matching - operator . + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . properties: effect: - description: Effect indicates the taint effect to match. Empty - means match all taint effects. When specified, allowed values - are NoSchedule, PreferNoSchedule and NoExecute. + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. type: string key: - description: Key is the taint key that the toleration applies - to. Empty means match all taint keys. If the key is empty, - operator must be Exists; this combination means to match all - values and all keys. + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. type: string operator: - description: Operator represents a key's relationship to the - value. Valid operators are Exists and Equal. Defaults to Equal. - Exists is equivalent to wildcard for value, so that a pod - can tolerate all taints of a particular category. + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists and Equal. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. type: string tolerationSeconds: - description: TolerationSeconds represents the period of time - the toleration (which must be of effect NoExecute, otherwise - this field is ignored) tolerates the taint. By default, it - is not set, which means tolerate the taint forever (do not - evict). Zero and negative values will be treated as 0 (evict - immediately) by the system. + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. format: int64 type: integer value: - description: Value is the taint value the toleration matches - to. If the operator is Exists, the value should be empty, - otherwise just a regular string. + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. type: string type: object type: array @@ -3052,7 +3646,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.13.0 + controller-gen.kubebuilder.io/version: v0.14.0 name: udproutes.stunner.l7mp.io spec: group: stunner.l7mp.io @@ -3072,21 +3666,31 @@ spec: name: v1 schema: openAPIV3Schema: - description: "UDPRoute provides a way to route UDP traffic. When combined - with a Gateway listener, it can be used to forward traffic on the port specified - by the listener to a set of backends specified by the UDPRoute. \n Differences - from Gateway API UDPRoutes - port-ranges are correctly handled ([port, endPort]) - - port is not mandatory - backend weight is not supported" + description: |- + UDPRoute provides a way to route UDP traffic. When combined with a Gateway listener, it can be + used to forward traffic on the port specified by the listener to a set of backends specified by + the UDPRoute. + + + Differences from Gateway API UDPRoutes + - port-ranges are correctly handled ([port, endPort]) + - port is not mandatory + - backend weight is not supported properties: apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object @@ -3094,204 +3698,252 @@ spec: description: Spec defines the desired state of UDPRoute. properties: parentRefs: - description: "ParentRefs references the resources (usually Gateways) - that a Route wants to be attached to. Note that the referenced parent - resource needs to allow this for the attachment to be complete. - For Gateways, that means the Gateway needs to allow attachment from - Routes of this kind and namespace. For Services, that means the - Service must either be in the same namespace for a \"producer\" - route, or the mesh implementation must support and allow \"consumer\" - routes for the referenced Service. ReferenceGrant is not applicable - for governing ParentRefs to Services - it is not possible to create - a \"producer\" route for a Service in a different namespace from - the Route. \n There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) - * Service (Mesh conformance profile, experimental, ClusterIP Services - only) This API may be extended - in the future to support additional kinds of parent resources. \n - ParentRefs must be _distinct_. This means either that: \n * They - select different objects. If this is the case, then parentRef entries - are distinct. In terms of fields, this means that the multi-part - key defined by `group`, `kind`, `namespace`, and `name` must be - unique across all parentRef entries in the Route. * They do not - select different objects, but for each optional field used, each - ParentRef that selects the same object must set the same set of - optional fields to different values. If one ParentRef sets a combination - of optional fields, all must set the same combination. \n Some examples: - \n * If one ParentRef sets `sectionName`, all ParentRefs referencing - the same object must also set `sectionName`. * If one ParentRef - sets `port`, all ParentRefs referencing the same object must also - set `port`. * If one ParentRef sets `sectionName` and `port`, all - ParentRefs referencing the same object must also set `sectionName` - and `port`. \n It is possible to separately reference multiple distinct - objects that may be collapsed by an implementation. For example, - some implementations may choose to merge compatible Gateway Listeners - together. If that is the case, the list of routes attached to those - resources should also be merged. \n Note that for ParentRefs that - cross namespace boundaries, there are specific rules. Cross-namespace - references are only valid if they are explicitly allowed by something - in the namespace they are referring to. For example, Gateway has - the AllowedRoutes field, and ReferenceGrant provides a generic way - to enable other kinds of cross-namespace reference. \n - ParentRefs from a Route to a Service in the same namespace are \"producer\" - routes, which apply default routing rules to inbound connections - from any namespace to the Service. \n ParentRefs from a Route to - a Service in a different namespace are \"consumer\" routes, and - these routing rules are only applied to outbound connections originating - from the same namespace as the Route, for which the intended destination - of the connections are a Service targeted as a ParentRef of the - Route. \n - - - " + description: |- + ParentRefs references the resources (usually Gateways) that a Route wants + to be attached to. Note that the referenced parent resource needs to + allow this for the attachment to be complete. For Gateways, that means + the Gateway needs to allow attachment from Routes of this kind and + namespace. For Services, that means the Service must either be in the same + namespace for a "producer" route, or the mesh implementation must support + and allow "consumer" routes for the referenced Service. ReferenceGrant is + not applicable for governing ParentRefs to Services - it is not possible to + create a "producer" route for a Service in a different namespace from the + Route. + + + There are two kinds of parent resources with "Core" support: + + + * Gateway (Gateway conformance profile) + + * Service (Mesh conformance profile, experimental, ClusterIP Services only) + + This API may be extended in the future to support additional kinds of parent + resources. + + + ParentRefs must be _distinct_. This means either that: + + + * They select different objects. If this is the case, then parentRef + entries are distinct. In terms of fields, this means that the + multi-part key defined by `group`, `kind`, `namespace`, and `name` must + be unique across all parentRef entries in the Route. + * They do not select different objects, but for each optional field used, + each ParentRef that selects the same object must set the same set of + optional fields to different values. If one ParentRef sets a + combination of optional fields, all must set the same combination. + + + Some examples: + + + * If one ParentRef sets `sectionName`, all ParentRefs referencing the + same object must also set `sectionName`. + * If one ParentRef sets `port`, all ParentRefs referencing the same + object must also set `port`. + * If one ParentRef sets `sectionName` and `port`, all ParentRefs + referencing the same object must also set `sectionName` and `port`. + + + It is possible to separately reference multiple distinct objects that may + be collapsed by an implementation. For example, some implementations may + choose to merge compatible Gateway Listeners together. If that is the + case, the list of routes attached to those resources should also be + merged. + + + Note that for ParentRefs that cross namespace boundaries, there are specific + rules. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For example, + Gateway has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable other kinds of cross-namespace reference. + + + + ParentRefs from a Route to a Service in the same namespace are "producer" + routes, which apply default routing rules to inbound connections from + any namespace to the Service. + + + ParentRefs from a Route to a Service in a different namespace are + "consumer" routes, and these routing rules are only applied to outbound + connections originating from the same namespace as the Route, for which + the intended destination of the connections are a Service targeted as a + ParentRef of the Route. + + + + + + + items: - description: "ParentReference identifies an API object (usually - a Gateway) that can be considered a parent of this resource (usually - a route). There are two kinds of parent resources with \"Core\" - support: \n * Gateway (Gateway conformance profile) * Service - (Mesh conformance profile, experimental, ClusterIP Services only) - \n This API may be extended in the future to support additional - kinds of parent resources. \n The API object must be valid in - the cluster; the Group and Kind must be registered in the cluster - for this reference to be valid." + description: |- + ParentReference identifies an API object (usually a Gateway) that can be considered + a parent of this resource (usually a route). There are two kinds of parent resources + with "Core" support: + + + * Gateway (Gateway conformance profile) + * Service (Mesh conformance profile, experimental, ClusterIP Services only) + + + This API may be extended in the future to support additional kinds of parent + resources. + + + The API object must be valid in the cluster; the Group and Kind must + be registered in the cluster for this reference to be valid. properties: group: default: gateway.networking.k8s.io - description: "Group is the group of the referent. When unspecified, - \"gateway.networking.k8s.io\" is inferred. To set the core - API group (such as for a \"Service\" kind referent), Group - must be explicitly set to \"\" (empty string). \n Support: - Core" + description: |- + Group is the group of the referent. + When unspecified, "gateway.networking.k8s.io" is inferred. + To set the core API group (such as for a "Service" kind referent), + Group must be explicitly set to "" (empty string). + + + Support: Core maxLength: 253 pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string kind: default: Gateway - description: "Kind is kind of the referent. \n There are two - kinds of parent resources with \"Core\" support: \n * Gateway - (Gateway conformance profile) * Service (Mesh conformance - profile, experimental, ClusterIP Services only) \n Support - for other resources is Implementation-Specific." + description: |- + Kind is kind of the referent. + + + There are two kinds of parent resources with "Core" support: + + + * Gateway (Gateway conformance profile) + * Service (Mesh conformance profile, experimental, ClusterIP Services only) + + + Support for other resources is Implementation-Specific. maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ type: string name: - description: "Name is the name of the referent. \n Support: - Core" + description: |- + Name is the name of the referent. + + + Support: Core maxLength: 253 minLength: 1 type: string namespace: - description: "Namespace is the namespace of the referent. When - unspecified, this refers to the local namespace of the Route. - \n Note that there are specific rules for ParentRefs which - cross namespace boundaries. Cross-namespace references are - only valid if they are explicitly allowed by something in - the namespace they are referring to. For example: Gateway - has the AllowedRoutes field, and ReferenceGrant provides a + description: |- + Namespace is the namespace of the referent. When unspecified, this refers + to the local namespace of the Route. + + + Note that there are specific rules for ParentRefs which cross namespace + boundaries. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For example: + Gateway has the AllowedRoutes field, and ReferenceGrant provides a generic way to enable any other kind of cross-namespace reference. - \n ParentRefs from a Route - to a Service in the same namespace are \"producer\" routes, - which apply default routing rules to inbound connections from - any namespace to the Service. \n ParentRefs from a Route to - a Service in a different namespace are \"consumer\" routes, - and these routing rules are only applied to outbound connections - originating from the same namespace as the Route, for which - the intended destination of the connections are a Service - targeted as a ParentRef of the Route. - \n Support: Core" + + + + ParentRefs from a Route to a Service in the same namespace are "producer" + routes, which apply default routing rules to inbound connections from + any namespace to the Service. + + + ParentRefs from a Route to a Service in a different namespace are + "consumer" routes, and these routing rules are only applied to outbound + connections originating from the same namespace as the Route, for which + the intended destination of the connections are a Service targeted as a + ParentRef of the Route. + + + + Support: Core maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ type: string port: - description: "Port is the network port this Route targets. It - can be interpreted differently based on the type of parent - resource. \n When the parent resource is a Gateway, this targets - all listeners listening on the specified port that also support - this kind of Route(and select this Route). It's not recommended - to set `Port` unless the networking behaviors specified in - a Route must apply to a specific port as opposed to a listener(s) - whose port(s) may be changed. When both Port and SectionName - are specified, the name and port of the selected listener - must match both specified values. \n - When the parent resource is a Service, this targets a specific - port in the Service spec. When both Port (experimental) and - SectionName are specified, the name and port of the selected - port must match both specified values. - \n Implementations MAY choose to support other parent resources. - Implementations supporting other types of parent resources - MUST clearly document how/if Port is interpreted. \n For the - purpose of status, an attachment is considered successful - as long as the parent resource accepts it partially. For example, - Gateway listeners can restrict which Routes can attach to - them by Route kind, namespace, or hostname. If 1 of 2 Gateway - listeners accept attachment from the referencing Route, the - Route MUST be considered successfully attached. If no Gateway - listeners accept attachment from this Route, the Route MUST - be considered detached from the Gateway. \n Support: Extended - \n " + description: |- + Port is the network port this Route targets. It can be interpreted + differently based on the type of parent resource. + + + When the parent resource is a Gateway, this targets all listeners + listening on the specified port that also support this kind of Route(and + select this Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to a specific port + as opposed to a listener(s) whose port(s) may be changed. When both Port + and SectionName are specified, the name and port of the selected listener + must match both specified values. + + + + When the parent resource is a Service, this targets a specific port in the + Service spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified values. + + + + Implementations MAY choose to support other parent resources. + Implementations supporting other types of parent resources MUST clearly + document how/if Port is interpreted. + + + For the purpose of status, an attachment is considered successful as + long as the parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept attachment + from the referencing Route, the Route MUST be considered successfully + attached. If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + + + Support: Extended + + + format: int32 maximum: 65535 minimum: 1 type: integer sectionName: - description: "SectionName is the name of a section within the - target resource. In the following resources, SectionName is - interpreted as the following: \n * Gateway: Listener Name. - When both Port (experimental) and SectionName are specified, - the name and port of the selected listener must match both - specified values. * Service: Port Name. When both Port (experimental) - and SectionName are specified, the name and port of the selected - listener must match both specified values. Note that attaching - Routes to Services as Parents is part of experimental Mesh - support and is not supported for any other purpose. \n Implementations - MAY choose to support attaching Routes to other resources. - If that is the case, they MUST clearly document how SectionName - is interpreted. \n When unspecified (empty string), this will - reference the entire resource. For the purpose of status, - an attachment is considered successful if at least one section - in the parent resource accepts it. For example, Gateway listeners - can restrict which Routes can attach to them by Route kind, - namespace, or hostname. If 1 of 2 Gateway listeners accept - attachment from the referencing Route, the Route MUST be considered - successfully attached. If no Gateway listeners accept attachment - from this Route, the Route MUST be considered detached from - the Gateway. \n Support: Core" + description: |- + SectionName is the name of a section within the target resource. In the + following resources, SectionName is interpreted as the following: + + + * Gateway: Listener Name. When both Port (experimental) and SectionName + are specified, the name and port of the selected listener must match + both specified values. + * Service: Port Name. When both Port (experimental) and SectionName + are specified, the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services as Parents + is part of experimental Mesh support and is not supported for any other + purpose. + + + Implementations MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName is + interpreted. + + + When unspecified (empty string), this will reference the entire resource. + For the purpose of status, an attachment is considered successful if at + least one section in the parent resource accepts it. For example, Gateway + listeners can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept attachment from + the referencing Route, the Route MUST be considered successfully + attached. If no Gateway listeners accept attachment from this Route, the + Route MUST be considered detached from the Gateway. + + + Support: Core maxLength: 253 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ @@ -3307,9 +3959,9 @@ spec: description: UDPRouteRule is the configuration for a given rule. properties: backendRefs: - description: BackendRefs defines the backend(s) where matching - requests should be sent. UDPRouteRules correctly handle port - ranges. + description: |- + BackendRefs defines the backend(s) where matching requests should be + sent. UDPRouteRules correctly handle port ranges. items: description: BackendRef defines how a Route should forward a request to a Kubernetes resource. @@ -3323,16 +3975,17 @@ spec: type: integer group: default: "" - description: Group is the group of the referent. For example, - "gateway.networking.k8s.io". When unspecified or empty - string, core API group is inferred. + description: |- + Group is the group of the referent. For example, "gateway.networking.k8s.io". + When unspecified or empty string, core API group is inferred. maxLength: 253 pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string kind: default: Service - description: Kind is the Kubernetes resource kind of the - referent. For example "Service". + description: |- + Kind is the Kubernetes resource kind of the referent. For example + "Service". maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ @@ -3343,18 +3996,18 @@ spec: minLength: 1 type: string namespace: - description: Namespace is the namespace of the backend. - When unspecified, the local namespace is inferred. + description: |- + Namespace is the namespace of the backend. When unspecified, the local + namespace is inferred. maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ type: string port: - description: Port specifies the destination port number - to use for this resource. If port is not specified, - all ports are allowed. If port is defined but endPort - is not, allow only access to the given port. If both - are specified, allows access in the port-range [port, + description: |- + Port specifies the destination port number to use for this resource. If port is not + specified, all ports are allowed. If port is defined but endPort is not, allow only + access to the given port. If both are specified, allows access in the port-range [port, endPort] inclusive. format: int32 maximum: 65535 @@ -3377,81 +4030,94 @@ spec: description: Status defines the current state of UDPRoute. properties: parents: - description: "Parents is a list of parent resources (usually Gateways) - that are associated with the route, and the status of the route - with respect to each parent. When this route attaches to a parent, - the controller that manages the parent must add an entry to this - list when the controller first sees the route and should update - the entry as appropriate when the route or gateway is modified. - \n Note that parent references that cannot be resolved by an implementation - of this API will not be added to this list. Implementations of this - API can only populate Route status for the Gateways/parent resources - they are responsible for. \n A maximum of 32 Gateways will be represented - in this list. An empty list means the route has not been attached - to any Gateway." + description: |- + Parents is a list of parent resources (usually Gateways) that are + associated with the route, and the status of the route with respect to + each parent. When this route attaches to a parent, the controller that + manages the parent must add an entry to this list when the controller + first sees the route and should update the entry as appropriate when the + route or gateway is modified. + + + Note that parent references that cannot be resolved by an implementation + of this API will not be added to this list. Implementations of this API + can only populate Route status for the Gateways/parent resources they are + responsible for. + + + A maximum of 32 Gateways will be represented in this list. An empty list + means the route has not been attached to any Gateway. items: - description: RouteParentStatus describes the status of a route with - respect to an associated Parent. + description: |- + RouteParentStatus describes the status of a route with respect to an + associated Parent. properties: conditions: - description: "Conditions describes the status of the route with - respect to the Gateway. Note that the route's availability - is also subject to the Gateway's own status conditions and - listener status. \n If the Route's ParentRef specifies an - existing Gateway that supports Routes of this kind AND that - Gateway's controller has sufficient access, then that Gateway's - controller MUST set the \"Accepted\" condition on the Route, - to indicate whether the route has been accepted or rejected - by the Gateway, and why. \n A Route MUST be considered \"Accepted\" - if at least one of the Route's rules is implemented by the - Gateway. \n There are a number of cases where the \"Accepted\" - condition may not be set due to lack of controller visibility, - that includes when: \n * The Route refers to a non-existent - parent. * The Route is of a type that the controller does - not support. * The Route is in a namespace the controller - does not have access to." + description: |- + Conditions describes the status of the route with respect to the Gateway. + Note that the route's availability is also subject to the Gateway's own + status conditions and listener status. + + + If the Route's ParentRef specifies an existing Gateway that supports + Routes of this kind AND that Gateway's controller has sufficient access, + then that Gateway's controller MUST set the "Accepted" condition on the + Route, to indicate whether the route has been accepted or rejected by the + Gateway, and why. + + + A Route MUST be considered "Accepted" if at least one of the Route's + rules is implemented by the Gateway. + + + There are a number of cases where the "Accepted" condition may not be set + due to lack of controller visibility, that includes when: + + + * The Route refers to a non-existent parent. + * The Route is of a type that the controller does not support. + * The Route is in a namespace the controller does not have access to. items: description: "Condition contains details for one aspect of - the current state of this API Resource. --- This struct + the current state of this API Resource.\n---\nThis struct is intended for direct use as an array at the field path - .status.conditions. For example, \n type FooStatus struct{ - // Represents the observations of a foo's current state. - // Known .status.conditions.type are: \"Available\", \"Progressing\", - and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge - // +listType=map // +listMapKey=type Conditions []metav1.Condition - `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\" - protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields - }" + .status.conditions. For example,\n\n\n\ttype FooStatus + struct{\n\t // Represents the observations of a foo's + current state.\n\t // Known .status.conditions.type are: + \"Available\", \"Progressing\", and \"Degraded\"\n\t // + +patchMergeKey=type\n\t // +patchStrategy=merge\n\t // + +listType=map\n\t // +listMapKey=type\n\t Conditions + []metav1.Condition `json:\"conditions,omitempty\" patchStrategy:\"merge\" + patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t + \ // other fields\n\t}" properties: lastTransitionTime: - description: lastTransitionTime is the last time the condition - transitioned from one status to another. This should - be when the underlying condition changed. If that is - not known, then using the time when the API field changed - is acceptable. + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: - description: message is a human readable message indicating - details about the transition. This may be an empty string. + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. maxLength: 32768 type: string observedGeneration: - description: observedGeneration represents the .metadata.generation - that the condition was set based upon. For instance, - if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration - is 9, the condition is out of date with respect to the - current state of the instance. + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: - description: reason contains a programmatic identifier - indicating the reason for the condition's last transition. - Producers of specific condition types may define expected - values and meanings for this field, and whether the - values are considered a guaranteed API. The value should - be a CamelCase string. This field may not be empty. + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ @@ -3465,12 +4131,12 @@ spec: - Unknown type: string type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - --- Many .condition.type values are consistent across - resources like Available, but because arbitrary conditions - can be useful (see .node.status.conditions), the ability - to deconflict is important. The regex it matches is - (dns1123SubdomainFmt/)?(qualifiedNameFmt) + description: |- + type of condition in CamelCase or in foo.example.com/CamelCase. + --- + Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be + useful (see .node.status.conditions), the ability to deconflict is important. + The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt) maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string @@ -3488,131 +4154,180 @@ spec: - type x-kubernetes-list-type: map controllerName: - description: "ControllerName is a domain/path string that indicates - the name of the controller that wrote this status. This corresponds - with the controllerName field on GatewayClass. \n Example: - \"example.net/gateway-controller\". \n The format of this - field is DOMAIN \"/\" PATH, where DOMAIN and PATH are valid - Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). - \n Controllers MUST populate this field when writing status. - Controllers should ensure that entries to status populated - with their ControllerName are cleaned up when they are no - longer necessary." + description: |- + ControllerName is a domain/path string that indicates the name of the + controller that wrote this status. This corresponds with the + controllerName field on GatewayClass. + + + Example: "example.net/gateway-controller". + + + The format of this field is DOMAIN "/" PATH, where DOMAIN and PATH are + valid Kubernetes names + (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). + + + Controllers MUST populate this field when writing status. Controllers should ensure that + entries to status populated with their ControllerName are cleaned up when they are no + longer necessary. maxLength: 253 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$ type: string parentRef: - description: ParentRef corresponds with a ParentRef in the spec - that this RouteParentStatus struct describes the status of. + description: |- + ParentRef corresponds with a ParentRef in the spec that this + RouteParentStatus struct describes the status of. properties: group: default: gateway.networking.k8s.io - description: "Group is the group of the referent. When unspecified, - \"gateway.networking.k8s.io\" is inferred. To set the - core API group (such as for a \"Service\" kind referent), - Group must be explicitly set to \"\" (empty string). \n - Support: Core" + description: |- + Group is the group of the referent. + When unspecified, "gateway.networking.k8s.io" is inferred. + To set the core API group (such as for a "Service" kind referent), + Group must be explicitly set to "" (empty string). + + + Support: Core maxLength: 253 pattern: ^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string kind: default: Gateway - description: "Kind is kind of the referent. \n There are - two kinds of parent resources with \"Core\" support: \n - * Gateway (Gateway conformance profile) * Service (Mesh - conformance profile, experimental, ClusterIP Services - only) \n Support for other resources is Implementation-Specific." + description: |- + Kind is kind of the referent. + + + There are two kinds of parent resources with "Core" support: + + + * Gateway (Gateway conformance profile) + * Service (Mesh conformance profile, experimental, ClusterIP Services only) + + + Support for other resources is Implementation-Specific. maxLength: 63 minLength: 1 pattern: ^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ type: string name: - description: "Name is the name of the referent. \n Support: - Core" + description: |- + Name is the name of the referent. + + + Support: Core maxLength: 253 minLength: 1 type: string namespace: - description: "Namespace is the namespace of the referent. - When unspecified, this refers to the local namespace of - the Route. \n Note that there are specific rules for ParentRefs - which cross namespace boundaries. Cross-namespace references - are only valid if they are explicitly allowed by something - in the namespace they are referring to. For example: Gateway - has the AllowedRoutes field, and ReferenceGrant provides - a generic way to enable any other kind of cross-namespace - reference. \n ParentRefs - from a Route to a Service in the same namespace are \"producer\" - routes, which apply default routing rules to inbound connections - from any namespace to the Service. \n ParentRefs from - a Route to a Service in a different namespace are \"consumer\" - routes, and these routing rules are only applied to outbound - connections originating from the same namespace as the - Route, for which the intended destination of the connections - are a Service targeted as a ParentRef of the Route. - \n Support: Core" + description: |- + Namespace is the namespace of the referent. When unspecified, this refers + to the local namespace of the Route. + + + Note that there are specific rules for ParentRefs which cross namespace + boundaries. Cross-namespace references are only valid if they are explicitly + allowed by something in the namespace they are referring to. For example: + Gateway has the AllowedRoutes field, and ReferenceGrant provides a + generic way to enable any other kind of cross-namespace reference. + + + + ParentRefs from a Route to a Service in the same namespace are "producer" + routes, which apply default routing rules to inbound connections from + any namespace to the Service. + + + ParentRefs from a Route to a Service in a different namespace are + "consumer" routes, and these routing rules are only applied to outbound + connections originating from the same namespace as the Route, for which + the intended destination of the connections are a Service targeted as a + ParentRef of the Route. + + + + Support: Core maxLength: 63 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ type: string port: - description: "Port is the network port this Route targets. - It can be interpreted differently based on the type of - parent resource. \n When the parent resource is a Gateway, - this targets all listeners listening on the specified - port that also support this kind of Route(and select this - Route). It's not recommended to set `Port` unless the - networking behaviors specified in a Route must apply to - a specific port as opposed to a listener(s) whose port(s) - may be changed. When both Port and SectionName are specified, - the name and port of the selected listener must match - both specified values. \n - When the parent resource is a Service, this targets a - specific port in the Service spec. When both Port (experimental) - and SectionName are specified, the name and port of the - selected port must match both specified values. - \n Implementations MAY choose to support other parent - resources. Implementations supporting other types of parent - resources MUST clearly document how/if Port is interpreted. - \n For the purpose of status, an attachment is considered - successful as long as the parent resource accepts it partially. - For example, Gateway listeners can restrict which Routes - can attach to them by Route kind, namespace, or hostname. - If 1 of 2 Gateway listeners accept attachment from the - referencing Route, the Route MUST be considered successfully - attached. If no Gateway listeners accept attachment from - this Route, the Route MUST be considered detached from - the Gateway. \n Support: Extended \n " + description: |- + Port is the network port this Route targets. It can be interpreted + differently based on the type of parent resource. + + + When the parent resource is a Gateway, this targets all listeners + listening on the specified port that also support this kind of Route(and + select this Route). It's not recommended to set `Port` unless the + networking behaviors specified in a Route must apply to a specific port + as opposed to a listener(s) whose port(s) may be changed. When both Port + and SectionName are specified, the name and port of the selected listener + must match both specified values. + + + + When the parent resource is a Service, this targets a specific port in the + Service spec. When both Port (experimental) and SectionName are specified, + the name and port of the selected port must match both specified values. + + + + Implementations MAY choose to support other parent resources. + Implementations supporting other types of parent resources MUST clearly + document how/if Port is interpreted. + + + For the purpose of status, an attachment is considered successful as + long as the parent resource accepts it partially. For example, Gateway + listeners can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept attachment + from the referencing Route, the Route MUST be considered successfully + attached. If no Gateway listeners accept attachment from this Route, + the Route MUST be considered detached from the Gateway. + + + Support: Extended + + + format: int32 maximum: 65535 minimum: 1 type: integer sectionName: - description: "SectionName is the name of a section within - the target resource. In the following resources, SectionName - is interpreted as the following: \n * Gateway: Listener - Name. When both Port (experimental) and SectionName are - specified, the name and port of the selected listener - must match both specified values. * Service: Port Name. - When both Port (experimental) and SectionName are specified, - the name and port of the selected listener must match - both specified values. Note that attaching Routes to Services - as Parents is part of experimental Mesh support and is - not supported for any other purpose. \n Implementations - MAY choose to support attaching Routes to other resources. - If that is the case, they MUST clearly document how SectionName - is interpreted. \n When unspecified (empty string), this - will reference the entire resource. For the purpose of - status, an attachment is considered successful if at least - one section in the parent resource accepts it. For example, - Gateway listeners can restrict which Routes can attach - to them by Route kind, namespace, or hostname. If 1 of - 2 Gateway listeners accept attachment from the referencing - Route, the Route MUST be considered successfully attached. - If no Gateway listeners accept attachment from this Route, - the Route MUST be considered detached from the Gateway. - \n Support: Core" + description: |- + SectionName is the name of a section within the target resource. In the + following resources, SectionName is interpreted as the following: + + + * Gateway: Listener Name. When both Port (experimental) and SectionName + are specified, the name and port of the selected listener must match + both specified values. + * Service: Port Name. When both Port (experimental) and SectionName + are specified, the name and port of the selected listener must match + both specified values. Note that attaching Routes to Services as Parents + is part of experimental Mesh support and is not supported for any other + purpose. + + + Implementations MAY choose to support attaching Routes to other resources. + If that is the case, they MUST clearly document how SectionName is + interpreted. + + + When unspecified (empty string), this will reference the entire resource. + For the purpose of status, an attachment is considered successful if at + least one section in the parent resource accepts it. For example, Gateway + listeners can restrict which Routes can attach to them by Route kind, + namespace, or hostname. If 1 of 2 Gateway listeners accept attachment from + the referencing Route, the Route MUST be considered successfully + attached. If no Gateway listeners accept attachment from this Route, the + Route MUST be considered detached from the Gateway. + + + Support: Core maxLength: 253 minLength: 1 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ From 1272d5fcfda232f7f4b62dcdb047b765bcec4ba2 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 28 May 2024 18:57:32 +0200 Subject: [PATCH 176/243] doc: Update milestones --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 48e33cfc..23b492d3 100644 --- a/README.md +++ b/README.md @@ -610,7 +610,9 @@ notable limitations at this point are as follows. * v0.13: Observability: Prometheus + Grafana dashboard. * v0.15: Performance: Per-allocation CPU load-balancing for UDP * v0.16: Management: Managed STUNner dataplane. -* v0.17: Stabilization: All Gateway and STUNner APIs move to V1. +* v0.17: First release candidate: All Gateway and STUNner APIs move to v1. +* v0.18: Stabilization: Second release candidate. +* v0.19: The missing pieces: Third release candidate. * v1.0: GA ## Help From e4fc45b0fc754371908d9a4e5b8a35e2e265db3b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 28 May 2024 19:01:45 +0200 Subject: [PATCH 177/243] chore: Remove doc disclaimer before release --- README.md | 2 -- docs/README.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/README.md b/README.md index 23b492d3..60dfe962 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,6 @@

-*This is the documentation for the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 5adbc162..03638016 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,6 @@ STUNner

-*Note: The below documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io).* - ## Overview * [Why STUNner](WHY.md) From 241e9e91a818b3bbf970dddc0f489ac11072372f Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 28 May 2024 19:18:00 +0200 Subject: [PATCH 178/243] chore: Restore doc disclaimers --- README.md | 2 ++ docs/README.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/README.md b/README.md index 60dfe962..464f9810 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 03638016..bf4c7f51 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,8 @@ STUNner

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + ## Overview * [Why STUNner](WHY.md) From 6b66aa4668366112ec4ca1a9bce3403860e7e3b7 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 29 May 2024 14:32:01 +0200 Subject: [PATCH 179/243] doc: Update release instructions --- docs/RELEASE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 484e0353..407402ed 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -14,6 +14,7 @@ Let the new version be vX.Y.Z. - `git push` if there are local changes - release vX.Y.Z on github (so that we can publish the release notes) - wait until the CI/CD pipeline goes green (may take a while) +- restore doc disclaimers ## STUNner auth service From 512ce555aac9859d4d36fd540a1fff89923b207b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Wed, 29 May 2024 14:32:39 +0200 Subject: [PATCH 180/243] fix(turncat): Report error in turncat when invoked with empty peer IP --- turncat.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/turncat.go b/turncat.go index 0e835462..ea10ac3c 100644 --- a/turncat.go +++ b/turncat.go @@ -107,6 +107,9 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { if err != nil { return nil, fmt.Errorf("error resolving peer address %q: %w", config.PeerAddr, err) } + if peerAddress == nil || peerAddress.IP == nil { + return nil, fmt.Errorf("empty IP address in peer URL %q", config.PeerAddr) + } if config.Realm == "" { config.Realm = stnrv1.DefaultRealm From 246913fd0b761775f42209933e6f9f27ef83c243 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Mon, 10 Jun 2024 12:14:15 +0200 Subject: [PATCH 181/243] feature(turncat): Set SNI on TURN/TLS client connections --- cmd/turncat/main.go | 5 ++++- turncat.go | 5 +++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index d1beeb55..f6720dce 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -58,8 +58,10 @@ func main() { cdsConfigFlags = cdsclient.NewCDSConfigFlags() cdsConfigFlags.AddFlags(flag.CommandLine) - var level = flag.StringP("log", "l", "all:WARN", "Log level") + var serverName string + flag.StringVar(&serverName, "sni", "", "Server name (SNI) for TURN/TLS client connections") var insecure = flag.BoolP("insecure", "i", false, "Insecure TLS mode, accept self-signed TURN server certificates (default: false)") + var level = flag.StringP("log", "l", "all:WARN", "Log level") var verbose = flag.BoolP("verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") var help = flag.BoolP("help", "h", false, "Display this help text and exit") @@ -114,6 +116,7 @@ func main() { PeerAddr: flag.Arg(2), Realm: config.Auth.Realm, AuthGen: authGen, + ServerName: serverName, InsecureMode: *insecure, LoggerFactory: loggerFactory, } diff --git a/turncat.go b/turncat.go index ea10ac3c..792d684a 100644 --- a/turncat.go +++ b/turncat.go @@ -35,6 +35,8 @@ type TurncatConfig struct { Realm string // AuthGet specifies the function to generate auth tokens. AuthGen AuthGen + // ServerName is the SNI used for virtual hosting (unless it is an IP address). + ServerName string // InsecureMode controls whether self-signed TLS certificates are accepted by the TURN // client. InsecureMode bool @@ -53,6 +55,7 @@ type Turncat struct { connTrack map[string]*connection // Conntrack table. lock *sync.Mutex // Sync access to the conntrack state. authGen AuthGen // Generate auth tokens. + serverName string insecure bool loggerFactory logging.LoggerFactory log logging.LeveledLogger @@ -165,6 +168,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { lock: new(sync.Mutex), realm: config.Realm, authGen: config.AuthGen, + serverName: config.ServerName, insecure: config.InsecureMode, loggerFactory: loggerFactory, log: log, @@ -263,6 +267,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { // assert.NoError(t, err, "cannot create certificate for TLS client socket") c, err := tls.Dial("tcp", t.serverAddr.String(), &tls.Config{ MinVersion: tls.VersionTLS10, + ServerName: t.serverName, InsecureSkipVerify: t.insecure, }) if err != nil { From 904b35230e9ac55c0f7a3d81bbbdd04dcea73e76 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 11 Jun 2024 16:20:29 +0200 Subject: [PATCH 182/243] chore: Improve logging format consistency --- cmd/stunnerd/main.go | 38 ++++++++++---------- internal/object/errors.go | 2 +- internal/resolver/mock_test.go | 2 +- internal/resolver/resolver.go | 14 ++++---- pkg/apis/v1/errors.go | 8 ++--- pkg/config/client/client.go | 2 +- pkg/config/client/file_client.go | 36 +++++++++---------- pkg/config/client/k8s_client.go | 10 +++--- pkg/config/server/config.go | 17 +++++---- pkg/config/server/handler.go | 13 +++---- pkg/config/server/server.go | 24 ++++++------- reconcile.go | 32 ++++++++--------- relay.go | 8 ++--- server_unix_test.go | 8 ++--- stunner.go | 4 +-- turncat.go | 60 ++++++++++++++++---------------- 16 files changed, 137 insertions(+), 141 deletions(-) diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index ccab9821..744d1232 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -80,19 +80,19 @@ func main() { log := st.GetLogger().NewLogger("stunnerd") buildInfo := buildinfo.BuildInfo{Version: version, CommitHash: commitHash, BuildDate: buildDate} - log.Infof("starting stunnerd id %q, STUNner %s ", st.GetId(), buildInfo.String()) + log.Infof("Starting stunnerd id %q, STUNner %s ", st.GetId(), buildInfo.String()) conf := make(chan *stnrv1.StunnerConfig, 1) defer close(conf) var cancelConfigLoader context.CancelFunc if flag.NArg() == 1 { - log.Infof("starting %s with default configuration at TURN URI: %s", + log.Infof("Starting %s with default configuration at TURN URI: %s", os.Args[0], flag.Arg(0)) c, err := stunner.NewDefaultConfig(flag.Arg(0)) if err != nil { - log.Errorf("could not load default STUNner config: %s", err.Error()) + log.Errorf("Could not load default STUNner config: %s", err.Error()) os.Exit(1) } @@ -102,17 +102,17 @@ func main() { ctx, cancel := context.WithCancel(context.Background()) if configOrigin == "k8s" { - log.Info("discovering configuration from Kubernetes") + log.Info("Discovering configuration from Kubernetes") cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, st.GetLogger().NewLogger("cds-fwd")) if err != nil { - log.Errorf("error searching for CDS server: %s", err.Error()) + log.Errorf("Error searching for CDS server: %s", err.Error()) os.Exit(1) } configOrigin = cdsAddr.Addr } - log.Infof("loading configuration from origin %q", configOrigin) + log.Infof("Loading configuration from origin %q", configOrigin) c, err := st.LoadConfig(configOrigin) if err != nil { log.Error(err.Error()) @@ -123,7 +123,7 @@ func main() { conf <- c } else if *watch { - log.Info("bootstrapping with minimal config") + log.Info("Bootstrapping stunnerd with minimal config") z := cdsclient.ZeroConfig(st.GetId()) conf <- z @@ -132,19 +132,19 @@ func main() { cancelConfigLoader = cancel if configOrigin == "k8s" { - log.Info("discovering configuration from Kubernetes") + log.Info("Discovering configuration from Kubernetes") cdsAddr, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, st.GetLogger().NewLogger("cds-fwd")) if err != nil { - log.Errorf("error searching for CDS server: %s", err.Error()) + log.Errorf("Error searching for CDS server: %s", err.Error()) os.Exit(1) } configOrigin = cdsAddr.Addr } - log.Infof("watching configuration at origin %q", configOrigin) + log.Infof("Watching configuration at origin %q", configOrigin) if err := st.WatchConfig(ctx, configOrigin, conf); err != nil { - log.Errorf("could not run config watcher: %s", err.Error()) + log.Errorf("Could not run config watcher: %s", err.Error()) os.Exit(1) } } else { @@ -162,16 +162,16 @@ func main() { for { select { case <-exit: - log.Info("normal exit on graceful shutdown") + log.Info("Normal exit on graceful shutdown") os.Exit(0) case <-sigterm: - log.Infof("performing a graceful shutdown with %d active connection(s)", + log.Infof("Commencing graceful shutdown with %d active connection(s)", st.AllocationCount()) st.Shutdown() if cancelConfigLoader != nil { - log.Info("canceling config loader") + log.Info("Canceling config loader") cancelConfigLoader() cancelConfigLoader = nil } @@ -188,7 +188,7 @@ func main() { }() case c := <-conf: - log.Infof("new configuration available: %q", c.String()) + log.Infof("New configuration available: %q", c.String()) // command line loglevel overrides config if *verbose || *level != "" { @@ -196,14 +196,14 @@ func main() { } // we have working stunnerd: reconcile - log.Debug("initiating reconciliation") + log.Debug("Initiating reconciliation") err := st.Reconcile(c) - log.Trace("reconciliation ready") + log.Trace("Reconciliation ready") if err != nil { if e, ok := err.(stnrv1.ErrRestarted); ok { - log.Debugf("reconciliation ready: %s", e.Error()) + log.Debugf("Reconciliation ready: %s", e.Error()) } else { - log.Errorf("could not reconcile new configuration "+ + log.Errorf("Could not reconcile new configuration "+ "(running configuration unchanged): %s", err.Error()) } } diff --git a/internal/object/errors.go b/internal/object/errors.go index 2b984680..c7ee7542 100644 --- a/internal/object/errors.go +++ b/internal/object/errors.go @@ -4,5 +4,5 @@ import "errors" var ( // ErrRestartRequired indicates that an object needs to be restarted for reconciliation. - ErrRestartRequired = errors.New("restart required") + ErrRestartRequired = errors.New("Restart required") ) diff --git a/internal/resolver/mock_test.go b/internal/resolver/mock_test.go index c24f6484..8e3b0ab4 100644 --- a/internal/resolver/mock_test.go +++ b/internal/resolver/mock_test.go @@ -16,7 +16,7 @@ func TestMockResolver(t *testing.T) { loggerFactory := logger.NewLoggerFactory(resolverTestLoglevel) log := loggerFactory.NewLogger("resolver-test") - log.Debug("setting up the mock DNS") + log.Debug("Setting up the mock DNS") mockDns := NewMockResolver(map[string]([]string){ "stunner.l7mp.io": []string{"1.2.3.4"}, "echo-server.l7mp.io": []string{"1.2.3.5"}, diff --git a/internal/resolver/resolver.go b/internal/resolver/resolver.go index 568cbe96..7b7a305c 100644 --- a/internal/resolver/resolver.go +++ b/internal/resolver/resolver.go @@ -85,13 +85,13 @@ func (r *dnsResolverImpl) Register(domain string) error { // the resolver goroutine func startResolver(e *serviceEntry, log logging.LeveledLogger) { - log.Infof("resolver thread starting for domain %q, DNS update interval: %v", + log.Infof("Resolver thread starting for domain %q, DNS update interval: %v", e.domain, dnsUpdateInterval) if err := doResolve(e); err != nil { - log.Debugf("initial resolution failed for domain %q: %s", e.domain, err.Error()) + log.Debugf("Initial resolution failed for domain %q: %s", e.domain, err.Error()) } - log.Tracef("initial resolution ready for domain %q, found %d endpoints", e.domain, + log.Tracef("Initial resolution ready for domain %q, found %d endpoints", e.domain, len(e.hostNames)) ticker := time.NewTicker(dnsUpdateInterval) @@ -100,15 +100,15 @@ func startResolver(e *serviceEntry, log logging.LeveledLogger) { for { select { case <-e.ctx.Done(): - log.Debugf("resolver thread exiting for domain %q", e.domain) + log.Debugf("Resolver thread exiting for domain %q", e.domain) return case <-ticker.C: - log.Tracef("resolving for domain %q", e.domain) + log.Tracef("Resolving for domain %q", e.domain) if err := doResolve(e); err != nil { - log.Debugf("resolution failed for domain %q: %s", + log.Debugf("Resolution failed for domain %q: %s", e.domain, err.Error()) } - log.Tracef("periodic resolution ready for domain %q, found %d endpoints", e.domain, + log.Tracef("Periodic resolution ready for domain %q, found %d endpoints", e.domain, len(e.hostNames)) } } diff --git a/pkg/apis/v1/errors.go b/pkg/apis/v1/errors.go index 2b5862d3..a9b4f71b 100644 --- a/pkg/apis/v1/errors.go +++ b/pkg/apis/v1/errors.go @@ -7,10 +7,10 @@ import ( ) var ( - ErrInvalidConf = errors.New("invalid configuration") - ErrNoSuchListener = errors.New("no such listener") - ErrNoSuchCluster = errors.New("no such cluster") - // ErrInvalidRoute = errors.New("invalid route") + ErrInvalidConf = errors.New("Invalid configuration") + ErrNoSuchListener = errors.New("No such listener") + ErrNoSuchCluster = errors.New("No such cluster") + // ErrInvalidRoute = errors.New("Invalid route") ) type ErrRestarted struct { diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index 1b38605c..f1c2a863 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -11,7 +11,7 @@ import ( "github.com/pion/logging" ) -var errFileTruncated = errors.New("zero-length config file") +var errFileTruncated = errors.New("Zero-length config file") var ( // Send pings to the CDS server with this period. Must be less than PongWait. diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index dedd7a01..3f0e7b2c 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -68,14 +68,14 @@ func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { // periodically retry until the file appears. func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { if w.configFile == "" { - return errors.New("uninitialized config file path") + return errors.New("Uninitialized config file path") } go func() { for { // try to watch if err := w.Poll(ctx, ch); err != nil { - w.log.Warnf("error loading config file %q: %s", + w.log.Warnf("Error loading config file %q: %s", w.configFile, err.Error()) } else { return @@ -113,7 +113,7 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo return err } - w.log.Debugf("initial config file successfully loaded from %q: %s", config, c.String()) + w.log.Debugf("Initial config file successfully loaded from %q: %s", config, c.String()) ch <- c @@ -128,31 +128,31 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo case e, ok := <-watcher.Events: if !ok { - return errors.New("config watcher event handler received invalid event") + return errors.New("Config watcher received invalid event") } - w.log.Debugf("received watcher event: %s", e.String()) + w.log.Debugf("Received watch event: %s", e.String()) if e.Has(fsnotify.Remove) { if err := watcher.Remove(config); err != nil { - w.log.Debugf("could not remove config file %q watcher: %s", + w.log.Debugf("Could not remove config file %q watcher: %s", config, err.Error()) } - return fmt.Errorf("config file deleted %q, disabling watcher", e.Op.String()) + return fmt.Errorf("Config file deleted %q, disabling watcher", e.Op.String()) } if !e.Has(fsnotify.Write) { - w.log.Debugf("unhandled notify op on config file %q (ignoring): %s", + w.log.Debugf("Unhandled notify op on config file %q (ignoring): %s", e.Name, e.Op.String()) continue } - w.log.Debugf("loading configuration file: %s", config) + w.log.Debugf("Loading configuration file: %s", config) c, err := w.Load() if err != nil { if errors.Is(err, errFileTruncated) { - w.log.Debugf("ignoring: %s", err.Error()) + w.log.Debugf("Ignoring: %s", err.Error()) continue } return err @@ -160,11 +160,11 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo // suppress repeated events if c.DeepEqual(&prev) { - w.log.Debugf("ignoring recurrent notify event for the same config file") + w.log.Debugf("Ignoring recurrent notify event for the same config file") continue } - w.log.Debugf("config file successfully loaded from %q: %s", config, c.String()) + w.log.Debugf("Config file successfully loaded from %q: %s", config, c.String()) ch <- c @@ -173,15 +173,15 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo case err, ok := <-watcher.Errors: if !ok { - return errors.New("config watcher error handler received invalid error") + return errors.New("Config watcher error handler received invalid error") } if err := watcher.Remove(config); err != nil { - w.log.Debugf("could not remove config file %q watcher: %s", + w.log.Debugf("Could not remove config file %q watcher: %s", config, err.Error()) } - return fmt.Errorf("watcher error, deactivating watcher: %w", err) + return fmt.Errorf("Config watcher error, deactivating: %w", err) } } } @@ -202,16 +202,16 @@ func (w *ConfigFileClient) tryWatchConfig(ctx context.Context) bool { return false case <-ticker.C: - w.log.Debugf("trying to read config file %q from periodic timer", + w.log.Debugf("Trying to read config file %q from periodic timer", config) // check if config file exists and it is readable if _, err := os.Stat(config); errors.Is(err, os.ErrNotExist) { - w.log.Debugf("config file %q does not exist", config) + w.log.Debugf("Config file %q does not exist", config) // report status in every 10th second if time.Now().Second()%10 == 0 { - w.log.Warnf("waiting for config file %q", config) + w.log.Warnf("Waiting for config file %q", config) } continue diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go index fca5f310..f32db5a7 100644 --- a/pkg/config/client/k8s_client.go +++ b/pkg/config/client/k8s_client.go @@ -298,7 +298,7 @@ func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, p, err := d.portfwd(ctx, &pod, podFlags.Port) if err != nil { - d.log.Errorf("failed to create port-forwarder to stunnerd pod %s/%s: %s", + d.log.Errorf("Failed to create port-forwarder to stunnerd pod %s/%s: %s", pod.GetNamespace(), pod.GetName(), err.Error()) return } @@ -344,7 +344,7 @@ func DiscoverK8sAuthServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, au LabelSelector: label, }) if err != nil { - return PodInfo{}, fmt.Errorf("failed to query Kubernetes API server: %w", err) + return PodInfo{}, fmt.Errorf("Failed to query Kubernetes API server: %w", err) } if len(pods.Items) == 0 { @@ -352,7 +352,7 @@ func DiscoverK8sAuthServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, au } if len(pods.Items) > 1 { - d.log.Infof("mulitple (%d) authentication service instances found, using the first one", len(pods.Items)) + d.log.Infof("Mulitple (%d) authentication service instances found, using the first one", len(pods.Items)) } return d.portfwd(ctx, &pods.Items[0], authFlags.Port) @@ -387,12 +387,12 @@ func (d *PodConnector) portfwd(ctx context.Context, pod *corev1.Pod, port int) ( out, errOut := new(bytes.Buffer), new(bytes.Buffer) fw, err := portforward.New(dialer, []string{remoteAddr}, stopChan, readyChan, out, errOut) if err != nil { - return PodInfo{}, fmt.Errorf("error creating port-forwarder: %w", err) + return PodInfo{}, fmt.Errorf("Error creating port-forwarder: %w", err) } go func() { if err := fw.ForwardPorts(); err != nil { - d.log.Errorf("error setting up port-forwarder: %s", err.Error()) + d.log.Errorf("Error setting up port-forwarder: %s", err.Error()) os.Exit(1) } }() diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go index fe0b6c4a..9b0d3ca2 100644 --- a/pkg/config/server/config.go +++ b/pkg/config/server/config.go @@ -39,13 +39,12 @@ func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { func (s *Server) DeleteConfig(id string) { s.configs.Delete(id) if ConfigDeletionUpdateDelay == 0 { - s.log.Info("suppressing config update for deleted config", - "client", id) + s.log.Info("Suppressing config update for deleted config", "client", id) return } - s.log.Info("delaying config update for deleted config", - "client", id, "delay", ConfigDeletionUpdateDelay) + s.log.Info("Delaying config update for deleted config", "client", id, + "delay", ConfigDeletionUpdateDelay) s.deleteCh <- Config{Id: id, Config: client.ZeroConfig(id)} } @@ -53,7 +52,7 @@ func (s *Server) DeleteConfig(id string) { // UpdateConfig receives a set of ids and newConfigs that represent the state-of-the-world at a // particular instance of time and generates an update per each change. func (s *Server) UpdateConfig(newConfigs []Config) error { - s.log.V(4).Info("processing config updates", "num-configs", len(newConfigs)) + s.log.V(4).Info("Processing config updates", "num-configs", len(newConfigs)) oldConfigs := s.configs.Snapshot() for _, oldC := range oldConfigs { @@ -61,11 +60,11 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { for _, newC := range newConfigs { if oldC.Id == newC.Id { if !oldC.Config.DeepEqual(newC.Config) { - s.log.V(2).Info("updating config", "client", newC.Id, "config", + s.log.V(2).Info("Updating config", "client", newC.Id, "config", newC.Config.String()) s.UpsertConfig(newC.Id, newC.Config) } else { - s.log.V(2).Info("config not updated", "client", newC.Id, + s.log.V(2).Info("Config not updated", "client", newC.Id, "old-config", oldC.Config.String(), "new-config", newC.Config.String()) } @@ -75,7 +74,7 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { } if !found { - s.log.V(2).Info("removing config", "client", oldC.Id) + s.log.V(2).Info("Removing config", "client", oldC.Id) s.DeleteConfig(oldC.Id) } } @@ -90,7 +89,7 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { } if !found { - s.log.V(2).Info("adding config", "client", newC.Id, "config", newC.Config) + s.log.V(2).Info("Adding config", "client", newC.Id, "config", newC.Config) s.UpsertConfig(newC.Id, newC.Config) } } diff --git a/pkg/config/server/handler.go b/pkg/config/server/handler.go index 5f72e2e6..c2e5892d 100644 --- a/pkg/config/server/handler.go +++ b/pkg/config/server/handler.go @@ -22,7 +22,7 @@ type ConfigPatcher func(conf *stnrv1.StunnerConfig, node string) (*stnrv1.Stunne // (GET /api/v1/configs) func (s *Server) ListV1Configs(ctx context.Context, request api.ListV1ConfigsRequestObject) (api.ListV1ConfigsResponseObject, error) { - s.log.V(1).Info("handling ListV1Configs API call") + s.log.V(1).Info("Handling ListV1Configs API call") configs := s.configs.Snapshot() response := ConfigList{Version: "v1", Items: []stnrv1.StunnerConfig{}} @@ -32,16 +32,14 @@ func (s *Server) ListV1Configs(ctx context.Context, request api.ListV1ConfigsReq response.Items = append(response.Items, cpy) } - s.log.V(3).Info("ListV1Configs API handler: ready", - "configlist-len", len(configs)) + s.log.V(3).Info("ListV1Configs API handler: ready", "configlist-len", len(configs)) return api.ListV1Configs200JSONResponse(response), nil } // (GET /api/v1/configs/{namespace}) func (s *Server) ListV1ConfigsNamespace(ctx context.Context, request api.ListV1ConfigsNamespaceRequestObject) (api.ListV1ConfigsNamespaceResponseObject, error) { - s.log.V(1).Info("handling ListV1ConfigsNamespace API call", - "namespace", request.Namespace) + s.log.V(1).Info("Handling ListV1ConfigsNamespace API call", "namespace", request.Namespace) configs := s.configs.Snapshot() response := ConfigList{Version: "v1", Items: []stnrv1.StunnerConfig{}} @@ -54,8 +52,7 @@ func (s *Server) ListV1ConfigsNamespace(ctx context.Context, request api.ListV1C } } - s.log.V(3).Info("ListV1ConfigsNamespace API handler: ready", - "configlist-len", len(configs)) + s.log.V(3).Info("ListV1ConfigsNamespace API handler: ready", "configlist-len", len(configs)) return api.ListV1ConfigsNamespace200JSONResponse(response), nil } @@ -63,7 +60,7 @@ func (s *Server) ListV1ConfigsNamespace(ctx context.Context, request api.ListV1C // (GET /api/v1/configs/{namespace}/{name}) func (s *Server) GetV1ConfigNamespaceName(ctx context.Context, request api.GetV1ConfigNamespaceNameRequestObject) (api.GetV1ConfigNamespaceNameResponseObject, error) { namespace, name := request.Namespace, request.Name - s.log.V(1).Info("handling GetV1ConfigNamespaceName API call", "namespace", namespace, + s.log.V(1).Info("Handling GetV1ConfigNamespaceName API call", "namespace", namespace, "name", name) id := fmt.Sprintf("%s/%s", namespace, name) diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index b900913e..f554cf52 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -62,14 +62,14 @@ func (s *Server) Start(ctx context.Context) error { s.Server = &http.Server{Addr: s.addr, Handler: s.router} go func() { - s.log.Info("starting CDS server", "address", s.addr, "patch", s.patch != nil) + s.log.Info("Starting CDS server", "address", s.addr, "patch", s.patch != nil) err := s.ListenAndServe() if err != nil { if errors.Is(err, net.ErrClosed) || errors.Is(err, http.ErrServerClosed) { - s.log.Info("closing config discovery server") + s.log.Info("Closing config discovery server") } else { - s.log.Error(err, "error closing config discovery server", "error", err.Error()) + s.log.Error(err, "Error closing config discovery server", "error", err.Error()) } return } @@ -86,7 +86,7 @@ func (s *Server) Start(ctx context.Context) error { s.broadcastConfig(c) case c := <-s.deleteCh: - s.log.V(2).Info("initiating deleyed config deletion", "id", c.Id) + s.log.V(2).Info("Initiating deleyed config deletion", "id", c.Id) go func() { select { @@ -136,7 +136,7 @@ func (s *Server) GetConnTrack() *ConnTrack { // RemoveClient forcefully closes a client connection. This is used mainly for testing. func (s *Server) RemoveClient(id string) { if c := s.conns.Get(id); c != nil { - s.log.V(1).Info("forcefully removing client connection", "client", id) + s.log.V(1).Info("Forcefully removing client connection", "client", id) s.closeConn(c) } } @@ -165,7 +165,7 @@ func (s *Server) handleConn(reqCtx context.Context, wsConn *websocket.Conn, oper return conn.WriteMessage(websocket.PongMessage, []byte("keepalive")) }) - s.log.V(1).Info("new config stream connection", "api", operationID, "client", conn.Id()) + s.log.V(1).Info("New config stream connection", "api", operationID, "client", conn.Id()) // send initial config(list) for _, conf := range s.configs.Snapshot() { @@ -178,7 +178,7 @@ func (s *Server) handleConn(reqCtx context.Context, wsConn *websocket.Conn, oper // the running connections) <-ctx.Done() - s.log.V(1).Info("client connection closed", "api", operationID, "client", conn.Id()) + s.log.V(1).Info("Client connection closed", "api", operationID, "client", conn.Id()) conn.Close() } @@ -199,7 +199,7 @@ func (s *Server) sendConfig(conn *Conn, e *stnrv1.StunnerConfig) { if conn.patch != nil { newC, err := conn.patch(c) if err != nil { - s.log.Error(err, "cannot patch config", "event", e.String()) + s.log.Error(err, "Cannot patch config", "event", e.String()) return } c = newC @@ -207,21 +207,21 @@ func (s *Server) sendConfig(conn *Conn, e *stnrv1.StunnerConfig) { json, err := json.Marshal(c) if err != nil { - s.log.Error(err, "cannot JSON serialize config", "event", e.String()) + s.log.Error(err, "Cannot JSON serialize config", "event", e.String()) return } - s.log.V(2).Info("sending configuration to client", "client", conn.Id(), + s.log.V(2).Info("Sending configuration to client", "client", conn.Id(), "config", c.String()) if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { - s.log.Error(err, "error sending config update", "client", conn.Id()) + s.log.Error(err, "Error sending config update", "client", conn.Id()) s.closeConn(conn) } } func (s *Server) closeConn(conn *Conn) { - s.log.V(1).Info("closing client connection", "client", conn.Id()) + s.log.V(1).Info("Closing client connection", "client", conn.Id()) conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck diff --git a/reconcile.go b/reconcile.go index 165ef456..20c4fa5b 100644 --- a/reconcile.go +++ b/reconcile.go @@ -27,7 +27,7 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo return err } - s.log.Debugf("reconciling STUNner for config: %s ", req.String()) + s.log.Debugf("Reconciling STUNner for config: %s ", req.String()) rollback := s.GetConfig() toBeStarted, toBeRestarted := []object.Object{}, []object.Object{} @@ -85,7 +85,7 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo // find all objects (listeners) to be restarted and stop each if !s.dryRun { if err := s.stop(toBeRestarted); err != nil { - s.log.Errorf("could not stop object: %s", err.Error()) + s.log.Errorf("Could not stop object: %s", err.Error()) errFinal = err if !inRollback { goto rollback @@ -94,13 +94,13 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo } } - s.log.Tracef("reconciliation preparation ready") + s.log.Tracef("Reconciliation preparation ready") // finish reconciliation // admin err = s.adminManager.FinishReconciliation(adminState) if err != nil { - s.log.Errorf("could not reconcile admin config: %s", err.Error()) + s.log.Errorf("Could not reconcile admin config: %s", err.Error()) errFinal = err if !inRollback { goto rollback @@ -109,13 +109,13 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo } toBeStarted = append(toBeStarted, adminState.ToBeStarted...) - s.log.Infof("setting loglevel to %q", s.GetAdmin().LogLevel) + s.log.Infof("Setting loglevel to %q", s.GetAdmin().LogLevel) s.logger.SetLevel(s.GetAdmin().LogLevel) // auth err = s.authManager.FinishReconciliation(authState) if err != nil { - s.log.Errorf("could not reconcile auth config: %s", err.Error()) + s.log.Errorf("Could not reconcile auth config: %s", err.Error()) errFinal = err if !inRollback { goto rollback @@ -127,7 +127,7 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo // listener err = s.listenerManager.FinishReconciliation(listenerState) if err != nil { - s.log.Errorf("could not reconcile listener config: %s", err.Error()) + s.log.Errorf("Could not reconcile listener config: %s", err.Error()) errFinal = err if !inRollback { goto rollback @@ -137,13 +137,13 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo toBeStarted = append(toBeStarted, listenerState.ToBeStarted...) if len(s.listenerManager.Keys()) == 0 { - s.log.Warn("running with no listeners") + s.log.Warn("Running with no listeners") } // cluster err = s.clusterManager.FinishReconciliation(clusterState) if err != nil { - s.log.Errorf("could not reconcile cluster config: %s", err.Error()) + s.log.Errorf("Could not reconcile cluster config: %s", err.Error()) errFinal = err if !inRollback { goto rollback @@ -153,13 +153,13 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo toBeStarted = append(toBeStarted, clusterState.ToBeStarted...) if len(s.clusterManager.Keys()) == 0 { - s.log.Warn("running with no clusters: all traffic will be dropped") + s.log.Warn("Running with no clusters: all traffic will be dropped") } // find all objects (listeners) to be started or restarted and start each if !s.dryRun { if err := s.start(toBeStarted, toBeRestarted); err != nil { - s.log.Errorf("could not start object: %s", err.Error()) + s.log.Errorf("Could not start object: %s", err.Error()) errFinal = err if !inRollback { goto rollback @@ -173,11 +173,11 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo s.ready = true } - s.log.Infof("reconciliation ready: new objects: %d, changed objects: %d, "+ + s.log.Infof("Reconciliation ready: new objects: %d, changed objects: %d, "+ "deleted objects: %d, started objects: %d, restarted objects: %d", new, changed, deleted, len(toBeStarted), len(toBeRestarted)) - s.log.Infof("new dataplane status: %s", s.Status().String()) + s.log.Infof("New dataplane status: %s", s.Status().String()) if len(toBeRestarted) > 0 { names := make([]string, len(toBeRestarted)) @@ -192,7 +192,7 @@ func (s *Stunner) reconcileWithRollback(req *stnrv1.StunnerConfig, inRollback bo rollback: if !s.suppressRollback { - s.log.Infof("rolling back to previous configuration: %s", rollback.String()) + s.log.Infof("Rolling back to previous configuration: %s", rollback.String()) return s.reconcileWithRollback(rollback, true) } @@ -207,7 +207,7 @@ func (s *Stunner) stop(restarted []object.Object) error { return err } default: - s.log.Errorf("internal error: stop() is not implemented for object %q", + s.log.Errorf("Internal error: stop() is not implemented for object %q", o.ObjectName()) } } @@ -223,7 +223,7 @@ func (s *Stunner) start(started, restarted []object.Object) error { return err } default: - s.log.Errorf("internal error: start() is not implemented for object %q", + s.log.Errorf("Internal error: start() is not implemented for object %q", o.ObjectName()) } } diff --git a/relay.go b/relay.go index 14ac6b45..953cfe68 100644 --- a/relay.go +++ b/relay.go @@ -21,13 +21,13 @@ import ( const ClusterCacheSize = 512 var ( - errNilConn = errors.New("cannot allocate relay connection") - errTodo = errors.New("relay to Net.Conn not implemented") + errNilConn = errors.New("Cannot allocate relay connection") + errTodo = errors.New("Relay to Net.Conn not implemented") ) var ( - ErrPortProhibited = errors.New("peer port administratively prohibited") - ErrInvalidPeerProtocol = errors.New("unknown peer transport protocol") + ErrPortProhibited = errors.New("Peer port administratively prohibited") + ErrInvalidPeerProtocol = errors.New("Unknown peer transport protocol") ) type PortRangeChecker = func(addr net.Addr) (*object.Cluster, bool) diff --git a/server_unix_test.go b/server_unix_test.go index 1075ea44..a00d5424 100644 --- a/server_unix_test.go +++ b/server_unix_test.go @@ -60,7 +60,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { initSeq := []byte("init-data") testSeq := []byte("benchmark-data") - log.Debug("creating a stunnerd") + log.Debug("Creating a stunnerd") stunner := NewStunner(Options{ LogLevel: stunnerTestLoglevel, SuppressRollback: true, @@ -68,7 +68,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { }) defer stunner.Close() - log.Debug("starting stunnerd") + log.Debug("Starting stunnerd") err := stunner.Reconcile(&stnrv1.StunnerConfig{ ApiVersion: stnrv1.ApiVersion, Admin: stnrv1.AdminConfig{ @@ -100,7 +100,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { b.Fatalf("Failed to start stunnerd: %s", err) } - log.Debug("creating a sink") + log.Debug("Creating a sink") sinkAddr, err := net.ResolveUDPAddr("udp4", "0.0.0.0:65432") if err != nil { b.Fatalf("Failed to resolve sink address: %s", err) @@ -150,7 +150,7 @@ func RunBenchmarkServer(b *testing.B, proto string, udpThreadNum int) { defer turncat.Close() // test with 20 clients - log.Debugf("creating %d senders", clientNum) + log.Debugf("Creating %d senders", clientNum) clients := make([]net.Conn, clientNum) for i := 0; i < clientNum; i++ { var client net.Conn diff --git a/stunner.go b/stunner.go index 0cd23bf7..acc1a73a 100644 --- a/stunner.go +++ b/stunner.go @@ -57,13 +57,13 @@ func NewStunner(options Options) *Stunner { if options.Net == nil { net, err := stdnet.NewNet() // defaults to native operation if err != nil { - log.Error("could not create stdnet.NewNet") + log.Error("Could not create vnet") return nil } vnet = net } else { vnet = options.Net - log.Warn("vnet is enabled") + log.Warn("Virtual net (vnet) is enabled") } udpThreadNum := 0 diff --git a/turncat.go b/turncat.go index 792d684a..c9e8a649 100644 --- a/turncat.go +++ b/turncat.go @@ -186,11 +186,11 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { // client connection is file go t.runListenFile() default: - t.log.Errorf("internal error: unknown client protocol %q for client %s:%s", + t.log.Errorf("Internal error: unknown client protocol %q for client %s:%s", listenerAddress.Network(), listenerAddress.Network(), listenerAddress.String()) } - log.Infof("Turncat client listening on %s, TURN server: %s, peer: %s:%s", + log.Infof("Client listening on %s, TURN server: %s, peer: %s:%s", config.ListenerAddr, config.ServerAddr, peerAddress.Network(), peerAddress.String()) @@ -200,7 +200,7 @@ func NewTurncat(config *TurncatConfig) (*Turncat, error) { // Close terminates all relay connections created via turncat and deletes the relay. Errors in this // phase are not critical and not propagated back to the caller. func (t *Turncat) Close() { - t.log.Info("closing Turncat") + t.log.Info("Closing Turncat") // close all active connections for _, conn := range t.connTrack { @@ -210,28 +210,28 @@ func (t *Turncat) Close() { // close the global listener socket switch t.listenerConn.(type) { case net.Listener: - t.log.Tracef("closing turncat listener connection") + t.log.Tracef("Closing turncat listener connection") l := t.listenerConn.(net.Listener) if err := l.Close(); err != nil { - t.log.Warnf("error closing listener connection: %s", err.Error()) + t.log.Warnf("Error closing listener connection: %s", err.Error()) } case net.PacketConn: - t.log.Tracef("closing turncat packet listener connection") + t.log.Tracef("Closing turncat packet listener connection") l := t.listenerConn.(net.PacketConn) if err := l.Close(); err != nil { - t.log.Warnf("error closing listener packet connection: %s", err.Error()) + t.log.Warnf("Error closing listener packet connection: %s", err.Error()) } case *util.FileConn: // do nothing default: - t.log.Error("internal error: unknown listener socket type") + t.log.Error("Internal error: unknown listener socket type") } } // Generate a new connection by opening a UDP connection to the server func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { clientAddr := clientConn.RemoteAddr() - t.log.Debugf("new connection from client %s", clientAddr.String()) + t.log.Debugf("New connection from client %s", clientAddr.String()) conn := new(connection) conn.clientAddr = clientAddr @@ -241,7 +241,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { user, passwd, errAuth := t.authGen() if errAuth != nil { - return nil, fmt.Errorf("cannot generate username/password pair for client %s:%s: %s", + return nil, fmt.Errorf("Cannot generate username/password pair for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), errAuth) } @@ -303,7 +303,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { }) if err != nil { turnConn.Close() - return nil, fmt.Errorf("cannot allocate TURN client for client %s:%s: %s", + return nil, fmt.Errorf("Cannot allocate TURN client for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } conn.turnConn = turnConn @@ -327,7 +327,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { // The relayConn's local address is actually the transport // address assigned on the TURN server. - t.log.Infof("new connection: client-address=%s, relayed-address=%s", + t.log.Infof("New connection: client-address=%s, relayed-address=%s", clientAddr.String(), conn.serverConn.LocalAddr().String()) return conn, nil @@ -347,14 +347,14 @@ func (t *Turncat) deleteConnection(conn *connection) { delete(t.connTrack, caddr) t.lock.Unlock() - t.log.Infof("closing client connection to %s", caddr) + t.log.Infof("Closing client connection to %s", caddr) if err := conn.clientConn.Close(); err != nil { - t.log.Warnf("error closing client connection for %s:%s: %s", + t.log.Warnf("Error closing client connection for %s:%s: %s", conn.clientAddr.Network(), conn.clientAddr.String(), err.Error()) } if err := conn.serverConn.Close(); err != nil { - t.log.Warnf("error closing relayed TURN server connection for %s:%s: %s", + t.log.Warnf("Error closing relayed TURN server connection for %s:%s: %s", conn.clientAddr.Network(), conn.clientAddr.String(), err.Error()) } @@ -371,7 +371,7 @@ func (t *Turncat) runConnection(conn *connection) { n, peerAddr, readErr := conn.serverConn.ReadFrom(buffer[0:]) if readErr != nil { if !util.IsClosedErr(readErr) { - t.log.Debugf("cannot read from TURN relay connection for client %s:%s: %s", + t.log.Debugf("Cannot read from TURN relay connection for client %s:%s: %s", conn.clientAddr.Network(), conn.clientAddr.String(), readErr.Error()) t.deleteConnection(conn) } @@ -380,7 +380,7 @@ func (t *Turncat) runConnection(conn *connection) { // TODO: not sure if this is the recommended way to compare net.Addrs if peerAddr.Network() != t.peerAddr.Network() || peerAddr.String() != t.peerAddr.String() { - t.log.Debugf("received packet of %d bytes from unknown peer %s:%s (expected: "+ + t.log.Debugf("Received packet of %d bytes from unknown peer %s:%s (expected: "+ "%s:%s) on TURN relay connection for client %s:%s: ignoring", n, peerAddr.Network(), peerAddr.String(), t.peerAddr.Network(), t.peerAddr.String(), @@ -388,12 +388,12 @@ func (t *Turncat) runConnection(conn *connection) { continue } - t.log.Tracef("forwarding packet of %d bytes from peer %s:%s on TURN relay connection "+ + t.log.Tracef("Forwarding packet of %d bytes from peer %s:%s on TURN relay connection "+ "for client %s:%s", n, peerAddr.Network(), peerAddr.String(), conn.clientAddr.Network(), conn.clientAddr.String()) if _, writeErr := conn.clientConn.Write(buffer[0:n]); writeErr != nil { - t.log.Debugf("cannot write to client connection for client %s:%s: %s", + t.log.Debugf("Cannot write to client connection for client %s:%s: %s", conn.clientAddr.Network(), conn.clientAddr.String(), writeErr.Error()) t.deleteConnection(conn) return @@ -408,19 +408,19 @@ func (t *Turncat) runConnection(conn *connection) { n, readErr := conn.clientConn.Read(buffer[0:]) if readErr != nil { if !util.IsClosedErr(readErr) { - t.log.Debugf("cannot read from client connection for client %s:%s (likely hamrless): %s", + t.log.Debugf("Cannot read from client connection for client %s:%s (likely hamrless): %s", conn.clientAddr.Network(), conn.clientAddr.String(), readErr.Error()) t.deleteConnection(conn) } return } - t.log.Tracef("forwarding packet of %d bytes from client %s:%s to peer %s:%s on TURN relay connection", + t.log.Tracef("Forwarding packet of %d bytes from client %s:%s to peer %s:%s on TURN relay connection", n, conn.clientAddr.Network(), conn.clientAddr.String(), t.peerAddr.Network(), t.peerAddr.String()) if _, writeErr := conn.serverConn.WriteTo(buffer[0:n], t.peerAddr); writeErr != nil { - t.log.Debugf("cannot write to TURN relay connection for client %s (likely harmless): %s", + t.log.Debugf("Cannot write to TURN relay connection for client %s (likely harmless): %s", conn.clientAddr.String(), writeErr.Error()) t.deleteConnection(conn) return @@ -432,7 +432,7 @@ func (t *Turncat) runConnection(conn *connection) { func (t *Turncat) runListenPacket() { listenerConn, ok := t.listenerConn.(net.PacketConn) if !ok { - t.log.Error("cannot listen on client connection: expected net.PacketConn") + t.log.Error("Cannot listen on client connection: expected net.PacketConn") // terminate go routine return } @@ -442,7 +442,7 @@ func (t *Turncat) runListenPacket() { n, clientAddr, err := listenerConn.ReadFrom(buffer[0:]) if err != nil { if !util.IsClosedErr(err) { - t.log.Warnf("cannot read from listener connection: %s", err.Error()) + t.log.Warnf("Cannot read from listener connection: %s", err.Error()) } return } @@ -452,15 +452,15 @@ func (t *Turncat) runListenPacket() { caddr := fmt.Sprintf("%s:%s", clientAddr.Network(), clientAddr.String()) trackConn, found := t.connTrack[caddr] if !found { - t.log.Tracef("new client connection: read initial packet of %d bytes on listener"+ + t.log.Tracef("New client connection: read initial packet of %d bytes on listener"+ "connnection from client %s", n, caddr) // create per-client connection, connect back to client, then call runConnection - t.log.Tracef("connnecting back to client %s", caddr) + t.log.Tracef("Connnecting back to client %s", caddr) dialer := &net.Dialer{LocalAddr: t.listenerAddr, Control: reuseAddr} clientConn, clientErr := dialer.Dial(clientAddr.Network(), clientAddr.String()) if clientErr != nil { - t.log.Warnf("cannot connect back to client %s:%s: %s", + t.log.Warnf("Cannot connect back to client %s:%s: %s", clientAddr.Network(), clientAddr.String(), clientErr.Error()) continue } @@ -468,7 +468,7 @@ func (t *Turncat) runListenPacket() { conn, err := t.newConnection(clientConn) if err != nil { t.lock.Unlock() - t.log.Warnf("relay setup failed for client %s: %s", caddr, err.Error()) + t.log.Warnf("Relay setup failed for client %s: %s", caddr, err.Error()) continue } @@ -481,7 +481,7 @@ func (t *Turncat) runListenPacket() { // and send the packet out if _, err := conn.serverConn.WriteTo(buffer[0:n], t.peerAddr); err != nil { - t.log.Warnf("cannot write initial packet to TURN relay connection for client %s: %s", + t.log.Warnf("Cannot write initial packet to TURN relay connection for client %s: %s", caddr, err.Error()) t.deleteConnection(conn) continue @@ -492,7 +492,7 @@ func (t *Turncat) runListenPacket() { // packets are left stuck in the global listener socket t.lock.Unlock() - t.log.Debugf("received packet from a known client %s on the global listener connection, sender too fast?", + t.log.Debugf("Received packet from a known client %s on the global listener connection, sender too fast?", caddr) // send out anyway if _, err := trackConn.serverConn.WriteTo(buffer[0:n], t.peerAddr); err != nil { From 012f8c008ada522ee9132020b4713880827c359c Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sat, 15 Jun 2024 13:38:42 +0200 Subject: [PATCH 183/243] fix: Prevent concurrent WS socket write errors in the CDS client --- pkg/config/client/cds_api.go | 5 ++++- pkg/config/server/conn.go | 26 ++++++-------------------- pkg/config/util/conn.go | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+), 21 deletions(-) create mode 100644 pkg/config/util/conn.go diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go index f83ea86e..39103e82 100644 --- a/pkg/config/client/cds_api.go +++ b/pkg/config/client/cds_api.go @@ -12,6 +12,7 @@ import ( "github.com/gorilla/websocket" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" "github.com/l7mp/stunner/pkg/config/client/api" + "github.com/l7mp/stunner/pkg/config/util" "github.com/pion/logging" ) @@ -268,10 +269,12 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error _, url := a.Endpoint() a.Tracef("poll: trying to open connection to CDS server at %s", url) - conn, _, err := websocket.DefaultDialer.DialContext(ctx, url, makeHeader(url)) + wc, _, err := websocket.DefaultDialer.DialContext(ctx, url, makeHeader(url)) if err != nil { return err } + // wrap with a locker to prevent concurrent writes + conn := util.NewConn(wc) defer conn.Close() // this will close the poller goroutine a.Infof("connection successfully opened to config discovery server at %s", url) diff --git a/pkg/config/server/conn.go b/pkg/config/server/conn.go index 74a962f1..2d143b29 100644 --- a/pkg/config/server/conn.go +++ b/pkg/config/server/conn.go @@ -7,23 +7,23 @@ import ( "github.com/gorilla/websocket" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/config/util" ) type ClientConfigPatcher func(conf *stnrv1.StunnerConfig) (*stnrv1.StunnerConfig, error) // Conn represents a client WebSocket connection. type Conn struct { - *websocket.Conn - Filter ConfigFilter - patch ClientConfigPatcher - cancel context.CancelFunc - readLock, writeLock sync.Mutex // for writemessage + *util.Conn + Filter ConfigFilter + patch ClientConfigPatcher + cancel context.CancelFunc } // NewConn wraps a WebSocket connection. func NewConn(conn *websocket.Conn, filter ConfigFilter, patch ClientConfigPatcher, cancel context.CancelFunc) *Conn { return &Conn{ - Conn: conn, + Conn: util.NewConn(conn), Filter: filter, patch: patch, cancel: cancel, @@ -35,20 +35,6 @@ func (c *Conn) Id() string { return fmt.Sprintf("%s:%s", c.RemoteAddr().Network(), c.RemoteAddr().String()) } -// WriteMessage writes a message to the client connection with proper locking. -func (c *Conn) WriteMessage(messageType int, data []byte) error { - c.writeLock.Lock() - defer c.writeLock.Unlock() - return c.Conn.WriteMessage(messageType, data) -} - -// ReadMessage reads a message from the client connection with proper locking. -func (c *Conn) ReadMessage() (int, []byte, error) { - c.readLock.Lock() - defer c.readLock.Unlock() - return c.Conn.ReadMessage() -} - // ConnTrack represents the server's connection tracking table. type ConnTrack struct { conns []*Conn diff --git a/pkg/config/util/conn.go b/pkg/config/util/conn.go new file mode 100644 index 00000000..f6171573 --- /dev/null +++ b/pkg/config/util/conn.go @@ -0,0 +1,34 @@ +package util + +import ( + "sync" + + "github.com/gorilla/websocket" +) + +// Conn represents a client WebSocket connection. An added lock guards the underlying connection +// from concurrent write to websocket connection errors. +type Conn struct { + *websocket.Conn + readLock, writeLock sync.Mutex // for writemessage + +} + +// NewConn wraps a WebSocket connection. +func NewConn(conn *websocket.Conn) *Conn { + return &Conn{Conn: conn} +} + +// WriteMessage writes a message to the client connection with proper locking. +func (c *Conn) WriteMessage(messageType int, data []byte) error { + c.writeLock.Lock() + defer c.writeLock.Unlock() + return c.Conn.WriteMessage(messageType, data) +} + +// ReadMessage reads a message from the client connection with proper locking. +func (c *Conn) ReadMessage() (int, []byte, error) { + c.readLock.Lock() + defer c.readLock.Unlock() + return c.Conn.ReadMessage() +} From b11bdf46f8ecc5209c0df9a80254b069e36fbfb6 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 27 Jun 2024 15:20:23 +0200 Subject: [PATCH 184/243] chore: Update static manifests --- deploy/manifests/static/stunner-crd.yaml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/deploy/manifests/static/stunner-crd.yaml b/deploy/manifests/static/stunner-crd.yaml index f4a60abc..db856cb4 100644 --- a/deploy/manifests/static/stunner-crd.yaml +++ b/deploy/manifests/static/stunner-crd.yaml @@ -1464,6 +1464,15 @@ spec: type: array type: object type: object + annotations: + additionalProperties: + type: string + description: |- + Custom annotations to add to dataplane pods. Note that this does not affect the + annotations added to the Deployment (this come from the correspnding Gateway), just the + pods. Note also that mandatory pod annotations override whatever you set here on + conflict, and the annotations set here override annotations manually added to the pods. + type: object args: description: Arguments to the entrypoint. items: @@ -1838,6 +1847,15 @@ spec: type: object x-kubernetes-map-type: atomic type: array + labels: + additionalProperties: + type: string + description: |- + Custom labels to add to dataplane pods. Note that this does not affect the labels added + to the Deployment (those come from the Gateway), just the pods. Note also that mandatory + pod labels override whatever you set here on conflict. The only way to set pod labels is + here: whatever you set manually on the dataplane pod will be reset by the opetator. + type: object replicas: description: |- Number of desired pods. If empty or set to 1, use whatever is in the target Deployment. From 96c0ce256c1a17203959284b1a0fecdfcd441428 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 27 Jun 2024 15:22:08 +0200 Subject: [PATCH 185/243] doc: Document stunnerd pod label/annotation customization --- docs/GATEWAY.md | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 8b7ff618..43b4dbb6 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -181,7 +181,7 @@ For TURN-TLS/TURN-DTLS listeners, `tls.mode` must be set to `Terminate` or omitt STUNner will automatically generate a Kubernetes LoadBalancer Service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN over UDP and TURN over DTLS listeners are exposed as UDP services, TURN-TCP and TURN-TLS listeners are exposed as TCP. -STUNner implements two ways to customize the automatically created Service, both involving certain pre-defined [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations). First, you can add global annotations to the `loadBalancerServiceAnnotations` field of the [GatewayConfig spec](#gatewayconfig), which affect the Service created for each Gateway that links to the GatewayConfig (via the GatewayClass `parametersRef`). To customize annotations on a per-gateway status, you can also add specific annotations to the Gateway itself. Gateway annotations override the global annotations on conflict. +STUNner implements several ways to customize the automatically created Service, each involving certain pre-defined [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations). First, you can add global annotations to the `loadBalancerServiceAnnotations` field of the [GatewayConfig spec](#gatewayconfig), which affect the Service created for each Gateway that links to the GatewayConfig (via the GatewayClass `parametersRef`). To customize annotations on a per-gateway status, you can also add specific annotations to the Gateway itself. Gateway annotations override the global annotations on conflict. Finally, you can also add custom labels/annotations to the automatically created STUNner Services manually, these are retained on Service update (unless there is a conflict). Note that labels/annotations added via the Gateway and the GatewayConfig are also propagated to the corresponding `stunnerd` Deployments. The following rules apply: - Each annotation is copied verbatim into the Service created for any Gateway. This can be used, for instance, to specify health-check settings on the load-balancer Service (using the `service.beta.kubernetes.io/*-loadbalancer-healthcheck-*` annotations, see above). @@ -360,26 +360,28 @@ spec: The following fields can be set in the Dataplane `spec` to customize the provisioning of `stunnerd` pods. -| Field | Type | Description | Required | -|:--------------------------------|:----------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| -| `image` | `string` | The container image. | Yes | -| `imagePullPolicy` | `string` | Policy for if/when to pull the [`stunnerd`](/cmd/stunnerd/README.md), either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if the `latest` tag is specified on the image, `IfNotPresent` otherwise. | No | -| `command` | `list` | Entrypoints for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | -| `args` | `list` | Command line arguments for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | -| `envFrom` | `list` | List of sources to populate environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | -| `env` | `list` | List of environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | -| `containerSecurityContext` | `object` | Container-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | -| `replicas` | `int` | Number of dataplane pods per Gateway to provision. Not enforced if the [dataplane](/cmd/stunnerd/README.md) Deployment replica count is overwritten manually or by an autoscaler. Default: 1. | No | -| `imagePullSecrets` | `list` | List of Secret references to use for pulling the `stunnerd` image. Each ref is a secret name, namespace is the same as that of the Gateway on behalf of which the dataplane is deployed. | No | -| `hostNetwork` | `bool` | Deploy the [dataplane](/cmd/stunnerd/README.md) into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | -| `resources` | `object` | Compute resources per [dataplane](/cmd/stunnerd/README.md) pod. Default: none. | No | -| `affinity` | `object` | Scheduling constraints for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | -| `tolerations` | `object` | Tolerations for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | -| `securityContext` | `object` | Pod-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | -| `topologySpreadConstraints` | `object` | Description of how the [dataplane](/cmd/stunnerd/README.md) pods for a Gateway ought to spread across topology domains. Default: none. | No | -| `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | -| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | -| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | +| Field | Type | Description | Required | +|:--------------------------------|:----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| +| `image` | `string` | The container image. | Yes | +| `imagePullPolicy` | `string` | Policy for if/when to pull the [`stunnerd`](/cmd/stunnerd/README.md), either `Always`, `Never`, or `IfNotPresent`. Default: `Always` if the `latest` tag is specified on the image, `IfNotPresent` otherwise. | No | +| `imagePullSecrets` | `list` | List of Secret references to use for pulling the `stunnerd` image. Each ref is a secret name, namespace is the same as that of the Gateway on behalf of which the dataplane is deployed. | No | +| `labels` | `map` | Custom labels added to `stunnerd` pods. Mandatory labels override whatever is set here. Changing pod labels triggers full dataplane restart for the affected Gateways. | No | +| `annotations` | `map` | Custom annotations added to `stunnerd` pods. Mandatory annotations override whatever is set here, which in turn override manually added annotations. Changes trigger full dataplane restart for the affected Gateways. | No | +| `command` | `list` | Entrypoints for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | +| `args` | `list` | Command line arguments for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | +| `envFrom` | `list` | List of sources to populate environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | +| `env` | `list` | List of environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | +| `containerSecurityContext` | `object` | Container-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `replicas` | `int` | Number of dataplane pods per Gateway to provision. Not enforced if the [dataplane](/cmd/stunnerd/README.md) Deployment replica count is overwritten manually or by an autoscaler. Default: 1. | No | +| `hostNetwork` | `bool` | Deploy the [dataplane](/cmd/stunnerd/README.md) into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | +| `resources` | `object` | Compute resources per [dataplane](/cmd/stunnerd/README.md) pod. Default: none. | No | +| `affinity` | `object` | Scheduling constraints for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `tolerations` | `object` | Tolerations for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `securityContext` | `object` | Pod-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `topologySpreadConstraints` | `object` | Description of how the [dataplane](/cmd/stunnerd/README.md) pods for a Gateway ought to spread across topology domains. Default: none. | No | +| `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | +| `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | +| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the GatewayConfig to choose the Dataplane per each STUNner install. From 0c17a0f3edf7ce9d0373bd4e45b07e7c480c1e57 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 27 Jun 2024 16:37:35 +0200 Subject: [PATCH 186/243] doc: Document if/when the health-check port is exposed --- docs/GATEWAY.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 43b4dbb6..ef0e6eee 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -191,8 +191,8 @@ STUNner defines the following special annotations: 1. **Service type:** The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). -1. **Mixed-protocol support:** Currently, STUNner limits each Gateway to a single transport protocol, e.g., UDP or TCP. This is intended to improve the consistency across the Kubernetes services of different cloud providers, which provide varying support for [mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types). If you still want to expose a UDP and a TCP port on the same IP using a single Gateway, add the annotation `stunner.l7mp.io/enable-mixed-protocol-lb: true` to the Gateway. Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature. - +1. **Mixed-protocol support:** Currently, STUNner limits each Gateway to a single transport protocol, e.g., UDP or TCP. This is intended to improve the consistency across the Kubernetes services of different cloud providers, which provide varying support for [mixed multi-protocol LoadBalancer Services](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types). If you still want to expose a UDP and a TCP port on the same IP using a single Gateway, add the annotation `stunner.l7mp.io/enable-mixed-protocol-lb: true` to the Gateway. Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature. + The below Gateway will expose both ports with their respective protocols. ```yaml @@ -219,6 +219,8 @@ STUNner defines the following special annotations: 1. **Selecting the NodePort:** By default, Kubernetes assigns a random external port from the range [32000-32767] to each listener of a Gateway exposed with a NodePort Service. This requires all ports in the [32000-32767] range to be opened on the external firewall, which may raise security concerns for hardened deployments. In order to assign specific nodeports to particular listeners, add the annotation `stunner.l7mp.io/nodeport:` `{listener_name_1:nodeport_1,listener_name_2:nodeport_2,...}` to the Gateway, where each key-value pair is a name of a listener and the selected (numeric) NodePort. The value itself must be proper a JSON map. Unknown listeners are silently ignored. Note that STUNner makes no specific effort to reconcile conflicting NodePorts: whenever the selected NodePort is unavailable Kubernetes will silently reject the Service, which can lead to hard-to-debug failures. Use this feature at your own risk. +1. **Disabling the exposition of the health-check port:** Some older Kubernetes load-balancer providers required the exposition of the health-check port on LoadBalancer Services for UDP listeners to become externally reachable. Therefore, by default STUNner adds the health-check port (usually set via specific Gateway annotations) to the service-ports in automatically created LoadBalancer services. This has the unfortunate consequence that the health-check port becomes publicly reachable, which is considered a security issue by some, see https://github.com/l7mp/stunner-gateway-operator/issues/49. To prevent STUNner from exposing the health-check port, add the annotation `stunner.l7mp.io/disable-health-check-expose: true` to the corresponding Gateway. Note that this may cause TURN/UDP listeners unreachable on the Gateway, so use this only if you know that this setting will work with your Kubernetes provider. + The below table summarizes the Gateway annotations supported by STUNner. | Key/value | Description | Default | @@ -228,6 +230,8 @@ The below table summarizes the Gateway annotations supported by STUNner. | `stunner.l7mp.io/external-traffic-policy: ` | Se the value to `Local` to preserve clients' source IP at the load balancer. | `Cluster` | | `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway. | False | | `stunner.l7mp.io/nodeport: ` | Request a specific NodePort for particular listeners. Value is a JSON map of listener-nodeport key-value pairs. | None | +| `stunner.l7mp.io/disable-health-check-expose: true` | Disable the default exposition of the health-check port (if any). | False | + ## UDPRoute From 1613a5d0cef330536bee2169eced0775bcc985de Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 2 Jul 2024 11:03:48 +0200 Subject: [PATCH 187/243] fix(stunnerd.conf): Update protocol names --- cmd/stunnerd/stunnerd.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cmd/stunnerd/stunnerd.conf b/cmd/stunnerd/stunnerd.conf index 8d130fa6..bee46539 100644 --- a/cmd/stunnerd/stunnerd.conf +++ b/cmd/stunnerd/stunnerd.conf @@ -23,7 +23,7 @@ listeners: - name: stunnerd-udp public_address: "$STUNNER_ADDR" address: "$STUNNER_ADDR" - protocol: udp + protocol: TURN-UDP port: $STUNNER_PORT min_relay_port: $STUNNER_MIN_PORT max_relay_port: $STUNNER_MAX_PORT @@ -33,7 +33,7 @@ listeners: - name: stunnerd-tcp public_address: "$STUNNER_ADDR" address: "$STUNNER_ADDR" - protocol: tcp + protocol: TURN-TCP port: $STUNNER_PORT min_relay_port: $STUNNER_MIN_PORT max_relay_port: $STUNNER_MAX_PORT @@ -41,7 +41,7 @@ listeners: - open-cluster # - media-server-cluster # - name: stunnerd-tls - # protocol: tls + # protocol: TURN-TLS # port: 3479 # cert: # key: From ea06087b9938567baa327cf91c428b111f30e61b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 23 Jul 2024 12:51:23 +0200 Subject: [PATCH 188/243] doc: Update stunnerd README --- cmd/stunnerd/README.md | 70 +++++++++++++++++++++++++----------------- 1 file changed, 42 insertions(+), 28 deletions(-) diff --git a/cmd/stunnerd/README.md b/cmd/stunnerd/README.md index 5d532836..ba47d719 100644 --- a/cmd/stunnerd/README.md +++ b/cmd/stunnerd/README.md @@ -33,6 +33,7 @@ dataplane using STUNner's config discovery service. ### Installation As easy as with any Go program. + ```console cd stunner go build -o stunnerd cmd/stunnerd/main.go @@ -49,7 +50,7 @@ The below command will open a `stunnerd` UDP listener at `127.0.0.1:5000`, set ` Alternatively, run `stunnerd` in verbose mode with the config file taken from `cmd/stunnerd/stunnerd.conf`. Adding the flag `-w` will enable watch mode. ```console -./stunnerd -v -w -c cmd/stunnerd/stunnerd.conf +./stunnerd -v -w -c file://cmd/stunnerd/stunnerd.conf ``` Type `./stunnerd -h` to get a short description of the supported command line arguments. @@ -61,37 +62,50 @@ In practice, you'll rarely need to run `stunnerd` directly: just fire up the [pr Using the below configuration, `stunnerd` will open 4 STUNner listeners: two for accepting unencrypted connections at UDP/3478 and TCP/3478, and two for encrypted connections at TLS/TCP/3479 and DTLS/UDP/3479. The daemon will use `ephemeral` authentication, with the shared secret taken from the environment variable `$STUNNER_SHARED_SECRET` during initialization. The relay address will be taken from the `$STUNNER_ADDR` environment variable. ``` yaml -version: v1alpha1 +version: v1 admin: name: my-stunnerd logLevel: all:DEBUG realm: "my-realm.example.com" -static: - auth: - type: ephemeral - credentials: - secret: $STUNNER_SHARED_SECRET - listeners: - - name: stunnerd-udp - address: "$STUNNER_ADDR" - protocol: turn-udp - port: 3478 - - name: stunnerd-tcp - address: "$STUNNER_ADDR" - protocol: turn-tcp - port: 3478 - - name: stunnerd-tls - address: "$STUNNER_ADDR" - protocol: turn-tls - port: 3479 - cert: "my-cert.cert" - key: "my-key.key" - - name: stunnerd-dtls - address: "$STUNNER_ADDR" - protocol: turn-dtls - port: 3479 - cert: "my-cert.cert" - key: "my-key.key" +auth: + type: ephemeral + credentials: + secret: $STUNNER_SHARED_SECRET +listeners: + - name: stunnerd-udp + address: "$STUNNER_ADDR" + protocol: turn-udp + port: 3478 + routes: + - default/media-plane + - name: stunnerd-tcp + address: "$STUNNER_ADDR" + protocol: turn-tcp + port: 3478 + routes: + - default/media-plane + - name: stunnerd-tls + address: "$STUNNER_ADDR" + protocol: turn-tls + port: 3479 + cert: "my-cert.cert" + key: "my-key.key" + routes: + - default/media-plane + - name: stunnerd-dtls + address: "$STUNNER_ADDR" + protocol: turn-dtls + port: 3479 + cert: "my-cert.cert" + key: "my-key.key" + routes: + - default/media-plane +clusters: + - name: stunner/iperf-server + protocol: UDP + type: STATIC + endpoints: + - 127.0.0.1 ``` STUNner can run multiple parallel readloops for TURN/UDP listeners, which allows it to scale to practically any number of CPUs and brings massive performance improvements for UDP workloads. This can be achieved by creating a configurable number of UDP readloop threads over the same TURN listener. The kernel will load-balance allocations across the readloops per the IP 5-tuple and so the same allocation will always stay at the same CPU, which is important for correct TURN operations. From d16deb44a1c06962da3ee9b24c66ea9bfff75d4d Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 23 Jul 2024 21:06:01 +0200 Subject: [PATCH 189/243] doc: Document the annotation for selecting the targetport --- docs/GATEWAY.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index ef0e6eee..4625b63b 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -219,6 +219,28 @@ STUNner defines the following special annotations: 1. **Selecting the NodePort:** By default, Kubernetes assigns a random external port from the range [32000-32767] to each listener of a Gateway exposed with a NodePort Service. This requires all ports in the [32000-32767] range to be opened on the external firewall, which may raise security concerns for hardened deployments. In order to assign specific nodeports to particular listeners, add the annotation `stunner.l7mp.io/nodeport:` `{listener_name_1:nodeport_1,listener_name_2:nodeport_2,...}` to the Gateway, where each key-value pair is a name of a listener and the selected (numeric) NodePort. The value itself must be proper a JSON map. Unknown listeners are silently ignored. Note that STUNner makes no specific effort to reconcile conflicting NodePorts: whenever the selected NodePort is unavailable Kubernetes will silently reject the Service, which can lead to hard-to-debug failures. Use this feature at your own risk. +1. **Selecting the target port:** Some hardened Kubernetes deployments prohibit containers opening privileged ports (i.e., everything under 1024). This causes problems when one wants to ingest TURN over the standard TCP/TLS port 443. Kubernetes lets Services to choose an arbitrary [target port](https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service) for each service port, which makes it possible to map a particular external port to an arbitrary (potentially non-privileged) port in the containers. In order to enforce a particular target port per listener, add the annotation `stunner.l7mp.io/targetport:` `{listener_name_1:targetport_1,...}` to the corresponding Gateway (the syntax and semantics is the same as those for the nodeport annotation). For instance, the below Gateway would expose the TURN TCP/TLS on port 443, but it would map the egress port to the target port 44321 in the STUNner dataplane container: + + ```yaml + apiVersion: gateway.networking.k8s.io/v1 + kind: Gateway + metadata: + name: tls-gateway + annotations: + stunner.l7mp.io/targetport: "{\"tls-listener\":44321}" + spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: tls-listener + port: 443 + protocol: TURN-TLS + tls: + certificateRefs: + - kind: Secret + namespace: stunner + name: tls-secret + ``` + 1. **Disabling the exposition of the health-check port:** Some older Kubernetes load-balancer providers required the exposition of the health-check port on LoadBalancer Services for UDP listeners to become externally reachable. Therefore, by default STUNner adds the health-check port (usually set via specific Gateway annotations) to the service-ports in automatically created LoadBalancer services. This has the unfortunate consequence that the health-check port becomes publicly reachable, which is considered a security issue by some, see https://github.com/l7mp/stunner-gateway-operator/issues/49. To prevent STUNner from exposing the health-check port, add the annotation `stunner.l7mp.io/disable-health-check-expose: true` to the corresponding Gateway. Note that this may cause TURN/UDP listeners unreachable on the Gateway, so use this only if you know that this setting will work with your Kubernetes provider. The below table summarizes the Gateway annotations supported by STUNner. @@ -230,6 +252,7 @@ The below table summarizes the Gateway annotations supported by STUNner. | `stunner.l7mp.io/external-traffic-policy: ` | Se the value to `Local` to preserve clients' source IP at the load balancer. | `Cluster` | | `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway. | False | | `stunner.l7mp.io/nodeport: ` | Request a specific NodePort for particular listeners. Value is a JSON map of listener-nodeport key-value pairs. | None | +| `stunner.l7mp.io/targetport: ` | Request a specific target port for particular listeners. Value is a JSON map of listener-targetport key-value pairs. | None | | `stunner.l7mp.io/disable-health-check-expose: true` | Disable the default exposition of the health-check port (if any). | False | From 69dff751588111382b833e3514f0e5215c360926 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 8 Aug 2024 12:28:12 +0200 Subject: [PATCH 190/243] doc: Add link to blog post describing the STUN service use case --- README.md | 12 ++++++++++++ docs/GATEWAY.md | 2 +- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 464f9810..491be8a1 100644 --- a/README.md +++ b/README.md @@ -151,6 +151,18 @@ way. lines of fully open-source code. The server is extremely lightweight: the typical STUNner container image size is only 15 Mbytes. +The main uses of STUNner are running a scalable STUN service in Kubernetes to make it possible for +clients to create peer-to-peer WebRTC media connections between themselves (the so called +[*headless deployment +model*](https://github.com/l7mp/stunner/blob/main/docs/DEPLOYMENT.md#headless-deployment-model)), +or as a scalable TURN gateway service for load-balancing clients' media connections across a pool +of WebRTC media servers hosted in Kubernetes pods (called the [*media-plane deployment +model*](https://github.com/l7mp/stunner/blob/main/docs/DEPLOYMENT.md#media-plane-deployment-model)). While +both use cases are fully supported, below we concentrate on the latter use case: deploying STUNner +as a media gateway (see [our blog +post](https://medium.com/l7mp-technologies/deploying-a-scalable-stun-service-in-kubernetes-c7b9726fa41d) +to learn how to deploy a STUN service with STUNner). + ## Getting Started STUNner comes with a [Helm](https://helm.sh) chart to fire up a fully functional STUNner-based diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 4625b63b..73dc5956 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -219,7 +219,7 @@ STUNner defines the following special annotations: 1. **Selecting the NodePort:** By default, Kubernetes assigns a random external port from the range [32000-32767] to each listener of a Gateway exposed with a NodePort Service. This requires all ports in the [32000-32767] range to be opened on the external firewall, which may raise security concerns for hardened deployments. In order to assign specific nodeports to particular listeners, add the annotation `stunner.l7mp.io/nodeport:` `{listener_name_1:nodeport_1,listener_name_2:nodeport_2,...}` to the Gateway, where each key-value pair is a name of a listener and the selected (numeric) NodePort. The value itself must be proper a JSON map. Unknown listeners are silently ignored. Note that STUNner makes no specific effort to reconcile conflicting NodePorts: whenever the selected NodePort is unavailable Kubernetes will silently reject the Service, which can lead to hard-to-debug failures. Use this feature at your own risk. -1. **Selecting the target port:** Some hardened Kubernetes deployments prohibit containers opening privileged ports (i.e., everything under 1024). This causes problems when one wants to ingest TURN over the standard TCP/TLS port 443. Kubernetes lets Services to choose an arbitrary [target port](https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service) for each service port, which makes it possible to map a particular external port to an arbitrary (potentially non-privileged) port in the containers. In order to enforce a particular target port per listener, add the annotation `stunner.l7mp.io/targetport:` `{listener_name_1:targetport_1,...}` to the corresponding Gateway (the syntax and semantics is the same as those for the nodeport annotation). For instance, the below Gateway would expose the TURN TCP/TLS on port 443, but it would map the egress port to the target port 44321 in the STUNner dataplane container: +1. **Selecting the target port:** Some hardened Kubernetes deployments prohibit containers to open privileged ports (i.e., everything under 1024). This causes problems when one wants to ingest TURN over the standard TCP/TLS port 443. Kubernetes lets Services to choose an arbitrary [target port](https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service) for each service port, which makes it possible to map a particular external port to an arbitrary (potentially non-privileged) port in the containers. In order to enforce a particular target port per listener, add the annotation `stunner.l7mp.io/targetport:` `{listener_name_1:targetport_1,...}` to the corresponding Gateway (the syntax and semantics are the same as those for the nodeport annotation). For instance, the below Gateway would expose the TURN TCP/TLS on port 443, but it would map the egress port to the target port 44321 in the STUNner dataplane container: ```yaml apiVersion: gateway.networking.k8s.io/v1 From f533fea0548f6ef5ba6930e2065a87016930b773 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 19 Aug 2024 20:24:46 +0200 Subject: [PATCH 191/243] chore(CI/CD): Disable Docker build in other repos Due to the lack of credentials, the automatically triggered Docker push will fail in forks. Disabling them fixes the false positive "GitHub action failed" notifications. --- .github/workflows/publish-dev.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 7e98ace5..53007d11 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -19,6 +19,7 @@ jobs: push_to_registry: name: Push Docker image to DockerHub needs: run_tests + if: github.repository == 'l7mp/stunner' runs-on: ubuntu-latest steps: - name: Checkout @@ -55,6 +56,7 @@ jobs: push_chart: name: Push helm charts to the repo + if: github.repository == 'l7mp/stunner' needs: push_to_registry uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: From 963508351bf5959130087c23c670a9734feb6d33 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Sun, 25 Aug 2024 14:05:46 +0200 Subject: [PATCH 192/243] doc: Add SECURITY.md Github reads security policy from the SECURITY.md files. To let GitHub use a security file on the main page instead of the docs/SECURITY.md file, this commit adds a new file that both describes security issue reporting and links the existing security guide. Refs - https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository --- SECURITY.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 SECURITY.md diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..6f97cd62 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,9 @@ +# Reporting Security Issues + +If you encounter any security issues, please get in touch with us in any of the following ways: +- open a ticket at [GitHub Security Advisories](https://github.com/l7mp/stunner/security/advisories), +- e-mail core developers at [info@l7mp.io](mailto:info@l7mp.io). + +## Learning More About Security + +To learn more about securing a STUNner deployment, please see the [security documentation](docs/SECURITY.md). From 18aa9c1f43d068af752498f1184feac92a56023e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 27 Aug 2024 18:08:29 +0200 Subject: [PATCH 193/243] chore: Fix linter error --- server.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server.go b/server.go index 56c91061..1ed2dc4a 100644 --- a/server.go +++ b/server.go @@ -141,7 +141,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { l.Conns = append(l.Conns, conn) default: - return fmt.Errorf("internal error: unknown listener protocol " + l.Proto.String()) + return fmt.Errorf("internal error: unknown listener protocol %q", l.Proto.String()) } // start the TURN server if there are actual listeners configured From 81f018d345b8784f0bd6b565b96dbf64157d1bba Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 29 Aug 2024 20:54:58 +0200 Subject: [PATCH 194/243] doc: Add warning for manual dataplane restart on helm upgrade --- docs/INSTALL.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index f5f8f28c..34b4c295 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -39,6 +39,13 @@ helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --cre --namespace=stunner-system ``` +After upgrading the operator from the dev channel you may need to manually restart the dataplane +for each of your Gateways: + +```console +kubectl -n rollout restart deployment +``` + ### Legacy mode In the default *managed dataplane mode*, the STUNner gateway operator automatically provisions the dataplane, which substantially simplifies operations and removes lot of manual and repetitive work. For compatibility reasons the traditional operational model, called the *legacy mode*, is still available. In this mode the user is responsible for provisioning both the control plane, by installing the `stunner-gateway-operator` Helm chart, and the dataplane(s), by helm-installing the `stunner` chart possibly multiple times. From c22ba333a1915b51d9a5542ad46a038312c9c5d1 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 30 Aug 2024 14:41:47 +0200 Subject: [PATCH 195/243] doc: Minor fixes --- docs/GATEWAY.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index 73dc5956..e9191d80 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -398,17 +398,17 @@ The following fields can be set in the Dataplane `spec` to customize the provisi | `args` | `list` | Command line arguments for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). | No | | `envFrom` | `list` | List of sources to populate environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | | `env` | `list` | List of environment variables for the [dataplane container](https://pkg.go.dev/k8s.io/api/core/v1#Container). Default: empty. | No | -| `containerSecurityContext` | `object` | Container-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | | `replicas` | `int` | Number of dataplane pods per Gateway to provision. Not enforced if the [dataplane](/cmd/stunnerd/README.md) Deployment replica count is overwritten manually or by an autoscaler. Default: 1. | No | -| `hostNetwork` | `bool` | Deploy the [dataplane](/cmd/stunnerd/README.md) into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | | `resources` | `object` | Compute resources per [dataplane](/cmd/stunnerd/README.md) pod. Default: none. | No | | `affinity` | `object` | Scheduling constraints for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | | `tolerations` | `object` | Tolerations for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | | `securityContext` | `object` | Pod-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | +| `containerSecurityContext` | `object` | Container-level security attributes for the [dataplane](/cmd/stunnerd/README.md) pods. Default: none. | No | | `topologySpreadConstraints` | `object` | Description of how the [dataplane](/cmd/stunnerd/README.md) pods for a Gateway ought to spread across topology domains. Default: none. | No | +| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | +| `hostNetwork` | `bool` | Deploy the [dataplane](/cmd/stunnerd/README.md) into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | | `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | | `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | -| `terminationGracePeriodSeconds` | `duration` | Optional duration in seconds for `stunnerd` to terminate gracefully. Default: 30 seconds. | No | There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the GatewayConfig to choose the Dataplane per each STUNner install. From 5176898474be41e451c03f8b4a5a2a8ed4a70a8a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 30 Aug 2024 15:44:44 +0200 Subject: [PATCH 196/243] chore: Remove doc disclaimer before release --- README.md | 2 -- docs/README.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/README.md b/README.md index 491be8a1..edb06e1f 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,6 @@

-*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index bf4c7f51..03638016 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,6 @@ STUNner

-*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - ## Overview * [Why STUNner](WHY.md) From 1b190b16616095aa1fbe8cb24491404aca4a090b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 30 Aug 2024 15:53:50 +0200 Subject: [PATCH 197/243] chore: Restore doc disclaimers --- README.md | 2 ++ docs/README.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/README.md b/README.md index edb06e1f..491be8a1 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 03638016..bf4c7f51 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,8 @@ STUNner

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + ## Overview * [Why STUNner](WHY.md) From a6b3de4bb72da23723b5450486a3d35a0e4e5e73 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Tue, 3 Sep 2024 18:54:37 +0200 Subject: [PATCH 198/243] chore: Bump pion/turn to v4 and update some deps (#156) --- cmd/turncat/main.go | 2 +- go.mod | 47 ++++++------ go.sum | 149 ++++++++++++------------------------ handlers_test.go | 2 +- internal/object/listener.go | 2 +- pkg/authentication/auth.go | 2 +- server.go | 4 +- stunner_test.go | 4 +- turncat.go | 4 +- turncat_test.go | 2 +- 10 files changed, 84 insertions(+), 134 deletions(-) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index f6720dce..69f63cc1 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -10,7 +10,7 @@ import ( "time" "github.com/pion/logging" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" flag "github.com/spf13/pflag" cliopt "k8s.io/cli-runtime/pkg/genericclioptions" diff --git a/go.mod b/go.mod index 2c3f3563..22168e75 100644 --- a/go.mod +++ b/go.mod @@ -8,22 +8,22 @@ require ( github.com/deepmap/oapi-codegen/v2 v2.1.0 github.com/fsnotify/fsnotify v1.7.0 github.com/getkin/kin-openapi v0.123.0 - github.com/go-logr/logr v1.4.1 + github.com/go-logr/logr v1.4.2 github.com/go-logr/zapr v1.3.0 github.com/google/uuid v1.6.0 github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.1 github.com/oapi-codegen/runtime v1.1.1 - github.com/pion/dtls/v2 v2.2.11 + github.com/pion/dtls/v3 v3.0.2 github.com/pion/logging v0.2.2 - github.com/pion/transport/v3 v3.0.2 - github.com/pion/turn/v3 v3.0.3 - github.com/prometheus/client_golang v1.19.1 - github.com/spf13/cobra v1.8.0 + github.com/pion/transport/v3 v3.0.7 + github.com/pion/turn/v4 v4.0.0 + github.com/prometheus/client_golang v1.20.2 + github.com/spf13/cobra v1.8.1 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 go.uber.org/zap v1.26.0 - golang.org/x/sys v0.18.0 + golang.org/x/sys v0.23.0 golang.org/x/time v0.5.0 k8s.io/api v0.29.1 k8s.io/apimachinery v0.29.1 @@ -37,7 +37,7 @@ require ( github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/beorn7/perks v1.0.1 // indirect - github.com/cespare/xxhash/v2 v2.2.0 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/emicklei/go-restful/v3 v3.11.2 // indirect github.com/evanphx/json-patch v5.9.0+incompatible // indirect @@ -57,6 +57,8 @@ require ( github.com/invopop/yaml v0.2.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect + github.com/klauspost/compress v1.17.9 // indirect + github.com/kylelemons/godebug v1.1.0 // indirect github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect github.com/mailru/easyjson v0.7.7 // indirect github.com/moby/spdystream v0.2.0 // indirect @@ -70,26 +72,25 @@ require ( github.com/perimeterx/marshmallow v1.1.5 // indirect github.com/peterbourgon/diskv v2.0.1+incompatible // indirect github.com/pion/randutil v0.1.0 // indirect - github.com/pion/stun/v2 v2.0.0 // indirect - github.com/pion/transport/v2 v2.2.4 // indirect + github.com/pion/stun/v3 v3.0.0 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect - github.com/prometheus/client_model v0.5.0 // indirect - github.com/prometheus/common v0.48.0 // indirect - github.com/prometheus/procfs v0.12.0 // indirect + github.com/prometheus/client_model v0.6.1 // indirect + github.com/prometheus/common v0.58.0 // indirect + github.com/prometheus/procfs v0.15.1 // indirect + github.com/wlynxg/anet v0.0.3 // indirect github.com/xlab/treeprint v1.2.0 // indirect go.starlark.net v0.0.0-20240123142251-f86470692795 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.21.0 // indirect - golang.org/x/mod v0.14.0 // indirect - golang.org/x/net v0.22.0 // indirect - golang.org/x/oauth2 v0.16.0 // indirect - golang.org/x/sync v0.6.0 // indirect - golang.org/x/term v0.18.0 // indirect - golang.org/x/text v0.14.0 // indirect - golang.org/x/tools v0.17.0 // indirect - google.golang.org/appengine v1.6.8 // indirect - google.golang.org/protobuf v1.33.0 // indirect + golang.org/x/crypto v0.26.0 // indirect + golang.org/x/mod v0.17.0 // indirect + golang.org/x/net v0.28.0 // indirect + golang.org/x/oauth2 v0.22.0 // indirect + golang.org/x/sync v0.8.0 // indirect + golang.org/x/term v0.23.0 // indirect + golang.org/x/text v0.17.0 // indirect + golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect + google.golang.org/protobuf v1.34.2 // indirect gopkg.in/evanphx/json-patch.v5 v5.9.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/go.sum b/go.sum index e2de97fc..647996fb 100644 --- a/go.sum +++ b/go.sum @@ -8,9 +8,9 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bmatcuk/doublestar v1.1.1/go.mod h1:UD6OnuiIn0yFxxA2le/rnRU1G4RaI4UvFv1sNto9p6w= -github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= -github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY= github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -28,8 +28,8 @@ github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mN github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM= github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk= github.com/go-errors/errors v1.5.1/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= -github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ= -github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= +github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q= @@ -45,7 +45,6 @@ github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncV github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/google/btree v1.1.2 h1:xf4v41cLI2Z6FxbKm+8Bu+m8ifhj15JuZ9sa0jZCMUU= @@ -85,10 +84,14 @@ github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHm github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA= +github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0= github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= @@ -120,65 +123,55 @@ github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= -github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= -github.com/pion/dtls/v2 v2.2.11 h1:9U/dpCYl1ySttROPWJgqWKEylUdT0fXp/xst6JwY5Ks= -github.com/pion/dtls/v2 v2.2.11/go.mod h1:d9SYc9fch0CqK90mRk1dC7AkzzpwJj6u2GU3u+9pqFE= +github.com/pion/dtls/v3 v3.0.2 h1:425DEeJ/jfuTTghhUDW0GtYZYIwwMtnKKJNMcWccTX0= +github.com/pion/dtls/v3 v3.0.2/go.mod h1:dfIXcFkKoujDQ+jtd8M6RgqKK3DuaUilm3YatAbGp5k= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8= -github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= -github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= -github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= -github.com/pion/transport/v2 v2.2.4 h1:41JJK6DZQYSeVLxILA2+F4ZkKb4Xd/tFJZRFZQ9QAlo= -github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= -github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= -github.com/pion/transport/v3 v3.0.2 h1:r+40RJR25S9w3jbA6/5uEPTzcdn7ncyU44RWCbHkLg4= -github.com/pion/transport/v3 v3.0.2/go.mod h1:nIToODoOlb5If2jF9y2Igfx3PFYWfuXi37m0IlWa/D0= -github.com/pion/turn/v3 v3.0.3 h1:1e3GVk8gHZLPBA5LqadWYV60lmaKUaHCkm9DX9CkGcE= -github.com/pion/turn/v3 v3.0.3/go.mod h1:vw0Dz420q7VYAF3J4wJKzReLHIo2LGp4ev8nXQexYsc= +github.com/pion/stun/v3 v3.0.0 h1:4h1gwhWLWuZWOJIJR9s2ferRO+W3zA/b6ijOI6mKzUw= +github.com/pion/stun/v3 v3.0.0/go.mod h1:HvCN8txt8mwi4FBvS3EmDghW6aQJ24T+y+1TKjB5jyU= +github.com/pion/transport/v3 v3.0.7 h1:iRbMH05BzSNwhILHoBoAPxoB9xQgOaJk+591KC9P1o0= +github.com/pion/transport/v3 v3.0.7/go.mod h1:YleKiTZ4vqNxVwh77Z0zytYi7rXHl7j6uPLGhhz9rwo= +github.com/pion/turn/v4 v4.0.0 h1:qxplo3Rxa9Yg1xXDxxH8xaqcyGUtbHYw4QSCvmFWvhM= +github.com/pion/turn/v4 v4.0.0/go.mod h1:MuPDkm15nYSklKpN8vWJ9W2M0PlyQZqYt1McGuxG7mA= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.19.1 h1:wZWJDwK+NameRJuPGDhlnFgx8e8HN3XHQeLaYJFJBOE= -github.com/prometheus/client_golang v1.19.1/go.mod h1:mP78NwGzrVks5S2H6ab8+ZZGJLZUq1hoULYBAYBw1Ho= -github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= -github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= -github.com/prometheus/common v0.48.0 h1:QO8U2CdOzSn1BBsmXJXduaaW+dY/5QLjfB8svtSzKKE= -github.com/prometheus/common v0.48.0/go.mod h1:0/KsvlIEfPQCQ5I2iNSAWKPZziNCvRs5EC6ILDTlAPc= -github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= -github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= +github.com/prometheus/client_golang v1.20.2 h1:5ctymQzZlyOON1666svgwn3s6IKWgfbjsejTMiXIyjg= +github.com/prometheus/client_golang v1.20.2/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE= +github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E= +github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY= +github.com/prometheus/common v0.58.0 h1:N+N8vY4/23r6iYfD3UQZUoJPnUYAo7v6LG5XZxjZTXo= +github.com/prometheus/common v0.58.0/go.mod h1:GpWM7dewqmVYcd7SmRaiWVe9SSqjf0UrwnYnpEZNuT0= +github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc= +github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= -github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= -github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= +github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM= +github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= -github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY= github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU= github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg= +github.com/wlynxg/anet v0.0.3 h1:PvR53psxFXstc12jelG6f1Lv4MWqE0tI76/hHGjh9rg= +github.com/wlynxg/anet v0.0.3/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA= github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= go.starlark.net v0.0.0-20240123142251-f86470692795 h1:LmbG8Pq7KDGkglKVn8VpZOZj6vb9b8nKEGcg9l03epM= go.starlark.net v0.0.0-20240123142251-f86470692795/go.mod h1:LcLNIzVOMp4oV+uusnpk+VU+SzXaJakUuBjoCSWH5dM= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= @@ -190,99 +183,55 @@ go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= -golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= -golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= -golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA= -golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs= +golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw= +golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= -golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA= +golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= -golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= -golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= -golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= -golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= -golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc= -golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= -golang.org/x/oauth2 v0.16.0 h1:aDkGMBSYxElaoP81NpoUoz2oo2R2wHdZpGToUxfyQrQ= -golang.org/x/oauth2 v0.16.0/go.mod h1:hqZ+0LWXsiVoZpeld6jVt06P3adbS2Uu911W1SsJv2o= +golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE= +golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg= +golang.org/x/oauth2 v0.22.0 h1:BzDx2FehcG7jJwgWLELCdmLuxk2i+x9UDpSiss2u0ZA= +golang.org/x/oauth2 v0.22.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ= -golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ= +golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4= -golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= -golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= -golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= -golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= -golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= -golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk= -golang.org/x/term v0.18.0 h1:FcHjZXDMxI8mM3nwhX9HlKop4C0YQvCVCdwYl2wOtE8= -golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58= +golang.org/x/sys v0.23.0 h1:YfKFowiIMvtgl1UERQoTPPToxltDeZfbj4H7dVUCwmM= +golang.org/x/sys v0.23.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/term v0.23.0 h1:F6D4vR+EHoL9/sWAWgAR1H2DcHr4PareCbAaCo1RpuU= +golang.org/x/term v0.23.0/go.mod h1:DgV24QBUrK6jhZXl+20l6UWznPlwAHm1Q1mGHtydmSk= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= -golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= -golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= -golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= -golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc= +golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc= -golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAsM= -google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI= -google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= +google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg= +google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= diff --git a/handlers_test.go b/handlers_test.go index 7eef6269..4dd4fe38 100644 --- a/handlers_test.go +++ b/handlers_test.go @@ -11,7 +11,7 @@ import ( "time" "github.com/pion/transport/v3/test" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" "github.com/stretchr/testify/assert" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" diff --git a/internal/object/listener.go b/internal/object/listener.go index b07f2ca3..2fb059a1 100644 --- a/internal/object/listener.go +++ b/internal/object/listener.go @@ -10,7 +10,7 @@ import ( "github.com/pion/logging" "github.com/pion/transport/v3" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" "github.com/l7mp/stunner/internal/util" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" diff --git a/pkg/authentication/auth.go b/pkg/authentication/auth.go index 8441d70b..c1092a6e 100644 --- a/pkg/authentication/auth.go +++ b/pkg/authentication/auth.go @@ -10,7 +10,7 @@ import ( "strings" "time" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" ) // UsernameSeparator is the separator character used in time-windowed TURN authentication as diff --git a/server.go b/server.go index 1ed2dc4a..6c072eb1 100644 --- a/server.go +++ b/server.go @@ -5,8 +5,8 @@ import ( "fmt" "net" - "github.com/pion/dtls/v2" - "github.com/pion/turn/v3" + "github.com/pion/dtls/v3" + "github.com/pion/turn/v4" "golang.org/x/time/rate" "github.com/l7mp/stunner/internal/object" diff --git a/stunner_test.go b/stunner_test.go index efea6a3f..5b4f3360 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -12,13 +12,13 @@ import ( "testing" "time" - "github.com/pion/dtls/v2" + "github.com/pion/dtls/v3" "github.com/pion/logging" "github.com/pion/transport/v3" "github.com/pion/transport/v3/stdnet" "github.com/pion/transport/v3/test" "github.com/pion/transport/v3/vnet" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/assert" diff --git a/turncat.go b/turncat.go index c9e8a649..b8872d0d 100644 --- a/turncat.go +++ b/turncat.go @@ -10,9 +10,9 @@ import ( "strings" "sync" - "github.com/pion/dtls/v2" + "github.com/pion/dtls/v3" "github.com/pion/logging" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" "github.com/l7mp/stunner/internal/util" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" diff --git a/turncat_test.go b/turncat_test.go index dcaf07a4..f776522a 100644 --- a/turncat_test.go +++ b/turncat_test.go @@ -10,7 +10,7 @@ import ( "github.com/pion/logging" "github.com/pion/transport/v3/test" - "github.com/pion/turn/v3" + "github.com/pion/turn/v4" "github.com/stretchr/testify/assert" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" From 0e3cc2eaa2cd1d6b7681c994b200b47768ee581f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Thu, 5 Sep 2024 18:27:55 +0200 Subject: [PATCH 199/243] chore(rtd): Sync with the official template --- .readthedocs.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index c7006c23..37ceb77e 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -7,9 +7,9 @@ version: 2 # Set the version of Python and other tools you might need build: - os: "ubuntu-22.04" + os: ubuntu-22.04 tools: - python: "3.11" + python: "3.12" mkdocs: configuration: mkdocs.yml From d7d7f660a81336c65bd4a49bdfc1dc2e5b3f7abb Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 5 Sep 2024 19:51:50 +0200 Subject: [PATCH 200/243] chore(rtd): Update rtd dependencies --- docs/requirements.in | 3 ++ docs/requirements.txt | 66 ++++++++++++++++++++++++++----------------- 2 files changed, 43 insertions(+), 26 deletions(-) create mode 100644 docs/requirements.in diff --git a/docs/requirements.in b/docs/requirements.in new file mode 100644 index 00000000..4ad415d6 --- /dev/null +++ b/docs/requirements.in @@ -0,0 +1,3 @@ +markdown-include +mkdocs +mkdocstrings[python] \ No newline at end of file diff --git a/docs/requirements.txt b/docs/requirements.txt index f6cb652b..2acd6ad1 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,66 +1,80 @@ # -# This file is autogenerated by pip-compile with python 3.10 -# To update, run: +# This file is autogenerated by pip-compile with Python 3.12 +# by the following command: # # pip-compile docs/requirements.in # -click==8.1.3 - # via mkdocs +click==8.1.7 + # via + # mkdocs + # mkdocstrings +colorama==0.4.6 + # via griffe ghp-import==2.1.0 # via mkdocs -griffe==0.22.0 +griffe==1.2.0 # via mkdocstrings-python -importlib-metadata==4.12.0 - # via mkdocs -jinja2==3.1.2 +jinja2==3.1.4 # via # mkdocs # mkdocstrings -markdown==3.3.7 +markdown==3.7 # via # markdown-include # mkdocs # mkdocs-autorefs # mkdocstrings # pymdown-extensions -markdown-include==0.6.0 +markdown-include==0.8.1 # via -r docs/requirements.in -markupsafe==2.1.1 +markupsafe==2.1.5 # via # jinja2 + # mkdocs + # mkdocs-autorefs # mkdocstrings mergedeep==1.3.4 - # via mkdocs -mkdocs==1.3.0 + # via + # mkdocs + # mkdocs-get-deps +mkdocs==1.6.1 # via # -r docs/requirements.in # mkdocs-autorefs # mkdocstrings -mkdocs-autorefs==0.4.1 - # via mkdocstrings -mkdocstrings[python]==0.19.0 +mkdocs-autorefs==1.2.0 + # via + # mkdocstrings + # mkdocstrings-python +mkdocs-get-deps==0.2.0 + # via mkdocs +mkdocstrings[python]==0.26.0 # via # -r docs/requirements.in # mkdocstrings-python -mkdocstrings-python==0.7.1 +mkdocstrings-python==1.11.1 # via mkdocstrings -packaging==21.3 +packaging==24.1 # via mkdocs -pymdown-extensions==9.5 +pathspec==0.12.1 + # via mkdocs +platformdirs==4.2.2 + # via + # mkdocs-get-deps + # mkdocstrings +pymdown-extensions==10.9 # via mkdocstrings -pyparsing==3.0.9 - # via packaging -python-dateutil==2.8.2 +python-dateutil==2.9.0.post0 # via ghp-import -pyyaml==6.0 +pyyaml==6.0.2 # via # mkdocs + # mkdocs-get-deps + # pymdown-extensions # pyyaml-env-tag pyyaml-env-tag==0.1 # via mkdocs six==1.16.0 # via python-dateutil -watchdog==2.1.9 +watchdog==5.0.2 # via mkdocs -zipp==3.8.0 - # via importlib-metadata From eeb59a73a978804833dadef7e161e1a069bff87d Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 6 Sep 2024 10:08:00 +0200 Subject: [PATCH 201/243] chore(rtd): Remove non-working pages --- mkdocs.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index be3a8d85..fa27b57b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,7 +9,6 @@ theme: name: readthedocs highlightjs: true nav: -- 'STUNner documentation': 'README.md' - 'Why STUNner': 'WHY.md' - 'Deployment models': 'DEPLOYMENT.md' - 'Concepts': 'CONCEPTS.md' @@ -19,7 +18,6 @@ nav: - 'Authentication': 'AUTH.md' - 'Scaling': 'SCALING.md' - 'Monitoring': 'MONITORING.md' -- 'Standalone mode': 'OBSOLETE.md' - 'Examples': - 'Simple tunnel': 'examples/simple-tunnel/README.md' - 'Direct one to one call': 'examples/direct-one2one-call/README.md' From 03d2b102b94241832048e2f18afddedb4bee4522 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 17 Sep 2024 10:13:15 +0200 Subject: [PATCH 202/243] chore(rtd): Fix CloudRetro name in the Examples --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index fa27b57b..a31b1400 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -26,7 +26,7 @@ nav: - 'LiveKit': 'examples/livekit/README.md' - 'Jitsi': 'examples/jitsi/README.md' - 'mediasoup': 'examples/mediasoup/README.md' - - 'Cloud Retro': 'examples/cloudretro/README.md' + - 'CloudRetro': 'examples/cloudretro/README.md' - 'N.eko': 'examples/neko/README.md' - 'Benchmark': 'examples/benchmark/README.md' - 'CMD': From 77b00e57063d344cac7de9f52474d159de263d11 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 18 Sep 2024 10:29:09 +0200 Subject: [PATCH 203/243] chore: Add stunnerctl bin to gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index c7339885..5d20408a 100644 --- a/.gitignore +++ b/.gitignore @@ -25,4 +25,5 @@ go.work # Our binaries turncat +stunnerctl stunnerd \ No newline at end of file From 5d8acafd1baab75605d0f39513f65dc3870218d2 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Wed, 18 Sep 2024 11:11:56 +0200 Subject: [PATCH 204/243] chore: Add buildinfo to stunnerctl binary too --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 7b2389fe..ba85bf1e 100644 --- a/Makefile +++ b/Makefile @@ -42,7 +42,7 @@ build: generate fmt vet build-bin build-bin: go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerd cmd/stunnerd/main.go go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/turncat cmd/turncat/main.go - go build ${GOARGS} -o ${BUILD_DIR}/stunnerctl cmd/stunnerctl/main.go + go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerctl cmd/stunnerctl/main.go .PHONY: clean clean: From e1760bfae6373f4a05d40f7395cd4a74e0aff620 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 20 Sep 2024 15:03:24 +0200 Subject: [PATCH 205/243] doc: Document the disable-session-affinity:true annotation --- docs/GATEWAY.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index e9191d80..57ef7415 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -243,6 +243,8 @@ STUNner defines the following special annotations: 1. **Disabling the exposition of the health-check port:** Some older Kubernetes load-balancer providers required the exposition of the health-check port on LoadBalancer Services for UDP listeners to become externally reachable. Therefore, by default STUNner adds the health-check port (usually set via specific Gateway annotations) to the service-ports in automatically created LoadBalancer services. This has the unfortunate consequence that the health-check port becomes publicly reachable, which is considered a security issue by some, see https://github.com/l7mp/stunner-gateway-operator/issues/49. To prevent STUNner from exposing the health-check port, add the annotation `stunner.l7mp.io/disable-health-check-expose: true` to the corresponding Gateway. Note that this may cause TURN/UDP listeners unreachable on the Gateway, so use this only if you know that this setting will work with your Kubernetes provider. +1. **Disabling session affinity:** By default STUNner applies the `sessionAffinity: ClientIP` setting on the LB services it creates to expose Gateways. Normally this setting improves stability by ensuring that each TURN session is safely pinned to the same dataplane pod for its entire lifetime. Certain hosted Kubernetes platforms, however, seem to reject UDP LB services that have this setting on, [breaking STUNner deployments](https://github.com/l7mp/stunner/issues/155) on these systems. In order to prevent STUNner from enforcing session affinity on the LB Service corresponding to a Gateway, just set the `stunner.l7mp.io/disable-session-affinity: true` annotation on the Gateway. Otherwise, session affinity is turned on. + The below table summarizes the Gateway annotations supported by STUNner. | Key/value | Description | Default | @@ -254,7 +256,7 @@ The below table summarizes the Gateway annotations supported by STUNner. | `stunner.l7mp.io/nodeport: ` | Request a specific NodePort for particular listeners. Value is a JSON map of listener-nodeport key-value pairs. | None | | `stunner.l7mp.io/targetport: ` | Request a specific target port for particular listeners. Value is a JSON map of listener-targetport key-value pairs. | None | | `stunner.l7mp.io/disable-health-check-expose: true` | Disable the default exposition of the health-check port (if any). | False | - +| `stunner.l7mp.io/disable-session-affinity: true` | Disable session affinity for a Gateway. | False | ## UDPRoute From 5eede986b4e1c3b9f57782d29f896e0d91059a8d Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 20 Sep 2024 15:23:32 +0200 Subject: [PATCH 206/243] chore: Remove doc disclaimer before release --- README.md | 2 -- docs/README.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/README.md b/README.md index 491be8a1..edb06e1f 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,6 @@

-*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index bf4c7f51..03638016 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,6 @@ STUNner

-*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - ## Overview * [Why STUNner](WHY.md) From 6647128f2451b248865266024ca3e0c7699e2972 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 20 Sep 2024 17:04:42 +0200 Subject: [PATCH 207/243] chore: Restore doc disclaimers --- README.md | 2 ++ docs/README.md | 2 ++ docs/RELEASE.md | 4 ++-- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index edb06e1f..491be8a1 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 03638016..bf4c7f51 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,8 @@ STUNner

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + ## Overview * [Why STUNner](WHY.md) diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 407402ed..4e7ac876 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -24,7 +24,7 @@ Let the new version be vX.Y.Z. - `go mod tidy` - make sure `go test ./... -count 1` passes - make sure `golangci-lint run` passes -- `git push` if there are local changes +- `git push` to send the updated `go.mod/go.sum` files - release vX.Y.Z (try to have the same version as the main stunner repo, if possible) on github (so that we can publish the release notes) @@ -38,7 +38,7 @@ Let the new version be vX.Y.Z. - `go mod tidy` - make sure `make test` passes - make sure `golangci-lint run` passes -- `git push` if there are local changes +- `git push` to send the updated `go.mod/go.sum` files - release vX.Y.Z (try to have the same version as the main stunner repo, if possible) on github (so that we can publish the release notes) From b64b2ea5ad2df74da67a84b2d29ba64ef143963b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Korn=C3=A9l=20D=C3=A1vid?= <47784437+davidkornel@users.noreply.github.com> Date: Tue, 24 Sep 2024 07:00:08 +0200 Subject: [PATCH 208/243] docs: Add Janus example (#158) --- docs/examples/janus/DOCKERFILE-janus-gateway | 123 ++++++++++ docs/examples/janus/DOCKERFILE-janus-web | 2 + docs/examples/janus/README.md | 213 +++++++++++++++++ docs/examples/janus/janus-call-stunner.yaml | 51 ++++ docs/examples/janus/janus-server.yaml | 235 +++++++++++++++++++ docs/img/stunner_janus_arch.svg | 4 + 6 files changed, 628 insertions(+) create mode 100644 docs/examples/janus/DOCKERFILE-janus-gateway create mode 100644 docs/examples/janus/DOCKERFILE-janus-web create mode 100644 docs/examples/janus/README.md create mode 100644 docs/examples/janus/janus-call-stunner.yaml create mode 100644 docs/examples/janus/janus-server.yaml create mode 100644 docs/img/stunner_janus_arch.svg diff --git a/docs/examples/janus/DOCKERFILE-janus-gateway b/docs/examples/janus/DOCKERFILE-janus-gateway new file mode 100644 index 00000000..08b214bc --- /dev/null +++ b/docs/examples/janus/DOCKERFILE-janus-gateway @@ -0,0 +1,123 @@ +FROM debian:bullseye-slim + +RUN apt-get -y update && \ + apt-get install -y \ + libavutil-dev \ + libavformat-dev \ + libavcodec-dev \ + libmicrohttpd-dev \ + libjansson-dev \ + libssl-dev \ + libsofia-sip-ua-dev \ + libglib2.0-dev \ + libopus-dev \ + libogg-dev \ + libcurl4-openssl-dev \ + liblua5.3-dev \ + libconfig-dev \ + libusrsctp-dev \ + libwebsockets-dev \ + libnanomsg-dev \ + librabbitmq-dev \ + pkg-config \ + gengetopt \ + libtool \ + automake \ + build-essential \ + wget \ + git \ + gtk-doc-tools && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + + +RUN cd /tmp && \ + wget https://github.com/cisco/libsrtp/archive/v2.3.0.tar.gz && \ + tar xfv v2.3.0.tar.gz && \ + cd libsrtp-2.3.0 && \ + ./configure --prefix=/usr --enable-openssl && \ + make shared_library && \ + make install + +RUN cd /tmp && \ + git clone https://gitlab.freedesktop.org/libnice/libnice && \ + cd libnice && \ + git checkout 0.1.17 && \ + ./autogen.sh && \ + ./configure --prefix=/usr && \ + make && \ + make install + +COPY . /usr/local/src/janus-gateway + +RUN cd /usr/local/src/janus-gateway && \ + sh autogen.sh && \ + ./configure --enable-post-processing --prefix=/usr/local && \ + make && \ + make install && \ + make configs + +FROM debian:bullseye-slim + +ARG BUILD_DATE="undefined" +ARG GIT_BRANCH="undefined" +ARG GIT_COMMIT="undefined" +ARG VERSION="undefined" + +LABEL build_date=${BUILD_DATE} +LABEL git_branch=${GIT_BRANCH} +LABEL git_commit=${GIT_COMMIT} +LABEL version=${VERSION} + +RUN apt-get -y update && \ + apt-get install -y \ + libmicrohttpd12 \ + libavutil-dev \ + libavformat-dev \ + libavcodec-dev \ + libjansson4 \ + libssl1.1 \ + libsofia-sip-ua0 \ + libglib2.0-0 \ + libopus0 \ + libogg0 \ + libcurl4 \ + liblua5.3-0 \ + libconfig9 \ + libusrsctp1 \ + libwebsockets16 \ + libnanomsg5 \ + librabbitmq4 && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +COPY --from=0 /usr/lib/libsrtp2.so.1 /usr/lib/libsrtp2.so.1 +RUN ln -s /usr/lib/libsrtp2.so.1 /usr/lib/libsrtp2.so + +COPY --from=0 /usr/lib/libnice.la /usr/lib/libnice.la +COPY --from=0 /usr/lib/libnice.so.10.10.0 /usr/lib/libnice.so.10.10.0 +RUN ln -s /usr/lib/libnice.so.10.10.0 /usr/lib/libnice.so.10 +RUN ln -s /usr/lib/libnice.so.10.10.0 /usr/lib/libnice.so + +COPY --from=0 /usr/local/bin/janus /usr/local/bin/janus +COPY --from=0 /usr/local/bin/janus-pp-rec /usr/local/bin/janus-pp-rec +COPY --from=0 /usr/local/bin/janus-cfgconv /usr/local/bin/janus-cfgconv +COPY --from=0 /usr/local/etc/janus /usr/local/etc/janus +COPY --from=0 /usr/local/lib/janus /usr/local/lib/janus +COPY --from=0 /usr/local/share/janus /usr/local/share/janus + +ENV BUILD_DATE=${BUILD_DATE} +ENV GIT_BRANCH=${GIT_BRANCH} +ENV GIT_COMMIT=${GIT_COMMIT} +ENV VERSION=${VERSION} + +EXPOSE 10000-10200/udp +EXPOSE 8188 +EXPOSE 8088 +EXPOSE 8089 +EXPOSE 8889 +EXPOSE 8000 +EXPOSE 7088 +EXPOSE 7089 + +CMD ["/usr/local/bin/janus"] \ No newline at end of file diff --git a/docs/examples/janus/DOCKERFILE-janus-web b/docs/examples/janus/DOCKERFILE-janus-web new file mode 100644 index 00000000..7aa5364f --- /dev/null +++ b/docs/examples/janus/DOCKERFILE-janus-web @@ -0,0 +1,2 @@ +FROM nginx:1.27.1 +COPY . /usr/share/nginx/html \ No newline at end of file diff --git a/docs/examples/janus/README.md b/docs/examples/janus/README.md new file mode 100644 index 00000000..d7e87c5d --- /dev/null +++ b/docs/examples/janus/README.md @@ -0,0 +1,213 @@ +# STUNner demo: Video-conferencing with Janus + +This document guides you through the installation of [Janus](https://janus.conf.meetecho.com/) by [Meetecho](https://www.meetecho.com/en/) into Kubernetes, when it is used together with the STUNner WebRTC media gateway. + +In this demo you will learn to: + +- integrate a typical WebRTC application with STUNner, +- obtain a valid TLS certificate to secure the signaling plane, +- deploy the Janus WebRTC server into Kubernetes, and +- configure STUNner to expose Janus to clients. + +## Prerequisites + +The below installation instructions require an operational cluster running a supported version of Kubernetes (>1.22). Most hosted or private Kubernetes cluster services will work, but make sure that the cluster comes with a functional load-balancer integration (all major hosted Kubernetes services should support this). Otherwise, STUNner will not be able to allocate a public IP address for clients to reach your WebRTC infra. As a regrettable exception, Minikube is unfortunately not supported for this demo. The reason is that [Let's Encrypt certificate issuance is not available with nip.io](https://medium.com/@EmiiKhaos/there-is-no-possibility-that-you-can-get-lets-encrypt-certificate-with-nip-io-7483663e0c1b); later on you will learn more about why this is crucial above. + +## Setup + +The recommended way (or at least the possible way, [link](https://janus.discourse.group/t/janus-with-kubernetes-demystifying-the-myths/938), [link](https://bugraoz93.medium.com/active-passive-highly-availability-janus-gateway-on-kubernetes-2189256e5525)) to install Janus into Kubernetes is deploying the media servers into the host-network namespace of the Kubernetes nodes (`hostNetwork: true`). This deployment model, however, comes with a set of uncanny [operational limitations and security concerns](../../WHY.md). Using STUNner, however, media servers can be deployed into ordinary Kubernetes pods and run over a private IP network, like any "normal" Kubernetes workload. + +The figure below shows Janus deployed into regular Kubernetes pods behind STUNner without the host-networking hack. Here, Janus is deployed behind STUNner in the [*media-plane deployment model*](../../DEPLOYMENT.md), so that STUNner acts as a "local" STUN/TURN server for Janus, saving the overhead of using public a 3rd party STUN/TURN server for NAT traversal. + +![STUNner Janus integration deployment architecture](../../img/stunner_janus_arch.svg) + +In this tutorial we deploy [Janus Gateway](https://github.com/meetecho/janus-gateway/tree/master) with a set of [preimplemented and packaged server plugins](https://janus.conf.meetecho.com/docs/pluginslist.html) for media exchange, a [Janus Web Demo](https://github.com/meetecho/janus-gateway/tree/master/html), a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the Janus server pool to clients. + +## Installation + +> [!NOTE] +> +> Let's start with a disclaimer. The Janus client-side application must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. In this demo, we will aim to obtain a proper CA-signed certificate (self-signed certificates haven't been tested). Obtaining a valid TLS certificate is a challenge. Thus, the majority of the below installation guide will be about securing client connections to Janus over TLS; as it turns out, once HTTPS is correctly working integrating Janus with STUNner is very simple. + +In the below example, STUNner will be installed into the identically named namespace (`stunner`), while Janus and the Ingress gateway will live in the `default` namespace. + +### Docker images + +Janus does not come with an official Docker image; thus, we built one using a self-made Dockerfile based on the available documents in the official [Janus repository](https://github.com/meetecho/janus-gateway). Actually, we've made two Dockerfiles. One for the Janus Gateway server and one for the Janus Web Demos. The [Janus Gateway server Dockerfile](./DOCKERFILE-janus-gateway) should be ran in the root directory of the [Janus repository](https://github.com/meetecho/janus-gateway). The Janus Web Demos Dockerfile should be used in the `/html` directory of the [same repository](https://github.com/meetecho/janus-gateway/tree/master/html). The images (`l7mp/janus-gateway:v1.2.4` and `l7mp/janus-web:latest`) used in the following demo are hosted on Docker Hub under the L7MP organization. + + +### TLS certificates + +As mentioned above, the Janus WebRTC server will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't then [nip.io](https://nip.io) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for Janus. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. + +Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of `nip.io`. + +### Ingress + +The first step of obtaining a valid cert is to install a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. + +Install an ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but this is not required. + +```console +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm repo update +helm install ingress-nginx ingress-nginx/ingress-nginx +``` + +Wait until Kubernetes assigns an external IP to the Ingress. + +```console +until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +``` + +Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. + +```console +kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' +export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') +``` + +### Cert manager + +We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. + +Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: + +```console +helm repo add cert-manager https://charts.jetstack.io +helm repo update +helm install cert-manager jetstack/cert-manager --namespace cert-manager \ + --create-namespace --set global.leaderElection.namespace=cert-manager \ + --set crds.enabled=true --timeout 600s +``` + +At this point we have all the necessary boilerplate set up to automate TLS issuance for Janus. + +### STUNner + +Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy (after some minor modifications) the [manifest](janus-server.yaml) packaged with STUNner. + +Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): + +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner +``` + +Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Janus Gateway pods. + +```console +git clone https://github.com/l7mp/stunner +cd stunner +kubectl apply -f docs/examples/janus/janus-call-stunner.yaml +``` + +The relevant parts here are the STUNner [Gateway definition](../../GATEWAY.md#gateway), which exposes the STUNner STUN/TURN server over UDP:3478 to the Internet, and the [UDPRoute definition](../../GATEWAY.md#udproute), which takes care of routing media to the pods running the Janus Gateway service. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: UDP +--- +apiVersion: stunner.l7mp.io/v1 +kind: UDPRoute +metadata: + name: janus + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - kind: Service + name: janus-gateway + namespace: default +``` + +Once the Gateway resource is installed into Kubernetes, STUNner will create a Kubernetes LoadBalancer for the Gateway to expose the TURN server on UDP:3478 to clients. It can take up to a minute for Kubernetes to allocate a public external IP for the service. + +Wait until Kubernetes assigns an external IP and store the external IP assigned by Kubernetes to +STUNner in an environment variable for later use. + +```console +until [ -n "$(kubectl get svc udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +export STUNNERIP=$(kubectl get service udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +``` + +### Janus + +The crucial step of integrating *any* WebRTC media server with STUNner is to ensure that the server instructs the clients to use STUNner as the STUN/TURN server. In order to achieve this, first we patch the public IP address of the STUNner STUN/TURN server we have learned above into our Janus Web and Gateway deployment manifests: + +```console +sed -i "s/stunner_ip/$STUNNERIP/g" docs/examples/janus/janus-server.yaml +``` + +Janus Web tells the connected clients where to look for the Janus Gateway server and which ICE servers should be used for ICE negotiation. Assuming that Kubernetes assigns the IP address 1.2.3.4 to STUNner (i.e., `STUNNERIP=1.2.3.4`), the relevant part of the Janus Web config would be something like the below: + +```yaml +... + settings.js: | + var server = "wss://server-$INGRESSIP.nip.io" + var iceServers = [{urls: "turn:1.2.3.4:3478?transport=udp", username: "user-1", credential: "pass-1"}] +``` + +This will make sure that Janus Web tells the clients to use STUNner as the STUN/TURN server. If unsure about the STUNner settings to use, you can always use the handy [`stunnerctl` CLI tool](/cmd/stunnerctl/README.md) to dump the running STUNner configuration. + +``` console +stunnerctl -n stunner config udp-gateway +Gateway: stunner/udp-gateway (loglevel: "all:INFO") +Authentication type: static, username/password: user-1/pass-1 +Listeners: + - Name: stunner/udp-gateway/udp-listener + Protocol: TURN-UDP + Public address:port: 34.118.88.91:3478 + Routes: [stunner/iperf-server] + Endpoints: [10.76.1.4, 10.80.4.47] +``` + +Note that Janus itself will not use STUNner as a TURN server (that would amount to a less efficient [symmetric ICE mode](../../DEPLOYMENT.md)); with the above configuration we are just telling Janus Web to instruct its clients to use STUNner to reach the Janus Gateway server. + +We also need the Ingress external IP address we have stored previously: this will make sure that the TLS certificate created by cert-manager will be bound to the proper `nip.io` domain and IP address. + +```console +sed -i "s/ingressserviceip/$INGRESSIP/g" docs/examples/janus/janus-server.yaml +``` + +Finally, fire up Janus. + +```console +kubectl apply -f docs/examples/janus/janus-server.yaml +``` + +The demo installation bundle includes a lot of resources to deploy Janus: + +- a Janus Gateway server, +- a web server serving the landing page using [Janus Web Demos](https://github.com/meetecho/janus-gateway/tree/master/html) +- a cluster issuer for the TLS certificates, +- an Ingress resource to terminate the secure connections between your browser and the Kubernetes cluster. + +Wait until all pods become operational and jump right into testing! + +## Test + +After installing everything, execute the following command to retrieve the URL of your freshly deployed Janus demo app: + +```console +echo client-$INGRESSIP.nip.io +``` + +Copy the URL into your browser, and now you should be greeted with the About page. On the landing page navigate to the Video call plugin demo (`/demos/videocall.html`). Duplicate the tab and register two users in the system and make a call. If everything is set up correctly, you should be able to connect to a room. If you repeat the procedure in a separate browser tab you can enjoy a nice video-conferencing session with yourself, with the twist that all media between the browser tabs is flowing through STUNner and the Janus Gateway server deployed in you Kubernetes cluster. + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/janus/janus-call-stunner.yaml b/docs/examples/janus/janus-call-stunner.yaml new file mode 100644 index 00000000..df5d4e09 --- /dev/null +++ b/docs/examples/janus/janus-call-stunner.yaml @@ -0,0 +1,51 @@ +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: stunner-gatewayclass +spec: + controllerName: "stunner.l7mp.io/gateway-operator" + parametersRef: + group: "stunner.l7mp.io" + kind: GatewayConfig + name: stunner-gatewayconfig + namespace: stunner + description: "STUNner is a WebRTC ingress gateway for Kubernetes" + +--- +apiVersion: stunner.l7mp.io/v1 +kind: GatewayConfig +metadata: + name: stunner-gatewayconfig + namespace: stunner +spec: + realm: stunner.l7mp.io + authType: static + userName: "user-1" + password: "pass-1" + +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: TURN-UDP +--- +apiVersion: stunner.l7mp.io/v1 +kind: UDPRoute +metadata: + name: janus + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - kind: Service + name: janus-gateway + namespace: default \ No newline at end of file diff --git a/docs/examples/janus/janus-server.yaml b/docs/examples/janus/janus-server.yaml new file mode 100644 index 00000000..1d6a6993 --- /dev/null +++ b/docs/examples/janus/janus-server.yaml @@ -0,0 +1,235 @@ +# Janus Gateway +apiVersion: v1 +kind: ConfigMap +metadata: + name: janus-gateway +data: + janus.jcfg: | + general: { + configs_folder = "@confdir@" + plugins_folder = "@plugindir@" + transports_folder = "@transportdir@" + events_folder = "@eventdir@" + loggers_folder = "@loggerdir@" + debug_level = 4 + admin_secret = "janusoverlord" + protected_folders = [ + "/bin", + "/boot", + "/dev", + "/etc", + "/initrd", + "/lib", + "/lib32", + "/lib64", + "/proc", + "/sbin", + "/sys", + "/usr", + "/var", + "/opt/janus/bin", + "/opt/janus/etc", + "/opt/janus/include", + "/opt/janus/lib", + "/opt/janus/lib32", + "/opt/janus/lib64", + "/opt/janus/sbin" + } + certificates: { + } + media: { + } + nat: { + nice_debug = false + ice_ignore_list = "vmnet" + plugins: { + + } + transports: { + + } + loggers: { + + } + events: { + } +--- +apiVersion: v1 +kind: Service +metadata: + name: janus-gateway + labels: + app.kubernetes.io/name: janus-gateway + app.kubernetes.io/instance: janus + app.kubernetes.io/version: "v1.2.4" +spec: + type: ClusterIP + ports: + - port: 8088 + targetPort: 8088 + protocol: TCP + name: http + - port: 8188 + targetPort: 8188 + protocol: TCP + name: websocket + selector: + app.kubernetes.io/name: janus-gateway + app.kubernetes.io/instance: janus +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: janus-gateway + labels: + app.kubernetes.io/name: janus-gateway + app.kubernetes.io/instance: janus + app.kubernetes.io/version: "v1.2.4" +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: janus-gateway + app.kubernetes.io/instance: janus + template: + metadata: + labels: + app.kubernetes.io/name: janus-gateway + app.kubernetes.io/instance: janus + spec: + terminationGracePeriodSeconds: 18000 # 5 hours + containers: + - name: janus-gateway + image: "l7mp/janus-gateway:v1.2.4" + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 8088 + protocol: TCP + - name: websocket + containerPort: 8188 + protocol: TCP +--- +# Janus Web Demos +apiVersion: v1 +kind: ConfigMap +metadata: + name: janus-web +data: + settings.js: | + var server = "wss://server-ingressserviceip.nip.io" + var iceServers = [{urls: "turn:stunner_ip:3478?transport=udp", username: "user-1", credential: "pass-1"}] +--- +apiVersion: v1 +kind: Service +metadata: + name: janus-web + labels: + app.kubernetes.io/name: janus-web + app.kubernetes.io/instance: janus-web + app.kubernetes.io/version: "v1.2.4" +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 80 + protocol: TCP + name: http + selector: + app.kubernetes.io/name: janus-web + app.kubernetes.io/instance: janus-web +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: janus-web + labels: + app.kubernetes.io/name: janus-web + app.kubernetes.io/instance: janus-web + app.kubernetes.io/version: "v1.2.4" +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: janus-web + app.kubernetes.io/instance: janus-web + template: + metadata: + labels: + app.kubernetes.io/name: janus-web + app.kubernetes.io/instance: janus-web + spec: + containers: + - name: janus-web + image: "l7mp/janus-web:latest" + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 80 + protocol: TCP + volumeMounts: + - name: config + mountPath: /usr/share/nginx/html/demos/settings.js + subPath: settings.js + volumes: + - name: config + configMap: + name: janus-web + items: + - key: settings.js + path: settings.js +--- +# Ingress for both Janus Gateway and Janus Web Demos +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: janus-web-demos + annotations: + cert-manager.io/cluster-issuer: "letsencrypt-prod" + nginx.ingress.kubernetes.io/enable-cors: "true" + nginx.ingress.kubernetes.io/cors-allow-origin: "*" + nginx.ingress.kubernetes.io/ssl-redirect: "true" +spec: + ingressClassName: nginx + tls: + - hosts: + - client-ingressserviceip.nip.io + - server-ingressserviceip.nip.io + secretName: janus-web-secret-tls + rules: + - host: server-ingressserviceip.nip.io + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: janus-gateway + port: + number: 8188 + - host: client-ingressserviceip.nip.io + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: janus-web + port: + number: 80 +--- +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: letsencrypt-prod +spec: + acme: + email: info@l7mp.io + server: https://acme-v02.api.letsencrypt.org/directory + privateKeySecretRef: + name: letsencrypt-secret-prod + solvers: + - http01: + ingress: + class: nginx +--- \ No newline at end of file diff --git a/docs/img/stunner_janus_arch.svg b/docs/img/stunner_janus_arch.svg new file mode 100644 index 00000000..a11d2428 --- /dev/null +++ b/docs/img/stunner_janus_arch.svg @@ -0,0 +1,4 @@ + + + +
Kubernetes  cluster
HTTPS
Ingress
UDP/RTP
Secure WebSocket
(wss)
STUN/TURN
Janus Web Demo
 \ No newline at end of file From b41c19ae6ca6e5a4c147e67f8fc693e9cbcde3b0 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Sep 2024 10:20:12 +0200 Subject: [PATCH 209/243] docs(rtd): Add Janus example to readthedocs --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index a31b1400..6ec4d400 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -24,6 +24,7 @@ nav: - 'Kurento one to one call': 'examples/kurento-one2one-call/README.md' - 'Kurento Magic Mirror': 'examples/kurento-magic-mirror/README.md' - 'LiveKit': 'examples/livekit/README.md' + - 'Janus': 'examples/janus/README.md' - 'Jitsi': 'examples/jitsi/README.md' - 'mediasoup': 'examples/mediasoup/README.md' - 'CloudRetro': 'examples/cloudretro/README.md' From c9c2c2ee063d21a5fa019e98f5513b819d5f298f Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 24 Sep 2024 11:24:40 +0200 Subject: [PATCH 210/243] docs: Mention Janus example --- README.md | 19 ++++++++++++------- docs/README.md | 3 ++- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 491be8a1..696cbc0e 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ minutes. The simplest way to deploy STUNner is through [Helm](https://helm.sh). STUNner configuration parameters are available for customization as [Helm -Values](https://helm.sh/docs/chart_template_guide/values_files). +Values](https://helm.sh/docs/chart_template_guide/values_files). ```console helm repo add stunner https://l7mp.io/stunner @@ -256,7 +256,7 @@ stunner` if it does not exist. namespace: stunner spec: realm: stunner.l7mp.io - authRef: + authRef: name: stunner-auth-secret namespace: stunner EOF @@ -315,7 +315,7 @@ stunner` if it does not exist. Gateway. This occurs by attaching a [UDPRoute](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1alpha2.UDPRoute) resource to the Gateway by setting the `parentRef` to the Gateway's name and specifying the - target service in the `backendRef`. + target service in the `backendRef`. The below UDPRoute will configure STUNner to [route client connections](/docs/GATEWAY.md#udproute) received on the Gateway called `udp-gateway` to the @@ -374,7 +374,7 @@ Listeners: ``` As it turns out, STUNner has successfully assigned a public IP and port to our Gateway and set the -STUN/TURN credentials based on the GatewayConfig. +STUN/TURN credentials based on the GatewayConfig. ### Testing @@ -571,14 +571,19 @@ applications into Kubernetes. obtain a valid TLS certificate to secure your signaling connections, courtesy of the [cert-manager](https://cert-manager.io) project, [nip.io](https://nip.io) and [Let's Encrypt](https://letsencrypt.org). +* [Video-conferencing with Janus](/docs/examples/janus/README.md): This tutorial helps you deploy a + fully fledged [Janus](https://janus.conf.meetecho.com/) video-conferencing service into Kubernetes + behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling + connections, using [cert-manager](https://cert-manager.io), [nip.io](https://nip.io) and [Let's + Encrypt](https://letsencrypt.org). * [Video-conferencing with Jitsi](/docs/examples/jitsi/README.md): This tutorial helps you deploy a fully fledged [Jitsi](https://jitsi.org) video-conferencing service into Kubernetes behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling connections, using [cert-manager](https://cert-manager.io), [nip.io](https://nip.io) and [Let's Encrypt](https://letsencrypt.org). -* [Video-conferencing with mediasoup](/docs/examples/mediasoup/README.md): This tutorial helps you deploy - the [mediasoup](https://mediasoup.org/) WebRTC media server behind STUNner. The docs also show how to - obtain a valid TLS certificate to secure your signaling connections, courtesy of the +* [Video-conferencing with mediasoup](/docs/examples/mediasoup/README.md): This tutorial helps you + deploy the [mediasoup](https://mediasoup.org/) WebRTC media server behind STUNner. The docs also + show how to obtain a valid TLS certificate to secure your signaling connections, courtesy of the [cert-manager](https://cert-manager.io) project, [nip.io](https://nip.io) and [Let's Encrypt](https://letsencrypt.org). * [Cloud-gaming with Cloudretro](/docs/examples/cloudretro/README.md): This tutorial lets you play Super diff --git a/docs/README.md b/docs/README.md index bf4c7f51..adc9cf73 100644 --- a/docs/README.md +++ b/docs/README.md @@ -40,8 +40,9 @@ * [Magic mirror with Kurento](examples/kurento-magic-mirror/README.md) * [Video-conferencing with LiveKit](examples/livekit/README.md) * [Video-conferencing with mediasoup](examples/mediasoup/README.md) +* [Video-conferencing with Janus](examples/janus/README.md) * [Video-conferencing with Jitsi](examples/jitsi/README.md) -* [Cloud-gaming with Cloudretro](examples/cloudretro/README.md) +* [Cloud-gaming with CloudRetro](examples/cloudretro/README.md) * [Remote desktop access with Neko](examples/neko/README.md) ## Manuals From fd58039d05b72ef46e119cfecf53ff257932a59e Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Sun, 29 Sep 2024 18:45:05 +0200 Subject: [PATCH 211/243] chore(rtd): Update deps --- docs/requirements.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/requirements.txt b/docs/requirements.txt index 2acd6ad1..f9065bfe 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -12,7 +12,7 @@ colorama==0.4.6 # via griffe ghp-import==2.1.0 # via mkdocs -griffe==1.2.0 +griffe==1.3.1 # via mkdocstrings-python jinja2==3.1.4 # via @@ -48,7 +48,7 @@ mkdocs-autorefs==1.2.0 # mkdocstrings-python mkdocs-get-deps==0.2.0 # via mkdocs -mkdocstrings[python]==0.26.0 +mkdocstrings[python]==0.26.1 # via # -r docs/requirements.in # mkdocstrings-python @@ -58,11 +58,11 @@ packaging==24.1 # via mkdocs pathspec==0.12.1 # via mkdocs -platformdirs==4.2.2 +platformdirs==4.3.6 # via # mkdocs-get-deps # mkdocstrings -pymdown-extensions==10.9 +pymdown-extensions==10.11 # via mkdocstrings python-dateutil==2.9.0.post0 # via ghp-import @@ -76,5 +76,5 @@ pyyaml-env-tag==0.1 # via mkdocs six==1.16.0 # via python-dateutil -watchdog==5.0.2 +watchdog==5.0.3 # via mkdocs From 5596d84d81d286ab9e36b2f341525df25234a43d Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 30 Sep 2024 10:14:49 +0200 Subject: [PATCH 212/243] chore(rtd): Fix github alert rendering --- docs/requirements.in | 1 + docs/requirements.txt | 5 ++++- mkdocs.yml | 1 + 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/requirements.in b/docs/requirements.in index 4ad415d6..e5bcdbf9 100644 --- a/docs/requirements.in +++ b/docs/requirements.in @@ -1,3 +1,4 @@ markdown-include +markdown-gfm-admonition mkdocs mkdocstrings[python] \ No newline at end of file diff --git a/docs/requirements.txt b/docs/requirements.txt index f9065bfe..32921e25 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -20,11 +20,14 @@ jinja2==3.1.4 # mkdocstrings markdown==3.7 # via + # markdown-gfm-admonition # markdown-include # mkdocs # mkdocs-autorefs # mkdocstrings # pymdown-extensions +markdown-gfm-admonition==0.1.1 + # via -r docs/requirements.in markdown-include==0.8.1 # via -r docs/requirements.in markupsafe==2.1.5 @@ -62,7 +65,7 @@ platformdirs==4.3.6 # via # mkdocs-get-deps # mkdocstrings -pymdown-extensions==10.11 +pymdown-extensions==10.11.1 # via mkdocstrings python-dateutil==2.9.0.post0 # via ghp-import diff --git a/mkdocs.yml b/mkdocs.yml index 6ec4d400..a9ef0a6c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -53,3 +53,4 @@ markdown_extensions: - pymdownx.inlinehilite - pymdownx.snippets - pymdownx.superfences + - gfm_admonition From c52752c825dc184279631827c93e4f9e2a5010c5 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 1 Oct 2024 17:39:43 +0200 Subject: [PATCH 213/243] doc: Fix wrong reg, fix #165 --- docs/AUTH.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/AUTH.md b/docs/AUTH.md index 360a719a..329b4e1b 100644 --- a/docs/AUTH.md +++ b/docs/AUTH.md @@ -66,7 +66,7 @@ kubectl -n stunner create secret generic stunner-auth-secret --from-literal=type --from-literal=username=my-user --from-literal=password=my-password ``` -Then, we update the [GatewayConfig](REFERENCE.md) to refer STUNner to this Secret for setting authentication credentials. +Then, we update the [GatewayConfig](GATEWAY.md) to refer STUNner to this Secret for setting authentication credentials. ```yaml apiVersion: stunner.l7mp.io/v1 From f677a943108415a12e645bf1b5123f1abfd04f8c Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 10 Oct 2024 17:27:36 +0200 Subject: [PATCH 214/243] chore(CI/CD): Bump helm to v3.16.2 --- .github/workflows/e2e-test-dev.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/e2e-test-dev.yaml b/.github/workflows/e2e-test-dev.yaml index fd7974de..109554a7 100644 --- a/.github/workflows/e2e-test-dev.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -38,7 +38,7 @@ jobs: - name: Set up Helm uses: azure/setup-helm@v4 with: - version: v3.14.2 + version: v3.16.2 - name: Install STUNner run: | diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index e1630eab..0f3b280d 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -37,7 +37,7 @@ jobs: - name: Set up Helm uses: azure/setup-helm@v4 with: - version: v3.14.2 + version: v3.16.2 - name: Install STUNner run: | From 9d1583de2e7b4a3660b7858dbe2d0b41c49eb1a8 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 24 Oct 2024 12:41:59 +0200 Subject: [PATCH 215/243] doc: Fix dead links --- README.md | 2 +- docs/examples/simple-tunnel/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 696cbc0e..f0137ed6 100644 --- a/README.md +++ b/README.md @@ -401,7 +401,7 @@ a heartwarming welcome message. ``` 1. We also need a STUN/TURN client to actually initiate a connection. STUNner comes with a handy - STUN/TURN client called [`turncat`](cmd/turncat/README.md) for this purpose. Once + STUN/TURN client called [`turncat`](docs/cmd/turncat.md) for this purpose. Once [installed](cmd/turncat/README.md#installation), you can fire up `turncat` to listen on the standard input and send everything it receives to STUNner. Type any input and press Enter, and you should see a nice greeting from your cluster! diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index 76773f90..b5c91f32 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -18,7 +18,7 @@ The tutorial assumes a fresh STUNner installation; see the [STUNner installation ### Setup -In this tutorial we perform a quick Kubernetes/STUNner benchmark: we fire up an iperf server inside the cluster and perform a speed test from the local console. We will use the [`turncat`](../../cmd/turncat) client utility to tunnel test traffic to the iperf server via STUNner acting as a STUN/TURN gateway. +In this tutorial we perform a quick Kubernetes/STUNner benchmark: we fire up an iperf server inside the cluster and perform a speed test from the local console. We will use the [`turncat`](../../cmd/turncat.md) client utility to tunnel test traffic to the iperf server via STUNner acting as a STUN/TURN gateway. ![STUNner benchmarks setup](../../img/stunner_benchmark.svg) From 79e322de31c39326f43ea6755e9cc645a929deec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Korn=C3=A9l=20D=C3=A1vid?= <47784437+davidkornel@users.noreply.github.com> Date: Mon, 28 Oct 2024 14:51:27 +0100 Subject: [PATCH 216/243] doc: Refactor examples and extend install guide (#164) --- README.md | 28 ++-- docs/INSTALL.md | 18 +- docs/README.md | 5 +- docs/examples/TLS.md | 103 ++++++++++++ docs/examples/benchmark/README.md | 4 + docs/examples/cloudretro/README.md | 17 +- docs/examples/direct-one2one-call/README.md | 4 + docs/examples/elixir-webrtc/README.md | 157 ++++++++++++++++++ .../elixir-webrtc/nexus-call-stunner.yaml | 51 ++++++ docs/examples/elixir-webrtc/nexus-server.yaml | 100 +++++++++++ docs/examples/janus/README.md | 79 ++------- docs/examples/jitsi/README.md | 91 ++-------- docs/examples/kurento-magic-mirror/README.md | 4 + docs/examples/kurento-one2one-call/README.md | 4 + docs/examples/livekit/README.md | 91 ++-------- docs/examples/livekit/livekit-server.yaml | 6 +- docs/examples/mediasoup/README.md | 90 ++-------- docs/examples/neko/README.md | 15 +- docs/examples/simple-tunnel/README.md | 6 +- docs/img/stunner_mediasoup.svg | 4 + mkdocs.yml | 6 +- 21 files changed, 549 insertions(+), 334 deletions(-) create mode 100644 docs/examples/TLS.md create mode 100644 docs/examples/elixir-webrtc/README.md create mode 100644 docs/examples/elixir-webrtc/nexus-call-stunner.yaml create mode 100644 docs/examples/elixir-webrtc/nexus-server.yaml create mode 100644 docs/img/stunner_mediasoup.svg diff --git a/README.md b/README.md index f0137ed6..6638aedf 100644 --- a/README.md +++ b/README.md @@ -555,17 +555,6 @@ applications into Kubernetes. ### Media-plane deployment model -* [One to one video call with Kurento](/docs/examples/kurento-one2one-call/README.md): This tutorial - shows how to use STUNner to connect WebRTC clients to a media server deployed into Kubernetes - behind STUNner in the [media-plane deployment model](/docs/DEPLOYMENT.md). All this happens - *without* modifying the media server code in any way, just by adding 5-10 lines of - straightforward JavaScript to configure clients to use STUNner as the TURN server. -* [Magic mirror with Kurento](/docs/examples/kurento-magic-mirror/README.md): This tutorial has been - adopted from the [Kurento](https://www.kurento.org) [magic - mirror](https://doc-kurento.readthedocs.io/en/stable/tutorials/node/tutorial-magicmirror.html) - demo, deploying a basic WebRTC loopback server behind STUNner with some media processing - added. In particular, the application uses computer vision and augmented reality techniques to - add a funny hat on top of faces. * [Video-conferencing with LiveKit](/docs/examples/livekit/README.md): This tutorial helps you deploy the [LiveKit](https://livekit.io) WebRTC media server behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling connections, courtesy of the @@ -576,6 +565,12 @@ applications into Kubernetes. behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling connections, using [cert-manager](https://cert-manager.io), [nip.io](https://nip.io) and [Let's Encrypt](https://letsencrypt.org). +* [Video-conferencing with Elixir WebRTC](/docs/examples/elixir-webrtc/README.md): This tutorial helps + you deploy a fully fledged [Elixir WebRTC](https://elixir-webrtc.org/) video-conferencing room called + [Nexus](https://github.com/elixir-webrtc/apps/tree/master/nexus) into Kubernetes + behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling + connections, using [cert-manager](https://cert-manager.io), [nip.io](https://nip.io) and [Let's + Encrypt](https://letsencrypt.org). * [Video-conferencing with Jitsi](/docs/examples/jitsi/README.md): This tutorial helps you deploy a fully fledged [Jitsi](https://jitsi.org) video-conferencing service into Kubernetes behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling @@ -595,6 +590,17 @@ applications into Kubernetes. providing an ingress gateway service to a remote desktop application. We use [neko.io](https://neko.m1k1o.net) to run a browser in a secure container inside the Kubernetes cluster, and stream the desktop to clients via STUNner. +* [One to one video call with Kurento](/docs/examples/kurento-one2one-call/README.md): This tutorial + shows how to use STUNner to connect WebRTC clients to a media server deployed into Kubernetes + behind STUNner in the [media-plane deployment model](/docs/DEPLOYMENT.md). All this happens + *without* modifying the media server code in any way, just by adding 5-10 lines of + straightforward JavaScript to configure clients to use STUNner as the TURN server. +* [Magic mirror with Kurento](/docs/examples/kurento-magic-mirror/README.md): This tutorial has been + adopted from the [Kurento](https://www.kurento.org) [magic + mirror](https://doc-kurento.readthedocs.io/en/stable/tutorials/node/tutorial-magicmirror.html) + demo, deploying a basic WebRTC loopback server behind STUNner with some media processing + added. In particular, the application uses computer vision and augmented reality techniques to + add a funny hat on top of faces. ## Documentation diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 34b4c295..22771a9a 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -28,7 +28,7 @@ helm install stunner-gateway-operator stunner/stunner-gateway-operator --create- --namespace=stunner-system ``` -And that's all: you don't need to install the dataplane separately, this is handled automatically by the operator. The `stunnerd` pods created by the operator can be customized using the Dataplane custom resource: you can specify the `stunnerd` container image version, provision resources per each `stunenrd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1alpha1#DataplaneSpec). +And that's all: you don't need to install the dataplane separately, this is handled automatically by the operator. The `stunnerd` pods created by the operator can be customized using the Dataplane custom resource: you can specify the `stunnerd` container image version, provision resources per each `stunnerd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1alpha1#DataplaneSpec). ### Development version @@ -63,6 +63,22 @@ helm install stunner-prod stunner/stunner --create-namespace --namespace=stunner helm install stunner-dev stunner/stunner --create-namespace --namespace=stunner-dev ``` +### Skip install CRDs + +You can install the STUNner chart without the Gateway API CRDs and STUNner CRDs with the `--skip-crds` flag. However, ensure that the CRDs are already present in the cluster, as the STUNner Gateway Operator will fail to start without them. + +```console +helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace \ + --namespace=stunner-system --skip-crds +``` + +To manually install the CRDs: + +```console +kubectl apply -k github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.0.0 +kubectl apply -f https://raw.githubusercontent.com/l7mp/stunner-helm/refs/heads/main/helm/stunner-gateway-operator/crds/stunner-crd.yaml +``` + ## Customization The Helm charts let you fine-tune STUNner features, including [compute resources](#resources) provisioned for `stunnerd` pods, [UDP multithreading](#udp-multithreading), and[graceful shutdown](#graceful-shutdown). diff --git a/docs/README.md b/docs/README.md index adc9cf73..14a15cef 100644 --- a/docs/README.md +++ b/docs/README.md @@ -36,14 +36,15 @@ ### Media-plane deployment model -* [One to one video call with Kurento](examples/kurento-one2one-call) -* [Magic mirror with Kurento](examples/kurento-magic-mirror/README.md) * [Video-conferencing with LiveKit](examples/livekit/README.md) * [Video-conferencing with mediasoup](examples/mediasoup/README.md) * [Video-conferencing with Janus](examples/janus/README.md) +* [Video-conferencing with Elixir WebRTC](examples/elixir-webrtc/README.md) * [Video-conferencing with Jitsi](examples/jitsi/README.md) * [Cloud-gaming with CloudRetro](examples/cloudretro/README.md) * [Remote desktop access with Neko](examples/neko/README.md) +* [One to one video call with Kurento](examples/kurento-one2one-call) +* [Magic mirror with Kurento](examples/kurento-magic-mirror/README.md) ## Manuals diff --git a/docs/examples/TLS.md b/docs/examples/TLS.md new file mode 100644 index 00000000..34a55025 --- /dev/null +++ b/docs/examples/TLS.md @@ -0,0 +1,103 @@ +# TLS + +This documentation sums up the TLS and certificate issues you will encounter deploying the examples. + +## The issue + +Some client-side application must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. In the demos, we will aim to obtain a proper CA-signed certificate (self-signed certificates haven't been tested). Obtaining a valid TLS certificate is a challenge. Thus, the majority of the installation guides will be about securing client connections to the client-side apps and the WebRTC mediaservers over TLS. Once HTTPS is correctly working, integrating the mediaservers with STUNner is very simple. + +## TLS certificates + +Some WebRTC servers will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't, we still have a solution for that. + +> [!NOTE] +> +> By default, the examples and commands snippets assume you don't own a domain. + +### If you don't have your own domain + +[nip.io](https://nip.io) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for the mediaserver. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. + +### If you have your own domain + +We use `nip.io` to "own a domain" in some examples. To replace it with your own domain, you must locate the corresponding lines in the specific mediaserver's configuration file and overwrite them. + +> [!NOTE] +> +> Although they might look similar, every mediaserver has a different configuration. You might need to (re)configure more things in one mediaserver than another. + +> [!NOTE] +> +> Make sure to set up your Ingress correctly and do not forget to create a new DNS record pointing to your Ingress' IP address! + +## Installation + +### Ingress + +The first step of secured traffic ingestion is obtaining a valid cert by installing a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. + +Install an Ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but other Ingress implementations might work (check their documentation for install steps). + +```console +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm repo update +helm install ingress-nginx ingress-nginx/ingress-nginx +``` + +Wait until Kubernetes assigns an external IP to the Ingress. + +```console +until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +``` + +Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. + +```console +kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' +export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') +``` + +### Cert manager + +We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. + +Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: + +```console +helm repo add cert-manager https://charts.jetstack.io +helm repo update +helm install cert-manager jetstack/cert-manager --namespace cert-manager \ + --create-namespace --set global.leaderElection.namespace=cert-manager \ + --set crds.enabled=true --timeout 600s +``` + +At this point we have all the necessary boilerplate set up to automate TLS issuance for the demo. + +## Troubleshooting + +#### Wildcard DNS domain rate limiting + +Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of `nip.io`. + +#### Certificate issuance + +If you work with certificates you must be aware that signing a certificate request takes some time and it differs for every CA (certificate authority). If you sense there is a problem with the certificate being signed or issued, you can check it directly and see what is going on. + +First, you'll need to find the certificate and its related resources in your cluster. +```console +kubectl get certificate -A +kubectl get certificaterequests.cert-manager.io -A +kubectl get certificatesigningrequests.certificates.k8s.io +``` + +To find more information about them +```console +kubectl describe certificate -A +kubectl describe certificaterequests.cert-manager.io -A +kubectl describe certificatesigningrequests.certificates.k8s.io +``` + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/benchmark/README.md b/docs/examples/benchmark/README.md index 605abe70..75497e97 100644 --- a/docs/examples/benchmark/README.md +++ b/docs/examples/benchmark/README.md @@ -165,3 +165,7 @@ Notice that the average packets/second rate will be slightly lower in case of a * It is advised to repeat the measurment with different packet sizes. Recommended packet sizes in bytes are 64, 128, 256, 512, 1024, and 1200. Small packet sizes result lower effective throughput (when packet drop is < 1%). * Measuring [measuring one-way latency](https://stackoverflow.com/questions/63793030/iperf2-latency-is-a-two-way-or-one-way-latency) with `iperf` requires the clocks at the iperf client and server to be synchronized. Without this the results may be corrupted, and you may even see negative latencies. + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/cloudretro/README.md b/docs/examples/cloudretro/README.md index 206aa256..79797f33 100644 --- a/docs/examples/cloudretro/README.md +++ b/docs/examples/cloudretro/README.md @@ -15,11 +15,7 @@ In this demo you will learn how to: ## Prerequisites -The below installation instructions require an operational cluster running a supported version of -Kubernetes (>1.22). Most hosted or private Kubernetes cluster services will work, but make sure -that the cluster comes with a functional load-balancer integration (all major hosted Kubernetes -services should support this). Otherwise, STUNner will not be able to allocate a public IP address -for clients to reach your WebRTC infra. +See prerequisites [here](../../INSTALL.md#prerequisites). You will need a basic familiarity [with the CloudRetro architecture](https://webrtchacks.com/open-source-cloud-gaming-with-webrtc), especially the concept @@ -79,12 +75,7 @@ the CloudRetro servers running on a private pod IP address. That is where STUNne ### STUNner -First we install STUNner gateway operator with helm ([more info](https://github.com/l7mp/stunner-helm)): -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner -``` +First we install the stable version of STUNner, please follow the instructions in [this section](../../INSTALL.md#installation-1). Wait until all the necessary resources are up and running, then you are ready to continue. @@ -242,3 +233,7 @@ kubectl delete -f cloudretro-setup-coordinator.yaml kubectl delete -f cloudretro-setup-workers.yaml kubectl delete -f cloudretro-stunner-cleanup.yaml ``` + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index 398ab618..30dca9e2 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -296,3 +296,7 @@ Delete the demo deployment using the below command: kubectl delete -f docs/examples/direct-one2one-call/direct-one2one-call-server.yaml kubectl delete -f docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml ``` + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/elixir-webrtc/README.md b/docs/examples/elixir-webrtc/README.md new file mode 100644 index 00000000..38c22d9f --- /dev/null +++ b/docs/examples/elixir-webrtc/README.md @@ -0,0 +1,157 @@ +# STUNner demo: Video-conferencing with Nexus + +This document guides you through the installation of [Elixir WebRTC](https://elixir-webrtc.org/), called [Nexus](https://github.com/elixir-webrtc/apps/tree/master/nexus) into Kubernetes, when it is used together with the STUNner WebRTC media gateway. + +In this demo you will learn to: + +- integrate a typical WebRTC application with STUNner, +- obtain a valid TLS certificate to secure the signaling plane, +- deploy the Nexus app server into Kubernetes, and +- configure STUNner to expose Nexus to clients. + +## Prerequisites + +To run this example, you need: +* a [Kubernetes cluster](../../INSTALL.md#prerequisites), +* a [deployed STUNner](../../INSTALL.md#installation-1) (presumably the latest stable version), +* an [Ingress controller](../TLS.md#ingress) to ingest traffic into the cluster, +* a [Cert-manager](../TLS.md#cert-manager) to automate TLS certificate management. + +> [!NOTE] +> +> If you have your own TLS certificate, put it in a `Secret` [resource](https://kubernetes.io/docs/concepts/configuration/secret/) and deploy it into the `default` namespace under the `nexus-secret-tls` name. + +### STUNner + +Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy (after some minor modifications) the [manifest](livekit-server.yaml) packaged with STUNner. + +To install the stable version of STUNner, please follow the instructions in [this section](../../INSTALL.md#installation-1). + +Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Nexus pods. + +```console +git clone https://github.com/l7mp/stunner +cd stunner +kubectl apply -f docs/examples/elixir-webrtc/nexus-call-stunner.yaml +``` + +The relevant parts here are the STUNner [Gateway definition](../../GATEWAY.md#gateway), which exposes the STUNner STUN/TURN server over UDP:3478 to the Internet, and the [UDPRoute definition](../../GATEWAY.md#udproute), which takes care of routing media to the pods running the Nexus Gateway service. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: UDP +--- +apiVersion: stunner.l7mp.io/v1 +kind: UDPRoute +metadata: + name: nexus + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - kind: Service + name: nexus + namespace: default +``` + +Once the Gateway resource is installed into Kubernetes, STUNner will create a Kubernetes LoadBalancer for the Gateway to expose the TURN server on UDP:3478 to clients. It can take up to a minute for Kubernetes to allocate a public external IP for the service. + +Wait until Kubernetes assigns an external IP and store the external IP assigned by Kubernetes to +STUNner in an environment variable for later use. + +```console +until [ -n "$(kubectl get svc udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done +export STUNNERIP=$(kubectl get service udp-gateway -n stunner -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +``` +### Nexus Docker images + +The crucial step of integrating *any* WebRTC media server with STUNner is to ensure that the server instructs the clients to use STUNner as the STUN/TURN server. +Unfortunately, currently the [official Nexus Docker image](ghcr.io/elixir-webrtc/apps/nexus) does not support this configuration in runtime (by default Google's STUN server is hardcoded into it). +Therefore, we have to modify this setting to STUNner's IP and build a new Docker image. + +In order to achieve this, first clone the Elixir WebRTC sample app repository: + +```console +git clone https://github.com/elixir-webrtc/apps/ +cd apps/nexus +``` + +You have to modify the ICE config to use STUNner with the given credentials in two files: + +`assets/js/home.js`: +``` +... +# const pcConfig = { iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] }; + +# change to: + + const pcConfig = { iceServers: [{ urls: 'turn::3478?transport=udp', username: 'user-1', credential: 'pass-1',iceTransportPolicy: 'relay'}] }; +``` + +`lib/nexus/peer.ex`: +``` +... +# ice_servers: [%{urls: "stun:stun.l.google.com:19302"}], + +# change to: + + ice_servers: [%{urls: "turn::3478?transport=udp", "username": "user-1", "credential": "pass-1", "iceTransportPolicy": "relay"}], +``` + +Now rebuild the Docker image, and push it into your image repository: +``` +export MYREPO=myrepo # use your own Docker repository name! +sudo docker build -t $MYREPO/nexus . +sudo docker push $MYREPO/nexus +``` + +After uploading the image, you also have to modify the Nexus image repo location in the Kubernetes deployment file. + +```console +sed -i "s/l7mp/$MYREPO/g" docs/examples/elixir-webrtc/nexus-server.yaml +``` + +We also need the Ingress external IP address we have stored previously: this will make sure that the TLS certificate created by cert-manager will be bound to the proper `nip.io` domain and IP address. + +```console +sed -i "s/ingressserviceip/$INGRESSIP/g" docs/examples/elixir-webrtc/nexus-server.yaml +``` + +Finally, fire up Nexus. + +```console +kubectl apply -f docs/examples/elixir-webrtc/nexus-server.yaml +``` + +The demo installation bundle includes a few resources to deploy Nexus: + +- Nexus deployment and service, +- a cluster issuer for the TLS certificates, +- an Ingress resource to terminate the secure connections between your browser and the Kubernetes cluster. + +Wait until all pods become operational and jump right into testing! + +## Test + +After installing everything, execute the following command to retrieve the URL of your freshly deployed Nexus demo app: + +```console +echo INGRESSIP.nip.io +``` + +Copy the URL into your browser, and if everything is set up correctly, you should be able to connect to a video room. If you repeat the procedure in a separate browser tab you can enjoy a nice video-conferencing session with yourself, with the twist that all media between the browser tabs is flowing through STUNner and the Nexus server deployed in you Kubernetes cluster. + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/elixir-webrtc/nexus-call-stunner.yaml b/docs/examples/elixir-webrtc/nexus-call-stunner.yaml new file mode 100644 index 00000000..ed052177 --- /dev/null +++ b/docs/examples/elixir-webrtc/nexus-call-stunner.yaml @@ -0,0 +1,51 @@ +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: stunner-gatewayclass +spec: + controllerName: "stunner.l7mp.io/gateway-operator" + parametersRef: + group: "stunner.l7mp.io" + kind: GatewayConfig + name: stunner-gatewayconfig + namespace: stunner + description: "STUNner is a WebRTC ingress gateway for Kubernetes" + +--- +apiVersion: stunner.l7mp.io/v1 +kind: GatewayConfig +metadata: + name: stunner-gatewayconfig + namespace: stunner +spec: + realm: stunner.l7mp.io + authType: static + userName: "user-1" + password: "pass-1" + +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: udp-gateway + namespace: stunner +spec: + gatewayClassName: stunner-gatewayclass + listeners: + - name: udp-listener + port: 3478 + protocol: TURN-UDP +--- +apiVersion: stunner.l7mp.io/v1 +kind: UDPRoute +metadata: + name: nexus + namespace: stunner +spec: + parentRefs: + - name: udp-gateway + rules: + - backendRefs: + - kind: Service + name: nexus + namespace: default \ No newline at end of file diff --git a/docs/examples/elixir-webrtc/nexus-server.yaml b/docs/examples/elixir-webrtc/nexus-server.yaml new file mode 100644 index 00000000..dd0db130 --- /dev/null +++ b/docs/examples/elixir-webrtc/nexus-server.yaml @@ -0,0 +1,100 @@ +apiVersion: v1 +kind: Service +metadata: + name: nexus + labels: + app.kubernetes.io/name: nexus +spec: + type: ClusterIP + ports: + - port: 4000 + targetPort: 4000 + protocol: TCP + name: http + selector: + app.kubernetes.io/name: nexus +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nexus + labels: + app.kubernetes.io/name: nexus +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: nexus + template: + metadata: + labels: + app.kubernetes.io/name: nexus + spec: + terminationGracePeriodSeconds: 18000 # 5 hours + containers: + - name: nexus + image: "l7mp/l7mp/elixir-webrtc-nexus" + imagePullPolicy: IfNotPresent + env: + - name: SECRET_KEY_BASE + value: f0e01531666a6ccd5b15879a3c00a15f8feaaea86230cf35d96fe0b8e468bc553549e47edb737a4d1530d1c56a3a7a3827443e1bd00aa46adb44dc6aa087ff9f + - name: PHX_SERVER + value: "1" + - name: PHX_HOST + value: ingressserviceip.nip.io + - name: ADMIN_USERNAME + value: admin + - name: ADMIN_PASSWORD + value: admin + - name: ICE_PORT_RANGE + value: 62000-63000 + - name: NEXUS_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + ports: + - name: http + containerPort: 4000 + protocol: TCP +--- +# Ingress for both Nexus +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: nexus + annotations: + cert-manager.io/cluster-issuer: "letsencrypt-prod" + nginx.ingress.kubernetes.io/ssl-redirect: "true" +spec: + ingressClassName: nginx + tls: + - hosts: + - ingressserviceip.nip.io + secretName: nexus-secret-tls + rules: + - host: ingressserviceip.nip.io + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: nexus + port: + number: 4000 +--- +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: letsencrypt-prod +spec: + acme: + email: info@l7mp.io + server: https://acme-v02.api.letsencrypt.org/directory + privateKeySecretRef: + name: letsencrypt-secret-prod + solvers: + - http01: + ingress: + class: nginx +--- \ No newline at end of file diff --git a/docs/examples/janus/README.md b/docs/examples/janus/README.md index d7e87c5d..8a128704 100644 --- a/docs/examples/janus/README.md +++ b/docs/examples/janus/README.md @@ -11,9 +11,17 @@ In this demo you will learn to: ## Prerequisites -The below installation instructions require an operational cluster running a supported version of Kubernetes (>1.22). Most hosted or private Kubernetes cluster services will work, but make sure that the cluster comes with a functional load-balancer integration (all major hosted Kubernetes services should support this). Otherwise, STUNner will not be able to allocate a public IP address for clients to reach your WebRTC infra. As a regrettable exception, Minikube is unfortunately not supported for this demo. The reason is that [Let's Encrypt certificate issuance is not available with nip.io](https://medium.com/@EmiiKhaos/there-is-no-possibility-that-you-can-get-lets-encrypt-certificate-with-nip-io-7483663e0c1b); later on you will learn more about why this is crucial above. +To run this example, you need: +* a [Kubernetes cluster](../../INSTALL.md#prerequisites), +* a [deployed STUNner](../../INSTALL.md#installation-1) (presumably the latest stable version), +* an [Ingress controller](../TLS.md#ingress) to ingest traffic into the cluster, +* a [Cert-manager](../TLS.md#cert-manager) to automate TLS certificate management. -## Setup +> [!NOTE] +> +> If you have your own TLS certificate, put it in a `Secret` [resource](https://kubernetes.io/docs/concepts/configuration/secret/) and deploy it into the `default` namespace under the `janus-web-secret-tls` name. + +## Description The recommended way (or at least the possible way, [link](https://janus.discourse.group/t/janus-with-kubernetes-demystifying-the-myths/938), [link](https://bugraoz93.medium.com/active-passive-highly-availability-janus-gateway-on-kubernetes-2189256e5525)) to install Janus into Kubernetes is deploying the media servers into the host-network namespace of the Kubernetes nodes (`hostNetwork: true`). This deployment model, however, comes with a set of uncanny [operational limitations and security concerns](../../WHY.md). Using STUNner, however, media servers can be deployed into ordinary Kubernetes pods and run over a private IP network, like any "normal" Kubernetes workload. @@ -23,78 +31,15 @@ The figure below shows Janus deployed into regular Kubernetes pods behind STUNne In this tutorial we deploy [Janus Gateway](https://github.com/meetecho/janus-gateway/tree/master) with a set of [preimplemented and packaged server plugins](https://janus.conf.meetecho.com/docs/pluginslist.html) for media exchange, a [Janus Web Demo](https://github.com/meetecho/janus-gateway/tree/master/html), a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the Janus server pool to clients. -## Installation - -> [!NOTE] -> -> Let's start with a disclaimer. The Janus client-side application must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. In this demo, we will aim to obtain a proper CA-signed certificate (self-signed certificates haven't been tested). Obtaining a valid TLS certificate is a challenge. Thus, the majority of the below installation guide will be about securing client connections to Janus over TLS; as it turns out, once HTTPS is correctly working integrating Janus with STUNner is very simple. - -In the below example, STUNner will be installed into the identically named namespace (`stunner`), while Janus and the Ingress gateway will live in the `default` namespace. - ### Docker images Janus does not come with an official Docker image; thus, we built one using a self-made Dockerfile based on the available documents in the official [Janus repository](https://github.com/meetecho/janus-gateway). Actually, we've made two Dockerfiles. One for the Janus Gateway server and one for the Janus Web Demos. The [Janus Gateway server Dockerfile](./DOCKERFILE-janus-gateway) should be ran in the root directory of the [Janus repository](https://github.com/meetecho/janus-gateway). The Janus Web Demos Dockerfile should be used in the `/html` directory of the [same repository](https://github.com/meetecho/janus-gateway/tree/master/html). The images (`l7mp/janus-gateway:v1.2.4` and `l7mp/janus-web:latest`) used in the following demo are hosted on Docker Hub under the L7MP organization. - -### TLS certificates - -As mentioned above, the Janus WebRTC server will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't then [nip.io](https://nip.io) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for Janus. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. - -Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of `nip.io`. - -### Ingress - -The first step of obtaining a valid cert is to install a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. - -Install an ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but this is not required. - -```console -helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx -helm repo update -helm install ingress-nginx ingress-nginx/ingress-nginx -``` - -Wait until Kubernetes assigns an external IP to the Ingress. - -```console -until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -``` - -Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. - -```console -kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' -export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') -``` - -### Cert manager - -We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. - -Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: - -```console -helm repo add cert-manager https://charts.jetstack.io -helm repo update -helm install cert-manager jetstack/cert-manager --namespace cert-manager \ - --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set crds.enabled=true --timeout 600s -``` - -At this point we have all the necessary boilerplate set up to automate TLS issuance for Janus. - ### STUNner Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy (after some minor modifications) the [manifest](janus-server.yaml) packaged with STUNner. -Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner -``` +To install the stable version of STUNner, please follow the instructions in [this section](../../INSTALL.md#installation-1). Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Janus Gateway pods. @@ -206,7 +151,7 @@ After installing everything, execute the following command to retrieve the URL o echo client-$INGRESSIP.nip.io ``` -Copy the URL into your browser, and now you should be greeted with the About page. On the landing page navigate to the Video call plugin demo (`/demos/videocall.html`). Duplicate the tab and register two users in the system and make a call. If everything is set up correctly, you should be able to connect to a room. If you repeat the procedure in a separate browser tab you can enjoy a nice video-conferencing session with yourself, with the twist that all media between the browser tabs is flowing through STUNner and the Janus Gateway server deployed in you Kubernetes cluster. +Copy the URL into your browser, and now you should be greeted with the About page. On the landing page navigate to the Video call plugin demo (`/demos/videocall.html`). Duplicate the tab and register two users in the system and make a call. If everything is set up correctly, you should be able to connect to a room. If you repeat the procedure in a separate browser tab you can enjoy a nice video-conferencing session with yourself, with the twist that all media between the browser tabs is flowing through STUNner and the Janus Gateway server deployed in you Kubernetes cluster. # Help diff --git a/docs/examples/jitsi/README.md b/docs/examples/jitsi/README.md index 380aaaf4..aa779736 100644 --- a/docs/examples/jitsi/README.md +++ b/docs/examples/jitsi/README.md @@ -11,11 +11,17 @@ In this demo you will learn to: ## Prerequisites -The tutorial assumes a fresh STUNner installation; see the STUNner installation and configuration guide. Create a namespace called stunner if there is none. You need a WebRTC-compatible browser to run this tutorial. Basically any modern browser will do; we usually test our WebRTC applications with Firefox and Chrome. +To run this example, you need: +* a [Kubernetes cluster](../../INSTALL.md#prerequisites), +* a [deployed STUNner](../../INSTALL.md#installation-1) (presumably the latest stable version), +* an [Ingress controller](../TLS.md#ingress) to ingest traffic into the cluster, +* a [Cert-manager](../TLS.md#cert-manager) to automate TLS certificate management. -As a regrettable exception, Minikube is unfortunately not supported for this demo. The reason is that [Let's Encrypt certificate issuance is not available with nip.io](https://medium.com/@EmiiKhaos/there-is-no-possibility-that-you-can-get-lets-encrypt-certificate-with-nip-io-7483663e0c1b); late on you will learn more about why this is crucial above. +> [!NOTE] +> +> If you have your own TLS certificate, put it in a `Secret` [resource](https://kubernetes.io/docs/concepts/configuration/secret/) and deploy it into the `default` namespace under the `jitsi-secret` name. -## Setup +## Description The recommended way to install Jitsi into Kubernetes is deploying the media servers into the host-network namespace of the Kubernetes nodes (`hostNetwork: true`), or using a NodePort service or a dedicated Ingress to ingest WebRTC media traffic into the network. However, these options allow only one JVB instance per Kubernetes node and the host-network deployment model also comes with a set of uncanny [operational limitations and security concerns](../../WHY.md). Using STUNner, however, media servers can be deployed into ordinary Kubernetes pods and run over a private IP network, like any "normal" Kubernetes workload. @@ -25,82 +31,11 @@ The figure below shows Jitsi deployed into regular Kubernetes pods behind STUNne In this tutorial we deploy a video room example using the [Jitsi framework](https://jitsi.github.io/handbook/docs/architecture) for media exchange, a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the Jitsi JVB to clients. -## Installation - -Let's start with a disclaimer. The Jitsi client example browser must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. Unfortunately, self-signed TLS certs will not work, so we have to come up with a way to provide our clients with a valid TLS cert. - -In the below example, STUNner will be installed into the identically named namespace, while Jitsi and the Ingress gateway will live in the default namespace. - -### TLS certificates - -As mentioned above, the Jitsi server will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't then [nip.io](https://nip.io/) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for Jitsi. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. - -Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of nip.io. - -### Ingress - -The first step of obtaining a valid cert is to install a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. - -Install an ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but this is not required. - -```console -helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx -helm repo update -helm install ingress-nginx ingress-nginx/ingress-nginx -``` - -Wait until Kubernetes assigns an external IP to the Ingress. - -```console -until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -``` - -Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. - -```console -kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' -export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') -``` - -### Cert manger - -We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. - -Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: - -```console -helm repo add cert-manager https://charts.jetstack.io -helm repo update -helm install cert-manager jetstack/cert-manager --namespace cert-manager \ - --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set installCRDs=true --timeout 600s -``` - -At this point we have all the necessary boilerplate set up to automate TLS issuance for Jitsi. - ### STUNner -Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy the [manifest](jitsi-server.yaml) packaged with STUNner. - -Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): +Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy (after some minor modifications) the [manifest](jitsi-server.yaml) packaged with STUNner. -Legacy mode: - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy -helm install stunner stunner/stunner --create-namespace --namespace=stunner -``` - -Managed mode: - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system -``` +To install the stable version of STUNner, please follow the instructions in [this section](../../INSTALL.md#installation-1). Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the Jitsi server pods. @@ -205,3 +140,7 @@ echo $INGRESSIP.nip.io ``` Copy the URL into your browser, and now you should be greeted with the Jitsi webpage. In the landing page you should create a room first. After you created a room you can set your username and join the room. On another page you have to open this page again and you should see the previously created room in the list. You only have to connect this room with another user. + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/kurento-magic-mirror/README.md b/docs/examples/kurento-magic-mirror/README.md index 4ef29050..dbffe576 100644 --- a/docs/examples/kurento-magic-mirror/README.md +++ b/docs/examples/kurento-magic-mirror/README.md @@ -79,3 +79,7 @@ health-checks and load-balancing should just work as expected. ```console $ kubectl scale deployment kms --replicas=4 ``` + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/kurento-one2one-call/README.md b/docs/examples/kurento-one2one-call/README.md index fa5c827c..bb5dc3a9 100644 --- a/docs/examples/kurento-one2one-call/README.md +++ b/docs/examples/kurento-one2one-call/README.md @@ -371,3 +371,7 @@ Delete the demo deployment using the below command: kubectl delete -f docs/examples/kurento-one2one-call/kurento-one2one-call-server.yaml kubectl delete -f docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml ``` + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/livekit/README.md b/docs/examples/livekit/README.md index 60a3468e..99aa02c2 100644 --- a/docs/examples/livekit/README.md +++ b/docs/examples/livekit/README.md @@ -11,9 +11,18 @@ In this demo you will learn to: ## Prerequisites -The below installation instructions require an operational cluster running a supported version of Kubernetes (>1.22). Most hosted or private Kubernetes cluster services will work, but make sure that the cluster comes with a functional load-balancer integration (all major hosted Kubernetes services should support this). Otherwise, STUNner will not be able to allocate a public IP address for clients to reach your WebRTC infra. As a regrettable exception, Minikube is unfortunately not supported for this demo. The reason is that [Let's Encrypt certificate issuance is not available with nip.io](https://medium.com/@EmiiKhaos/there-is-no-possibility-that-you-can-get-lets-encrypt-certificate-with-nip-io-7483663e0c1b); late on you will learn more about why this is crucial above. +To run this example, you need: +* a [Kubernetes cluster](../../INSTALL.md#prerequisites), +* a [deployed STUNner](../../INSTALL.md#installation-1) (presumably the latest stable version), +* an [Ingress controller](../TLS.md#ingress) to ingest traffic into the cluster, +* a [Cert-manager](../TLS.md#cert-manager) to automate TLS certificate management. -## Setup +> [!NOTE] +> +> If you have your own TLS certificate, put it in a `Secret` [resource](https://kubernetes.io/docs/concepts/configuration/secret/) and deploy it into the `default` namespace under the `livekit-secret-tls` name. + + +## Description The recommended way to install LiveKit into Kubernetes is deploying the media servers into the host-network namespace of the Kubernetes nodes (`hostNetwork: true`). This deployment model, however, comes with a set of uncanny [operational limitations and security concerns](../../WHY.md). Using STUNner, however, media servers can be deployed into ordinary Kubernetes pods and run over a private IP network, like any "normal" Kubernetes workload. @@ -23,82 +32,10 @@ The figure below shows LiveKit deployed into regular Kubernetes pods behind STUN In this tutorial we deploy a video room example using [LiveKit's React SDK](https://github.com/livekit/livekit-react/tree/master/example), the [LiveKit server](https://github.com/livekit/livekit) for media exchange, a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the LiveKit server pool to clients. -## Installation - -Let's start with a disclaimer. The LiveKit client example browser must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. Unfortunately, self-signed TLS certs [will not work](https://docs.livekit.io/deploy/#domain,-ssl-certificates,-and-load-balancer), so we have to come up with a way to provide our clients with a valid TLS cert. This will have the unfortunate consequence that the majority of the below installation guide will be about securing client connections to LiveKit over TLS; as it turns out, once HTTPS is correctly working integrating LiveKit with STUNner is very simple. - -In the below example, STUNner will be installed into the identically named namespace, while LiveKit and the Ingress gateway will live in the default namespace. - -### TLS certificates - -As mentioned above, the LiveKit server will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't then [nip.io](https://nip.io) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for LiveKit. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. - -Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of `nip.io`. - -### Ingress - -The first step of obtaining a valid cert is to install a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. - -Install an ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but this is not required. - -```console -helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx -helm repo update -helm install ingress-nginx ingress-nginx/ingress-nginx -``` - -Wait until Kubernetes assigns an external IP to the Ingress. - -```console -until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -``` - -Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. - -```console -kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' -export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') -``` - -### Cert manager - -We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. - -Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: - -```console -helm repo add cert-manager https://charts.jetstack.io -helm repo update -helm install cert-manager jetstack/cert-manager --namespace cert-manager \ - --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set installCRDs=true --timeout 600s -``` - -At this point we have all the necessary boilerplate set up to automate TLS issuance for LiveKit. ### STUNner -Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy the [manifest](livekit-server.yaml) packaged with STUNner. - -Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): - -Legacy mode: - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy -helm install stunner stunner/stunner --create-namespace --namespace=stunner -``` - -Managed mode: - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system -``` +Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy (after some minor modifications) the [manifest](livekit-server.yaml) packaged with STUNner. Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the LiveKit server pods. @@ -231,3 +168,7 @@ livekit-cli create-token \ ``` Copy the access token into the token field and hit the Connect button. If everything is set up correctly, you should be able to connect to a room. If you repeat the procedure in a separate browser tab you can enjoy a nice video-conferencing session with yourself, with the twist that all media between the browser tabs is flowing through STUNner and the LiveKit-server deployed in you Kubernetes cluster. + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). \ No newline at end of file diff --git a/docs/examples/livekit/livekit-server.yaml b/docs/examples/livekit/livekit-server.yaml index 04f6ea02..2304b493 100644 --- a/docs/examples/livekit/livekit-server.yaml +++ b/docs/examples/livekit/livekit-server.yaml @@ -85,7 +85,7 @@ data: port_range_start: 50000 tcp_port: 7801 stun_servers: - - stunner_ip:3478 + - stunner_ip:3478 turn_servers: - credential: pass-1 host: stunner_ip @@ -213,10 +213,10 @@ metadata: name: tls-example-ingress namespace: default annotations: - kubernetes.io/ingress.class: "nginx" + # kubernetes.io/ingress.class: "nginx" cert-manager.io/cluster-issuer: "letsencrypt-prod" spec: - # ingressClassName: nginx + ingressClassName: nginx tls: - hosts: - client-ingressserviceip.nip.io diff --git a/docs/examples/mediasoup/README.md b/docs/examples/mediasoup/README.md index ec0ed408..f3e6bcf8 100644 --- a/docs/examples/mediasoup/README.md +++ b/docs/examples/mediasoup/README.md @@ -11,14 +11,25 @@ In this demo you will learn to: ## Prerequisites -The below installation instructions require an operational cluster running a supported version of Kubernetes (>1.22). Most hosted or private Kubernetes cluster services will work, but make sure that the cluster comes with a functional load-balancer integration (all major hosted Kubernetes services should support this). Otherwise, STUNner will not be able to allocate a public IP address for clients to reach your WebRTC infra. As a regrettable exception, Minikube is unfortunately not supported for this demo. The reason is that [Let's Encrypt certificate issuance is not available with nip.io when using local/private IPs](https://medium.com/@EmiiKhaos/there-is-no-possibility-that-you-can-get-lets-encrypt-certificate-with-nip-io-7483663e0c1b); later on, you will learn more about why this is crucial above. +To run this example, you need: +* a [Kubernetes cluster](../../INSTALL.md#prerequisites), +* a [deployed STUNner](../../INSTALL.md#installation-1) (presumably the latest stable version), +* an [Ingress controller](../TLS.md#ingress) to ingest traffic into the cluster, +* a [Cert-manager](../TLS.md#cert-manager) to automate TLS certificate management. -## Setup +> [!NOTE] +> +> If you have your own TLS certificate, put it in a `Secret` [resource](https://kubernetes.io/docs/concepts/configuration/secret/) and deploy it into the `default` namespace under the `mediasoup-demo-tls` name. + + +## Description The recommended way to install mediasoup ([link](https://mediasoup.discourse.group/t/server-in-kubernetes-with-turn/3434),[link](https://www.reddit.com/r/kubernetes/comments/sdkhwn/deploying_mediasoup_webrtc_sfu_in_kubernetes/)) into Kubernetes is deploying the media servers into the host-network namespace of the Kubernetes nodes (`hostNetwork: true`). This deployment model, however, comes with a set of uncanny [operational limitations and security concerns](../../WHY.md). Using STUNner, however, media servers can be deployed into ordinary Kubernetes pods and run over a private IP network, like any "normal" Kubernetes workload. The figure below shows mediasoup deployed into regular Kubernetes pods behind STUNner without the host-networking hack. Here, mediasoup is deployed behind STUNner in the [*media-plane deployment model*](../../DEPLOYMENT.md), so that STUNner acts as a "local" STUN/TURN server for mediasoup, saving the overhead of using public a 3rd party STUN/TURN server for NAT traversal. +![STUNner mediasoup integration deployment architecture](../../img/stunner_mediasoup.svg) + In this tutorial we deploy a video room example using [mediasoup's demo application](https://github.com/versatica/mediasoup-demo/) with slight modifications (more on these below), the [mediasoup server](https://github.com/versatica/mediasoup/) for media exchange, a Kubernetes Ingress gateway to secure signaling connections and handle TLS, and STUNner as a media gateway to expose the mediasoup server pool to clients. ### Modifications on the mediasoup demo @@ -41,82 +52,11 @@ with an additional environment variable we can load the pod's private IP into th - Added the parsing of url parameters to configure TURN server and a simple if/else in server/app/lib/RoomClient.js. The mediasoup clients will use the configured TURN server to gather the ICE candidates. Example: https://mediasoup-demo.example.com/?enableIceServer=yes&iceServerHost=100.100.100.100&iceServerPort=3478&iceServerProto=udp&iceServerUser=user-1&iceServerPass=pass-1 -## Installation - -Let's start with a disclaimer. The mediasoup demo example must work over a secure HTTPS connection, because [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#browser_compatibility) is available only in secure contexts. This implies that the client-server signaling connection must be secure too. According to the [documentation](https://github.com/versatica/mediasoup-demo/blob/a59c6ab8e50fb950c3df54f4b85167a4e3f8497a/README.md?plain=1#L96) mediasoup should work with self-signed certs, however this haven't been tested. In the following we will deploy mediasoup configured with a valid signed TLS certificate. This will have the unfortunate consequence that the majority of the below installation guide will be about securing client connections to mediasoup over TLS; as it turns out, once HTTPS is correctly working integrating mediasoup with STUNner is very simple. - -In the below example, STUNner will be installed into the identically named namespace, while mediasoup and the Ingress gateway will live in the default namespace. - -### TLS certificates - -As mentioned above, the mediasoup server will need a valid TLS cert, which means it must run behind an existing DNS domain name backed by a CA signed TLS certificate. This is simple if you have your own domain, but if you don't then [nip.io](https://nip.io) provides a dead simple wildcard DNS for any IP address. We will use this to "own a domain" and obtain a CA signed certificate for mediasoup. This will allow us to point the domain name `client-.nip.io` to an ingress HTTP gateway in our Kubernetes cluster, which will then use some automation (namely, cert-manager) to obtain a valid CA signed cert. - -Note that public wildcard DNS domains might run into [rate limiting](https://letsencrypt.org/docs/rate-limits/) issues. If this occurs you can try [alternative services](https://moss.sh/free-wildcard-dns-services/) instead of `nip.io`. - -### Ingress - -The first step of obtaining a valid cert is to install a Kubernetes Ingress: this will be used during the validation of our certificates and to terminate client TLS encrypted contexts. - -Install an ingress controller into your cluster. We used the official [nginx ingress](https://github.com/kubernetes/ingress-nginx), but this is not required. - -```console -helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx -helm repo update -helm install ingress-nginx ingress-nginx/ingress-nginx -``` - -Wait until Kubernetes assigns an external IP to the Ingress. - -```console -until [ -n "$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do sleep 1; done -``` - -Store the Ingress IP address Kubernetes assigned to our Ingress; this will be needed later when we configure the validation pipeline for our TLS certs. - -```console -kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' -export INGRESSIP=$(kubectl get service ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') -export INGRESSIP=$(echo $INGRESSIP | sed 's/\./-/g') -``` - -### Cert manager - -We use the official [cert-manager](https://cert-manager.io) to automate TLS certificate management. - -Add the Helm repository, which contains the cert-manager Helm chart, and install the charts: - -```console -helm repo add cert-manager https://charts.jetstack.io -helm repo update -helm install cert-manager jetstack/cert-manager --namespace cert-manager \ - --create-namespace --set global.leaderElection.namespace=cert-manager \ - --set installCRDs=true --timeout 600s -``` - -At this point we have all the necessary boilerplate set up to automate TLS issuance for mediasoup. - ### STUNner -Now comes the fun part. - -Install the STUNner gateway operator and STUNner via [Helm](https://github.com/l7mp/stunner-helm): - -Legacy mode: +Now comes the fun part. The simplest way to run this demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy (after some minor modifications) the [manifest](mediasoup-server.yaml) packaged with STUNner. -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system --set stunnerGatewayOperator.dataplane.mode=legacy -helm install stunner stunner/stunner-dev --create-namespace --namespace=stunner-system -``` - -Managed mode (recommended): - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator-dev --create-namespace --namespace=stunner-system -``` +To install the stable version of STUNner, please follow the instructions in [this section](../../INSTALL.md#installation-1). Configure STUNner to act as a STUN/TURN server to clients, and route all received media to the mediasoup server pods. diff --git a/docs/examples/neko/README.md b/docs/examples/neko/README.md index 2a641597..0777cc4f 100644 --- a/docs/examples/neko/README.md +++ b/docs/examples/neko/README.md @@ -15,20 +15,15 @@ In this demo you will learn the following steps to: ### Prerequisites -Consult the [STUNner installation and configuration guide](/docs/INSTALL.md) to set up STUNner. +To run this example, you need: +* a [Kubernetes cluster](../../INSTALL.md#prerequisites), +* a [deployed STUNner](../../INSTALL.md#installation-1) (presumably the latest stable version), +* optionally, an [Ingress controller](../TLS.md#ingress) to ingest traffic into the cluster. ### Quick installation The simplest way to deploy the demo is to clone the [STUNner git repository](https://github.com/l7mp/stunner) and deploy the [manifest](neko.yaml) packaged with STUNner. -Install the STUNner gateway operator and STUNner ([more info](https://github.com/l7mp/stunner-helm)): - -```console -helm repo add stunner https://l7mp.io/stunner -helm repo update -helm install stunner-gateway-operator stunner/stunner-gateway-operator --create-namespace --namespace=stunner-system -``` - Configure STUNner to act as a STUN server towards clients, and to let media reach the media server. ```console @@ -45,7 +40,7 @@ kubectl get service udp-gateway -n default -o jsonpath='{.status.loadBalancer.in STUNNERIP=$(kubectl get service udp-gateway -n default -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` -> **Note** +> [!NOTE] > This IP should be accessible from your browser. If that "public IP" is behind a NAT, you can overwrite it with the actual public IP that routes to the service by hand (e.g. `STUNNERIP=`). We need to give this public IP the Neko configuration in the `NEKO_ICESERVERS` environment variable, inside the `json` content (basically this will tell you browser to use STUNner as a STUN/TURN server). diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index b5c91f32..ffbe2092 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -158,7 +158,7 @@ We will need to learn the ClusterIP assigned by Kubernetes to the `iperf-server` export IPERF_ADDR=$(kubectl get svc iperf-server -o jsonpath="{.spec.clusterIP}") ``` -Next, set up `turncat` to listen on `UDP:127.0.0.1:5000` and tunnel connections from this listener via the STUNner STUN/TURN listener `udp-listener` to the iperf server. Luckily, `turncat` is clever enough to [parse the running STUNner configuration](../../cmd/turncat) from Kubernetes and set the STUN/TURN server public address/port and the authentication credentials accordingly. +Next, set up `turncat` to listen on `UDP:127.0.0.1:5000` and tunnel connections from this listener via the STUNner STUN/TURN listener `udp-listener` to the iperf server. Luckily, `turncat` is clever enough to [parse the running STUNner configuration](../../cmd/turncat.md) from Kubernetes and set the STUN/TURN server public address/port and the authentication credentials accordingly. ``` console bin/turncat --log=all:INFO udp://127.0.0.1:5000 k8s://stunner/udp-gateway:udp-listener \ @@ -215,3 +215,7 @@ Stop `turncat` and wipe all Kubernetes configuration. kubectl delete -f docs/examples/simple-tunnel/iperf-server.yaml kubectl delete -f docs/examples/simple-tunnel/iperf-stunner.yaml ``` + +# Help + +STUNner development is coordinated in Discord, feel free to [join](https://discord.gg/DyPgEsbwzc). diff --git a/docs/img/stunner_mediasoup.svg b/docs/img/stunner_mediasoup.svg new file mode 100644 index 00000000..5949b7bf --- /dev/null +++ b/docs/img/stunner_mediasoup.svg @@ -0,0 +1,4 @@ + + + +
Kubernetes  cluster
Kubernetes  cluster
HTTPS
HTTPS
Ingress
Ingress
UDP/RTP
UDP/RTP
STUN/TURN
STUN/TURN \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index a9ef0a6c..c3c40de8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,16 +19,18 @@ nav: - 'Scaling': 'SCALING.md' - 'Monitoring': 'MONITORING.md' - 'Examples': + - 'Configuring TLS for examples': 'examples/TLS.md' - 'Simple tunnel': 'examples/simple-tunnel/README.md' - 'Direct one to one call': 'examples/direct-one2one-call/README.md' - - 'Kurento one to one call': 'examples/kurento-one2one-call/README.md' - - 'Kurento Magic Mirror': 'examples/kurento-magic-mirror/README.md' - 'LiveKit': 'examples/livekit/README.md' - 'Janus': 'examples/janus/README.md' + - 'Elixir WebRTC': 'examples/elixir-webrtc/README.md' - 'Jitsi': 'examples/jitsi/README.md' - 'mediasoup': 'examples/mediasoup/README.md' - 'CloudRetro': 'examples/cloudretro/README.md' - 'N.eko': 'examples/neko/README.md' + - 'Kurento one to one call': 'examples/kurento-one2one-call/README.md' + - 'Kurento Magic Mirror': 'examples/kurento-magic-mirror/README.md' - 'Benchmark': 'examples/benchmark/README.md' - 'CMD': - 'stunnerctl': 'cmd/stunnerctl.md' From 2cd71f88d29c910f99f0c8630b335733a6aa2fde Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Mon, 28 Oct 2024 15:01:03 +0100 Subject: [PATCH 217/243] docs(examples/neko): Remove extra iceTransportPolicy --- docs/examples/neko/neko.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/examples/neko/neko.yaml b/docs/examples/neko/neko.yaml index 7292dbcd..b99a0a68 100644 --- a/docs/examples/neko/neko.yaml +++ b/docs/examples/neko/neko.yaml @@ -35,7 +35,7 @@ spec: fieldRef: fieldPath: status.podIP - name: NEKO_ICESERVERS - value: '[{"urls": ["turn:1.1.1.1:3478?transport=udp"], "username": "user-1", "credential": "pass-1","iceTransportPolicy": "all"}]' + value: '[{"urls": ["turn:1.1.1.1:3478?transport=udp"], "username": "user-1", "credential": "pass-1"}]' ports: - name: http containerPort: 8080 From 42158065815c0c5861232c198fbc6f0939070470 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 31 Oct 2024 12:41:49 +0100 Subject: [PATCH 218/243] doc: Massive doc reorg --- README.md | 460 ++---------------- .../stunner-standalone-tls.yaml.template | 220 --------- docs/README.md | 1 + docs/examples/simple-tunnel/README.md | 6 +- docs/examples/udp-echo/README.md | 342 +++++++++++++ .../examples/udp-echo}/udp-greeter.yaml | 0 6 files changed, 398 insertions(+), 631 deletions(-) delete mode 100644 deploy/manifests/stunner-standalone-tls.yaml.template create mode 100644 docs/examples/udp-echo/README.md rename {deploy/manifests => docs/examples/udp-echo}/udp-greeter.yaml (100%) diff --git a/README.md b/README.md index 6638aedf..6346c382 100644 --- a/README.md +++ b/README.md @@ -62,9 +62,8 @@ style via Kubernetes manifests. 1. [Description](#description) 1. [Features](#features) 1. [Getting started](#getting-started) -1. [Tutorials](#tutorials) +1. [Usage](#usage) 1. [Documentation](#documentation) -1. [Caveats](#caveats) 1. [Milestones](#milestones) ## Description @@ -136,11 +135,21 @@ way. can connect to it directly without the use of *any* external STUN/TURN service whatsoever, apart from STUNner itself. -* **Easily scale your WebRTC infrastructure.** Tired of manually provisioning your WebRTC media - servers? STUNner lets you deploy the entire WebRTC infrastructure into ordinary Kubernetes pods, - thus [scaling the media plane](docs/SCALING.md) is as easy as issuing a `kubectl scale` - command. Or you can use the built in Kubernetes horizontal autoscaler to *automatically* resize - your workload based on demand. +* **Scale your WebRTC infrastructure.** Tired of manually provisioning your WebRTC media servers? + STUNner lets you deploy the entire WebRTC infrastructure into ordinary Kubernetes pods, thus + [scaling the media plane](docs/SCALING.md) is as easy as issuing a `kubectl scale` command. Or + you can use the built in Kubernetes horizontal autoscaler to *automatically* resize your workload + based on demand. + +* **Minimal client-side configuration.** STUNner comes with a built-in [authentication + service](https://github.com/l7mp/stunner-auth-service) that can be used to generate time-windowed + per-user TURN credentials through a [standards + compliant](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00) HTTP [REST + API](/docs/AUTH.md). Just set the generated [ICE + configuration](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#configuration) + in the [`PeerConnection` JavaScript + API](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection) and + your clients will readily start using your Kubernetes-based WebRTC service via STUNner. * **Secure perimeter defense.** No need to open thousands of UDP/TCP ports on your media server for potentially malicious access; with STUNner *all* media is received through a single ingress port @@ -151,38 +160,29 @@ way. lines of fully open-source code. The server is extremely lightweight: the typical STUNner container image size is only 15 Mbytes. -The main uses of STUNner are running a scalable STUN service in Kubernetes to make it possible for -clients to create peer-to-peer WebRTC media connections between themselves (the so called -[*headless deployment -model*](https://github.com/l7mp/stunner/blob/main/docs/DEPLOYMENT.md#headless-deployment-model)), -or as a scalable TURN gateway service for load-balancing clients' media connections across a pool -of WebRTC media servers hosted in Kubernetes pods (called the [*media-plane deployment -model*](https://github.com/l7mp/stunner/blob/main/docs/DEPLOYMENT.md#media-plane-deployment-model)). While -both use cases are fully supported, below we concentrate on the latter use case: deploying STUNner -as a media gateway (see [our blog -post](https://medium.com/l7mp-technologies/deploying-a-scalable-stun-service-in-kubernetes-c7b9726fa41d) -to learn how to deploy a STUN service with STUNner). +The main uses of STUNner are [hosting a scalable STUN server +pool](https://medium.com/l7mp-technologies/deploying-a-scalable-stun-service-in-kubernetes-c7b9726fa41d) +in Kubernetes, as a public Kubernetes-based [TURN +service](https://github.com/l7mp/stunner/blob/main/docs/DEPLOYMENT.md#headless-deployment-model), +or as a fully-fledged [gateway +service](https://github.com/l7mp/stunner/blob/main/docs/DEPLOYMENT.md#media-plane-deployment-model) +for ingesting and load-balancing clients' media connections across a pool of WebRTC media servers +hosted in ordinary Kubernetes pods. ## Getting Started -STUNner comes with a [Helm](https://helm.sh) chart to fire up a fully functional STUNner-based -WebRTC media gateway in minutes. Note that the default installation does not contain an application -server and a media server: STUNner is not a WebRTC service, it is merely an *enabler* for you to -deploy your *own* WebRTC infrastructure into Kubernetes. Once installed, STUNner makes sure that -your media servers are readily reachable to WebRTC clients, despite running with a private IP -address inside a Kubernetes pod. See the [tutorials](#tutorials) for some ideas on how to deploy an -actual WebRTC application behind STUNner. - With a minimal understanding of WebRTC and Kubernetes, deploying STUNner should take less than 5 -minutes. +minutes, in five simple steps. -* [Customize STUNner and deploy it](#installation) into your Kubernetes cluster. -* Optionally [deploy a WebRTC media server](docs/examples/kurento-one2one-call). -* [Set STUNner as the ICE server](#configuring-webrtc-clients) in your WebRTC clients. +* [Customize STUNner and deploy it](/docs/INSTALL.md) into your Kubernetes cluster. +* Optionally [deploy a WebRTC media server](/docs/README.md#media-plane-deployment-model). +* [Set STUNner as the ICE server](/docs/AUTH.md) in your WebRTC clients. * ... * Profit!! -### Installation +Note that the default installation does not contain an application server and a media server: +STUNner is not a WebRTC service in itself, it is merely an *enabler* for you to deploy your *own* +WebRTC infrastructure into Kubernetes. The simplest way to deploy STUNner is through [Helm](https://helm.sh). STUNner configuration parameters are available for customization as [Helm @@ -197,364 +197,25 @@ helm install stunner-gateway-operator stunner/stunner-gateway-operator --create- Find out more about the charts in the [STUNner-helm repository](https://github.com/l7mp/stunner-helm). -### Configuration - -The standard way to interact with STUNner is via the standard Kubernetes [Gateway -API](https://gateway-api.sigs.k8s.io). This is much akin to the way you configure *all* Kubernetes -workloads: specify your intents in YAML files and issue a `kubectl apply`, and the [STUNner gateway -operator](https://github.com/l7mp/stunner-gateway-operator) will automatically create the STUNner -dataplane (that is, the `stunnerd` pods that implement the STUN/TURN service) and downloads the new -configuration to the dataplane pods. - -It is generally a good idea to maintain STUNner configuration into a separate Kubernetes -namespace. Below we will use the `stunner` namespace; create it with `kubectl create namespace -stunner` if it does not exist. - -1. Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway - API. This amounts to creating a - [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.GatewayClass), - which serves as the [root level configuration](/docs/GATEWAY.md#gatewayclass) for your STUNner - deployment. - - Each GatewayClass must specify a controller that will manage the Gateway objects created under - the class hierarchy. This must be set to `stunner.l7mp.io/gateway-operator` in order for STUNner - to pick up the GatewayClass. In addition, a GatewayClass can refer to further - implementation-specific configuration via a reference called `parametersRef`; in our case, this - will be a GatewayConfig object to be specified next. - - ``` console - kubectl apply -f - <`) with the correct configuration -from the running STUNner config; don't forget that `stunnerctl` is always there for you to help. - -```js -var ICE_config = { - iceServers: [ - { - url: 'turn::?transport=udp', - username: , - credential: , - }, - ], -}; -var pc = new RTCPeerConnection(ICE_config); -``` - -Note that STUNner comes with a built-in [authentication -service](https://github.com/l7mp/stunner-auth-service) that can be used to generate a complete ICE -configuration for reaching STUNner through a [standards -compliant](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00) HTTP [REST -API](docs/AUTH.md). - -## Tutorials - -The below series of tutorials demonstrates how to leverage STUNner to deploy different WebRTC -applications into Kubernetes. - -### Basics - -* [Opening a UDP tunnel via STUNner](/docs/examples/simple-tunnel/README.md): This introductory tutorial - shows how to tunnel an external connection via STUNner to a UDP service deployed into - Kubernetes. The demo can be used to quickly check and benchmark a STUNner installation. - -### Headless deployment mode - +## Usage + +STUNner comes with a wide selection of tutorials and demos that teach you how to deploy all kinds +of WebRTC services into Kubernetes. The first couple of tutorials present the basic concepts, +especially the use of the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) to configure +STUNner and the [`turncat`](/docs/cmd/turncat.md) utility to test it. Each subsequent demo +showcases a specific WebRTC application, from desktop streaming and video-conferencing to +cloud-gaming, and goes from a clean Kubernetes cluster to a working and usable publicly available +WebRTC service in 5-10 minutes using a purely declarative configuration. + +* [Deploying a UDP echo service behind STUNner](/docs/examples/udp-echo/README.md): This + introductory tutorial shows how to deploy a simple UDP echo service into Kubernetes and expose it + via STUNner. If you read just one STUNner tutorial, this should be it. +* [Opening a UDP tunnel via STUNner](/docs/examples/simple-tunnel/README.md): This tutorial shows + how to tunnel an external UDP client via STUNner to a standard iperf server deployed into + Kubernetes. The demo can be used to benchmark your STUNner installation. * [Direct one to one video call via STUNner](/docs/examples/direct-one2one-call/README.md): This tutorial showcases STUNner acting as a TURN server for two WebRTC clients to establish connections between themselves, without the mediation of a media server. - -### Media-plane deployment model - * [Video-conferencing with LiveKit](/docs/examples/livekit/README.md): This tutorial helps you deploy the [LiveKit](https://livekit.io) WebRTC media server behind STUNner. The docs also show how to obtain a valid TLS certificate to secure your signaling connections, courtesy of the @@ -581,8 +242,8 @@ applications into Kubernetes. show how to obtain a valid TLS certificate to secure your signaling connections, courtesy of the [cert-manager](https://cert-manager.io) project, [nip.io](https://nip.io) and [Let's Encrypt](https://letsencrypt.org). -* [Cloud-gaming with Cloudretro](/docs/examples/cloudretro/README.md): This tutorial lets you play Super - Mario or Street Fighter in your browser, courtesy of the amazing +* [Cloud-gaming with Cloudretro](/docs/examples/cloudretro/README.md): This tutorial lets you play + Super Mario or Street Fighter in your browser, courtesy of the amazing [CloudRetro](https://cloudretro.io) project and, of course, STUNner. The demo also presents a simple multi-cluster setup, where clients can reach the game-servers in their geographical locality to minimize latency. @@ -607,27 +268,10 @@ applications into Kubernetes. The documentation of the stable release can be found [here](https://docs.l7mp.io/en/stable). The documentation for the latest development release can be found [here](/docs/README.md). -## Caveats - -STUNner is a work-in-progress. Some features are missing, others may not work as expected. The -notable limitations at this point are as follows. - -* STUNner targets only a *partial implementation of the Kubernetes Gateway API.* In particular, - only GatewayClass, Gateway and UDPRoute resources are supported. This is intended: STUNner - deliberately ignores some complexity in the [Gateway API](https://gateway-api.sigs.k8s.io) and - deviates from the prescribed behavior in some cases, all in the name of simplifying the - configuration process. The [STUNner Kubernetes gateway - operator](https://github.com/l7mp/stunner-gateway-operator) docs contain a [detailed - list](https://github.com/l7mp/stunner-gateway-operator#caveats) on the differences. -* STUNner *lacks officially support for IPv6*. Clients and peers reachable only on IPv6 may or may - not be able connect to STUNner depending on the version you're using. Please file a bug if you - absolutely need IPv6 support. - ## Milestones * v0.9: Demo release: STUNner basic UDP/TURN connectivity + helm chart + tutorials. -* v0.10: Dataplane: Long-term STUN/TURN credentials and [STUN/TURN over - TCP/TLS/DTLS](https://www.rfc-editor.org/rfc/rfc6062.txt) in standalone mode. +* v0.10: Dataplane: Long-term STUN/TURN credentials and [STUN/TURN over TCP/TLS/DTLS](https://www.rfc-editor.org/rfc/rfc6062.txt) in standalone mode. * v0.11: Control plane: Kubernetes gateway operator and dataplane reconciliation. * v0.12: Security: Expose TLS/DTLS settings via the Gateway API. * v0.13: Observability: Prometheus + Grafana dashboard. @@ -636,7 +280,9 @@ notable limitations at this point are as follows. * v0.17: First release candidate: All Gateway and STUNner APIs move to v1. * v0.18: Stabilization: Second release candidate. * v0.19: The missing pieces: Third release candidate. -* v1.0: GA +* v0.20: Final stabilization: Fourth stable release candidate +* v0.21: Towards v1: Fifth stable release candidate +* v1.0: STUNner goes GA! ## Help diff --git a/deploy/manifests/stunner-standalone-tls.yaml.template b/deploy/manifests/stunner-standalone-tls.yaml.template deleted file mode 100644 index 22741af2..00000000 --- a/deploy/manifests/stunner-standalone-tls.yaml.template +++ /dev/null @@ -1,220 +0,0 @@ -# Stunner: An ingress gateway for WebRTC: TLS/DTLS in standalone mode ---- -# STUN/TURN server config: Make sure to customize the below configurations, see the README.pm for -# more details -apiVersion: v1 -kind: ConfigMap -metadata: - name: stunner-config -data: - # * `STUNNER_PUBLIC_ADDR` (no default): The public IP address clients can use to reach - # STUNner. By default, the public IP address will be dynamically assigned by the Kubernetes - # LoadBalancer service. The Helm installation script takes care of updating the configuration - # with the correct value. However, if installing from the static manifests then the external IP - # must be set manually. - STUNNER_PUBLIC_ADDR: "A.B.C.D" - - # * `STUNNER_PUBLIC_PORT` (default: 3478): The public port used by clients to reach STUNner. It - # is important that applications use the public port as found in the configuration, since the - # Helm installation scripts may overwrite this configuration. This occurs when the installation - # falls back to a NodePort service (i.e., when STUNner fails to obtain an external IP from the - # load-balancer). - STUNNER_PUBLIC_PORT: "443" - - # * `STUNNER_PORT` (default: 3478): The internal port used by STUNner for communication inside - # the cluster. It is safe to set this to the public port. - STUNNER_PORT: "443" - - # * `STUNNER_REALM` (default `stunner.l7mp.io`): the REALM used to guide the user agent in - # authenticating with STUNner. - STUNNER_REALM: "stunner.l7mp.io" - - # * `STUNNER_AUTH_TYPE` (default: `plaintext`): the STUN/TURN authentication mode, either - # "plaintext" over the username/password pair $STUNNER_USERNAME/$STUNNER_PASSWORD, or - # "longterm", using $STUNNER_SECRET. Make sure to customize! - STUNNER_AUTH_TYPE: "plaintext" - - # * `STUNNER_USERNAME` (default: `user`): the USERNAME attribute clients can use the authenticate - # with STUNner over plain-text authentication. Make sure to customize! - STUNNER_USERNAME: "user1" - - # * `STUNNER_PASSWORD` (default: `pass`): the password clients can use to authenticate with - # STUNner over plain-text authentication. Make sure to customize! - STUNNER_PASSWORD: "passwd1" - - # * `STUNNER_SHARED_SECRET`: the shared secret used for longterm authentication. - STUNNER_SHARED_SECRET: "my-shared-secret" - - # * `STUNNER_DURATION` (default: `86400`, i.e., one day): the lifetime of STUNner credentials - # * over longterm authentication. - STUNNER_DURATION: "86400" - - # * `STUNNER_LOGLEVEL` (default: `all:WARN`): the default log level used by the STUNner daemons. - STUNNER_LOGLEVEL: "all:INFO" - - # * `STUNNER_MIN_PORT` (default: 10000): smallest relay transport port assigned by STUNner. - STUNNER_MIN_PORT: "10000" - - # * `STUNNER_MAX_PORT` (default: 20000): highest relay transport port assigned by STUNner. - STUNNER_MAX_PORT: "20000" - - STUNNER_TLS_KEY: | - XXXXXXX - STUNNER_TLS_CERT: | - YYYYYYY - ---- -## custom static stunnerd conf for TLS/DTLS -apiVersion: v1 -kind: ConfigMap -metadata: - name: stunnerd-conf -data: - "stunnerd.conf" : | - version: v1alpha1 - admin: - name: stunnerd - loglevel: $STUNNER_LOGLEVEL - metrics_endpoint: "http://0.0.0.0:8080/metrics" - auth: - type: $STUNNER_AUTH_TYPE - realm: $STUNNER_REALM - credentials: - username: $STUNNER_USERNAME - password: $STUNNER_PASSWORD - secret: $STUNNER_SHARED_SECRET - clusters: - - name: media-plane - type: STRICT_DNS - endpoints: - - media-plane.default.svc.cluster.local - listeners: - - name: stunner-dtls - public_address: "$STUNNER_PUBLIC_ADDR" - public_port: $STUNNER_PUBLIC_PORT - address: "$STUNNER_ADDR" - port: $STUNNER_PORT - protocol: dtls - min_port: $STUNNER_MIN_PORT - max_port: $STUNNER_MAX_PORT - key: $STUNNER_TLS_KEY - cert: $STUNNER_TLS_CERT - routes: - - media-plane - - name: stunner-tls - public_address: "$STUNNER_PUBLIC_ADDR" - public_port: $STUNNER_PUBLIC_PORT - address: "$STUNNER_ADDR" - port: $STUNNER_PORT - protocol: tls - min_port: $STUNNER_MIN_PORT - max_port: $STUNNER_MAX_PORT - key: $STUNNER_TLS_KEY - cert: $STUNNER_TLS_CERT - routes: - - media-plane - ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: stunner -spec: - selector: - matchLabels: - app: stunner - replicas: 1 - template: - metadata: - labels: - app: stunner - spec: - containers: - - name: stunnerd - image: l7mp/stunnerd:latest - imagePullPolicy: Always - command: ["stunnerd"] - args: ["-c", "/etc/stunnerd/stunnerd.conf"] - envFrom: - - configMapRef: - name: stunner-config - env: - - name: STUNNER_ADDR # we use the POD IP - valueFrom: - fieldRef: - fieldPath: status.podIP - volumeMounts: - - name: stunnerd-config-volume - mountPath: /etc/stunnerd - readOnly: true - - name: stunnerd-cert-volume - mountPath: /etc/ssl/certs - readOnly: true - # Uncomment this if you want to deploy a sidecar container with stunner to sniff traffic - # - name: net-debug - # image: l7mp/net-debug:latest - # command: ["/bin/sh"] - # args: ["-c", "while true; do echo hello; sleep 10;done"] - volumes: - - name: stunnerd-config-volume - configMap: - name: stunnerd-conf - optional: true - - name: stunnerd-cert-volume - secret: - secretName: stunner-tls - optional: true - ---- -apiVersion: v1 -kind: Service -metadata: - name: stunner-dtls - labels: - app: stunner -spec: - ports: - - port: 443 - targetPort: 443 - protocol: UDP - name: stunner-dtls - type: LoadBalancer - selector: - app: stunner - ---- -apiVersion: v1 -kind: Service -metadata: - name: stunner-tls - labels: - app: stunner -spec: - ports: - - port: 443 - targetPort: 443 - protocol: TCP - name: stunner-tls - type: LoadBalancer - selector: - app: stunner - ---- -# lock down access from the TURN server to anywhere! -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: stunner-network-policy -spec: - podSelector: - matchLabels: - app: stunner - policyTypes: - - Egress - egress: - - to: - - podSelector: - matchLabels: - app: media-plane - ports: - - protocol: UDP diff --git a/docs/README.md b/docs/README.md index 14a15cef..ab28c7ce 100644 --- a/docs/README.md +++ b/docs/README.md @@ -28,6 +28,7 @@ ### Basics +* [Deploying a UDP echo service behind](examples/udp-echo) * [Opening a UDP tunnel via STUNner](examples/simple-tunnel) ### Headless deployment model diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index ffbe2092..dbd10150 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -1,8 +1,6 @@ -# STUNner Tutorial +# STUNner Tutorial: STUNner benchmark with iperf -# Open a tunnel via STUNner - -This tutorial shows how to tunnel an external connection via STUNner to a UDP service deployed into Kubernetes. The tutorial can also be used to quickly check a STUNner installation. +This tutorial shows how to make a simple benchmark via STUNner to evaluate your cloud provider's UDP and TCP stack. In this tutorial you will learn how to: * configure a UDP service in Kubernetes, diff --git a/docs/examples/udp-echo/README.md b/docs/examples/udp-echo/README.md new file mode 100644 index 00000000..692ef50c --- /dev/null +++ b/docs/examples/udp-echo/README.md @@ -0,0 +1,342 @@ +# STUNner Tutorial: Deploy a UDP echo service behind STUNner + +This tutorial shows how to tunnel an external connection via STUNner to a UDP service deployed into Kubernetes. The tutorial can also be used as an introduction to the main concepts in STUNner and to quickly check a STUNner installation. + +In this tutorial you will learn how to: +* configure a UDP service in Kubernetes, +* configure STUNner to expose the service to clients, +* use [`turncat`](../../cmd/turncat.md) to connect to the UDP service via STUNner, +* benchmark your cloud-setup with [`iperfv2`](https://iperf.fr). + +## Prerequisites + +The tutorial assumes a fresh STUNner installation; see the [STUNner installation and configuration +guide](../../INSTALL.md). + +## Configuration + +The standard way to interact with STUNner is via the standard Kubernetes [Gateway +API](https://gateway-api.sigs.k8s.io). This is much akin to the way you configure *all* Kubernetes +workloads: specify your intents in YAML files and issue a `kubectl apply`, and the [STUNner gateway +operator](https://github.com/l7mp/stunner-gateway-operator) will automatically create the STUNner +dataplane (that is, the `stunnerd` pods that implement the STUN/TURN service) and downloads the new +configuration to the dataplane pods. + +It is generally a good idea to maintain STUNner configuration into a separate Kubernetes +namespace. Below we will use the `stunner` namespace; create it with `kubectl create namespace +stunner` if it does not exist. + +1. Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway + API. This amounts to creating a + [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.GatewayClass), + which serves as the [root level configuration](/docs/GATEWAY.md#gatewayclass) for your STUNner + deployment. + + Each GatewayClass must specify a controller that will manage the Gateway objects created under + the class hierarchy. This must be set to `stunner.l7mp.io/gateway-operator` in order for STUNner + to pick up the GatewayClass. In addition, a GatewayClass can refer to further + implementation-specific configuration via a reference called `parametersRef`; in our case, this + will be a GatewayConfig object to be specified next. + + ``` console + kubectl apply -f - < Date: Thu, 31 Oct 2024 12:55:47 +0100 Subject: [PATCH 219/243] chore: Remove doc disclaimer before release --- README.md | 2 -- docs/README.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/README.md b/README.md index 6346c382..6e581492 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,6 @@

-*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index ab28c7ce..99d235ef 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,6 @@ STUNner

-*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* - ## Overview * [Why STUNner](WHY.md) From bb5043558bc0d085fc71d8c3599f7a92c750354c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tam=C3=A1s=20L=C3=A9vai?= Date: Thu, 31 Oct 2024 13:16:28 +0100 Subject: [PATCH 220/243] chore(rtd): Add udp-echo example --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index c3c40de8..580c3175 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,7 @@ nav: - 'Monitoring': 'MONITORING.md' - 'Examples': - 'Configuring TLS for examples': 'examples/TLS.md' + - 'UDP echo': 'examples/udp-echo/README.md' - 'Simple tunnel': 'examples/simple-tunnel/README.md' - 'Direct one to one call': 'examples/direct-one2one-call/README.md' - 'LiveKit': 'examples/livekit/README.md' From 4061e988f6be1650c0eba247b1153940d47cabf3 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 31 Oct 2024 13:21:56 +0100 Subject: [PATCH 221/243] chore: Restore doc disclaimers --- README.md | 2 ++ docs/README.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/README.md b/README.md index 6e581492..6346c382 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + # STUNner: A Kubernetes media gateway for WebRTC Ever wondered how to [deploy your WebRTC infrastructure into the diff --git a/docs/README.md b/docs/README.md index 99d235ef..ab28c7ce 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,8 @@ STUNner

+*Note: This page documents the latest development version of STUNner. See the documentation for the stable version [here](https://docs.l7mp.io/en/stable).* + ## Overview * [Why STUNner](WHY.md) From 9d8d0071a1e4911f32875af67f2a95c4ccb8ba09 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 31 Oct 2024 14:31:40 +0100 Subject: [PATCH 222/243] doc: Fix typo --- docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index ab28c7ce..cb81f66f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -28,7 +28,7 @@ ### Basics -* [Deploying a UDP echo service behind](examples/udp-echo) +* [Deploying a UDP echo service behind STUNner](examples/udp-echo) * [Opening a UDP tunnel via STUNner](examples/simple-tunnel) ### Headless deployment model From 609ca1027e8d426874cfbaa3b46a35066075627f Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 31 Oct 2024 20:45:49 +0100 Subject: [PATCH 223/243] chore(CI/CD): Bump go version to v1.23 --- .github/workflows/e2e-test-dev.yaml | 2 +- .github/workflows/e2e-test-stable.yml | 2 +- .github/workflows/lint.yml | 2 +- .github/workflows/publish--add-binaries.yml | 2 +- .github/workflows/test.yml | 4 ++-- Dockerfile | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/e2e-test-dev.yaml b/.github/workflows/e2e-test-dev.yaml index 109554a7..cc14c0ad 100644 --- a/.github/workflows/e2e-test-dev.yaml +++ b/.github/workflows/e2e-test-dev.yaml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version: '1.23' - name: Download modules run: go mod download diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index 0f3b280d..bae52d0a 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -14,7 +14,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version: '1.23' - name: Download modules run: go mod download diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 4bf38dc7..2341c8ef 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -20,7 +20,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version: '1.23' - name: Checkout code uses: actions/checkout@v4 - name: Run linters diff --git a/.github/workflows/publish--add-binaries.yml b/.github/workflows/publish--add-binaries.yml index 14c71c09..733cdb97 100644 --- a/.github/workflows/publish--add-binaries.yml +++ b/.github/workflows/publish--add-binaries.yml @@ -42,7 +42,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version: '1.23' - name: Build binaries run: | diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 95a3174b..156a8d7f 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -23,7 +23,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version: '1.23' - name: Download modules run: go mod download - name: Go install @@ -38,7 +38,7 @@ jobs: if: success() uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version: '1.23' - name: Checkout code uses: actions/checkout@v4 - name: Calc coverage diff --git a/Dockerfile b/Dockerfile index c2b2d18c..63d52e97 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ ########### # BUILD -FROM golang:1.22-alpine as builder +FROM golang:1.23-alpine as builder WORKDIR /app From 5c8acf9f579298d25927b43a3de3fde569bc9b6f Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 31 Oct 2024 20:45:56 +0100 Subject: [PATCH 224/243] chore(rtd): Update deps --- docs/requirements.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/requirements.txt b/docs/requirements.txt index 32921e25..e6fff250 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -12,7 +12,7 @@ colorama==0.4.6 # via griffe ghp-import==2.1.0 # via mkdocs -griffe==1.3.1 +griffe==1.5.1 # via mkdocstrings-python jinja2==3.1.4 # via @@ -30,7 +30,7 @@ markdown-gfm-admonition==0.1.1 # via -r docs/requirements.in markdown-include==0.8.1 # via -r docs/requirements.in -markupsafe==2.1.5 +markupsafe==3.0.2 # via # jinja2 # mkdocs @@ -51,11 +51,11 @@ mkdocs-autorefs==1.2.0 # mkdocstrings-python mkdocs-get-deps==0.2.0 # via mkdocs -mkdocstrings[python]==0.26.1 +mkdocstrings[python]==0.26.2 # via # -r docs/requirements.in # mkdocstrings-python -mkdocstrings-python==1.11.1 +mkdocstrings-python==1.12.2 # via mkdocstrings packaging==24.1 # via mkdocs @@ -65,7 +65,7 @@ platformdirs==4.3.6 # via # mkdocs-get-deps # mkdocstrings -pymdown-extensions==10.11.1 +pymdown-extensions==10.11.2 # via mkdocstrings python-dateutil==2.9.0.post0 # via ghp-import From b843a01f7c9c64ca0f3c6bedc880444485160314 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Thu, 31 Oct 2024 21:06:47 +0100 Subject: [PATCH 225/243] chore(e2e-test-stable): Use latest turncat binary --- .github/workflows/e2e-test-stable.yml | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/.github/workflows/e2e-test-stable.yml b/.github/workflows/e2e-test-stable.yml index bae52d0a..e28b6ad5 100644 --- a/.github/workflows/e2e-test-stable.yml +++ b/.github/workflows/e2e-test-stable.yml @@ -11,17 +11,11 @@ jobs: steps: - uses: actions/checkout@v4 - - name: Set up Go - uses: actions/setup-go@v5 - with: - go-version: '1.23' - - - name: Download modules - run: go mod download - - name: Go install - run: go install - - name: Build turncat - run: CGO_ENABLED=0 go build -ldflags="-w -s" -o turncat cmd/turncat/main.go + - name: Install turncat + run: | + URL=`wget -q -O - https://api.github.com/repos/l7mp/stunner/releases/latest | jq -r '.assets[] | select(.name | contains ("turncat")) | select(.name | contains ("linux")) | select(.name | contains ("amd64")) | .browser_download_url'` + wget $URL -O turncat + chmod a+x turncat - name: Start minikube uses: medyagh/setup-minikube@master From c7a5112e82b242b4b51cecbdeae35136bcff0058 Mon Sep 17 00:00:00 2001 From: Basavaraju B K Date: Fri, 8 Nov 2024 15:32:19 +0530 Subject: [PATCH 226/243] chore(examples/livekit): Upgrade LiveKit Server to v1.8.0 (#175) --- docs/examples/livekit/livekit-server.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/examples/livekit/livekit-server.yaml b/docs/examples/livekit/livekit-server.yaml index 2304b493..08cebbba 100644 --- a/docs/examples/livekit/livekit-server.yaml +++ b/docs/examples/livekit/livekit-server.yaml @@ -104,7 +104,7 @@ metadata: labels: app.kubernetes.io/name: livekit-server app.kubernetes.io/instance: livekit - app.kubernetes.io/version: "v1.4.2" + app.kubernetes.io/version: "v1.8.0" spec: type: LoadBalancer ports: @@ -127,7 +127,7 @@ metadata: labels: app.kubernetes.io/name: livekit-server app.kubernetes.io/instance: livekit - app.kubernetes.io/version: "v1.4.2" + app.kubernetes.io/version: "v1.8.0" spec: replicas: 1 selector: @@ -147,7 +147,7 @@ spec: terminationGracePeriodSeconds: 18000 # 5 hours containers: - name: livekit-server - image: "livekit/livekit-server:v1.4.2" + image: "livekit/livekit-server:v1.8.0" imagePullPolicy: IfNotPresent args: ["--disable-strict-config"] env: @@ -258,4 +258,4 @@ spec: - http01: ingress: class: nginx ---- \ No newline at end of file +--- From a9a8164d0b8ac36cb81a4cd3d79e20ca456edc44 Mon Sep 17 00:00:00 2001 From: Mathias Durat <118441536+duratm@users.noreply.github.com> Date: Tue, 12 Nov 2024 10:47:32 +0100 Subject: [PATCH 227/243] docs(examples/elixir-webrtc): fix iceTransportPolicy syntax (#178) --- docs/examples/elixir-webrtc/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/examples/elixir-webrtc/README.md b/docs/examples/elixir-webrtc/README.md index 38c22d9f..3088a2a0 100644 --- a/docs/examples/elixir-webrtc/README.md +++ b/docs/examples/elixir-webrtc/README.md @@ -96,7 +96,7 @@ You have to modify the ICE config to use STUNner with the given credentials in t # change to: - const pcConfig = { iceServers: [{ urls: 'turn::3478?transport=udp', username: 'user-1', credential: 'pass-1',iceTransportPolicy: 'relay'}] }; + const pcConfig = { iceServers: [{ urls: 'turn::3478?transport=udp', username: 'user-1', credential: 'pass-1'}], iceTransportPolicy: 'relay' }; ``` `lib/nexus/peer.ex`: @@ -106,7 +106,8 @@ You have to modify the ICE config to use STUNner with the given credentials in t # change to: - ice_servers: [%{urls: "turn::3478?transport=udp", "username": "user-1", "credential": "pass-1", "iceTransportPolicy": "relay"}], + ice_servers: [%{urls: "turn::3478?transport=udp", "username": "user-1", "credential": "pass-1"}], + ice_transport_policy: :relay, ``` Now rebuild the Docker image, and push it into your image repository: From 1d3f5cfaf93dd4f2466df3f0130aa5cc8c33dd44 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Tue, 12 Nov 2024 10:56:50 +0100 Subject: [PATCH 228/243] docs(INSTALL): Fix gateway API CRD install URLs Fixes #176 --- docs/INSTALL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 22771a9a..c91b837b 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -75,7 +75,7 @@ helm install stunner-gateway-operator stunner/stunner-gateway-operator --create- To manually install the CRDs: ```console -kubectl apply -k github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.0.0 +kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml kubectl apply -f https://raw.githubusercontent.com/l7mp/stunner-helm/refs/heads/main/helm/stunner-gateway-operator/crds/stunner-crd.yaml ``` From fd61fbfdfda94d1478d4de41ea50fdbe5e5c0243 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Tue, 12 Nov 2024 20:01:48 +0100 Subject: [PATCH 229/243] refactor: Rewrite metric exporting to Open Telemetry --- go.mod | 30 +-- go.sum | 51 +++-- internal/telemetry/metrics.go | 170 --------------- internal/telemetry/statsconn.go | 53 ++--- internal/telemetry/telemetry.go | 233 ++++++++++++++++++++ internal/telemetry/tester/tester.go | 119 +++++++++++ internal/util/conn.go | 93 -------- internal/util/file_conn.go | 55 +++++ internal/util/socketpool.go | 46 +++- internal/util/socketpool_nonunix.go | 20 ++ internal/util/socketpool_unix.go | 8 +- relay.go | 21 +- relay_test.go | 22 +- server.go | 10 +- stunner.go | 23 +- stunner_test.go | 316 +++++++++++++++------------- 16 files changed, 760 insertions(+), 510 deletions(-) delete mode 100644 internal/telemetry/metrics.go create mode 100644 internal/telemetry/telemetry.go create mode 100644 internal/telemetry/tester/tester.go delete mode 100644 internal/util/conn.go create mode 100644 internal/util/file_conn.go create mode 100644 internal/util/socketpool_nonunix.go diff --git a/go.mod b/go.mod index 22168e75..58c96984 100644 --- a/go.mod +++ b/go.mod @@ -1,8 +1,8 @@ module github.com/l7mp/stunner -go 1.21 +go 1.22 -toolchain go1.21.5 +toolchain go1.22.4 require ( github.com/deepmap/oapi-codegen/v2 v2.1.0 @@ -18,12 +18,17 @@ require ( github.com/pion/logging v0.2.2 github.com/pion/transport/v3 v3.0.7 github.com/pion/turn/v4 v4.0.0 - github.com/prometheus/client_golang v1.20.2 + github.com/prometheus/client_golang v1.20.4 github.com/spf13/cobra v1.8.1 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 + go.opentelemetry.io/otel v1.31.0 + go.opentelemetry.io/otel/exporters/prometheus v0.53.0 + go.opentelemetry.io/otel/metric v1.31.0 + go.opentelemetry.io/otel/sdk v1.31.0 + go.opentelemetry.io/otel/sdk/metric v1.31.0 go.uber.org/zap v1.26.0 - golang.org/x/sys v0.23.0 + golang.org/x/sys v0.26.0 golang.org/x/time v0.5.0 k8s.io/api v0.29.1 k8s.io/apimachinery v0.29.1 @@ -42,6 +47,7 @@ require ( github.com/emicklei/go-restful/v3 v3.11.2 // indirect github.com/evanphx/json-patch v5.9.0+incompatible // indirect github.com/go-errors/errors v1.5.1 // indirect + github.com/go-logr/stdr v1.2.2 // indirect github.com/go-openapi/jsonpointer v0.20.2 // indirect github.com/go-openapi/jsonreference v0.20.4 // indirect github.com/go-openapi/swag v0.22.9 // indirect @@ -58,7 +64,6 @@ require ( github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/compress v1.17.9 // indirect - github.com/kylelemons/godebug v1.1.0 // indirect github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect github.com/mailru/easyjson v0.7.7 // indirect github.com/moby/spdystream v0.2.0 // indirect @@ -76,21 +81,22 @@ require ( github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.6.1 // indirect - github.com/prometheus/common v0.58.0 // indirect + github.com/prometheus/common v0.60.0 // indirect github.com/prometheus/procfs v0.15.1 // indirect github.com/wlynxg/anet v0.0.3 // indirect github.com/xlab/treeprint v1.2.0 // indirect + go.opentelemetry.io/otel/trace v1.31.0 // indirect go.starlark.net v0.0.0-20240123142251-f86470692795 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.26.0 // indirect + golang.org/x/crypto v0.27.0 // indirect golang.org/x/mod v0.17.0 // indirect - golang.org/x/net v0.28.0 // indirect - golang.org/x/oauth2 v0.22.0 // indirect + golang.org/x/net v0.29.0 // indirect + golang.org/x/oauth2 v0.23.0 // indirect golang.org/x/sync v0.8.0 // indirect - golang.org/x/term v0.23.0 // indirect - golang.org/x/text v0.17.0 // indirect + golang.org/x/term v0.24.0 // indirect + golang.org/x/text v0.18.0 // indirect golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect - google.golang.org/protobuf v1.34.2 // indirect + google.golang.org/protobuf v1.35.1 // indirect gopkg.in/evanphx/json-patch.v5 v5.9.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/go.sum b/go.sum index 647996fb..bc4e93a5 100644 --- a/go.sum +++ b/go.sum @@ -28,8 +28,11 @@ github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mN github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM= github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk= github.com/go-errors/errors v1.5.1/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q= @@ -139,12 +142,12 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.20.2 h1:5ctymQzZlyOON1666svgwn3s6IKWgfbjsejTMiXIyjg= -github.com/prometheus/client_golang v1.20.2/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE= +github.com/prometheus/client_golang v1.20.4 h1:Tgh3Yr67PaOv/uTqloMsCEdeuFTatm5zIq5+qNN23vI= +github.com/prometheus/client_golang v1.20.4/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE= github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E= github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY= -github.com/prometheus/common v0.58.0 h1:N+N8vY4/23r6iYfD3UQZUoJPnUYAo7v6LG5XZxjZTXo= -github.com/prometheus/common v0.58.0/go.mod h1:GpWM7dewqmVYcd7SmRaiWVe9SSqjf0UrwnYnpEZNuT0= +github.com/prometheus/common v0.60.0 h1:+V9PAREWNvJMAuJ1x1BaWl9dewMW4YrHZQbx0sJNllA= +github.com/prometheus/common v0.60.0/go.mod h1:h0LYf1R1deLSKtD4Vdg8gy4RuOvENW2J/h19V5NADQw= github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc= github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= @@ -172,6 +175,18 @@ github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.opentelemetry.io/otel v1.31.0 h1:NsJcKPIW0D0H3NgzPDHmo0WW6SptzPdqg/L1zsIm2hY= +go.opentelemetry.io/otel v1.31.0/go.mod h1:O0C14Yl9FgkjqcCZAsE053C13OaddMYr/hz6clDkEJE= +go.opentelemetry.io/otel/exporters/prometheus v0.53.0 h1:QXobPHrwiGLM4ufrY3EOmDPJpo2P90UuFau4CDPJA/I= +go.opentelemetry.io/otel/exporters/prometheus v0.53.0/go.mod h1:WOAXGr3D00CfzmFxtTV1eR0GpoHuPEu+HJT8UWW2SIU= +go.opentelemetry.io/otel/metric v1.31.0 h1:FSErL0ATQAmYHUIzSezZibnyVlft1ybhy4ozRPcF2fE= +go.opentelemetry.io/otel/metric v1.31.0/go.mod h1:C3dEloVbLuYoX41KpmAhOqNriGbA+qqH6PQ5E5mUfnY= +go.opentelemetry.io/otel/sdk v1.31.0 h1:xLY3abVHYZ5HSfOg3l2E5LUj2Cwva5Y7yGxnSW9H5Gk= +go.opentelemetry.io/otel/sdk v1.31.0/go.mod h1:TfRbMdhvxIIr/B2N2LQW2S5v9m3gOQ/08KsbbO5BPT0= +go.opentelemetry.io/otel/sdk/metric v1.31.0 h1:i9hxxLJF/9kkvfHppyLL55aW7iIJz4JjxTeYusH7zMc= +go.opentelemetry.io/otel/sdk/metric v1.31.0/go.mod h1:CRInTMVvNhUKgSAMbKyTMxqOBC0zgyxzW55lZzX43Y8= +go.opentelemetry.io/otel/trace v1.31.0 h1:ffjsj1aRouKewfr85U2aGagJ46+MvodynlQ1HYdmJys= +go.opentelemetry.io/otel/trace v1.31.0/go.mod h1:TXZkRk7SM2ZQLtR6eoAWQFIHPvzQ06FJAsO1tJg480A= go.starlark.net v0.0.0-20240123142251-f86470692795 h1:LmbG8Pq7KDGkglKVn8VpZOZj6vb9b8nKEGcg9l03epM= go.starlark.net v0.0.0-20240123142251-f86470692795/go.mod h1:LcLNIzVOMp4oV+uusnpk+VU+SzXaJakUuBjoCSWH5dM= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= @@ -183,8 +198,8 @@ go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw= -golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54= +golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A= +golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -195,10 +210,10 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE= -golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg= -golang.org/x/oauth2 v0.22.0 h1:BzDx2FehcG7jJwgWLELCdmLuxk2i+x9UDpSiss2u0ZA= -golang.org/x/oauth2 v0.22.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI= +golang.org/x/net v0.29.0 h1:5ORfpBpCs4HzDYoodCDBbwHzdR5UrLBZ3sOnUJmFoHo= +golang.org/x/net v0.29.0/go.mod h1:gLkgy8jTGERgjzMic6DS9+SP0ajcu6Xu3Orq/SpETg0= +golang.org/x/oauth2 v0.23.0 h1:PbgcYx2W7i4LvjJWEbf0ngHV6qJYr86PkAV3bXdLEbs= +golang.org/x/oauth2 v0.23.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -208,14 +223,14 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.23.0 h1:YfKFowiIMvtgl1UERQoTPPToxltDeZfbj4H7dVUCwmM= -golang.org/x/sys v0.23.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/term v0.23.0 h1:F6D4vR+EHoL9/sWAWgAR1H2DcHr4PareCbAaCo1RpuU= -golang.org/x/term v0.23.0/go.mod h1:DgV24QBUrK6jhZXl+20l6UWznPlwAHm1Q1mGHtydmSk= +golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo= +golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/term v0.24.0 h1:Mh5cbb+Zk2hqqXNO7S1iTjEphVL+jb8ZWaqh/g+JWkM= +golang.org/x/term v0.24.0/go.mod h1:lOBK/LVxemqiMij05LGJ0tzNr8xlmwBRJ81PX6wVLH8= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc= -golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= +golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224= +golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -230,8 +245,8 @@ golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg= -google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw= +google.golang.org/protobuf v1.35.1 h1:m3LfL6/Ca+fqnjnlqQXNpFPABW1UD7mjh8KO2mKFytA= +google.golang.org/protobuf v1.35.1/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= diff --git a/internal/telemetry/metrics.go b/internal/telemetry/metrics.go deleted file mode 100644 index 80fca0d4..00000000 --- a/internal/telemetry/metrics.go +++ /dev/null @@ -1,170 +0,0 @@ -package telemetry - -import ( - "github.com/pion/logging" - "github.com/prometheus/client_golang/prometheus" -) - -const ( - stunnerNamespace = "stunner" -) - -var ( - ConnLabels = []string{"name"} - CounterLabels = []string{"name", "direction"} - AllocActiveGauge prometheus.GaugeFunc - ListenerPacketsTotal *prometheus.CounterVec - ListenerBytesTotal *prometheus.CounterVec - ListenerConnsTotal *prometheus.CounterVec - ListenerConnsActive *prometheus.GaugeVec - ClusterPacketsTotal *prometheus.CounterVec - ClusterBytesTotal *prometheus.CounterVec - // promClusterConnsTotal *prometheus.CounterVec - // promClusterConnsActive *prometheus.GaugeVec -) - -func Init() { - // listener stats - ListenerConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ - Namespace: stunnerNamespace, - Subsystem: "listener", - Name: "connections", - Help: "Number of active downstream connections at a listener.", - }, ConnLabels) - ListenerConnsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: stunnerNamespace, - Subsystem: "listener", - Name: "connections_total", - Help: "Number of downstream connections at a listener.", - }, ConnLabels) - ListenerPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: stunnerNamespace, - Subsystem: "listener", - Name: "packets_total", - Help: "Number of datagrams sent or received at a listener.", - }, CounterLabels) - ListenerBytesTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: stunnerNamespace, - Subsystem: "listener", - Name: "bytes_total", - Help: "Number of bytes sent or received at a listener.", - }, CounterLabels) - - prometheus.MustRegister(ListenerPacketsTotal) - prometheus.MustRegister(ListenerBytesTotal) - prometheus.MustRegister(ListenerConnsTotal) - prometheus.MustRegister(ListenerConnsActive) - - // cluster stats - // promClusterConnsActive = prometheus.NewGaugeVec(prometheus.GaugeOpts{ - // Namespace: stunnerNamespace, - // Subsystem: "cluster", - // Name: "connections", - // Help: "Number of active upstream connections on behalf of a listener", - // }, promConnLabels) - // promClusterConnsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - // Namespace: stunnerNamespace, - // Subsystem: "cluster", - // Name: "connections_total", - // Help: "Number of upstream connections on behalf of a listener.", - // }, promConnLabels) - ClusterPacketsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: stunnerNamespace, - Subsystem: "cluster", - Name: "packets_total", - Help: "Number of datagrams sent to backends or received from backends on behalf of a listener", - }, CounterLabels) - ClusterBytesTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: stunnerNamespace, - Subsystem: "cluster", - Name: "bytes_total", - Help: "Number of bytes sent to backends or received from backends on behalf of a listener.", - }, CounterLabels) - - prometheus.MustRegister(ClusterPacketsTotal) - prometheus.MustRegister(ClusterBytesTotal) - // prometheus.MustRegister(promClusterConnsTotal) - // prometheus.MustRegister(promClusterConnsActive) -} - -func Close() { - _ = prometheus.Unregister(ListenerPacketsTotal) - _ = prometheus.Unregister(ListenerBytesTotal) - _ = prometheus.Unregister(ListenerConnsTotal) - _ = prometheus.Unregister(ListenerConnsActive) - _ = prometheus.Unregister(ClusterPacketsTotal) - _ = prometheus.Unregister(ClusterBytesTotal) - // _ = prometheus.Unregister(promClusterConnsTotal) - // _ = prometheus.Unregister(promClusterConnsActive) -} - -func IncrementPackets(n string, c ConnType, d Direction, count uint64) { - switch c { - case ListenerType: - ListenerPacketsTotal.WithLabelValues(n, d.String()).Add(float64(count)) - case ClusterType: - ClusterPacketsTotal.WithLabelValues(n, d.String()).Add(float64(count)) - } -} - -func IncrementBytes(n string, c ConnType, d Direction, count uint64) { - switch c { - case ListenerType: - ListenerBytesTotal.WithLabelValues(n, d.String()).Add(float64(count)) - case ClusterType: - ClusterBytesTotal.WithLabelValues(n, d.String()).Add(float64(count)) - } -} - -func AddConnection(n string, c ConnType) { - switch c { - case ListenerType: - ListenerConnsActive.WithLabelValues(n).Add(1) - ListenerConnsTotal.WithLabelValues(n).Add(1) - case ClusterType: - // promClusterConnsActive.WithLabelValues(n).Add(1) - // promClusterConnsTotal.WithLabelValues(n).Add(1) - } -} - -func SubConnection(n string, c ConnType) { - switch c { - case ListenerType: - ListenerConnsActive.WithLabelValues(n).Sub(1) - case ClusterType: - // promClusterConnsActive.WithLabelValues(n).Sub(1) - } -} - -func RegisterAllocationMetric(log logging.LeveledLogger, GetAllocationCount func() float64) { - AllocActiveGauge = prometheus.NewGaugeFunc( - prometheus.GaugeOpts{ - Namespace: stunnerNamespace, - Name: "allocations_active", - Help: "Number of active allocations.", - }, - GetAllocationCount, - ) - if err := prometheus.Register(AllocActiveGauge); err == nil { - log.Debug("GaugeFunc 'stunner_allocations_active' registered.") - } else { - log.Warn("GaugeFunc 'stunner_allocations_active' cannot be registered.") - } -} - -func UnregisterAllocationMetric(log logging.LeveledLogger) { - if AllocActiveGauge != nil { - if success := prometheus.Unregister(AllocActiveGauge); success { - log.Debug("GaugeFunc 'stunner_allocations_active' unregistered.") - return - } - } - log.Warn("GaugeFunc 'stunner_allocations_active' cannot be unregistered.") -} - -// func GetListenerPacketsTotal(ch chan prometheus.Metric) { -// go func() { -// defer close(ch) -// promListenerPacketsTotal.Collect(ch) -// }() -// } diff --git a/internal/telemetry/statsconn.go b/internal/telemetry/statsconn.go index a83565ed..36ac9665 100644 --- a/internal/telemetry/statsconn.go +++ b/internal/telemetry/statsconn.go @@ -9,13 +9,14 @@ import ( // Listener is a net.Listener that knows how to report to Prometheus. type Listener struct { net.Listener - name string - connType ConnType + name string + connType ConnType + telemetry *Telemetry } // NewListener creates a net.Listener that knows its name and type. -func NewListener(l net.Listener, n string, t ConnType) *Listener { - return &Listener{Listener: l, name: n, connType: t} +func NewListener(l net.Listener, n string, t ConnType, tm *Telemetry) *Listener { + return &Listener{Listener: l, name: n, connType: t, telemetry: tm} } // Accept accepts a new connection on a Listener. @@ -25,28 +26,29 @@ func (l *Listener) Accept() (net.Conn, error) { return nil, err } - return NewConn(conn, l.name, l.connType), nil + return NewConn(conn, l.name, l.connType, l.telemetry), nil } // Conn is a net.Conn that knows how to report to Prometheus. type Conn struct { net.Conn - name string - connType ConnType + name string + connType ConnType + telemetry *Telemetry } // NewConn allocates a stats conn that knows its name and type. -func NewConn(c net.Conn, n string, t ConnType) *Conn { - AddConnection(n, t) - return &Conn{Conn: c, name: n, connType: t} +func NewConn(c net.Conn, n string, t ConnType, tm *Telemetry) *Conn { + tm.AddConnection(n, t) + return &Conn{Conn: c, name: n, connType: t, telemetry: tm} } // Read reads from the Conn. func (c *Conn) Read(b []byte) (n int, err error) { n, err = c.Conn.Read(b) if n > 0 { - IncrementBytes(c.name, c.connType, Incoming, uint64(n)) - IncrementPackets(c.name, c.connType, Incoming, 1) + c.telemetry.IncrementBytes(c.name, c.connType, Incoming, uint64(n)) + c.telemetry.IncrementPackets(c.name, c.connType, Incoming, 1) } return } @@ -55,37 +57,38 @@ func (c *Conn) Read(b []byte) (n int, err error) { func (c *Conn) Write(b []byte) (n int, err error) { n, err = c.Conn.Write(b) if n > 0 { - IncrementBytes(c.name, c.connType, Outgoing, uint64(n)) - IncrementPackets(c.name, c.connType, Outgoing, 1) + c.telemetry.IncrementBytes(c.name, c.connType, Outgoing, uint64(n)) + c.telemetry.IncrementPackets(c.name, c.connType, Outgoing, 1) } return } // Close closes the Conn. func (c *Conn) Close() error { - SubConnection(c.name, c.connType) + c.telemetry.SubConnection(c.name, c.connType) return c.Conn.Close() } // PacketConn is a net.PacketConn that knows how to report to Prometheus. type PacketConn struct { net.PacketConn - name string - connType ConnType + name string + connType ConnType + telemetry *Telemetry } // NewPacketConn decorates a PacketConnn with metric reporting. -func NewPacketConn(c net.PacketConn, n string, t ConnType) *PacketConn { - AddConnection(n, t) - return &PacketConn{PacketConn: c, name: n, connType: t} +func NewPacketConn(c net.PacketConn, n string, t ConnType, tm *Telemetry) *PacketConn { + tm.AddConnection(n, t) + return &PacketConn{PacketConn: c, name: n, connType: t, telemetry: tm} } // ReadFrom reads from the PacketConn. func (c *PacketConn) ReadFrom(p []byte) (n int, addr net.Addr, err error) { n, addr, err = c.PacketConn.ReadFrom(p) if n > 0 { - IncrementBytes(c.name, c.connType, Incoming, uint64(n)) - IncrementPackets(c.name, c.connType, Incoming, 1) + c.telemetry.IncrementBytes(c.name, c.connType, Incoming, uint64(n)) + c.telemetry.IncrementPackets(c.name, c.connType, Incoming, 1) } return } @@ -94,8 +97,8 @@ func (c *PacketConn) ReadFrom(p []byte) (n int, addr net.Addr, err error) { func (c *PacketConn) WriteTo(p []byte, addr net.Addr) (n int, err error) { n, err = c.PacketConn.WriteTo(p, addr) if n > 0 { - IncrementBytes(c.name, c.connType, Outgoing, uint64(n)) - IncrementPackets(c.name, c.connType, Outgoing, 1) + c.telemetry.IncrementBytes(c.name, c.connType, Outgoing, uint64(n)) + c.telemetry.IncrementPackets(c.name, c.connType, Outgoing, 1) } return } @@ -104,6 +107,6 @@ func (c *PacketConn) WriteTo(p []byte, addr net.Addr) (n int, err error) { // WriteTo writes to the PacketConn. // Close closes the PacketConn. func (c *PacketConn) Close() error { - SubConnection(c.name, c.connType) + c.telemetry.SubConnection(c.name, c.connType) return c.PacketConn.Close() } diff --git a/internal/telemetry/telemetry.go b/internal/telemetry/telemetry.go new file mode 100644 index 00000000..4a186c92 --- /dev/null +++ b/internal/telemetry/telemetry.go @@ -0,0 +1,233 @@ +package telemetry + +import ( + "context" + "fmt" + "time" + + "github.com/pion/logging" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/prometheus" + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" + semconv "go.opentelemetry.io/otel/semconv/v1.26.0" +) + +const ( + stunnerInstrumentName = "stunner" + closeTimeout = 250 * time.Millisecond +) + +// Callbacks lets the caller to define various callbacks for reporting active metrics from an +// object that cannot be reached from this subpackage. This interface allows to easily add new +// metric reporters. +type Callbacks struct { + // GetAllocationCount should map to the total allocation counter of the server. + GetAllocationCount func() int64 +} + +type Telemetry struct { + sdkmetric.Reader + + meter metric.Meter + provider *sdkmetric.MeterProvider + ctx context.Context + cancel context.CancelFunc + + // Metrics instruments + ListenerPacketsCounter metric.Int64Counter + ListenerBytesCounter metric.Int64Counter + ListenerConnsCounter metric.Int64Counter + ListenerConnsGauge metric.Int64UpDownCounter + ClusterPacketsCounter metric.Int64Counter + ClusterBytesCounter metric.Int64Counter + AllocationsGauge metric.Int64ObservableGauge + + callbacks Callbacks + + log logging.LeveledLogger +} + +func New(callbacks Callbacks, dryRun bool, log logging.LeveledLogger) (*Telemetry, error) { + var reader sdkmetric.Reader + + resource, err := resource.Merge(resource.Default(), + resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceNameKey.String("stunner"))) + if err != nil { + return nil, fmt.Errorf("could not create OTEL resource: %w", err) + } + + if dryRun { + // Use manual collection mode + reader = sdkmetric.NewManualReader() + } else { + // Create a new Prometheus exporter (starts background collection) + exporter, err := prometheus.New() + if err != nil { + return nil, err + } + reader = exporter + } + + // Create a new MeterProvider with the Prometheus exporter + provider := sdkmetric.NewMeterProvider( + sdkmetric.WithResource(resource), + sdkmetric.WithReader(reader), + ) + + ctx, cancel := context.WithCancel(context.Background()) + t := &Telemetry{ + Reader: reader, + meter: provider.Meter(stunnerInstrumentName), + provider: provider, + callbacks: callbacks, + ctx: ctx, + cancel: cancel, + log: log, + } + + if err := t.init(); err != nil { + return nil, err + } + return t, nil +} + +// Close cleanly shuts down the meter provider and blocks until the shutdown cycle is finished or a +// timout expires. +func (t *Telemetry) Close() error { + ctx, cancel := context.WithTimeout(t.ctx, closeTimeout) + defer cancel() + defer t.cancel() + return t.provider.Shutdown(ctx) +} + +func (t *Telemetry) Ctx() context.Context { + return t.ctx +} + +func (t *Telemetry) init() error { + var err error + + // Initialize listener metrics + t.ListenerPacketsCounter, err = t.meter.Int64Counter( + stunnerInstrumentName+"_listener_packets_total", + metric.WithDescription("Number of datagrams sent or received at a listener"), + ) + if err != nil { + return err + } + + t.ListenerBytesCounter, err = t.meter.Int64Counter( + stunnerInstrumentName+"_listener_bytes_total", + metric.WithDescription("Number of bytes sent or received at a listener"), + ) + if err != nil { + return err + } + + t.ListenerConnsCounter, err = t.meter.Int64Counter( + stunnerInstrumentName+"_listener_connections_total", + metric.WithDescription("Number of all downstream connections observed at a listener"), + ) + if err != nil { + return err + } + + t.ListenerConnsGauge, err = t.meter.Int64UpDownCounter( + stunnerInstrumentName+"_listener_connections", + metric.WithDescription("Number of active downstream connections at a listener"), + ) + if err != nil { + return err + } + + // Initialize cluster metrics + t.ClusterPacketsCounter, err = t.meter.Int64Counter( + stunnerInstrumentName+"_cluster_packets_total", + metric.WithDescription("Number of datagrams sent to or received from backends"), + ) + if err != nil { + return err + } + + t.ClusterBytesCounter, err = t.meter.Int64Counter( + stunnerInstrumentName+"_cluster_bytes_total", + metric.WithDescription("Number of bytes sent to or received from backends"), + ) + if err != nil { + return err + } + + t.AllocationsGauge, err = t.meter.Int64ObservableGauge( + stunnerInstrumentName+"_allocations_active", + metric.WithDescription("Number of active allocations"), + ) + if err != nil { + return err + } + + _, err = t.meter.RegisterCallback( + func(_ context.Context, o metric.Observer) error { + o.ObserveInt64(t.AllocationsGauge, t.callbacks.GetAllocationCount()) + return nil + }, + t.AllocationsGauge, + ) + if err != nil { + return err + } + + return nil +} + +func (t *Telemetry) IncrementPackets(n string, c ConnType, d Direction, count uint64) { + attrs := metric.WithAttributes( + attribute.String("name", n), + attribute.String("direction", d.String()), + ) + + switch c { + case ListenerType: + t.ListenerPacketsCounter.Add(t.ctx, int64(count), attrs) + case ClusterType: + t.ClusterPacketsCounter.Add(t.ctx, int64(count), attrs) + } +} + +func (t *Telemetry) IncrementBytes(n string, c ConnType, d Direction, count uint64) { + attrs := metric.WithAttributes( + attribute.String("name", n), + attribute.String("direction", d.String()), + ) + + switch c { + case ListenerType: + t.ListenerBytesCounter.Add(t.ctx, int64(count), attrs) + case ClusterType: + t.ClusterBytesCounter.Add(t.ctx, int64(count), attrs) + } +} + +func (t *Telemetry) AddConnection(n string, c ConnType) { + attrs := metric.WithAttributes(attribute.String("name", n)) + + switch c { + case ListenerType: + t.ListenerConnsGauge.Add(t.ctx, 1, attrs) + t.ListenerConnsCounter.Add(t.ctx, 1, attrs) + case ClusterType: + // Cluster connection metrics are disabled + } +} + +func (t *Telemetry) SubConnection(n string, c ConnType) { + attrs := metric.WithAttributes(attribute.String("name", n)) + + switch c { + case ListenerType: + t.ListenerConnsGauge.Add(t.ctx, -1, attrs) + case ClusterType: + // Cluster connection metrics are disabled + } +} diff --git a/internal/telemetry/tester/tester.go b/internal/telemetry/tester/tester.go new file mode 100644 index 00000000..55336322 --- /dev/null +++ b/internal/telemetry/tester/tester.go @@ -0,0 +1,119 @@ +package tester + +import ( + "context" + "fmt" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" +) + +// Tester provides testing utilities for OpenTelemetry metrics +type Tester struct { + sdkmetric.Reader + *testing.T +} + +// New creates a new test helper and returns both the helper and a MeterProvider +// that can be used to initialize the system under test +func New(reader sdkmetric.Reader, t *testing.T) *Tester { + return &Tester{Reader: reader, T: t} +} + +// CollectAndCount returns the number of metrics with the given name and attributes +func (h *Tester) CollectAndCount(name string) int { + metrics := &metricdata.ResourceMetrics{} + err := h.Collect(context.Background(), metrics) + assert.NoError(h, err, "failed to collect metrics: %v") + + for _, scope := range metrics.ScopeMetrics { + for _, m := range scope.Metrics { + if m.Name != name { + continue + } + + sum, ok := m.Data.(metricdata.Sum[int64]) + assert.True(h, ok, fmt.Sprintf("metric %s is not a Sum", name)) + + return len(sum.DataPoints) + } + } + + return 0 +} + +// CollectAndGetInt returns the value of the metric with given name and attributes. +func (h *Tester) CollectAndGetInt(name string, attrs ...string) int { + assert.True(h, len(attrs)%2 == 0, "odd number of attribute key-value pairs") + + metrics := &metricdata.ResourceMetrics{} + err := h.Collect(context.Background(), metrics) + assert.NoError(h, err, "failed to collect metrics: %v") + + for _, scope := range metrics.ScopeMetrics { + for _, m := range scope.Metrics { + if m.Name != name { + continue + } + + sum, ok := m.Data.(metricdata.Sum[int64]) + assert.True(h, ok, fmt.Sprintf("metric %s is not a Sum", name)) + + for _, dp := range sum.DataPoints { + matches := true + for i := 0; i < len(attrs); i += 2 { + if val, ok := dp.Attributes.Value(attribute.Key(attrs[i])); !ok || val.AsString() != attrs[i+1] { + matches = false + break + } + } + if matches { + return int(dp.Value) + } + } + } + } + + return 0 +} + +// CollectAndDump returns the metrics with the given name and attributes as a string. +func (h *Tester) CollectAndDump(name string, attrs ...string) string { + metrics := &metricdata.ResourceMetrics{} + if err := h.Collect(context.Background(), metrics); err != nil { + return "" + } + + ret := []string{} + for _, scope := range metrics.ScopeMetrics { + for _, m := range scope.Metrics { + if m.Name != name { + continue + } + + sum, ok := m.Data.(metricdata.Sum[int64]) + if !ok { + return "" + } + + for _, dp := range sum.DataPoints { + matches := true + for i := 0; i < len(attrs); i += 2 { + if val, ok := dp.Attributes.Value(attribute.Key(attrs[i])); !ok || val.AsString() != attrs[i+1] { + matches = false + break + } + } + if matches { + ret = append(ret, fmt.Sprintf("%v", dp)) + } + } + } + } + + return strings.Join(ret, ",") +} diff --git a/internal/util/conn.go b/internal/util/conn.go deleted file mode 100644 index 459d9bd8..00000000 --- a/internal/util/conn.go +++ /dev/null @@ -1,93 +0,0 @@ -package util - -import ( - "fmt" - "net" - "os" - "time" - - "github.com/l7mp/stunner/internal/telemetry" - "github.com/pion/transport/v3" -) - -type FileConnAddr struct { - File *os.File -} - -func (s *FileConnAddr) Network() string { return "file" } -func (s *FileConnAddr) String() string { return s.File.Name() } - -type FileConn struct { - file *os.File -} - -func (f *FileConn) Read(b []byte) (n int, err error) { - return f.file.Read(b) -} - -func (f *FileConn) Write(b []byte) (n int, err error) { - return f.file.Write(b) -} - -func (f *FileConn) Close() error { - return f.file.Close() -} - -func (f *FileConn) LocalAddr() net.Addr { - return &FileConnAddr{File: f.file} -} - -func (f *FileConn) RemoteAddr() net.Addr { - return &FileConnAddr{File: f.file} -} - -func (f *FileConn) SetDeadline(t time.Time) error { - return nil -} - -func (f *FileConn) SetReadDeadline(t time.Time) error { - return nil -} - -func (f *FileConn) SetWriteDeadline(t time.Time) error { - return nil -} - -// NewFileConn returns a wrapper that shows an os.File as a net.Conn. -func NewFileConn(file *os.File) net.Conn { - return &FileConn{file: file} -} - -// PacketConnPool is a factory to create pools of related PacketConns, which may either be a set of -// PacketConns bound to the same local IP using SO_REUSEPORT (on unix, under certain circumstances) -// that can do multithreaded readloops, or a single PacketConn as a fallback for non-unic -// architectures and for testing. -type PacketConnPool interface { - // Make creates a PacketConnPool, caller must make sure to close the sockets. - Make(network, address string) ([]net.PacketConn, error) - // Size returns the number of sockets in the pool. - Size() int -} - -// defaultPacketConPool implements a socketpool that consists of only a single socket, used as a fallback for architectures that do not support SO_REUSEPORT or when socket pooling is disabled. -type defaultPacketConnPool struct { - transport.Net - listenerName string -} - -// Make creates a PacketConnPool, caller must make sure to close the sockets. -func (p *defaultPacketConnPool) Make(network, address string) ([]net.PacketConn, error) { - conns := []net.PacketConn{} - - conn, err := p.ListenPacket(network, address) - if err != nil { - return []net.PacketConn{}, fmt.Errorf("failed to create PacketConn at %s "+ - "(REUSEPORT: false): %s", address, err) - } - - conn = telemetry.NewPacketConn(conn, p.listenerName, telemetry.ListenerType) - conns = append(conns, conn) - return conns, nil -} - -func (p *defaultPacketConnPool) Size() int { return 1 } diff --git a/internal/util/file_conn.go b/internal/util/file_conn.go new file mode 100644 index 00000000..d7b8c5c7 --- /dev/null +++ b/internal/util/file_conn.go @@ -0,0 +1,55 @@ +package util + +import ( + "net" + "os" + "time" +) + +type FileConnAddr struct { + File *os.File +} + +func (s *FileConnAddr) Network() string { return "file" } +func (s *FileConnAddr) String() string { return s.File.Name() } + +type FileConn struct { + file *os.File +} + +func (f *FileConn) Read(b []byte) (n int, err error) { + return f.file.Read(b) +} + +func (f *FileConn) Write(b []byte) (n int, err error) { + return f.file.Write(b) +} + +func (f *FileConn) Close() error { + return f.file.Close() +} + +func (f *FileConn) LocalAddr() net.Addr { + return &FileConnAddr{File: f.file} +} + +func (f *FileConn) RemoteAddr() net.Addr { + return &FileConnAddr{File: f.file} +} + +func (f *FileConn) SetDeadline(t time.Time) error { + return nil +} + +func (f *FileConn) SetReadDeadline(t time.Time) error { + return nil +} + +func (f *FileConn) SetWriteDeadline(t time.Time) error { + return nil +} + +// NewFileConn returns a wrapper that shows an os.File as a net.Conn. +func NewFileConn(file *os.File) net.Conn { + return &FileConn{file: file} +} diff --git a/internal/util/socketpool.go b/internal/util/socketpool.go index c02b9281..2c63a6d3 100644 --- a/internal/util/socketpool.go +++ b/internal/util/socketpool.go @@ -1,15 +1,45 @@ -//go:build !linux - package util import ( + "fmt" + "net" + + "github.com/l7mp/stunner/internal/telemetry" "github.com/pion/transport/v3" ) -// NewPacketConnPool creates a new packet connection pool which is fixed to a single connection, -// used if threadNum is zero or if we are running on top of transport.VNet (which does not support -// reuseport), or if we are on non-unix, see the fallback in socketpool.go. -func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int) PacketConnPool { - // default to a single socket for vnet or if udp multithreading is disabled - return &defaultPacketConnPool{Net: vnet, listenerName: listenerName} +// PacketConnPool is a factory to create pools of related PacketConns, which may either be a set of +// PacketConns bound to the same local IP using SO_REUSEPORT (on unix, under certain circumstances) +// that can do multithreaded readloops, or a single PacketConn as a fallback for non-unic +// architectures and for testing. +type PacketConnPool interface { + // Make creates a PacketConnPool, caller must make sure to close the sockets. + Make(network, address string) ([]net.PacketConn, error) + // Size returns the number of sockets in the pool. + Size() int } + +// defaultPacketConPool implements a socketpool that consists of only a single socket, used as a +// fallback for architectures that do not support SO_REUSEPORT or when socket pooling is disabled. +type defaultPacketConnPool struct { + transport.Net + listenerName string + telemetry *telemetry.Telemetry +} + +// Make creates a PacketConnPool, caller must make sure to close the sockets. +func (p *defaultPacketConnPool) Make(network, address string) ([]net.PacketConn, error) { + conns := []net.PacketConn{} + + conn, err := p.ListenPacket(network, address) + if err != nil { + return []net.PacketConn{}, fmt.Errorf("failed to create PacketConn at %s "+ + "(REUSEPORT: false): %s", address, err) + } + + conn = telemetry.NewPacketConn(conn, p.listenerName, telemetry.ListenerType, p.telemetry) + conns = append(conns, conn) + return conns, nil +} + +func (p *defaultPacketConnPool) Size() int { return 1 } diff --git a/internal/util/socketpool_nonunix.go b/internal/util/socketpool_nonunix.go new file mode 100644 index 00000000..23df79fe --- /dev/null +++ b/internal/util/socketpool_nonunix.go @@ -0,0 +1,20 @@ +//go:build !linux + +package util + +import ( + "github.com/l7mp/stunner/internal/telemetry" + "github.com/pion/transport/v3" +) + +// NewPacketConnPool creates a new packet connection pool which is fixed to a single connection, +// used if threadNum is zero or if we are running on top of transport.VNet (which does not support +// reuseport), or if we are on non-unix, see the fallback in socketpool.go. +func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int, t *telemetry.Telemetry) PacketConnPool { + // default to a single socket for vnet or if udp multithreading is disabled + return &defaultPacketConnPool{ + Net: vnet, + listenerName: listenerName, + telemetry: t, + } +} diff --git a/internal/util/socketpool_unix.go b/internal/util/socketpool_unix.go index d6f09764..46476256 100644 --- a/internal/util/socketpool_unix.go +++ b/internal/util/socketpool_unix.go @@ -20,12 +20,13 @@ type unixPacketConnPool struct { net.ListenConfig listenerName string size int + telemetry *telemetry.Telemetry } // NewPacketConnPool creates a new packet connection pool. Pooling is disabled if threadNum is zero // or if we are running on top of transport.VNet (which does not support reuseport), or if we are // on non-unix, see the fallback in socketpool.go. -func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int) PacketConnPool { +func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int, t *telemetry.Telemetry) PacketConnPool { // default to a single socket for vnet or if udp multithreading is disabled _, ok := vnet.(*stdnet.Net) if ok && threadNum > 0 { @@ -45,9 +46,10 @@ func NewPacketConnPool(listenerName string, vnet transport.Net, threadNum int) P }, size: threadNum, listenerName: listenerName, + telemetry: t, } } else { - return &defaultPacketConnPool{listenerName: listenerName, Net: vnet} + return &defaultPacketConnPool{listenerName: listenerName, Net: vnet, telemetry: t} } } @@ -62,7 +64,7 @@ func (p *unixPacketConnPool) Make(network, address string) ([]net.PacketConn, er return []net.PacketConn{}, fmt.Errorf("failed to create PacketConn "+ "%d at %s (REUSEPORT: %t): %s", i, address, (p.size > 0), err) } - conn = telemetry.NewPacketConn(conn, p.listenerName, telemetry.ListenerType) + conn = telemetry.NewPacketConn(conn, p.listenerName, telemetry.ListenerType, p.telemetry) conns = append(conns, conn) } diff --git a/relay.go b/relay.go index 953cfe68..c93a1f1f 100644 --- a/relay.go +++ b/relay.go @@ -56,9 +56,11 @@ type RelayGen struct { // Logger is a logger factory we can use to generate per-listener relay loggers. Logger *logger.LeveledLoggerFactory + + telemetry *telemetry.Telemetry } -func NewRelayGen(l *object.Listener, logger *logger.LeveledLoggerFactory) *RelayGen { +func NewRelayGen(l *object.Listener, t *telemetry.Telemetry, logger *logger.LeveledLoggerFactory) *RelayGen { return &RelayGen{ Listener: l, RelayAddress: l.Addr, @@ -66,6 +68,7 @@ func NewRelayGen(l *object.Listener, logger *logger.LeveledLoggerFactory) *Relay ClusterCache: lru.New(ClusterCacheSize), Net: l.Net, Logger: logger, + telemetry: t, } } @@ -86,7 +89,7 @@ func (r *RelayGen) AllocatePacketConn(network string, requestedPort int) (net.Pa return nil, nil, err } - conn = NewPortRangePacketConn(conn, r.PortRangeChecker, + conn = NewPortRangePacketConn(conn, r.PortRangeChecker, r.telemetry, r.Logger.NewLogger(fmt.Sprintf("relay-%s", r.Listener.Name))) relayAddr, ok := conn.LocalAddr().(*net.UDPAddr) @@ -145,18 +148,20 @@ func (s *Stunner) GenPortRangeChecker(g *RelayGen) PortRangeChecker { type PortRangePacketConn struct { net.PacketConn checker PortRangeChecker - log logging.LeveledLogger readDeadline time.Time + telemetry *telemetry.Telemetry lock sync.Mutex + log logging.LeveledLogger } // NewPortRangePacketConn decorates a PacketConn with filtering on a target port range. Errors are reported per listener name. -func NewPortRangePacketConn(c net.PacketConn, checker PortRangeChecker, log logging.LeveledLogger) net.PacketConn { +func NewPortRangePacketConn(c net.PacketConn, checker PortRangeChecker, t *telemetry.Telemetry, log logging.LeveledLogger) net.PacketConn { // cluster add/sub connection is not tracked // AddConnection(n, t) r := PortRangePacketConn{ PacketConn: c, checker: checker, + telemetry: t, log: log, } @@ -172,8 +177,8 @@ func (c *PortRangePacketConn) WriteTo(p []byte, peerAddr net.Addr) (int, error) n, err := c.PacketConn.WriteTo(p, peerAddr) if n > 0 { - telemetry.IncrementBytes(cluster.Name, telemetry.ClusterType, telemetry.Outgoing, uint64(n)) - telemetry.IncrementPackets(cluster.Name, telemetry.ClusterType, telemetry.Outgoing, 1) + c.telemetry.IncrementBytes(cluster.Name, telemetry.ClusterType, telemetry.Outgoing, uint64(n)) + c.telemetry.IncrementPackets(cluster.Name, telemetry.ClusterType, telemetry.Outgoing, 1) } return n, err @@ -204,8 +209,8 @@ func (c *PortRangePacketConn) ReadFrom(p []byte) (int, net.Addr, error) { } if n > 0 { - telemetry.IncrementBytes(cluster.Name, telemetry.ClusterType, telemetry.Incoming, uint64(n)) - telemetry.IncrementPackets(cluster.Name, telemetry.ClusterType, telemetry.Incoming, 1) + c.telemetry.IncrementBytes(cluster.Name, telemetry.ClusterType, telemetry.Incoming, uint64(n)) + c.telemetry.IncrementPackets(cluster.Name, telemetry.ClusterType, telemetry.Incoming, 1) } return n, peerAddr, nil diff --git a/relay_test.go b/relay_test.go index b9845570..09e665cb 100644 --- a/relay_test.go +++ b/relay_test.go @@ -35,9 +35,6 @@ func getChecker(minPort, maxPort int) PortRangeChecker { } func TestPortRangePacketConn(t *testing.T) { - telemetry.Init() - defer telemetry.Close() - lim := test.TimeOut(time.Second * 30) defer lim.Stop() @@ -53,6 +50,10 @@ func TestPortRangePacketConn(t *testing.T) { return } + tm, err := telemetry.New(telemetry.Callbacks{}, false, loggerFactory.NewLogger("metric")) + assert.NoError(t, err, "should succeed") + defer tm.Close() + t.Run("LoopbackOnValidPort", func(t *testing.T) { log.Debug("Creating base socket") addr := "127.0.0.1:15000" @@ -61,7 +62,7 @@ func TestPortRangePacketConn(t *testing.T) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := NewPortRangePacketConn(baseConn, getChecker(10000, 20000), log) + conn := NewPortRangePacketConn(baseConn, getChecker(10000, 20000), tm, log) assert.NoError(t, err, "should create port-range filtered packetconn") log.Debug("Sending packet") @@ -91,7 +92,7 @@ func TestPortRangePacketConn(t *testing.T) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := NewPortRangePacketConn(baseConn, getChecker(10000, 20000), log) + conn := NewPortRangePacketConn(baseConn, getChecker(10000, 20000), tm, log) assert.NoError(t, err, "should create port-range filtered packetconn") log.Debug("Sending packet") @@ -120,7 +121,7 @@ func TestPortRangePacketConn(t *testing.T) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := NewPortRangePacketConn(baseConn, getChecker(15000, 15000), log) + conn := NewPortRangePacketConn(baseConn, getChecker(15000, 15000), tm, log) assert.NoError(t, err, "should create port-range filtered packetconn") log.Debug("Sending packet") @@ -145,9 +146,6 @@ func TestPortRangePacketConn(t *testing.T) { // BenchmarkPortRangePacketConn sends lots of invalid packets: this is mostly for testing the logger func BenchmarkPortRangePacketConn(b *testing.B) { - telemetry.Init() - defer telemetry.Close() - loggerFactory := logger.NewLoggerFactory(connTestLoglevel) log := loggerFactory.NewLogger("test") // relayLog := loggerFactory.WithRateLimiter(.25, 1).NewLogger("relay") @@ -159,6 +157,10 @@ func BenchmarkPortRangePacketConn(b *testing.B) { b.Fatalf("Cannot allocate vnet: %s", err.Error()) } + tm, err := telemetry.New(telemetry.Callbacks{}, false, loggerFactory.NewLogger("metric")) + assert.NoError(b, err, "should succeed") + defer tm.Close() + log.Debug("Creating base socket") addr := "127.0.0.1:25000" baseConn, err := nw.ListenPacket("udp", addr) @@ -168,7 +170,7 @@ func BenchmarkPortRangePacketConn(b *testing.B) { msg := "PING!" log.Debug("Creating filtered packet conn wrappeer socket") - conn := WithCounter(NewPortRangePacketConn(baseConn, getChecker(15000, 15000), relayLog)) + conn := WithCounter(NewPortRangePacketConn(baseConn, getChecker(15000, 15000), tm, relayLog)) if err != nil { b.Fatalf("Cannot create port-range packetconn: %s", err.Error()) } diff --git a/server.go b/server.go index 6c072eb1..b3b5613a 100644 --- a/server.go +++ b/server.go @@ -31,7 +31,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { var pConns []turn.PacketConnConfig var lConns []turn.ListenerConfig - relay := NewRelayGen(l, s.logger) + relay := NewRelayGen(l, s.telemetry, s.logger) relay.PortRangeChecker = s.GenPortRangeChecker(relay) permissionHandler := s.NewPermissionHandler(l) @@ -40,7 +40,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { switch l.Proto { case stnrv1.ListenerProtocolTURNUDP: - socketPool := util.NewPacketConnPool(l.Name, l.Net, s.udpThreadNum) + socketPool := util.NewPacketConnPool(l.Name, l.Net, s.udpThreadNum, s.telemetry) s.log.Infof("setting up UDP listener socket pool at %s with %d readloop threads", addr, socketPool.Size()) @@ -68,7 +68,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { return fmt.Errorf("failed to create TCP listener at %s: %s", addr, err) } - tcpListener = telemetry.NewListener(tcpListener, l.Name, telemetry.ListenerType) + tcpListener = telemetry.NewListener(tcpListener, l.Name, telemetry.ListenerType, s.telemetry) conn := turn.ListenerConfig{ Listener: tcpListener, @@ -96,7 +96,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { return fmt.Errorf("failed to create TLS listener at %s: %s", addr, err) } - tlsListener = telemetry.NewListener(tlsListener, l.Name, telemetry.ListenerType) + tlsListener = telemetry.NewListener(tlsListener, l.Name, telemetry.ListenerType, s.telemetry) conn := turn.ListenerConfig{ Listener: tlsListener, @@ -129,7 +129,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { return fmt.Errorf("failed to create DTLS listener at %s: %s", addr, err) } - dtlsListener = telemetry.NewListener(dtlsListener, l.Name, telemetry.ListenerType) + dtlsListener = telemetry.NewListener(dtlsListener, l.Name, telemetry.ListenerType, s.telemetry) conn := turn.ListenerConfig{ Listener: dtlsListener, diff --git a/stunner.go b/stunner.go index acc1a73a..8e4c3905 100644 --- a/stunner.go +++ b/stunner.go @@ -29,6 +29,7 @@ type Stunner struct { suppressRollback, dryRun bool resolver resolver.DnsResolver udpThreadNum int + telemetry *telemetry.Telemetry logger *logger.LeveledLoggerFactory log logging.LeveledLogger net transport.Net @@ -101,13 +102,20 @@ func NewStunner(options Options) *Stunner { s.clusterManager = manager.NewManager("cluster-manager", object.NewClusterFactory(r, logger), logger) + telemetryCallbacks := telemetry.Callbacks{ + GetAllocationCount: func() int64 { return s.GetActiveConnections() }, + } + t, err := telemetry.New(telemetryCallbacks, s.dryRun, logger.NewLogger("metrics")) + if err != nil { + log.Errorf("Could not initialize metric provider: %s", err.Error()) + return nil + } + s.telemetry = t + if !s.dryRun { s.resolver.Start() - telemetry.Init() - telemetry.RegisterAllocationMetric(s.log, s.GetActiveConnections) } - // TODO: remove this when STUNner gains self-managed dataplanes s.ready = true return s @@ -275,16 +283,15 @@ func (s *Stunner) Close() { } } - telemetry.UnregisterAllocationMetric(s.log) - if !s.dryRun { - telemetry.Close() + if err := s.telemetry.Close(); err != nil { // blocks until finished + s.log.Errorf("Could not shutdown metric provider cleanly: %s", err.Error()) } s.resolver.Close() } // GetActiveConnections returns the number of active downstream (listener-side) TURN allocations. -func (s *Stunner) GetActiveConnections() float64 { +func (s *Stunner) GetActiveConnections() int64 { count := s.AllocationCount() - return float64(count) + return int64(count) } diff --git a/stunner_test.go b/stunner_test.go index 5b4f3360..b10e6b54 100644 --- a/stunner_test.go +++ b/stunner_test.go @@ -19,11 +19,10 @@ import ( "github.com/pion/transport/v3/test" "github.com/pion/transport/v3/vnet" "github.com/pion/turn/v4" - "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/assert" "github.com/l7mp/stunner/internal/resolver" - "github.com/l7mp/stunner/internal/telemetry" + telemetrytester "github.com/l7mp/stunner/internal/telemetry/tester" "github.com/l7mp/stunner/pkg/logger" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" @@ -32,12 +31,17 @@ import ( cfgclient "github.com/l7mp/stunner/pkg/config/client" ) -var stunnerTestLoglevel string = "all:ERROR" +const ( + // timeout = 200 * time.Millisecond + // interval = 50 * time.Millisecond -// var stunnerTestLoglevel string = stnrv1.DefaultLogLevel -// var stunnerTestLoglevel string = "all:INFO" -// var stunnerTestLoglevel string = "all:TRACE" -// var stunnerTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" + stunnerTestLoglevel string = "all:ERROR" + + // stunnerTestLoglevel string = stnrv1.DefaultLogLevel + // stunnerTestLoglevel string = "all:INFO" + // stunnerTestLoglevel string = "all:TRACE" + // stunnerTestLoglevel string = "all:TRACE,vnet:INFO,turn:ERROR,turnc:ERROR" +) var certPem, keyPem, _ = GenerateSelfSignedKey() var certPem64 = base64.StdEncoding.EncodeToString(certPem) @@ -176,7 +180,7 @@ func stunnerEchoTest(conf echoTestConfig) { assert.NoError(t, echoConn.Close(), "cannot close echo server connection") } } - time.Sleep(150 * time.Millisecond) + time.Sleep(50 * time.Millisecond) client.Close() } @@ -655,7 +659,7 @@ type StunnerTestClusterConfig struct { config stnrv1.StunnerConfig echoServerAddr string result bool - tester func(t *testing.T) + tester func(h *telemetrytester.Tester) } var testClusterConfigsWithVNet = []StunnerTestClusterConfig{ @@ -1198,34 +1202,36 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: true, - tester: func(t *testing.T) { - c := telemetry.ListenerConnsTotal - assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) - - g := telemetry.ListenerConnsActive - assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) - - c = telemetry.ListenerPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(500)) - - c = telemetry.ListenerBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(2000)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(2000)) - - c = telemetry.ClusterPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(500)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(500)) - - c = telemetry.ClusterBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(2000)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) + tester: func(h *telemetrytester.Tester) { + // stunner_listener_connections_total + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections_total")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + + // stunner_listener_connections + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + + // stunner_listener_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "rx"), 20) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "tx"), 20) + + // stunner_listener_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "rx"), 20) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "tx"), 20) + + // stunner_cluster_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_cluster_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "rx"), 20) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "tx"), 20) + + // stunner_cluster_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_cluster_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "rx"), 20) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "tx"), 20) }, }, { @@ -1265,34 +1271,36 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: true, - tester: func(t *testing.T) { - c := telemetry.ListenerConnsTotal - assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) - - g := telemetry.ListenerConnsActive - assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) - - c = telemetry.ListenerPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(500)) - - c = telemetry.ListenerBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(2000)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(2000)) - - c = telemetry.ClusterPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(500)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(500)) - - c = telemetry.ClusterBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(2000)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) + tester: func(h *telemetrytester.Tester) { + // stunner_listener_connections_total + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections_total")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + + // stunner_listener_connections + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + + // stunner_listener_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "rx"), 200) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "tx"), 200) + + // stunner_listener_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "rx"), 2000) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "tx"), 2000) + + // stunner_cluster_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_cluster_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "rx"), 200) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "tx"), 200) + + // stunner_cluster_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_cluster_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "rx"), 2000) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "tx"), 2000) }, }, { @@ -1328,34 +1336,36 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: false, - tester: func(t *testing.T) { - c := telemetry.ListenerConnsTotal - assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) - - g := telemetry.ListenerConnsActive - assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) - - c = telemetry.ListenerPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data - assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling - - c = telemetry.ListenerBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data - assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling - - c = telemetry.ClusterPacketsTotal - assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) - - c = telemetry.ClusterBytesTotal - assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + tester: func(h *telemetrytester.Tester) { + // stunner_listener_connections_total + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections_total")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + + // stunner_listener_connections + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + + // stunner_listener_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "rx"), 500) // signaling+data + assert.Less(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "tx"), 50) // just signaling + + // stunner_listener_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "rx"), 1000) // signaling+data + assert.Less(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "tx"), 1000) // just signaling + + // stunner_cluster_packets_total + assert.Equal(h, 0, h.CollectAndCount("stunner_cluster_packets_total")) + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "rx")) // fail + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "tx")) // fail + + // stunner_cluster_bytes_total + assert.Equal(h, 0, h.CollectAndCount("stunner_cluster_bytes_total")) + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "rx")) // fail + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "tx")) // fail }, }, { @@ -1391,34 +1401,36 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: false, - tester: func(t *testing.T) { - c := telemetry.ListenerConnsTotal - assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) - - g := telemetry.ListenerConnsActive - assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) - - c = telemetry.ListenerPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) // signaling+data - assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(50)) // just signaling - - c = telemetry.ListenerBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(1000)) // signaling+data - assert.LessOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(1000)) // just signaling - - c = telemetry.ClusterPacketsTotal - assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) - - c = telemetry.ClusterBytesTotal - assert.Equal(t, 0, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx"))) - assert.Equal(t, float64(0), testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx"))) + tester: func(h *telemetrytester.Tester) { + // stunner_listener_connections_total + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections_total")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + + // stunner_listener_connections + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + + // stunner_listener_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "rx"), 500) // signaling+data + assert.Less(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "tx"), 50) // just signaling + + // stunner_listener_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "rx"), 1000) // signaling+data + assert.Less(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "tx"), 1000) // just signaling + + // stunner_cluster_packets_total + assert.Equal(h, 0, h.CollectAndCount("stunner_cluster_packets_total")) + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "rx")) // fail + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "tx")) // fail + + // stunner_cluster_bytes_total + assert.Equal(h, 0, h.CollectAndCount("stunner_cluster_bytes_total")) + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "rx")) // fail + assert.Equal(h, 0, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "tx")) // fail }, }, { @@ -1458,34 +1470,36 @@ var testPortRangeConfigsWithVNet = []StunnerTestClusterConfig{ }, echoServerAddr: "1.2.3.5:5678", result: true, - tester: func(t *testing.T) { - c := telemetry.ListenerConnsTotal - assert.Equal(t, 1, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(c.WithLabelValues("udp"))) - - g := telemetry.ListenerConnsActive - assert.Equal(t, 1, testutil.CollectAndCount(g), "ListenerConnsTotal") - assert.Equal(t, float64(1), testutil.ToFloat64(g.WithLabelValues("udp"))) - - c = telemetry.ListenerPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(500)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(500)) - - c = telemetry.ListenerBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "rx")), float64(2000)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("udp", "tx")), float64(2000)) - - c = telemetry.ClusterPacketsTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(500)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(500)) - - c = telemetry.ClusterBytesTotal - assert.Equal(t, 2, testutil.CollectAndCount(c), "ListenerConnsTotal") - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "rx")), float64(2000)) - assert.GreaterOrEqual(t, testutil.ToFloat64(c.WithLabelValues("echo-server-cluster", "tx")), float64(2000)) + tester: func(h *telemetrytester.Tester) { + // stunner_listener_connections_total + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections_total")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections_total", "name", "udp")) + + // stunner_listener_connections + assert.Equal(h, 1, h.CollectAndCount("stunner_listener_connections")) // name: udp + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + assert.Equal(h, 1, h.CollectAndGetInt("stunner_listener_connections", "name", "udp")) + + // stunner_listener_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "rx"), 200) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_packets_total", "name", "udp", "direction", "tx"), 200) + + // stunner_listener_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_listener_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "rx"), 2000) + assert.Greater(h, h.CollectAndGetInt("stunner_listener_bytes_total", "name", "udp", "direction", "tx"), 2000) + + // stunner_cluster_packets_total + assert.Equal(h, 2, h.CollectAndCount("stunner_cluster_packets_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "rx"), 200) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_packets_total", "name", "echo-server-cluster", "direction", "tx"), 200) + + // stunner_cluster_bytes_total + assert.Equal(h, 2, h.CollectAndCount("stunner_cluster_bytes_total")) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "rx"), 2000) + assert.Greater(h, h.CollectAndGetInt("stunner_cluster_bytes_total", "name", "echo-server-cluster", "direction", "tx"), 2000) }, }, // TODO: implement port-range filtering for DNS clusters @@ -1590,6 +1604,7 @@ func TestStunnerPortRangeWithVNet(t *testing.T) { stunner := NewStunner(Options{ LogLevel: stunnerTestLoglevel, SuppressRollback: true, + DryRun: false, Resolver: mockDns, Net: v.podnet, }) @@ -1620,7 +1635,7 @@ func TestStunnerPortRangeWithVNet(t *testing.T) { stunnerEchoFloodTest(testConfig) if c.tester != nil { - c.tester(t) + c.tester(telemetrytester.New(stunner.telemetry, t)) } assert.NoError(t, lconn.Close(), "cannot close TURN client connection") @@ -1632,6 +1647,7 @@ func TestStunnerPortRangeWithVNet(t *testing.T) { func stunnerEchoFloodTest(conf echoTestConfig) { t := conf.t + t.Helper() log := conf.loggerFactory.NewLogger("test") client, err := turn.NewClient(&turn.ClientConfig{ From 5b871592071668a39ef9da7254b433f625f0992b Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Nov 2024 18:13:33 +0100 Subject: [PATCH 230/243] test: Mark telemetry tester helpers as such --- internal/telemetry/tester/tester.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/internal/telemetry/tester/tester.go b/internal/telemetry/tester/tester.go index 55336322..d24b4a9b 100644 --- a/internal/telemetry/tester/tester.go +++ b/internal/telemetry/tester/tester.go @@ -26,6 +26,8 @@ func New(reader sdkmetric.Reader, t *testing.T) *Tester { // CollectAndCount returns the number of metrics with the given name and attributes func (h *Tester) CollectAndCount(name string) int { + h.Helper() + metrics := &metricdata.ResourceMetrics{} err := h.Collect(context.Background(), metrics) assert.NoError(h, err, "failed to collect metrics: %v") @@ -48,6 +50,8 @@ func (h *Tester) CollectAndCount(name string) int { // CollectAndGetInt returns the value of the metric with given name and attributes. func (h *Tester) CollectAndGetInt(name string, attrs ...string) int { + h.Helper() + assert.True(h, len(attrs)%2 == 0, "odd number of attribute key-value pairs") metrics := &metricdata.ResourceMetrics{} @@ -83,6 +87,8 @@ func (h *Tester) CollectAndGetInt(name string, attrs ...string) int { // CollectAndDump returns the metrics with the given name and attributes as a string. func (h *Tester) CollectAndDump(name string, attrs ...string) string { + h.Helper() + metrics := &metricdata.ResourceMetrics{} if err := h.Collect(context.Background(), metrics); err != nil { return "" From 5e476c822b44c0046ca6a416a47c2d8cee41899e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Nov 2024 18:13:45 +0100 Subject: [PATCH 231/243] fix: Improve CDS server port randomization on tests --- pkg/config/cds_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index c53daa7b..c8519181 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -27,7 +27,7 @@ const stunnerLogLevel = "all:ERROR" // run on random port func getRandCDSAddr() string { - rndPort := rand.Intn(10000) + 50000 + rndPort := rand.Intn(10000) + 20000 return fmt.Sprintf(":%d", rndPort) } From 5646b6e772cc2f5bc9d44a87445dacf0fe1ebcd6 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Nov 2024 18:13:58 +0100 Subject: [PATCH 232/243] fix: Fix race condition in logger --- pkg/logger/logger.go | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pkg/logger/logger.go b/pkg/logger/logger.go index 9f5d694a..9f21f34f 100644 --- a/pkg/logger/logger.go +++ b/pkg/logger/logger.go @@ -7,6 +7,7 @@ import ( "log" "os" "strings" + "sync" "github.com/pion/logging" "golang.org/x/time/rate" @@ -42,6 +43,7 @@ type LeveledLoggerFactory struct { DefaultLogLevel logging.LogLevel ScopeLevels map[string]logging.LogLevel Loggers map[string]*RateLimitedLogger + lock sync.RWMutex } // NewLoggerFactory sets up a scoped logger for STUNner. @@ -67,6 +69,9 @@ func (f *LeveledLoggerFactory) NewLogger(scope string) logging.LeveledLogger { // SetLevel sets the loglevel. func (f *LeveledLoggerFactory) SetLevel(levelSpec string) { + f.lock.Lock() + defer f.lock.Unlock() + levels := strings.Split(levelSpec, ",") for _, s := range levels { scopedLevel := strings.SplitN(s, ":", 2) @@ -88,7 +93,6 @@ func (f *LeveledLoggerFactory) SetLevel(levelSpec string) { f.DefaultLogLevel = l continue } - f.ScopeLevels[scope] = l } @@ -109,6 +113,9 @@ func (f *LeveledLoggerFactory) SetLevel(levelSpec string) { // GetLevel gets the loglevel for the given scope. func (f *LeveledLoggerFactory) GetLevel(scope string) string { + f.lock.RLock() + defer f.lock.RUnlock() + logLevel := f.DefaultLogLevel scopeLevel, found := f.ScopeLevels[scope] if found { @@ -178,6 +185,9 @@ func NewRateLimitedLoggerForScope(scope string, level logging.LogLevel, writer i // newLogger knows how to emit rate-limited loggers. func (f *LeveledLoggerFactory) newLogger(scope string, limit rate.Limit, burst int) *RateLimitedLogger { + f.lock.Lock() + defer f.lock.Unlock() + logger, found := f.Loggers[scope] if found { return logger From 79f1b329d18faf5ad3502aefecbe3667b2a7f34e Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Fri, 15 Nov 2024 18:17:16 +0100 Subject: [PATCH 233/243] feature: Implement ICE tester server and client --- Makefile | 6 +- go.mod | 18 +- go.sum | 42 +++- pkg/icetester/dialer.go | 369 +++++++++++++++++++++++++++++++ pkg/icetester/icetester_test.go | 248 +++++++++++++++++++++ pkg/icetester/listener.go | 374 ++++++++++++++++++++++++++++++++ pkg/icetester/types.go | 49 +++++ 7 files changed, 1091 insertions(+), 15 deletions(-) create mode 100644 pkg/icetester/dialer.go create mode 100644 pkg/icetester/icetester_test.go create mode 100644 pkg/icetester/listener.go create mode 100644 pkg/icetester/types.go diff --git a/Makefile b/Makefile index ba85bf1e..3da34954 100644 --- a/Makefile +++ b/Makefile @@ -46,6 +46,6 @@ build-bin: .PHONY: clean clean: - echo 'Use "make generate` to autogenerate server code' > pkg/server/server.go - echo 'Use "make generate` to autogenerate client code' > pkg/client/client.go - echo 'Use "make generate` to autogenerate client code' > pkg/types/types.go + # echo 'Use "make generate" to autogenerate server code' > pkg/server/server.go + # echo 'Use "make generate" to autogenerate client code' > pkg/client/client.go + # echo 'Use "make generate" to autogenerate client code' > pkg/types/types.go diff --git a/go.mod b/go.mod index 58c96984..295026c6 100644 --- a/go.mod +++ b/go.mod @@ -14,10 +14,12 @@ require ( github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.1 github.com/oapi-codegen/runtime v1.1.1 - github.com/pion/dtls/v3 v3.0.2 + github.com/pion/datachannel v1.5.9 + github.com/pion/dtls/v3 v3.0.3 github.com/pion/logging v0.2.2 github.com/pion/transport/v3 v3.0.7 github.com/pion/turn/v4 v4.0.0 + github.com/pion/webrtc/v4 v4.0.1 github.com/prometheus/client_golang v1.20.4 github.com/spf13/cobra v1.8.1 github.com/spf13/pflag v1.0.5 @@ -76,7 +78,15 @@ require ( github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect github.com/perimeterx/marshmallow v1.1.5 // indirect github.com/peterbourgon/diskv v2.0.1+incompatible // indirect + github.com/pion/ice/v4 v4.0.2 // indirect + github.com/pion/interceptor v0.1.37 // indirect + github.com/pion/mdns/v2 v2.0.7 // indirect github.com/pion/randutil v0.1.0 // indirect + github.com/pion/rtcp v1.2.14 // indirect + github.com/pion/rtp v1.8.9 // indirect + github.com/pion/sctp v1.8.33 // indirect + github.com/pion/sdp/v3 v3.0.9 // indirect + github.com/pion/srtp/v3 v3.0.4 // indirect github.com/pion/stun/v3 v3.0.0 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect @@ -88,13 +98,13 @@ require ( go.opentelemetry.io/otel/trace v1.31.0 // indirect go.starlark.net v0.0.0-20240123142251-f86470692795 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/crypto v0.27.0 // indirect + golang.org/x/crypto v0.28.0 // indirect golang.org/x/mod v0.17.0 // indirect golang.org/x/net v0.29.0 // indirect golang.org/x/oauth2 v0.23.0 // indirect golang.org/x/sync v0.8.0 // indirect - golang.org/x/term v0.24.0 // indirect - golang.org/x/text v0.18.0 // indirect + golang.org/x/term v0.25.0 // indirect + golang.org/x/text v0.19.0 // indirect golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect google.golang.org/protobuf v1.35.1 // indirect gopkg.in/evanphx/json-patch.v5 v5.9.0 // indirect diff --git a/go.sum b/go.sum index bc4e93a5..343cffa8 100644 --- a/go.sum +++ b/go.sum @@ -118,6 +118,7 @@ github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= github.com/oapi-codegen/runtime v1.1.1 h1:EXLHh0DXIJnWhdRPN2w4MXAzFyE4CskzhNLUmtpMYro= github.com/oapi-codegen/runtime v1.1.1/go.mod h1:SK9X900oXmPWilYR5/WKPzt3Kqxn/uS/+lbpREv+eCg= +github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= github.com/onsi/ginkgo/v2 v2.13.0 h1:0jY9lJquiL8fcf3M4LAXN5aMlS/b2BV86HFFPCPMgE4= github.com/onsi/ginkgo/v2 v2.13.0/go.mod h1:TE309ZR8s5FsKKpuB1YAQYBzCaAfUgatB/xlT/ETL/o= github.com/onsi/gomega v1.29.0 h1:KIA/t2t5UBzoirT4H9tsML45GEbo3ouUnBHsCfD2tVg= @@ -126,18 +127,38 @@ github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= -github.com/pion/dtls/v3 v3.0.2 h1:425DEeJ/jfuTTghhUDW0GtYZYIwwMtnKKJNMcWccTX0= -github.com/pion/dtls/v3 v3.0.2/go.mod h1:dfIXcFkKoujDQ+jtd8M6RgqKK3DuaUilm3YatAbGp5k= +github.com/pion/datachannel v1.5.9 h1:LpIWAOYPyDrXtU+BW7X0Yt/vGtYxtXQ8ql7dFfYUVZA= +github.com/pion/datachannel v1.5.9/go.mod h1:kDUuk4CU4Uxp82NH4LQZbISULkX/HtzKa4P7ldf9izE= +github.com/pion/dtls/v3 v3.0.3 h1:j5ajZbQwff7Z8k3pE3S+rQ4STvKvXUdKsi/07ka+OWM= +github.com/pion/dtls/v3 v3.0.3/go.mod h1:weOTUyIV4z0bQaVzKe8kpaP17+us3yAuiQsEAG1STMU= +github.com/pion/ice/v4 v4.0.2 h1:1JhBRX8iQLi0+TfcavTjPjI6GO41MFn4CeTBX+Y9h5s= +github.com/pion/ice/v4 v4.0.2/go.mod h1:DCdqyzgtsDNYN6/3U8044j3U7qsJ9KFJC92VnOWHvXg= +github.com/pion/interceptor v0.1.37 h1:aRA8Zpab/wE7/c0O3fh1PqY0AJI3fCSEM5lRWJVorwI= +github.com/pion/interceptor v0.1.37/go.mod h1:JzxbJ4umVTlZAf+/utHzNesY8tmRkM2lVmkS82TTj8Y= github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= +github.com/pion/mdns/v2 v2.0.7 h1:c9kM8ewCgjslaAmicYMFQIde2H9/lrZpjBkN8VwoVtM= +github.com/pion/mdns/v2 v2.0.7/go.mod h1:vAdSYNAT0Jy3Ru0zl2YiW3Rm/fJCwIeM0nToenfOJKA= github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA= github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8= +github.com/pion/rtcp v1.2.14 h1:KCkGV3vJ+4DAJmvP0vaQShsb0xkRfWkO540Gy102KyE= +github.com/pion/rtcp v1.2.14/go.mod h1:sn6qjxvnwyAkkPzPULIbVqSKI5Dv54Rv7VG0kNxh9L4= +github.com/pion/rtp v1.8.9 h1:E2HX740TZKaqdcPmf4pw6ZZuG8u5RlMMt+l3dxeu6Wk= +github.com/pion/rtp v1.8.9/go.mod h1:pBGHaFt/yW7bf1jjWAoUjpSNoDnw98KTMg+jWWvziqU= +github.com/pion/sctp v1.8.33 h1:dSE4wX6uTJBcNm8+YlMg7lw1wqyKHggsP5uKbdj+NZw= +github.com/pion/sctp v1.8.33/go.mod h1:beTnqSzewI53KWoG3nqB282oDMGrhNxBdb+JZnkCwRM= +github.com/pion/sdp/v3 v3.0.9 h1:pX++dCHoHUwq43kuwf3PyJfHlwIj4hXA7Vrifiq0IJY= +github.com/pion/sdp/v3 v3.0.9/go.mod h1:B5xmvENq5IXJimIO4zfp6LAe1fD9N+kFv+V/1lOdz8M= +github.com/pion/srtp/v3 v3.0.4 h1:2Z6vDVxzrX3UHEgrUyIGM4rRouoC7v+NiF1IHtp9B5M= +github.com/pion/srtp/v3 v3.0.4/go.mod h1:1Jx3FwDoxpRaTh1oRV8A/6G1BnFL+QI82eK4ms8EEJQ= github.com/pion/stun/v3 v3.0.0 h1:4h1gwhWLWuZWOJIJR9s2ferRO+W3zA/b6ijOI6mKzUw= github.com/pion/stun/v3 v3.0.0/go.mod h1:HvCN8txt8mwi4FBvS3EmDghW6aQJ24T+y+1TKjB5jyU= github.com/pion/transport/v3 v3.0.7 h1:iRbMH05BzSNwhILHoBoAPxoB9xQgOaJk+591KC9P1o0= github.com/pion/transport/v3 v3.0.7/go.mod h1:YleKiTZ4vqNxVwh77Z0zytYi7rXHl7j6uPLGhhz9rwo= github.com/pion/turn/v4 v4.0.0 h1:qxplo3Rxa9Yg1xXDxxH8xaqcyGUtbHYw4QSCvmFWvhM= github.com/pion/turn/v4 v4.0.0/go.mod h1:MuPDkm15nYSklKpN8vWJ9W2M0PlyQZqYt1McGuxG7mA= +github.com/pion/webrtc/v4 v4.0.1 h1:6Unwc6JzoTsjxetcAIoWH81RUM4K5dBc1BbJGcF9WVE= +github.com/pion/webrtc/v4 v4.0.1/go.mod h1:SfNn8CcFxR6OUVjLXVslAQ3a3994JhyE3Hw1jAuqEto= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= @@ -161,10 +182,15 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY= github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU= @@ -198,8 +224,8 @@ go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A= -golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70= +golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw= +golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -225,12 +251,12 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo= golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/term v0.24.0 h1:Mh5cbb+Zk2hqqXNO7S1iTjEphVL+jb8ZWaqh/g+JWkM= -golang.org/x/term v0.24.0/go.mod h1:lOBK/LVxemqiMij05LGJ0tzNr8xlmwBRJ81PX6wVLH8= +golang.org/x/term v0.25.0 h1:WtHI/ltw4NvSUig5KARz9h521QvRC8RmF/cuYqifU24= +golang.org/x/term v0.25.0/go.mod h1:RPyXicDX+6vLxogjjRxjgD2TKtmAO6NZBsBRfrOLu7M= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224= -golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= +golang.org/x/text v0.19.0 h1:kTxAhCbGbxhK0IwgSKiMO5awPoDQ0RpfiVYBfK860YM= +golang.org/x/text v0.19.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= diff --git a/pkg/icetester/dialer.go b/pkg/icetester/dialer.go new file mode 100644 index 00000000..dc977a87 --- /dev/null +++ b/pkg/icetester/dialer.go @@ -0,0 +1,369 @@ +package icetester + +import ( + "context" + "encoding/json" + "fmt" + "net" + "net/http" + "net/url" + "strings" + "time" + + "github.com/gorilla/websocket" + "github.com/pion/datachannel" + "github.com/pion/logging" + "github.com/pion/webrtc/v4" +) + +var ( + // Send pings to the CDS server with this period. Must be less than PongWait. + PingPeriod = 5 * time.Second + + // Time allowed to read the next pong message from the CDS server. + PongWait = 8 * time.Second + + // Time allowed to write a message to the CDS server. + WriteWait = 2 * time.Second + + // Period for retrying failed CDS connections. + RetryPeriod = 1 * time.Second +) + +var _ net.Conn = &dialerConn{} + +type Dialer struct { + iceConfig webrtc.Configuration + api *webrtc.API + logger logging.LoggerFactory + log logging.LeveledLogger +} + +func NewDialer(iceConfig webrtc.Configuration, logger logging.LoggerFactory) *Dialer { + e := webrtc.SettingEngine{} + e.DetachDataChannels() + + return &Dialer{ + iceConfig: iceConfig, + api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), + logger: logger, + log: logger.NewLogger("tester-dialer"), + } +} + +func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) { + signalingServerURI, err := getURI(addr) + if err != nil { + return nil, fmt.Errorf("Failed to parse server address %q: %w", addr, err) + } + rawWsConn, _, err := websocket.DefaultDialer.DialContext(ctx, signalingServerURI.String(), makeHeader(signalingServerURI)) + if err != nil { + return nil, fmt.Errorf("Failed to connect to singlaing server at %q: %w", + signalingServerURI.String(), err) + } + // wrap with a locker to prevent concurrent writes + wsConn := &ThreadSafeWriter{Conn: rawWsConn} + + conn := &dialerConn{ + wsConn: wsConn, + log: d.logger.NewLogger(fmt.Sprintf("tester-client-%s-%s", + wsConn.LocalAddr(), wsConn.RemoteAddr())), + } + + conn.log.Debugf("Signaling connection successfully opened to tester server at %s", signalingServerURI.String()) + + // Start pinger thread: lifetime is the connection's lifetime, stops when connCtx is + // canceled + pingTicker := time.NewTicker(PingPeriod) + pingerCtx, pingerCancel := context.WithCancel(context.Background()) + conn.pingerCancel = pingerCancel + go func() { + defer pingTicker.Stop() + + for { + select { + case <-pingTicker.C: + wsConn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck + if err := wsConn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { + conn.log.Errorf("Could not ping tester signaling server at %q: %s", + wsConn.RemoteAddr(), err.Error()) + return + } + case <-pingerCtx.Done(): + conn.log.Tracef("Closing pinger thread WS connection %s-%s", + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) + return + } + } + }() + + conn.log.Tracef("Creating new PeerConnection for WS connection %s-%s", + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) + peerConn, err := d.api.NewPeerConnection(d.iceConfig) + if err != nil { + return nil, fmt.Errorf("Failed to create a PeerConnection: %w", err) + } + conn.peerConn = peerConn + + // Trickle ICE: emit server candidates to client, errors are not fatal + peerConn.OnICECandidate(func(i *webrtc.ICECandidate) { + if i == nil { + return + } + // When serializing a candidate use ToJSON, othwerwise json.Marshal will result in + // errors around `sdpMid` + candidateString, err := json.Marshal(i.ToJSON()) + if err != nil { + conn.log.Errorf("Failed to marshal candidate to json: %v", err) + return + } + + conn.log.Infof("Local candidate: %s", candidateString) + + if writeErr := wsConn.WriteJSON(&Message{ + Type: MessageTypeIceCandidate, + Data: string(candidateString), + }); writeErr != nil { + conn.log.Errorf("Failed to write JSON: %v", writeErr) + return + } + }) + + // If PeerConnection is closed, close the client + peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { + conn.log.Infof("Connection State has changed: %s", p) + if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { + conn.Close() //nolint + } + }) + + // the next pong must arrive within the PongWait period + wsConn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + // reinit the deadline when we get a pong + wsConn.SetPongHandler(func(string) error { + // a.Tracef("Got PONG from server %q", url) + wsConn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck + return nil + }) + + // Register data channel creation handling + connCh := make(chan any, 1) + defer close(connCh) + errCh := make(chan error) + + peerConn.OnDataChannel(func(dataChannel *webrtc.DataChannel) { + conn.log.Tracef("New DataChannel %s %d for WS connection %s-%s", dataChannel.Label(), + dataChannel.ID(), wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) + + // Register channel opening handling + dataChannel.OnOpen(func() { + conn.log.Debugf("Data channel '%s'-'%d' open for WS connection %s-%s", + dataChannel.Label(), dataChannel.ID(), + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) + + raw, err := dataChannel.Detach() + if err != nil { + errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) + return + } + conn.dataConn = raw + + connCh <- struct{}{} + }) + }) + + candidateCache := []webrtc.ICECandidateInit{} + // Start signaling client speaker: lifetime is the connection's lifetime, stops when wsConn + // is closed + go func() { + defer close(errCh) + + message := &Message{} + for { + // ping-pong deadline misses will end up being caught here as a read beyond + // the deadline + msgType, raw, err := wsConn.ReadMessage() + if err != nil { + errCh <- err + return + } + + // Decoding errors are not fatal + if msgType != websocket.TextMessage { + conn.log.Errorf("Unexpected message type (code: %d) from client %q", + msgType, wsConn.RemoteAddr().String()) + continue + } + + if err := json.Unmarshal(raw, &message); err != nil { + conn.log.Errorf("Failed to unmarshal json to message: %v", err) + continue + } + + conn.log.Tracef("Got signaling message WS connection %s-%s: %v", + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String(), message) + + switch message.Type { + case MessageTypeIceCandidate: + candidate := webrtc.ICECandidateInit{} + if err := json.Unmarshal([]byte(message.Data), &candidate); err != nil { + conn.log.Errorf("Failed to unmarshal json to candidate: %v", err) + continue + } + + conn.log.Infof("Remote candidate: %s", message.Data) + + if peerConn.RemoteDescription() == nil { + // cannot set candidates yet: cache candidate + candidateCache = append(candidateCache, candidate) + } else if err := peerConn.AddICECandidate(candidate); err != nil { + errCh <- fmt.Errorf("Failed to add ICE candidate: %w", err) + } + + case MessageTypeOffer: + offer := webrtc.SessionDescription{} + if err := json.Unmarshal([]byte(message.Data), &offer); err != nil { + conn.log.Errorf("Failed to unmarshal json to Offer: %v", err) + continue + } + + conn.log.Debugf("Got Offer on WS connection %s-%s: %v", + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String(), + offer) + + if err := peerConn.SetRemoteDescription(offer); err != nil { + errCh <- fmt.Errorf("Failed to set remote description: %w", err) + return + } + + // flush candidate cache + for _, candidate := range candidateCache { + if err := peerConn.AddICECandidate(candidate); err != nil { + errCh <- fmt.Errorf("Failed to add cached ICE candidate: %w", err) + } + } + candidateCache = []webrtc.ICECandidateInit{} + + // Create an offer to send to the other process + conn.log.Tracef("Creating Answer WS connection %s-%s", + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) + answer, err := peerConn.CreateAnswer(nil) + if err != nil { + errCh <- fmt.Errorf("Failed to create Answer: %w", err) + return + } + + // Sets the LocalDescription, and starts our UDP listeners + // Note: this will start the gathering of ICE candidates + if err = peerConn.SetLocalDescription(answer); err != nil { + errCh <- fmt.Errorf("Failed to set local description: %w", err) + return + } + + payload, err := json.Marshal(answer) + if err != nil { + errCh <- fmt.Errorf("Failed to JSON encode Answer: %v", err) + return + } + + conn.log.Debugf("Sending Answer on WS connection %s-%s: %v", + wsConn.LocalAddr().String(), wsConn.RemoteAddr().String(), + answer) + + if writeErr := wsConn.WriteJSON(&Message{ + Type: MessageTypeAnswer, + Data: string(payload), + }); writeErr != nil { + errCh <- fmt.Errorf("Failed to write Answer: %v", writeErr) + } + default: + conn.log.Errorf("unknown message: %+v", message) + } + } + }() + + select { + case <-connCh: + d.log.Infof("Creating new connection %s", conn.String()) + return conn, nil + case err := <-errCh: + conn.Close() + return nil, err + } +} + +type dialerConn struct { + pingerCancel context.CancelFunc + wsConn *ThreadSafeWriter + peerConn *webrtc.PeerConnection + dataConn datachannel.ReadWriteCloser + closed bool + log logging.LeveledLogger +} + +func (c *dialerConn) Close() error { + c.log.Tracef("Closing tester client connection %s", c.String()) + + if c.closed { + return nil + } + c.closed = true + + // Close the pinger thread + c.pingerCancel() + + // Close the WebSocket signaling connection: closes the signaling thread + c.wsConn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck + c.wsConn.Close() + + // Close the peerconnection + if err := c.peerConn.Close(); err != nil { + return fmt.Errorf("Failed to close PeerConnection: %w", err) + } + + return nil +} + +func (c *dialerConn) Read(b []byte) (int, error) { + return c.dataConn.Read(b) +} + +func (c *dialerConn) Write(b []byte) (int, error) { + return c.dataConn.Write(b) +} + +// TODO: implement +func (c *dialerConn) LocalAddr() net.Addr { return nil } +func (c *dialerConn) RemoteAddr() net.Addr { return nil } +func (c *dialerConn) SetDeadline(t time.Time) error { return nil } +func (c *dialerConn) SetReadDeadline(t time.Time) error { return nil } +func (c *dialerConn) SetWriteDeadline(t time.Time) error { return nil } + +// String returns a unique identifier for the connection based on the underlying signaling connection. +func (c *dialerConn) String() string { + return fmt.Sprintf("%s-%s", c.wsConn.LocalAddr(), c.wsConn.RemoteAddr()) +} + +// creates an origin header +func makeHeader(url *url.URL) http.Header { + header := http.Header{} + origin := *url + origin.Scheme = "http" + origin.Path = "" + header.Set("origin", origin.String()) + return header +} + +func getURI(addr string) (*url.URL, error) { + if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") && + !strings.HasPrefix(addr, "ws://") && !strings.HasPrefix(addr, "wss://") { + addr = "ws://" + addr + } + + url, err := url.Parse(addr) + if err != nil { + return nil, err + } + url.Path = signalingServerPath + return url, nil +} diff --git a/pkg/icetester/icetester_test.go b/pkg/icetester/icetester_test.go new file mode 100644 index 00000000..0c8614e0 --- /dev/null +++ b/pkg/icetester/icetester_test.go @@ -0,0 +1,248 @@ +package icetester + +import ( + "context" + "errors" + "net" + "net/http" + "sync" + "testing" + "time" + + "github.com/pion/logging" + "github.com/pion/webrtc/v4" + "github.com/stretchr/testify/assert" + + slogger "github.com/l7mp/stunner/pkg/logger" +) + +var ( + testerLogLevel = "all:WARN" + // testerLogLevel = "all:TRACE" + // testerLogLevel = "all:INFO" + addr = "localhost:12345" + timeout = 5 * time.Second + interval = 50 * time.Millisecond + logger logging.LoggerFactory = slogger.NewLoggerFactory(testerLogLevel) + log logging.LeveledLogger = logger.NewLogger("test") +) + +// func init() { +// // setup a fast pinger so that we get a timely error notification +// PingPeriod = 500 * time.Millisecond +// PongWait = 800 * time.Millisecond +// WriteWait = 200 * time.Millisecond +// RetryPeriod = 250 * time.Millisecond +// } + +func echoTest(t *testing.T, conn net.Conn, content string) { + t.Helper() + + n, err := conn.Write([]byte(content)) + assert.NoError(t, err) + assert.Equal(t, len(content), n) + + buf := make([]byte, 2048) + n, err = conn.Read(buf) + assert.NoError(t, err) + assert.Equal(t, content, string(buf[:n])) +} + +var testerTestCases = []struct { + name string + tester func(t *testing.T, ctx context.Context, l *Listener) +}{ + { + name: "Basic connectivity", + tester: func(t *testing.T, ctx context.Context, l *Listener) { + log.Debug("Creating dialer") + d := NewDialer(webrtc.Configuration{}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(ctx, addr) + assert.NoError(t, err) + + log.Debug("Echo test round 1") + echoTest(t, clientConn, "test1") + log.Debug("Echo test round 2") + echoTest(t, clientConn, "test2") + + assert.NoError(t, clientConn.Close(), "client conn close") + }, + }, { + name: "Closing dialer does not close client connection", + tester: func(t *testing.T, serverCtx context.Context, l *Listener) { + // a new context for the dialer + dialerCtx, dialerCancel := context.WithCancel(context.Background()) + + log.Debug("Creating dialer") + d := NewDialer(webrtc.Configuration{}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(dialerCtx, addr) + assert.NoError(t, err) + + log.Debug("Echo test round 1") + echoTest(t, clientConn, "test1") + + log.Debug("Closing dialer") + dialerCancel() + + log.Debug("Echo test round 2") + echoTest(t, clientConn, "test2") + }, + }, { + name: "Client side close closes server", + tester: func(t *testing.T, serverCtx context.Context, l *Listener) { + log.Debug("Creating dialer") + d := NewDialer(webrtc.Configuration{}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(serverCtx, addr) + assert.NoError(t, err) + + assert.Eventually(t, func() bool { return l.activeConns == 1 }, timeout, interval) + + log.Debug("Closing client connection") + assert.NoError(t, clientConn.Close()) + + // should close the server conn too + assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + }, + }, { + name: "Server side close closes client", + tester: func(t *testing.T, serverCtx context.Context, l *Listener) { + clientCtx, clientCancel := context.WithCancel(context.Background()) + defer clientCancel() + + log.Debug("Creating dialer") + d := NewDialer(webrtc.Configuration{}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(clientCtx, addr) + assert.NoError(t, err) + + assert.Eventually(t, func() bool { return l.activeConns == 1 }, timeout, interval) + + log.Debug("Closing server connections") + for _, lConn := range l.Conns() { + // log.Infof("------------ %s", lConn.String()) + assert.NoError(t, lConn.Close()) + } + + log.Info("&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&") + + assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + + // should close the client conn too + assert.Eventually(t, func() bool { return clientConn.(*dialerConn).closed == true }, timeout, interval) + }, + }, { + name: "Multiple connections", + tester: func(t *testing.T, ctx context.Context, l *Listener) { + log.Debug("Creating dialer") + d := NewDialer(webrtc.Configuration{}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing: creating 5 connections") + var wg sync.WaitGroup + wg.Add(5) + connChan := make(chan net.Conn, 5) + for i := 0; i < 5; i++ { + go func() { + defer wg.Done() + + clientConn, err := d.DialContext(ctx, addr) + assert.NoError(t, err) + + log.Debug("Echo test round 1") + echoTest(t, clientConn, "test1111") + + log.Debug("Echo test round 2") + echoTest(t, clientConn, "test2222") + + connChan <- clientConn + }() + } + + wg.Wait() + close(connChan) + + assert.Eventually(t, func() bool { return l.activeConns == 5 }, timeout, interval) + + for c := range connChan { + c.Close() + } + + assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + }, + }, +} + +func TestTesterConn(t *testing.T) { + for _, c := range testerTestCases { + var l *Listener + + t.Run(c.name, func(t *testing.T) { + log.Infof("--------------------- %s ----------------------", c.name) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + log.Debug("Creating listener") + listener, err := NewListener(addr, webrtc.Configuration{}, logger) + assert.NoError(t, err) + l = listener + assert.NotNil(t, l) + + log.Debug("Creating echo services") + go func() { + for { + conn, err := l.Accept() + if err != nil { + return + } + + log.Debug("Accepting server connection") + + // responder + go func() { + buf := make([]byte, 100) + for { + n, err := conn.Read(buf) + if err != nil { + return + } + + _, err = conn.Write(buf[:n]) + assert.NoError(t, err) + } + }() + + // closer + go func() { + <-ctx.Done() + + if err := conn.Close(); err != nil && !errors.Is(err, net.ErrClosed) && + !errors.Is(err, http.ErrServerClosed) { + t.Error("server conn close") + } + + // close listener + l.Close() //nolint + }() + } + }() + + c.tester(t, ctx, l) + }) + + log.Debug("Waiting for connections to close") + if l != nil { + assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + } + } +} diff --git a/pkg/icetester/listener.go b/pkg/icetester/listener.go new file mode 100644 index 00000000..19a0f528 --- /dev/null +++ b/pkg/icetester/listener.go @@ -0,0 +1,374 @@ +package icetester + +import ( + "encoding/json" + "errors" + "fmt" + "net" + "net/http" + "sync" + "time" + + "github.com/gorilla/mux" + "github.com/gorilla/websocket" + "github.com/pion/datachannel" + "github.com/pion/logging" + "github.com/pion/webrtc/v4" +) + +const ( + messageSize = 2048 + signalingServerPath = "websocket" +) + +var _ net.Listener = &Listener{} +var _ net.Conn = &listenerConn{} + +// nolint +var ( + upgrader = websocket.Upgrader{ + CheckOrigin: func(r *http.Request) bool { return true }, + } + serverDataPeriod = 100 * time.Millisecond // 10 pkt/sec +) + +type Listener struct { + *http.Server + addr string + iceConfig webrtc.Configuration + errCh chan error + connCh chan *listenerConn + api *webrtc.API + conns map[string]*listenerConn + lock sync.Mutex + activeConns int + logger logging.LoggerFactory + log logging.LeveledLogger +} + +func NewListener(addr string, iceConfig webrtc.Configuration, logger logging.LoggerFactory) (*Listener, error) { + e := webrtc.SettingEngine{} + e.DetachDataChannels() + l := &Listener{ + addr: addr, + iceConfig: iceConfig, + api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), + errCh: make(chan error, 5), + connCh: make(chan *listenerConn, 128), + conns: map[string]*listenerConn{}, + logger: logger, + log: logger.NewLogger("tester-listener"), + } + + router := mux.NewRouter() + router.HandleFunc("/"+signalingServerPath, l.ServeHTTP) + l.Server = &http.Server{Addr: addr, Handler: router} + + c, err := net.Listen("tcp", addr) + if err != nil { + return nil, fmt.Errorf("WS tester signaling server on %s: %w", addr, err) + } + + go func() { + defer close(l.errCh) + defer close(l.connCh) + + if err := l.Server.Serve(c); err != nil { + l.errCh <- err + } + }() + + return l, nil +} + +func (l *Listener) Accept() (net.Conn, error) { + l.log.Trace("Accept: waiting for new connection") + + select { + case err := <-l.errCh: + return nil, err + case conn := <-l.connCh: + l.log.Infof("Accepting connection for WS connection %s-%s", + conn.wsConn.RemoteAddr(), conn.wsConn.LocalAddr()) + + l.lock.Lock() + l.activeConns += 1 + l.conns[conn.String()] = conn + l.lock.Unlock() + + return conn, nil + } +} + +func (l *Listener) Close() error { + l.log.Tracef("Closing tester server listener at address %s", l.addr) + defer l.Server.Close() + + select { + case err := <-l.errCh: + return err + default: + return nil + } +} + +func (l *Listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { + // upgrade to webSocket + upgrader := websocket.Upgrader{ + ReadBufferSize: 1024, + WriteBufferSize: 1024, + } + + rawWsConn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + l.errCh <- fmt.Errorf("Failed to upgrade WebSocket connection: %w", err) + return + } + + wsConn := &ThreadSafeWriter{Conn: rawWsConn} + + l.log.Tracef("Tester listener received new connection from client %s", wsConn.RemoteAddr()) + connLog := l.logger.NewLogger(fmt.Sprintf("tester-lconn-%s-%s", + wsConn.RemoteAddr(), wsConn.LocalAddr())) + + wsConn.SetPingHandler(func(string) error { + return wsConn.WriteMessage(websocket.PongMessage, []byte("keepalive")) + }) + + connLog.Trace("Creating PeerConnection") + peerConn, err := l.api.NewPeerConnection(l.iceConfig) + if err != nil { + l.errCh <- fmt.Errorf("Failed to create a PeerConnection: %w", err) + return + } + + connLog.Tracef("Creating DataChannel") + dataChannel, err := peerConn.CreateDataChannel("data", nil) + if err != nil { + l.errCh <- fmt.Errorf("Failed to create DataChannel: %w", err) + return + } + + peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { + connLog.Debugf("Connection State has changed: %s", p.String()) + if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { + l.errCh <- errors.New("ICE connection closed prematurely") + return + } + }) + + dataChannel.OnOpen(func() { + connLog.Tracef("Data channel '%s'-'%d' open", dataChannel.Label(), dataChannel.ID()) + + raw, dErr := dataChannel.Detach() + if dErr != nil { + l.errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) + return + } + + l.log.Infof("Creating new connection for WS connection %s-%s", + wsConn.RemoteAddr(), wsConn.LocalAddr()) + + conn := &listenerConn{ + listener: l, + wsConn: wsConn, + peerConn: peerConn, + dataChan: dataChannel, + dataConn: raw, + log: connLog, + } + + l.connCh <- conn + + // redo ICE state change callback to react to the peerconnection going away + peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { + conn.Close() //nolint:errcheck + }) + }) + + peerConn.OnICECandidate(func(i *webrtc.ICECandidate) { + if i == nil { + return + } + + // When serializing a candidate use ToJSON, othwerwise json.Marshal will result in + // errors around `sdpMid` + candidateString, err := json.Marshal(i.ToJSON()) + if err != nil { + connLog.Errorf("Failed to marshal candidate to json: %v", err) + return + } + + l.log.Debugf("Sending candidate: %s", candidateString) + + if writeErr := wsConn.WriteJSON(&Message{ + Type: MessageTypeIceCandidate, + Data: string(candidateString), + }); writeErr != nil { + connLog.Errorf("Failed to write JSON: %v", writeErr) + } + }) + + connLog.Trace("Creating Offer for client") + offer, err := peerConn.CreateOffer(nil) + if err != nil { + l.errCh <- fmt.Errorf("Failed to create Offer: %w", err) + return + } + + // Note: this will start the gathering of ICE candidates + connLog.Tracef("Setting local descrition (Offer)") + if err = peerConn.SetLocalDescription(offer); err != nil { + l.errCh <- fmt.Errorf("Failed to set local description: %w", err) + return + } + + payload, err := json.Marshal(offer) + if err != nil { + l.errCh <- fmt.Errorf("Failed to JSON encode Offer: %v", err) + return + } + + connLog.Tracef("Sending Offer: %s", offer) + if writeErr := wsConn.WriteJSON(&Message{ + Type: MessageTypeOffer, + Data: string(payload), + }); writeErr != nil { + l.errCh <- fmt.Errorf("Failed to write Offer: %v", writeErr) + return + } + + message := &Message{} + for { + _, raw, err := wsConn.ReadMessage() + if err != nil { + l.errCh <- fmt.Errorf("Failed to read message: %v", err) + return + } + + connLog.Tracef("Got message: %s", raw) + + if err := json.Unmarshal(raw, &message); err != nil { + l.errCh <- fmt.Errorf("Failed to unmarshal json to message: %v", err) + return + } + + switch message.Type { + case MessageTypeIceCandidate: + candidate := webrtc.ICECandidateInit{} + if err := json.Unmarshal([]byte(message.Data), &candidate); err != nil { + l.errCh <- fmt.Errorf("Failed to unmarshal json to candidate: %v", err) + return + } + + connLog.Debugf("Got ICE candidate: %v", candidate) + + if err := peerConn.AddICECandidate(candidate); err != nil { + l.errCh <- fmt.Errorf("Failed to add ICE candidate: %v", err) + return + } + + case MessageTypeAnswer: + answer := webrtc.SessionDescription{} + if err := json.Unmarshal([]byte(message.Data), &answer); err != nil { + l.errCh <- fmt.Errorf("Failed to unmarshal json to answer: %v", err) + return + } + + connLog.Debugf("Got Answer: %v", answer) + + connLog.Tracef("Setting remote descrition (Answer)") + if err := peerConn.SetRemoteDescription(answer); err != nil { + l.errCh <- fmt.Errorf("Failed to set remote description: %v", err) + return + } + default: + l.log.Errorf("unknown message: %+v", message) + } + } +} + +func (_ *Listener) Addr() net.Addr { + //TODO + return nil +} + +func (l *Listener) Conns() []*listenerConn { + l.lock.Lock() + defer l.lock.Unlock() + ret := []*listenerConn{} + for _, c := range l.conns { + ret = append(ret, c) + } + return ret +} + +type listenerConn struct { + listener *Listener + wsConn *ThreadSafeWriter + peerConn *webrtc.PeerConnection + dataChan *webrtc.DataChannel + dataConn datachannel.ReadWriteCloser + closed bool + log logging.LeveledLogger +} + +func (c *listenerConn) Close() error { + c.log.Tracef("Closing tester server listener connection %s", c.String()) + + if c.closed { + return nil + } + c.closed = true + + // Close the datachannel + var err error + if c.dataChan.ReadyState() == webrtc.DataChannelStateOpen { + if err = c.dataConn.Close(); err != nil { + c.log.Debugf("Error closing DataChannel for client %s: %s", + c.wsConn.RemoteAddr().String(), err.Error()) + } + } + + // Close the peer connection too + err = c.peerConn.Close() + if err != nil { + c.log.Debugf("Error closing PeerConnection for client %s: %s", + c.wsConn.RemoteAddr().String(), err.Error()) + } + + // Close the websocket, this will exit the peerconnection and the http handler + err = c.wsConn.Close() + if err != nil { + c.log.Debugf("Error closing WS connection for client %s: %s", + c.wsConn.RemoteAddr().String(), err.Error()) + } + + c.listener.lock.Lock() + c.listener.activeConns -= 1 + delete(c.listener.conns, c.String()) + c.listener.lock.Unlock() + + // Return the last error + return err +} + +func (c *listenerConn) Read(b []byte) (int, error) { + return c.dataConn.Read(b) +} + +func (c *listenerConn) Write(b []byte) (int, error) { + return c.dataConn.Write(b) +} + +// TODO: implement +func (c *listenerConn) LocalAddr() net.Addr { return nil } +func (c *listenerConn) RemoteAddr() net.Addr { return nil } +func (c *listenerConn) SetDeadline(t time.Time) error { return nil } +func (c *listenerConn) SetReadDeadline(t time.Time) error { return nil } +func (c *listenerConn) SetWriteDeadline(t time.Time) error { return nil } + +// String returns a unique identifier for the connection based on the underlying signaling connection. +func (c *listenerConn) String() string { + return fmt.Sprintf("%s-%s", c.wsConn.RemoteAddr(), c.wsConn.LocalAddr()) +} diff --git a/pkg/icetester/types.go b/pkg/icetester/types.go new file mode 100644 index 00000000..495a7174 --- /dev/null +++ b/pkg/icetester/types.go @@ -0,0 +1,49 @@ +package icetester + +import ( + "sync" + + "github.com/gorilla/websocket" +) + +type MessageType int + +const ( + MessageTypeUnknown MessageType = iota + MessageTypeIceCandidate + MessageTypeOffer + MessageTypeAnswer + MessageTypeClose +) + +type Message struct { + Type MessageType `json:"type"` + Data string `json:"data"` +} + +// ThreadSafeWriter represents a client WebSocket connection. An added lock guards the underlying connection +// from concurrent write to websocket connection errors. +type ThreadSafeWriter struct { + *websocket.Conn + readLock, writeLock sync.Mutex // for writemessage +} + +// WriteMessage writes a message to the client connection with proper locking. +func (c *ThreadSafeWriter) WriteMessage(messageType int, data []byte) error { + c.writeLock.Lock() + defer c.writeLock.Unlock() + return c.Conn.WriteMessage(messageType, data) +} + +func (c *ThreadSafeWriter) WriteJSON(v any) error { + c.writeLock.Lock() + defer c.writeLock.Unlock() + return c.Conn.WriteJSON(v) +} + +// ReadMessage reads a message from the client connection with proper locking. +func (c *ThreadSafeWriter) ReadMessage() (int, []byte, error) { + c.readLock.Lock() + defer c.readLock.Unlock() + return c.Conn.ReadMessage() +} From 3451422db0fd0ce64377893d9f34649f656eb705 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Sun, 17 Nov 2024 21:10:32 +0100 Subject: [PATCH 234/243] refactor: Reimplement ICE tester on top of standard WHIP --- pkg/icetester/dialer.go | 369 ----------------- pkg/icetester/listener.go | 374 ------------------ pkg/icetester/types.go | 49 --- pkg/whipconn/config.go | 38 ++ pkg/whipconn/dialer.go | 237 +++++++++++ pkg/whipconn/listener.go | 352 +++++++++++++++++ .../whipconn_test.go} | 121 +++--- 7 files changed, 698 insertions(+), 842 deletions(-) delete mode 100644 pkg/icetester/dialer.go delete mode 100644 pkg/icetester/listener.go delete mode 100644 pkg/icetester/types.go create mode 100644 pkg/whipconn/config.go create mode 100644 pkg/whipconn/dialer.go create mode 100644 pkg/whipconn/listener.go rename pkg/{icetester/icetester_test.go => whipconn/whipconn_test.go} (62%) diff --git a/pkg/icetester/dialer.go b/pkg/icetester/dialer.go deleted file mode 100644 index dc977a87..00000000 --- a/pkg/icetester/dialer.go +++ /dev/null @@ -1,369 +0,0 @@ -package icetester - -import ( - "context" - "encoding/json" - "fmt" - "net" - "net/http" - "net/url" - "strings" - "time" - - "github.com/gorilla/websocket" - "github.com/pion/datachannel" - "github.com/pion/logging" - "github.com/pion/webrtc/v4" -) - -var ( - // Send pings to the CDS server with this period. Must be less than PongWait. - PingPeriod = 5 * time.Second - - // Time allowed to read the next pong message from the CDS server. - PongWait = 8 * time.Second - - // Time allowed to write a message to the CDS server. - WriteWait = 2 * time.Second - - // Period for retrying failed CDS connections. - RetryPeriod = 1 * time.Second -) - -var _ net.Conn = &dialerConn{} - -type Dialer struct { - iceConfig webrtc.Configuration - api *webrtc.API - logger logging.LoggerFactory - log logging.LeveledLogger -} - -func NewDialer(iceConfig webrtc.Configuration, logger logging.LoggerFactory) *Dialer { - e := webrtc.SettingEngine{} - e.DetachDataChannels() - - return &Dialer{ - iceConfig: iceConfig, - api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), - logger: logger, - log: logger.NewLogger("tester-dialer"), - } -} - -func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) { - signalingServerURI, err := getURI(addr) - if err != nil { - return nil, fmt.Errorf("Failed to parse server address %q: %w", addr, err) - } - rawWsConn, _, err := websocket.DefaultDialer.DialContext(ctx, signalingServerURI.String(), makeHeader(signalingServerURI)) - if err != nil { - return nil, fmt.Errorf("Failed to connect to singlaing server at %q: %w", - signalingServerURI.String(), err) - } - // wrap with a locker to prevent concurrent writes - wsConn := &ThreadSafeWriter{Conn: rawWsConn} - - conn := &dialerConn{ - wsConn: wsConn, - log: d.logger.NewLogger(fmt.Sprintf("tester-client-%s-%s", - wsConn.LocalAddr(), wsConn.RemoteAddr())), - } - - conn.log.Debugf("Signaling connection successfully opened to tester server at %s", signalingServerURI.String()) - - // Start pinger thread: lifetime is the connection's lifetime, stops when connCtx is - // canceled - pingTicker := time.NewTicker(PingPeriod) - pingerCtx, pingerCancel := context.WithCancel(context.Background()) - conn.pingerCancel = pingerCancel - go func() { - defer pingTicker.Stop() - - for { - select { - case <-pingTicker.C: - wsConn.SetWriteDeadline(time.Now().Add(WriteWait)) //nolint:errcheck - if err := wsConn.WriteMessage(websocket.PingMessage, []byte("keepalive")); err != nil { - conn.log.Errorf("Could not ping tester signaling server at %q: %s", - wsConn.RemoteAddr(), err.Error()) - return - } - case <-pingerCtx.Done(): - conn.log.Tracef("Closing pinger thread WS connection %s-%s", - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) - return - } - } - }() - - conn.log.Tracef("Creating new PeerConnection for WS connection %s-%s", - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) - peerConn, err := d.api.NewPeerConnection(d.iceConfig) - if err != nil { - return nil, fmt.Errorf("Failed to create a PeerConnection: %w", err) - } - conn.peerConn = peerConn - - // Trickle ICE: emit server candidates to client, errors are not fatal - peerConn.OnICECandidate(func(i *webrtc.ICECandidate) { - if i == nil { - return - } - // When serializing a candidate use ToJSON, othwerwise json.Marshal will result in - // errors around `sdpMid` - candidateString, err := json.Marshal(i.ToJSON()) - if err != nil { - conn.log.Errorf("Failed to marshal candidate to json: %v", err) - return - } - - conn.log.Infof("Local candidate: %s", candidateString) - - if writeErr := wsConn.WriteJSON(&Message{ - Type: MessageTypeIceCandidate, - Data: string(candidateString), - }); writeErr != nil { - conn.log.Errorf("Failed to write JSON: %v", writeErr) - return - } - }) - - // If PeerConnection is closed, close the client - peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { - conn.log.Infof("Connection State has changed: %s", p) - if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { - conn.Close() //nolint - } - }) - - // the next pong must arrive within the PongWait period - wsConn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck - // reinit the deadline when we get a pong - wsConn.SetPongHandler(func(string) error { - // a.Tracef("Got PONG from server %q", url) - wsConn.SetReadDeadline(time.Now().Add(PongWait)) //nolint:errcheck - return nil - }) - - // Register data channel creation handling - connCh := make(chan any, 1) - defer close(connCh) - errCh := make(chan error) - - peerConn.OnDataChannel(func(dataChannel *webrtc.DataChannel) { - conn.log.Tracef("New DataChannel %s %d for WS connection %s-%s", dataChannel.Label(), - dataChannel.ID(), wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) - - // Register channel opening handling - dataChannel.OnOpen(func() { - conn.log.Debugf("Data channel '%s'-'%d' open for WS connection %s-%s", - dataChannel.Label(), dataChannel.ID(), - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) - - raw, err := dataChannel.Detach() - if err != nil { - errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) - return - } - conn.dataConn = raw - - connCh <- struct{}{} - }) - }) - - candidateCache := []webrtc.ICECandidateInit{} - // Start signaling client speaker: lifetime is the connection's lifetime, stops when wsConn - // is closed - go func() { - defer close(errCh) - - message := &Message{} - for { - // ping-pong deadline misses will end up being caught here as a read beyond - // the deadline - msgType, raw, err := wsConn.ReadMessage() - if err != nil { - errCh <- err - return - } - - // Decoding errors are not fatal - if msgType != websocket.TextMessage { - conn.log.Errorf("Unexpected message type (code: %d) from client %q", - msgType, wsConn.RemoteAddr().String()) - continue - } - - if err := json.Unmarshal(raw, &message); err != nil { - conn.log.Errorf("Failed to unmarshal json to message: %v", err) - continue - } - - conn.log.Tracef("Got signaling message WS connection %s-%s: %v", - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String(), message) - - switch message.Type { - case MessageTypeIceCandidate: - candidate := webrtc.ICECandidateInit{} - if err := json.Unmarshal([]byte(message.Data), &candidate); err != nil { - conn.log.Errorf("Failed to unmarshal json to candidate: %v", err) - continue - } - - conn.log.Infof("Remote candidate: %s", message.Data) - - if peerConn.RemoteDescription() == nil { - // cannot set candidates yet: cache candidate - candidateCache = append(candidateCache, candidate) - } else if err := peerConn.AddICECandidate(candidate); err != nil { - errCh <- fmt.Errorf("Failed to add ICE candidate: %w", err) - } - - case MessageTypeOffer: - offer := webrtc.SessionDescription{} - if err := json.Unmarshal([]byte(message.Data), &offer); err != nil { - conn.log.Errorf("Failed to unmarshal json to Offer: %v", err) - continue - } - - conn.log.Debugf("Got Offer on WS connection %s-%s: %v", - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String(), - offer) - - if err := peerConn.SetRemoteDescription(offer); err != nil { - errCh <- fmt.Errorf("Failed to set remote description: %w", err) - return - } - - // flush candidate cache - for _, candidate := range candidateCache { - if err := peerConn.AddICECandidate(candidate); err != nil { - errCh <- fmt.Errorf("Failed to add cached ICE candidate: %w", err) - } - } - candidateCache = []webrtc.ICECandidateInit{} - - // Create an offer to send to the other process - conn.log.Tracef("Creating Answer WS connection %s-%s", - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String()) - answer, err := peerConn.CreateAnswer(nil) - if err != nil { - errCh <- fmt.Errorf("Failed to create Answer: %w", err) - return - } - - // Sets the LocalDescription, and starts our UDP listeners - // Note: this will start the gathering of ICE candidates - if err = peerConn.SetLocalDescription(answer); err != nil { - errCh <- fmt.Errorf("Failed to set local description: %w", err) - return - } - - payload, err := json.Marshal(answer) - if err != nil { - errCh <- fmt.Errorf("Failed to JSON encode Answer: %v", err) - return - } - - conn.log.Debugf("Sending Answer on WS connection %s-%s: %v", - wsConn.LocalAddr().String(), wsConn.RemoteAddr().String(), - answer) - - if writeErr := wsConn.WriteJSON(&Message{ - Type: MessageTypeAnswer, - Data: string(payload), - }); writeErr != nil { - errCh <- fmt.Errorf("Failed to write Answer: %v", writeErr) - } - default: - conn.log.Errorf("unknown message: %+v", message) - } - } - }() - - select { - case <-connCh: - d.log.Infof("Creating new connection %s", conn.String()) - return conn, nil - case err := <-errCh: - conn.Close() - return nil, err - } -} - -type dialerConn struct { - pingerCancel context.CancelFunc - wsConn *ThreadSafeWriter - peerConn *webrtc.PeerConnection - dataConn datachannel.ReadWriteCloser - closed bool - log logging.LeveledLogger -} - -func (c *dialerConn) Close() error { - c.log.Tracef("Closing tester client connection %s", c.String()) - - if c.closed { - return nil - } - c.closed = true - - // Close the pinger thread - c.pingerCancel() - - // Close the WebSocket signaling connection: closes the signaling thread - c.wsConn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck - c.wsConn.Close() - - // Close the peerconnection - if err := c.peerConn.Close(); err != nil { - return fmt.Errorf("Failed to close PeerConnection: %w", err) - } - - return nil -} - -func (c *dialerConn) Read(b []byte) (int, error) { - return c.dataConn.Read(b) -} - -func (c *dialerConn) Write(b []byte) (int, error) { - return c.dataConn.Write(b) -} - -// TODO: implement -func (c *dialerConn) LocalAddr() net.Addr { return nil } -func (c *dialerConn) RemoteAddr() net.Addr { return nil } -func (c *dialerConn) SetDeadline(t time.Time) error { return nil } -func (c *dialerConn) SetReadDeadline(t time.Time) error { return nil } -func (c *dialerConn) SetWriteDeadline(t time.Time) error { return nil } - -// String returns a unique identifier for the connection based on the underlying signaling connection. -func (c *dialerConn) String() string { - return fmt.Sprintf("%s-%s", c.wsConn.LocalAddr(), c.wsConn.RemoteAddr()) -} - -// creates an origin header -func makeHeader(url *url.URL) http.Header { - header := http.Header{} - origin := *url - origin.Scheme = "http" - origin.Path = "" - header.Set("origin", origin.String()) - return header -} - -func getURI(addr string) (*url.URL, error) { - if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") && - !strings.HasPrefix(addr, "ws://") && !strings.HasPrefix(addr, "wss://") { - addr = "ws://" + addr - } - - url, err := url.Parse(addr) - if err != nil { - return nil, err - } - url.Path = signalingServerPath - return url, nil -} diff --git a/pkg/icetester/listener.go b/pkg/icetester/listener.go deleted file mode 100644 index 19a0f528..00000000 --- a/pkg/icetester/listener.go +++ /dev/null @@ -1,374 +0,0 @@ -package icetester - -import ( - "encoding/json" - "errors" - "fmt" - "net" - "net/http" - "sync" - "time" - - "github.com/gorilla/mux" - "github.com/gorilla/websocket" - "github.com/pion/datachannel" - "github.com/pion/logging" - "github.com/pion/webrtc/v4" -) - -const ( - messageSize = 2048 - signalingServerPath = "websocket" -) - -var _ net.Listener = &Listener{} -var _ net.Conn = &listenerConn{} - -// nolint -var ( - upgrader = websocket.Upgrader{ - CheckOrigin: func(r *http.Request) bool { return true }, - } - serverDataPeriod = 100 * time.Millisecond // 10 pkt/sec -) - -type Listener struct { - *http.Server - addr string - iceConfig webrtc.Configuration - errCh chan error - connCh chan *listenerConn - api *webrtc.API - conns map[string]*listenerConn - lock sync.Mutex - activeConns int - logger logging.LoggerFactory - log logging.LeveledLogger -} - -func NewListener(addr string, iceConfig webrtc.Configuration, logger logging.LoggerFactory) (*Listener, error) { - e := webrtc.SettingEngine{} - e.DetachDataChannels() - l := &Listener{ - addr: addr, - iceConfig: iceConfig, - api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), - errCh: make(chan error, 5), - connCh: make(chan *listenerConn, 128), - conns: map[string]*listenerConn{}, - logger: logger, - log: logger.NewLogger("tester-listener"), - } - - router := mux.NewRouter() - router.HandleFunc("/"+signalingServerPath, l.ServeHTTP) - l.Server = &http.Server{Addr: addr, Handler: router} - - c, err := net.Listen("tcp", addr) - if err != nil { - return nil, fmt.Errorf("WS tester signaling server on %s: %w", addr, err) - } - - go func() { - defer close(l.errCh) - defer close(l.connCh) - - if err := l.Server.Serve(c); err != nil { - l.errCh <- err - } - }() - - return l, nil -} - -func (l *Listener) Accept() (net.Conn, error) { - l.log.Trace("Accept: waiting for new connection") - - select { - case err := <-l.errCh: - return nil, err - case conn := <-l.connCh: - l.log.Infof("Accepting connection for WS connection %s-%s", - conn.wsConn.RemoteAddr(), conn.wsConn.LocalAddr()) - - l.lock.Lock() - l.activeConns += 1 - l.conns[conn.String()] = conn - l.lock.Unlock() - - return conn, nil - } -} - -func (l *Listener) Close() error { - l.log.Tracef("Closing tester server listener at address %s", l.addr) - defer l.Server.Close() - - select { - case err := <-l.errCh: - return err - default: - return nil - } -} - -func (l *Listener) ServeHTTP(w http.ResponseWriter, r *http.Request) { - // upgrade to webSocket - upgrader := websocket.Upgrader{ - ReadBufferSize: 1024, - WriteBufferSize: 1024, - } - - rawWsConn, err := upgrader.Upgrade(w, r, nil) - if err != nil { - l.errCh <- fmt.Errorf("Failed to upgrade WebSocket connection: %w", err) - return - } - - wsConn := &ThreadSafeWriter{Conn: rawWsConn} - - l.log.Tracef("Tester listener received new connection from client %s", wsConn.RemoteAddr()) - connLog := l.logger.NewLogger(fmt.Sprintf("tester-lconn-%s-%s", - wsConn.RemoteAddr(), wsConn.LocalAddr())) - - wsConn.SetPingHandler(func(string) error { - return wsConn.WriteMessage(websocket.PongMessage, []byte("keepalive")) - }) - - connLog.Trace("Creating PeerConnection") - peerConn, err := l.api.NewPeerConnection(l.iceConfig) - if err != nil { - l.errCh <- fmt.Errorf("Failed to create a PeerConnection: %w", err) - return - } - - connLog.Tracef("Creating DataChannel") - dataChannel, err := peerConn.CreateDataChannel("data", nil) - if err != nil { - l.errCh <- fmt.Errorf("Failed to create DataChannel: %w", err) - return - } - - peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { - connLog.Debugf("Connection State has changed: %s", p.String()) - if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { - l.errCh <- errors.New("ICE connection closed prematurely") - return - } - }) - - dataChannel.OnOpen(func() { - connLog.Tracef("Data channel '%s'-'%d' open", dataChannel.Label(), dataChannel.ID()) - - raw, dErr := dataChannel.Detach() - if dErr != nil { - l.errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) - return - } - - l.log.Infof("Creating new connection for WS connection %s-%s", - wsConn.RemoteAddr(), wsConn.LocalAddr()) - - conn := &listenerConn{ - listener: l, - wsConn: wsConn, - peerConn: peerConn, - dataChan: dataChannel, - dataConn: raw, - log: connLog, - } - - l.connCh <- conn - - // redo ICE state change callback to react to the peerconnection going away - peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { - conn.Close() //nolint:errcheck - }) - }) - - peerConn.OnICECandidate(func(i *webrtc.ICECandidate) { - if i == nil { - return - } - - // When serializing a candidate use ToJSON, othwerwise json.Marshal will result in - // errors around `sdpMid` - candidateString, err := json.Marshal(i.ToJSON()) - if err != nil { - connLog.Errorf("Failed to marshal candidate to json: %v", err) - return - } - - l.log.Debugf("Sending candidate: %s", candidateString) - - if writeErr := wsConn.WriteJSON(&Message{ - Type: MessageTypeIceCandidate, - Data: string(candidateString), - }); writeErr != nil { - connLog.Errorf("Failed to write JSON: %v", writeErr) - } - }) - - connLog.Trace("Creating Offer for client") - offer, err := peerConn.CreateOffer(nil) - if err != nil { - l.errCh <- fmt.Errorf("Failed to create Offer: %w", err) - return - } - - // Note: this will start the gathering of ICE candidates - connLog.Tracef("Setting local descrition (Offer)") - if err = peerConn.SetLocalDescription(offer); err != nil { - l.errCh <- fmt.Errorf("Failed to set local description: %w", err) - return - } - - payload, err := json.Marshal(offer) - if err != nil { - l.errCh <- fmt.Errorf("Failed to JSON encode Offer: %v", err) - return - } - - connLog.Tracef("Sending Offer: %s", offer) - if writeErr := wsConn.WriteJSON(&Message{ - Type: MessageTypeOffer, - Data: string(payload), - }); writeErr != nil { - l.errCh <- fmt.Errorf("Failed to write Offer: %v", writeErr) - return - } - - message := &Message{} - for { - _, raw, err := wsConn.ReadMessage() - if err != nil { - l.errCh <- fmt.Errorf("Failed to read message: %v", err) - return - } - - connLog.Tracef("Got message: %s", raw) - - if err := json.Unmarshal(raw, &message); err != nil { - l.errCh <- fmt.Errorf("Failed to unmarshal json to message: %v", err) - return - } - - switch message.Type { - case MessageTypeIceCandidate: - candidate := webrtc.ICECandidateInit{} - if err := json.Unmarshal([]byte(message.Data), &candidate); err != nil { - l.errCh <- fmt.Errorf("Failed to unmarshal json to candidate: %v", err) - return - } - - connLog.Debugf("Got ICE candidate: %v", candidate) - - if err := peerConn.AddICECandidate(candidate); err != nil { - l.errCh <- fmt.Errorf("Failed to add ICE candidate: %v", err) - return - } - - case MessageTypeAnswer: - answer := webrtc.SessionDescription{} - if err := json.Unmarshal([]byte(message.Data), &answer); err != nil { - l.errCh <- fmt.Errorf("Failed to unmarshal json to answer: %v", err) - return - } - - connLog.Debugf("Got Answer: %v", answer) - - connLog.Tracef("Setting remote descrition (Answer)") - if err := peerConn.SetRemoteDescription(answer); err != nil { - l.errCh <- fmt.Errorf("Failed to set remote description: %v", err) - return - } - default: - l.log.Errorf("unknown message: %+v", message) - } - } -} - -func (_ *Listener) Addr() net.Addr { - //TODO - return nil -} - -func (l *Listener) Conns() []*listenerConn { - l.lock.Lock() - defer l.lock.Unlock() - ret := []*listenerConn{} - for _, c := range l.conns { - ret = append(ret, c) - } - return ret -} - -type listenerConn struct { - listener *Listener - wsConn *ThreadSafeWriter - peerConn *webrtc.PeerConnection - dataChan *webrtc.DataChannel - dataConn datachannel.ReadWriteCloser - closed bool - log logging.LeveledLogger -} - -func (c *listenerConn) Close() error { - c.log.Tracef("Closing tester server listener connection %s", c.String()) - - if c.closed { - return nil - } - c.closed = true - - // Close the datachannel - var err error - if c.dataChan.ReadyState() == webrtc.DataChannelStateOpen { - if err = c.dataConn.Close(); err != nil { - c.log.Debugf("Error closing DataChannel for client %s: %s", - c.wsConn.RemoteAddr().String(), err.Error()) - } - } - - // Close the peer connection too - err = c.peerConn.Close() - if err != nil { - c.log.Debugf("Error closing PeerConnection for client %s: %s", - c.wsConn.RemoteAddr().String(), err.Error()) - } - - // Close the websocket, this will exit the peerconnection and the http handler - err = c.wsConn.Close() - if err != nil { - c.log.Debugf("Error closing WS connection for client %s: %s", - c.wsConn.RemoteAddr().String(), err.Error()) - } - - c.listener.lock.Lock() - c.listener.activeConns -= 1 - delete(c.listener.conns, c.String()) - c.listener.lock.Unlock() - - // Return the last error - return err -} - -func (c *listenerConn) Read(b []byte) (int, error) { - return c.dataConn.Read(b) -} - -func (c *listenerConn) Write(b []byte) (int, error) { - return c.dataConn.Write(b) -} - -// TODO: implement -func (c *listenerConn) LocalAddr() net.Addr { return nil } -func (c *listenerConn) RemoteAddr() net.Addr { return nil } -func (c *listenerConn) SetDeadline(t time.Time) error { return nil } -func (c *listenerConn) SetReadDeadline(t time.Time) error { return nil } -func (c *listenerConn) SetWriteDeadline(t time.Time) error { return nil } - -// String returns a unique identifier for the connection based on the underlying signaling connection. -func (c *listenerConn) String() string { - return fmt.Sprintf("%s-%s", c.wsConn.RemoteAddr(), c.wsConn.LocalAddr()) -} diff --git a/pkg/icetester/types.go b/pkg/icetester/types.go deleted file mode 100644 index 495a7174..00000000 --- a/pkg/icetester/types.go +++ /dev/null @@ -1,49 +0,0 @@ -package icetester - -import ( - "sync" - - "github.com/gorilla/websocket" -) - -type MessageType int - -const ( - MessageTypeUnknown MessageType = iota - MessageTypeIceCandidate - MessageTypeOffer - MessageTypeAnswer - MessageTypeClose -) - -type Message struct { - Type MessageType `json:"type"` - Data string `json:"data"` -} - -// ThreadSafeWriter represents a client WebSocket connection. An added lock guards the underlying connection -// from concurrent write to websocket connection errors. -type ThreadSafeWriter struct { - *websocket.Conn - readLock, writeLock sync.Mutex // for writemessage -} - -// WriteMessage writes a message to the client connection with proper locking. -func (c *ThreadSafeWriter) WriteMessage(messageType int, data []byte) error { - c.writeLock.Lock() - defer c.writeLock.Unlock() - return c.Conn.WriteMessage(messageType, data) -} - -func (c *ThreadSafeWriter) WriteJSON(v any) error { - c.writeLock.Lock() - defer c.writeLock.Unlock() - return c.Conn.WriteJSON(v) -} - -// ReadMessage reads a message from the client connection with proper locking. -func (c *ThreadSafeWriter) ReadMessage() (int, []byte, error) { - c.readLock.Lock() - defer c.readLock.Unlock() - return c.Conn.ReadMessage() -} diff --git a/pkg/whipconn/config.go b/pkg/whipconn/config.go new file mode 100644 index 00000000..50ffb2ed --- /dev/null +++ b/pkg/whipconn/config.go @@ -0,0 +1,38 @@ +package whipconn + +import ( + "fmt" + "hash/fnv" + "net/url" + + "github.com/pion/webrtc/v4" +) + +const ( + messageSize = 2048 + whipEndpoint = "/whip" +) + +type Config struct { + ICEServers []webrtc.ICEServer + ICETransportPolicy webrtc.ICETransportPolicy + Token, Endpoint string +} + +func makeURL(addr, endpoint string) *url.URL { + return &url.URL{ + Scheme: "http", + Host: addr, + Path: endpoint, + } +} + +func resourceHash(s string) string { + h := fnv.New32a() + h.Write([]byte(s)) + return fmt.Sprintf("/%d", h.Sum32()) +} + +func makeResourceURL(endpoint, id string) string { + return endpoint + id +} diff --git a/pkg/whipconn/dialer.go b/pkg/whipconn/dialer.go new file mode 100644 index 00000000..ca2252d0 --- /dev/null +++ b/pkg/whipconn/dialer.go @@ -0,0 +1,237 @@ +// A simple WHIP client and server that implement a stream connection abstraction on top of a WebRTC data channel published via WHIP. +// Client adopted from https://github.com/ggarber/whip-go/ +package whipconn + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "net" + "net/http" + "time" + + "github.com/pion/datachannel" + "github.com/pion/logging" + "github.com/pion/webrtc/v4" +) + +var _ net.Conn = &dialerConn{} + +type Dialer struct { + config Config + api *webrtc.API + logger logging.LoggerFactory + log, connLog logging.LeveledLogger +} + +func NewDialer(config Config, logger logging.LoggerFactory) *Dialer { + e := webrtc.SettingEngine{} + e.DetachDataChannels() + + if config.Endpoint == "" { + config.Endpoint = whipEndpoint + } + + return &Dialer{ + config: config, + api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), + logger: logger, + log: logger.NewLogger("whip-dialer"), + connLog: logger.NewLogger("whip-conn"), + } +} + +func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) { + peerConn, err := d.api.NewPeerConnection(webrtc.Configuration{ + ICEServers: d.config.ICEServers, + ICETransportPolicy: d.config.ICETransportPolicy, + }) + if err != nil { + return nil, fmt.Errorf("Failed to create PeerConnection: %w", err) + } + + d.log.Trace("Creating DataChannel") + dataChannel, err := peerConn.CreateDataChannel("whipconn", nil) + if err != nil { + return nil, fmt.Errorf("Failed to create DataChannel: %w", err) + } + + conn := &dialerConn{ + dialer: d, + addr: addr, + peerConn: peerConn, + log: d.connLog, + } + + connCh := make(chan any, 1) + defer close(connCh) + errCh := make(chan error) + defer close(errCh) + + // Register channel opening handling + dataChannel.OnOpen(func() { + conn.log.Debugf("Creating new connection in data channel %s-%d", + dataChannel.Label(), dataChannel.ID()) + + raw, err := dataChannel.Detach() + if err != nil { + errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) + } + conn.dataConn = raw + + connCh <- struct{}{} + }) + + // If PeerConnection is closed, close the client + peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { + conn.log.Infof("Connection state has changed: %s", p) + if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { + conn.Close() //nolint + } + }) + + offer, err := peerConn.CreateOffer(nil) + if err != nil { + conn.Close() //nolint + return nil, fmt.Errorf("Failed to create offer: %w", err) + } + + err = peerConn.SetLocalDescription(offer) + if err != nil { + conn.Close() //nolint + return nil, fmt.Errorf("Failed to set local SDP (Offer): %w", err) + } + + // Block until ICE Gathering is complete, disabling trickle ICE + // we do this because we only can exchange one signaling message + // in a production application you should exchange ICE Candidates via OnICECandidate + gatherComplete := webrtc.GatheringCompletePromise(peerConn) + <-gatherComplete + + d.log.Debugf("ICE gathering complete: %s", peerConn.LocalDescription().SDP) + + var sdp = []byte(peerConn.LocalDescription().SDP) + + req, err := http.NewRequestWithContext(ctx, http.MethodPost, + makeURL(addr, d.config.Endpoint).String(), bytes.NewBuffer(sdp)) + if err != nil { + conn.Close() //nolint + return nil, fmt.Errorf("Unexpected error building HTTP request: %w", err) + } + + req.Header.Add("Content-Type", "application/sdp") + if d.config.Token != "" { + req.Header.Add("Authorization", "Bearer "+d.config.Token) + } + + resp, err := http.DefaultClient.Do(req) + if err != nil { + conn.Close() //nolint + return nil, fmt.Errorf("Failed to POST WHIP request: %w", err) + } + + d.log.Tracef("Received POST response with status code: %d", resp.StatusCode) + + if resp.StatusCode != 201 { + conn.Close() //nolint + return nil, fmt.Errorf("POST request returned invalid status: %d", resp.StatusCode) + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + conn.Close() //nolint + return nil, fmt.Errorf("Failed to read HTTP response body: %w", err) + } + defer resp.Body.Close() + + resourceId := resp.Header.Get("Location") + if resourceId == "" { + conn.Close() //nolint + return nil, errors.New("Empty resource id in POST response") + } + conn.resourceId = resourceId + + answer := webrtc.SessionDescription{} + answer.Type = webrtc.SDPTypeAnswer + answer.SDP = string(body) + + err = peerConn.SetRemoteDescription(answer) + if err != nil { + conn.Close() //nolint + return nil, fmt.Errorf("Failed to set remote SDP (Answer): %w", err) + } + + // Waiting for the connection or errors surfaced from the callbacks + select { + case <-connCh: + d.log.Infof("Creating new connection %s", conn.String()) + return conn, nil + case err := <-errCh: + conn.Close() + return nil, err + case <-ctx.Done(): + conn.Close() + return nil, ctx.Err() + } +} + +type dialerConn struct { + dialer *Dialer + addr string + peerConn *webrtc.PeerConnection + dataConn datachannel.ReadWriteCloser + resourceId string + closed bool + log logging.LeveledLogger +} + +func (c *dialerConn) Close() error { + if c.closed { + return nil + } + c.closed = true + + c.log.Trace("Closing WHIP client connection") + + uri := makeURL(c.addr, makeResourceURL(c.dialer.config.Endpoint, c.resourceId)) + req, err := http.NewRequest("DELETE", uri.String(), nil) + if err != nil { + return fmt.Errorf("Unexpected error building http request: %w", err) + } + if c.dialer.config.Token != "" { + req.Header.Add("Authorization", "Bearer "+c.dialer.config.Token) + } + + if _, err = http.DefaultClient.Do(req); err != nil { + return fmt.Errorf("Failed WHIP DELETE request: %w", err) + } + + // Close the peerconnection + if err := c.peerConn.Close(); err != nil { + return fmt.Errorf("Failed to close PeerConnection: %w", err) + } + + return nil +} + +func (c *dialerConn) Read(b []byte) (int, error) { + return c.dataConn.Read(b) +} + +func (c *dialerConn) Write(b []byte) (int, error) { + return c.dataConn.Write(b) +} + +// TODO: implement +func (c *dialerConn) LocalAddr() net.Addr { return nil } +func (c *dialerConn) RemoteAddr() net.Addr { return nil } +func (c *dialerConn) SetDeadline(t time.Time) error { return nil } +func (c *dialerConn) SetReadDeadline(t time.Time) error { return nil } +func (c *dialerConn) SetWriteDeadline(t time.Time) error { return nil } + +// String returns a unique identifier for the connection based on the underlying signaling connection. +func (c *dialerConn) String() string { + return c.resourceId +} diff --git a/pkg/whipconn/listener.go b/pkg/whipconn/listener.go new file mode 100644 index 00000000..c9169a1f --- /dev/null +++ b/pkg/whipconn/listener.go @@ -0,0 +1,352 @@ +// A simple WHIP client and server that implement a stream connection abstraction on top of a WebRTC data channel published via WHIP. +// Adopted from https://github.com/pion/webrtc/tree/master/examples/whip-whep +package whipconn + +import ( + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "sync" + "time" + + "github.com/pion/datachannel" + "github.com/pion/logging" + "github.com/pion/webrtc/v4" +) + +var _ net.Listener = &Listener{} +var _ net.Conn = &listenerConn{} + +type Listener struct { + api *webrtc.API + config Config + addr string + server *http.Server + errCh chan error + connCh chan *listenerConn + conns map[string]*listenerConn + lock sync.Mutex + logger logging.LoggerFactory + log, connLog logging.LeveledLogger + closed bool +} + +func NewListener(addr string, config Config, logger logging.LoggerFactory) (*Listener, error) { + e := webrtc.SettingEngine{} + e.DetachDataChannels() + + if config.Endpoint == "" { + config.Endpoint = whipEndpoint + } + + l := &Listener{ + addr: addr, + config: config, + api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), + errCh: make(chan error, 5), + connCh: make(chan *listenerConn, 128), + conns: map[string]*listenerConn{}, + logger: logger, + log: logger.NewLogger("whip-listener"), + connLog: logger.NewLogger("whip-conn"), + } + + mux := http.NewServeMux() + mux.HandleFunc("/config", l.configHandler) + + deletePatternWithoutSlash := fmt.Sprintf("DELETE %s/{resourceId}", config.Endpoint) + mux.HandleFunc(deletePatternWithoutSlash, l.whipDeleteHandler) + deletePatternWithSlash := fmt.Sprintf("DELETE %s/{resourceId}/{$}", config.Endpoint) + mux.HandleFunc(deletePatternWithSlash, l.whipDeleteHandler) + + requestPatternWithoutSlash := fmt.Sprintf("POST %s", config.Endpoint) + mux.HandleFunc(requestPatternWithoutSlash, l.whipRequestHandler) + requestPatternWithSlash := fmt.Sprintf("POST %s/{$}", config.Endpoint) + mux.HandleFunc(requestPatternWithSlash, l.whipRequestHandler) + + c, err := net.Listen("tcp", addr) + if err != nil { + return nil, fmt.Errorf("Cannot open WHIP server socket on %s: %w", addr, err) + } + l.server = &http.Server{Addr: addr, Handler: mux} + go func() { + defer close(l.errCh) + defer close(l.connCh) + + if err := l.server.Serve(c); err != nil { + l.errCh <- err + } + }() + + return l, nil +} + +func (l *Listener) Accept() (net.Conn, error) { + l.log.Trace("Accept: waiting for new connection") + + select { + case err := <-l.errCh: + l.log.Tracef("Accept error: %s", err.Error()) + return nil, err + case conn := <-l.connCh: + l.log.Info("Accept: New connection") + + l.lock.Lock() + l.conns[conn.String()] = conn + l.lock.Unlock() + + return conn, nil + } +} + +func (l *Listener) Close() error { + if l.closed { + return nil + } + l.closed = true + + l.log.Tracef("Closing WHIP server listener at address %s", l.addr) + + // Send an error to stop any Accept() calls running + l.errCh <- net.ErrClosed + + return l.server.Close() +} + +func (l *Listener) configHandler(w http.ResponseWriter, r *http.Request) { + l.log.Tracef("New Config request from client %s", r.RemoteAddr) + + var config Config + if err := json.NewDecoder(r.Body).Decode(&config); err != nil { + l.log.Errorf("Failed to decode config request %w from client %s: %s", + r.RemoteAddr, config, err.Error()) + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + l.config = config +} + +func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { + l.log.Tracef("New WHIP POST request from client %s", r.RemoteAddr) + + // Check bearer token + if l.config.Token != "" { + if token := r.Header.Get("Authorization"); token != "Bearer "+l.config.Token { + err := fmt.Errorf("Unauthorized WHIP request from client %s", r.RemoteAddr) + l.errCh <- err + http.Error(w, err.Error(), http.StatusUnauthorized) + return + } + } + + if ctype := r.Header.Get("Content-Type"); ctype != "application/sdp" { + err := fmt.Errorf("Invalid WHIP request from client %s, expected Content-Type=application/sdp", + r.RemoteAddr) + l.errCh <- err + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + // Read the offer from HTTP Request + offer, err := io.ReadAll(r.Body) + defer r.Body.Close() + if err != nil { + l.errCh <- fmt.Errorf("Failed to read WHIP request body: %w", err) + return + } + + conn := &listenerConn{ + listener: l, + log: l.connLog, + } + + conn.log.Tracef("Creating PeerConnection for client %s", r.RemoteAddr) + peerConn, err := l.api.NewPeerConnection(webrtc.Configuration{ + ICEServers: l.config.ICEServers, + ICETransportPolicy: l.config.ICETransportPolicy, + }) + if err != nil { + l.errCh <- fmt.Errorf("Failed to create a PeerConnection: %w", err) + return + } + conn.peerConn = peerConn + + peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { + conn.log.Debugf("PeerConnection state for client %s has changed: %s", r.RemoteAddr, p.String()) + if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { + conn.Close() // nolint:errcheck + return + } + }) + + peerConn.OnDataChannel(func(dataChannel *webrtc.DataChannel) { + conn.log.Tracef("New data channel %s-%d", dataChannel.Label(), dataChannel.ID()) + + dataChannel.OnOpen(func() { + conn.log.Tracef("Data channel %s-%d open for client %s", dataChannel.Label(), + dataChannel.ID(), r.RemoteAddr) + conn.dataChan = dataChannel + + raw, dErr := dataChannel.Detach() + if dErr != nil { + l.errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) + return + } + conn.dataConn = raw + conn.started = true + + l.log.Infof("Creating new connection for client %s", r.RemoteAddr) + l.connCh <- conn + }) + }) + + conn.log.Tracef("Set remote SDP (Offer) for client %s", r.RemoteAddr) + if err := peerConn.SetRemoteDescription(webrtc.SessionDescription{ + Type: webrtc.SDPTypeOffer, + SDP: string(offer), + }); err != nil { + l.errCh <- fmt.Errorf("Failed to set remote SDP (Offer): %w", err) + return + } + + // Create channel that is blocked until ICE Gathering is complete + gatherComplete := webrtc.GatheringCompletePromise(peerConn) + + // Create answer + answer, err := peerConn.CreateAnswer(nil) + if err != nil { + l.errCh <- fmt.Errorf("Failed to create SDP (Answer): %w", err) + return + } else if err = peerConn.SetLocalDescription(answer); err != nil { + l.errCh <- fmt.Errorf("Failed to set local SDP (Answer): %w", err) + return + } + + // Block until ICE Gathering is complete, disabling trickle ICE + // we do this because we only can exchange one signaling message + // in a production application you should exchange ICE Candidates via OnICECandidate + <-gatherComplete + + sdp := peerConn.LocalDescription().SDP + l.log.Debugf("ICE gathering complete: %s", sdp) + + // WHIP expects a Location header: the hash of our local SDP + resourceId := resourceHash(sdp) + conn.resourceUrl = makeResourceURL(l.config.Endpoint, resourceId) + w.Header().Add("Location", resourceId) + + // WHIP+WHEP expects a HTTP Status Code of 201 + w.WriteHeader(http.StatusCreated) + + // Write Answer with Candidates as HTTP Response + fmt.Fprint(w, peerConn.LocalDescription().SDP) //nolint: errcheck +} + +func (l *Listener) whipDeleteHandler(w http.ResponseWriter, r *http.Request) { + l.log.Tracef("New WHIP DELETE request from client %s for resource id %q", + r.RemoteAddr, r.PathValue("resourceId")) + + resourceId := r.PathValue("resourceId") + if resourceId == "" { + http.Error(w, "Empty resource id", http.StatusBadRequest) + return + } + + l.lock.Lock() + conn, ok := l.conns[resourceId] + l.lock.Unlock() + + if !ok { + http.Error(w, "Unknown resource id", http.StatusNotFound) + return + } + + l.log.Infof("Deleting connection with resource id %q", resourceId) + + if err := conn.Close(); err != nil { + http.Error(w, fmt.Sprintf("Failed to close connection: %s", err.Error()), http.StatusInternalServerError) + return + } + + w.WriteHeader(http.StatusOK) + fmt.Fprint(w, fmt.Sprintf("Resource %s deleted", resourceId)) //nolint +} + +func (_ *Listener) Addr() net.Addr { + return nil +} + +func (l *Listener) Conns() []*listenerConn { + l.lock.Lock() + defer l.lock.Unlock() + ret := []*listenerConn{} + for _, c := range l.conns { + ret = append(ret, c) + } + return ret +} + +type listenerConn struct { + resourceUrl string + listener *Listener + peerConn *webrtc.PeerConnection + dataChan *webrtc.DataChannel + dataConn datachannel.ReadWriteCloser + started, closed bool + log logging.LeveledLogger +} + +func (c *listenerConn) Close() error { + c.log.Tracef("Closing WHIP listener connection %s", c.String()) + + if c.closed { + return nil + } + c.closed = true + + // Close the datachannel + var err error + if c.dataChan.ReadyState() == webrtc.DataChannelStateOpen { + if err = c.dataConn.Close(); err != nil { + c.log.Debugf("Error closing DataChannel: %s", err.Error()) + } + } + + // Close the peer connection + err = c.peerConn.Close() + if err != nil { + c.log.Debugf("Error closing PeerConnection: %s", err.Error()) + } + + if c.started { + c.listener.lock.Lock() + delete(c.listener.conns, c.String()) + c.listener.lock.Unlock() + } + + // Return the last error + return err +} + +func (c *listenerConn) Read(b []byte) (int, error) { + return c.dataConn.Read(b) +} + +func (c *listenerConn) Write(b []byte) (int, error) { + return c.dataConn.Write(b) +} + +// TODO: implement +func (c *listenerConn) LocalAddr() net.Addr { return nil } +func (c *listenerConn) RemoteAddr() net.Addr { return nil } +func (c *listenerConn) SetDeadline(t time.Time) error { return nil } +func (c *listenerConn) SetReadDeadline(t time.Time) error { return nil } +func (c *listenerConn) SetWriteDeadline(t time.Time) error { return nil } + +// String returns a unique identifier for the connection based on the underlying signaling connection. +func (c *listenerConn) String() string { + return c.resourceUrl +} diff --git a/pkg/icetester/icetester_test.go b/pkg/whipconn/whipconn_test.go similarity index 62% rename from pkg/icetester/icetester_test.go rename to pkg/whipconn/whipconn_test.go index 0c8614e0..170cb534 100644 --- a/pkg/icetester/icetester_test.go +++ b/pkg/whipconn/whipconn_test.go @@ -1,8 +1,8 @@ -package icetester +package whipconn import ( "context" - "errors" + "fmt" "net" "net/http" "sync" @@ -10,7 +10,6 @@ import ( "time" "github.com/pion/logging" - "github.com/pion/webrtc/v4" "github.com/stretchr/testify/assert" slogger "github.com/l7mp/stunner/pkg/logger" @@ -20,21 +19,14 @@ var ( testerLogLevel = "all:WARN" // testerLogLevel = "all:TRACE" // testerLogLevel = "all:INFO" - addr = "localhost:12345" - timeout = 5 * time.Second - interval = 50 * time.Millisecond - logger logging.LoggerFactory = slogger.NewLoggerFactory(testerLogLevel) - log logging.LeveledLogger = logger.NewLogger("test") + addr = "localhost:12345" + timeout = 5 * time.Second + interval = 50 * time.Millisecond + defaultConfig = Config{Token: "whiptoken"} + logger logging.LoggerFactory = slogger.NewLoggerFactory(testerLogLevel) + log logging.LeveledLogger = logger.NewLogger("test") ) -// func init() { -// // setup a fast pinger so that we get a timely error notification -// PingPeriod = 500 * time.Millisecond -// PongWait = 800 * time.Millisecond -// WriteWait = 200 * time.Millisecond -// RetryPeriod = 250 * time.Millisecond -// } - func echoTest(t *testing.T, conn net.Conn, content string) { t.Helper() @@ -50,13 +42,14 @@ func echoTest(t *testing.T, conn net.Conn, content string) { var testerTestCases = []struct { name string + config *Config tester func(t *testing.T, ctx context.Context, l *Listener) }{ { name: "Basic connectivity", tester: func(t *testing.T, ctx context.Context, l *Listener) { log.Debug("Creating dialer") - d := NewDialer(webrtc.Configuration{}, logger) + d := NewDialer(defaultConfig, logger) assert.NotNil(t, d) log.Debug("Dialing") @@ -70,6 +63,17 @@ var testerTestCases = []struct { assert.NoError(t, clientConn.Close(), "client conn close") }, + }, { + name: "Invalid bearer token refused", + tester: func(t *testing.T, ctx context.Context, l *Listener) { + log.Debug("Creating dialer") + d := NewDialer(Config{Token: "dummy-token"}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + _, err := d.DialContext(ctx, addr) + assert.Error(t, err) + }, }, { name: "Closing dialer does not close client connection", tester: func(t *testing.T, serverCtx context.Context, l *Listener) { @@ -77,7 +81,7 @@ var testerTestCases = []struct { dialerCtx, dialerCancel := context.WithCancel(context.Background()) log.Debug("Creating dialer") - d := NewDialer(webrtc.Configuration{}, logger) + d := NewDialer(defaultConfig, logger) assert.NotNil(t, d) log.Debug("Dialing") @@ -97,20 +101,20 @@ var testerTestCases = []struct { name: "Client side close closes server", tester: func(t *testing.T, serverCtx context.Context, l *Listener) { log.Debug("Creating dialer") - d := NewDialer(webrtc.Configuration{}, logger) + d := NewDialer(defaultConfig, logger) assert.NotNil(t, d) log.Debug("Dialing") clientConn, err := d.DialContext(serverCtx, addr) assert.NoError(t, err) - assert.Eventually(t, func() bool { return l.activeConns == 1 }, timeout, interval) + assert.Eventually(t, func() bool { return len(l.conns) == 1 }, timeout, interval) log.Debug("Closing client connection") assert.NoError(t, clientConn.Close()) // should close the server conn too - assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + assert.Eventually(t, func() bool { return len(l.conns) == 0 }, timeout, interval) }, }, { name: "Server side close closes client", @@ -119,24 +123,21 @@ var testerTestCases = []struct { defer clientCancel() log.Debug("Creating dialer") - d := NewDialer(webrtc.Configuration{}, logger) + d := NewDialer(defaultConfig, logger) assert.NotNil(t, d) log.Debug("Dialing") clientConn, err := d.DialContext(clientCtx, addr) assert.NoError(t, err) - assert.Eventually(t, func() bool { return l.activeConns == 1 }, timeout, interval) + assert.Eventually(t, func() bool { return len(l.conns) == 1 }, timeout, interval) log.Debug("Closing server connections") for _, lConn := range l.Conns() { - // log.Infof("------------ %s", lConn.String()) assert.NoError(t, lConn.Close()) } - log.Info("&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&") - - assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + assert.Eventually(t, func() bool { return len(l.conns) == 0 }, timeout, interval) // should close the client conn too assert.Eventually(t, func() bool { return clientConn.(*dialerConn).closed == true }, timeout, interval) @@ -145,7 +146,7 @@ var testerTestCases = []struct { name: "Multiple connections", tester: func(t *testing.T, ctx context.Context, l *Listener) { log.Debug("Creating dialer") - d := NewDialer(webrtc.Configuration{}, logger) + d := NewDialer(defaultConfig, logger) assert.NotNil(t, d) log.Debug("Dialing: creating 5 connections") @@ -172,13 +173,44 @@ var testerTestCases = []struct { wg.Wait() close(connChan) - assert.Eventually(t, func() bool { return l.activeConns == 5 }, timeout, interval) + assert.Eventually(t, func() bool { return len(l.conns) == 5 }, timeout, interval) for c := range connChan { c.Close() } - assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) + assert.Eventually(t, func() bool { return len(l.conns) == 0 }, timeout, interval) + }, + }, { + name: "Closing invalid resource fails", + tester: func(t *testing.T, ctx context.Context, l *Listener) { + uri := fmt.Sprintf("http://%s/whip/dummy-id", addr) + req, err := http.NewRequest("DELETE", uri, nil) + assert.NoError(t, err) + req.Header.Add("Authorization", "Bearer "+defaultConfig.Token) + + r, err := http.DefaultClient.Do(req) + assert.NoError(t, err) + assert.Equal(t, http.StatusNotFound, r.StatusCode) + }, + }, { + name: "Connecting with custom path", + config: &Config{Endpoint: "/custompath"}, + tester: func(t *testing.T, ctx context.Context, l *Listener) { + log.Debug("Creating dialer") + d := NewDialer(Config{Endpoint: "/custompath"}, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(ctx, addr) + assert.NoError(t, err) + + log.Debug("Echo test round 1") + echoTest(t, clientConn, "test1") + log.Debug("Echo test round 2") + echoTest(t, clientConn, "test2") + + assert.NoError(t, clientConn.Close(), "client conn close") }, }, } @@ -192,8 +224,13 @@ func TestTesterConn(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() + config := defaultConfig + if c.config != nil { + config = *c.config + } + log.Debug("Creating listener") - listener, err := NewListener(addr, webrtc.Configuration{}, logger) + listener, err := NewListener(addr, config, logger) assert.NoError(t, err) l = listener assert.NotNil(t, l) @@ -208,7 +245,7 @@ func TestTesterConn(t *testing.T) { log.Debug("Accepting server connection") - // responder + // readloop go func() { buf := make([]byte, 100) for { @@ -221,28 +258,12 @@ func TestTesterConn(t *testing.T) { assert.NoError(t, err) } }() - - // closer - go func() { - <-ctx.Done() - - if err := conn.Close(); err != nil && !errors.Is(err, net.ErrClosed) && - !errors.Is(err, http.ErrServerClosed) { - t.Error("server conn close") - } - - // close listener - l.Close() //nolint - }() } }() c.tester(t, ctx, l) - }) - log.Debug("Waiting for connections to close") - if l != nil { - assert.Eventually(t, func() bool { return l.activeConns == 0 }, timeout, interval) - } + l.Close() //nolint + }) } } From 032619f2587c789d97aefe1bdd97f3b815de798a Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 28 Nov 2024 14:17:13 +0100 Subject: [PATCH 235/243] fix: No longer delay config-delete messages in the CDS server So far we have added a 5 sec delay before sending our config deletion messages on the CDS server watchers in order to prevent a race condition where a terminating client receives the config deletion message from the CDS server before it can actually enter the graceful shutdown cycle, which causes the immediate dropping of all active client connections. The delay allowed comfortable time to the client to start graceful shutdown. Unfortunately, the delay also caused unexpected resets in some hard-to-debug situations, where during adding the Gateway API resources there is a transient that causes a (delayed) delete due to a missing API resource, which then resets the full config later when all resources become available. This change removes the delay in the config-delete path of the CDS server, so all config-delete updates are immediately sent out. In order to prevent the race condition, the CDS client in stunnerd ignores delete-config CDS updates all together (but only in stunnerd, the auth-service and stunnerctl still process them): once started, there is no other way to stop a stunnerd pod than to delete the Gateway resource. --- cmd/stunnerctl/main.go | 2 +- cmd/stunnerd/main.go | 12 +- config.go | 8 +- config_test.go | 6 +- pkg/config/cds_test.go | 182 +++++++++++++++++++++++-------- pkg/config/client/cds_api.go | 45 ++++---- pkg/config/client/client.go | 11 +- pkg/config/client/file_client.go | 15 ++- pkg/config/server/config.go | 27 ++--- pkg/config/server/handler.go | 2 +- pkg/config/server/server.go | 42 +++---- 11 files changed, 220 insertions(+), 132 deletions(-) diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go index 41d2bf23..8c15df44 100644 --- a/cmd/stunnerctl/main.go +++ b/cmd/stunnerctl/main.go @@ -173,7 +173,7 @@ func runConfig(_ *cobra.Command, args []string) error { confChan := make(chan *stnrv1.StunnerConfig, 8) if watch { - err := cds.Watch(ctx, confChan) + err := cds.Watch(ctx, confChan, false) if err != nil { close(confChan) return err diff --git a/cmd/stunnerd/main.go b/cmd/stunnerd/main.go index 744d1232..462e79d5 100644 --- a/cmd/stunnerd/main.go +++ b/cmd/stunnerd/main.go @@ -142,8 +142,8 @@ func main() { configOrigin = cdsAddr.Addr } - log.Infof("Watching configuration at origin %q", configOrigin) - if err := st.WatchConfig(ctx, configOrigin, conf); err != nil { + log.Infof("Watching configuration at origin %q (ignoring delete-config updates)", configOrigin) + if err := st.WatchConfig(ctx, configOrigin, conf, true); err != nil { log.Errorf("Could not run config watcher: %s", err.Error()) os.Exit(1) } @@ -195,11 +195,9 @@ func main() { c.Admin.LogLevel = logLevel } - // we have working stunnerd: reconcile log.Debug("Initiating reconciliation") - err := st.Reconcile(c) - log.Trace("Reconciliation ready") - if err != nil { + + if err := st.Reconcile(c); err != nil { if e, ok := err.(stnrv1.ErrRestarted); ok { log.Debugf("Reconciliation ready: %s", e.Error()) } else { @@ -207,6 +205,8 @@ func main() { "(running configuration unchanged): %s", err.Error()) } } + + log.Trace("Reconciliation ready") } } } diff --git a/config.go b/config.go index 601e5039..e39675be 100644 --- a/config.go +++ b/config.go @@ -143,7 +143,7 @@ func (s *Stunner) GetConfig() *stnrv1.StunnerConfig { return &c } -// LoadConfig loads a configuration from an origin. This is a shim wrapper around ConfigOrigin.Load. +// LoadConfig loads a configuration from an origin. This is a shim wrapper around configclient.Load. func (s *Stunner) LoadConfig(origin string) (*stnrv1.StunnerConfig, error) { client, err := client.New(origin, s.name, s.logger) if err != nil { @@ -153,12 +153,12 @@ func (s *Stunner) LoadConfig(origin string) (*stnrv1.StunnerConfig, error) { return client.Load() } -// WatchConfig watches a configuration from an origin. This is a shim wrapper around ConfigOrigin.Watch. -func (s *Stunner) WatchConfig(ctx context.Context, origin string, ch chan<- *stnrv1.StunnerConfig) error { +// WatchConfig watches a configuration from an origin. This is a shim wrapper around configclient.Watch. +func (s *Stunner) WatchConfig(ctx context.Context, origin string, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { client, err := client.New(origin, s.name, s.logger) if err != nil { return err } - return client.Watch(ctx, ch) + return client.Watch(ctx, ch, suppressDelete) } diff --git a/config_test.go b/config_test.go index 96c62086..0a9b247d 100644 --- a/config_test.go +++ b/config_test.go @@ -154,7 +154,7 @@ func TestStunnerConfigFileWatcher(t *testing.T) { defer cancel() url := "file://" + file - err = stunner.WatchConfig(ctx, url, conf) + err = stunner.WatchConfig(ctx, url, conf, false) assert.NoError(t, err, "creating config watcher") // nothing should happen here: wait a bit so that the watcher has comfortable time to start @@ -269,7 +269,7 @@ func TestStunnerConfigFileWatcherMultiVersion(t *testing.T) { defer cancel() url := "file://" + file - err = stunner.WatchConfig(ctx, url, conf) + err = stunner.WatchConfig(ctx, url, conf, false) assert.NoError(t, err, "creating config watcher") // nothing should happen here: wait a bit so that the watcher has comfortable time to start @@ -410,7 +410,7 @@ func TestStunnerConfigPollerMultiVersion(t *testing.T) { defer close(conf) log.Debug("init config poller") - assert.NoError(t, stunner.WatchConfig(ctx, origin, conf), "creating config poller") + assert.NoError(t, stunner.WatchConfig(ctx, origin, conf, true), "creating config poller") c2, ok := <-conf assert.True(t, ok, "config emitted") diff --git a/pkg/config/cds_test.go b/pkg/config/cds_test.go index c8519181..50a178d0 100644 --- a/pkg/config/cds_test.go +++ b/pkg/config/cds_test.go @@ -51,9 +51,7 @@ func TestServerLoad(t *testing.T) { testLog := logger.NewLogger("test") // suppress deletions - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = 0 - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + server.SuppressConfigDeletion = true ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -144,11 +142,6 @@ func TestServerPoll(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") - // suppress deletions - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = 0 - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() - ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -178,15 +171,15 @@ func TestServerPoll(t *testing.T) { defer close(ch3) go func() { - err = client1.Poll(ctx, ch1) + err = client1.Poll(ctx, ch1, false) assert.NoError(t, err, "client 1 cancelled") }() go func() { - err = client2.Poll(ctx, ch2) + err = client2.Poll(ctx, ch2, false) assert.NoError(t, err, "client 2 cancelled") }() go func() { - err = client3.Poll(ctx, ch2) + err = client3.Poll(ctx, ch2, false) assert.NoError(t, err, "client 3 cancelled") }() @@ -253,11 +246,6 @@ func TestServerWatch(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") - // suppress deletions - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = 0 - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() - serverCtx, serverCancel := context.WithCancel(context.Background()) testCDSAddr := getRandCDSAddr() @@ -285,11 +273,11 @@ func TestServerWatch(t *testing.T) { clientCtx, clientCancel := context.WithCancel(context.Background()) defer clientCancel() - err = client1.Watch(clientCtx, ch1) + err = client1.Watch(clientCtx, ch1, false) assert.NoError(t, err, "client 1 watch") - err = client2.Watch(clientCtx, ch2) + err = client2.Watch(clientCtx, ch2, false) assert.NoError(t, err, "client 2 watch") - err = client3.Watch(clientCtx, ch3) + err = client3.Watch(clientCtx, ch3, false) assert.NoError(t, err, "client 3 watch") s := watchConfig(ch1, 150*time.Millisecond) @@ -435,11 +423,6 @@ func TestServerWatchBootstrap(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") - // suppress deletions - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = 0 - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() - serverCtx, serverCancel := context.WithCancel(context.Background()) defer serverCancel() @@ -479,7 +462,7 @@ func TestServerWatchBootstrap(t *testing.T) { clientCtx, clientCancel := context.WithCancel(context.Background()) defer clientCancel() - err = client1.Watch(clientCtx, ch1) + err = client1.Watch(clientCtx, ch1, false) assert.NoError(t, err, "client 1 watch") s := watchConfig(ch1, 1500*time.Millisecond) @@ -534,11 +517,6 @@ func TestServerAPI(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") - // suppress deletions - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = time.Millisecond - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() - serverCtx, serverCancel := context.WithCancel(context.Background()) testCDSAddr := getRandCDSAddr() @@ -570,13 +548,13 @@ func TestServerAPI(t *testing.T) { clientCtx, clientCancel := context.WithCancel(context.Background()) defer clientCancel() - err = client1.Watch(clientCtx, ch1) + err = client1.Watch(clientCtx, ch1, false) assert.NoError(t, err, "client 1 watch") - err = client2.Watch(clientCtx, ch2) + err = client2.Watch(clientCtx, ch2, false) assert.NoError(t, err, "client 2 watch") - err = client3.Watch(clientCtx, ch3) + err = client3.Watch(clientCtx, ch3, false) assert.NoError(t, err, "client 3 watch") - err = client4.Watch(clientCtx, ch4) + err = client4.Watch(clientCtx, ch4, false) assert.NoError(t, err, "client 4 watch") s := watchConfig(ch1, 50*time.Millisecond) @@ -877,6 +855,9 @@ func TestServerAPI(t *testing.T) { s = watchConfig(ch4, 50*time.Millisecond) assert.Nil(t, s) + // switch config deletions on + server.SuppressConfigDeletion = false + testLog.Debug("--------------------------------") testLog.Debug("Update1: ns1/gw1 + ns3/gw1 ") testLog.Debug("--------------------------------") @@ -953,8 +934,11 @@ func TestServerAPI(t *testing.T) { assert.NotNil(t, s1) s2 = watchConfig(ch2, 50*time.Millisecond) assert.NotNil(t, s2) - assert.True(t, s1.DeepEqual(sc1), "deepeq") - assert.True(t, client.IsConfigDeleted(s2), "deepeq") // deleted! + // we do not know the order + assert.True(t, s1.DeepEqual(sc1) || s2.DeepEqual(sc1), "config-deepeq") + assert.True(t, client.IsConfigDeleted(s1) || client.IsConfigDeleted(s2), "deleted") // deleted + // assert.True(t, s1.DeepEqual(sc1), "deepeq") + // assert.True(t, client.IsConfigDeleted(s2), "deepeq") // deleted! // no config from client3 watch s = watchConfig(ch3, 50*time.Millisecond) @@ -974,14 +958,12 @@ func TestClientReconnect(t *testing.T) { zlogger := zapr.NewLogger(z) log := zlogger.WithName("tester") - // suppress deletions - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = 0 - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() - logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") + // suppress deletions + server.SuppressConfigDeletion = true + serverCtx, serverCancel := context.WithCancel(context.Background()) defer serverCancel() @@ -1002,7 +984,7 @@ func TestClientReconnect(t *testing.T) { clientCtx, clientCancel := context.WithCancel(context.Background()) defer clientCancel() - err = client1.Watch(clientCtx, ch1) + err = client1.Watch(clientCtx, ch1, false) assert.NoError(t, err, "client 1 watch") s := watchConfig(ch1, 150*time.Millisecond) @@ -1055,10 +1037,8 @@ func TestServerUpdate(t *testing.T) { logger := logger.NewLoggerFactory(stunnerLogLevel) testLog := logger.NewLogger("test") - // make sure deletions are suppressed - deleteDelay := server.ConfigDeletionUpdateDelay - server.ConfigDeletionUpdateDelay = 0 - defer func() { server.ConfigDeletionUpdateDelay = deleteDelay }() + // suppress deletions + server.SuppressConfigDeletion = true serverCtx, serverCancel := context.WithCancel(context.Background()) defer serverCancel() @@ -1141,6 +1121,112 @@ func TestServerUpdate(t *testing.T) { assert.True(t, sc2.DeepEqual(tcpC), "deepeq") } +// Test various combinations of server-side "drop-delete" (server.SuppressConfigDeletion=true) and +// client-side "drop-delete" (client.Watch(..., suppressDelete=true)). +func TestDeleteConfigAPI(t *testing.T) { + zc := zap.NewProductionConfig() + zc.Level = zap.NewAtomicLevelAt(testerLogLevel) + z, err := zc.Build() + assert.NoError(t, err, "logger created") + zlogger := zapr.NewLogger(z) + log := zlogger.WithName("tester") + + logger := logger.NewLoggerFactory(stunnerLogLevel) + testLog := logger.NewLogger("test") + + saved := server.SuppressConfigDeletion + server.SuppressConfigDeletion = false + serverCtx, serverCancel := context.WithCancel(context.Background()) + defer serverCancel() + + testCDSAddr := getRandCDSAddr() + testLog.Debugf("create server on %s", testCDSAddr) + srv := server.New(testCDSAddr, nil, log) + assert.NotNil(t, srv, "server") + err = srv.Start(serverCtx) + assert.NoError(t, err, "start") + + testLog.Debug("create client") + c, err := client.New(testCDSAddr, "ns1/gw1", logger) + assert.NoError(t, err, "client") + + ch := make(chan *stnrv1.StunnerConfig, 8) + defer close(ch) + + for _, testCase := range []struct { + name string + serverDropDel, clientDropDel bool + tester func(t *testing.T) + }{ + { + name: "server sends delete - client handles delete", + serverDropDel: false, + clientDropDel: false, + tester: func(t *testing.T) { + conf := watchConfig(ch, 50*time.Millisecond) + assert.NotNil(t, conf, "config") + assert.True(t, client.IsConfigDeleted(conf)) + }, + }, + { + name: "server suppresses delete - client handles delete", + serverDropDel: true, + clientDropDel: false, + tester: func(t *testing.T) { + conf := watchConfig(ch, 50*time.Millisecond) + assert.Nil(t, conf, "config") + }, + }, + { + name: "server sends delete - client suppresses delete", + serverDropDel: false, + clientDropDel: true, + tester: func(t *testing.T) { + conf := watchConfig(ch, 50*time.Millisecond) + assert.Nil(t, conf, "config") + }, + }, + { + name: "server suppresses delete - client suppresses delete", + serverDropDel: true, + clientDropDel: true, + tester: func(t *testing.T) { + conf := watchConfig(ch, 50*time.Millisecond) + assert.Nil(t, conf, "config") + }, + }, + } { + testLog.Debugf("------------------------- %s ----------------------", testCase.name) + + server.SuppressConfigDeletion = testCase.serverDropDel + + clientCtx, clientCancel := context.WithCancel(context.Background()) + err = c.Watch(clientCtx, ch, testCase.clientDropDel) + assert.NoError(t, err, "client watch") + + conf := watchConfig(ch, 25*time.Millisecond) + assert.Nil(t, conf, "noconfig") + + testLog.Trace("Adding config") + testConf := testConfig("ns1/gw1", "realm1") + err = srv.UpdateConfig([]server.Config{testConf}) + assert.NoError(t, err, "update") + + conf = watchConfig(ch, 50*time.Millisecond) + assert.NotNil(t, conf) + assert.Equal(t, *testConf.Config, *conf) + + testLog.Trace("Deleting config") + err = srv.UpdateConfig([]server.Config{}) + assert.NoError(t, err, "update") + testCase.tester(t) + + clientCancel() + } + + server.SuppressConfigDeletion = saved +} + // func TestServerPatcher(t *testing.T) { // zc := zap.NewProductionConfig() // zc.Level = zap.NewAtomicLevelAt(testerLogLevel) @@ -1206,7 +1292,7 @@ func TestServerUpdate(t *testing.T) { // ch1 := make(chan *stnrv1.StunnerConfig, 8) // defer close(ch1) -// err = watcher1.Watch(watchCtx, ch1) +// err = watcher1.Watch(watchCtx, ch1, false) // assert.NoError(t, err, "client watch") // s := watchConfig(ch1, 100*time.Millisecond) @@ -1219,7 +1305,7 @@ func TestServerUpdate(t *testing.T) { // ch2 := make(chan *stnrv1.StunnerConfig, 8) // defer close(ch2) -// err = watcher2.Watch(watchCtx, ch2) +// err = watcher2.Watch(watchCtx, ch2, false) // assert.NoError(t, err, "client watch") // s = watchConfig(ch2, 100*time.Millisecond) diff --git a/pkg/config/client/cds_api.go b/pkg/config/client/cds_api.go index 39103e82..e017abe4 100644 --- a/pkg/config/client/cds_api.go +++ b/pkg/config/client/cds_api.go @@ -36,10 +36,12 @@ type CdsApi interface { // Get loads the config(s) from the API endpoint. Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error) // Watch watches config(s) from the API endpoint of a CDS server. If the server is not - // available watch will retry, and if the connection goes away it will create a new one. - Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error + // available watch will retry, and if the connection goes away it will create a new one. If + // set, the suppressDelete instructs the API to ignore config delete updates from the + // server. + Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error // Poll creates a one-shot config watcher without the retry mechanincs of Watch. - Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error + Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error logging.LeveledLogger } @@ -98,14 +100,14 @@ func (a *AllConfigsAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, error return decodeConfigList(r.Body) } -func (a *AllConfigsAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (a *AllConfigsAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { a.Debugf("WATCH: watching all configs from CDS server %s", a.wsURI) - return watch(ctx, a, ch) + return watch(ctx, a, ch, suppressDelete) } -func (a *AllConfigsAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (a *AllConfigsAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { a.Debugf("POLL: polling all configs from CDS server %s", a.wsURI) - return poll(ctx, a, ch) + return poll(ctx, a, ch, suppressDelete) } // ConfigsNamespaceAPI is the API for listing all configs in a namespace. @@ -163,16 +165,16 @@ func (a *ConfigsNamespaceAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConfig, return decodeConfigList(r.Body) } -func (a *ConfigsNamespaceAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (a *ConfigsNamespaceAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { a.Debugf("WATCH: watching all configs in namespace %s from CDS server %s", a.namespace, a.wsURI) - return watch(ctx, a, ch) + return watch(ctx, a, ch, suppressDelete) } -func (a *ConfigsNamespaceAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (a *ConfigsNamespaceAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { a.Debugf("POLL: polling all configs in namespace %s from CDS server %s", a.namespace, a.wsURI) - return poll(ctx, a, ch) + return poll(ctx, a, ch, suppressDelete) } type ConfigNamespaceNameAPI struct { @@ -231,22 +233,22 @@ func (a *ConfigNamespaceNameAPI) Get(ctx context.Context) ([]*stnrv1.StunnerConf return decodeConfig(r.Body) } -func (a *ConfigNamespaceNameAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (a *ConfigNamespaceNameAPI) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { a.Debugf("WATCH: watching config for gateway %s/%s from CDS server %s", a.namespace, a.name, a.wsURI) - return watch(ctx, a, ch) + return watch(ctx, a, ch, suppressDelete) } -func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (a *ConfigNamespaceNameAPI) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { a.Debugf("POLL: polling config for gateway %s/%s from CDS server %s", a.namespace, a.name, a.wsURI) - return poll(ctx, a, ch) + return poll(ctx, a, ch, suppressDelete) } -func watch(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { +func watch(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { go func() { for { - if err := poll(ctx, a, ch); err != nil { + if err := poll(ctx, a, ch, suppressDelete); err != nil { _, wsuri := a.Endpoint() a.Errorf("failed to init CDS watcher (url: %s): %s", wsuri, err.Error()) } else { @@ -265,7 +267,7 @@ func watch(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error // //////////// // API workers // //////////// -func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error { +func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { _, url := a.Endpoint() a.Tracef("poll: trying to open connection to CDS server at %s", url) @@ -354,6 +356,11 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error continue } + if suppressDelete && IsConfigDeleted(c) { + a.Infof("Ignoring delete configuration update from %q", url) + continue + } + a.Debugf("new config received from %q: %q", url, c.String()) ch <- c @@ -363,7 +370,7 @@ func poll(ctx context.Context, a CdsApi, ch chan<- *stnrv1.StunnerConfig) error // wait fo cancel for { defer func() { - a.Infof("closing connection for client %s", conn.RemoteAddr().String()) + a.Infof("closing connection to server %s", conn.RemoteAddr().String()) conn.WriteMessage(websocket.CloseMessage, []byte{}) //nolint:errcheck conn.Close() closePinger <- struct{}{} diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index f1c2a863..d54a843d 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -33,12 +33,13 @@ var ( type Client interface { // Load grabs a new configuration from the config client. Load() (*stnrv1.StunnerConfig, error) - // Watch grabs new configs from a config origin (config file or CDS server) and returns - // them on the channel. The context cancels the watcher. If the origin is not available - // watch will retry. - Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error + // Watch listens to new configs from a config origin (config file or CDS server) and + // returns them on the given channel. The context cancels the watcher. If the origin is not + // available watch will retry. If set, the suppressDelete flag instructs the client to + // ignore delete config (essentially zero-configs) from the origin. + Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error // Poll creates a one-shot config watcher without the retry mechanincs of Watch. - Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error + Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error fmt.Stringer } diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index 3f0e7b2c..0ead3bfc 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -66,15 +66,15 @@ func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { // Watch watches a configuration file for changes. If no file exists at the given path, it will // periodically retry until the file appears. -func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { if w.configFile == "" { - return errors.New("Uninitialized config file path") + return errors.New("uninitialized config file path") } go func() { for { // try to watch - if err := w.Poll(ctx, ch); err != nil { + if err := w.Poll(ctx, ch, suppressDelete); err != nil { w.log.Warnf("Error loading config file %q: %s", w.configFile, err.Error()) } else { @@ -92,7 +92,7 @@ func (w *ConfigFileClient) Watch(ctx context.Context, ch chan<- *stnrv1.StunnerC // Poll watches the config file and emits new configs on the specified channel. Returns an error if // further action is needed (tryWatchConfig is to be started) or nil on normal exit. -func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig) error { +func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerConfig, suppressDelete bool) error { w.log.Tracef("configWatcher") // create a new watcher @@ -160,7 +160,12 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo // suppress repeated events if c.DeepEqual(&prev) { - w.log.Debugf("Ignoring recurrent notify event for the same config file") + w.log.Debug("Ignoring recurrent notify event for the same config file") + continue + } + + if suppressDelete && IsConfigDeleted(c) { + w.log.Info("Ignoring deleted configuration") continue } diff --git a/pkg/config/server/config.go b/pkg/config/server/config.go index 9b0d3ca2..159fcd61 100644 --- a/pkg/config/server/config.go +++ b/pkg/config/server/config.go @@ -28,24 +28,19 @@ func (s *Server) UpsertConfig(id string, c *stnrv1.StunnerConfig) { s.configCh <- Config{Id: id, Config: cpy} } -// DeleteConfig removes a config from the client. Theoretically, this should send the client a -// zero-config immediately. However, in order to avoid that a client being removed and entering the +// DeleteConfig removes a config from clients by sending a zero-config. Clients may decide to +// ignore the delete operation by (1) using client.IsConfigDeleted() to identify whether a config +// is being deleted and (2) selectively ignoring config delete updates based on the result. This is +// needed, e.g., in stunnerd, in order to avoid that a client being removed and entering the // graceful shutdown cycle receive a zeroconfig and abruprly kill all listeners with all active -// connections allocated to them, we actually delay sending the zeroconfig with a configurable time -// (default is 5 sec, but a zero delay will suppress sending the zero-config all together). This -// should allow the client comfortable time to enter the grafeul shutdown cycle. Note that clients -// should stop actively reconciling config updates once they initiated graceful shutdown for this -// to work. +// connections allocated to them. func (s *Server) DeleteConfig(id string) { s.configs.Delete(id) - if ConfigDeletionUpdateDelay == 0 { - s.log.Info("Suppressing config update for deleted config", "client", id) + if SuppressConfigDeletion { + s.log.Info("Suppressing config update for deleted config", "config-id", id) return } - s.log.Info("Delaying config update for deleted config", "client", id, - "delay", ConfigDeletionUpdateDelay) - s.deleteCh <- Config{Id: id, Config: client.ZeroConfig(id)} } @@ -60,11 +55,11 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { for _, newC := range newConfigs { if oldC.Id == newC.Id { if !oldC.Config.DeepEqual(newC.Config) { - s.log.V(2).Info("Updating config", "client", newC.Id, "config", + s.log.V(2).Info("Updating config", "config-id", newC.Id, "config", newC.Config.String()) s.UpsertConfig(newC.Id, newC.Config) } else { - s.log.V(2).Info("Config not updated", "client", newC.Id, + s.log.V(2).Info("Config unchanged", "config-id", newC.Id, "old-config", oldC.Config.String(), "new-config", newC.Config.String()) } @@ -74,7 +69,7 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { } if !found { - s.log.V(2).Info("Removing config", "client", oldC.Id) + s.log.V(2).Info("Removing config", "config-id", oldC.Id) s.DeleteConfig(oldC.Id) } } @@ -89,7 +84,7 @@ func (s *Server) UpdateConfig(newConfigs []Config) error { } if !found { - s.log.V(2).Info("Adding config", "client", newC.Id, "config", newC.Config) + s.log.V(2).Info("Adding config", "config-id", newC.Id, "config", newC.Config) s.UpsertConfig(newC.Id, newC.Config) } } diff --git a/pkg/config/server/handler.go b/pkg/config/server/handler.go index c2e5892d..f97e81c0 100644 --- a/pkg/config/server/handler.go +++ b/pkg/config/server/handler.go @@ -66,7 +66,7 @@ func (s *Server) GetV1ConfigNamespaceName(ctx context.Context, request api.GetV1 id := fmt.Sprintf("%s/%s", namespace, name) c := s.configs.Get(id) if c == nil { - s.log.V(1).Info("GetV1ConfigNamespaceName: Config not found", "client", id) + s.log.V(1).Info("GetV1ConfigNamespaceName: Config not found", "config-id", id) return api.GetV1ConfigNamespaceName404JSONResponse{ Code: http.StatusNotFound, Message: fmt.Sprintf("Config not found for ID %q", id), diff --git a/pkg/config/server/server.go b/pkg/config/server/server.go index f554cf52..f986bdfe 100644 --- a/pkg/config/server/server.go +++ b/pkg/config/server/server.go @@ -5,9 +5,9 @@ import ( "context" "encoding/json" "errors" + "fmt" "net" "net/http" - "time" "github.com/go-logr/logr" "github.com/gorilla/mux" @@ -18,10 +18,9 @@ import ( ) var ( - // ConfigDeletionUpdateDelay is the delay between deleting a config from the server and - // sending the corresponing zero-config to the client. Set this to zero to suppress sending - // the zero-config all together. - ConfigDeletionUpdateDelay = 5 * time.Second + // SuppressConfigDeletion allows the server to suppress config deletions all together. Used + // mostly for testing. + SuppressConfigDeletion = false ) // Server is a generic config discovery server implementation. @@ -60,11 +59,15 @@ func (s *Server) Start(ctx context.Context) error { handler := api.NewStrictHandler(s, []api.StrictMiddlewareFunc{s.WSUpgradeMiddleware}) api.HandlerFromMux(handler, s.router) s.Server = &http.Server{Addr: s.addr, Handler: s.router} + l, err := net.Listen("tcp", s.addr) + if err != nil { + return fmt.Errorf("CDS server failed to listen: %w", err) + } go func() { s.log.Info("Starting CDS server", "address", s.addr, "patch", s.patch != nil) - err := s.ListenAndServe() + err := s.Serve(l) if err != nil { if errors.Is(err, net.ErrClosed) || errors.Is(err, http.ErrServerClosed) { s.log.Info("Closing config discovery server") @@ -83,21 +86,11 @@ func (s *Server) Start(ctx context.Context) error { for { select { case c := <-s.configCh: + s.log.V(2).Info("Sending config update event", "config-id", c.Id) s.broadcastConfig(c) - case c := <-s.deleteCh: - s.log.V(2).Info("Initiating deleyed config deletion", "id", c.Id) - - go func() { - select { - case <-ctx.Done(): - return - case <-time.After(ConfigDeletionUpdateDelay): - s.configCh <- Config{Id: c.Id, Config: c.Config} - return - } - }() - + s.log.V(2).Info("Sending config delete event", "config-id", c.Id) + s.broadcastConfig(c) case <-ctx.Done(): return } @@ -135,9 +128,10 @@ func (s *Server) GetConnTrack() *ConnTrack { // RemoveClient forcefully closes a client connection. This is used mainly for testing. func (s *Server) RemoveClient(id string) { - if c := s.conns.Get(id); c != nil { - s.log.V(1).Info("Forcefully removing client connection", "client", id) - s.closeConn(c) + if conn := s.conns.Get(id); conn != nil { + s.log.V(1).Info("Forcefully removing client connection", "config-id", id, + "client", conn.RemoteAddr().String()) + s.closeConn(conn) } } @@ -156,6 +150,7 @@ func (s *Server) handleConn(reqCtx context.Context, wsConn *websocket.Conn, oper // drop anything we receive _, _, err := conn.ReadMessage() if err != nil { + s.closeConn(conn) return } } @@ -211,8 +206,7 @@ func (s *Server) sendConfig(conn *Conn, e *stnrv1.StunnerConfig) { return } - s.log.V(2).Info("Sending configuration to client", "client", conn.Id(), - "config", c.String()) + s.log.V(2).Info("Sending configuration to client", "client", conn.Id()) if err := conn.WriteMessage(websocket.TextMessage, json); err != nil { s.log.Error(err, "Error sending config update", "client", conn.Id()) From 74d54804a847824ebb602b742906eedd89d80970 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 28 Nov 2024 14:24:34 +0100 Subject: [PATCH 236/243] chore: Improve error message consistency --- cmd/turncat/main.go | 8 ++++---- internal/object/admin.go | 2 +- internal/object/errors.go | 2 +- internal/resolver/mock.go | 2 +- internal/resolver/resolver.go | 6 +++--- internal/util/endpoint.go | 6 +++--- pkg/apis/v1/auth.go | 3 +-- pkg/apis/v1/errors.go | 8 ++++---- pkg/config/client/client.go | 4 ++-- pkg/config/client/file_client.go | 12 ++++++------ pkg/config/client/k8s_client.go | 14 +++++++------- pkg/whipconn/dialer.go | 26 +++++++++++++------------- pkg/whipconn/listener.go | 18 +++++++++--------- relay.go | 8 ++++---- turncat.go | 16 ++++++++-------- uri.go | 4 ++-- 16 files changed, 69 insertions(+), 70 deletions(-) diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 69f63cc1..93c8349e 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -146,14 +146,14 @@ func getStunnerConf(uri string) (*stnrv1.StunnerConfig, error) { case "k8s": conf, err := getStunnerConfFromK8s(def) if err != nil { - return nil, fmt.Errorf("Could not read running STUNner configuration from "+ + return nil, fmt.Errorf("could not read running STUNner configuration from "+ "Kubernetes: %w", err) } return conf, nil case "turn": conf, err := getStunnerConfFromCLI(def) if err != nil { - return nil, fmt.Errorf("Could not generate STUNner configuration from "+ + return nil, fmt.Errorf("could not generate STUNner configuration from "+ "URI %q: %w", uri, err) } return conf, nil @@ -237,11 +237,11 @@ func getStunnerConfFromCLI(def string) (*stnrv1.StunnerConfig, error) { u, err := stunner.ParseUri(uri) if err != nil { - return nil, fmt.Errorf("Invalid STUNner URI %q: %s", uri, err) + return nil, fmt.Errorf("invalid STUNner URI %q: %s", uri, err) } if u.Username == "" || u.Password == "" { - return nil, fmt.Errorf("Username/password must be set: '%s'", uri) + return nil, fmt.Errorf("username/password must be set: '%s'", uri) } conf.Listeners[0].PublicAddr = u.Address diff --git a/internal/object/admin.go b/internal/object/admin.go index 418449c2..9bb5c7eb 100644 --- a/internal/object/admin.go +++ b/internal/object/admin.go @@ -198,7 +198,7 @@ func (a *Admin) reconcileMetrics(req *stnrv1.AdminConfig) error { a.log.Tracef("closing metrics server at %s", mEndpoint) if err := a.metricsServer.Shutdown(context.Background()); err != nil { - return fmt.Errorf("error stopping metrics server at %s: %w", + return fmt.Errorf("failed to stop metrics server at %s: %w", mEndpoint, err) } a.metricsServer = nil diff --git a/internal/object/errors.go b/internal/object/errors.go index c7ee7542..2b984680 100644 --- a/internal/object/errors.go +++ b/internal/object/errors.go @@ -4,5 +4,5 @@ import "errors" var ( // ErrRestartRequired indicates that an object needs to be restarted for reconciliation. - ErrRestartRequired = errors.New("Restart required") + ErrRestartRequired = errors.New("restart required") ) diff --git a/internal/resolver/mock.go b/internal/resolver/mock.go index d25aa437..2d2960a0 100644 --- a/internal/resolver/mock.go +++ b/internal/resolver/mock.go @@ -63,5 +63,5 @@ func (m *MockResolver) Lookup(domain string) ([]net.IP, error) { } } - return []net.IP{}, fmt.Errorf("Host %q not found: 3(NXDOMAIN)", domain) + return []net.IP{}, fmt.Errorf("host %q not found: 3(NXDOMAIN)", domain) } diff --git a/internal/resolver/resolver.go b/internal/resolver/resolver.go index 7b7a305c..6b1f8328 100644 --- a/internal/resolver/resolver.go +++ b/internal/resolver/resolver.go @@ -119,7 +119,7 @@ func doResolve(e *serviceEntry) error { if e.cname == "" { cname, err := e.resolver.LookupCNAME(e.ctx, e.domain) if err != nil { - return fmt.Errorf("Cannot resolve CNAME for domain %q: %s", + return fmt.Errorf("failed to resolve CNAME for domain %q: %s", e.domain, err.Error()) } e.cname = cname @@ -127,7 +127,7 @@ func doResolve(e *serviceEntry) error { hosts, err := e.resolver.LookupHost(e.ctx, e.domain) if err != nil { - return fmt.Errorf("Cannot resolve CNAME for domain %q: %s", + return fmt.Errorf("failed to resolve CNAME for domain %q: %s", e.domain, err.Error()) } @@ -175,7 +175,7 @@ func (r *dnsResolverImpl) Lookup(domain string) ([]net.IP, error) { e, found := r.register[domain] if !found { - return []net.IP{}, fmt.Errorf("Unknown domain name: %q", domain) + return []net.IP{}, fmt.Errorf("unknown domain name: %q", domain) } e.lock.RLock() diff --git a/internal/util/endpoint.go b/internal/util/endpoint.go index f33b466e..9d5d1961 100644 --- a/internal/util/endpoint.go +++ b/internal/util/endpoint.go @@ -33,11 +33,11 @@ func ParseEndpoint(ep string) (*Endpoint, error) { } else { port, err = strconv.Atoi(m[2]) if err != nil { - return nil, fmt.Errorf("Invalid port in endpoint %q: %w", ep, err) + return nil, fmt.Errorf("invalid port in endpoint %q: %w", ep, err) } endPort, err = strconv.Atoi(m[3]) if err != nil { - return nil, fmt.Errorf("Invalid end-port in endpoint %q: %w", ep, err) + return nil, fmt.Errorf("invalid end-port in endpoint %q: %w", ep, err) } cidr = m[1] hasPort = true @@ -56,7 +56,7 @@ func ParseEndpoint(ep string) (*Endpoint, error) { // convert to an actual prefix _, ipnet, err := net.ParseCIDR(cidr) if err != nil { - return nil, fmt.Errorf("Invalid endpoint %q: %w", ep, err) + return nil, fmt.Errorf("invalid endpoint %q: %w", ep, err) } return &Endpoint{ diff --git a/pkg/apis/v1/auth.go b/pkg/apis/v1/auth.go index 4f37c5a3..a46cba87 100644 --- a/pkg/apis/v1/auth.go +++ b/pkg/apis/v1/auth.go @@ -44,8 +44,7 @@ func (req *AuthConfig) Validate() error { case AuthTypeEphemeral: _, secretFound := req.Credentials["secret"] if !secretFound { - return fmt.Errorf("cannot handle auth config for type %s: invalid secret", - atype.String()) + return fmt.Errorf("no secret found in %s auth config", atype.String()) } default: return fmt.Errorf("invalid authentication type %q", req.Type) diff --git a/pkg/apis/v1/errors.go b/pkg/apis/v1/errors.go index a9b4f71b..2b5862d3 100644 --- a/pkg/apis/v1/errors.go +++ b/pkg/apis/v1/errors.go @@ -7,10 +7,10 @@ import ( ) var ( - ErrInvalidConf = errors.New("Invalid configuration") - ErrNoSuchListener = errors.New("No such listener") - ErrNoSuchCluster = errors.New("No such cluster") - // ErrInvalidRoute = errors.New("Invalid route") + ErrInvalidConf = errors.New("invalid configuration") + ErrNoSuchListener = errors.New("no such listener") + ErrNoSuchCluster = errors.New("no such cluster") + // ErrInvalidRoute = errors.New("invalid route") ) type ErrRestarted struct { diff --git a/pkg/config/client/client.go b/pkg/config/client/client.go index d54a843d..9becd437 100644 --- a/pkg/config/client/client.go +++ b/pkg/config/client/client.go @@ -11,7 +11,7 @@ import ( "github.com/pion/logging" ) -var errFileTruncated = errors.New("Zero-length config file") +var errFileTruncated = errors.New("zero-length config file") var ( // Send pings to the CDS server with this period. Must be less than PongWait. @@ -49,7 +49,7 @@ type Client interface { func New(origin string, id string, logger logging.LoggerFactory) (Client, error) { u, err := getURI(origin) if err != nil { - return nil, fmt.Errorf("could not parse config origin URI %q: %w", origin, err) + return nil, fmt.Errorf("failed to parse config origin URI %q: %w", origin, err) } switch strings.ToLower(u.Scheme) { diff --git a/pkg/config/client/file_client.go b/pkg/config/client/file_client.go index 0ead3bfc..adb0e8d8 100644 --- a/pkg/config/client/file_client.go +++ b/pkg/config/client/file_client.go @@ -45,7 +45,7 @@ func (w *ConfigFileClient) String() string { func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { b, err := os.ReadFile(w.configFile) if err != nil { - return nil, fmt.Errorf("could not read config file %q: %s", w.configFile, err.Error()) + return nil, fmt.Errorf("failed to read config file %q: %s", w.configFile, err.Error()) } if len(b) == 0 { @@ -54,7 +54,7 @@ func (w *ConfigFileClient) Load() (*stnrv1.StunnerConfig, error) { c, err := ParseConfig(b) if err != nil { - return nil, fmt.Errorf("could not parse config: %w", err) + return nil, fmt.Errorf("failed to parse config: %w", err) } if err := c.Validate(); err != nil { @@ -128,7 +128,7 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo case e, ok := <-watcher.Events: if !ok { - return errors.New("Config watcher received invalid event") + return errors.New("config watcher received invalid event") } w.log.Debugf("Received watch event: %s", e.String()) @@ -139,7 +139,7 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo config, err.Error()) } - return fmt.Errorf("Config file deleted %q, disabling watcher", e.Op.String()) + return fmt.Errorf("config file deleted %q, disabling watcher", e.Op.String()) } if !e.Has(fsnotify.Write) { @@ -178,7 +178,7 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo case err, ok := <-watcher.Errors: if !ok { - return errors.New("Config watcher error handler received invalid error") + return errors.New("config watcher error handler received invalid error") } if err := watcher.Remove(config); err != nil { @@ -186,7 +186,7 @@ func (w *ConfigFileClient) Poll(ctx context.Context, ch chan<- *stnrv1.StunnerCo config, err.Error()) } - return fmt.Errorf("Config watcher error, deactivating: %w", err) + return fmt.Errorf("config watcher error, deactivating: %w", err) } } } diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go index f32db5a7..c18752a1 100644 --- a/pkg/config/client/k8s_client.go +++ b/pkg/config/client/k8s_client.go @@ -237,7 +237,7 @@ func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, appLabel, err := labels.NewRequirement(stnrv1.DefaultAppLabelKey, selection.Equals, []string{stnrv1.DefaultAppLabelValue}) if err != nil { - return ps, fmt.Errorf("cannot create app label selector: %w", err) + return ps, fmt.Errorf("failed to create app label selector: %w", err) } selector = selector.Add(*appLabel) @@ -245,7 +245,7 @@ func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, nsLabel, err := labels.NewRequirement(stnrv1.DefaultRelatedGatewayNamespace, selection.Equals, []string{gwNs}) if err != nil { - return ps, fmt.Errorf("cannot create namespace label selector: %w", err) + return ps, fmt.Errorf("failed to create namespace label selector: %w", err) } selector = selector.Add(*nsLabel) @@ -253,7 +253,7 @@ func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, gwLabel, err := labels.NewRequirement(stnrv1.DefaultRelatedGatewayKey, selection.Equals, []string{gw}) if err != nil { - return ps, fmt.Errorf("cannot create namespace label selector: %w", err) + return ps, fmt.Errorf("failed to create namespace label selector: %w", err) } selector = selector.Add(*gwLabel) } @@ -344,7 +344,7 @@ func DiscoverK8sAuthServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, au LabelSelector: label, }) if err != nil { - return PodInfo{}, fmt.Errorf("Failed to query Kubernetes API server: %w", err) + return PodInfo{}, fmt.Errorf("failed to query Kubernetes API server: %w", err) } if len(pods.Items) == 0 { @@ -376,7 +376,7 @@ func (d *PodConnector) portfwd(ctx context.Context, pod *corev1.Pod, port int) ( d.log.Debugf("Creating a SPDY stream to API server using URL %q", req.URL().String()) transport, upgrader, err := spdy.RoundTripperFor(d.config) if err != nil { - return PodInfo{}, fmt.Errorf("error getting transport/upgrader from restconfig: %w", err) + return PodInfo{}, fmt.Errorf("failed to get transport/upgrader from restconfig: %w", err) } dialer := spdy.NewDialer(upgrader, &http.Client{Transport: transport}, http.MethodPost, req.URL()) @@ -387,12 +387,12 @@ func (d *PodConnector) portfwd(ctx context.Context, pod *corev1.Pod, port int) ( out, errOut := new(bytes.Buffer), new(bytes.Buffer) fw, err := portforward.New(dialer, []string{remoteAddr}, stopChan, readyChan, out, errOut) if err != nil { - return PodInfo{}, fmt.Errorf("Error creating port-forwarder: %w", err) + return PodInfo{}, fmt.Errorf("failed to create port-forwarder: %w", err) } go func() { if err := fw.ForwardPorts(); err != nil { - d.log.Errorf("Error setting up port-forwarder: %s", err.Error()) + d.log.Errorf("failed to set up port-forwarder: %s", err.Error()) os.Exit(1) } }() diff --git a/pkg/whipconn/dialer.go b/pkg/whipconn/dialer.go index ca2252d0..c447a945 100644 --- a/pkg/whipconn/dialer.go +++ b/pkg/whipconn/dialer.go @@ -49,13 +49,13 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) ICETransportPolicy: d.config.ICETransportPolicy, }) if err != nil { - return nil, fmt.Errorf("Failed to create PeerConnection: %w", err) + return nil, fmt.Errorf("failed to create PeerConnection: %w", err) } d.log.Trace("Creating DataChannel") dataChannel, err := peerConn.CreateDataChannel("whipconn", nil) if err != nil { - return nil, fmt.Errorf("Failed to create DataChannel: %w", err) + return nil, fmt.Errorf("failed to create DataChannel: %w", err) } conn := &dialerConn{ @@ -77,7 +77,7 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) raw, err := dataChannel.Detach() if err != nil { - errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) + errCh <- fmt.Errorf("failed to detach DataChannel: %w", err) } conn.dataConn = raw @@ -95,13 +95,13 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) offer, err := peerConn.CreateOffer(nil) if err != nil { conn.Close() //nolint - return nil, fmt.Errorf("Failed to create offer: %w", err) + return nil, fmt.Errorf("failed to create offer: %w", err) } err = peerConn.SetLocalDescription(offer) if err != nil { conn.Close() //nolint - return nil, fmt.Errorf("Failed to set local SDP (Offer): %w", err) + return nil, fmt.Errorf("failed to set local SDP (Offer): %w", err) } // Block until ICE Gathering is complete, disabling trickle ICE @@ -118,7 +118,7 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) makeURL(addr, d.config.Endpoint).String(), bytes.NewBuffer(sdp)) if err != nil { conn.Close() //nolint - return nil, fmt.Errorf("Unexpected error building HTTP request: %w", err) + return nil, fmt.Errorf("unexpected error building HTTP request: %w", err) } req.Header.Add("Content-Type", "application/sdp") @@ -129,7 +129,7 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) resp, err := http.DefaultClient.Do(req) if err != nil { conn.Close() //nolint - return nil, fmt.Errorf("Failed to POST WHIP request: %w", err) + return nil, fmt.Errorf("failed to POST WHIP request: %w", err) } d.log.Tracef("Received POST response with status code: %d", resp.StatusCode) @@ -142,14 +142,14 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) body, err := io.ReadAll(resp.Body) if err != nil { conn.Close() //nolint - return nil, fmt.Errorf("Failed to read HTTP response body: %w", err) + return nil, fmt.Errorf("failed to read HTTP response body: %w", err) } defer resp.Body.Close() resourceId := resp.Header.Get("Location") if resourceId == "" { conn.Close() //nolint - return nil, errors.New("Empty resource id in POST response") + return nil, errors.New("empty resource id in POST response") } conn.resourceId = resourceId @@ -160,7 +160,7 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) err = peerConn.SetRemoteDescription(answer) if err != nil { conn.Close() //nolint - return nil, fmt.Errorf("Failed to set remote SDP (Answer): %w", err) + return nil, fmt.Errorf("failed to set remote SDP (Answer): %w", err) } // Waiting for the connection or errors surfaced from the callbacks @@ -198,19 +198,19 @@ func (c *dialerConn) Close() error { uri := makeURL(c.addr, makeResourceURL(c.dialer.config.Endpoint, c.resourceId)) req, err := http.NewRequest("DELETE", uri.String(), nil) if err != nil { - return fmt.Errorf("Unexpected error building http request: %w", err) + return fmt.Errorf("unexpected error building http request: %w", err) } if c.dialer.config.Token != "" { req.Header.Add("Authorization", "Bearer "+c.dialer.config.Token) } if _, err = http.DefaultClient.Do(req); err != nil { - return fmt.Errorf("Failed WHIP DELETE request: %w", err) + return fmt.Errorf("failed WHIP DELETE request: %w", err) } // Close the peerconnection if err := c.peerConn.Close(); err != nil { - return fmt.Errorf("Failed to close PeerConnection: %w", err) + return fmt.Errorf("failed to close PeerConnection: %w", err) } return nil diff --git a/pkg/whipconn/listener.go b/pkg/whipconn/listener.go index c9169a1f..86be4447 100644 --- a/pkg/whipconn/listener.go +++ b/pkg/whipconn/listener.go @@ -68,7 +68,7 @@ func NewListener(addr string, config Config, logger logging.LoggerFactory) (*Lis c, err := net.Listen("tcp", addr) if err != nil { - return nil, fmt.Errorf("Cannot open WHIP server socket on %s: %w", addr, err) + return nil, fmt.Errorf("failed to open WHIP server socket on %s: %w", addr, err) } l.server = &http.Server{Addr: addr, Handler: mux} go func() { @@ -135,7 +135,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { // Check bearer token if l.config.Token != "" { if token := r.Header.Get("Authorization"); token != "Bearer "+l.config.Token { - err := fmt.Errorf("Unauthorized WHIP request from client %s", r.RemoteAddr) + err := fmt.Errorf("unauthorized WHIP request from client %s", r.RemoteAddr) l.errCh <- err http.Error(w, err.Error(), http.StatusUnauthorized) return @@ -143,7 +143,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { } if ctype := r.Header.Get("Content-Type"); ctype != "application/sdp" { - err := fmt.Errorf("Invalid WHIP request from client %s, expected Content-Type=application/sdp", + err := fmt.Errorf("invalid WHIP request from client %s, expected Content-Type=application/sdp", r.RemoteAddr) l.errCh <- err http.Error(w, err.Error(), http.StatusBadRequest) @@ -154,7 +154,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { offer, err := io.ReadAll(r.Body) defer r.Body.Close() if err != nil { - l.errCh <- fmt.Errorf("Failed to read WHIP request body: %w", err) + l.errCh <- fmt.Errorf("failed to read WHIP request body: %w", err) return } @@ -169,7 +169,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { ICETransportPolicy: l.config.ICETransportPolicy, }) if err != nil { - l.errCh <- fmt.Errorf("Failed to create a PeerConnection: %w", err) + l.errCh <- fmt.Errorf("failed to create a PeerConnection: %w", err) return } conn.peerConn = peerConn @@ -192,7 +192,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { raw, dErr := dataChannel.Detach() if dErr != nil { - l.errCh <- fmt.Errorf("Failed to detach DataChannel: %w", err) + l.errCh <- fmt.Errorf("failed to detach DataChannel: %w", err) return } conn.dataConn = raw @@ -208,7 +208,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { Type: webrtc.SDPTypeOffer, SDP: string(offer), }); err != nil { - l.errCh <- fmt.Errorf("Failed to set remote SDP (Offer): %w", err) + l.errCh <- fmt.Errorf("failed to set remote SDP (Offer): %w", err) return } @@ -218,10 +218,10 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { // Create answer answer, err := peerConn.CreateAnswer(nil) if err != nil { - l.errCh <- fmt.Errorf("Failed to create SDP (Answer): %w", err) + l.errCh <- fmt.Errorf("failed to create SDP (Answer): %w", err) return } else if err = peerConn.SetLocalDescription(answer); err != nil { - l.errCh <- fmt.Errorf("Failed to set local SDP (Answer): %w", err) + l.errCh <- fmt.Errorf("failed to set local SDP (Answer): %w", err) return } diff --git a/relay.go b/relay.go index c93a1f1f..fa9e3036 100644 --- a/relay.go +++ b/relay.go @@ -21,13 +21,13 @@ import ( const ClusterCacheSize = 512 var ( - errNilConn = errors.New("Cannot allocate relay connection") - errTodo = errors.New("Relay to Net.Conn not implemented") + errNilConn = errors.New("cannot allocate relay connection") + errTodo = errors.New("relay to Net.Conn not implemented") ) var ( - ErrPortProhibited = errors.New("Peer port administratively prohibited") - ErrInvalidPeerProtocol = errors.New("Unknown peer transport protocol") + ErrPortProhibited = errors.New("peer port administratively prohibited") + ErrInvalidPeerProtocol = errors.New("unknown peer transport protocol") ) type PortRangeChecker = func(addr net.Addr) (*object.Cluster, bool) diff --git a/turncat.go b/turncat.go index b8872d0d..7dd9dcce 100644 --- a/turncat.go +++ b/turncat.go @@ -241,7 +241,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { user, passwd, errAuth := t.authGen() if errAuth != nil { - return nil, fmt.Errorf("Cannot generate username/password pair for client %s:%s: %s", + return nil, fmt.Errorf("failed to generate username/password pair for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), errAuth) } @@ -251,14 +251,14 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { case "turn-udp": t, err := net.ListenPacket(t.serverAddr.Network(), "0.0.0.0:0") if err != nil { - return nil, fmt.Errorf("cannot allocate TURN listening packet socket for client %s:%s: %s", + return nil, fmt.Errorf("failed to allocate TURN listening packet socket for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } turnConn = t case "turn-tcp": c, err := net.Dial(t.serverAddr.Network(), t.serverAddr.String()) if err != nil { - return nil, fmt.Errorf("cannot allocate TURN socket for client %s:%s: %s", + return nil, fmt.Errorf("failed to allocate TURN socket for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } turnConn = turn.NewSTUNConn(c) @@ -271,7 +271,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { InsecureSkipVerify: t.insecure, }) if err != nil { - return nil, fmt.Errorf("cannot allocate TURN/TLS socket for client %s:%s: %s", + return nil, fmt.Errorf("failed to allocate TURN/TLS socket for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } turnConn = turn.NewSTUNConn(c) @@ -283,7 +283,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { InsecureSkipVerify: t.insecure, }) if err != nil { - return nil, fmt.Errorf("cannot allocate TURN/DTLS socket for client %s:%s: %s", + return nil, fmt.Errorf("failed to allocate TURN/DTLS socket for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } turnConn = turn.NewSTUNConn(conn) @@ -303,7 +303,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { }) if err != nil { turnConn.Close() - return nil, fmt.Errorf("Cannot allocate TURN client for client %s:%s: %s", + return nil, fmt.Errorf("failed to allocate TURN client for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), err) } conn.turnConn = turnConn @@ -311,7 +311,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { // Start the TURN client if err = turnClient.Listen(); err != nil { turnConn.Close() - return nil, fmt.Errorf("cannot listen on TURN client: %s", err) + return nil, fmt.Errorf("failed to listen on TURN client: %s", err) } conn.turnClient = turnClient @@ -320,7 +320,7 @@ func (t *Turncat) newConnection(clientConn net.Conn) (*connection, error) { serverConn, serverErr := turnClient.Allocate() if serverErr != nil { turnClient.Close() - return nil, fmt.Errorf("could not allocate new TURN relay transport for client %s:%s: %s", + return nil, fmt.Errorf("failed to allocate TURN relay transport for client %s:%s: %s", clientAddr.Network(), clientAddr.String(), serverErr.Error()) } conn.serverConn = serverConn diff --git a/uri.go b/uri.go index d4ac5817..3258419e 100644 --- a/uri.go +++ b/uri.go @@ -34,7 +34,7 @@ func ParseUri(uri string) (*StunnerUri, error) { u, err := url.Parse(uri) if err != nil { - return nil, fmt.Errorf("Invalid URI '%s': %s", uri, err) + return nil, fmt.Errorf("invalid URI '%s': %s", uri, err) } s.Address = u.Hostname() @@ -163,5 +163,5 @@ func getStunnerProtoForURI(u *url.URL) (string, error) { return "TURN-TLS", nil } - return "", fmt.Errorf("Invalid scheme/protocol in URI %q", u.String()) + return "", fmt.Errorf("invalid scheme/protocol in URI %q", u.String()) } From f48067f973428b730d2ab31e75424fd8a757f6e4 Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 28 Nov 2024 20:59:33 +0100 Subject: [PATCH 237/243] feature: Implement ICE tester in stunnerctl --- Dockerfile.icetester | 39 ++ Makefile | 4 +- cmd/icetester/README.md | 42 ++ cmd/icetester/icetester_test.go | 84 ++++ cmd/icetester/main.go | 166 ++++++++ cmd/stunnerctl/README.md | 63 ++- cmd/stunnerctl/auth.go | 67 ++++ cmd/stunnerctl/config.go | 123 ++++++ cmd/stunnerctl/icetest.go | 187 +++++++++ cmd/stunnerctl/main.go | 358 +++-------------- cmd/stunnerctl/status.go | 118 ++++++ cmd/turncat/main.go | 2 +- go.mod | 28 +- go.sum | 73 ++-- internal/icetester/artifacts.go | 398 +++++++++++++++++++ internal/icetester/events.go | 75 ++++ internal/icetester/floodtest.go | 136 +++++++ internal/icetester/icetester.go | 685 ++++++++++++++++++++++++++++++++ internal/icetester/utils.go | 183 +++++++++ pkg/apis/v1/default.go | 1 + pkg/config/client/k8s_client.go | 32 +- pkg/logger/logger.go | 58 ++- pkg/logger/logger_test.go | 4 +- pkg/whipconn/config.go | 8 +- pkg/whipconn/dialer.go | 66 +-- pkg/whipconn/listener.go | 134 ++++--- pkg/whipconn/whipconn_test.go | 121 +++++- relay.go | 4 +- server.go | 3 +- stunner.go | 2 +- 30 files changed, 2801 insertions(+), 463 deletions(-) create mode 100644 Dockerfile.icetester create mode 100644 cmd/icetester/README.md create mode 100644 cmd/icetester/icetester_test.go create mode 100644 cmd/icetester/main.go create mode 100644 cmd/stunnerctl/auth.go create mode 100644 cmd/stunnerctl/config.go create mode 100644 cmd/stunnerctl/icetest.go create mode 100644 cmd/stunnerctl/status.go create mode 100644 internal/icetester/artifacts.go create mode 100644 internal/icetester/events.go create mode 100644 internal/icetester/floodtest.go create mode 100644 internal/icetester/icetester.go create mode 100644 internal/icetester/utils.go diff --git a/Dockerfile.icetester b/Dockerfile.icetester new file mode 100644 index 00000000..ce243532 --- /dev/null +++ b/Dockerfile.icetester @@ -0,0 +1,39 @@ +########### +# BUILD +FROM docker.io/golang:1.23-alpine as builder + +WORKDIR /app + +COPY go.mod ./ +COPY go.sum ./ + +COPY *.go ./ +COPY internal/ internal/ +COPY pkg/ pkg/ + +COPY cmd/ cmd/ + +COPY .git ./ +COPY Makefile ./ +RUN apk add --no-cache git make + +RUN apkArch="$(apk --print-arch)"; \ + case "$apkArch" in \ + aarch64) export GOARCH='arm64' ;; \ + *) export GOARCH='amd64' ;; \ + esac; \ + export CGO_ENABLED=0; \ + export GOOS=linux; \ + make build-bin + +########### +# STUNNERD +FROM scratch + +WORKDIR /app + +COPY --from=builder /app/bin/icetester /usr/bin/ + +EXPOSE 8089/tcp + +CMD [ "icetester", "-l", "all:INFO" ] diff --git a/Makefile b/Makefile index 3da34954..15838cab 100644 --- a/Makefile +++ b/Makefile @@ -39,10 +39,12 @@ test: generate fmt vet build: generate fmt vet build-bin .PHONY: build-bin +bin: build-bin build-bin: go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerd cmd/stunnerd/main.go go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/turncat cmd/turncat/main.go - go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerctl cmd/stunnerctl/main.go + go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/stunnerctl cmd/stunnerctl/*.go + go build ${GOARGS} -ldflags "${LDFLAGS}" -o ${BUILD_DIR}/icetester cmd/icetester/main.go .PHONY: clean clean: diff --git a/cmd/icetester/README.md b/cmd/icetester/README.md new file mode 100644 index 00000000..e75fbf69 --- /dev/null +++ b/cmd/icetester/README.md @@ -0,0 +1,42 @@ +# icetester: Universal UDP echo service using WebRTC/ICE + +`icetester` is test server that can be used WebRTC/ICE connectivity. The tester serves a simple +WebSocket/JSON API server that clients can use to create a WebRTC data channel. whatever is +received by `icetester` on the data channel will be echoed back to the client over the data channel. + +While `icetester` can be used as a standalone too, the intended use is via `stunnerctl icetest`. + +## Installation + +Install `icetest` using the standard Go toolchain and add it to `$PATH`. + +```console +go install github.com/l7mp/stunner/cmd/icetest@latest +``` + +Building from source is as easy as it usually gets with Go: + +```console +cd stunner +go build -o turncat cmd/turncat/main.go +``` + +The containerized version is available as `docker.io/l7mp/icester`. + +## Usage + +Deploy a STUNner gateway and test is via UDP and TCP through `stunnerctl`: + +```console +stunnerctl icetest +``` + +## License + +Copyright 2021-2024 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). + +MIT License - see [LICENSE](../../LICENSE) for full text. + +## Acknowledgments + +Initial code adopted from [pion/stun](https://github.com/pion/stun) and [pion/turn](https://github.com/pion/turn). diff --git a/cmd/icetester/icetester_test.go b/cmd/icetester/icetester_test.go new file mode 100644 index 00000000..2cc53bc3 --- /dev/null +++ b/cmd/icetester/icetester_test.go @@ -0,0 +1,84 @@ +package main + +import ( + "context" + "net" + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/l7mp/stunner/pkg/logger" + "github.com/l7mp/stunner/pkg/whipconn" +) + +var ( + testerLogLevel = "all:WARN" + // testerLogLevel = "all:TRACE" + // testerLogLevel = "all:INFO" + defaultConfig = whipconn.Config{BearerToken: "whiptoken"} +) + +func echoTest(t *testing.T, conn net.Conn, content string) { + t.Helper() + + n, err := conn.Write([]byte(content)) + assert.NoError(t, err) + assert.Equal(t, len(content), n) + + buf := make([]byte, 2048) + n, err = conn.Read(buf) + assert.NoError(t, err) + assert.Equal(t, content, string(buf[:n])) +} + +var testerTestCases = []struct { + name string + config *whipconn.Config + tester func(t *testing.T, ctx context.Context) +}{ + { + name: "Basic connectivity", + tester: func(t *testing.T, ctx context.Context) { + log.Debug("Creating dialer") + d := whipconn.NewDialer(defaultConfig, loggerFactory) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(ctx, defaultICETesterAddr) + assert.NoError(t, err) + + log.Debug("Echo test round 1") + echoTest(t, clientConn, "test1") + log.Debug("Echo test round 2") + echoTest(t, clientConn, "test2") + + assert.NoError(t, clientConn.Close(), "client conn close") + }, + }, +} + +func TestICETesterConn(t *testing.T) { + loggerFactory = logger.NewLoggerFactory(testerLogLevel) + log = loggerFactory.NewLogger("icester") + + for _, c := range testerTestCases { + t.Run(c.name, func(t *testing.T) { + log.Infof("--------------------- %s ----------------------", c.name) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + config := defaultConfig + if c.config != nil { + config = *c.config + } + + log.Debug("Running listener loop") + go func() { + err := runICETesterListener(ctx, defaultICETesterAddr, config) + assert.NoError(t, err) + }() + + c.tester(t, ctx) + }) + } +} diff --git a/cmd/icetester/main.go b/cmd/icetester/main.go new file mode 100644 index 00000000..a40b9c51 --- /dev/null +++ b/cmd/icetester/main.go @@ -0,0 +1,166 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net" + "net/http" + "os" + "os/signal" + + "github.com/pion/logging" + "github.com/pion/webrtc/v4" + flag "github.com/spf13/pflag" + + v1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/buildinfo" + "github.com/l7mp/stunner/pkg/logger" + "github.com/l7mp/stunner/pkg/whipconn" +) + +const ( + // Name of the environment variable specifying the list of ICE servers, default is no ICE servers. + EnvVarNameICEServers = "ICE_SERVERS" + + // Name of the environment variable specifying the ICE transport policy (either "relay" or "all"), default is "all". + EnvVarNameICETransportPolicy = "ICE_TRANSPORT_POLICY" + + // HIP bearer token for authenticating WHIP requests, default is no bearer token. + EnvVarNameBearerToken = "BEARER_TOKEN" + + // WHIP API endpoint, default is "/whip". Must include the leading slash ("/"). + EnvVarNameWHIPEndpoint = "WHIP_ENDPOINT" +) + +var ( + version = "dev" + commitHash = "n/a" + buildDate = "" + + defaultICEServers = []webrtc.ICEServer{} + defaultICETransportPolicy = webrtc.NewICETransportPolicy("all") + defaultBearerToken = "" + defaultWHIPEndpoint = "/whip" + defaultICETesterAddr = fmt.Sprintf(":%d", v1.DefaultICETesterPort) + + loggerFactory logging.LoggerFactory + log logging.LeveledLogger +) + +func main() { + os.Args[0] = "icetester" + var whipServerAddr = flag.StringP("addr", "a", defaultICETesterAddr, "WHIP server listener address") + var level = flag.StringP("log", "l", "all:WARN", "Log level") + var verbose = flag.BoolP("verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") + + flag.Parse() + + if *verbose { + *level = "all:DEBUG" + } + + loggerFactory = logger.NewLoggerFactory(*level) + log = loggerFactory.NewLogger("icester") + + buildInfo := buildinfo.BuildInfo{Version: version, CommitHash: commitHash, BuildDate: buildDate} + log.Debugf("Starting icetester %s", buildInfo.String()) + + iceServers := defaultICEServers + if os.Getenv(EnvVarNameICEServers) != "" { + s := []webrtc.ICEServer{} + if err := json.Unmarshal([]byte(os.Getenv(EnvVarNameICEServers)), &s); err != nil { + log.Errorf("Environment ICE_SERVERS is invalid: %s", err.Error()) + os.Exit(1) + } + iceServers = s + } + + iceTransportPolicy := defaultICETransportPolicy + if os.Getenv(EnvVarNameICETransportPolicy) != "" { + iceTransportPolicy = webrtc.NewICETransportPolicy(os.Getenv(EnvVarNameICETransportPolicy)) + } + + token := defaultBearerToken + if os.Getenv(EnvVarNameBearerToken) != "" { + token = os.Getenv(EnvVarNameBearerToken) + } + + whipEndpoint := defaultWHIPEndpoint + if os.Getenv(EnvVarNameWHIPEndpoint) != "" { + endpoint := os.Getenv(EnvVarNameWHIPEndpoint) + if endpoint[0] != '/' { + log.Errorf("Environment WHIP_ENDPOINT is invalid: %s, expecting a leading slash '/'", endpoint) + os.Exit(1) + } + whipEndpoint = endpoint + } + + whipServerConfig := whipconn.Config{ + ICEServers: iceServers, + ICETransportPolicy: iceTransportPolicy, + BearerToken: token, + WHIPEndpoint: whipEndpoint, + } + + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() + + if err := runICETesterListener(ctx, *whipServerAddr, whipServerConfig); err != nil { + log.Errorf("Could not create WHIP server listener: %s", err.Error()) + os.Exit(1) + } + + os.Exit(0) +} + +func runICETesterListener(ctx context.Context, addr string, config whipconn.Config) error { + log.Infof("Creating WHIP server listener with config %#v", config) + l, err := whipconn.NewListener(addr, config, loggerFactory) + if err != nil { + return fmt.Errorf("Could not create WHIP server listener: %s", err.Error()) + } + + log.Debug("Creating echo service") + go func() { + for { + conn, err := l.Accept() + if err != nil { + return + } + + log.Debugf("Accepting WHIP server connection with resource ID: %s", + conn.(*whipconn.ListenerConn).ResourceUrl) + + // readloop + go func() { + buf := make([]byte, 100) + for { + n, err := conn.Read(buf) + if err != nil { + return + } + + _, err = conn.Write(buf[:n]) + if err != nil { + return + } + } + }() + } + }() + + <-ctx.Done() + + for _, conn := range l.GetConns() { + if err := conn.Close(); err != nil && !errors.Is(err, net.ErrClosed) && + !errors.Is(err, http.ErrServerClosed) { + return fmt.Errorf("WHIP connection close error: %s", err.Error()) + } + } + + l.Close() + + return nil +} diff --git a/cmd/stunnerctl/README.md b/cmd/stunnerctl/README.md index 7b63962c..02079af1 100644 --- a/cmd/stunnerctl/README.md +++ b/cmd/stunnerctl/README.md @@ -1,11 +1,11 @@ # stunnerctl: Command line toolbox for STUNner -A CLI tool to simplify the interaction with STUNner. +A CLI tool to simplify the interaction with STUNner. The prominent use of `stunnerctl` is to load or watch STUNner dataplane configurations from a Kubernetes cluster for debugging and troubleshooting, or just for checking whether everything is configured the way it should be. ## Installation -Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. +Install the `stunnerctl` binary using the standard Go toolchain and add it to `$PATH`. ```console go install github.com/l7mp/stunner/cmd/stunnerctl@latest @@ -26,7 +26,7 @@ go build -o stunnerctl cmd/stunnerctl/main.go ## Usage -Type `stunnerctl` to get a glimpse of the sub-commands and features provided. +Type `stunnerctl` to get a glimpse of the sub-commands and features provided. ### Config @@ -70,7 +70,7 @@ The `config` sub-command is used to load or watch running dataplane configs from ``` For those who don't have the Go toolchain available to run `go install`, STUNner provides a minimalistic `stunnerctl` replacement called `stunnerctl.sh`. -This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. +This script requires nothing else than `bash`, `kubectl`, `curl` and `jq` to work. The below will dump the running config of `tcp-gateway` deployed into the `stunner` namespace: @@ -154,6 +154,61 @@ The `auth` sub-command can be used to obtain a TURN credential or a full ICE ser {"password":"pass-1","ttl":86400,"uris":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"} ``` +### ICE test + +The `icetest` sub-command can be used to run a full-blown ICE test. This command is intended for +users to check a STUNner installation and pinpoint installation errors. + +The tester will fire up a WHIP server in the cluster, configure a UDP and a TCP gateway to expose +it, makes a PeerConnection to the WHIP server via the gateways, and performs a quick test by +sending a set of packets via a data channel created over the PeerConnection and measures loss and +latency using the packets echoed back by the WHIP server. If successful, the tester will output the +measured statistics, otherwise it reports the error that stopped the ICE test and provides some +diagnostics that to help troubleshooting. + +- Run a dataplane test over UDP and TCP: + + ``` console + stunnerctl icetest + Initializing... completed + Checking installation... completed + Checking Gateway... completed + Obtaining ICE server configuration... completed + Running asymmetric ICE test over TURN-UDP... completed + Statistics: rate=48.65pps, loss=0/973pkts=0.00%, RTT:mean=20.67ms/median=20.54ms/P95=22.23ms/P99=23.34ms + LocalICECandidates: + * udp4 relay 10.244.0.24:43988 related 0.0.0.0:43716 (resolved: 10.244.0.24:43988) + RemoteICECandidates: + * udp4 host 10.244.0.163:35242 (resolved: 10.244.0.163:35242) + Running asymmetric ICE test over TURN-TCP... completed + Statistics: rate=48.55pps, loss=0/971pkts=0.00%, RTT:mean=21.00ms/median=20.89ms/P95=22.45ms/P99=23.47ms + LocalICECandidates: + * udp4 relay 10.244.0.162:45090 related 0.0.0.0:45654 (resolved: 10.244.0.162:45090) + RemoteICECandidates: + * udp4 host 10.244.0.163:51653 (resolved: 10.244.0.163:51653) + Running symmetric ICE test over TURN-UDP... completed + Statistics: rate=48.65pps, loss=0/973pkts=0.00%, RTT:mean=20.63ms/median=20.47ms/P95=21.85ms/P99=23.01ms + LocalICECandidates: + * udp4 relay 10.244.0.24:47282 related 0.0.0.0:55122 (resolved: 10.244.0.24:47282) + RemoteICECandidates: + * udp4 relay 10.244.0.24:47367 related 0.0.0.0:51777 (resolved: 10.244.0.24:47367) + Running symmetric ICE test over TURN-TCP... completed + Statistics: rate=48.60pps, loss=0/972pkts=0.00%, RTT:mean=24.61ms/median=20.56ms/P95=39.96ms/P99=133.40ms + LocalICECandidates: + * udp4 relay 10.244.0.162:56555 related 0.0.0.0:42600 (resolved: 10.244.0.162:56555) + RemoteICECandidates: + * udp4 relay 10.244.0.162:33397 related 10.244.0.163:53124 (resolved: 10.244.0.162:33397) + ``` + +- Clean up the Kubernetes resources the tester might have left behind on a previous run and perform + the test only on TURN-UDP with at a rate of 100 packets per second using a 2 minute timeout: + + ``` console + stunnerctl icetest --force-cleanup -packet-rate 100 --timeout 2m udp + ``` + +Run `stunnerctl icetest --help` for further useful command line arguments. + ## License Copyright 2021-2023 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). diff --git a/cmd/stunnerctl/auth.go b/cmd/stunnerctl/auth.go new file mode 100644 index 00000000..61efc6fa --- /dev/null +++ b/cmd/stunnerctl/auth.go @@ -0,0 +1,67 @@ +package main + +import ( + "context" + "fmt" + "io" + "net/http" + "net/url" + + "github.com/spf13/cobra" + + cdsclient "github.com/l7mp/stunner/pkg/config/client" +) + +func runAuth(_ *cobra.Command, args []string) error { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + log.Debug("Searching for authentication server") + pod, err := cdsclient.DiscoverK8sAuthServer(ctx, k8sConfigFlags, authConfigFlags, + loggerFactory.NewLogger("auth-fwd")) + if err != nil { + return fmt.Errorf("error searching for auth service: %w", err) + } + + u := url.URL{ + Scheme: "http", + Host: pod.Addr, + Path: "/ice", + } + q := u.Query() + q.Set("service", "turn") + u.RawQuery = q.Encode() + + if authConfigFlags.TurnAuth { + // enforce TURN credential format + u.Path = "" + } + + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { + q := u.Query() + q.Set("namespace", *k8sConfigFlags.Namespace) + if len(args) > 0 { + q.Set("gateway", args[0]) + } + u.RawQuery = q.Encode() + } + + log.Debugf("Querying to authentication server %s using URL %q", pod.String(), u.String()) + res, err := http.Get(u.String()) + if err != nil { + return fmt.Errorf("error querying auth service %s: %w", pod.String(), err) + } + if res.StatusCode != http.StatusOK { + return fmt.Errorf("HTTP error querying auth service %s: expected status %d, got %d", + pod.String(), http.StatusOK, res.StatusCode) + } + + b, err := io.ReadAll(res.Body) + if err != nil { + return fmt.Errorf("cannot read HTTP response: %w", err) + } + + fmt.Println(string(b)) + + return nil +} diff --git a/cmd/stunnerctl/config.go b/cmd/stunnerctl/config.go new file mode 100644 index 00000000..a77da115 --- /dev/null +++ b/cmd/stunnerctl/config.go @@ -0,0 +1,123 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "os" + "os/signal" + "syscall" + + "github.com/spf13/cobra" + "sigs.k8s.io/yaml" + + stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" +) + +func runConfig(_ *cobra.Command, args []string) error { + gwNs := "default" + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { + gwNs = *k8sConfigFlags.Namespace + } + + jsonQuery, output, err := ParseJSONPathFlag(output) + if err != nil { + return err + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + log.Debug("Searching for CDS server") + pod, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, + loggerFactory.NewLogger("cds-fwd")) + if err != nil { + return fmt.Errorf("error searching for CDS server: %w", err) + } + + log.Debugf("Connecting to CDS server: %s", pod.String()) + var cds cdsclient.CdsApi + cdslog := loggerFactory.NewLogger("cds-client") + if all { + cds, err = cdsclient.NewAllConfigsAPI(pod.Addr, cdslog) + } else if len(args) == 0 { + cds, err = cdsclient.NewConfigsNamespaceAPI(pod.Addr, gwNs, cdslog) + } else { + gwName := args[0] + cds, err = cdsclient.NewConfigNamespaceNameAPI(pod.Addr, gwNs, gwName, cdslog) + } + + if err != nil { + return fmt.Errorf("error creating CDS client: %w", err) + } + + confChan := make(chan *stnrv1.StunnerConfig, 8) + if watch { + err := cds.Watch(ctx, confChan, false) + if err != nil { + close(confChan) + return err + } + + go func() { + sigs := make(chan os.Signal, 1) + signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM) + <-sigs + close(confChan) + }() + } else { + resp, err := cds.Get(ctx) + if err != nil { + close(confChan) + return err + } + for _, c := range resp { + confChan <- c + } + + close(confChan) + } + + for c := range confChan { + if cdsclient.IsConfigDeleted(c) { + fmt.Printf("Gateway: %s \n", c.Admin.Name) + continue + } + switch output { + case "yaml": + if out, err := yaml.Marshal(c); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "json": + if out, err := json.Marshal(c); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "jsonpath": + values, err := jsonQuery.FindResults(c) + if err != nil { + return err + } + + if len(values) == 0 || len(values[0]) == 0 { + fmt.Println("") + } + + for arrIx := range values { + for valIx := range values[arrIx] { + fmt.Printf("%v\n", values[arrIx][valIx].Interface()) + } + } + case "summary": + fmt.Print(c.Summary()) + default: + fmt.Println(c.String()) + } + } + + return nil +} diff --git a/cmd/stunnerctl/icetest.go b/cmd/stunnerctl/icetest.go new file mode 100644 index 00000000..a2d81efe --- /dev/null +++ b/cmd/stunnerctl/icetest.go @@ -0,0 +1,187 @@ +package main + +import ( + "bytes" + "context" + "fmt" + "io" + "os" + "os/signal" + "sync" + "time" + + "github.com/spf13/cobra" + + "github.com/l7mp/stunner/internal/icetester" + v1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/logger" +) + +const DefaultTestNamespace = "icetest" + +func runICETest(_ *cobra.Command, args []string) error { + ns := DefaultTestNamespace + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { + ns = *k8sConfigFlags.Namespace + } + + turnTransports := []v1.ListenerProtocol{} + protos := args + if len(protos) == 0 { + // run all tests for all transports if no specific transport is provided + protos = []string{"udp", "tcp"} + } + for _, arg := range protos { + proto, err := v1.NewListenerProtocol(arg) + if err != nil { + return err + } + switch proto { + case v1.ListenerProtocolUDP: + turnTransports = append(turnTransports, v1.ListenerProtocolTURNUDP) + case v1.ListenerProtocolTCP: + turnTransports = append(turnTransports, v1.ListenerProtocolTURNTCP) + case v1.ListenerProtocolTURNUDP, v1.ListenerProtocolTURNTCP: + turnTransports = append(turnTransports, proto) + default: + return fmt.Errorf("ICE test is currently not available on TURN transport protocol %s", proto) + } + } + + // Create a buffered logger for the tests + logBuffer := &bytes.Buffer{} + bufferedLoggerFactory := logger.NewLoggerFactory("all:TRACE") // hardcode highest loglevel + bufferedLoggerFactory.SetWriter(logBuffer) + + eventCh := make(chan icetester.Event, 12) + defer close(eventCh) + tester, err := icetester.NewICETester(icetester.Config{ + EventChannel: eventCh, + + K8sConfigFlags: k8sConfigFlags, + CDSConfigFlags: cdsConfigFlags, + AuthConfigFlags: authConfigFlags, + + Namespace: ns, + TURNTransports: turnTransports, + ICETesterImage: iceTesterImage, + ForceCleanup: forceCleanup, + PacketRate: iceTesterPacketRate, + + Logger: bufferedLoggerFactory, + }) + if err != nil { + return fmt.Errorf("Failed to create ICE tester: %w", err) + } + + // run for at most 5 minutes + ctx, cancel := context.WithTimeout(context.Background(), iceTesterTimeout) + + // stop on interrupt as well + ctx, stop := signal.NotifyContext(ctx, os.Interrupt) + defer stop() + + // event handler + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + for { + select { + case e := <-eventCh: + printEvent(e, logBuffer) + + case <-ctx.Done(): + return + } + } + }() + + err = tester.Start(ctx) + if err == nil { + err = ctx.Err() + } + + if err != nil { + switch ctx.Err() { + case context.DeadlineExceeded: + fmt.Printf("\nICE tester timed out after %s\n", iceTesterTimeout) + printLogs(logBuffer) + case context.Canceled: + fmt.Printf("\nICE tester stopped due to user interrupt\n") + default: + fmt.Printf("\nICE tester error: %s\n", err.Error()) + } + } + + cancel() + + // wait until the printer finishes + wg.Wait() + + return nil +} + +func printEvent(e icetester.Event, logbuf io.ReadWriter) { + // started + if e.InProgress { + proto := "" + if arg, ok := e.Args["ICETransport"]; ok { + if p, ok := arg.(string); ok { + proto = fmt.Sprintf(" over %s", p) + } + } + + fmt.Printf("%s: %s%s... ", e.Timestamp.Format(time.RFC822), e.Type.String(), proto) + return + } + + // completed successfully + if e.Error == nil { + fmt.Println("completed") + if arg, ok := e.Args["Stats"]; ok { + if s, ok := arg.(*icetester.Stats); ok { + fmt.Printf("\tStatistics: rate=%0.2fpps, loss=%d/%dpkts=%0.2f%%, "+ + "RTT:mean=%0.2fms/median=%0.2fms/P95=%0.2fms/P99=%0.2fms\n", + s.SendRate, s.PacketsSent-s.PacketsReceived, s.PacketsSent, + s.LossRate, s.MeanLatency, s.MedianLatency, s.P95Latency, + s.P99Latency) + } + } + + for _, ctype := range []string{"LocalICECandidates", "RemoteICECandidates"} { + if arg, ok := e.Args[ctype]; ok { + if cs, ok := arg.([]icetester.CandidateDesc); ok { + fmt.Printf("\t%s:\n", ctype) + for _, c := range cs { + sel := " " + if c.Selected { + sel = "* " + } + fmt.Printf("\t %s%s\n", sel, c.Candidate) + } + } + } + } + + return + } + + // completed with error + fmt.Printf("error\nError: %q\nTimeStamp: %s\n", e.Error.Error(), + e.Timestamp.Format(time.DateTime)) + if e.Diagnostics != "" { + fmt.Printf("Diagnostics: %s\n", e.Diagnostics) + } + printLogs(logbuf) +} + +func printLogs(logbuf io.ReadWriter) { + logs, err := io.ReadAll(logbuf) + if err != nil { + fmt.Printf("Logs not available due to error when reading log buffer: %s", err.Error()) + } else if len(logs) != 0 { + fmt.Println("Detailed logs") + fmt.Print(string(logs)) + } +} diff --git a/cmd/stunnerctl/main.go b/cmd/stunnerctl/main.go index 8c15df44..15632b0e 100644 --- a/cmd/stunnerctl/main.go +++ b/cmd/stunnerctl/main.go @@ -1,26 +1,18 @@ package main import ( - "context" - "encoding/json" "fmt" - "io" - "net/http" - "net/url" "os" - "os/signal" "regexp" "strings" - "syscall" "time" "github.com/pion/logging" "github.com/spf13/cobra" cliopt "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/client-go/util/jsonpath" - "sigs.k8s.io/yaml" - stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/internal/icetester" cdsclient "github.com/l7mp/stunner/pkg/config/client" "github.com/l7mp/stunner/pkg/logger" ) @@ -31,20 +23,31 @@ import ( // get config for stunner/udp-gateway in yaml format: stunnerctl -n stunner get config udp-gateway --output yaml var ( - output string - watch, all, verbose bool - k8sConfigFlags *cliopt.ConfigFlags - cdsConfigFlags *cdsclient.CDSConfigFlags - authConfigFlags *cdsclient.AuthConfigFlags - podConfigFlags *cdsclient.PodConfigFlags - loggerFactory *logger.LeveledLoggerFactory - log logging.LeveledLogger + output, iceTesterImage, loglevel string + watch, all, verbose, forceCleanup bool + k8sConfigFlags *cliopt.ConfigFlags + cdsConfigFlags *cdsclient.CDSConfigFlags + authConfigFlags *cdsclient.AuthConfigFlags + podConfigFlags *cdsclient.PodConfigFlags + iceTesterTimeout time.Duration + iceTesterPacketRate int + + loggerFactory logger.LoggerFactory + log logging.LeveledLogger rootCmd = &cobra.Command{ Use: "stunnerctl", Short: "A command line utility to inspect STUNner dataplane .", Long: "The stunnerctl tool is a CLI for inspecting, watching and troublehssooting STUNner gateways", DisableAutoGenTag: true, + PersistentPreRun: func(_ *cobra.Command, _ []string) { + if verbose { + loglevel = "all:TRACE" + } + + loggerFactory = logger.NewLoggerFactory(loglevel) + log = loggerFactory.NewLogger("stunnerctl") + }, } ) @@ -52,7 +55,7 @@ var ( configCmd = &cobra.Command{ Use: "config", Aliases: []string{"stunner-config"}, - Short: "Get or watch dataplane configs", + Short: "Get or watch dataplane configs of a gateway", Args: cobra.RangeArgs(0, 1), DisableAutoGenTag: true, Run: func(cmd *cobra.Command, args []string) { @@ -63,9 +66,9 @@ var ( }, } statusCmd = &cobra.Command{ - Use: "status", + Use: "status [gateway]", Aliases: []string{"dataplane-status"}, - Short: "Read status from dataplane pods", + Short: "Read status from dataplane pods for a gateway", Args: cobra.RangeArgs(0, 1), DisableAutoGenTag: true, Run: func(cmd *cobra.Command, args []string) { @@ -78,7 +81,7 @@ var ( authCmd = &cobra.Command{ Use: "auth", Aliases: []string{"get-credential"}, - Short: "Obtain authenticaction credentials to a gateway", + Short: "Obtain authenticaction credentials for a gateway", Args: cobra.RangeArgs(0, 1), DisableAutoGenTag: true, Run: func(cmd *cobra.Command, args []string) { @@ -88,12 +91,24 @@ var ( } }, } + iceTestCmd = &cobra.Command{ + Use: "icetest [udp, tcp, ...]", + Short: "Test ICE connectivity with the specified transports", + DisableAutoGenTag: true, + Run: func(cmd *cobra.Command, args []string) { + if err := runICETest(cmd, args); err != nil { + fmt.Println(err) + os.Exit(1) + } + }, + } ) func init() { rootCmd.PersistentFlags().BoolVarP(&all, "all-namespaces", "a", false, "Consider all namespaces") rootCmd.PersistentFlags().StringVarP(&output, "output", "o", "summary", "Output format, either json, yaml, summary or jsonpath=template") - rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG") + rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Enable verbose logging, identical to -l all:DEBUG (overrides -l)") + rootCmd.PersistentFlags().StringVarP(&loglevel, "loglevel", "l", "all:WARN", "Set loglevel (format: :, overrides: PION_LOG_*, default: all:WARN)") // Kubernetes config flags: persistent, all commands k8sConfigFlags = cliopt.NewConfigFlags(true) @@ -114,10 +129,24 @@ func init() { authConfigFlags = cdsclient.NewAuthConfigFlags() authConfigFlags.AddFlags(authCmd.Flags()) + // ICE test: uses CDS and auth args + cdsConfigFlags.AddFlags(iceTestCmd.Flags()) + authConfigFlags.AddFlags(iceTestCmd.Flags()) + + // ICE test timeout + iceTestCmd.Flags().IntVarP(&iceTesterPacketRate, "packet-rate", "r", 50, + "Packet rate [pkts/sec], 0 means flood test (Default: 50)") + iceTestCmd.Flags().DurationVarP(&iceTesterTimeout, "timeout", "t", icetester.DefaultICETesterTimeout, + "Timeout") + iceTestCmd.Flags().StringVar(&iceTesterImage, "ice-tester-image", icetester.DefaultICETesterImage, + "Default icetester container image") + iceTestCmd.Flags().BoolVar(&forceCleanup, "force-cleanup", false, "Remove tester namespace if it exists") + // Add commands rootCmd.AddCommand(configCmd) rootCmd.AddCommand(statusCmd) rootCmd.AddCommand(authCmd) + rootCmd.AddCommand(iceTestCmd) } func main() { @@ -127,291 +156,6 @@ func main() { } } -func runConfig(_ *cobra.Command, args []string) error { - loglevel := "all:WARN" - if verbose { - loglevel = "all:TRACE" - } - loggerFactory = logger.NewLoggerFactory(loglevel) - log = loggerFactory.NewLogger("stunnerctl") - - gwNs := "default" - if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { - gwNs = *k8sConfigFlags.Namespace - } - - jsonQuery, output, err := ParseJSONPathFlag(output) - if err != nil { - return err - } - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - log.Debug("Searching for CDS server") - pod, err := cdsclient.DiscoverK8sCDSServer(ctx, k8sConfigFlags, cdsConfigFlags, - loggerFactory.NewLogger("cds-fwd")) - if err != nil { - return fmt.Errorf("error searching for CDS server: %w", err) - } - - log.Debugf("Connecting to CDS server: %s", pod.String()) - var cds cdsclient.CdsApi - cdslog := loggerFactory.NewLogger("cds-client") - if all { - cds, err = cdsclient.NewAllConfigsAPI(pod.Addr, cdslog) - } else if len(args) == 0 { - cds, err = cdsclient.NewConfigsNamespaceAPI(pod.Addr, gwNs, cdslog) - } else { - gwName := args[0] - cds, err = cdsclient.NewConfigNamespaceNameAPI(pod.Addr, gwNs, gwName, cdslog) - } - - if err != nil { - return fmt.Errorf("error creating CDS client: %w", err) - } - - confChan := make(chan *stnrv1.StunnerConfig, 8) - if watch { - err := cds.Watch(ctx, confChan, false) - if err != nil { - close(confChan) - return err - } - - go func() { - sigs := make(chan os.Signal, 1) - signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM) - <-sigs - close(confChan) - }() - } else { - resp, err := cds.Get(ctx) - if err != nil { - close(confChan) - return err - } - for _, c := range resp { - confChan <- c - } - - close(confChan) - } - - for c := range confChan { - if cdsclient.IsConfigDeleted(c) { - fmt.Printf("Gateway: %s \n", c.Admin.Name) - continue - } - switch output { - case "yaml": - if out, err := yaml.Marshal(c); err != nil { - return err - } else { - fmt.Println(string(out)) - } - case "json": - if out, err := json.Marshal(c); err != nil { - return err - } else { - fmt.Println(string(out)) - } - case "jsonpath": - values, err := jsonQuery.FindResults(c) - if err != nil { - return err - } - - if len(values) == 0 || len(values[0]) == 0 { - fmt.Println("") - } - - for arrIx := range values { - for valIx := range values[arrIx] { - fmt.Printf("%v\n", values[arrIx][valIx].Interface()) - } - } - case "summary": - fmt.Print(c.Summary()) - default: - fmt.Println(c.String()) - } - } - - return nil -} - -func runStatus(_ *cobra.Command, args []string) error { - loglevel := "all:WARN" - if verbose { - loglevel = "all:TRACE" - } - loggerFactory = logger.NewLoggerFactory(loglevel) - log = loggerFactory.NewLogger("stunnerctl") - - jsonQuery, output, err := ParseJSONPathFlag(output) - if err != nil { - return err - } - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - gwNs := "default" - extraLog := "in namespace default" - if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { - gwNs = *k8sConfigFlags.Namespace - extraLog = fmt.Sprintf("in namespace %s", gwNs) - } - // --all-namespaces overrides -n - if all { - gwNs = "" - extraLog = "in all namespaces" - } - - gw := "" - if len(args) > 0 { - gw = args[0] - } - if gwNs != "" && gw != "" { - extraLog += fmt.Sprintf("for gateway %s", gw) - } - - log.Debug("Searching for dataplane pods " + extraLog) - pods, err := cdsclient.DiscoverK8sStunnerdPods(ctx, k8sConfigFlags, podConfigFlags, - gwNs, gw, loggerFactory.NewLogger("stunnerd-fwd")) - if err != nil { - return fmt.Errorf("error searching for stunnerd pods: %w", err) - } - - for _, pod := range pods { - client := http.Client{ - Timeout: 5 * time.Second, - } - url := fmt.Sprintf("http://%s/status", pod.Addr) - res, err := client.Get(url) - if err != nil { - log.Errorf("Error querying status for stunnerd pod at URL %q on %s: %s", - url, pod.String(), err.Error()) - continue - } - - if res.StatusCode != http.StatusOK { - log.Errorf("Status query failed on %s with HTTP error code %s", - pod.String(), res.Status) - continue - } - - s := stnrv1.StunnerStatus{} - err = json.NewDecoder(res.Body).Decode(&s) - if err != nil { - log.Errorf("Could not decode status response: %s", err.Error()) - continue - } - - switch output { - case "yaml": - if out, err := yaml.Marshal(s); err != nil { - return err - } else { - fmt.Println(string(out)) - } - case "json": - if out, err := json.Marshal(s); err != nil { - return err - } else { - fmt.Println(string(out)) - } - case "jsonpath": - values, err := jsonQuery.FindResults(s) - if err != nil { - return err - } - - if len(values) == 0 || len(values[0]) == 0 { - fmt.Println("") - } - - for arrIx := range values { - for valIx := range values[arrIx] { - fmt.Printf("%v\n", values[arrIx][valIx].Interface()) - } - } - case "summary": - fallthrough - default: - if pod.Proxy { - fmt.Printf("%s/%s:\n\t%s\n", pod.Namespace, pod.Name, s.Summary()) - } else { - fmt.Printf("%s:\n\t%s\n", pod.Addr, s.Summary()) - } - } - } - - return nil -} - -func runAuth(cmd *cobra.Command, args []string) error { - loglevel := "all:WARN" - if verbose { - loglevel = "all:TRACE" - } - loggerFactory = logger.NewLoggerFactory(loglevel) - log = loggerFactory.NewLogger("stunnerctl") - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - log.Debug("Searching for authentication server") - pod, err := cdsclient.DiscoverK8sAuthServer(ctx, k8sConfigFlags, authConfigFlags, - loggerFactory.NewLogger("auth-fwd")) - if err != nil { - return fmt.Errorf("error searching for auth service: %w", err) - } - - u := url.URL{ - Scheme: "http", - Host: pod.Addr, - Path: "/ice", - } - q := u.Query() - q.Set("service", "turn") - u.RawQuery = q.Encode() - - if authConfigFlags.TurnAuth { - // enforce TURN credential format - u.Path = "" - } - - if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { - q := u.Query() - q.Set("namespace", *k8sConfigFlags.Namespace) - if len(args) > 0 { - q.Set("gateway", args[0]) - } - u.RawQuery = q.Encode() - } - - log.Debugf("Querying to authentication server %s using URL %q", pod.String(), u.String()) - res, err := http.Get(u.String()) - if err != nil { - return fmt.Errorf("error querying auth service %s: %w", pod.String(), err) - } - if res.StatusCode != http.StatusOK { - return fmt.Errorf("HTTP error querying auth service %s: expected status %d, got %d", - pod.String(), http.StatusOK, res.StatusCode) - } - - b, err := io.ReadAll(res.Body) - if err != nil { - return fmt.Errorf("cannot read HTTP response: %w", err) - } - - fmt.Println(string(b)) - - return nil -} - // //////////////////////// var jsonRegexp = regexp.MustCompile(`^\{\.?([^{}]+)\}$|^\.?([^{}]+)$`) diff --git a/cmd/stunnerctl/status.go b/cmd/stunnerctl/status.go new file mode 100644 index 00000000..ce4adc78 --- /dev/null +++ b/cmd/stunnerctl/status.go @@ -0,0 +1,118 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "time" + + "github.com/spf13/cobra" + "sigs.k8s.io/yaml" + + v1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" +) + +func runStatus(_ *cobra.Command, args []string) error { + jsonQuery, output, err := ParseJSONPathFlag(output) + if err != nil { + return err + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + gwNs := "default" + extraLog := "in namespace default" + if k8sConfigFlags.Namespace != nil && *k8sConfigFlags.Namespace != "" { + gwNs = *k8sConfigFlags.Namespace + extraLog = fmt.Sprintf("in namespace %s", gwNs) + } + // --all-namespaces overrides -n + if all { + gwNs = "" + extraLog = "in all namespaces" + } + + gw := "" + if len(args) > 0 { + gw = args[0] + } + if gwNs != "" && gw != "" { + extraLog += fmt.Sprintf("for gateway %s", gw) + } + + log.Debug("Searching for dataplane pods " + extraLog) + pods, err := cdsclient.DiscoverK8sStunnerdPods(ctx, k8sConfigFlags, podConfigFlags, + gwNs, gw, loggerFactory.NewLogger("stunnerd-fwd")) + if err != nil { + return fmt.Errorf("error searching for stunnerd pods: %w", err) + } + + for _, pod := range pods { + client := http.Client{ + Timeout: 5 * time.Second, + } + url := fmt.Sprintf("http://%s/status", pod.Addr) + res, err := client.Get(url) + if err != nil { + log.Errorf("Error querying status for stunnerd pod at URL %q on %s: %s", + url, pod.String(), err.Error()) + continue + } + + if res.StatusCode != http.StatusOK { + log.Errorf("Status query failed on %s with HTTP error code %s", + pod.String(), res.Status) + continue + } + + s := v1.StunnerStatus{} + err = json.NewDecoder(res.Body).Decode(&s) + if err != nil { + log.Errorf("Could not decode status response: %s", err.Error()) + continue + } + + switch output { + case "yaml": + if out, err := yaml.Marshal(s); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "json": + if out, err := json.Marshal(s); err != nil { + return err + } else { + fmt.Println(string(out)) + } + case "jsonpath": + values, err := jsonQuery.FindResults(s) + if err != nil { + return err + } + + if len(values) == 0 || len(values[0]) == 0 { + fmt.Println("") + } + + for arrIx := range values { + for valIx := range values[arrIx] { + fmt.Printf("%v\n", values[arrIx][valIx].Interface()) + } + } + case "summary": + fallthrough + default: + if pod.Proxy { + fmt.Printf("%s/%s:\n\t%s\n", pod.Namespace, pod.Name, s.Summary()) + } else { + fmt.Printf("%s:\n\t%s\n", pod.Addr, s.Summary()) + } + } + } + + return nil +} diff --git a/cmd/turncat/main.go b/cmd/turncat/main.go index 93c8349e..9f305daa 100644 --- a/cmd/turncat/main.go +++ b/cmd/turncat/main.go @@ -34,7 +34,7 @@ var ( cdsConfigFlags *cdsclient.CDSConfigFlags log logging.LeveledLogger defaultDuration time.Duration - loggerFactory *logger.LeveledLoggerFactory + loggerFactory logger.LoggerFactory version = "dev" commitHash = "n/a" diff --git a/go.mod b/go.mod index 295026c6..3b089f2e 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/l7mp/stunner -go 1.22 +go 1.22.0 toolchain go1.22.4 @@ -32,11 +32,12 @@ require ( go.uber.org/zap v1.26.0 golang.org/x/sys v0.26.0 golang.org/x/time v0.5.0 - k8s.io/api v0.29.1 - k8s.io/apimachinery v0.29.1 + gonum.org/v1/gonum v0.15.1 + k8s.io/api v0.31.3 + k8s.io/apimachinery v0.31.3 k8s.io/cli-runtime v0.29.1 - k8s.io/client-go v0.29.1 - k8s.io/utils v0.0.0-20240102154912-e7106e64919e + k8s.io/client-go v0.31.3 + k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 sigs.k8s.io/yaml v1.4.0 ) @@ -45,18 +46,20 @@ require ( github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect - github.com/davecgh/go-spew v1.1.1 // indirect + github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect github.com/emicklei/go-restful/v3 v3.11.2 // indirect github.com/evanphx/json-patch v5.9.0+incompatible // indirect + github.com/fxamacker/cbor/v2 v2.7.0 // indirect github.com/go-errors/errors v1.5.1 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-openapi/jsonpointer v0.20.2 // indirect github.com/go-openapi/jsonreference v0.20.4 // indirect github.com/go-openapi/swag v0.22.9 // indirect github.com/gogo/protobuf v1.3.2 // indirect - github.com/golang/protobuf v1.5.3 // indirect + github.com/golang/protobuf v1.5.4 // indirect github.com/google/btree v1.1.2 // indirect github.com/google/gnostic-models v0.6.8 // indirect + github.com/google/go-cmp v0.6.0 // indirect github.com/google/gofuzz v1.2.0 // indirect github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect @@ -68,7 +71,7 @@ require ( github.com/klauspost/compress v1.17.9 // indirect github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect github.com/mailru/easyjson v0.7.7 // indirect - github.com/moby/spdystream v0.2.0 // indirect + github.com/moby/spdystream v0.4.0 // indirect github.com/moby/term v0.5.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect @@ -89,15 +92,16 @@ require ( github.com/pion/srtp/v3 v3.0.4 // indirect github.com/pion/stun/v3 v3.0.0 // indirect github.com/pkg/errors v0.9.1 // indirect - github.com/pmezard/go-difflib v1.0.0 // indirect + github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/prometheus/client_model v0.6.1 // indirect github.com/prometheus/common v0.60.0 // indirect github.com/prometheus/procfs v0.15.1 // indirect github.com/wlynxg/anet v0.0.3 // indirect + github.com/x448/float16 v0.8.4 // indirect github.com/xlab/treeprint v1.2.0 // indirect go.opentelemetry.io/otel/trace v1.31.0 // indirect go.starlark.net v0.0.0-20240123142251-f86470692795 // indirect - go.uber.org/multierr v1.10.0 // indirect + go.uber.org/multierr v1.11.0 // indirect golang.org/x/crypto v0.28.0 // indirect golang.org/x/mod v0.17.0 // indirect golang.org/x/net v0.29.0 // indirect @@ -111,8 +115,8 @@ require ( gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect - k8s.io/klog/v2 v2.120.1 // indirect - k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec // indirect + k8s.io/klog/v2 v2.130.1 // indirect + k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect sigs.k8s.io/kustomize/api v0.16.0 // indirect sigs.k8s.io/kustomize/kyaml v0.16.0 // indirect diff --git a/go.sum b/go.sum index 343cffa8..f4688bd9 100644 --- a/go.sum +++ b/go.sum @@ -14,8 +14,9 @@ github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46t github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY= github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/deepmap/oapi-codegen/v2 v2.1.0 h1:I/NMVhJCtuvL9x+S2QzZKpSjGi33oDZwPRdemvOZWyQ= github.com/deepmap/oapi-codegen/v2 v2.1.0/go.mod h1:R1wL226vc5VmCNJUvMyYr3hJMm5reyv25j952zAVXZ8= github.com/emicklei/go-restful/v3 v3.11.2 h1:1onLa9DcsMYO9P+CXaL0dStDqQ2EHHXLiz+BtnqkLAU= @@ -24,6 +25,8 @@ github.com/evanphx/json-patch v5.9.0+incompatible h1:fBXyNpNMuTTDdquAq/uisOr2lSh github.com/evanphx/json-patch v5.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= +github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E= +github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ= github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mNrPhs8= github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM= github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk= @@ -41,35 +44,32 @@ github.com/go-openapi/jsonreference v0.20.4 h1:bKlDxQxQJgwpUSgOENiMPzCTBVuc7vTdX github.com/go-openapi/jsonreference v0.20.4/go.mod h1:5pZJyJP2MnYCpoeoMAql78cCHauHj0V9Lhc506VOpw4= github.com/go-openapi/swag v0.22.9 h1:XX2DssF+mQKM2DHsbgZK74y/zj4mo9I99+89xUmuZCE= github.com/go-openapi/swag v0.22.9/go.mod h1:3/OXnFfnMAwBD099SwYRk7GD3xOrr1iL7d/XNLXVVwE= -github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= -github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM= github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= -github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= github.com/google/btree v1.1.2 h1:xf4v41cLI2Z6FxbKm+8Bu+m8ifhj15JuZ9sa0jZCMUU= github.com/google/btree v1.1.2/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= -github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec= -github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM= +github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY= github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ= -github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY= github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY= github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 h1:+ngKgrYPPJrOjhax5N+uePQ0Fh1Z7PheYoUI/0nzkPA= @@ -99,8 +99,8 @@ github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhn github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE= github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= -github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8= -github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= +github.com/moby/spdystream v0.4.0 h1:Vy79D6mHeJJjiPdFEL2yku1kl0chZpJfZcPpb16BRl8= +github.com/moby/spdystream v0.4.0/go.mod h1:xBAYlnt/ay+11ShkdFKNAG7LsyK/tmNBVvVOwrfMgdI= github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0= github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -119,10 +119,10 @@ github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+ github.com/oapi-codegen/runtime v1.1.1 h1:EXLHh0DXIJnWhdRPN2w4MXAzFyE4CskzhNLUmtpMYro= github.com/oapi-codegen/runtime v1.1.1/go.mod h1:SK9X900oXmPWilYR5/WKPzt3Kqxn/uS/+lbpREv+eCg= github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= -github.com/onsi/ginkgo/v2 v2.13.0 h1:0jY9lJquiL8fcf3M4LAXN5aMlS/b2BV86HFFPCPMgE4= -github.com/onsi/ginkgo/v2 v2.13.0/go.mod h1:TE309ZR8s5FsKKpuB1YAQYBzCaAfUgatB/xlT/ETL/o= -github.com/onsi/gomega v1.29.0 h1:KIA/t2t5UBzoirT4H9tsML45GEbo3ouUnBHsCfD2tVg= -github.com/onsi/gomega v1.29.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= +github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA= +github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To= +github.com/onsi/gomega v1.19.0 h1:4ieX6qQjPP/BfC3mpsAtIGGlxTWPeA3Inl/7DtXw1tw= +github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro= github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s= github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw= github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= @@ -161,8 +161,9 @@ github.com/pion/webrtc/v4 v4.0.1 h1:6Unwc6JzoTsjxetcAIoWH81RUM4K5dBc1BbJGcF9WVE= github.com/pion/webrtc/v4 v4.0.1/go.mod h1:SfNn8CcFxR6OUVjLXVslAQ3a3994JhyE3Hw1jAuqEto= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/prometheus/client_golang v1.20.4 h1:Tgh3Yr67PaOv/uTqloMsCEdeuFTatm5zIq5+qNN23vI= github.com/prometheus/client_golang v1.20.4/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE= github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E= @@ -197,6 +198,8 @@ github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4d github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg= github.com/wlynxg/anet v0.0.3 h1:PvR53psxFXstc12jelG6f1Lv4MWqE0tI76/hHGjh9rg= github.com/wlynxg/anet v0.0.3/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA= +github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= +github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ= github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -217,8 +220,8 @@ go.starlark.net v0.0.0-20240123142251-f86470692795 h1:LmbG8Pq7KDGkglKVn8VpZOZj6v go.starlark.net v0.0.0-20240123142251-f86470692795/go.mod h1:LcLNIzVOMp4oV+uusnpk+VU+SzXaJakUuBjoCSWH5dM= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= -go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= -go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= +go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= @@ -226,6 +229,8 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw= golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U= +golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ= +golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 h1:VLliZ0d+/avPrXXH+OakdXhpJuEoBZuwh1m2j7U6Iug= golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -269,13 +274,15 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= -google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +gonum.org/v1/gonum v0.15.1 h1:FNy7N6OUZVUaWG9pTiD+jlhdQ3lMP+/LcTpJ6+a8sQ0= +gonum.org/v1/gonum v0.15.1/go.mod h1:eZTZuRFrzu5pcyjN5wJhcIhnUdNijYxX1T2IcrOGY0o= google.golang.org/protobuf v1.35.1 h1:m3LfL6/Ca+fqnjnlqQXNpFPABW1UD7mjh8KO2mKFytA= google.golang.org/protobuf v1.35.1/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4= +gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= gopkg.in/evanphx/json-patch.v5 v5.9.0 h1:hx1VU2SGj4F8r9b8GUwJLdc8DNO8sy79ZGui0G05GLo= gopkg.in/evanphx/json-patch.v5 v5.9.0/go.mod h1:/kvTRh1TVm5wuM6OkHxqXtE/1nUZZpihg29RtuIyfvk= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= @@ -287,20 +294,20 @@ gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -k8s.io/api v0.29.1 h1:DAjwWX/9YT7NQD4INu49ROJuZAAAP/Ijki48GUPzxqw= -k8s.io/api v0.29.1/go.mod h1:7Kl10vBRUXhnQQI8YR/R327zXC8eJ7887/+Ybta+RoQ= -k8s.io/apimachinery v0.29.1 h1:KY4/E6km/wLBguvCZv8cKTeOwwOBqFNjwJIdMkMbbRc= -k8s.io/apimachinery v0.29.1/go.mod h1:6HVkd1FwxIagpYrHSwJlQqZI3G9LfYWRPAkUvLnXTKU= +k8s.io/api v0.31.3 h1:umzm5o8lFbdN/hIXbrK9oRpOproJO62CV1zqxXrLgk8= +k8s.io/api v0.31.3/go.mod h1:UJrkIp9pnMOI9K2nlL6vwpxRzzEX5sWgn8kGQe92kCE= +k8s.io/apimachinery v0.31.3 h1:6l0WhcYgasZ/wk9ktLq5vLaoXJJr5ts6lkaQzgeYPq4= +k8s.io/apimachinery v0.31.3/go.mod h1:rsPdaZJfTfLsNJSQzNHQvYoTmxhoOEofxtOsF3rtsMo= k8s.io/cli-runtime v0.29.1 h1:By3WVOlEWYfyxhGko0f/IuAOLQcbBSMzwSaDren2JUs= k8s.io/cli-runtime v0.29.1/go.mod h1:vjEY9slFp8j8UoMhV5AlO8uulX9xk6ogfIesHobyBDU= -k8s.io/client-go v0.29.1 h1:19B/+2NGEwnFLzt0uB5kNJnfTsbV8w6TgQRz9l7ti7A= -k8s.io/client-go v0.29.1/go.mod h1:TDG/psL9hdet0TI9mGyHJSgRkW3H9JZk2dNEUS7bRks= -k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw= -k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= -k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec h1:iGTel2aR8vCZdxJDgmbeY0zrlXy9Qcvyw4R2sB4HLrA= -k8s.io/kube-openapi v0.0.0-20240126223410-2919ad4fcfec/go.mod h1:Pa1PvrP7ACSkuX6I7KYomY6cmMA0Tx86waBhDUgoKPw= -k8s.io/utils v0.0.0-20240102154912-e7106e64919e h1:eQ/4ljkx21sObifjzXwlPKpdGLrCfRziVtos3ofG/sQ= -k8s.io/utils v0.0.0-20240102154912-e7106e64919e/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= +k8s.io/client-go v0.31.3 h1:CAlZuM+PH2cm+86LOBemaJI/lQ5linJ6UFxKX/SoG+4= +k8s.io/client-go v0.31.3/go.mod h1:2CgjPUTpv3fE5dNygAr2NcM8nhHzXvxB8KL5gYc3kJs= +k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= +k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= +k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag= +k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98= +k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 h1:pUdcCO1Lk/tbT5ztQWOBi5HBgbBP1J8+AsQnQCKsi8A= +k8s.io/utils v0.0.0-20240711033017-18e509b52bc8/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/kustomize/api v0.16.0 h1:/zAR4FOQDCkgSDmVzV2uiFbuy9bhu3jEzthrHCuvm1g= diff --git a/internal/icetester/artifacts.go b/internal/icetester/artifacts.go new file mode 100644 index 00000000..be9c9eb2 --- /dev/null +++ b/internal/icetester/artifacts.go @@ -0,0 +1,398 @@ +package icetester + +import ( + "context" + "fmt" + "strings" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + + v1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +const ( + icetesterDataplaneName = "icetester-dataplane" +) + +func newICETesterNamespace(name string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": map[string]any{ + "name": name, + }, + }, + } +} + +func newICETesterGatewayClass(namespace string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "gateway.networking.k8s.io/v1", + "kind": "GatewayClass", + "metadata": map[string]any{ + "name": "icetest-gatewayclass", + }, + "spec": map[string]any{ + "controllerName": "stunner.l7mp.io/gateway-operator", + "parametersRef": map[string]any{ + "group": "stunner.l7mp.io", + "kind": "GatewayConfig", + "name": "icetest-gatewayconfig", + "namespace": namespace, + }, + "description": "STUNner is a WebRTC ingress gateway for Kubernetes", + }, + }, + } +} + +func newICETesterGatewayConfig(namespace string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "stunner.l7mp.io/v1", + "kind": "GatewayConfig", + "metadata": map[string]any{ + "name": "icetest-gatewayconfig", + "namespace": namespace, + }, + "spec": map[string]any{ + "realm": "icetest.l7mp.io", + "authType": "ephemeral", + "sharedSecret": "icetest-secret", + "dataplane": icetesterDataplaneName, + "logLevel": "all:INFO", + }, + }, + } +} + +func newICETesterUDPGateway(namespace string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "gateway.networking.k8s.io/v1", + "kind": "Gateway", + "metadata": map[string]any{ + "name": "icetest-udp-gateway", + "namespace": namespace, + }, + "spec": map[string]any{ + "gatewayClassName": "icetest-gatewayclass", + "listeners": []any{ + map[string]any{ + "name": "icetest-udp-listener", + "port": int64(3478), + "protocol": "TURN-UDP", + }, + }, + }, + }, + } +} + +func newICETesterTCPGateway(namespace string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "gateway.networking.k8s.io/v1", + "kind": "Gateway", + "metadata": map[string]any{ + "name": "icetest-tcp-gateway", + "namespace": namespace, + }, + "spec": map[string]any{ + "gatewayClassName": "icetest-gatewayclass", + "listeners": []any{ + map[string]any{ + "name": "icetest-tcp-listener", + "port": int64(3478), + "protocol": "TURN-TCP", + }, + }, + }, + }, + } +} + +func newICETesterUDPRoute(namespace string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "stunner.l7mp.io/v1", + "kind": "UDPRoute", + "metadata": map[string]any{ + "name": "icetest-route", + "namespace": namespace, + }, + "spec": map[string]any{ + "parentRefs": []any{ + map[string]any{ + "name": "icetest-udp-gateway", + }, + map[string]any{ + "name": "icetest-tcp-gateway", + }, + }, + "rules": []any{ + map[string]any{ + "backendRefs": []any{ + map[string]any{ + "name": "icetest-backend", + }, + map[string]any{ + "name": "icetest-udp-gateway", + }, + map[string]any{ + "name": "icetest-tcp-gateway", + }, + }, + }, + }, + }, + }, + } +} + +func newICETesterBackendPod(namespace, iceTesterImage string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "/v1", + "kind": "Pod", + "metadata": map[string]any{ + "name": "icetest-backend", + "namespace": namespace, + "labels": map[string]any{ + "app": "icetester", + }, + }, + "spec": map[string]any{ + "containers": []any{ + map[string]any{ + "name": "icetester", + "image": iceTesterImage, + "command": []any{"icetester"}, + "args": []any{"-l", "all:DEBUG"}, + }, + }, + }, + }, + } +} + +func newICETesterBackendService(namespace string) *unstructured.Unstructured { + return &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "v1", + "kind": "Service", + "metadata": map[string]any{ + "name": "icetest-backend", + "namespace": namespace, + "labels": map[string]any{ + "app": "icetester", + }, + }, + "spec": map[string]any{ + "ports": []any{ + map[string]any{ + "name": "whip-port", + "port": int64(v1.DefaultICETesterPort), + "protocol": "TCP", + }, + }, + "selector": map[string]any{ + "app": "icetester", + }, + }, + }, + } +} + +func newICETesterICETesterResources(ns, iceTesterImage string) []*unstructured.Unstructured { + return []*unstructured.Unstructured{ + newICETesterGatewayClass(ns), + newICETesterGatewayConfig(ns), + newICETesterUDPGateway(ns), + newICETesterTCPGateway(ns), + newICETesterUDPRoute(ns), + newICETesterBackendPod(ns, iceTesterImage), + newICETesterBackendService(ns), + } +} + +func (t *iceTester) makeDataplane(ctx context.Context) (*unstructured.Unstructured, error) { + obj := &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "stunner.l7mp.io/v1", + "kind": "Dataplane", + "metadata": map[string]any{ + "name": "default", + }, + }, + } + d, err := t.get(ctx, obj, metav1.GetOptions{}) + if err != nil { + return nil, fmt.Errorf("Failed to query default Dataplane: %w", err) + } + + // customize + d.SetName(icetesterDataplaneName) + d.SetResourceVersion("") + for _, s := range []struct { + path []string + value any + }{ + {path: []string{"spec", "terminationGracePeriodSeconds"}, value: int64(0)}, + {path: []string{"spec", "resources", "requests", "cpu"}, value: "250m"}, + {path: []string{"spec", "resources", "requests", "memory"}, value: "256Mi"}, + {path: []string{"spec", "resources", "limits", "cpu"}, value: "250m"}, + {path: []string{"spec", "resources", "limits", "memory"}, value: "256Mi"}, + } { + if err := unstructured.SetNestedField(d.Object, s.value, s.path...); err != nil { + return nil, fmt.Errorf("Failed to set field <%s> to %s: %w", + strings.Join(s.path, "."), s.value, err) + } + } + + if err := t.createOrUpdate(ctx, d, metav1.GetOptions{}, metav1.CreateOptions{}, metav1.UpdateOptions{}); err != nil { + return nil, fmt.Errorf("Failed to create/update tester Dataplane: %w", err) + } + + return d, nil +} + +// k8s client funcs +func (t *iceTester) create(ctx context.Context, obj *unstructured.Unstructured, opts metav1.CreateOptions) error { + gvr, err := getGVR(t.k8sConfig, obj) + if err != nil { + return err + } + + if obj.GetNamespace() != "" { + // For namespaced resources + _, err = t.Resource(gvr).Namespace(obj.GetNamespace()).Create(ctx, obj, opts) + } else { + // For cluster-scoped resources + _, err = t.Resource(gvr).Create(ctx, obj, opts) + } + + return err +} + +func (t *iceTester) get(ctx context.Context, obj *unstructured.Unstructured, opts metav1.GetOptions) (*unstructured.Unstructured, error) { + gvr, err := getGVR(t.k8sConfig, obj) + if err != nil { + return nil, err + } + + var getObj *unstructured.Unstructured + if obj.GetNamespace() != "" { + // For namespaced resources + getObj, err = t.Resource(gvr).Namespace(obj.GetNamespace()).Get(ctx, obj.GetName(), opts) + } else { + // For cluster-scoped resources + getObj, err = t.Resource(gvr).Get(ctx, obj.GetName(), opts) + } + + return getObj, err +} + +func (t *iceTester) getCRD(ctx context.Context, name string, opts metav1.GetOptions) (*unstructured.Unstructured, error) { + gvr := schema.GroupVersionResource{ + Group: "apiextensions.k8s.io", + Version: "v1", + Resource: "customresourcedefinitions", + } + + return t.Resource(gvr).Get(ctx, name, opts) +} + +func (t *iceTester) list(ctx context.Context, obj *unstructured.Unstructured, opts metav1.ListOptions) ([]*unstructured.Unstructured, error) { + gvr, err := getGVR(t.k8sConfig, obj) + if err != nil { + return nil, err + } + + var listObj *unstructured.UnstructuredList + if obj.GetNamespace() != "" { + // For namespaced resources + listObj, err = t.Resource(gvr).Namespace(obj.GetNamespace()).List(ctx, opts) + } else { + // For cluster-scoped resources + listObj, err = t.Resource(gvr).List(ctx, opts) + } + if err != nil { + return nil, err + } + + var list = make([]*unstructured.Unstructured, len(listObj.Items)) + for i, o := range listObj.Items { + list[i] = &o + } + + return list, err +} + +func (t *iceTester) delete(ctx context.Context, obj *unstructured.Unstructured, opts metav1.DeleteOptions) error { + gvr, err := getGVR(t.k8sConfig, obj) + if err != nil { + return err + } + + if obj.GetNamespace() != "" { + // For namespaced resources + return t.Resource(gvr).Namespace(obj.GetNamespace()).Delete(ctx, obj.GetName(), opts) + } else { + // For cluster-scoped resources + return t.Resource(gvr). + Delete(ctx, obj.GetName(), opts) + } +} + +func (t *iceTester) update(ctx context.Context, obj *unstructured.Unstructured, opts metav1.UpdateOptions) (*unstructured.Unstructured, error) { + gvr, err := getGVR(t.k8sConfig, obj) + if err != nil { + return nil, err + } + + current, err := t.get(ctx, obj, metav1.GetOptions{}) + if err != nil { + return nil, err + } + + // Copy the ResourceVersion to ensure proper update + obj.SetResourceVersion(current.GetResourceVersion()) + + // Perform the update + if obj.GetNamespace() != "" { + return t.Resource(gvr).Namespace(obj.GetNamespace()).Update(ctx, obj, opts) + } + + return t.Resource(gvr).Update(ctx, obj, opts) +} + +func (t *iceTester) createOrUpdate(ctx context.Context, obj *unstructured.Unstructured, gopts metav1.GetOptions, copts metav1.CreateOptions, uopts metav1.UpdateOptions) error { + if _, err := t.get(ctx, obj, gopts); err != nil { + if apierrors.IsNotFound(err) { + return t.create(ctx, obj, copts) + } else { + _, err := t.update(ctx, obj, uopts) + return err + } + } else { + return err + } +} + +func (t *iceTester) safelyRemove(ctx context.Context, obj *unstructured.Unstructured, gopts metav1.GetOptions, dopts metav1.DeleteOptions) error { + _ = t.delete(ctx, obj, dopts) // nolint:errcheck + return eventually(ctx, func(ctx context.Context) (bool, error) { + if _, err := t.get(ctx, obj, gopts); err != nil && apierrors.IsNotFound(err) { + return true, nil + } else { + return false, err + } + }, 30*time.Second, 250*time.Millisecond) +} diff --git a/internal/icetester/events.go b/internal/icetester/events.go new file mode 100644 index 00000000..14b64a25 --- /dev/null +++ b/internal/icetester/events.go @@ -0,0 +1,75 @@ +package icetester + +import "time" + +// ICE test steps: +// 1. Check install: CDS server and API service available, GW resources and icetest-backend created +// 2. Wait until a GW public IP becomes available: emit one event for all configured TURN protocols +// 3. Run ICE test: emit one event per each configured TURN protocol +// 4. Delete K8s resources: Un-apply static manifests + +type EventType int + +const ( + EventInit EventType = iota + EventInstallationComplete + EventGatewayAvailable + EventICEConfigAvailable + EventAsymmetricICETest + EventSymmetricICETest +) + +func (t EventType) String() string { + switch t { + case EventInit: + return "Initializing" + case EventInstallationComplete: + return "Checking installation" + case EventGatewayAvailable: + return "Checking Gateway" + case EventICEConfigAvailable: + return "Obtaining ICE server configuration" + case EventAsymmetricICETest: + return "Running asymmetric ICE test" + case EventSymmetricICETest: + return "Running symmetric ICE test" + default: + return "N/A" + } +} + +type Event struct { + Type EventType + Error error + Diagnostics string + Args map[string]any + Timestamp time.Time + InProgress bool +} + +const ( + // init + diagK8sConfigUnavailable = "The Kubernetes configuration is unavailable. Please check whether kubectl works first." + + diagK8sClientError = "Kubernetes client is dysfunctional or the Kubernetes API server is unreachable. Does kubectl work? Is the Kubernetes context set for the right cluster?" + + diagNamespaceAlreadyExists = "The Kubernetes namespace to be used for the test already exists, most probably due to an earlier unclean exit. The tester refuses to run in such cases in order to avoid interference with existing workload. If you are sure about that the namespace is unused use '--force-cleanup' to remove it before running the test." + + diagFailedToCreateNamespace = "A namespace for running the test could not be created. This either means that the namespace already exists (e.g., if the icetester could not exit cleanly) or you spefified an existing namespace (the tester refuses to run in an existing namespace in order to avoid interfering with the resources existing there), or the current Kubernetes user does not have enough rigts to create a namespace. Does 'kubectl create namespace my-namespace' work?" + + // install + diagFailedToQueryOrCreateArtifacts = "Some Kubernetes resources needed for running the tests could not be queried or created. Typically, this occurs because the Gateway API custom resources or STUNNer's own custom resources have not been installed, or the current Kubernetes user does not have enough rigts to get or create the resource, or some other error occurred." + + diagCDSServerUnavailable = "The STUNner gateway operator is not installed or the installation is incomplete. Is the gateway operator pod running? It is usually called 'stunner-gateway-operator-controller-manager-XXX' in the 'stunner-system', or in the namespace you installed STUNner. What is the pod status?" + + diagAuthServiceUnavailable = "The STUNner auth service is not installed or the installation is incomplete. Is the STUNner authentication server pod running? Search for the service called 'stunner-auth' in the 'stunner-system', or in the namespace you installed STUNner. What is the pod status of the pod called 'stunner-auth-XXX' in the same namespace?" + + diagICETesterBackendUnavailable = "The ICE tester backend is not inserted into the Kubernetes cluster or it is dysfunctional. This is usually a bug in the 'stunnerctl', please file an issue." + + diagCDSServerConnectionFailed = "The STUNner gateway operator is installed but it is dysfunctional. This often occurs because the Gateway API CRDs are missing or are of the wrong version and thus the operator fails to start, or the operator does not have enough RBAC permissions to access the Kubernetes resources it works on." + + diagPublicAddrNotFound = "At least one Gateway could not be exposed on a public IP/port. This is the most typical problem you will see with STUNner: it usually means that the load-balancer integration in your Kubernetes cluster does not work, or, if you are on NodePorts, none of the Kubernetes nodes have a publicly avaalble external IP (look for ExternalIP in your node status)." + + // test + diagICETestFailed = "The ICE test has failed. Check the reported error!" +) diff --git a/internal/icetester/floodtest.go b/internal/icetester/floodtest.go new file mode 100644 index 00000000..bbdee640 --- /dev/null +++ b/internal/icetester/floodtest.go @@ -0,0 +1,136 @@ +package icetester + +import ( + "context" + "encoding/binary" + "net" + "sort" + "sync" + "sync/atomic" + "time" + + "gonum.org/v1/gonum/stat" +) + +const ( + MaxPacketCount = 10000 +) + +type Stats struct { + SendRate float64 // packets per second + LossRate float64 // percentage + MeanLatency float64 // milliseconds + MedianLatency float64 // milliseconds + P95Latency float64 // milliseconds + P99Latency float64 // milliseconds + PacketsSent uint32 + PacketsReceived uint32 + Duration time.Duration +} + +type Packet struct { + SeqNum uint32 + Timestamp int64 +} + +func FloodTest(ctx context.Context, conn net.Conn, interval time.Duration, packetSize int) (*Stats, error) { + // Prepare buffer pool + bufferPool := sync.Pool{ + New: func() interface{} { + return make([]byte, packetSize) + }, + } + + // Stats + received := make(map[uint32]int64) + latencies := make([]float64, 0) + + // Atomic counter for sequence numbers + var currentSeq uint32 + + // Start receiver goroutine + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + buffer := bufferPool.Get().([]byte) + defer bufferPool.Put(buffer) //nolint:staticcheck + + for { + _, err := conn.Read(buffer) + if err != nil { + return + } + + seqNum := binary.BigEndian.Uint32(buffer[0:4]) + timestamp := int64(binary.BigEndian.Uint64(buffer[4:12])) + received[seqNum] = timestamp + latency := float64(time.Now().UnixNano()-timestamp) / float64(time.Millisecond) + latencies = append(latencies, latency) + } + }() + + // Start sender goroutine + wg.Add(1) + var startTime, endTime time.Time + go func() { + defer wg.Done() + startTime = time.Now() + defer func() { endTime = time.Now() }() + + for { + buffer := bufferPool.Get().([]byte) + seq := atomic.AddUint32(¤tSeq, 1) - 1 + + binary.BigEndian.PutUint32(buffer[0:4], seq) + binary.BigEndian.PutUint64(buffer[4:12], uint64(time.Now().UnixNano())) + + // Fill rest of buffer with sequence number for verification + for j := 12; j < packetSize; j++ { + buffer[j] = byte(seq) + } + + _, err := conn.Write(buffer) + if err != nil { + return + } + bufferPool.Put(buffer) //nolint:staticcheck + + if interval != 0 { + select { + case <-time.After(interval): + case <-ctx.Done(): + return + } + } + } + }() + + // Wait for test duration plus a grace period for receiving remaining packets + <-ctx.Done() + conn.Close() // this will stop the goroutines + + wg.Wait() + + // Calculate statistics + packetsSent := atomic.LoadUint32(¤tSeq) + duration := endTime.Sub(startTime) + stats := &Stats{ + PacketsSent: packetsSent, + PacketsReceived: uint32(len(received)), + Duration: duration, + SendRate: float64(packetsSent) / duration.Seconds(), + LossRate: (float64(packetsSent) - float64(len(received))) / float64(packetsSent) * 100, + } + + sort.Float64s(latencies) + if len(latencies) > 0 { + // Convert slice to float64 slice if not already + stats.MeanLatency = stat.Mean(latencies, nil) + stats.MedianLatency = stat.Quantile(0.5, stat.Empirical, latencies, nil) + stats.P95Latency = stat.Quantile(0.95, stat.Empirical, latencies, nil) + stats.P99Latency = stat.Quantile(0.99, stat.Empirical, latencies, nil) + } + + return stats, nil +} diff --git a/internal/icetester/icetester.go b/internal/icetester/icetester.go new file mode 100644 index 00000000..10957b45 --- /dev/null +++ b/internal/icetester/icetester.go @@ -0,0 +1,685 @@ +package icetester + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "net" + "net/http" + "net/url" + "slices" + "time" + + "github.com/pion/ice/v4" + "github.com/pion/logging" + "github.com/pion/webrtc/v4" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + cliopt "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/rest" + + v1 "github.com/l7mp/stunner/pkg/apis/v1" + cdsclient "github.com/l7mp/stunner/pkg/config/client" + "github.com/l7mp/stunner/pkg/logger" + "github.com/l7mp/stunner/pkg/whipconn" +) + +const ( + DefaultICETesterImage = "docker.io/l7mp/icetester:latest" + DefaultICETesterTimeout = 5 * time.Minute + DefaultICETesterPacketRate time.Duration = 0 + + floodTestPacketSize = 100 + floodTestTimeout = 20 * time.Second +) + +var ( + crdCheckList = []string{ + // Gateway API + "gatewayclasses.gateway.networking.k8s.io", + "gateways.gateway.networking.k8s.io", + "tcproutes.gateway.networking.k8s.io", + "udproutes.gateway.networking.k8s.io", + // STUNner + "dataplanes.stunner.l7mp.io", + "gatewayconfigs.stunner.l7mp.io", + "staticservices.stunner.l7mp.io", + "udproutes.stunner.l7mp.io", + } +) + +type ICETestType int + +const ( + ICETestAsymmetric ICETestType = iota + ICETestSymmetric +) + +func (t ICETestType) String() string { + switch t { + case ICETestAsymmetric: + return "Asymmetric" + case ICETestSymmetric: + return "Symmetric" + default: + return "N/A" + } +} + +type Config struct { + EventChannel chan Event + + K8sConfigFlags *cliopt.ConfigFlags + CDSConfigFlags *cdsclient.CDSConfigFlags + AuthConfigFlags *cdsclient.AuthConfigFlags + + Namespace string + TURNTransports []v1.ListenerProtocol + ICETesterImage string + ForceCleanup bool + PacketRate int + + Logger logger.LoggerFactory +} + +type iceTester struct { + k8sConfig *rest.Config + *dynamic.DynamicClient + + eventCh chan Event + + k8sConfigFlags *cliopt.ConfigFlags + cdsConfigFlags *cdsclient.CDSConfigFlags + authConfigFlags *cdsclient.AuthConfigFlags + + namespace string + transports []v1.ListenerProtocol + iceTesterImage string + forceCleanup bool + floodTestSendInterval time.Duration + + logger logger.LoggerFactory + log logging.LeveledLogger +} + +func NewICETester(config Config) (*iceTester, error) { + logr := config.Logger + if logr == nil { + logr = logger.NewLoggerFactory("all:INFO") + } + + image := DefaultICETesterImage + if config.ICETesterImage != "" { + image = config.ICETesterImage + } + + var sendInterval time.Duration + if config.PacketRate == 0 { + sendInterval = 0 + } else { + sendInterval = time.Duration(int64(float64(time.Second) / float64(config.PacketRate))) + } + + tester := &iceTester{ + eventCh: config.EventChannel, + + k8sConfigFlags: config.K8sConfigFlags, + cdsConfigFlags: config.CDSConfigFlags, + authConfigFlags: config.AuthConfigFlags, + + namespace: config.Namespace, + transports: config.TURNTransports, + iceTesterImage: image, + forceCleanup: config.ForceCleanup, + floodTestSendInterval: sendInterval, + + logger: logr, + log: logr.NewLogger("icetester"), + } + + return tester, nil +} + +func (t *iceTester) Start(ctx context.Context) error { + log := t.log + + ///////// EventInit in-progress + t.sendEventInit(EventInit, nil) //nolint:errcheck + + log.Infof("Creating a Kubernetes client") + k8sConfig, err := t.k8sConfigFlags.ToRESTConfig() + if err != nil { + return t.sendEventComplete(EventInit, + fmt.Errorf("Error building a Kubernetes config: %w", err), + diagK8sConfigUnavailable, + nil, + ) + } + t.k8sConfig = k8sConfig + + cs, err := dynamic.NewForConfig(k8sConfig) + if err != nil { + return t.sendEventComplete(EventInit, + fmt.Errorf("Error creating a Kubernetes client: %w", err), + diagK8sClientError, + nil, + ) + } + t.DynamicClient = cs + + // log.Infof("Checking basic connectivity") + // apiGroupList := &unstructured.Unstructured{} + // apiGroupList.SetGroupVersionKind(schema.GroupVersionKind{Group: "apiextensions.k8s.io", Version: "v1", Kind: "APIGroup"}) + // if _, err := t.get(ctx, apiGroupList, metav1.GetOptions{}); err != nil { + // return t.sendEventComplete(EventInit, + // fmt.Errorf("Failed to query API server: %w", err), + // diagK8sClientError, + // nil, + // ) + // } + + log.Infof("Creating a namespace for testing") + ns := newICETesterNamespace(t.namespace) + if _, err := t.get(ctx, ns, metav1.GetOptions{}); err == nil { + if t.forceCleanup { + if err := t.safelyRemove(ctx, ns, metav1.GetOptions{}, metav1.DeleteOptions{}); err != nil { + return t.sendEventComplete(EventInit, + fmt.Errorf("Failed to clean up tester namespace %s (--force-cleanup enforced): %w", + ns.GetName(), err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + } else { + return t.sendEventComplete(EventInit, + fmt.Errorf("Namespace %s already exists, halting test", ns.GetName()), + diagNamespaceAlreadyExists, + nil, + ) + } + } else if !apierrors.IsNotFound(err) { + return t.sendEventComplete(EventInit, + fmt.Errorf("Error querying testing namespace %s: %w", ns.GetName(), err), + diagFailedToCreateNamespace, + nil, + ) + } + + if err := t.create(ctx, ns, metav1.CreateOptions{}); err != nil { + return t.sendEventComplete(EventInit, + fmt.Errorf("Error creating a namespace for running the tests: %w", err), + diagFailedToCreateNamespace, + nil, + ) + } + defer func() { + // do not use ctx: it might have timed out + if err := t.delete(context.TODO(), ns, metav1.DeleteOptions{}); err != nil { + log.Errorf("Error deleting namespace: %s", err.Error()) + } + }() + + log.Infof("Creating custom Dataplane") + d, err := t.makeDataplane(ctx) + if err != nil { + return t.sendEventComplete(EventInit, + fmt.Errorf("Failed to create custom Dataplane: %w", err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + defer func() { + // do not use ctx: it might have timed out + if err := t.delete(context.TODO(), d, metav1.DeleteOptions{}); err != nil { + log.Errorf("Error deleting custom Dataplane: %s", err.Error()) + } + }() + + ///////// EventInit ready + t.sendEventComplete(EventInit, nil, "", nil) //nolint:errcheck + + ///////// EventInstallationComplete in-progress + t.sendEventInit(EventInstallationComplete, nil) //nolint:errcheck + + log.Infof("Querying CRDs") + for _, crd := range crdCheckList { + if _, err := t.getCRD(ctx, crd, metav1.GetOptions{}); err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Failed to query CRD %s: %w", crd, err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + } + + log.Infof("Inserting tester artifacts") + for _, obj := range newICETesterICETesterResources(t.namespace, t.iceTesterImage) { + if err := t.safelyRemove(ctx, obj, metav1.GetOptions{}, metav1.DeleteOptions{}); err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Failed to clean up resource %s/%s of kind %s: %w", + obj.GetNamespace(), obj.GetName(), obj.GetKind(), err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + + if err := t.create(ctx, obj, metav1.CreateOptions{}); err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Failed to create resource %s/%s of kind %s: %w", + obj.GetNamespace(), obj.GetName(), obj.GetKind(), err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + + log.Debugf("Created resource %s/%s of kind %s", obj.GetNamespace(), + obj.GetName(), obj.GetKind()) + } + defer func() { + for _, obj := range newICETesterICETesterResources(t.namespace, t.iceTesterImage) { + // do not use ctx: it might have timed out + if err := t.delete(context.TODO(), obj, metav1.DeleteOptions{}); err != nil { + log.Errorf("Error deleting resource: %s", err.Error()) + } + } + }() + + log.Infof("Checking tester backend") + iceTesterPod := newICETesterBackendPod(t.namespace, t.iceTesterImage) + if err := eventually(ctx, t.podStatusChecker(iceTesterPod, metav1.GetOptions{}), 30*time.Second, 250*time.Millisecond); err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("ICE tester backend %s/%s not running or ready: %w", + iceTesterPod.GetNamespace(), iceTesterPod.GetName(), err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + + whipEndpoint, err := cdsclient.DiscoverK8sPod(ctx, t.k8sConfigFlags, t.namespace, "app=icetester", v1.DefaultICETesterPort, + t.logger.NewLogger("auth-fwd")) + if err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Error searching for ICE tester backend pod: %w", err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + + log.Info("Searching for CDS server") + cdsPod, err := cdsclient.DiscoverK8sCDSServer(ctx, t.k8sConfigFlags, t.cdsConfigFlags, + t.logger.NewLogger("cds-fwd")) + if err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Error searching for CDS server: %w", err), + diagCDSServerUnavailable, + nil, + ) + } + + log.Info("Searching for authentication service") + authPod, err := cdsclient.DiscoverK8sAuthServer(ctx, t.k8sConfigFlags, t.authConfigFlags, + t.logger.NewLogger("auth-fwd")) + if err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Error searching for auth service: %w", err), + diagAuthServiceUnavailable, + nil, + ) + } + + ///////// EventInstallationComplete ready + t.sendEventComplete(EventInstallationComplete, nil, "", nil) //nolint:errcheck + + ///////// EventGatewayAvailable in-progress + t.sendEventInit(EventGatewayAvailable, nil) //nolint:errcheck + + log.Info("Checking dataplane") + for _, proto := range t.transports { + gw := gwFromProto(proto, t.namespace) + + log.Infof("Checing public address for Gateway %s", gw.GetName()) + cds, err := cdsclient.NewConfigNamespaceNameAPI(cdsPod.Addr, t.namespace, gw.GetName(), + t.logger.NewLogger("cds-client")) + if err != nil { + return t.sendEventComplete(EventGatewayAvailable, + fmt.Errorf("Could not connect to CDS server for obtaning the configuration of Gateway %s: %w", + gw.GetName(), err), + diagCDSServerConnectionFailed, + nil, + ) + } + + if err := eventually(ctx, func(ctx context.Context) (bool, error) { + confs, err := cds.Get(ctx) + if err != nil { + // log.Tracef("Could not get dataplane config: %w", err) + return false, nil + } + + if len(confs) != 1 { + return false, errors.New("Expected exactly one dataplane config") // this should never rhappen + } + + found := false + for _, c := range confs[0].Clusters { + if len(c.Endpoints) != 0 { + found = true + break + } + } + if !found { + // errors.New("No clusters found") + return false, nil // operator not ready yet: retry + + } + + for _, l := range confs[0].Listeners { + if l.PublicAddr == "" || l.PublicPort == 0 { + return false, nil // no public address yet: retry + } + } + + return true, nil + }, 60*time.Second, 250*time.Millisecond); err != nil { + return t.sendEventComplete(EventGatewayAvailable, + fmt.Errorf("Failed to find public address for Gateway %s/%s: %w", t.namespace, gw.GetName(), err), + diagPublicAddrNotFound, + nil, + ) + } + + log.Infof("Checking dataplane pod for Gateway %s/%s", gw.GetNamespace(), gw.GetName()) + // name is unstable, choose by label + opts := metav1.ListOptions{LabelSelector: makeSelector(map[string]string{"app": "my-app", "some-label": "some-value"})} + if err := eventually(ctx, t.podListStatusChecker(opts), 60*time.Second, 250*time.Millisecond); err != nil { + return t.sendEventComplete(EventGatewayAvailable, + fmt.Errorf("Dataplane pod for Gateway %s/%s not running or ready: %w", + iceTesterPod.GetNamespace(), iceTesterPod.GetName(), err), + diagFailedToQueryOrCreateArtifacts, + nil, + ) + } + } + + ///////// EventGatewayAvailable ready + t.sendEventComplete(EventGatewayAvailable, nil, "", nil) //nolint:errcheck + + ///////// EventICEConfigAvailable in-progress + t.sendEventInit(EventICEConfigAvailable, nil) //nolint:errcheck + + iceServers := map[v1.ListenerProtocol]webrtc.Configuration{} + for _, proto := range t.transports { + log.Infof("Testing ICE connection over TURN transport %s", proto.String()) + + gw := gwFromProto(proto, t.namespace) + + log.Infof("Obtaining ICE server config for Gateway %s/%s", t.namespace, gw.GetName()) + if err := eventually(ctx, func(ctx context.Context) (bool, error) { + u := url.URL{ + Scheme: "http", + Host: authPod.Addr, + Path: "/ice", + } + q := u.Query() + q.Set("service", "turn") + q.Set("namespace", t.namespace) + q.Set("gateway", gw.GetName()) + u.RawQuery = q.Encode() + + req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil) + if err != nil { + return false, fmt.Errorf("Error preparing auth service request: %w", err) + } + + res, err := http.DefaultClient.Do(req) + if err != nil { + return false, fmt.Errorf("Failed to query auth service: %w", err) + } + + if res.StatusCode != http.StatusOK { + return false, nil + // return false, fmt.Errorf("Wrong HTTP status querying auth service: %d", res.StatusCode) // this may return false for a while + } + + var iceconf struct { + ICEServers []webrtc.ICEServer + ICETransportPolicy webrtc.ICETransportPolicy + } + if err := json.NewDecoder(res.Body).Decode(&iceconf); err != nil { // Handle errors + return false, fmt.Errorf("Failed to parse the ICE server config obtained from the auth service: %w", err) + } + + iceServers[proto] = webrtc.Configuration{ + ICEServers: iceconf.ICEServers, + ICETransportPolicy: iceconf.ICETransportPolicy, + } + + return true, nil + }, 10*time.Second, 250*time.Millisecond); err != nil { + return t.sendEventComplete(EventInstallationComplete, + fmt.Errorf("Could not obtain ICE server config for Gateway %s/%s: %w", t.namespace, gw.GetName(), err), + diagCDSServerConnectionFailed, + nil, + ) + } + } + + ///////// EventICEServerConfigAvailable ready + t.sendEventComplete(EventICEConfigAvailable, nil, "", nil) //nolint:errcheck + + for _, iceTestType := range []ICETestType{ICETestAsymmetric, ICETestSymmetric} { + var eventType EventType + switch iceTestType { + case ICETestAsymmetric: + eventType = EventAsymmetricICETest + case ICETestSymmetric: + eventType = EventSymmetricICETest + default: + } + for _, proto := range t.transports { + iceConfig := iceServers[proto] + + ///////// EventICETestComplete in-progress + t.sendEventInit(eventType, map[string]any{"ICETransport": proto.String()}) //nolint:errcheck + + log.Infof("Performing %s ICE test for ICE transport %s", iceTestType.String(), proto.String()) + + listenerConfig := whipconn.Config{} + // in symmetric tests the listener uses the same ICE servers as the dialer + if iceTestType == ICETestSymmetric { + listenerConfig.ICEServers = t.updateICEServerAddr(iceConfig.ICEServers, proto) + listenerConfig.ICETransportPolicy = webrtc.ICETransportPolicyRelay + } + log.Debugf("Setting ICE tester listener config: %#v", listenerConfig) + if err := eventually(ctx, func(ctx context.Context) (bool, error) { + uri := url.URL{ + Scheme: "http", + Host: whipEndpoint.Addr, + Path: "/config", + } + + b, err := json.Marshal(listenerConfig) + if err != nil { + return false, fmt.Errorf("Error preparing config: %w", err) // should never happen + } + + _, err = http.Post(uri.String(), "application/json", bytes.NewReader(b)) + if err != nil { + return false, fmt.Errorf("Failed to POST config: %w", err) // should never happen + } + + // query back + req, err := http.NewRequest(http.MethodGet, uri.String(), nil) + if err != nil { + return false, fmt.Errorf("Error preparing GET request for querying the config: %w", err) + } + req.Header.Add("Content-Type", "application/json") + + res, err := http.DefaultClient.Do(req) + if err != nil { + return false, fmt.Errorf("Failed to GET config: %w", err) + } + + err = json.NewDecoder(res.Body).Decode(&listenerConfig) + if err != nil { + return false, fmt.Errorf("Failed to decode config: %w", err) + } + + return true, nil + }, 10*time.Second, 250*time.Millisecond); err != nil { + return t.sendEventComplete(eventType, + fmt.Errorf("Could not update config on ICE tester backend: %w", err), + diagICETesterBackendUnavailable, + nil, + ) + } + + dialerConfig := listenerConfig + dialerConfig.ICEServers = iceConfig.ICEServers + dialerConfig.ICETransportPolicy = webrtc.ICETransportPolicyRelay + dialerConfig.WHIPEndpoint = whipconn.WhipEndpoint + log.Debugf("Using ICE tester dialer config: %#v", dialerConfig) + + log.Debug("Dialing") + var clientConn net.Conn + if err := eventually(ctx, func(ctx context.Context) (bool, error) { + conn, err := whipconn.NewDialer(dialerConfig, t.logger).DialContext(ctx, whipEndpoint.Addr) + if err != nil { + return false, nil // not fatal, should be retried + } + clientConn = conn + return true, nil + }, 90*time.Second, 1000*time.Millisecond); err != nil { + return t.sendEventComplete(EventAsymmetricICETest, + fmt.Errorf("Could not send WHIP request to ICE tester backend: %w", err), + diagICETestFailed, + map[string]any{"ICETransport": proto.String()}) + } + + localSelected, remoteSelected, err := t.getSelectedICECandidates(clientConn) + if err != nil { + return t.sendEventComplete(eventType, + fmt.Errorf("Failed to find selected ICE candidate pair: %w", err), + diagICETestFailed, + map[string]any{"ICETransport": proto.String()}) + } + localCandidates, remoteCandidates, err := t.getICECandidates(clientConn, localSelected, remoteSelected) + if err != nil { + return t.sendEventComplete(eventType, + fmt.Errorf("Failed to find ICE candidates: %w", err), + diagICETestFailed, + map[string]any{"ICETransport": proto.String()}) + } + + timeout, stop := context.WithTimeout(ctx, floodTestTimeout) + defer stop() // useless + stats, err := FloodTest(timeout, clientConn, t.floodTestSendInterval, floodTestPacketSize) + if err != nil { + return t.sendEventComplete(eventType, + fmt.Errorf("Flood test failed: %w", err), + diagICETestFailed, + map[string]any{"ICETransport": proto.String()}) + } + + // floodtest closes the connection on normal exit, but not on error + clientConn.Close() + + ///////// EventICETestComplete ready + //nolint:errcheck + t.sendEventComplete(eventType, nil, "", map[string]any{ + "ICETransport": proto.String(), + "Stats": stats, + "LocalICECandidates": localCandidates, + "RemoteICECandidates": remoteCandidates, + }) + } + } + + return nil +} + +func (t *iceTester) sendEventInit(typ EventType, args map[string]any) { + t.eventCh <- Event{Type: typ, InProgress: true, Timestamp: time.Now(), Args: args} +} + +func (t *iceTester) sendEventComplete(typ EventType, err error, diag string, args map[string]any) error { + t.eventCh <- Event{Type: typ, Error: err, Timestamp: time.Now(), Diagnostics: diag, Args: args} + return err +} + +func (t *iceTester) getSelectedICECandidates(conn net.Conn) (string, string, error) { + whipconn, ok := conn.(*whipconn.DialerConn) + if !ok { + return "", "", errors.New("failed to cast net.Conn to whipconn") + } + + peerConn := whipconn.GetPeerConnection() + transport := peerConn.SCTP().Transport().ICETransport() + selectedPair, err := transport.GetSelectedCandidatePair() + if err != nil { + return "", "", err + } + + return selectedPair.Local.String(), selectedPair.Remote.String(), nil +} + +// parse from the sdps +type CandidateDesc struct { + Candidate string + Selected bool +} + +func (t *iceTester) getICECandidates(conn net.Conn, localSelected, remoteSelected string) ([]CandidateDesc, []CandidateDesc, error) { + local, remote := []CandidateDesc{}, []CandidateDesc{} + + whipconn, ok := conn.(*whipconn.DialerConn) + if !ok { + return local, remote, errors.New("failed to cast net.Conn to whipconn") + } + + peerConn := whipconn.GetPeerConnection() + + getcands := func(desc *webrtc.SessionDescription, selected string) []CandidateDesc { + ret := []CandidateDesc{} + sdp, err := desc.Unmarshal() + if err != nil { + return ret + } + + for _, m := range sdp.MediaDescriptions { + for _, attr := range m.Attributes { + if attr.IsICECandidate() { + ice, err := ice.UnmarshalCandidate(attr.String()) + if err == nil { + cand := ice.String() + if !ContainsDesc(ret, cand) { // candidates are often dumplicated for some reason + mark := false + if cand == selected { + mark = true + } + ret = append(ret, CandidateDesc{Candidate: cand, Selected: mark}) + } + } + } + } + } + + return ret + } + + local = getcands(peerConn.LocalDescription(), localSelected) + remote = getcands(peerConn.RemoteDescription(), remoteSelected) + + return local, remote, nil +} + +func ContainsDesc(cs []CandidateDesc, c string) bool { + ss := make([]string, len(cs)) + for i, c := range cs { + ss[i] = c.Candidate + } + return slices.Contains(ss, c) +} diff --git a/internal/icetester/utils.go b/internal/icetester/utils.go new file mode 100644 index 00000000..bb271bfe --- /dev/null +++ b/internal/icetester/utils.go @@ -0,0 +1,183 @@ +package icetester + +import ( + "context" + "fmt" + "regexp" + "time" + + "github.com/pion/webrtc/v4" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/discovery" + memory "k8s.io/client-go/discovery/cached" + "k8s.io/client-go/rest" + "k8s.io/client-go/restmapper" + + v1 "github.com/l7mp/stunner/pkg/apis/v1" +) + +var turnURIAddrRegexp = regexp.MustCompile(`:(\d+.\d+.\d+.\d+):`) + +func getGVR(config *rest.Config, obj *unstructured.Unstructured) (schema.GroupVersionResource, error) { + dc, err := discovery.NewDiscoveryClientForConfig(config) + if err != nil { + return schema.GroupVersionResource{}, err + } + + mapper := restmapper.NewDeferredDiscoveryRESTMapper(memory.NewMemCacheClient(dc)) + + gv, err := schema.ParseGroupVersion(obj.GetAPIVersion()) + if err != nil { + return schema.GroupVersionResource{}, err + } + + gvk := gv.WithKind(obj.GetKind()) + mapping, err := mapper.RESTMapping(gvk.GroupKind(), gvk.Version) + if err != nil { + return schema.GroupVersionResource{}, err + } + + return mapping.Resource, nil +} + +func gwFromProto(proto v1.ListenerProtocol, ns string) *unstructured.Unstructured { + switch proto { + case v1.ListenerProtocolTURNUDP, v1.ListenerProtocolUDP: + return newICETesterUDPGateway(ns) + case v1.ListenerProtocolTCP, v1.ListenerProtocolTURNTCP: + return newICETesterTCPGateway(ns) + default: + return nil + } +} + +// updateICEServerAddr modifies the cluster-side ICE server config for symmetric ICe tests so that +// the backend will be configured with the gateway as a TURN server +func (t *iceTester) updateICEServerAddr(ss []webrtc.ICEServer, proto v1.ListenerProtocol) []webrtc.ICEServer { + // use service as the TURN server address + gw := gwFromProto(proto, t.namespace) + ret := []webrtc.ICEServer{} + for _, s := range ss { + urls := []string{} + for _, u := range s.URLs { + urls = append(urls, turnURIAddrRegexp.ReplaceAllString(u, + fmt.Sprintf(":%s.%s.svc.cluster.local:", gw.GetName(), gw.GetNamespace()))) + } + ret = append(ret, webrtc.ICEServer{ + URLs: urls, + Username: s.Username, + Credential: s.Credential, + }) + } + return ret +} + +func makeSelector(matcher map[string]string) string { + labelSelector := metav1.LabelSelector{MatchLabels: matcher} + selector, err := metav1.LabelSelectorAsSelector(&labelSelector) + if err != nil { + panic(err) + } + return selector.String() +} + +// checkers +type checkerFunc func(ctx context.Context) (bool, error) + +func eventually(ctx context.Context, condition checkerFunc, timeout, interval time.Duration) error { + timeoutCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + ticker := time.NewTicker(interval) + defer ticker.Stop() + + for { + ok, err := condition(ctx) + if err != nil { + return err + } + if ok { + return nil + } + + select { + case <-timeoutCtx.Done(): + return timeoutCtx.Err() + case <-ticker.C: + continue + } + } +} + +func (t *iceTester) podStatusChecker(obj *unstructured.Unstructured, opts metav1.GetOptions) checkerFunc { + return func(ctx context.Context) (bool, error) { + pod, err := t.get(ctx, obj, opts) + if err != nil { + if !apierrors.IsNotFound(err) { + return false, err + } + return false, nil //retry + } + + phase, found, err := unstructured.NestedString(pod.Object, "status", "phase") + if err != nil || !found { + return false, nil // retry + } + + if phase != "Running" { + return false, nil // retry + } + + conditions, found, err := unstructured.NestedSlice(pod.Object, "status", "conditions") + if err != nil || !found { + return false, nil // retry + } + + ready := false + for _, c := range conditions { + condition, ok := c.(map[string]any) + if !ok { + continue + } + if condition["type"] == "Ready" && condition["status"] == "True" { + ready = true + break + } + } + + return ready, nil + } +} + +// check status of all pods in the podlist, best used for querying pods by a label-selector +func (t *iceTester) podListStatusChecker(opts metav1.ListOptions) checkerFunc { + return func(ctx context.Context) (bool, error) { + obj := &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "/v1", + "kind": "Pod", + }, + } + + pods, err := t.list(ctx, obj, opts) + if err != nil { + return false, err + } + + for _, p := range pods { + chk := t.podStatusChecker(p, metav1.GetOptions{}) + ok, err := chk(ctx) + if err != nil { + return false, err + } + if !ok { + return false, nil + } + } + + return true, nil + } +} diff --git a/pkg/apis/v1/default.go b/pkg/apis/v1/default.go index 8207d38c..835911b0 100644 --- a/pkg/apis/v1/default.go +++ b/pkg/apis/v1/default.go @@ -22,6 +22,7 @@ const ( DefaultMetricsPort int = 8080 DefaultHealthCheckPort int = 8086 DefaultAuthServicePort int = 8088 + DefaultICETesterPort int = 8089 ) // Label/annotation defaults diff --git a/pkg/config/client/k8s_client.go b/pkg/config/client/k8s_client.go index c18752a1..989ea527 100644 --- a/pkg/config/client/k8s_client.go +++ b/pkg/config/client/k8s_client.go @@ -3,6 +3,7 @@ package client import ( "bytes" "context" + "errors" "fmt" "net/http" "os" @@ -210,7 +211,7 @@ func DiscoverK8sCDSServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, cds return PodInfo{}, fmt.Errorf("too many CDS servers") } - return d.portfwd(ctx, &pods.Items[0], cdsFlags.Port) + return d.PortFwd(ctx, &pods.Items[0], cdsFlags.Port) } // DiscoverK8sStunnerdPods discovers the stunnerd pods in a Kubernetes cluster, opens a @@ -296,7 +297,7 @@ func DiscoverK8sStunnerdPods(ctx context.Context, k8sFlags *cliopt.ConfigFlags, defer wg.Done() pod := pods.Items[j] - p, err := d.portfwd(ctx, &pod, podFlags.Port) + p, err := d.PortFwd(ctx, &pod, podFlags.Port) if err != nil { d.log.Errorf("Failed to create port-forwarder to stunnerd pod %s/%s: %s", pod.GetNamespace(), pod.GetName(), err.Error()) @@ -355,10 +356,33 @@ func DiscoverK8sAuthServer(ctx context.Context, k8sFlags *cliopt.ConfigFlags, au d.log.Infof("Mulitple (%d) authentication service instances found, using the first one", len(pods.Items)) } - return d.portfwd(ctx, &pods.Items[0], authFlags.Port) + return d.PortFwd(ctx, &pods.Items[0], authFlags.Port) } -func (d *PodConnector) portfwd(ctx context.Context, pod *corev1.Pod, port int) (PodInfo, error) { +// DiscoverK8sPod discovers an arbitrary pod. +func DiscoverK8sPod(ctx context.Context, k8sFlags *cliopt.ConfigFlags, namespace, labelSelector string, port int, log logging.LeveledLogger) (PodInfo, error) { + d, err := NewK8sDiscoverer(k8sFlags, log) + if err != nil { + return PodInfo{}, fmt.Errorf("Failed to K8s discovery client: %w", err) + } + + pods, err := d.cs.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{LabelSelector: labelSelector}) + if err != nil { + return PodInfo{}, fmt.Errorf("Failed to query Kubernetes API server: %w", err) + } + + if len(pods.Items) == 0 { + return PodInfo{}, errors.New("No pod found") + } + + if len(pods.Items) > 1 { + d.log.Infof("Mulitple (%d) pods found, using the first one", len(pods.Items)) + } + + return d.PortFwd(ctx, &pods.Items[0], port) +} + +func (d *PodConnector) PortFwd(ctx context.Context, pod *corev1.Pod, port int) (PodInfo, error) { p := PodInfo{ Name: pod.GetName(), Namespace: pod.GetNamespace(), diff --git a/pkg/logger/logger.go b/pkg/logger/logger.go index 9f21f34f..515ef937 100644 --- a/pkg/logger/logger.go +++ b/pkg/logger/logger.go @@ -1,6 +1,7 @@ package logger import ( + "bufio" "bytes" "fmt" "io" @@ -35,6 +36,8 @@ type LoggerFactory interface { SetLevel(levelSpec string) // GetLevel gets the loglevel for the given scope. GetLevel(scope string) string + // SetWriter decorates a logger factory with a writer. + SetWriter(w io.Writer) } // LeveledLoggerFactory defines levels by scopes and creates new LeveledLoggers that can dynamically change their own loglevels. @@ -47,12 +50,14 @@ type LeveledLoggerFactory struct { } // NewLoggerFactory sets up a scoped logger for STUNner. -func NewLoggerFactory(levelSpec string) *LeveledLoggerFactory { +func NewLoggerFactory(levelSpec string) LoggerFactory { logger := LeveledLoggerFactory{} logger.DefaultLogLevel = logging.LogLevelError logger.ScopeLevels = make(map[string]logging.LogLevel) logger.Loggers = make(map[string]*RateLimitedLogger) - logger.Writer = os.Stdout + + // Set writer + logger.SetWriter(os.Stdout) // resets all child loggers logger.SetLevel(levelSpec) @@ -67,6 +72,11 @@ func (f *LeveledLoggerFactory) NewLogger(scope string) logging.LeveledLogger { return logger } +// SetWriter sets the writer underlying the logger. +func (f *LeveledLoggerFactory) SetWriter(w io.Writer) { + f.Writer = w +} + // SetLevel sets the loglevel. func (f *LeveledLoggerFactory) SetLevel(levelSpec string) { f.lock.Lock() @@ -137,9 +147,13 @@ type RateLimitedLoggerFactory struct { // WithRateLimiter decorates a logger factory with a rate-limiter. All loggers emitted by the // factory will be automatically rate-limited. -func (f *LeveledLoggerFactory) WithRateLimiter(limit rate.Limit, burst int) *RateLimitedLoggerFactory { +func NewRateLimitedLoggerFactory(logger LoggerFactory, limit rate.Limit, burst int) *RateLimitedLoggerFactory { + leveledF, ok := logger.(*LeveledLoggerFactory) + if !ok { + return nil // this will blow up + } return &RateLimitedLoggerFactory{ - LeveledLoggerFactory: f, + LeveledLoggerFactory: leveledF, Limit: limit, Burst: burst, } @@ -253,27 +267,51 @@ func (w *RateLimitedWriter) Write(p []byte) (int, error) { // RateLimiter is a token bucket that can be disabled. type RateLimiter struct { *rate.Limiter - EnableRateLimiterd bool + Enabled bool } func NewRateLimiter(r rate.Limit, b int) *RateLimiter { return &RateLimiter{ - Limiter: rate.NewLimiter(r, b), - EnableRateLimiterd: false, + Limiter: rate.NewLimiter(r, b), + Enabled: false, } } func (l *RateLimiter) EnableRateLimiter() { - l.EnableRateLimiterd = true + l.Enabled = true } func (l *RateLimiter) DisableRateLimiter() { - l.EnableRateLimiterd = false + l.Enabled = false } func (l *RateLimiter) Allow() bool { - if !l.EnableRateLimiterd { + if !l.Enabled { return true } return l.Limiter.Allow() } + +// AutoFlushWriter wraps a bufio.Writer and ensures that Flush is called after every Write +// operation. +type AutoFlushWriter struct { + *bufio.Writer +} + +// NewAutoFlushWriter creates a new AutoFlushWriter. +func NewAutoFlushWriter(w io.Writer) *AutoFlushWriter { + return &AutoFlushWriter{ + Writer: bufio.NewWriter(w), + } +} + +// Write writes the data and immediately flushes the buffer +func (w *AutoFlushWriter) Write(p []byte) (n int, err error) { + n, err = w.Writer.Write(p) + if err != nil { + return n, err + } + + err = w.Flush() + return n, err +} diff --git a/pkg/logger/logger_test.go b/pkg/logger/logger_test.go index f7c74c4f..253b0b0a 100644 --- a/pkg/logger/logger_test.go +++ b/pkg/logger/logger_test.go @@ -373,7 +373,7 @@ func TestLogger(t *testing.T) { // t.Logf("-------------- Running test: %s -------------", c.name) // create - loggerFactory := NewLoggerFactory(c.defaultLogLevel) + loggerFactory := NewLoggerFactory(c.defaultLogLevel).(*LeveledLoggerFactory) loggerFactory.Writer = logBuffer logreset() @@ -554,7 +554,7 @@ func TestRateLimitedLogger(t *testing.T) { // t.Logf("-------------- Running test: %s -------------", c.name) // create - loggerFactory := NewLoggerFactory(c.level).WithRateLimiter(c.limit, c.burst) + loggerFactory := NewRateLimitedLoggerFactory(NewLoggerFactory(c.level), c.limit, c.burst) loggerFactory.Writer = logBuffer logreset() diff --git a/pkg/whipconn/config.go b/pkg/whipconn/config.go index 50ffb2ed..d750dc63 100644 --- a/pkg/whipconn/config.go +++ b/pkg/whipconn/config.go @@ -10,13 +10,13 @@ import ( const ( messageSize = 2048 - whipEndpoint = "/whip" + WhipEndpoint = "/whip" ) type Config struct { - ICEServers []webrtc.ICEServer - ICETransportPolicy webrtc.ICETransportPolicy - Token, Endpoint string + ICEServers []webrtc.ICEServer + ICETransportPolicy webrtc.ICETransportPolicy + BearerToken, WHIPEndpoint string } func makeURL(addr, endpoint string) *url.URL { diff --git a/pkg/whipconn/dialer.go b/pkg/whipconn/dialer.go index c447a945..76bea810 100644 --- a/pkg/whipconn/dialer.go +++ b/pkg/whipconn/dialer.go @@ -17,7 +17,7 @@ import ( "github.com/pion/webrtc/v4" ) -var _ net.Conn = &dialerConn{} +var _ net.Conn = &DialerConn{} type Dialer struct { config Config @@ -27,23 +27,31 @@ type Dialer struct { } func NewDialer(config Config, logger logging.LoggerFactory) *Dialer { - e := webrtc.SettingEngine{} + e := webrtc.SettingEngine{LoggerFactory: logger} e.DetachDataChannels() - if config.Endpoint == "" { - config.Endpoint = whipEndpoint + if config.WHIPEndpoint == "" { + config.WHIPEndpoint = WhipEndpoint } return &Dialer{ config: config, - api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), + api: webrtc.NewAPI(webrtc.WithSettingEngine(e), webrtc.WithMediaEngine(&webrtc.MediaEngine{})), logger: logger, log: logger.NewLogger("whip-dialer"), connLog: logger.NewLogger("whip-conn"), } } +func (d *Dialer) WithSettingEngine(e webrtc.SettingEngine) *Dialer { + e.DetachDataChannels() // make sure this is set + d.api = webrtc.NewAPI(webrtc.WithSettingEngine(e), webrtc.WithMediaEngine(&webrtc.MediaEngine{})) + return d +} + func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) { + stopped := false + defer func() { stopped = true }() peerConn, err := d.api.NewPeerConnection(webrtc.Configuration{ ICEServers: d.config.ICEServers, ICETransportPolicy: d.config.ICETransportPolicy, @@ -58,7 +66,7 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) return nil, fmt.Errorf("failed to create DataChannel: %w", err) } - conn := &dialerConn{ + conn := &DialerConn{ dialer: d, addr: addr, peerConn: peerConn, @@ -88,7 +96,11 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { conn.log.Infof("Connection state has changed: %s", p) if p == webrtc.PeerConnectionStateFailed || p == webrtc.PeerConnectionStateClosed { - conn.Close() //nolint + if stopped { + conn.Close() //nolint + } else { + errCh <- fmt.Errorf("ICE connection terminated with state: %s", p.String()) + } } }) @@ -112,18 +124,17 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) d.log.Debugf("ICE gathering complete: %s", peerConn.LocalDescription().SDP) - var sdp = []byte(peerConn.LocalDescription().SDP) - + sdp := []byte(peerConn.LocalDescription().SDP) req, err := http.NewRequestWithContext(ctx, http.MethodPost, - makeURL(addr, d.config.Endpoint).String(), bytes.NewBuffer(sdp)) + makeURL(addr, d.config.WHIPEndpoint).String(), bytes.NewBuffer(sdp)) if err != nil { conn.Close() //nolint return nil, fmt.Errorf("unexpected error building HTTP request: %w", err) } req.Header.Add("Content-Type", "application/sdp") - if d.config.Token != "" { - req.Header.Add("Authorization", "Bearer "+d.config.Token) + if d.config.BearerToken != "" { + req.Header.Add("Authorization", "Bearer "+d.config.BearerToken) } resp, err := http.DefaultClient.Do(req) @@ -177,7 +188,7 @@ func (d *Dialer) DialContext(ctx context.Context, addr string) (net.Conn, error) } } -type dialerConn struct { +type DialerConn struct { dialer *Dialer addr string peerConn *webrtc.PeerConnection @@ -187,7 +198,7 @@ type dialerConn struct { log logging.LeveledLogger } -func (c *dialerConn) Close() error { +func (c *DialerConn) Close() error { if c.closed { return nil } @@ -195,13 +206,13 @@ func (c *dialerConn) Close() error { c.log.Trace("Closing WHIP client connection") - uri := makeURL(c.addr, makeResourceURL(c.dialer.config.Endpoint, c.resourceId)) + uri := makeURL(c.addr, makeResourceURL(c.dialer.config.WHIPEndpoint, c.resourceId)) req, err := http.NewRequest("DELETE", uri.String(), nil) if err != nil { return fmt.Errorf("unexpected error building http request: %w", err) } - if c.dialer.config.Token != "" { - req.Header.Add("Authorization", "Bearer "+c.dialer.config.Token) + if c.dialer.config.BearerToken != "" { + req.Header.Add("Authorization", "Bearer "+c.dialer.config.BearerToken) } if _, err = http.DefaultClient.Do(req); err != nil { @@ -216,22 +227,27 @@ func (c *dialerConn) Close() error { return nil } -func (c *dialerConn) Read(b []byte) (int, error) { +func (c *DialerConn) Read(b []byte) (int, error) { return c.dataConn.Read(b) } -func (c *dialerConn) Write(b []byte) (int, error) { +func (c *DialerConn) Write(b []byte) (int, error) { return c.dataConn.Write(b) } // TODO: implement -func (c *dialerConn) LocalAddr() net.Addr { return nil } -func (c *dialerConn) RemoteAddr() net.Addr { return nil } -func (c *dialerConn) SetDeadline(t time.Time) error { return nil } -func (c *dialerConn) SetReadDeadline(t time.Time) error { return nil } -func (c *dialerConn) SetWriteDeadline(t time.Time) error { return nil } +func (c *DialerConn) LocalAddr() net.Addr { return nil } +func (c *DialerConn) RemoteAddr() net.Addr { return nil } +func (c *DialerConn) SetDeadline(t time.Time) error { return nil } +func (c *DialerConn) SetReadDeadline(t time.Time) error { return nil } +func (c *DialerConn) SetWriteDeadline(t time.Time) error { return nil } // String returns a unique identifier for the connection based on the underlying signaling connection. -func (c *dialerConn) String() string { +func (c *DialerConn) String() string { return c.resourceId } + +// GetPeerConnection returns the PeerConnection underlying the data channnel. Useful for checking ICE candidate status. +func (c *DialerConn) GetPeerConnection() *webrtc.PeerConnection { + return c.peerConn +} diff --git a/pkg/whipconn/listener.go b/pkg/whipconn/listener.go index 86be4447..398253cf 100644 --- a/pkg/whipconn/listener.go +++ b/pkg/whipconn/listener.go @@ -17,7 +17,7 @@ import ( ) var _ net.Listener = &Listener{} -var _ net.Conn = &listenerConn{} +var _ net.Conn = &ListenerConn{} type Listener struct { api *webrtc.API @@ -25,8 +25,8 @@ type Listener struct { addr string server *http.Server errCh chan error - connCh chan *listenerConn - conns map[string]*listenerConn + connCh chan *ListenerConn + conns map[string]*ListenerConn lock sync.Mutex logger logging.LoggerFactory log, connLog logging.LeveledLogger @@ -37,33 +37,34 @@ func NewListener(addr string, config Config, logger logging.LoggerFactory) (*Lis e := webrtc.SettingEngine{} e.DetachDataChannels() - if config.Endpoint == "" { - config.Endpoint = whipEndpoint + if config.WHIPEndpoint == "" { + config.WHIPEndpoint = WhipEndpoint } l := &Listener{ addr: addr, config: config, - api: webrtc.NewAPI(webrtc.WithSettingEngine(e)), + api: webrtc.NewAPI(webrtc.WithSettingEngine(e), webrtc.WithMediaEngine(&webrtc.MediaEngine{})), errCh: make(chan error, 5), - connCh: make(chan *listenerConn, 128), - conns: map[string]*listenerConn{}, + connCh: make(chan *ListenerConn, 128), + conns: map[string]*ListenerConn{}, logger: logger, log: logger.NewLogger("whip-listener"), connLog: logger.NewLogger("whip-conn"), } mux := http.NewServeMux() - mux.HandleFunc("/config", l.configHandler) + mux.HandleFunc("GET /config", l.configGetHandler) + mux.HandleFunc("POST /config", l.configPostHandler) - deletePatternWithoutSlash := fmt.Sprintf("DELETE %s/{resourceId}", config.Endpoint) + deletePatternWithoutSlash := fmt.Sprintf("DELETE %s/{resourceId}", config.WHIPEndpoint) mux.HandleFunc(deletePatternWithoutSlash, l.whipDeleteHandler) - deletePatternWithSlash := fmt.Sprintf("DELETE %s/{resourceId}/{$}", config.Endpoint) + deletePatternWithSlash := fmt.Sprintf("DELETE %s/{resourceId}/{$}", config.WHIPEndpoint) mux.HandleFunc(deletePatternWithSlash, l.whipDeleteHandler) - requestPatternWithoutSlash := fmt.Sprintf("POST %s", config.Endpoint) + requestPatternWithoutSlash := fmt.Sprintf("POST %s", config.WHIPEndpoint) mux.HandleFunc(requestPatternWithoutSlash, l.whipRequestHandler) - requestPatternWithSlash := fmt.Sprintf("POST %s/{$}", config.Endpoint) + requestPatternWithSlash := fmt.Sprintf("POST %s/{$}", config.WHIPEndpoint) mux.HandleFunc(requestPatternWithSlash, l.whipRequestHandler) c, err := net.Listen("tcp", addr) @@ -115,27 +116,64 @@ func (l *Listener) Close() error { return l.server.Close() } -func (l *Listener) configHandler(w http.ResponseWriter, r *http.Request) { - l.log.Tracef("New Config request from client %s", r.RemoteAddr) +func (l *Listener) configGetHandler(w http.ResponseWriter, r *http.Request) { + l.log.Infof("New Config GET request from client %s", r.RemoteAddr) + + if r.Header.Get("Content-Type") != "application/json" { + err := fmt.Errorf("Expected Content-Type:application/json, got %q", r.Header.Get("Content-Type")) + l.log.Error(err.Error()) + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + if err := json.NewEncoder(w).Encode(l.config); err != nil { + l.log.Errorf("Failed to encode config %#v for client %s: %s", + l.config, r.RemoteAddr, err.Error()) + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } +} + +// Note: is is unsafe to update the whip endpoint without restarting the listener +func (l *Listener) configPostHandler(w http.ResponseWriter, r *http.Request) { + l.log.Infof("New Config POST request from client %s", r.RemoteAddr) + + if r.Header.Get("Content-Type") != "application/json" { + err := fmt.Errorf("Expected Content-Type:application/json, got %q", r.Header.Get("Content-Type")) + l.log.Error(err.Error()) + http.Error(w, err.Error(), http.StatusBadRequest) + return + } var config Config if err := json.NewDecoder(r.Body).Decode(&config); err != nil { - l.log.Errorf("Failed to decode config request %w from client %s: %s", - r.RemoteAddr, config, err.Error()) + l.log.Errorf("Failed to decode config request %#v from client %s: %s", + config, r.RemoteAddr, err.Error()) http.Error(w, err.Error(), http.StatusBadRequest) return } - l.config = config + l.config.ICEServers = config.ICEServers + if config.ICETransportPolicy != l.config.ICETransportPolicy { + l.config.ICETransportPolicy = config.ICETransportPolicy + } + if config.BearerToken != "" { + l.config.BearerToken = config.BearerToken + } + if config.WHIPEndpoint != "" { + l.log.Debugf("Ignoring WHIP endpoint in received config: %s", config.WHIPEndpoint) + } + + l.log.Infof("Using new config: %#v", l.config) } func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { - l.log.Tracef("New WHIP POST request from client %s", r.RemoteAddr) + l.log.Infof("New WHIP POST request from client %s", r.RemoteAddr) // Check bearer token - if l.config.Token != "" { - if token := r.Header.Get("Authorization"); token != "Bearer "+l.config.Token { - err := fmt.Errorf("unauthorized WHIP request from client %s", r.RemoteAddr) + if l.config.BearerToken != "" { + if token := r.Header.Get("Authorization"); token != "Bearer "+l.config.BearerToken { + err := fmt.Errorf("Unauthorized WHIP request from client %s", r.RemoteAddr) l.errCh <- err http.Error(w, err.Error(), http.StatusUnauthorized) return @@ -158,7 +196,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { return } - conn := &listenerConn{ + conn := &ListenerConn{ listener: l, log: l.connLog, } @@ -172,7 +210,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { l.errCh <- fmt.Errorf("failed to create a PeerConnection: %w", err) return } - conn.peerConn = peerConn + conn.PeerConn = peerConn peerConn.OnConnectionStateChange(func(p webrtc.PeerConnectionState) { conn.log.Debugf("PeerConnection state for client %s has changed: %s", r.RemoteAddr, p.String()) @@ -195,7 +233,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { l.errCh <- fmt.Errorf("failed to detach DataChannel: %w", err) return } - conn.dataConn = raw + conn.DataConn = raw conn.started = true l.log.Infof("Creating new connection for client %s", r.RemoteAddr) @@ -235,7 +273,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { // WHIP expects a Location header: the hash of our local SDP resourceId := resourceHash(sdp) - conn.resourceUrl = makeResourceURL(l.config.Endpoint, resourceId) + conn.ResourceUrl = makeResourceURL(l.config.WHIPEndpoint, resourceId) w.Header().Add("Location", resourceId) // WHIP+WHEP expects a HTTP Status Code of 201 @@ -246,7 +284,7 @@ func (l *Listener) whipRequestHandler(w http.ResponseWriter, r *http.Request) { } func (l *Listener) whipDeleteHandler(w http.ResponseWriter, r *http.Request) { - l.log.Tracef("New WHIP DELETE request from client %s for resource id %q", + l.log.Infof("New WHIP DELETE request from client %s for resource id %q", r.RemoteAddr, r.PathValue("resourceId")) resourceId := r.PathValue("resourceId") @@ -279,27 +317,27 @@ func (_ *Listener) Addr() net.Addr { return nil } -func (l *Listener) Conns() []*listenerConn { +func (l *Listener) GetConns() []*ListenerConn { l.lock.Lock() defer l.lock.Unlock() - ret := []*listenerConn{} + ret := []*ListenerConn{} for _, c := range l.conns { ret = append(ret, c) } return ret } -type listenerConn struct { - resourceUrl string +type ListenerConn struct { + ResourceUrl string listener *Listener - peerConn *webrtc.PeerConnection + PeerConn *webrtc.PeerConnection dataChan *webrtc.DataChannel - dataConn datachannel.ReadWriteCloser + DataConn datachannel.ReadWriteCloser started, closed bool log logging.LeveledLogger } -func (c *listenerConn) Close() error { +func (c *ListenerConn) Close() error { c.log.Tracef("Closing WHIP listener connection %s", c.String()) if c.closed { @@ -309,14 +347,14 @@ func (c *listenerConn) Close() error { // Close the datachannel var err error - if c.dataChan.ReadyState() == webrtc.DataChannelStateOpen { - if err = c.dataConn.Close(); err != nil { + if c.dataChan != nil && c.dataChan.ReadyState() == webrtc.DataChannelStateOpen { + if err = c.DataConn.Close(); err != nil { c.log.Debugf("Error closing DataChannel: %s", err.Error()) } } // Close the peer connection - err = c.peerConn.Close() + err = c.PeerConn.Close() if err != nil { c.log.Debugf("Error closing PeerConnection: %s", err.Error()) } @@ -331,22 +369,22 @@ func (c *listenerConn) Close() error { return err } -func (c *listenerConn) Read(b []byte) (int, error) { - return c.dataConn.Read(b) +func (c *ListenerConn) Read(b []byte) (int, error) { + return c.DataConn.Read(b) } -func (c *listenerConn) Write(b []byte) (int, error) { - return c.dataConn.Write(b) +func (c *ListenerConn) Write(b []byte) (int, error) { + return c.DataConn.Write(b) } // TODO: implement -func (c *listenerConn) LocalAddr() net.Addr { return nil } -func (c *listenerConn) RemoteAddr() net.Addr { return nil } -func (c *listenerConn) SetDeadline(t time.Time) error { return nil } -func (c *listenerConn) SetReadDeadline(t time.Time) error { return nil } -func (c *listenerConn) SetWriteDeadline(t time.Time) error { return nil } +func (c *ListenerConn) LocalAddr() net.Addr { return nil } +func (c *ListenerConn) RemoteAddr() net.Addr { return nil } +func (c *ListenerConn) SetDeadline(t time.Time) error { return nil } +func (c *ListenerConn) SetReadDeadline(t time.Time) error { return nil } +func (c *ListenerConn) SetWriteDeadline(t time.Time) error { return nil } // String returns a unique identifier for the connection based on the underlying signaling connection. -func (c *listenerConn) String() string { - return c.resourceUrl +func (c *ListenerConn) String() string { + return c.ResourceUrl } diff --git a/pkg/whipconn/whipconn_test.go b/pkg/whipconn/whipconn_test.go index 170cb534..2befaf4b 100644 --- a/pkg/whipconn/whipconn_test.go +++ b/pkg/whipconn/whipconn_test.go @@ -1,7 +1,9 @@ package whipconn import ( + "bytes" "context" + "encoding/json" "fmt" "net" "net/http" @@ -10,19 +12,20 @@ import ( "time" "github.com/pion/logging" + "github.com/pion/webrtc/v4" "github.com/stretchr/testify/assert" slogger "github.com/l7mp/stunner/pkg/logger" ) var ( - testerLogLevel = "all:WARN" + testerLogLevel = "all:ERROR" // testerLogLevel = "all:TRACE" // testerLogLevel = "all:INFO" addr = "localhost:12345" timeout = 5 * time.Second interval = 50 * time.Millisecond - defaultConfig = Config{Token: "whiptoken"} + defaultConfig = Config{BearerToken: "whiptoken"} logger logging.LoggerFactory = slogger.NewLoggerFactory(testerLogLevel) log logging.LeveledLogger = logger.NewLogger("test") ) @@ -67,13 +70,34 @@ var testerTestCases = []struct { name: "Invalid bearer token refused", tester: func(t *testing.T, ctx context.Context, l *Listener) { log.Debug("Creating dialer") - d := NewDialer(Config{Token: "dummy-token"}, logger) + d := NewDialer(Config{BearerToken: "dummy-token"}, logger) assert.NotNil(t, d) log.Debug("Dialing") _, err := d.DialContext(ctx, addr) assert.Error(t, err) }, + }, { + name: "Empty bearer token accepted", + config: &Config{BearerToken: ""}, + tester: func(t *testing.T, ctx context.Context, l *Listener) { + log.Debug("Creating dialer") + config := defaultConfig + config.BearerToken = "" + d := NewDialer(config, logger) + assert.NotNil(t, d) + + log.Debug("Dialing") + clientConn, err := d.DialContext(ctx, addr) + assert.NoError(t, err) + + log.Debug("Echo test round 1") + echoTest(t, clientConn, "test1") + log.Debug("Echo test round 2") + echoTest(t, clientConn, "test2") + + assert.NoError(t, clientConn.Close(), "client conn close") + }, }, { name: "Closing dialer does not close client connection", tester: func(t *testing.T, serverCtx context.Context, l *Listener) { @@ -133,14 +157,14 @@ var testerTestCases = []struct { assert.Eventually(t, func() bool { return len(l.conns) == 1 }, timeout, interval) log.Debug("Closing server connections") - for _, lConn := range l.Conns() { + for _, lConn := range l.GetConns() { assert.NoError(t, lConn.Close()) } assert.Eventually(t, func() bool { return len(l.conns) == 0 }, timeout, interval) // should close the client conn too - assert.Eventually(t, func() bool { return clientConn.(*dialerConn).closed == true }, timeout, interval) + assert.Eventually(t, func() bool { return clientConn.(*DialerConn).closed == true }, timeout, interval) }, }, { name: "Multiple connections", @@ -187,7 +211,7 @@ var testerTestCases = []struct { uri := fmt.Sprintf("http://%s/whip/dummy-id", addr) req, err := http.NewRequest("DELETE", uri, nil) assert.NoError(t, err) - req.Header.Add("Authorization", "Bearer "+defaultConfig.Token) + req.Header.Add("Authorization", "Bearer "+defaultConfig.BearerToken) r, err := http.DefaultClient.Do(req) assert.NoError(t, err) @@ -195,10 +219,10 @@ var testerTestCases = []struct { }, }, { name: "Connecting with custom path", - config: &Config{Endpoint: "/custompath"}, + config: &Config{WHIPEndpoint: "/custompath"}, tester: func(t *testing.T, ctx context.Context, l *Listener) { log.Debug("Creating dialer") - d := NewDialer(Config{Endpoint: "/custompath"}, logger) + d := NewDialer(Config{WHIPEndpoint: "/custompath"}, logger) assert.NotNil(t, d) log.Debug("Dialing") @@ -212,6 +236,27 @@ var testerTestCases = []struct { assert.NoError(t, clientConn.Close(), "client conn close") }, + }, { + name: "Failed datachannel connection fails Dial", + tester: func(t *testing.T, ctx context.Context, l *Listener) { + log.Debug("Creating dialer") + // empty ICE servers and forcing relay policy will fail the dialer + d := NewDialer(Config{ + ICEServers: []webrtc.ICEServer{}, + ICETransportPolicy: webrtc.ICETransportPolicyRelay, + BearerToken: "whiptoken", + }, logger) + assert.NotNil(t, d) + + // set agggressive timeouts + e := webrtc.SettingEngine{} + e.SetICETimeouts(500*time.Millisecond, 1000*time.Millisecond, time.Second) + d.WithSettingEngine(e) + + log.Debug("Dialing") + _, err := d.DialContext(ctx, addr) + assert.Error(t, err) + }, }, } @@ -267,3 +312,63 @@ func TestTesterConn(t *testing.T) { }) } } + +func TestConfigEndpoint(t *testing.T) { + log.Debug("Creating listener") + l, err := NewListener(addr, defaultConfig, logger) + assert.NoError(t, err) + assert.NotNil(t, l) + + uri := "http://" + addr + "/config" + req, err := http.NewRequest(http.MethodGet, uri, nil) + req.Header.Add("Content-Type", "application/json") + assert.NoError(t, err) + res, err := http.DefaultClient.Do(req) + assert.NoError(t, err) + + config := Config{} + err = json.NewDecoder(res.Body).Decode(&config) + assert.NoError(t, err) + assert.Equal(t, defaultConfig.ICEServers, config.ICEServers) + assert.Equal(t, defaultConfig.ICETransportPolicy, config.ICETransportPolicy) + assert.Equal(t, defaultConfig.BearerToken, config.BearerToken) + assert.Equal(t, WhipEndpoint, config.WHIPEndpoint) + + ss := []webrtc.ICEServer{{ + URLs: []string{"a", "b"}, + Username: "test-user-1", + Credential: "test-passwd-1r", + }, { + URLs: []string{"c", "d"}, + Username: "test-user-2", + Credential: "test-passwd-2", + }} + + config = Config{ + ICEServers: ss, + ICETransportPolicy: webrtc.ICETransportPolicyRelay, + BearerToken: "some-token", + WHIPEndpoint: "will-be-ignored", + } + + b, err := json.Marshal(config) + assert.NoError(t, err) + _, err = http.Post(uri, "application/json", bytes.NewReader(b)) + assert.NoError(t, err) + + req, err = http.NewRequest(http.MethodGet, uri, nil) + req.Header.Add("Content-Type", "application/json") + assert.NoError(t, err) + res, err = http.DefaultClient.Do(req) + assert.NoError(t, err) + + newConfig := Config{} + err = json.NewDecoder(res.Body).Decode(&newConfig) + assert.NoError(t, err) + assert.Equal(t, config.ICEServers, newConfig.ICEServers) + assert.Equal(t, config.ICETransportPolicy, newConfig.ICETransportPolicy) + assert.Equal(t, config.BearerToken, newConfig.BearerToken) + assert.Equal(t, WhipEndpoint, newConfig.WHIPEndpoint) + + l.Close() //nolint +} diff --git a/relay.go b/relay.go index fa9e3036..5986113d 100644 --- a/relay.go +++ b/relay.go @@ -55,12 +55,12 @@ type RelayGen struct { Net transport.Net // Logger is a logger factory we can use to generate per-listener relay loggers. - Logger *logger.LeveledLoggerFactory + Logger logger.LoggerFactory telemetry *telemetry.Telemetry } -func NewRelayGen(l *object.Listener, t *telemetry.Telemetry, logger *logger.LeveledLoggerFactory) *RelayGen { +func NewRelayGen(l *object.Listener, t *telemetry.Telemetry, logger logger.LoggerFactory) *RelayGen { return &RelayGen{ Listener: l, RelayAddress: l.Addr, diff --git a/server.go b/server.go index b3b5613a..e3eda178 100644 --- a/server.go +++ b/server.go @@ -13,6 +13,7 @@ import ( "github.com/l7mp/stunner/internal/telemetry" "github.com/l7mp/stunner/internal/util" stnrv1 "github.com/l7mp/stunner/pkg/apis/v1" + "github.com/l7mp/stunner/pkg/logger" ) // Number of log events per second reported at ERROR, WARN and INFO loglevel (logging at DEBUG and @@ -155,7 +156,7 @@ func (s *Stunner) StartServer(l *object.Listener) error { AuthHandler: s.NewAuthHandler(), PacketConnConfigs: pConns, ListenerConfigs: lConns, - LoggerFactory: s.logger.WithRateLimiter(LogRateLimit, LogBurst), + LoggerFactory: logger.NewRateLimitedLoggerFactory(s.logger, LogRateLimit, LogBurst), }) if err != nil { return fmt.Errorf("cannot set up TURN server for listener %s: %w", diff --git a/stunner.go b/stunner.go index 8e4c3905..47941324 100644 --- a/stunner.go +++ b/stunner.go @@ -30,7 +30,7 @@ type Stunner struct { resolver resolver.DnsResolver udpThreadNum int telemetry *telemetry.Telemetry - logger *logger.LeveledLoggerFactory + logger logger.LoggerFactory log logging.LeveledLogger net transport.Net ready, shutdown bool From cf69799c4c4de6cb9d22a90433f8037952a2ec5e Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 29 Nov 2024 11:57:12 +0100 Subject: [PATCH 238/243] docs(icetester): Fix some typos/thinkos --- cmd/icetester/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cmd/icetester/README.md b/cmd/icetester/README.md index e75fbf69..a43eb208 100644 --- a/cmd/icetester/README.md +++ b/cmd/icetester/README.md @@ -8,24 +8,24 @@ While `icetester` can be used as a standalone too, the intended use is via `stun ## Installation -Install `icetest` using the standard Go toolchain and add it to `$PATH`. +Install `icetester` using the standard Go toolchain and add it to `$PATH`. ```console -go install github.com/l7mp/stunner/cmd/icetest@latest +go install github.com/l7mp/stunner/cmd/icetester@latest ``` Building from source is as easy as it usually gets with Go: ```console cd stunner -go build -o turncat cmd/turncat/main.go +go build -o turncat cmd/icetester/main.go ``` The containerized version is available as `docker.io/l7mp/icester`. ## Usage -Deploy a STUNner gateway and test is via UDP and TCP through `stunnerctl`: +Deploy a STUNner gateway and test is via UDP and TCP through `stunnerctl`: ```console stunnerctl icetest From fa30699f75f3e78714dc7b4d19b9708322029803 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 29 Nov 2024 15:40:42 +0100 Subject: [PATCH 239/243] chore(CI/CD): Build and publish icester images --- .github/workflows/publish-dev.yaml | 43 +++++++++++++++++++++++++++-- .github/workflows/publish.yml | 44 ++++++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 4 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index 53007d11..ffd7275d 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -16,8 +16,8 @@ jobs: name: Run tests uses: l7mp/stunner/.github/workflows/test.yml@main - push_to_registry: - name: Push Docker image to DockerHub + push_stunner_to_registry: + name: Push STUNner image to DockerHub needs: run_tests if: github.repository == 'l7mp/stunner' runs-on: ubuntu-latest @@ -62,3 +62,42 @@ jobs: with: dev: true secrets: inherit + + push_icetester_to_registry: + name: Push icetester image to DockerHub + needs: run_tests + if: github.repository == 'l7mp/stunner' + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Docker meta + id: meta + uses: docker/metadata-action@v5 + with: + images: l7mp/icetester + tags: | + type=raw,value=dev + + - name: Set up QEMU + uses: docker/setup-qemu-action@v3 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKER_USER }} + password: ${{ secrets.DOCKER_TOKEN }} + + - name: Build and Push + uses: docker/build-push-action@v5 + with: + file: Dockerfile.icetester + context: . + platforms: linux/amd64,linux/arm64 + push: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index ff6d0fe1..9820757c 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -10,8 +10,8 @@ jobs: name: Run tests uses: l7mp/stunner/.github/workflows/test.yml@main - push_to_registry: - name: Push Docker image to DockerHub + push_stunner_to_registry: + name: Push STUNner image to DockerHub needs: run_tests runs-on: ubuntu-latest steps: @@ -56,6 +56,46 @@ jobs: dev: false secrets: inherit + push_icetester_to_registry: + name: Push icetester image to DockerHub + needs: run_tests + if: github.repository == 'l7mp/stunner' + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Docker meta + id: meta + uses: docker/metadata-action@v5 + with: + images: l7mp/icetester + tags: | + type=semver,pattern={{version}} + type=raw,value=latest + + - name: Set up QEMU + uses: docker/setup-qemu-action@v3 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKER_USER }} + password: ${{ secrets.DOCKER_TOKEN }} + + - name: Build and Push + uses: docker/build-push-action@v5 + with: + file: Dockerfile.icetester + context: . + platforms: linux/amd64,linux/arm64 + push: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + add_binaries: name: Add binaries to release assets uses: l7mp/stunner/.github/workflows/publish--add-binaries.yml@main From 43cc5568d9f22a95c6fdd395eed8c6a4c32f386c Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 29 Nov 2024 15:41:12 +0100 Subject: [PATCH 240/243] chore(rtd): Update deps --- docs/requirements.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/requirements.txt b/docs/requirements.txt index e6fff250..32fc8dc3 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -51,13 +51,13 @@ mkdocs-autorefs==1.2.0 # mkdocstrings-python mkdocs-get-deps==0.2.0 # via mkdocs -mkdocstrings[python]==0.26.2 +mkdocstrings[python]==0.27.0 # via # -r docs/requirements.in # mkdocstrings-python mkdocstrings-python==1.12.2 # via mkdocstrings -packaging==24.1 +packaging==24.2 # via mkdocs pathspec==0.12.1 # via mkdocs @@ -65,7 +65,7 @@ platformdirs==4.3.6 # via # mkdocs-get-deps # mkdocstrings -pymdown-extensions==10.11.2 +pymdown-extensions==10.12 # via mkdocstrings python-dateutil==2.9.0.post0 # via ghp-import @@ -79,5 +79,5 @@ pyyaml-env-tag==0.1 # via mkdocs six==1.16.0 # via python-dateutil -watchdog==5.0.3 +watchdog==6.0.0 # via mkdocs From 0be4af0bdb5a470483d2ec7439a0b08099794631 Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 29 Nov 2024 15:42:27 +0100 Subject: [PATCH 241/243] chore(CI/CD): Fix task dependencies in publish --- .github/workflows/publish-dev.yaml | 2 +- .github/workflows/publish.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish-dev.yaml b/.github/workflows/publish-dev.yaml index ffd7275d..783d3cd0 100644 --- a/.github/workflows/publish-dev.yaml +++ b/.github/workflows/publish-dev.yaml @@ -57,7 +57,7 @@ jobs: push_chart: name: Push helm charts to the repo if: github.repository == 'l7mp/stunner' - needs: push_to_registry + needs: push_stunner_to_registry uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: dev: true diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 9820757c..d63d42f0 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -50,7 +50,7 @@ jobs: push_chart: name: Push helm charts to the repo - needs: push_to_registry + needs: push_stunner_to_registry uses: l7mp/stunner/.github/workflows/publish--push-charts.yml@main with: dev: false From 4812c10c1a5d85d60d14589ab46efb9351816b1d Mon Sep 17 00:00:00 2001 From: Tamas Levai Date: Fri, 29 Nov 2024 16:05:39 +0100 Subject: [PATCH 242/243] doc(icetester): Make it appear on the readthedocs --- docs/cmd/icetester.md | 42 ++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 43 insertions(+) create mode 100644 docs/cmd/icetester.md diff --git a/docs/cmd/icetester.md b/docs/cmd/icetester.md new file mode 100644 index 00000000..a43eb208 --- /dev/null +++ b/docs/cmd/icetester.md @@ -0,0 +1,42 @@ +# icetester: Universal UDP echo service using WebRTC/ICE + +`icetester` is test server that can be used WebRTC/ICE connectivity. The tester serves a simple +WebSocket/JSON API server that clients can use to create a WebRTC data channel. whatever is +received by `icetester` on the data channel will be echoed back to the client over the data channel. + +While `icetester` can be used as a standalone too, the intended use is via `stunnerctl icetest`. + +## Installation + +Install `icetester` using the standard Go toolchain and add it to `$PATH`. + +```console +go install github.com/l7mp/stunner/cmd/icetester@latest +``` + +Building from source is as easy as it usually gets with Go: + +```console +cd stunner +go build -o turncat cmd/icetester/main.go +``` + +The containerized version is available as `docker.io/l7mp/icester`. + +## Usage + +Deploy a STUNner gateway and test is via UDP and TCP through `stunnerctl`: + +```console +stunnerctl icetest +``` + +## License + +Copyright 2021-2024 by its authors. Some rights reserved. See [AUTHORS](../../AUTHORS). + +MIT License - see [LICENSE](../../LICENSE) for full text. + +## Acknowledgments + +Initial code adopted from [pion/stun](https://github.com/pion/stun) and [pion/turn](https://github.com/pion/turn). diff --git a/mkdocs.yml b/mkdocs.yml index 580c3175..9e401a58 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -37,6 +37,7 @@ nav: - 'stunnerctl': 'cmd/stunnerctl.md' - 'stunnerd': 'cmd/stunnerd.md' - 'turncat': 'cmd/turncat.md' + - 'icetester': 'cmd/icetester.md' plugins: - search - mkdocstrings: From 5d1a7b1c6fc6933ed98292302424da4b316a304c Mon Sep 17 00:00:00 2001 From: Kornel David Date: Sun, 1 Dec 2024 20:15:31 +0100 Subject: [PATCH 243/243] docs: Fix link to outdated api version Signed-off-by: Kornel David --- docs/INSTALL.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index c91b837b..5a1f1f09 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -28,7 +28,7 @@ helm install stunner-gateway-operator stunner/stunner-gateway-operator --create- --namespace=stunner-system ``` -And that's all: you don't need to install the dataplane separately, this is handled automatically by the operator. The `stunnerd` pods created by the operator can be customized using the Dataplane custom resource: you can specify the `stunnerd` container image version, provision resources per each `stunnerd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1alpha1#DataplaneSpec). +And that's all: you don't need to install the dataplane separately, this is handled automatically by the operator. The `stunnerd` pods created by the operator can be customized using the Dataplane custom resource: you can specify the `stunnerd` container image version, provision resources per each `stunnerd` pod, deploy into the host network namespace, etc.; see the documentation [here](https://pkg.go.dev/github.com/l7mp/stunner-gateway-operator/api/v1#DataplaneSpec). ### Development version @@ -85,7 +85,7 @@ The Helm charts let you fine-tune STUNner features, including [compute resources ### Resources requests/limits -it is important to manage the [amount of CPU and memory resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers) available for each `stunnerd` pod. The [default](https://github.com/l7mp/stunner-helm/blob/main/helm/stunner-gateway-operator/values.yaml) resource request and limit is set as follows: +it is important to manage the [amount of CPU and memory resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers) available for each `stunnerd` pod. The [default](https://github.com/l7mp/stunner-helm/blob/main/helm/stunner-gateway-operator/values.yaml) resource request and limit is set as follows: ```yaml resources: @@ -113,7 +113,6 @@ udpMultithreading: ### Graceful shutdown -STUNner has full support for [graceful shutdown](SCALING.md). This means that `stunner` pods will remain alive as long as there are active allocations in the embedded TURN server, and a pod will automatically remove itself once all allocations are deleted or time out. This enables the full support for graceful scale-down: the user can scale the number of `stunner` instances up and down and no harm should be made to active client connections meanwhile. +STUNner has full support for [graceful shutdown](SCALING.md). This means that `stunner` pods will remain alive as long as there are active allocations in the embedded TURN server, and a pod will automatically remove itself once all allocations are deleted or time out. This enables the full support for graceful scale-down: the user can scale the number of `stunner` instances up and down and no harm should be made to active client connections meanwhile. The default termination period is set to 3600 seconds (1 hour). To modify, use the `--set stunner.deployment.container.terminationGracePeriodSeconds=` flag. -