diff --git a/README.md b/README.md index 222bf7c..e03b824 100644 --- a/README.md +++ b/README.md @@ -76,6 +76,7 @@ See [Getting Started](./docs/getting-started.md). - Architecture - [Overview](./docs/architecture/overview.md) - Manage - - [Configure](./docs/configure.md) - - [Kubernetes](./docs/deploy/kubernetes.md) - - [Observability](./docs/deploy/observability.md) + - [Overview](./docs/manage/overview.md) + - [Configure](./docs/manage/configure.md) + - [Kubernetes](./docs/manage/kubernetes.md) + - [Observability](./docs/manage/observability.md) diff --git a/docs/configure.md b/docs/configure.md index 6b71c9f..50ab045 100644 --- a/docs/configure.md +++ b/docs/configure.md @@ -16,11 +16,12 @@ in the loaded YAML configuration. This will replace references to `${VAR}` and `$VAR` with the corresponding environment variable. If the environment variable is not defined, it will be replaced with an empty -string. You can also defined a default value using form `${VAR:default}`. +string. You can also define a default value using form `${VAR:default}`. ## Server -The Pico server is run using `pico server`. It has the following configuration: +A Pico server node is run using `pico server`. It has the following +configuration: ``` proxy: # The host/port to listen for incoming proxy HTTP requests. diff --git a/docs/manage/configure.md b/docs/manage/configure.md new file mode 100644 index 0000000..9dd96b5 --- /dev/null +++ b/docs/manage/configure.md @@ -0,0 +1,323 @@ +# Configure + +## Sources + +Pico server and agent support both YAML configuration and command-line flags. + +The YAML file path can be set using `--config.path`. + +See `pico server -h` and `pico agent -h` for the available configuration +options. + +### Variable Substitution + +When enabling `--config.expand-env`, Pico will expand environment variables +in the loaded YAML configuration. This will replace references to `${VAR}` +and `$VAR` with the corresponding environment variable. + +If the environment variable is not defined, it will be replaced with an empty +string. You can also define a default value using form `${VAR:default}`. + +## Server + +The Pico server is run using `pico server`. It has the following configuration: +``` +proxy: + # The host/port to listen for incoming proxy HTTP requests. + # + # If the host is unspecified it defaults to all listeners, such as + # '--proxy.bind-addr :8000' will listen on '0.0.0.0:8000'. + bind_addr: :8000 + + # Proxy listen address to advertise to other nodes in the cluster. This is the + # address other nodes will used to forward proxy requests. + # + # Such as if the listen address is ':8000', the advertised address may be + # '10.26.104.45:8000' or 'node1.cluster:8000'. + # + # By default, if the bind address includes an IP to bind to that will be used. + # If the bind address does not include an IP (such as ':8000') the nodes + # private IP will be used, such as a bind address of ':8000' may have an + # advertise address of '10.26.104.14:8000'. + advertise_addr: "" + + # The timeout when sending proxied requests to upstream listeners for forwarding + # to other nodes in the cluster. + # + # If the upstream does not respond within the given timeout a + # '504 Gateway Timeout' is returned to the client. + gateway_timeout: 15s + +upstream: + # The host/port to listen for connections from upstream listeners. + # + # If the host is unspecified it defaults to all listeners, such as + # '--upstream.bind-addr :8001' will listen on '0.0.0.0:8001'. + bind_addr: :8001 + + # Upstream listen address to advertise to other nodes in the cluster. + # + # Such as if the listen address is ':8001', the advertised address may be + # '10.26.104.45:8001' or 'node1.cluster:8001'. + # + # By default, if the bind address includes an IP to bind to that will be used. + # If the bind address does not include an IP (such as ':8001') the nodes + # private IP will be used, such as a bind address of ':8001' may have an + # advertise address of '10.16.104.14:8001'. + advertise_addr: "" + +admin: + # The host/port to listen for incoming admin connections. + # + # If the host is unspecified it defaults to all listeners, such as + # '--admin.bind-addr :8002' will listen on '0.0.0.0:8002'. + bind_addr: :8002 + + # Admin listen address to advertise to other nodes in the cluster. This is the + # address other nodes will used to forward admin requests. + # + # Such as if the listen address is ':8002', the advertised address may be + # '10.26.104.45:8002' or 'node1.cluster:8002'. + # + # By default, if the bind address includes an IP to bind to that will be used. + # If the bind address does not include an IP (such as ':8002') the nodes + # private IP will be used, such as a bind address of ':8002' may have an + # advertise address of '10.26.104.14:8002'. + advertise_addr: "" + +gossip: + # The host/port to listen for inter-node gossip traffic. + # + # If the host is unspecified it defaults to all listeners, such as + # '--gossip.bind-addr :8003' will listen on '0.0.0.0:8003'. + bind_addr: :8003 + + # Gossip listen address to advertise to other nodes in the cluster. This is the + # address other nodes will used to gossip with the node. + # + # Such as if the listen address is ':8003', the advertised address may be + # '10.26.104.45:8003' or 'node1.cluster:8003'. + # + # By default, if the bind address includes an IP to bind to that will be used. + # If the bind address does not include an IP (such as ':8003') the nodes + # private IP will be used, such as a bind address of ':8003' may have an + # advertise address of '10.26.104.14:8003'. + advertise_addr: "" + +cluster: + # A unique identifier for the node in the cluster. + # + # By default a random ID will be generated for the node. + node_id: "" + + # A prefix for the node ID. + # + # Pico will generate a unique random identifier for the node and append it to + # the given prefix. + # + # Such as you could use the node or pod name as a prefix, then add a unique + # identifier to ensure the node ID is unique across restarts. + node_id_prefix: "" + + # A list of addresses of members in the cluster to join. + # + # This may be either addresses of specific nodes, such as + # '--cluster.join 10.26.104.14,10.26.104.75', or a domain that resolves to + # the addresses of the nodes in the cluster (e.g. a Kubernetes headless + # service), such as '--cluster.join pico.prod-pico-ns'. + # + # Each address must include the host, and may optionally include a port. If no + # port is given, the gossip port of this node is used. + # + # Note each node propagates membership information to the other known nodes, + # so the initial set of configured members only needs to be a subset of nodes. + join: [] + + # Whether the server node should abort if it is configured with more than one + # node to join (excluding itself) but fails to join any members. + abort_if_join_fails: true + +auth: + # Secret key to authenticate HMAC endpoint connection JWTs. + token_hmac_secret_key: "" + + # Public key to authenticate RSA endpoint connection JWTs. + token_rsa_public_key: "" + + # Public key to authenticate ECDSA endpoint connection JWTs. + token_ecdsa_public_key: "" + + # Audience of endpoint connection JWT token to verify. + # + # If given the JWT 'aud' claim must match the given audience. Otherwise it + # is ignored. + token_audience: "" + + # Issuer of endpoint connection JWT token to verify. + # + # If given the JWT 'iss' claim must match the given issuer. Otherwise it + # is ignored. + token_issuer: "" + +server: + # Maximum duration after a shutdown signal is received (SIGTERM or + # SIGINT) to gracefully shutdown the server node before terminating. + # This includes handling in-progress HTTP requests, gracefully closing + # connections to upstream listeners, announcing to the cluster the node is + # leaving... + graceful_shutdown_timeout: 1m + +log: + # Minimum log level to output. + # + # The available levels are 'debug', 'info', 'warn' and 'error'. + level: info + + # Each log has a 'subsystem' field where the log occured. + # + # '--log.subsystems' enables all log levels for those given subsystems. This + # can be useful to debug a particular subsystem without having to enable all + # debug logs. + # + # Such as you can enable 'gossip' logs with '--log.subsystems gossip'. + subsystems: [] +``` + +### Cluster + +To deploy Pico as a cluster, configure `--cluster.join` to a list of cluster +members in the cluster to join. + +The addresses may be either addresses of specific nodes, such as +`10.26.104.14`, or a domain name that resolves to the IP addresses of all nodes +in the cluster. + +To deploy Pico to Kubernetes, you can create a headless service whose domain +resolves to the IP addresses of the pods in the service, such as +`pico.prod-pico-ns.svc.cluster.local`. When Pico starts, it will then attempt +to join the other pods. + +### Authentication + +To authenticate upstream endpoint connections, Pico can use a +[JSON Web Token (JWT)](https://jwt.io/) provided by your application. + +You configure Pico with the secret key or public key to verify the JWT, then +configure the upstream endpoint with the JWT. Pico will then verify endpoints +connection token. + +Pico supports HMAC, RSA, and ECDSA JWT algorithms, specifically HS256, HS384, +HS512, RSA256, RSA384, RSA512, EC256, EC384, and EC512. + +The server has the following configuration options to pass a secret key or +public key: +- `auth.token_hmac_secret_key`: Add HMAC secret key +- `auth.token_rsa_public_key`: Add RSA public key +- `auth.token_ecdsa_public_key`: Add ECDSA public key + +If no keys secret or public keys are given, Pico will allow unauthenticated +endpoint connections. + +Pico will verify the `exp` (expiry) and `iat` (issued at) claims if given, and +drop the connection to the upstream endpoint once its token expires. + +By default Pico will not verify the `aud` (audience) or `iss` (issuer) claims, +though you can enable these checks with `auth.token_audience` and +`auth.token_issuer` respectively. + +You may also include Pico specific fields in your JWT. Pico supports the +`pico.endpoints` claim which contains an array of endpoint IDs the token is +permitted to register. Such as if the JWT includes claim +`"pico": {"endpoints": ["endpoint-123"]}`, it will be permitted to register +endpoint ID `endpoint-123` but not `endpoint-xyz`. + +Note Pico does (yet) not authenticate proxy requests as proxy clients will +typically be deployed to the same network as the Pcio server. Your upstream +services may then authenticate incoming requests if needed after they've been +forwarded by Pico. + +## Agent + +The Pico agent is run using `pico agent`. It has the following configuration: +``` +# The endpoints to register with the Pico server. +# +# Each endpoint has an ID and forwarding address. The agent will register the +# endpoint with the Pico server then receive incoming requests for that endpoint +# and forward them to the configured address. +# +# '--endpoints' is a comma separated list of endpoints with format: +# '/'. Such as '--endpoints 6ae6db60/localhost:3000' +# will register the endpoint '6ae6db60' then forward incoming requests to +# 'localhost:3000'. +# +# You may register multiple endpoints which have their own connection to Pico, +# such as '--endpoints 6ae6db60/localhost:3000,941c3c2e/localhost:4000'. +# +# (Required). +endpoints: [] + +server: + # Pico server URL. + # + # The listener will add path /pico/v1/listener/:endpoint_id to the given URL, + # so if you include a path it will be used as a prefix. + # + # Note Pico connects to the server with WebSockets, so will replace http/https + # with ws/wss (you can configure either). + url: http://localhost:8001 + + # Heartbeat interval. + # + # To verify the connection to the server is ok, the listener sends a + # heartbeat to the upstream at the '--server.heartbeat-interval' + # interval, with a timeout of '--server.heartbeat-timeout'.`, + heartbeat_interval: 10s + + # Heartbeat timeout. + # + # To verify the connection to the server is ok, the listener sends a + # heartbeat to the upstream at the '--server.heartbeat-interval' + heartbeat_timeout: 10s + +auth: + # An API key to authenticate the connection to Pico. + api_key: "" + +forwarder: + # Forwarder timeout. + # + # This is the timeout between a listener receiving a request from Pico then + # forwarding it to the configured forward address, and receiving a response. + # + # If the upstream does not respond within the given timeout a + # '504 Gateway Timeout' is returned to the client. + timeout: 10s + +admin: + # The host/port to listen for incoming admin connections. + # + # If the host is unspecified it defaults to all listeners, such as + # '--admin.bind-addr :9000' will listen on '0.0.0.0:9000' + bind_addr: :9000 + +log: + # Minimum log level to output. + # + # The available levels are 'debug', 'info', 'warn' and 'error'. + level: info + + # Each log has a 'subsystem' field where the log occured. + # + # '--log.subsystems' enables all log levels for those given subsystems. This + # can be useful to debug a particular subsystem without having to enable all + # debug logs. + # + # Such as you can enable 'gossip' logs with '--log.subsystems gossip'. + subsystems: [] +``` + +### Authentication + +To authenticate the agent, include a JWT in `auth.api_key`. The supported JWT +formats are described above in the server configuration. diff --git a/docs/manage/kubernetes.md b/docs/manage/kubernetes.md index cbb9c17..080d30d 100644 --- a/docs/manage/kubernetes.md +++ b/docs/manage/kubernetes.md @@ -6,27 +6,126 @@ joining. ## Discovery -Pico is configured to join a set of known nodes in a cluster using -`--cluster.join`. Rather than maintain a static list of IP addresses, the -easiest option is to create a headless service for Pico. This will create a -DNS record that resolves to the addresses of each Pico pod. +To deploy Pico as a cluster, configure `--cluster.join` to a list of cluster +members in the cluster to join. -You can then configure `--cluster.join` with this DNS record, such as -`pico.pico-ns.svc.cluster.local`, then when Pico starts it will attempt to -join any existing nodes. +The addresses may be either addresses of specific nodes, such as +`10.26.104.14`, or a domain name that resolves to the IP addresses of all nodes +in the cluster. -## Ports +On Kubernetes, its best to create a headless service for Pico that resolves to +the IP addresses of each Pico pod in the cluster. -The proxy port accepts connections from proxy clients. It only -supports HTTP and defaults to port `8000`. +You can then configure `--cluster.join` with the service name, `pico`. -The upstream port accepts connections from upstream listeners via -WebSockets, so if you route using a HTTP load balancer or gateway, you must -ensure WebSockets are supported/enabled. It defaults to port `8001`. +## Example -The admin port accepts admin connections to inspect the status of the server. -This includes Prometheus metrics at `/metrics` and a status API at `/status` -which is used by the `pico status` CLI. It defaults to port `8002`. +This example creates a Pico cluster with 3 replicas using a StatefulSet. -Finally the gossip port is used for inter-node traffic to propagate the cluster -state which defaults to port `8003`. +First create a headless service which is used for cluster discovery: +``` +apiVersion: v1 +kind: Service +metadata: + name: pico + labels: + app: pico +spec: + ports: + - port: 8000 + name: proxy + - port: 8001 + name: upstream + - port: 8002 + name: admin + - port: 8003 + name: gossip + clusterIP: None + selector: + app: pico +``` + +Next create a YAML config map. To make debugging easier, this uses the pod name +as the Pico node ID prefix. Note the pod name must not be used as the node ID +itself since each restart requires a new node ID. This also configures cluster +discovery to use the service created above: + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: server-config +data: + server.yaml: | + cluster: + node_id_prefix: ${POD_NAME}- + join: + - pico +``` + +Finally create a StatefulSet with three replicas: + +``` +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: pico +spec: + selector: + matchLabels: + app: pico + serviceName: "pico" + replicas: 3 + template: + metadata: + labels: + app: pico + spec: + terminationGracePeriodSeconds: 60 + containers: + - name: pico + image: my-repo/pico:latest + ports: + - containerPort: 8000 + name: proxy + - containerPort: 8001 + name: upstream + - containerPort: 8002 + name: admin + - containerPort: 8003 + name: gossip + args: + - server + - --config.path + - /config/server.yaml + - --config.expand-env + resources: + limits: + cpu: 250m + ephemeral-storage: 1Gi + memory: 1Gi + requests: + cpu: 250m + ephemeral-storage: 1Gi + memory: 1Gi + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + volumeMounts: + - name: config + mountPath: "/config" + readOnly: true + volumes: + - name: config + configMap: + name: server-config + items: + - key: "server.yaml" + path: "server.yaml" +``` + +You can then setup the any required load balancers (such as a Kubernetes +Gatweay) or services to route requests to the server. +to Pico. diff --git a/docs/manage/observability.md b/docs/manage/observability.md index c45bf7f..bdfc2af 100644 --- a/docs/manage/observability.md +++ b/docs/manage/observability.md @@ -32,7 +32,7 @@ list. Such as `rpc` will match `rpc` but not `rpc.stream`. The Pico server exposes Prometheus on the admin port at `/metrics`. Pico also includes a number of Grafana dashboards at -[../../monitoring/dashboards](monitoring/dashboards). +[monitoring/dashboards](../../monitoring/dashboards). ## Status Pico includes a status CLI to inspect a Pico server. Servers register endpoints @@ -42,6 +42,6 @@ Such as to view the endpoints registers on a server use `pico status proxy endpoints`. Or to inspect the set of known nodes in the cluster use `pico status cluster nodes`. -Configure the server URL with `--server.url`. You can also forward to a -particular node ID using `--forward` (which can be useful when all nodes are -behind a load balancer). +Configure the server URL with `--server.url`. You can also forward the request + to a particular node ID using `--forward` (which can be useful when all nodes +are behind a load balancer). diff --git a/docs/manage/overview.md b/docs/manage/overview.md new file mode 100644 index 0000000..69e42c0 --- /dev/null +++ b/docs/manage/overview.md @@ -0,0 +1,96 @@ +# Overview + +This document provides an overview on setting up and managing a Pico +deployment. + +Also see [Architecture Overview](../architecture/overview.md). + +## Server +The Pico server is responsible for accepting outbound-only connections from +upstream services, then routing requests from proxy clients to the appropriate +upstream. + +The server is designed to be hosted as a cluster of nodes behind a HTTP load +balancer. + +Run the a server node using `pico server`. + +### Port + +The Pico server exposes 4 ports: +* Proxy port: Receives HTTP(S) requests from proxy clients which are routed to +an upstream service +* Upstream port: Accepts connections from upstream services +* Admin port: Exposes metrics and a status API to inspect the server state +* Gossip port: Used for inter-node gossip traffic + +The proxy port and upstream port are kept separate to support different access +to each port. Such as if you're using Pico to access external customer +networks, the upstream port may be exposed to the Internet for upstreams to +connect, but you may only allow proxy requests from clients in the same network +as Pico. Similarly the admin port should not be exposed to the Internet. + +The proxy, upstream and admin ports are all designed to be hosted behind a HTTP +load balancer. The upstream port uses WebSockets so you must ensure your load +balancer is configured correctly. + +### Cluster + +To deploy Pico as a cluster, configure `--cluster.join` to list the addresses +of existing cluster members to join. + +The addresses may be either addresses of specific nodes, such as +`10.26.104.14`, or a domain name that resolves to the IP addresses of all nodes +in the cluster. + +To deploy Pico to Kubernetes, you can create a headless service whose domain +resolves to the IP addresses of the pods in the service, such as +`pico.prod-pico-ns.svc.cluster.local`. When Pico starts, it will then attempt +to join the other pods in the cluster. See [Kubernetes](./kubernetes.md) for +details on hosting Pico on Kubernetes. + +See [Configure](./configure.md) for details. + +### Observability + +Each server node has an admin port (`8003` by default) which includes +Prometheus metrics at `/metrics`, a health endpoint at `/health`, and a status +API at `/status`. The status API exposes endpoints for inspecting the status of +a server node, which is used by the `pico status` CLI. + +See [Observability](./observability.md) for details. + +## Upstreams + +Upstream services open outbound-only connections to Pico and register an +endpoint ID. The connection is the ‘tunnel’ to Pico and is the only transport +that's used between Pico and the upstream. + +To add an upstream service, use the Pico agent. The agent is a lightweight +proxy that runs alongside your services, that opens a connection to Pico, +registers the configured endpoints, then forwards incoming requests to your +service. + +Such as you may configure the agent to register the endpoint `my-endpoint` then +forward requests to `localhost:4000` using +`pico agent –endpoints my-endpoint/localhost:4000`. + +See [Configure](./configure.md) for details. + +## Authentication + +Pico supports authenticating upstream services using a +[JSON Web Token (JWT)](https://jwt.io/) provided by your application. + +Pico supports HMAC, RSA, and ECDSA JWT algorithms, specifically HS256, HS384, +HS512, RSA256, RSA384, RSA512, EC256, EC384, and EC512. + +You configure the Pico server with the secret key (HMAC) or public key (RSA and +EC), then Pico will verify upstream connections using a valid JWT. Then +configure the Pico agent with a JWT API key which it will use when connecting +to Pico. + +The JWT can also include Pico specific claims, such as a list of endpoints the +upstream service is permitted to register. + +See [Configure](./configure.md) for details.