Sync with upstream (#10)

* add a custom interface for the resolver instead of forcing *net.Resolver (stripe#187) * feature/add prometheus metrics (stripe#179) * STORY-25143 - Add prometheus metrics to smokescreen * STORY-25143 - Cleanup * STORY-25143 - Fix tests to compare new metric labels * STORY-25143 - Host prometheus endpoint on separate port * STORY-25143 - Use value provided via command line flag * STORY-25143 - Add prometheus timing metrics * STORY-25143 - Fix nil map assignment and prometheus metric name sanitisation * STORY-25143 - Cleanup comments * STORY-25143 - Remove some repetition + add further unit testing * STORY-25143 - Document new prometheus features in README + add port flag to prometheus config * STORY-25143 - Make PR requested changes: * Don't export metrics list * Follow project sytlistic choices * STORY-25143 - Rename only one receiver * STORY-25143 - Add new `--expose-prometheus-metrics` flag to CLI to toggle exposing prometheus metrics * Small cleanup of timer metrics * Fix go module vendoring * Use ElementsMatch to ignore order * Just use require * Move the custom request handler call after the main acl check * Use local server instead of httpbin (stripe#192) * Do not return a denyError for DNS resolution failures (stripe#194) * dont return denial errors for dns resolution failures * fix test * move DNSError check into net.Error assertion, extend test * fix integration test * add AcceptResponseHandler to modify accepted responses (stripe#196) * add AcceptResponseHandler to modify accepted responses * customer->custom * Update docs to clarify global_deny_list (stripe#197) * update docs to clarify global_deny_list behavior * consistent example domain * be more concise * Use AcceptResponseHandler in goproxy https CONNECT hook (stripe#199) * pipe AcceptResponseHandler into new goproxy hook * update comment * go mod vendor * unit test * use smokescreenctx in acceptresponsehandler * fix unit tests * Export SmokescreenContext type (stripe#200) * export SmokescreenContext type * also export AclDecision * ResolvedAddr too * consistent caps * Update pkg/smokescreen/smokescreen.go Co-authored-by: jjiang-stripe <[email protected]> * export Decision --------- Co-authored-by: jjiang-stripe <[email protected]> * generate new test pki (stripe#206) * allow listen address specification for prom (stripe#203) * Bump golang.org/x/net from 0.7.0 to 0.17.0 (stripe#204) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.7.0 to 0.17.0. - [Commits](golang/net@v0.7.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bump go versions (stripe#207) * update dependency * configure addr in smokescreen and add unit test * use fmt * try this workaround * variable name change * Update docs to disambiguate ACL vs --deny-address behavior (stripe#210) * update docs to clarify how IP filtering works * fix fields bug * remove extra field setting * trigger build * Add support for Smokescreen -> HTTPS CONNECT Proxy ACLs (stripe#213) * Introduce CONNECT Proxy URL ACL Support Add gitignore debug changes WIP Basic concept working WIP Cleaned up some things prereview fixed tests Removed extraneous yaml file Add correctly failing test tmp WIP WIP WIP WIP WIP WIP * WIP * WIP * PR feedback 1 * Fixed tests * testing again * WIP * Added extra test * Bump goproxy version to incorporate CONNECT proxy header changes * WIP * Bump google.golang.org/protobuf from 1.28.1 to 1.33.0 (stripe#216) Bumps google.golang.org/protobuf from 1.28.1 to 1.33.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add support for username / password auth in URLs to external CONNECT proxies (stripe#222) * Add support for UN / PW Auth for External CONNECT Proxies * Fixed naming of log line * PR feedback * Debug commit * Removing modifications of vendor-ed code * Removed debug * Removed missed cruft * Fixed bug with env var proxy arg * Add failure kind * update goproxy version to master commit * Ensure proxy passed in X-Upstream-Https-Proxy is parsable * Update Github build workflows (stripe#228) Co-authored-by: Harold Simpson <[email protected]> * Use goveralls parallel build * go get -d github.com/stripe/goproxy@latest && go mod vendor * Add MITM support to Smokescreen * Use MitmTLSConfig in the config instead of MitmCa * PR feedback + remove CloseIdleConnections * Refactor allowed_domains_mitm to mitm_domains * Rename ValidateRule * Add Support for Reject Handler with Context * Update comment * Block smokescreen init incase of invalid config * fix: fix slice init length * Remove duplicate validation * Make SmokeScreen Fields Public * Revert Role fixes * Revert Role fixes * Update goproxy version to v0.0.0-20241017101008-e12ef0653f22 (stripe#235) * Adding [allow|deny]_addresses settings to yaml config file * Update goproxy version to v0.0.0-20241022131412-58117846327a (stripe#238) * Ignore goveralls --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: JulesD <[email protected]> Co-authored-by: Josh McConnell <[email protected]> Co-authored-by: Kevin Vincent <[email protected]> Co-authored-by: kevinv-stripe <[email protected]> Co-authored-by: Sergey Rud <[email protected]> Co-authored-by: cmoresco-stripe <[email protected]> Co-authored-by: Craig Shannon <[email protected]> Co-authored-by: jjiang-stripe <[email protected]> Co-authored-by: Timofey Bakunin <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yuxi Xie <[email protected]> Co-authored-by: xieyuxi-stripe <[email protected]> Co-authored-by: Jessica Jiang <[email protected]> Co-authored-by: pspieker-stripe <[email protected]> Co-authored-by: Patrick Spieker <[email protected]> Co-authored-by: Gautham Warrier <[email protected]> Co-authored-by: gauthamw-stripe <[email protected]> Co-authored-by: harold-stripe <[email protected]> Co-authored-by: Harold Simpson <[email protected]> Co-authored-by: Saurabh Bhatia <[email protected]> Co-authored-by: cui fliter <[email protected]> Co-authored-by: Bryan Eastes <[email protected]>
intercom · Jan 27, 2025 · 5ef31cf · 5ef31cf
1 parent 83ed067
commit 5ef31cf
Show file tree

Hide file tree

Showing 637 changed files with 113,497 additions and 5,673 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -1,14 +1,19 @@
 on: [push, pull_request]
 name: Test
+# Workaround for coveralls error "Can't add a job to a build that is already closed"
+# See https://github.com/lemurheavy/coveralls-public/issues/1716
+env:
+  COVERALLS_SERVICE_NUMBER: ${{ github.run_id }}-${{ github.run_attempt }}
+  COVERALLS_PARALLEL: true
 jobs:
   test:
     strategy:
       matrix:
-        go-version: ['1.19', '1.20']
+        go-version: ['1.21', '1.22', '1.23']
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-go@v3
+    - uses: actions/checkout@v4
+    - uses: actions/setup-go@v5
       with:
         go-version: ${{ matrix.go-version }}
     - name: Setup env
@@ -23,13 +28,24 @@ jobs:
         go test -race -v -timeout 2m -failfast -covermode atomic -coverprofile=.covprofile ./... -tags=nointegration
         # Run integration tests hermetically to avoid nondeterministic races on environment variables
         go test -race -v -timeout 2m -failfast ./cmd/... -run TestSmokescreenIntegration
+        go test -race -v -timeout 2m -failfast ./cmd/... -run TestInvalidUpstreamProxyConfiguratedFromEnv
         go test -race -v -timeout 2m -failfast ./cmd/... -run TestInvalidUpstreamProxyConfiguration
         go test -race -v -timeout 2m -failfast ./cmd/... -run TestClientHalfCloseConnection
-    - name: Install goveralls
-      env:
-        GO111MODULE: off
-      run: go get github.com/mattn/goveralls
-    - name: Send coverage
-      env:
-        COVERALLS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      run: goveralls -coverprofile=.covprofile -service=github
+    # - name: Install goveralls
+    #   run: go install github.com/mattn/goveralls@latest
+    # - name: Send coverage
+    #   env:
+    #     COVERALLS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    #   run: goveralls -coverprofile=.covprofile -service=github
+  finish:
+    needs: test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+      # - name: Install goveralls
+      #   run: go install github.com/mattn/goveralls@latest
+      # - name: Close goveralls parallel build
+      #   env:
+      #     COVERALLS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      #   run: goveralls -coverprofile=.covprofile -service=github -parallel-finish=true
diff --git a/.gitignore b/.gitignore
@@ -3,3 +3,4 @@
 *.swp
 *.swo
 *.swn
+*debug.test*
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -0,0 +1,16 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "smokescreen",
+            "type": "go",
+            "request": "launch",
+            "mode": "auto",
+            "program": "./",
+            "args": ["--config-file", "config.yaml", "--egress-acl-file", "acl.yaml"]
+        }
+    ]
+}
diff --git a/Development.md b/Development.md
@@ -0,0 +1,308 @@
+
+# Development and Testing
+
+## Testing
+```bash
+go test ./...
+```
+
+## Running locally
+
+This section describes how to run Smokescreen locally with different scenarios and using `curl` as a client.
+
+- [HTTP Proxy](#http-proxy)
+- [HTTP CONNECT Proxy](#http-connect-proxy)
+- [Monitor metrics Smokescreen emits](#monitor-metrics-smokescreen-emits)
+- [HTTP CONNECT Proxy over TLS](#http-connect-proxy-over-tls)
+- [MITM (Man in the middle) Proxy](#mitm-man-in-the-middle-proxy)
+- [MITM (Man in the middle) Proxy over TLS](#mitm-man-in-the-middle-proxy-over-tls)
+
+### HTTP Proxy
+
+#### Configurations
+
+```yaml
+# config.yaml
+---
+allow_missing_role: true  # skip mTLS client validation (use default ACL)
+```
+
+```yaml
+# acl.yaml
+---
+version: v1
+services: []
+default:
+  name: default
+  project: security
+  action: enforce
+  allowed_domains: 
+    - example.com
+```
+
+#### Run
+
+```bash
+# Run smokescreen (in a different shell)
+go run . --config-file config.yaml --egress-acl-file acl.yaml
+
+# Curl
+curl -x localhost:4750 http://example.com
+# Curl with ALL_PROXY
+ALL_PROXY=localhost:4750 curl -v http://example.com
+```
+
+### HTTP CONNECT Proxy
+
+#### Configurations
+
+```yaml
+# config.yaml
+---
+allow_missing_role: true  # skip mTLS client validation (use default ACL)
+```
+
+```yaml
+# acl.yaml
+---
+version: v1
+services: []
+default:
+  name: default
+  project: security
+  action: enforce
+  allowed_domains: 
+    - api.github.com
+```
+
+#### Run
+
+```bash
+# Run smokescreen (in a different shell)
+go run . --config-file config.yaml --egress-acl-file acl.yaml
+
+# Curl
+curl --proxytunnel -x localhost:4750 https://api.github.com/zen
+# Curl with HTTPS_PROXY
+HTTPS_PROXY=localhost:4750 curl https://api.github.com/zen
+```
+
+### Monitor metrics Smokescreen emits
+
+#### Configurations
+
+```yaml
+# config.yaml
+---
+allow_missing_role: true  # skip mTLS client validation (use default ACL)
+statsd_address: 127.0.0.1:8200
+```
+
+```yaml
+# acl.yaml
+---
+version: v1
+services: []
+default:
+  name: default
+  project: security
+  action: enforce
+  allowed_domains: 
+    - api.github.com
+```
+
+#### Run
+
+```bash
+# Listen to a local port with nc (in a different shell)
+nc -uklv 127.0.0.1 8200
+
+# Run smokescreen (in a different shell)
+go run . --config-file config.yaml --egress-acl-file acl.yaml
+
+# Curl
+curl --proxytunnel -x localhost:4750 https://api.github.com/zen
+# Curl with HTTPS_PROXY
+HTTPS_PROXY=localhost:4750 curl https://api.github.com/zen
+```
+
+### HTTP CONNECT Proxy over TLS
+
+#### Set-up
+
+##### Generate certificates
+```bash
+mkdir -p mtls_setup
+# Private keys for CAs
+openssl genrsa -out mtls_setup/server-ca.key 2048
+openssl genrsa -out mtls_setup/client-ca.key 2048
+
+# Generate client and server CA certificates
+openssl req -new -x509 -nodes -days 1000 -key mtls_setup/server-ca.key -out mtls_setup/server-ca.crt \
+    -subj "/C=AQ/ST=Petrel Island/L=Dumont-d'Urville
+/O=Penguin/OU=Publishing house/CN=server CA"
+
+openssl req -new -x509 -nodes -days 1000 -key mtls_setup/client-ca.key -out mtls_setup/client-ca.crt \
+    -subj "/C=MA/ST=Tarfaya/L=Tarfaya/O=Fennec/OU=Aviator/CN=Client CA"
+
+# Generate a certificate signing request (client CN is localhost which is used by smokescreen as the service name by default)
+openssl req -newkey rsa:2048 -nodes -keyout mtls_setup/server.key -out mtls_setup/server.req \
+    -subj "/C=AQ/ST=Petrel Island/L=Dumont-d'Urville/O=Chionis/OU=Publishing house/CN=server req"
+openssl req -newkey rsa:2048 -nodes -keyout mtls_setup/client.key -out mtls_setup/client.req \
+    -subj "/C=MA/ST=Tarfaya/L=Tarfaya/O=Addax/OU=Writer/CN=localhost"
+
+# Have the CA sign the certificate requests and output the certificates.
+echo "authorityKeyIdentifier=keyid,issuer
+basicConstraints=CA:FALSE
+keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = localhost
+" > mtls_setup/localhost.ext
+
+openssl x509 -req -in mtls_setup/server.req -days 1000 -CA mtls_setup/server-ca.crt -CAkey mtls_setup/server-ca.key -set_serial 01 -out mtls_setup/server.crt -extfile mtls_setup/localhost.ext
+
+openssl x509 -req -in mtls_setup/client.req -days 1000 -CA mtls_setup/client-ca.crt -CAkey mtls_setup/client-ca.key -set_serial 01 -out mtls_setup/client.crt
+```
+
+##### Configurations
+
+```yaml
+# config.yaml
+---
+tls:
+  cert_file: "mtls_setup/server.crt"
+  key_file: "mtls_setup/server.key"
+  client_ca_files:
+    - "mtls_setup/client-ca.crt"
+```
+
+```yaml
+# acl.yaml
+---
+version: v1
+services:
+  - name: localhost
+    project: github
+    action: enforce
+    allowed_domains:
+      - api.github.com
+default:
+  name: default
+  project: security
+  action: enforce
+  allowed_domains: []
+```
+
+#### Run
+
+```bash
+# Run smokescreen (in a different shell)
+go run . --config-file config.yaml --egress-acl-file acl.yaml
+
+# Curl
+curl --proxytunnel -x https://localhost:4750 --proxy-cacert mtls_setup/server-ca.crt --proxy-cert mtls_setup/client.crt --proxy-key mtls_setup/client.key https://api.github.com/zen
+# Curl with HTTPS_PROXY
+HTTPS_PROXY=https://localhost:4750 curl --proxy-cacert mtls_setup/server-ca.crt --proxy-cert mtls_setup/client.crt --proxy-key mtls_setup/client.key https://api.github.com/zen
+```
+
+### MITM (Man in the middle) Proxy
+
+#### Set-up
+
+```yaml
+# config.yaml
+---
+allow_missing_role: true  # skip mTLS client validation (use default ACL)
+# Re-using goproxy library CA and key
+mitm_ca_cert_file: "vendor/github.com/stripe/goproxy/ca.pem"
+mitm_ca_key_file: "vendor/github.com/stripe/goproxy/key.pem"
+```
+
+```yaml
+# acl.yaml
+---
+version: v1
+services: []
+default:
+  name: default
+  project: security
+  action: enforce
+  allowed_domains:
+    - wttr.in
+  mitm_domains:
+  - domain: wttr.in
+    add_headers:
+      Accept-Language: el
+    detailed_http_logs: true
+    detailed_http_logs_full_headers:
+      - User-Agent
+```
+
+#### Run
+
+```bash
+# Run smokescreen (in a different shell)
+go run . --config-file config.yaml --egress-acl-file acl.yaml
+
+# Curl (weather should be in Greek since we set the Accept-Language header)
+curl --proxytunnel -x localhost:4750 --cacert vendor/github.com/stripe/goproxy/ca.pem https://wttr.in
+# Curl with HTTPS_PROXY
+HTTPS_PROXY=localhost:4750 curl --cacert vendor/github.com/stripe/goproxy/ca.pem https://wttr.in
+```
+
+### MITM (Man in the middle) Proxy over TLS
+
+#### Set-up
+
+Please generate the certificates from the TLS Generate certificates section.
+
+```yaml
+# config.yaml
+---
+tls:
+  cert_file: "mtls_setup/server.crt"
+  key_file: "mtls_setup/server.key"
+  client_ca_files:
+    - "mtls_setup/client-ca.crt"
+# Re-using goproxy library CA and key
+mitm_ca_cert_file: "vendor/github.com/stripe/goproxy/ca.pem"
+mitm_ca_key_file: "vendor/github.com/stripe/goproxy/key.pem"
+```
+
+```yaml
+# acl.yaml
+---
+version: v1
+services:
+  - name: localhost
+    project: github
+    action: enforce
+    allowed_domains:
+      - wttr.in
+    mitm_domains:
+      - domain: wttr.in
+        add_headers:
+          Accept-Language: el
+        detailed_http_logs: true
+        detailed_http_logs_full_headers:
+          - User-Agent
+default:
+  name: default
+  project: security
+  action: enforce
+  allowed_domains: []
+```
+
+#### Run
+
+```bash
+# Run smokescreen (in a different shell)
+go run . --config-file config.yaml --egress-acl-file acl.yaml
+
+# Curl (weather should be in Greek since we set the Accept-Language header)
+curl --proxytunnel -x https://localhost:4750 --cacert vendor/github.com/stripe/goproxy/ca.pem --proxy-cacert mtls_setup/server-ca.crt --proxy-cert mtls_setup/client.crt --proxy-key mtls_setup/client.key https://wttr.in
+# Curl with HTTPS_PROXY
+HTTPS_PROXY=https://localhost:4750 curl --cacert vendor/github.com/stripe/goproxy/ca.pem --proxy-cacert mtls_setup/server-ca.crt --proxy-cert mtls_setup/client.crt --proxy-key mtls_setup/client.key https://wttr.in
+```
-Original file line number
+Diff line change
@@ Expand Up / @@ -3,3 +3,4 @@ @@
     *.swp
     *.swo
     *.swn
+    *debug.test*